FCI-Helwan blog

Just another FCI-H weblog

Iterative Development – there is a simpler way to work – Part II (finale)

NOTE: This text (part I and part II) is just an introduction to the existence of an alternative, not a guide to use that alternative.

Because I -personally- believe that an example is worth a thousand words, I will tell you my personal experience with iterative development; the story of our graduation project. We only tried iterative development when other ways failed.

Our graduation project was a clustered server for massive multiplayer online games. The entire period for the project to come out was approximately 10 months, overlapping an academic year.

We’ve spent almost 9 months of them in the design phase; the project turned out to be very big and needed some research and novel techniques. For 5-members team who doesn’t know anything about clustering we needed that much time.

Anyways, having only one month left to hand the project, there was no single line of code. We thought that we already have an exhaustive list of classes and functions and it’s only a matter of implementation.

Trying to implement the classes we designed, we had no idea how to start, and just -exactly- after 4 functions we totally lost contact with the design and didn’t know how to proceed. We were overwhelmed by the complexity and inter-dependencies between modules that it was NOT possible to see anything running before at least thousands of lines of code, which will need even more time to fix (cuz they were not tested).

That seemed inappropriate for the 1-month due time. Our main problem was the inter-dependencies of the modules. That’s where iterative development kicked in. (Note that changes in the server and the game affect each others so they have to be done according to each-other step by step).

You will notice in the following sequence that at any point of time we could
A. Hand something that worked, not complete, but works.
B. Features are added incrementally. That means at the dead line we will have N features working, compared to a complete set of features that doesn’t work at all.

1- The ping server
The first thing we did is to implement a simple server and test if the game (which was not practically a game then :D) can connect and exchange messages with it. – day 1

2- Simple actions
Then we implemented simple actions to make the game work and players interact – the server is not clustered and there is no state persistence or anything. – day 3
(Starting from here, the server development and game development was going in parallel)

A. Server development

3- Splitting the server
After that we implemented a simple layer above Java Remote Method Invocation (RMI) to be able to split the server on the set of computers in the clusters, and to make debugging easier (that layer could be disabled to run the whole server on a single machine). – day 9

4- Persisting the state
Also, we’ve developed something we called Object Store to provide transparent persistence using aspect oriented programming. – day 26

5- Work that was not complete until we handed the project
Proper Object Store synchronization between several machines, database replication, security checks, traffic control and traffic load balancing, server peer to peer load balancing, distributed administration.

B. Game development

3- Adding a few actions each day to extend the things the player can do. First walking, then attacking, then riding a car etc. (That’s very simple view, the actual actions are numerous and much more complicated than that). – Until we handed the project

Conclusion:
Using iterative development, we’ve faced the overwhelm we had from the extreme complexity and actually delivered something that was working and people can actually play and enjoy. Compared to waterfall model, nothing would have been delivered in the deadline, and there would have been an implementation paralysis, and an elongated testing phase which might require lots of parts to be reimplemented too.

It’s like splitting a big project into smaller projects with limited set of features, and doing that over and over until it’s enough or you meet the deadline.

Pros:
Early feedback
More visibility, for the developer and for the client, the client can make changes early (cuz client don’t know what they want !
Can deliver at any point of time

Cons:
Adding new features later can result in modifying existing code or removing it, which is the cost tradeoff of using iterative development

September 20, 2008 Posted by | Concepts, TeamDevelopment | Leave a comment

Iterative Development – there is a simpler way to work – Part I

NOTE: This text (part I and part II) is just an introduction to the existence of an alternative, not a guide to use that alternative.

Probably the only way you know when you want to start a new project (for college for example), is the famous Waterfall mode. That’s (Planning,Analysis,Design,Implementation,Testing).
When ever you get a project idea that you like and want to start in, your first thought is to gather all your team* to start doing the analysis.

You would probably be happy when you finish your analysis document(s). But now you want to do the design (most probably you understand design as “UML Class Diagram” only).
So you put the analysis documents aside, and perhaps this time let each team member tries to figure out the classes of some part of the project.

So far, although that it contains a handful of mistakes, the life is smiling at you. But…

Once you finished the so called “class diagram”, you start the implementation. Needless to say, you will end up probably nothing like the design, that’s if you were lucky and your project isn’t really a very complex one.

And because you end up with something different, i.e. changes are added randomly as you find them, the code complexity will increase, readability will decrease, lots of ad-hoc fixes**, and will take much longer than anticipated to get it to work as you wanted (and with high probability you will miss the dead-line, and deliver your project nonfunctional or with lots of bugs that makes it totally unusable).

Well, this mostly isn’t really because you are bad, or you did something wrong. But this is due to the nature of both the Waterfall model, and your experience in the kind of project you are trying to do. I will explain.

The Waterfall model assumes that you know exactly what you are doing, and that’s there will be no modifications -in the design- afterwards. To do that, you must have tried and did that -arguably exact- kind of projects several times before. Which in an academic settings is not always true; because in each subject you are assigned a project to apply what you’ve just learned. So the uncertainty factor is very high. Thus the Waterfall assumption fails, and now it is to hinder you more than to help you.

Fortunately, there is an alternative way that assumes the uncertainty and will help you in that regard; Iterative Development***.

I will talk in details about it in Part II isA.

* This in the context of under-graduate projects’ teams in our faculty.
** For people who don’t know what ad-hoc means, I quote from some of my friends 😉 = “طلسأ طلسأ يا عم الحج”. See it on wikipedia http://en.wikipedia.org/wiki/Ad_hoc . In Arabic it probably means اعتباطاً. It’s not always bad in some conditions, but in programming, it usually is.
*** Iterative Development isn’t a model or process by itself but it describes a feature of several different processes. Like -for example, not to list- the Rational Unified Process, Extreme Programming, SCRUM, and generally the agile software development frameworks, all of which, is beyond the scope of this text. This text is just an introduction to the existence of an alternative, not a guide to use it. If you were interested to use it, you should pursue a good book or a good reference in that regard.

August 8, 2008 Posted by | Concepts, TeamDevelopment | Leave a comment

Source Control and Team Development Part 4

See previous parts here

The Only Chapter

In the first meeting, you defined a set of features, classes, and functional tests. You got Class A and Class B. You go home. You start writing a set of unit tests for your features. You write a skeleton of your classes. Just empty functions that returns dummy values. You compile, it compiles fine. Now you Check-In your files to the Source Control Server; just a single click. Before the check-in starts, unit tests are run, of course non of them are passed; you only have skeleton classes. Now check-in is complete. You go to sleep.
Next day you wake up, open you PC, in your home. You click “Update” to get the latest version of other people source code. Then you continue developing your classes. You implement seven functions. You compile, test, and find that 30% of tests have been passed. You check-in your code with a message stating a description of the change.

After 3 days, you find that some previously-passed tests have failed. You browse the history of the class that have failed the test, you click a single button and you have a side-by-side view of the current file version and the last-working version. The changed lines are highlighted for you. You compare only 3 lines, and decide to change one of them, and viola: that test have been passed again. You check-in your code with a message stating the correction.

Now you go to the second failed test, you remember that you have made a change that corrected that before and that maybe you have removed that change forgetting what it was for. You perform a search of the check-in messages using the keyword you used for that change, you got the file version that has that bug corrected, the exact lines that you changed to fix that bug is highlighted for you. You fix the current version with that code, and here it is; the test have been passed. You check-in your code with a message saying that you re-fixed that bug again.

Someday you find something inside someone else’s code that preventing your code from working. You fix it in his file. You check his file in with a message describing your change. When he checks-in his file with different changes, the server merges both of your files if you haven’t modified the same lines. The server lets him know that he doesn’t have the latest version and that a merge will happen, he previews the merge and approves it.

The big day has come. You sit together, you don’t have anything to do. The project was already integrated each day, and all the tests have been passed, you just go to college and get the maximum grade and go home having 4 feelings:
1- I love programming
2- We CAN make things right
3- I WANT to make a much bigger project next time
4- “Weeee are the champions”

See:

Test-Driven Development
Source Control

Other team management stuff:

Scrum

Other development techniques:

Extreme Programming (XP)

Finally, this is all about Agile Software Development

Big thanks to the MDC 2007 😉

February 10, 2007 Posted by | TeamDevelopment | 1 Comment

Source Control and Team Development Part 3

See part 1 here
See part 2 here

Chapter 7

Suddenly some remembers that he has the flash memory that they used to transfer the last working version of the file. He gets is and replaces the non-working file, compiles, runs, and there it is. It works now. Still it doesn’t have the bugfix that they wanted to restore.

They accept that, and prepare to go to college, each one thinking how they are going to hide that bug from being noticed, or how they would convince the professor to have mercy on them if he noticed it.

Later that night, each one of them have four feelings:
1- I hate programming
2- We can never make anything right
3- I won’t spend much effort in such project again
4- WE ARE LOSEEERS 😥

The End

So what we have here ??
We have people who lost self-confidence thus will never be able to make something big, not because they can’t, but only because either they don’t want to, or they believe they can’t.

So, what’s the solution ? What do these people who make big software do ? Can this scenario ever be different ??

“Yes, sure it can be different, those people are just idiots!” Says someone smartly “They could’ve named the folders better!!”

Well, there is indeed a different scenario. But it is not like that smart one said.
This scenario is MUCH shorter than the former. It even consists of only one chapter.

To be con… Ops, sorry, I mean see the next article 😀


February 10, 2007 Posted by | TeamDevelopment | Leave a comment

Source Control and Team Development

I haven’t met a lot of teams who have actually used this feature. Since we are facing a new semester where we should start implementing our graduation project, I will introduce it.

You are in a team ? You divide the coding among you ? Then you definitely need Source Control.

Let me first tell you the scenario that you face each day of your team work:

Chapter One

Person A was assigned Class A, and Person B was assigned Class B. Till now everything is okay; each one works in him own home till he finishes, isn’t it ? Now they’ve finished. It is time to integrate work into a single project to start testing. They sit together in a single place, each one with his computer. They gather the source files and put them on someone machine. They create a project and add the source files. They hit “compile”. They get a huge list of errors.

Chapter Two

After the initial depression that their hopes and prayers that the project works from the first time painlessly have not been responded to, they start to fix the errors.

Each one boots his PC/laptop, gets the list of errors in his file, opens the IDE, and start fixing. After fixing all the errors he has, he gets his updated file to the main PC which they agreed to integrate their files using it. He does that by putting his file on a flash drive, or better, on the LAN.

Now the person on that PC, gets the new file and overwrites the old file. Then he tries to compile again, yet he still gets some errors from another one’s file. He waits till that person gets his files and compiles. And “Hooray!!” it has compiled “Thank GOD!” they say in relief.

Chapter 3

Now it is time to launch the project and see if it is working. They double-click the executable, and “BOOOOM!” they get an exception even before they see the application window.

The exception is in Class A, “Who’s class is this?!!” whispers someone in frustration. Person A replies “It’s mine”. He takes over the keyboard and traces till he find the error. Fixing it requires some modification in someone else’s work. He modifies his own file, while that someone goes to his PC and fixes his own. He gets the file back, he put it in the project folder, overwriting the old file while interrupting the one who is still fixing his file on that PC.

Now they compile again, “Thank GOD there is no compile-errors this time” someone says to himself proudly. They run the application. They see the application window this time. They tried to use it but it doesn’t make the expected behavior.

To be continued…

February 9, 2007 Posted by | TeamDevelopment | 2 Comments

Source Control and Team Development Part 2

see part 1 here

Chapter 4

They start thinking and analyze whose part is this?? Finally after a lot of argument, they decide that Person X and Y are responsible for that feature. Those two takes a side and start a conversation about who will modify which part of his code. They agree on some changes.

Each one goes to his PC, make the changes, and gets his file back to the main PC. “Stop! don’t overwrite the old files!!” Someone shouts nervously “Backup the old folder first, as the new changes might break more than they fix so we can return to the old files”. They create a new folder and name it “after changes”. They put the whole project in it and then overwrite the old files in this with the new ones in this new directory. They try that feature and it works.

Chapter 5

While they are partying the success in having a running feature, by playing and resting. Someone was playing around with their application happy that they created something that works. Suddenly he gets another exception. “Oh damn it!” he says in dissatisfaction “can’t we have a single happy moment that it works fine?!!”.

He gathers the playing members and wakes the sleeping ones telling them the awful news. When he shows them the exception, silence goes around. Each one of themselves wishing that it is not his part so that he can get away with it and go continue resting.

Once again, after discussions, they made modifications, created a new folder, and fixed the problem.

Chapter 6

Six days, Eighteen folders later…

This day is the big day; the deadline. They should go and give a working copy of the project to the supervisor. After a long sleepless night, they’ve got so many folders with a lot of names like “after modification”, “this one will work isA”, “KEFAAYAAA”, “at night”, “3 am version”, “last 1″, last 2”, “final”, and “the final last one”.

“YOU ARE A BIG *** **** MY OLD CODE HAD NO PROBLEM, WHY YOU MADE ME CHANGE IT”, the other one replies “GO TO HELL, IF YOU WAS NOT THAT STUPID YOU WOULD HAVE KNOWN THAT EARLIER”…
[cut: after 15 minutes of yelling and thousands of dirty words]
“HEY PEOPLE, LISTEN TO ME!!! We’ve only 30 minutes to go and this must be fixed! Stop fighting like kids and do something!!” says someone impatiently.

They stop fighting keeping in mind they will complete it later. “The correction of this was in one of the old versions, from two days ago, I know it” says A “let me try”. He takes over the keyboard again, keeps staring to the folder list trying to know which folder it was. He has 3 candidates, so he goes into each one of them opening the above-one-thousand-lines-of-code file and seeking for the correction desperately inside two or three functions in each file.

Finally after an hour and a half, and calls from friends at college saying that the professor is angry because there is no ready team and that he threatens to leave, and after a lot of “Com’on forget that stupid bugs, let’s just go we7na we 7azzena”, and after a lot of closed-eye concentration moments, he finds the correction and applies it to the final version of their application.

He runs the application to try it, and “DA-DOOOOM!!”, the application doesn’t work at all. You have seen so many different reactions in a single minute, someone sits on the floor crying, someone had his face blown in red out of anger, another one just hits his head against the wall.

To be continued…


February 9, 2007 Posted by | TeamDevelopment | 5 Comments