My Project is Failing!

Today, I read http://www.computerworld.com/s/article/71209/Why_Projects_Fail and kind of agree mostly with the reasons.

If we talk more about what and how to deliver in the next three weeks instead of 3 months (or one year and beyond), we will possibly do a much better work. Invariably, this problem exists universally!

Code Reviews - The Pain and Gain

Typically, when implementation is done, code is modularized into chunks. Each chunk that does some small meaningful stuff and that does not break the build qualifies to be committed to version control. There is a school of thought that every commit has to be code reviewed. Now, while only the developer knows what he is upto and how the feature is getting implemented, the code reviewer in this context can only review to ensure that this new addition does not cause any harm. He is in no position to comment upon the completeness since it is assumed that more code has to be committed eventually before we are feature complete. If this is how we work, we have a case for a deep-dive session on every feature once its complete on top of the reviews that happen. Now, this deep-dive where design and implementation is discussed becomes the actual review session where completeness, correctness and consistency are debated upon. If this kind of session is missed, we can only expect that the builds may break the essential 3Cs (Completeness, Correctness, Consistency) and keep coming back as bugs.

The factors that could hamper this process could be:
• My team is not intelligent enough. It’s a waste of time.
• Doing this only to the manager or only when a manager is present. We are counting upon the technical skills of a person who prefers management over technology.
• As such there are too many meetings.
• Never mind, I am never wrong.
• Our testers will catch it anyway.
• This is not such an important feature.
• Well, I myself do not understand how I made this work.

Campus offer from a great product company - Is this a good thing?

When I was a fresher, my list of dream companies included Sun, Yahoo, Microsoft and so on. I believe these days the list has grown to include companies like Facebook, Google, LinkedIn and Twitter. Of course, I did not make it to any of these from campus. After 11 years into industry, now I wonder if it was indeed sensible to get into any of these from campus.

Quite unarguably, all these big time product organizations provide some of the best perks and facilities. Most of them have good work life balance and very matured employee policies. They boast of some of the excellent technical challenges and coolest products. They have great people - many experienced and matured hands work together on different problems. They are always making news and promise great learning. All this, for sure, makes anyone fall for a job with these giants hoping for a great start to their professional profile.

My observations have been quite different. At the first place, I believe, these people are not tuned to learn by failing fast. And this to me, is a number one factor that kills newbie interest. Let me explain. Assume you are into a new team. You hope people to come to you, explain stuff that they do, assign tasks of reasonable complexity just enough to challenge your intellectual energy however assuring enough that you can complete it and boast of tiny success. This rarely happens. Most experienced heads are by passion diving into complex problems all the time that they are rarely bothered about your motivation or improvement. Of course, they may be available and willing to help to bail you out of trouble; but, then you will have to shout of trouble before you can hope to get their attention. This is always below our dignity and we tend to cover up and manage as long as possible and keep delivering some solution or other and thereby learning by ourselves. What a pity! Best of people around but none are utilised for your very own betterment! Let's say you are part of a web company and when you joined you hoped to learn how it works and be a great web developer in short future. You will be amused that there are departments such as SE (service engg), RE (Release Engg), Product Engineering, Program Management, QA, etc. And being part of just one of these organisations, you are pretty much solving one small problem of a small module that belongs to a bigger system. You pretty much cannot develop and deliver an end to end system! Well, thats not all. You are the last one who gets a chance to comment upon processes. Almost every superstar of the team believes his way is the right way and they will ensure that you are the last one to be asked of any opinion. Neither do you understand why things are structured this way nor are you encouraged to ask. In a small less structured organisation, focus is on developing and delivering an end to end solution. Of course, the lack of maturity of a good product company does put these small startups into trouble every now and then. But these people, who witness the pain, fail fast, learn and recover. When they get old, they know how software systems are engineered and maintained.

If you are fresher, and dream of an excellent software engineering career, you most probably wish to be a great developer. Someone, that this world could trust to develop complex challenging software. If you are one such person like this (like me), then this is what you will need:
Good peer community. Need not be super matured super stars. But, a group of people who are just like you. This should be a group that wants to learn, work hard, experiment and deliver.
* Good mentors. These should be people whose success depends on your success. Typically, in a hierarchical organization, a team lead is only successful when all his subordinates are successful. Hence, they are in a mode of constructive criticism and high quality mentorship/training.
* Strong visibility of end to end process and product development. This is particulary important to get to the bottom of all software engineering issues. You should "own" your product - end to end. You should be able to appreciate the associated pain and should understand why was a process step introduced or removed.
* Good hardware and software resources. This to my knowledge is not a problem with any company. The relative costs are much less to own legal and variety of software. However, I have seen some companies advocating star office instead of microsoft office, for example. Many companies do not have licenses of performance testing tools.
* Domain of your interest. A mobile OS development is greatly different from a banking or online retail solution. Typically, it takes years and lot of luck to experience multiple domains and pick one for future. Hardcore finance giants like goldman are known for very professional etiquette and formal clothing. Non-commerce online companies like Yahoo are known for a casual and fun-loving lifestyle. While mobile companies like Nokia and Samsung worry a lot about their products being protected from public eyes till they are released, there are open source companies who are interested in maximum outreach well before the product is ready for launch. Picking your company is not easy; however, mind you, your lifestyle should match the nature of the organization.

All said and done, for most people, first company is not a company of their choice but the best that they see (from factors such as compensation, location, role, etc) from a small lot that they have access to. It takes several years and several hops to figure out what is right and where to settle. Unfortunately, we are like doctors - we know and like our work but do not know where to practise. Its a clinic round the door sometime or a huge hospital in the heart of the city. It will take a while for us to appreciate the characteristics of our environment. There are many lazy people in us who just believe in retiring from where they joined. They are either too brilliant and adaptive that any environment works for them or they are totally passion-less and ambition-less that environment does not matter. Even if you are not interested in hopping jobs, I am sure you will find it interesting to discuss with your peers and knowing how his company works.

Dev Sprint, Test Sprint - Is this scrum or waterfall?

Measure the date of dev complete and test complete difference. This difference should be not more than 2 days. If it is so, it is waterfall, and not agile.

To make this happen, you will have to make smaller and definite releases. Test team should accept builds even if there is zero functional changes (there may be just some amount of code refactoring). Releases should be regular and frequent (like once every 3 weeks). Mind you this is not end of sprint; this is middle of sprint. If you have 3 weeks for a sprint, somewhere in second week, dev team must be done with its first cut work and then it should start working with test team for acceptance, and parallely keep rest of time for subsequent releases.

sprint 1 = release 1 + release 2 activities. sprint 1 = test preparation + release 1 testing activity.
sprint 2 = release 2 + release 3 activities. sprint 2 = release 2 deployment + release 3 prep + release 3 testing.

Why have a backlog?

An ever prioritized list of requirements - Does that not sound as a basic need for a successful team? The team need not be a scrum team, need not even be agile for that matter. Any software development team, anywhere in the world is asking itself everyday, what should be the next item to look at!

Being so fundamental, why do most matured organizations ignore its significance? In my opinion, a simple shared list of items that is editable for entire team is all that is required. A twiki and bugzilla combination should just work lovely. Yet, teams just dont do it. I strongly believe, this needs discipline, a cultural shift and a deep ownership/commitment to product and sense of belonging to a team.

If I ever become a manager, I will ask my team to:
* Create a ticket in bugzilla for every item that it works upon.
* Use a twiki with a table rule to pull all bugzilla items based on a definite tag that signifies that this item is a backlog item.
* Continuously someone owns and updates these tickets with priorities. Marks items for releases and calls out risks.

Now, for your team, if bugzilla is too much, you may just use a shared xls or any of the open and free scrum tools.

Not doing this activity seem to boost an individual team member's productivity; but, come on, there is so much work coming from so many stakeholders, that you are going to spend hours in status meetings. QA will get thoroughly confused on what is that they need to test. Dev team will never read their manager's mind properly and will show sheepish face on wrong item selection when manager is away.

If you are already a manager and nothing above convinces you, at least, for god's sake, you wish to have a self-sustaining team. This is the first step to get your team in that mode.

Top 3 reasons why you do not follow scrum?

Well, honestly, these are the answers:
* Bookish / Theoretical. Its not our dignity to try bookish things.
* Who said we do not? We do daily stand ups for one hour.
* My manager does not know it.

Oh man! We are still an uncivilized and under developed community (of software engineers)!

How can five brilliant people screw up a release?

Not sure if I should count myself as brilliant... so see it as four or five as you find appropriate. Anyways the point is, being part of a leading product development company, as you would expect, this team of five is technically impeccable. They are reliable and hard working wherever necessary. So, how could a release be goofed up with such a combination?

On retrospect, the answer is pretty simple. We were brilliant in our individual affairs and were not really working together towards a common vision. Each one had our set of tasks that we completed to "OUR" level of satisfaction. The "OUR" really demands an emphasis. There was absolutely no acceptance or definition of done. There were nobody to certify that this is actually done. Real stakeholders who were concerned or beneficiaries were never consulted. There were no testers in the team. The central group of testers sitting in another corner of the floor chip in only after the sprint is over or product is available as a build.

Although everyone in the team seem to feel that they could have better handled the process, nobody wants to meetup for defining any part of it!

I think my team is no different; half of world software development community works this way. Requirements come in from product manager, architects, engineering managers, service engineering team, customers, solutions team and sometimes from the dev team itself. The busy engineering team manager decides priorities on the fly and usually, the team is confused on what should go in the next release and is always contemplating where the new bug will find a place in the managers head (towards a release fitment).

Although it is ok for inexperienced teams to get started this way, I hope the team realises this fact that process also evolves just like the product and its important to define and execute a feedback loop to improve the way they do software development. A team that thinks meeting up for process improvement is not worth their time, is either not trusting its own members or just believes they have no control in influencing a process for themselves. In either case, if you are a team member of this kind of a team; your life is set. Its only difficult to be part of a high performance team; not a low performing one.

Lighter side apart, I believe the summary is this:

Some very basic laws that I learned due course:
* Respect your team members and build a team first and then worry about building your product.
* Evolve your product, slow but steady.
* Don't keep your testers far away. Having a central or a different testing team means there is no testing team.
* Don't try to influence your team too early. Win their trust before suggesting process changes.
* If you do not have time, find someone else to drive a release.
* If you believe teaching your team a good discipline of working (as in a process) is not worth it, you will pay for it any ways.
* More than doing everything right, learn to do the right things first.
* Don't fear losing your job today; fear about getting a job tomorrow.
* Know that your are engineering and not inventing a product. Engineering must be professional (predictable, efficient, repeatable and long-standing) and not innovative.
* Engineering best practices come from years of hard work and painful observation. If you think the authors are idiots and if someone had told you that theory should not be practised, please think again. You are screwing up your own life unnecessarily.

Here's the scenario: A new team replaces an existing team of four completely and all of a sudden. Existing team has disappeared for various reasons and even though some of them may be available in the same company, they are not freely available unless there is a disaster to manage. New team takes up a goal that is architecturally sensitive. Team has no idea how they are going to achieve it! Time is running out and there is a feeling that reputation is at stake!

Key contingencies are "Poor understanding of system", "Poor understanding of underlying technologies", "Demanding yet friendly eco-system" and "Poor / Virtually no process in place". Target is to survive and keep maintaining the system and impress with some cool features. It is portrayed that the senior manager is interested in knowing when the team could complete a particular goal of its interest.

Team does typical mistakes - "Making optimistic commitments", "Not planning time for learning activities", "Try, Fail, Discover, Try, Fail, Discover, ... Ultimately, it feels as though "Never Complete and all commitments broken!".

Oh NO!!! All the processes are not just flexed version of the Waterfall, just to support the client's need for viewing the product

I disagree. Everything in this world is process-driven. Whether you are traveling in a public transport vehicle or you have to book a gas cylinder or for that matter, even if you need admission in a college, you follow a process. Software development is no different. Every process has its own set of roles; each having some responsibilities. Every process has associated products that it uses and another set that it delivers. There are several stake-holders in all cases; client is just one stake-holder in the case of software product development. Processes are mandatory in a software eco-system to socially co-exist and run efficiently performing and highly impacting teams.

Coming to the nature of waterfall, it has been widely accepted that software systems are not typically built to an agreed specification. It is not like manufacturing a mobile phone, for example. In most cases, software systems evolve. For example, you do not make a N95 mobile, sell it to user, take it back, modify it again and ask the same user to now use it again! N95 does not evolve. Nokia builds N95 as a product towards a specification that it thinks will work the best and sells it hoping that it will work. If you consider a vocabulary tool as a product and if your team is an expert team in technology, product visualization, design, etc., you will probably be able to come up with a specification in just two days and implement to that specification in a matter of 2 or 3 weeks. In such cases, we could really stick to waterfall.

Processes are tools to work in an eco-system and socially co-exist in such a way that every stake-holder finds a satsifactory role/benefit.

All that said, processes should not be confused with lifecycle models. These are two different things. Ultimately every human being take birth, eats food, lives for some time and dies. This is a very high level visualization of human lifecycle model. At this height, all is same! Constructing a under-sea railway track is not like constructing a out-house. Ultimately you may still generalize in saying that its just about understanding what to build, how to build and then build it!! I am sure the nature of stake holders and their roles will be drastically different in that case.

I encourage you to read articles such as "Mythical Man-Month" and the case of "Killer Robot". These are beautiful manifestation of software engineering issues. They bring out the essence of troubles a software engineer has to go through to successfully survive. You should then start appreciating the wide basket of choices we have at each step of software engineering and how this decision making becomes important and difficult in no time.