Saturday, July 26, 2008

Leadership on a Self-organizing Team

Self-organization is at the core of a successful Agile/Scrum team. When a team is transitioning from a traditional team structure (hierarchical) to self-organizing team (flat), I have seen the team not sure how to accommodate traditional roles like team lead or architect.

Let's delve into how a traditional team works. The single point of accountability lies with a team lead who is expected to ensure the success for the team. He/she decides who does what and delegates work accordingly. Even though, a team lead may involve team in making these decisions, more often than not it is a one way street. When we are accountable for the outcome, instinctively we feel compelled to force our way of doing things. And the rest of the team takes a back-seat in the whole process.

In a self-organizing team, on the other hand, each team member is accountable for his or her work and collectively accountable for the success of the team. This collective accountability takes the center stage in a self-organizing team. There really is no single person to point finger to. As a result, the team collaboratively decides who gets to do what. It may be driven by the current skills, experience, as well as the need for cross-training and self-interest to learn new things. The leadership is not bestowed by a title, rather is earned over time through performance. In fact, the leadership becomes distributed based on specific expertise. For example, a team member might show a lot of interests in database technology. He/she establishes himself as the SME (subject matter expert) by voluntarily solving database related complex issues facing the team. Similarly, another person may show greater understanding of OO design and programming and the team might start to defer OO matters to him. By effectively bringing these individual expertise together, a team as a whole, could work as an architect and come up with a good architecture. This may not happen with a team that is working together for the first time. The Scrum Master will need to help the team become self-sufficient and self-confident to fill in the traditional roles of an architect or a team lead.

So, am I saying the need for an architect or a team lead is gone? The answer is "Yes." A team may choose to make the SMEs the spokespersons for the team for topics. The topics could be technology, or business. The SMEs will handle all questions/queries related to their own topic. It needs to be published to all stakeholders. Each SME will need to keep the responsible enterprise architect in the loop for his or her area of interest. For example, if the issue at hand is security related, the team SME for security will work with the enterprise architect.

I know you might be thinking that this is not possible on a real project. I agree it may not be the case when a project team is working together for the first time. We may need help from a solution architect who could help the team to weave all discplines into a cohesive solution. However, the goal of the solution architect should be to help the team get to a point where they can collectively drive the solution forward. The first step towards achiving this "team nirvana" is to put together a team with a collective knowledge and experinece covering all aspects of the project.

Saturday, July 12, 2008

User Stories vs. Use Cases

I am consulting on a large government project, where we are using Agile/Scrum. Lately we have been having passionate discussions around whether or not User Stories are enough for capturing requirements for large projects. My answer is yes User Stories are good enough for capturing requirements for any size projects. However, the product owners on the project are of the opinion that User Stories need to be complemented with Use Cases. We couldn't come to an agreement and the discussions continue. I recently came across Jeff Anderson's blog that exactly captures the questions/concerns that my co-workers on the project have raised. So, I thought let me share my view that I already did with my co-workers.

User Stories do not provide context. The way to add context to user stories and linkage among them is by associating stories with themes/tags. We could either associate a story with a single theme or multiple themes. For example, say a story is - "I as an Admin can approve registration requests." We could associate this story with "admin," and "registration." Additionally, we could associate the story with "architecture." Maybe this is the first story to address how we would implement admin module as well as security roles. So, it is architecturally significant. Later we could filter all stories that are architecturally significant. The themes can also be organized in an hierarchical fashion. However, I would argue that the same thing can be accomplished without explicitly creating this hierarchy. This will make life much simpler and provide greater flexibility in organizing and viewing requirements.

User Stories do not capture appropriate level of abstraction. I agree that the implementable stories might not be suitable for tracking and reporting to sponsors and management. However, if we think about how stories naturally evolve, then we would realize that most stories start at an epic level. Then over time, we break the epic stories down until we find implementable stories. The epic stories should provide the abstraction that we are looking for. We could report to the stakeholders/management on progress at the level that we feel appropriate. Of course, we would need a good tool support to help us with tracking and managing the relationships among stories. Watch for ScrumPad to provide this capabilities soon. However, we could also do this using plain old Excel.

User Stories do not capture enough details. Well, we could capture any and all information related to a story that we could in a Use Case. The amount and level of details that we should capture depends on many factors. The trick is though the way to put the details and being able to understand when is too much or too little. This is more art than science and depends on individual experiences and preferences. However, the intention should be to make a story card a starting point for conversation and collaboration between developers and product owner. I see a tendency among product owners coming from Use Case background to put in a lot of details as an alternative to conversation. The fallacy in that is the details are known, understood and can be unambiguously communicated in written format. We all know how that works.

However, on our project we are mapping stories to Use Cases where we need to interact with an external team that is not familiar with User Stories. In general, User Stories can be used to manage requirements for any size project.

Thursday, July 3, 2008

Architecture on Agile projects- define a baseline architecture

One of the questions that frequently comes up when people move from waterfall to iterative incremental software development (Scrum, XP), is how architecture fits into this new world. Most people of the opinion that architecture needs to be in place before you can start iterative incremental development. My answer is yes and no. I will try and explain what I mean by that. This will be first in the series.

Whether we are building a simple application or a complex enterprise application, we need to have a starting point. The starting point of an architecture is defining a baseline.

The baseline architecture. Yes, this needs to be in place before we start our first sprint. In sprint 0, we will define our baseline architecture as we build our product backlog. Some of the things that are included in this baseline at this stage are,

  1. What software platform to use? We need to decide whether to use Java/J2EE, or .Net, or Ruby on Rails, or some other technology. We also need to select what version of the technology to use. Do we want to use the latest stable release or beta release of the technology? The decision depends on many factors. Among them are type of application, existing standards (companies may already have made this decision as part of enterprise architecture), experience and expertise of the developers as well as support people, strategic partnership etc.

  2. What hardware platform to use? This decision depends on the decision from the previous question. If we select .Net, then we all know we will have to use Wintel hardware. If we select Java or RoR, the choices are more- Lintel, Sun, AIX, etc. However, we most likely will choose Lintel. We may also have to make a decision about the vendor we will get the hardware from. Most of the time companies have existing relationship with a vendor.

  3. Where to host? We know what technology we are using as well as which platform to run the application on. How about where will the servers be located? Do we want to put the server in-house or do we want to outsource them to one of those hosting providers. If we already have an in-house data center, the answer is clear (even though I am not sure why most companies would want to continue to own and operate their data center now-a-days). If we decide to go with a data center provider, we have a number of options at our disposal- shared, virtual dedicated, dedicated, and cloud. However, who do we select as our data center provider depends on whether or not the hosting provider can support the growth of your application (scalability) in addition to pricing, and reputation. We do not need to know the exact load profile of our application to understand how the platform will scale. But, we need to have an idea about the expected growth of usage of the application and the type of users.

  4. What architecture style and pattern to use? Architectural style sets the tone of the solution whereas Architectural patterns, like design patterns, provides a proven skeleton for implementing the solution following the style. The most popular style is n-tier architecture. In recent years, we have seen growing popularity of SOA (service oriented architecture), and RESTful(representational state transfer) style of architecture. However, the mother of all architecture pattern is MVC- model, view, controller. We cannot go wrong with this and it is applicable to most situations. Other patterns to consider in conjunction with MVC are hub-and-spoke, pipe-and-filter, and publish and subscribe.

  5. What technology tools to use? Most likely you would build the solution from using one or more off-the-shelf products/components/tools, commercial or open-source, instead of doing everything from scratch. Some of the type of tools that you would want to consider are- MVC framework, ORM framework, logging library, instrumentation framework, rule engine, workflow engine, integration engine, BPM, reporting engine, ETL etc. Some of these tool selection may involve a formal vendor selection process. However, the baseline architecture should include a short list of 2/3 choices.

  6. How to handle security of the solution? I have listed this as a separate item to consider for the baseline architecture because of its importance. Even though we all understand the importance of security of any solution, we tend to put it in the back burner. We developers usually like to use adminstrative access (turn off security) to avoid the headache of dealing with security issues when we write codes, which is at odds with "Principle of Least Privilege." The best practice is to start with no previliage and open up just enough to get the job done. The baseline, at a minimum, should define how to perform authentication and authorization as well as data encryption. For example, we need to decide whether to use windows or database or LDAP for authentication and authorization. Also, we need to decide whether to encrypt communication and when to encrypt (e.g. HTTPS).

  7. What development tools to use? Even though it is not part of the solution and hence is not part of the architecture, it assists in implementing the architecture. Also right development tools ensure efficient and quality delivery. The is why typically the architect is on the hook to drive these decisions. The tools are SCM (software configuration management), IDE, testing (unit, integration, functional) frameworks, build tools, project management & collaboration tool (e.g., ScrumPad), and design tool (e.g., Rational Rose, Visio).

  8. Setup of separate development, and test environments. Typically developers write codes on PC/laptop and then push the codes to integrate and run in a separate environment for testing before the application can be put in the production environment for actual use. The test environment must replicate the setup of the production environment. The only difference between the two environments is that the production is setup to handle the expected load. That is why it is important to define a unit footprint of the solution to help setup the test environment. This also allows you to define how to scale your solution in production as well as determine when to scale (in terms of the load supported by the unit of the solution). For example, say your unit of the solution can handle x load, and you need to support 5x load, you know you need to get 5 units of your solution. Even though it may seem common sense , you will be surprised to find out that how many companies do not have this in place. Yes, there are situations when you cannot have the test environment exactly replicate the production environment. This is usually the case when legacy systems are involved. For development environment, we also need to make sure that the developers can integrate with rest of the codes easily. We need to make sure a developer can have access to a representative instance of each element of the architecture. For example, if the solution is using SQL server enterprise edition, the developer may choose to use SQl server express. Sometimes, we may not have separate instance for each component of the solution for individual developers, we may need to make an effort to come up with a work-around. For example, you may need to integrate with a Mainframe application, but it may not be accessible from a developer's laptop because of the absence of network access. You could use the test environment to allow developers to access the mainframe application from their laptops.

I know some of these are common sense elements of any solution, yet I am surprised to see how many projects do not have these in place. We should spend a lot of time to define the baseline. Some of the decisions are related and one decisions will help narrow our choices for the others. Others are purely individual (or organizational) preferences and we should be fine either way. Once we have the baseline defined, the detailed as well as additional elements can be iteratively fleshed out. I will discuss how to evolve a baseline architecture iteratively.