Danilo Sato

Inspired by a post on the Lean Blog, Pat reminds us that you can’t measure everything effectively. Having written my Master’s thesis on metrics for Agile projects, I’ve learned and read about this in a lot of different places. One approach that is very known to Empirical Software Engineering researchers is the Goal-Question-Metric approach, first published by Vitor Basili et al. in the 90’s.

The GQM model suggests a hierarchical view of three levels to define which metrics to use:

• Conecptual level (goal): the motivation for measurement. Measuring things without a purpose and a thorough understanding of the problem will lead to meaningless metrics. This level imposes the hardest questions: what’s the purpose? what’s the object of measurement (your product, process, people)? what’s the motivation? who is interested in this goal? what are the quality attributes?
• Operational level (question): at this level, a set of questions are defined to try and correlate the object to the quality attributes we are interested in. These questions should help in understanding and assessing the current situation, but also in identifying ways to determine whether the goal is achieved.
• Quantitative level (metric): only then a set of metrics is associated to the questions, to try and find a quantitative way to measure and answer it. These can be objective (like code coverage), or subjetive (individual’s ranking of current code quality). Finding these metrics is not easy either.

It’s easy to try to cut corners and get into the things that are easy to measure first, specially when you can collect lots of quantitative data to work with these days. However, if you you don’t stop to think about the goals and motivations for measurement, it’s easy to forget the systemic complexity that surrounds us and look only for the easy-to-track numbers.

Lean management and problem solving is known for taking a very thorough and detailed approach in the understanding phase. To many people this is a paradigm-shift approach to management. Don’t let the numbers fool you, use them to your advantage.

Continuing with the series, this time I want to highlight a very dangerous anti-pattern: using velocity as a performance metric. Before getting into the examples of how it applies to velocity, I want to first explain my view on metrics. I am in favour of metrics and coming up with interesting ways of displaying data (information visualization is a very interesting topic). However, the problem lies in the way that these metrics are used. There are two main types of metrics that I like to categorise as:

• Diagnostics Metrics: these are informative measurements that the team uses to evaluate and improve it’s own process. The purpose of collecting them is to gain insight into where to improve, and to track whether the proposed improvements are taking effect. They are not associated to a particular individual or to how much value is being produced. They’re merely informative and should have a relatively short life-cycle. As soon as the process improves, another bottleneck will be identified and the team will propose new metrics to measure and improve that area.
• Performance Metrics: these are measurements of how much value your process is delivering. These are the ones you should use to track your organisation’s performance, but they should be chosen very carefully. A good approach is to “measure up”. Value should be measured at the highest level possible, so that it doesn’t fall into one team’s (or individual’s) span of control. People tend to behave according to how they’re measured and if this metric is easy to game, it will be gamed. There should also be just a few of these metrics. An example of one such metric would be a Net Promoter Score (that measures how much your custumer is willing to recommend you to a friend) or some financial metric like Net Present Value (read Software By Numbers if this interests you). As you can see, these are very much outside of a team’s control and to be able to score high on them, they should try and do a good job (instead of gaming the numbers).

Going back to velocity, a very common mistake is to use it as a performance metric instead of diagnostics. Velocity doesn’t satisfy my criteria for a good performance measure. Quite the opposite, it’s a very easy metric to game (as mentioned in my previous posts). When approached as a performance metric, it’s common to see things like:

• Comparing velocity between teams: “Why is Team A slower than Team B?” Maybe because they estimate in different scales? Maybe their iteration length is different? Maybe the team composition is different? So many factors can influence velocity that it’s only useful to compare it within the same team, and even then just to identify trends. The absolute value doesn’t mean much.
• Measuring individual velocity: as highlighted by Pat, this is a VERY DANGEROUS use of velocity, and it can actually harm your process and discourage collaboration.
• A push to always increase velocity: it’s common to have a lower velocity in the beginning of a project, and that it tends to increase after a number of iterations. Inspite of that, I’ve seen teams pushing themselves to improve it when they reach a natural limit (Who doesn’t want to go faster, right?). Velocity measures the capability of your team to deliver and, as such, tends to stabilise itself (if you have a stable process and the number is not being gamed). A Control Chart could help you visualise that. As noted by Deming, in a stable process, the way to improve is to change the process.

It’s important to remember that velocity is a by-product of your current reality (your team, your processes, your tools). You can only improve your process once it’s stable and you know it’s current capacity. Velocity is just a health-check number that will tell your team’s capability. It will not tell you about how much value is being delivered or how fast you’re going. You can deliver a lot of points and make trade-offs on quality which, no matter how you measure it, will impact your ability to go fast in the long run. As uncle Bob says:

“The way to go fast, is to go well”

So let’s stop using velocity to measure performance and look at it as a diagnostic metric to improve our software delivery process.

Dan North wrote an interesting post about the perils of estimation, questioning our approach to inceptions, release planning, and setting expectations about scope. This made me think about the implications of those factors once a project starts, and I came up with some anti-patterns on the usage of velocity to track progress. This is my first attempt at writing about them.

Before we start, it’s important to understand what velocity means. My simple definition of velocity is the total number of estimation units for the items delivered in an iteration. Estimation unit can be whatever the team chooses: ideal days, hours, pomodoros, or story points. The nature of items may vary as well: features, use cases, and user stories are common choices. Iteration is a fixed amount of time where the team will work on delivering those items. Sounds simple? Well… there’s one concept that is commonly overlooked and that’s the source of the first anti-pattern: what does delivered means?

One of the most common anti-patterns I’ve seen is not having a clear definition of done. Lean thinking tells us that nothing is really done until it’s delivering value, which in software means: code running in production and being used by real users. Although I know very few teams who can deploy code to production at the end of every iteration (some even do more than once per iteration), once a story is considered done, it could be potentially shipped, if the business decides so. There shouldn’t be a lot of extra work after that.

Another bad implication of this anti-pattern is that some teams decide to change the definition of done and count half-completed work to show progress. Some of the symptoms to help diagnose if your team is suffering from this anti-pattern are:

• The team starts tracking dev-complete stories
• “It’s done, but [we need to write the acceptance test/it’s not integrated with the other system/…]”
• “It’s done, but not done-done”
  
• It takes a lot of extra work to get the story deployed to production
• After finished, the story goes into the next team’s backlog
• Hearing terms like “development team velocity” or “test team velocity”
• Counting half-points or percentages because “if we don’t count it will look like we haven’t worked”

The solution? Remember that velocity is just a number that provides information for the team to understand and improve it’s process. Forget that you’re tracking it and focus on the entire Value Stream and on what’s really value-added to get things into production. Anything else is just waste. If it’s not done, it’s not done. Accept it, move on, and don’t overcomplicate, because it will only add noise and mask what could have been important information to the team.

Last week marked the 10th edition of the XP 200x conference, held in Sardinia, Italy. Me and Francisco were there to present an extended version of our Lego Lean Game. Being selected as the first session on the first day of the program, we were expecting a small audience, but it turned out to be quite well attended by about 20-25 people.

We started the session a bit delayed, due to the lack of room organisation: I was a bit shocked when we arrived and the room was arranged as a normal “lecture room”, rather than the usual group tables (that we requested a week before). Projector and flipchart were not available, so it took us about 15 minutes to have everything ready to begin.

The slow start, however, did not got in the way of the overall workshop. We have designed the activities in a flexible way that allow us to adapt their length just-in-time so we still managed to cover everything we wanted without having to rush.

The first half of the workshop was mostly the same version we presented last year in Buenos Aires, with slight modifications based on feedback we got from participants. The second half, however, was mostly new and we included an activity to allow each team to come up with their own processes (rather than following ours). This turned out to be a great success! Each group came up with different ideas and, by watching the other teams perform, we had an interesting discussion about the different approaches and results. We now think that the original version is too condensed :-)

The feedback we received after the session was great and a lot of people asked us for the material to run the session themselves. Me and Francisco have a “game package” that we can share for those interested in running the game. Get in touch with us if you’re interested! You can find more photos of our presentation here, here, and here (thanks to Hubert for sharing his pictures!). The slides are also available:

We are very interested in your feedback. So, if you were at the conference or want to use the material to run the workshop, please let us know! Share your experiences and help us make it better!

XP 2009 is happening between 25-29th of May in Sardinia, Italy and I will be there attending and presenting some interesting workshops:

• The Lean Lego Game: Me and Francisco have been improving our workshop since we last presented it at Agiles 2008 in Argentina, and we will be presenting a long version (180 minutes) on Monday, May 25th. We have just a few “seats” available to participate on the session (20-24) and it will be occupied in a first-come-first-serve basis. If more people show up we have plans to try and not reject anyone, though.

• Test Driven Development: Performing Art: Emily Bache kindly invited me to present a Prepared Kata at her workshop and I will be pairing with Francisco for 30-40 minutes, programming in Ruby with RSpec/Cucumber. Should be fun to “perform” and watch the other pairs as well. Looking forward to that session on Wednesday afternoon!

My fellow ThoughtWorker Pat Kua will be there presenting a workshop as well. I will try to brush up my (lately lazy) writting skills and publish some conference reports. And hope to see you all there in Sardinia!

The last conference I attended in October was Falando em Agile (Talking about Agile), a 2-day conference organized by Caelum. First of all I need to thank all the organizers for inviting me to speak and for putting together such a great conference.

The content was very mixed, with some people focusing on the management aspects of Agile, while others highlighting the importance of the technical side (with some in between, as well as interesting experience reports). About 200 participants showed up with great questions and experiences, which impressed me and showed how much Agile has grown in Brazil since last year.

David Anderson gave the opening keynote, which focused on his “Recipe for Success”. I find he’s a very opinionated speaker and he can get me both agreeing and disagreeing with him at the same time, which is interesting because we can always learn something. At the same time that I don’t believe in a recipe for success, I strongly agree with his points about how the focus on quality leads to fast delivery, focusing on the engineering aspects to build quality in.

The second talk was mine and Frankie’s about Agile adoption anti-patterns that we noticed in our experiences applying Agile in the “real world”. I think the presentation went really well and the feedback we got was very positive. A lot of people came to talk to us during the break to tell us they’ve identified themselves in a lot of our examples, and that they liked to see us talking about what didn’t work instead of the bright/good things we like about Agile. The slides are available here (in Portuguese).

After lunch, Adail gave a talk about Agile thinking, showing tools such as mind maps and theory of constraints trees. After that, the guys from SEA Technology presented a very interested case study of their attempts of applying Agile in a government project in Brasília. Very interesting to hear their experiences and how it’s possible to apply Agile even in the most adverse environments.

José Papo gave a talk about different types of contract and how their are suitable (or most commonly not suitable) for Agile. The closing talk of the day was given by my friend Guilherme Chapiewski about Agile Leadership. It was interesting to see that he gathered a lot of topics on the subject from different sources and his own experience, arguing about the conflicts of being a leader that provides technical versus process guidelines. I’ve been recently reading a lot about Lean, and my current thinking goes much towards a Chief Engineer role, with a strong technical background, but at the same point with a holistic view of the product and what is value from the customer’s perspective. I missed this topic from his talk, but it was overall really good.

The second day started with traffic again, so I missed Alexandre Magno’s talk about Scrum in PMI environments. Danilo Bardusco gave a great experience report of how they are adopting Scrum at Globo.com, the largest media company in Brazil. I think this kind of experience report is really important for people who are starting with Agile, because it shows how hard it is to change a company’s culture and some of the problems they might find on the way. The last talk before lunch was given by Prof. Fabio Kon and Daniel Cukier: they are both members of AgilCoop (which I’m also part of) and gave a great summary of Linda Rising’s and Mary Lynn’s book “Fearless Change”, showing patterns for introducing new ideas.

Because of the short lunch break (only 1 hour), I was late to watch Daniel Wildt’s talk about his experiences as an Agile coach at Dell. Antonio Carlos from Yahoo!, gave an interesting talk about the importance and the responsibilities of a Product Owner (which I would generalize as the customer in general). The closing keynote was from my co-worker Philip Calçado, where he shared his experience in 2 different projects where bad management decisions made an Agile adoption less successful. I particularly liked his point that “Agile is about inspecting and adapting, but you need to understand what you are doing”. Dropping practices or putting others in place without understanding why they are important may be more harmful to the team.

Overall the conference was really good and I enjoyed seeing my friends and meeting new people enthusiastic about Agile. I was really impressed with how Agile has grown in Brazil since last year. Next year’s conference is promising to be even better and I’m looking forward to participate again.

Continuing the series of conference reports, me and Frankie spend a day in Buenos Aires, Argentina to present a Lego(tm) Game we developed to teach some Lean practices and principles at Agiles 2008. The conference was held during the full week of 20-25 October, including three concurrent 2-day Certified Scrum Master trainings and Mary and Tom’s 2 day course on Lean Software Development.

Unfortunately, we were not able to stay for the whole week, but spending one day gave us a pretty good impression of how Agile is growing in Latin America. The conference had about 400 participants during these 5 days and they had to reject some of the 900 interested due to logistics constraints. As happened in the US and Europe, the major driver for Agile adoption is being Scrum and, as more people start adopting it, the more problems are uncovered about how they can improve on the “technical” engineering practices.

Mary’s opening keynote was exactly about that: how important it is to look at the engineering side of Agile to make its success sustainable. This was the same talk she gave at Agile 2008, highlighting the successes and failures (Plank Roads) of our short software engineering history. I thought it was much better than the last time I saw it, and she managed to convey her message in a much more clear way: focusing on processes/life-cycle has been fragile, while strong technical and engineering practices has shown success throughout our history.

Our workshop went really well: the number of participants and their level of knowledge on Lean matched exactly what we had in mind when we developed it. I’m not going to describe the dynamics of the session, because Frankie already did a good job in doing that. Suffice it to say that the feedback we received was great and that we already have some changes to make it even better. I’m also making the slides available here, although you would have to participate on the hands-on exercises to fully understand it.

I think the overall message of our session was to show how some of the Lean practices work in practice, but also highlight the importance of Systems Thinking and the principles behind the practices. Blindly applying a practice may give you marginal results, but to fully embrace a Lean philosophy you need to keep learning and improving. There’s not an easy recipe to success.

After a good lunch (with Argentinian steak and some Brazilian friends), we went back to the venue and didn’t have a lot of time to watch any other session, so we rushed to the airport. Next year’s conference is promissing to be even bigger, and besides wanting to stay for the whole week, I hope to see more Brazilians sharing their Agile experiences with the Latin American community.

My last session report on the series will be about the second half of the double session I attended on Friday afternoon, which was much more informative and funny. Kenji showed (with live translation) a 30 minutes japanese video about how a former Toyota leader conducted a plant transformation at a Sanyo cellphone factory in Japan.

It started with the successor of Taichi Ohno walking around the production line, followed by a bunch of people, and pointing out a lot of visible waste: unfinished products, piles of partially finished packages, and part-empty boxes. It really demonstrates the Toyota’s practice of Genchi Genbutsu, or “go and see for yourself”. By showing to the plant workers the visible wastes, he even claimed not having seen so much waste in a while :-)

His first approach to transform the plant was to physically reduce the production line size. He arrived on the next day and started to rip out the conveyor belt. The plant manager ran scared to the scene to see what was going on and was asked to get a stair and remove one of the signs hanging from the ceiling that read “Packaging Area”. The idea was to make the size of the production line smaller, so that people would have less space between then. That would require less people: the ones still working would have to do the job of more than one person, and the others could be moved to another department (such as product development).

As you would expect, the changes introduced some temporary chaos into the work place. Not everyone was happy with doing more than one job and the plant manager was worried they wouldn’t make their deadlines. After a heated discussion, he decided to go back to the old layout. However, he allowed that one of the workers participated in another improvement experiment: she would have to work alone in a work cell (yatai), doing the job who was previously performed by 6 or 7 people in the old production line.

At first, the time she took to assemble the cell phone was 2 minutes slower than the production line takt time. However, as she became more proficient, she started to suggest kaizen improvements, such as: raising the table and moving the pieces around to be closer to the place where they would be needed. Within 5 days, she managed to beat the productivity of the production line. The improvement was celebrated by the company’s leaders in an internal event, and she claimed that she felt that she could still improve and become faster.

My mains lessons from watching the video and seeing a lean transformation in practice were:

• Any change introduces a temporary period of chaos, which may cause the process to be reverted before the benefits are realized. Read more about the subject searching for Virginia Satir‘s work.
• Respect for People is clearly shown in the video, as the transformation leader stands besides the worker to support her while she is learning the new skills.
• The differences between Western and Eastern cultures: even though she was unhappy at first with the fact that she would have to do the job of 6 people, she respectfully acknowledged the new job and gave her best. It’s also clear that the Japanese have a more holistic view: companies collaborate with each other to improve the country’s industry as a whole. One of the “incentives” raised by the transformation leader was that if the quality of their products didn’t improve, their jobs would move to China.

This is the end of my Agile 2008 session reports. I hope it proves to be helpful for those who couldn’t attend the conference. Feel free to contact me or leave comments if you want to take these discussions further.

Kenji Hiranabe was awarded with this year’s Gordon Pask Award and 2 of his sessions were voted for a re-run on the last day of the conference, which he decided to present together. In this post I will summarize the first half of his presentation.

New Product Development @ Toyota

Kenji presented an english version of Nobuaki Katayama’s (a former chief engineer at Toyota) talk on a Japanese Agile conference. The video from the first run of this talk is already available on InfoQ, but there are some points that I think are worth highlighting:

• Product development is a phased process: the first phase (getting the concept right) is all about creativity and insights to arrive at an overall vision of the product. For example, the vision for the Prius was not to be a hybrid engine car (this was a decision made later, at the design phase), but to be energy-efficient. According to him, this phase should take as long as necessary to get the concept right. The second design phase is milestone-driven and the chief-engineer has an overall cost buffer that he can use when making trade-off decisions during the development process. The third phase is going into the manufacturing line. This phased approach was somehow brought up again by Alan Cooper on his closing keynote (commented slides are also available online).
• Leadership characteristics: Toyota doesn’t value leaders with a dictator attitude. What really surprised me is that they also don’t seek charism in a leader’s attitude. They should be there to enable teamwork and keep a constant focus on the macro view (product vision).
• Bad news first: Toyota leaders don’t like to hear what is going well. They trust that everyone is doing their best to keep doing the good things. They are there to remove impediments and help solve the problems, so they cultivate a culture of always giving the bad news first.

Another interesting fact mentioned by Kenji is that, after his presentation, the chief engineer watched two experience reports on the adoption of XP and Agile in Japan. He said that, even though we are following different practices, we are applying the same engineering thinking. We use changeability in software to defer the chance of changing things to the last responsible moment. In manufacturing, repetition (as in iterations) is considered a failure. But we use tests to continually keep quality high, allowing for late changes to be implemented without incurring high costs.

I attended this session on Friday morning. This time the subject was another Lean tool: kanban. Corey Ladas showed 3 different project scenarios and presented different approaches to implement a kanban system.

He started by talking about some important Lean concepts like one-piece flow, work-in-process (WIP), cycle time, and their relationship using a Cumulative Flow Diagram (or finger chart). He showed how a constraint in the system can cause disruptions to flow and cause more harm than benefit to downstream processes by building more and more inventory (WIP). He then went on to explain the benefits of using a kanban system to limit the amount of WIP in the different scenarios.

The first scenario was a traditional waterfall-style process. He used a value stream map to identify the amount of value-producing time in each phase of the process, and used that to calculate an initial buffer size for each kanban lane. Instead of trying to explain the whole idea here, I suggest you to read Corey’s series of 4 posts on the subject.

The second scenario was a transition from a Scrum process into using a structured kanban instead of a traditional story/task board. Again, the approach is explained in more detail in a post about what he called Scrum-ban.

The third scenario would be an improvement over an existing kanban process, but due the number of questions throughout the presentation (which generated some quite interesting discussions) he didn’t have enough time to go into more detail.

My overall impression of the session was that he did a good job explaining kanban as a tool, and the reasoning behind it to limit WIP and control the flow of your process. The discussions and questions were also very interesting, which demonstrated the audience was interested in the subject. The thing that I didn’t like so much was his argument in favor of having specialists in the team, and how to move into using kanban without too much change along the way. I think one of the fundamental Lean principles is kaizen, or continuous improvement, and it encourages the team to constantly search for better alternatives. Change is part of the process and should be seen as a Good Thing. And I have already shared some of my impressions about Generalist vs. Specialists, so I’m a bit biased :-)

An important thing to keep in mind is that kanban is just one of the tools in our Lean toolkit. I particularly like Kenji Hiranabe’s InfoQ article on the subject, where he explains kanban in a more broad context, acting as a balancing tool between reducing WIP and one-piece flow, as well as a way to visualize kaizen. Lean is full of contradictions, and you must understand the underlying principles to be able to apply (and change) the practices to a particular situation. Using kanban per-se shouldn’t be a goal for any agile team, but it is very important to understand how it works and the reasoning behind it. If you haven’t looked into kanban yet, you should definitely check out the references in this post.