Showing posts with label Agile Development. Show all posts
Showing posts with label Agile Development. Show all posts

Tuesday, April 06, 2010

Untapping your people’s passion

I’m in the midst of preparing for The Internet Show in Melbourne, 13-14 April 2010. As a sponsor my employer, Elcom Technology, has several slots for presentations. We’re doing a case study on a social networking startup and a CMS clinic, but most importantly for me I am also doing a session on Web 2.0 and Social Media within the enterprise. These are the key ideas behind that talk, although this is more detailed than I will be on the day.

Passion is one of those wonderful ideas, it is hard to quantify or measure, but unmistakable when you meet it. In the business world we tend to think of it as messy, disorganised, hard to manage, hard to inspire and hard to control.

Passion is messy

However, passion is one of the greatest competitive advantages, with something they are passionate about, people will spend more time, think more creatively and fight harder than they will with other things in their lives. It is also well-recognised that passionate employees create passionate customers, and that’s something any business should be interested in.

Deloitte’s 2009 Shift Index says that “Passion is when people discover the work that they love and when their job becomes more than a mode of income.” They describe the need for passion in terms of competing for employees:

“Why does passion matter? Because staying competitive in the newly globalized labor market requires all of us to constantly renew and update our professional skills and capabilities. The effort required to increase our rate of professional development is difficult to muster unless we are passionately engaged with our professional activities.”

A History Lesson

Around 100 years ago the Ford Motor Company implemented an assembly line in order to boost the production of the Model T. Initially, Ford raised the wages of assembly line workers to keep them interested in the innovations they were rolling out, but now we think of assembly lines as places where cost is minimised by reducing the complexity of the work across many simple steps, and therefore minimising the training (and wages) required for employees.

By simultaneously innovating to a lower price point and creating a middle class that could afford their product, Ford managed to reach a dominant early lead in the production of motor cars. This precipitated a movement from craft production to mass production and in large part was based on scientific management, a term coined by Frederick Taylor.

The motor car industry might have continued in this vein forever, except for the advent of the Toyota Production System (TPS). The founders of Toyota were unimpressed with Ford’s assembly line, but found a way to improve it greatly. Their ideas were wildly successful and a host of studies tried to understand the core ideas in the TPS in order to apply it in the western world. Thus we received just-in-time manufacturing, total quality management, continuous improvement and lean manufacturing.

Most of these ideas have failed to find as fertile soil in the west as they did in Japan – however the evidence is that the difference is not cultural (Toyota runs plants in America run as well as they do in Japan), predicated upon Japanese corporate solidarity, or based on some secret sauce hidden from researchers. In fact the key items were in plain view all the time, but hidden by the biases the researchers brought with them.

There are two key differences they ignored:

1. People Management

The first key difference was the way people were treated. In the TPS people at all levels are valued and given control of their area of work. For example, an assembly line worker is able to shut down the entire line if they find a defect, and will then be part of the team tasked with finding the root cause of that defect and solving it.

2. Practice Makes Perfect

Another key difference is that the Japanese understood W. Edward Deming’s statistical process control methodology, and saw their assembly lines as being imperfect and in need of continuous improvement, rather than the perfectly defined processes that scientific management assumed.

An illustration of this is the pottery class that was split 50/50 between students who were told they would be judged on the quality of a single pot at the end of the course, and ones that were told they would be judged on the quantity of the pots they produced. The ones who focussed on quantity ended up producing better quality pots because they learnt from each one they created, whilst the others got stuck in trying to create the single perfect pot first time.

Eventually in the mid-80’s, Ford asked Deming to help them turn around their quality problems. They were surprised when his focus was on their management methods, but his influence was quickly seen in the profitable Taurus-Sable line.

Today these principles are working their way into other areas of work, through Agile software development and the general concept of Lean. An important key concept is the idea of waste – whether it be spare inventory, transportation, defects, unnecessary motion, over-production or over-processing. The well-known Scrum methodology lists three important pillars for empirical process control:

  1. Transparency
    Know what is happening and be able to measure everything.
  2. Inspection
    Regular and frequent inspection of the process to ensure it is functioning without waste.
  3. Adaptation
    Adapting the process to remove any waste.

So What?

A big part of Deming’s management principles was helping people re-discover “pride of workmanship” (or passion!) in their work, and removing barriers to them doing this (e.g. work standards, quotas, annual ratings, management by objective, etc.), substituting leadership for those barriers.

Whilst Deming’s methods have transformed the manufacturing industry, with companies like Dell and Boeing taking lessons from the TPS and applying them well, there are many industries where Taylor’s scientific management methods still rule, mostly because of habit and management (mis)training.

I know that I learnt how to manage very much in line with scientific management techniques (mainly by osmosis from poor managers). Plan, control, punish, reward, etc. The reality is that there is a shift in our working lives, we need more passionate creatives than ever, more artisans or craftspeople than unthinking drones. There is scarcely any industry or business that does not need at least some of their people to be passionately involved, engaged and thinking outside the box.

That last point is key to applying this. For some organisations the amount of passionate creative work that overlaps their routine work is minimal, for others there is almost a total overlap. The trend is towards more passionate creative work, and we will see that increasingly the leaders in their field are organisations that tap into this and harness it properly. Ford’s initial success in keeping their assembly workers by doubling their pay was short-term, soon the rest of the employment market matched their wages and the middle class expanded rapidly thereafter.

Your efforts need to match the amount of passionate creative work that overlaps routine work in your industry – but don’t shy away from finding ways to move away from the industry average and get more creative work in there.

How Can We Do It?

The question is how do we help people tap their passionate creativity in the workplace so they want to stay?

We need to go back to the basics of good management again to find out what really motivates employees. Daniel Pink talked about this at TED in August last year.

 

“There is a mismatch between what science knows and what business does. And here is what science knows. One: Those 20th century rewards,those motivators we think are a natural part of business, do work, but only in a surprisingly narrow band of circumstances. Two: Those if-then rewards often destroy creativity. Three: The secret to high performance isn't rewards and punishments, but that unseen intrinsic drive. The drive to do things for their own sake. The drive to do things because they matter.”

In summary Daniel points to three key ideas for managing people in the 21st century:

  • Autonomy
    The urge to direct our work and have some say in what we do.
  • Mastery
    The desire to get better and better at something, to make progress and overcome challenges.
  • Purpose
    The yearning we have to work in the service of something greater than ourselves.

There was some interesting research reported in the Jan-Feb 2010 Harvard Business Review that supports the power of mastery in motivating employees:

“On days when workers have the sense they’re making headway in their jobs, or when they receive support that helps them overcome obstacles, their emotions are most positive and their drive to succeed is at its peak. On days when they feel they are spinning their wheels or encountering roadblocks to meaningful accomplishment, their moods and motivation are lowest.”

Deloitte’s 2009 Shift Index had similar findings about autonomy and mastery:

“24 percent responded that flexibility, freedom, and autonomy were the reasons they “loved their job.” Similarly, 23 percent of the respondents cited challenges and opportunities for problem solving and creativity as the reasons they loved their job.”

These keys can all be found within the TPS, and are the foundation of any great 21st century business. If they can work for assembly line workers in one of the world’s largest car manufacturers, then surely they can work in our office environments?

More to the point, if we don’t offer work environments that have these keys, then we can expect to see more of our best and brightest leaving to go freelance, or joining that new startup, or even finding a big company that gets this idea.

Getting a Leg Up from Enterprise 2.0

I’m a Technical Director, so of course at some point I want to see just how technology can help us, and in this area as so many others there are some interesting possibilities thanks to Enterprise 2.0.

Original photo from istockphoto.com

Learning

Modern creative work places a huge demand on people to be constantly learning, diversifying their experience and increasing their speed of uptake of new ideas and skills. More then ever before success demands that people self-educate themselves, and with information overload and the skyrocketing complexity of our global business environment (where your competitors may just as easily be half a world away as down the road) that is harder than ever.

Enterprise 2.0 offers lots of help in the area of learning and knowledge transfer.

  • Wikis
    These are great for helping people record their knowledge, and then allowing that knowledge to be reviewed, improved, updated and generally kept alive.
  • Blogs
    Fantastic for putting ideas out there and getting feedback on them. They can provide employees with acceptable avenues for bridging great gaps in relative seniority (e.g. letting a new hire question the corporate vision statement) as they can allow people other than the original author to respond and correct mistaken ideas. The ability to browse posts by chronology and/or tags means that content is more usefully categorised than it would be in an email inbox.
  • Social Bookmarking
    Whilst bookmarks can be overrated, if properly categorised and rated, they allow people to make new information available to their colleagues and can act as scent as to who is most interested or interesting.
  • Enterprise search
    Having an effective and useful enterprise search tool is a must-have Enterprise 2.0 tool. Most Enterprise 2.0 content is unstructured and a good search tool is essential to ensuring that people find the right content when they need it.
Networking

There is a need for us to connect with ever greater numbers of people, and being increasingly time-poor to find the right people to connect with. Research has shown that high performing individuals maintain better networks than others, and in particular that their networks are more diverse/disconnected. This allows them to “see the big picture better, generate innovative solutions by integrating the expertise of those with unique backgrounds, position their efforts well, bypass bureaucratic gridlock and obtain necessary resources and support.”

Enterprise 2.0 makes it possible to find the right people in your network by reviewing their ideas and following the commentary and discussion about those ideas. It also helps people develop connections beyond their immediate team or management hierarchy.

  • Corporate directory
    Having an active and useful corporate directory, one that tracks projects worked on, positions held and teams an individual has belonged to can help identify key people.
  • Employee pages
    Giving employees their own page to talk about themselves can be risky, but for those passionate creatives it can offer a way to get their ideas out there, attracting others that have similar interests and passions.
  • Blogs
    The commentary and discussion facilitated by blogs, along with their attachment to individuals (and hopefully the corporate directory), makes them an ideal place to connect with others and find out who else you should connect to.
  • Yammer
    Internal tweeting tools like Yammer offer interesting possibilities as people can follow who they like within the organisation and find out who is following them.
Managing Autonomy

One of the main reasons people fear worker autonomy is that they will lose oversight of what is going on. Obviously implementing transparency properly means this is not an issue, but there are problems with managing transparency when collaboration is limited to email inboxes and offline meetings.

Enterprise 2.0 offers ways for collaborative work to be both done at a distance (of either time or place), and to be recorded in ways that allow for relatively easy monitoring by managers who want to help ensure that responsibility is taken seriously and known problems are not repeated.

  • Wikis
    Tracking progress on a wiki gives managers insight into what is happening and an avenue for querying ideas and outcomes.
  • Online document collaboration
    Collaborating on a document online makes the draft version available to people beyond the immediate team, often in a read-only version.
  • Basecamp
    online project management tools like Basecamp are highly regarded for their ease of use, simplicity and ability to facilitate complex project discussions and task workflow.
  • Google Wave
    Still in Beta, and with real performance and usability problems still, Google Wave offers interesting possibilities with low-overhead asynchronous conversations organically evolving into rich document collaboration opportunities and even real-time conversation.
Making Progress

We have seen that a key motivator is the sense of making progress, achieving real results and conquering challenges. One way our current systems can rob us of this is by interrupting our state of flow and causing us to drop out to check something that turns out to be unnecessary – it is estimated that a 1 minute interruption kills 15 minutes of productive time. Two deadly forms of this are unimportant emails and face to face social desk visits.

Enterprise 2.0 tools can help reduce the amount of email flow by moving items to more useful forms of communication that rely on a pull method of distribution (where users choose to use it), rather than the invasive push method employed by email.

  • RSS feeds
    By eliminating the push nature of methods like email, RSS feeds give a generic method for tools like blogs and wikis to make their data available for the user when they want it.
  • Yammer
    By bringing micro-blogging to the enterprise tools like Yammer remove the instant interruption hassle of instant messaging (IM) and provide an outlet for broadcasts of ideas that may be useful, but don’t need to be considered immediately.

Enterprise 2.0 tools can help move social interaction away from invasive face to face time in the office to more asynchronous means, or at least ones that include less wasteful interruption time.

  • Yammer
    Social engagement via micro-blogs like Yammer is ideal because it is asynchronous, based on the push/broadcast mode of communication and carries an implicit understanding that it is not important content.
  • Intranets/Wikis
    Placing social event information online, and providing interactive photo galleries can help reduce the amount of social interaction that must occur face to face.

In Summary

Deciding to manage work empirically, through transparency, inspection and adaptation can help give your employees the sense of autonomy, mastery and purpose that is necessary to untap their passionate creativity.

There are key ideas I have not covered here, such as the willingness to accept failure as learning experiences, the need to align authority with responsibility, and responsibility with ability that must be addressed for this to work well.

However untapping passion is not only possible, but well within our reach. It fundamentally depends on the willingness of senior management to set the direction, free their people and unleash their own passion.

Finally, Enterprise 2.0 offers some interesting tools to help manage and enable this change in our organisations. But as ever, technology plays an enabling and supporting role rather than being the driver of true change.

Thanks

There is a lot packed into this particular post, but a few people were key in helping me understand these issues.

Jason Yip, from ThoughtWorks Australia, whose passion for Lean got me interested in looking at Agile software development and whose ideas and blog have helped me work out my own thinking.

Lachlan Heasman, also from ThoughtWorks Australia, who runs the Sydney Scrum Meetup, and is a good friend and great Scrum trainer.

Craig Bailey, who in a year as my boss ignited hope in me for great managers once again, and who has gone on to embody following your passionate creativity wherever it may lead.

Chris Scoggins, my first great manager and a great example to me of how to mentor and build leadership into your staff.

Daniel Pink for bringing the hidden truths about motivation into plain view, and pointing out the Emperor’s New Clothes of management.

My Dad, for always making me question what is true, and helping me understand my own passionate creativity.

Thursday, March 04, 2010

Sydney Scrum Meetup (March)

Jeff Sutherland and Jens Østergaard visited the Sydney Scrum Meetup last Thursday. We had some great pizza and then they got everyone doing the Nokia test!

My Test Results

Iteration length, 2 weeks – 10

Testing, customer acceptance testing – 8

Agile specifications, poor user stories – 4 (good enabling specifications might be 3-5 pages long – before sprint planning)

(for venture companies Jeff finds that 2 sprints planned is necessary)

Product owner, product owner with backlog – 5 (or 2 with last project!)

Product backlog, single product backlog - 3

Estimates, backlog estimated by BA – 0

(Jeff said that usually as teams get better their stories get smaller and eventually are about the size of tasks, and then estimating changes to use points vs hours – surprising what can be untangled and done separately)

Sprint burndown chart, no chart, but team knows velocity – 0 + 3 + 2

(Jeff mentioned that partial completion of tasks creates a high-risk environment)

Team disruption, project leaders telling people what to do – 3

(self-organise to maximise velocity)

Team, no emergent leadership - 1

Total = 4.33

Most of the room were between 2 and 5, and apparently, most of OpenView’s venture companies start out around 4, but when they work on it they end up around 6. It looks like TIDC might have a great Scrum team as someone from there scored 8 for their team! (they were the only one higher than 5)

If your reference stories change then so do the story points. But Jeff said it is really the delta in velocity that is interesting. One of the symptoms of hyper-productive teams is that you get asked to slow down!

  • Sustainable pace
  • Quality
  • Velocity
  • “Balanced Scorecard” (not right term, but Jeff couldn’t remember what it was)

Jens recommends the XP game for teams to learn about velocity.

How long does it take to fix a bug from CI/automated testing. If longer than 2 hours then create an impediment list and get the Scrum master to remove them. Simple metric, track the day you start a story and the day you finish a story and then compare to your standard process efficiency (usually around 20% – measures the quality of your backlog) {calc elapsed days versus theoretical ideal days based on velocity}. Raising the efficiency meant getting the backlog well enough detailed that you spend less time waiting to do a particular item. Measuring the time to Done.

Speed of testing is usually the bottleneck, and is more important than the speed of coding. So interrupt devs immediately with a bug, rather than leaving them to finish what they’re doing. The availability of testable code is another way of thinking of it.

Just evaluate the code in production is one way of incentivising people.

I asked about how to handle client projects using Scrum, Jeff mentioned that Systematic is doing big fixed price contracts using Scrum. They provide 2 bids on every project, with the Scrum bid at around half the price.

In response to someone’s question about Scrum, Jeff mentioned he’d written about the answer in an article titled Future of Scrum from the Agile 2005 conference.

Metrics to Live By

Jeff got me hooked on metrics as it is obvious when talking to him that empirical process control requires metrics to be successful. Some of these metrics assume that you use story points for estimating, which we have done with some success on one project. To do this well you need a range of reference stories of various sizes and rough sizing (e.g. Fibonacci numbers) that the team can agree on the size of.

Here are some of the metrics Jeff mentioned:

Backlog Velocity = story points/sprint

Sprint Velocity = story points ‘done’/day

Efficiency = story’s ideal days/actual elapsed days from ‘start’ to ‘done’
(if an item should have taken 2 ideal days, but it actually took 10 days from the ‘start’ to the ‘done’ date, then you have 20% efficiency)

Churn = % of ‘done’ items that testers send back to developers for fixing
(if all items churns once then you have 100% churn, if half of those churn again then you have 150% churn! Ignore size of item when calculating)

Jeff has an upcoming conference session on this very topic and he discusses it further in his Excel Spreadsheet for Hyperproductive Scrum Teams post. The spreadsheet seems a little hard to get into, but I’ve given it to our Solutions team to review.

Friday, January 29, 2010

Do I Need User Stories?

How do you create a worthwhile specification of the requirements for a system?

As an Agile Scrum practitioner I’m supposed to answer this by pointing to user stories, but as others have found before me, these are a slippery beastie to define. Cat Schwamm takes a stab at a definition, but finds Scott Bellware telling her she is following Scrum too blindly and should go change what she’s doing.

Scott’s point of view is interesting, but like a lot of his comments he is coming at the problem from the frame of his lean methodology, which is not fully explained and therefore extremely hard to argue with/understand. This is not Scott’s fault, he is after all busy consulting not writing Agile methodologies, but it does make his commentary a tad alarming when one first reads it.

The classic format of a user story is:

As a [user] I need/want [feature] so that [benefit].

Or as Cat puts it:

As a [role] I need [feature] so that [business value].

Cat points out that this is fairly barebones, and leaves lots of scope for you to choose the wrong level of granularity for your user story. She points out that her team also follows the INVEST acronym and use the concept of minimum marketable feature (MMF) as defined by James Shore.

INVEST = Independent, Negotiable, Valuable, Estimable, Small, Testable

MMF = “Smallest possible set of functionality that, by itself, has value in the marketplace”

This is obviously working for her team, but I suspect they benefit from having a single Business Analyst who works everything through for them and ensures some consistency across user stories. This is actually very Agile, as the manifesto points out we value “individuals and interactions over processes and tools”. However, we still need some processes/tools for those individuals.

Scott Bellware’s suggestion was to change the format to:

As a [role] I want to [goal] so that [motivation].

He wants us to avoid mentioning the software altogether and have the user story simply capture what the users actually do (or will do once they have a system to help them). He then divorces the user story from work items altogether, and simply relegates them to the role of being information about the goals of the system. This may work for him, but it then begs the question of how he defines what the work item is all about, which is where we started talking about the need for user stories …

I have been reading a great book called Designing for Interaction by Dan Saffer that gave me another viewpoint on this issue. One of the things I learnt from this book was that user-centred design (UCD) is not the best way to design interactions, it is simply one of the possible ways you could design them and its relevance depends upon the goals of the thing you are designing.

Dan Saffer lists 4 different approaches to interaction design:

  1. User-Centred Design
  2. Activity-Centred Design (aka Use-Centred Design ???)
  3. Systems Design (aka Service Design???)
  4. Genius Design (aka Rapid Expert Design???)

Cat’s user stories actually sounded like activity-centred design, and Scott sounds like he is more interested in user-centred design. Quite possibly they are both right in the contexts they are working in. Neither one may be better than the other (Scott at this point would probably call BS on me, he has previously talked about the need for user stories to communicate expectation).

Coming back to my own issues with user stories, I need to ask what sort of design approach am I taking, otherwise I will find my user stories being all over the place. Is the user story meant to encapsulate the MMF in a single story, or is it supposed to be used to specify all ways of looking at the functionality.

As an example, when looking at the search function for administrators of an organisation I have specified:

  • As an admin I want to be able to search for a member by their Member Id so that I can find a specific member quickly.
  • As an admin I want to be able to search for one or more members by their Member Type so that I can target an action against all members of a specific type.
  • As an admin I want to be able to search for one or more members by a wildcard search of first and last names so that I can find a member just by knowing their name.
  • As an admin I want to be able to search for one or more members by region or industry so that I can find members from specific industries or regions.
  • As an admin I want to be able to search for one or more members by a wildcard search of job title so that I can find members with specific job titles.
  • As an admin I want to be able to search for one or more members by member status for my organisation so that I can target an action against all members of a specific member status.
  • As an admin I want to be able to search for one or more members by activation status so that I can target an action against all members of a specific activation status.
  • As an admin I want to be able to conduct a search against multiple facets of a member at once (e.g. by member status, industry, member type and job title) so that I can get very specific search results.
  • As the system I want to give admins a common page which allows them to search for members for all sorts of reasons and then target actions against the members found.
  • As the system I want to ensure that admins can only ever see their organisation’s own members in search results.

You can see the format slipping towards the end of that list, with the “so that [benefit]” element disappearing. There is also no regard to the INVEST principles, particularly the independent one. If I adopted another (MMF?) approach I might reduce all of that to:

  • As an admin I need to be able to find the members I’m interested in from my organisation so that I can work with those members’ details.

Scott’s format might leave me with:

  • As an admin I want to work with my members so that I can help them use our organisation’s online tools.

I think this demonstrates that there is a lot of variance possible within this simple definition of user story. In the past I think I have over-specified functionality with user stories, leaving my developers with a weird tension between overly specific (and hence out of date) user stories and little or no supporting documentation around other design issues.

This actually speaks to the N in INVEST, as Cat explains it:

“Negotiable: User stories are not a contract; they are reminders for the team to discuss and collaborate to clarify the details near time of development

What a user story really is is a placeholder for a conversation.  We’ve gotten away from writing strict, rigid use cases and into these user stories, which are much better for the way the business world really works.  By writing a user story this way, I am opening up a higher bandwidth conversation.  Instead of a developer just reading what I have and doing it word for word, now we can talk about things and make sure everyone is on the same page and fully understands the business value and the needs of the user.  I like that Kelly specified “near time of development” though; that’s one thing you’ll always find: CHANGE IS A CONSTANT.  Friggin annoying, but it is inevitable.  The closer to development time you talk about something the more likely it is that you’ll have the best information and the most clarity on the item.”

I especially like the idea of a user story as a placeholder for a conversation. Of course this depends on your developers actually being willing to talk to anyone about the requirements (sometimes they still seem to think that it is faster to code away on assumptions and correct later than to have a 30 min conversation up front and ensure you know what you are doing!).

Scott Reynolds works with Cat, and he points out that not everything can be captured in user stories:

“Here’s the thing, stories and tasks are only part of the equation. Stories alone aren’t enough to define a system, and trying to define everything in text is a fool’s task. (I’ve been that fool). You need a full arsenal of specification tools to do the best job possible.”

I think I’ve been the fool Scott mentioned myself (once or twice). Scott (and Cat) mentions 3 other tools that they use as well as user stories:

  1. Mockups
    Basically a low-fidelity prototype. They are great for eliciting feedback about visual design and interaction issues without leaving anyone feeling like these ideas are final. They are poor for actually designing off of as they frequently ignore or skimp on the design of areas that were not pertinent to the original design issues being explored in them.
  2. High Fidelity Design
    High-fidelity screenshots showing exactly how the screen should look. These are great when you need signoff on something from outside the development team as they show everything that needs to be taken into account. They are poor when the design needs to change and you don’t have time to re-do those rich initial screenshots for the change request. Actually they are poor for Agile, full-stop, as they reek of BUFD.
  3. Conversation and Whiteboarding
    This is the higher bandwidth conversation that Cat referred to. This is a chance to meet, collaboratively engage each other and work out what the business need for this feature is. The caveat is that it is only as good as the conversation and whiteboarding ability of your developers, and it requires that the customer be available immediately when the developers need them (this is why Agile doesn’t work for many projects).

There are other good ideas for artifacts to capture interaction design ideas, such as:

  • Acceptance Tests
    Here I mean a light version of the full test scripts that are possible. Ones that match user stories come in the form:
    Given that [context], when [event/action], then [expected outcome].
    Like user stories they have the same problems with being at the right scale.
  • Key Scenarios
    Penny Hagen describes these as combinations of user stories that help define the strategic direction of the website. Whilst she uses them as a tool to help define design strategy early on, they have a useful purpose in helping developers understand the context of the user stories they are working on.
  • User Pathways
    Again Penny Hagen describes these as a detailed view of the pathway a user needs to follow for a key scenario. They don’t explicitly restrict themselves to pages, or data, but they do help identify the user’s view of what is happening when they interact with the system – and give insight into the user’s experience over time.

Personally I’ve used mockups, high fidelity design, conversation/whiteboarding and acceptance tests to varying levels of success. One of the problems you face in Agile development is that the more artifacts you ask for, the longer it takes to go from designing to building – and the greater the chance that your artifacts are out of date and, frankly, wrong.

I’m still not sure about how to set my user stories, I sure appreciate the discipline they grant even quick feature specifications, but I’m loathe now to simply bang them out when the reality is that I might need to use either less, or more, detail for a given feature. I’m going to keep experimenting because that’s what Agile is all about. In the meantime let me know if you find something better to use!

Thursday, November 19, 2009

When Agile Doesn’t Work

imageI am a huge fan of Agile software development and since becoming Elcom’s Technical Director I have made it my business to push us to become more Agile at every step. But a few months ago I realised I was pushing a square block into a round hole.

It started with a realisation that most proponents of Agile software development had a common single clause that modified their promises of great results.

Customers must get involved for Agile to work.

In Scrum it’s the Product Owner, and it must be the single wringable neck – the person accountable to the business for the project’s delivery of real value. It is the person who not only understands, but can make decisions about functionality, business needs and the user interface.

Oops.

While this sounds great, and you would think that people paying you tens (if not hundreds) of thousands of dollars would want to get involved, we had found all too often that big promises of ongoing involvement disappeared as soon as the project kicked off. Clients wanted to tell us what they wanted, approve our quote, pay us the money, and then pretend we didn’t exist until we had finished their project. Or they had a committee of thousands that wanted to give us their pent up IT requirements, regardless of whether it was in scope or not, and often in conflict with each other.

To be fair to our clients, we often are simply implementing our software and the amount of customer involvement is quite low – however that didn’t fit the Agile methodology we wanted to work within.

Things became clearer for me when I attended a ThoughtWorks Quarterly Technology Briefing in Sydney, and got the chance to hear Martin Fowler speak about how to fail in Agile adoption. During the Q&A I asked him how he suggested we handle customers who were reluctant to get involved.

Martin pointed out that there are in general two sorts of projects:

  1. Strategic: Key to business success, and as such important.
  2. Infrastructure: Necessary to do, but not seen as particularly vital for business success.

Immediately I saw that for many of our clients their projects were internally seen as necessary (new intranet, re-skinned website) but were not particularly risky or impactful on the organisation’s success. I mulled that answer over for a while and wondered what it might mean for our use of Agile.

MaisterProfessionalServicesFirm A few weekends later I was re-reading David Maister’s classic book Managing the Professional Service Firm and came across his definition of the three basic types of professional service delivery. He writes for any type of professional service and uses examples from disciplines as far apart as architecture, accounting, engineering and management consulting to make his point.

Brains/Expertise

  • Client problem is at forefront of professional or technical knowledge
  • “Hire us because we’re smart”
  • One-off projects
  • Highly skilled and highly paid professionals

Grey Hair/Experience

  • Clients require highly customised output, more familiar problems
  • “Hire us because we’ve done it before”
  • Specialised knowledge in project area
  • We sell our knowledge, experience and judgement

Procedure/Efficiency

  • Client problems are well-known and familiar, solutions also well-known
  • “Hire us because we can do it for you”
  • Repeat projects, well understood, standard solutions
  • Well trained and supervised junior staff

image

Maister points out that you can see a continuum from Brains to Grey Hair to Procedure for a range of different service features.

image

Using this idea, I could see that most of the “infrastructure” projects we had involved procedure or efficiency services. A simple deployment of CommunityManager.NET with implementation of a specified basic website design and some custom styling of forms and other modules. Agile simply slowed these projects down and was generally ignored by the team doing most of this work.

On the other hand some of our custom development work fell into the realm of grey hair or experience services – enhancing or extending CommunityManager.NET for a client. These usually did not need to be Agile projects, but were being forced into our product sprint cycle, even when they were a day or two of custom work not dependent upon enhancements to our core product.

Our product work deserves a mention here. Whilst not for any specific client, this is in fact the work that provides the greatest return on investment for the business. However a need for cash flow meant there was a limit to how much could be done at any given time. We had a settled sprint duration of 6 weeks (5 weeks dev + 1 week regression testing and release preparation), wit 1 week in between sprints, that the team had a good rhythm with. Agile was a great fit for this work.

After this analysis there was a small, but important part of our business. Small in terms of number of projects, but important in terms of revenue. This was the projects that involved significant custom development work and had little dependency upon CommunityManager.NET. Sometimes these jobs could have been delivered without our product at all, but using it gave a convenient head start to the project. These projects were strategic to the client, but as a result often had loose, or non-existent requirements because it was acknowledged that the requirements would change during the course of the project. This type of work suits Agile methodologies, but the business needed to recognise that this work shouldn’t be sold the same way our other services were.

Clearly Agile suited some parts of our business, but was a poor fit for others. Our sales material made a big deal of our Agile methodology, but it was not always relevant or appropriate for our clients. Worse, working in an Agile manner was causing many projects to have a greater chance of failure, and to delay client solutions.

Agile Wasn’t Working

I now knew why Agile wasn’t a great fit for much of our business, but it still had value in some areas. Maister also made the point that when for a professional service firms to mix Brains, Grey Hair and Procedure work successfully it is necessary to create divisions within the firm to allow each to have a management style and culture that fit their style of work.

I really like the way that Jurgen Appelo organises his development teams, with a project manager assigned 3-5 other staff as their permanent team, with them needing to complete each of their projects using that team. By pushing resource constraints to the lowest level (that of the project team) they ensure that other projects’ resource variances do not adversely affect them. However I know that Elcom is much smaller than Jurgen’s company, and such a profound shift would create more friction than it would generate in gains.

I discussed these ideas with my boss and some of my colleagues, but felt that we quickly needed to make a change as a large custom development project was about to kickoff its second stage.

The team structure we are now heading towards is this:

Delivery Team

This is a team made up of web designers, front-end web developers, and .NET analyst/programmers. Their role will be to implement new customers’ versions of CommunityManager.NET as painlessly and efficiently as possible. They will handle custom development work that is tightly integrated to, or built around, CommunityManager.NET. They work in traditional BUFD (big up-front design) projects with well-scoped requirements. Project duration would usually be anything from 1-8 weeks.

Solutions Team

This is a team made up of user experience (UX) consultants, designers, and web developers. It is the renaissance team that is responsible for finding new ways to deliver innovative solutions for strategic client problems using design thinking and Agile software development methodologies. They work in a more flexible manner, preferably with a pseudo time and materials approach, with the client purchasing entire sprint’s worth of work. Project duration would usually be more than 12 weeks, with usually two week sprints.

Product Team

This is a team made with a web designer, front-end web developer and .NET developers. It is responsible for enhancements to, and strategic development of our products, particularly CommunityManager.NET. They follow an Agile software development methodology and are very much the traditional product-centred Agile team. Sprint duration would depend upon the ability to regression test in a timely manner – but for the moment this would be around the same 6 week duration we had used previously.

Right Now

When we first made this change, there wasn’t enough strategic product work for us to justify having a dedicated Product Team. Instead members of the Delivery Team are made available as and when required to work on the product. We still have 6 week product sprints.

The Delivery Team has started transitioning away from Agile, and their time is now more easily shifted from one piece of client work to another (they previously were committed  up to 6 weeks in advance for a particular sprint). Our clients have started to notice that we have become more flexible and responsive to sudden requests. Our project managers have been learning a new resource planning tool which is slightly clunky (built in Excel, requiring double data entry) but that we soon hope to replace with a more effective system.

The Solutions Team has had some success working in a more Agile manner, but they need to learn to do it without the moral, emotional or practical support of the rest of the technical teams. By removing many of the restrictions that product work placed on them, they are now free to explore new technologies and test them in practice before we roll them into our products. For instance they have used Domain-Driven Design (DDD) and LINQ2SQL.

The teams are differentiated by more than just name. They have different KPIs, different management styles and different things to get good at.

The Delivery Team needs to become more process-driven. Too many things are left to individual choice when the best solution is already known and has been tested in practice. Also documenting requirements up front is a skill set that needs to be re-discovered and applied. After all if you are going to do BUFD projects then you need to be good at it.

The Solutions Team needs to become more Agile, and look at ideas like TDD/BDD and more comprehensive automated testing so that short sprints do not lead to a reduction in quality. Selling a job as an Agile development project is proving harder than we would have thought. It seems lip service to Agile is easy to sell, it sounds good and requires few changes in the way clients work. Real Agile development requires an altogether different mindset, and this is much harder to sell up front. We may well need to lead these projects with an initial fixed price project that uses BDUF, and then transition to a more Agile approach once we have some credits in the trust account with our client.

The Future

The Solutions Team will be undertaking a major strategic product initiative for Elcom over the Christmas period and will bring their Agile approach and technical expertise to bear in order to give us a major new element to our product for CommunityManager.NET version 7.0. After this we hope to have another major project lined up, but if not then they may become our Product Team.

If another Solutions Team project does come up, then the next step will be to create a full-time Product Team. We believe that version 7.0 will create enough demand that a full-time team will be justified. An important part of this strategy is to find partners that are comfortable enough implementing CommunityManager.NET that our Delivery Team is not a bottleneck on our sales. One of the keys to version 7’s success will be the way that it makes it far easier for digital agencies and systems integrators to use our product for their own web projects.

After that fame, fortune and fun might consume our days – or more likely we will find ourselves looking beyond enterprise content management to create a new market niche in the manner of good Blue Ocean strategists. Whatever we do, Agile will remain a part of it, but only when it deserves to be.

Tuesday, August 11, 2009

Software projects are like sailing voyages

I have been reading Greg Smith's new Becoming Agile book and was struck by the false analogy that is often applied to software projects, which compares them to construction projects, where Smith points out you typically "gather requirements, create a design, excavate, lay the foundation, and then build the walls."

Smith points out two problems with applying this analogy:

"1. Additional requirements will be discovered during design, coding, demonstration, and testing.
2. Software development does not require completion of each step before the following step can be initiated. You can start design and coding based on a few initial requirements. You can iteratively build out the system, revisiting it and going deeper on requirements, design, and code as you make discoveries and refine the vision for the project."

Analogies are dangerous because they build in these sorts of inaccuracies, and that can lead to dangerous assumptions and attitudes later on. However they are excellent for helping non-experts in a field understand complex topics. I dearly want a better analogy for software projects and I have never heard a really satisfactory one, so I got the mental juices going and looked for one that makes sense to me. This is what I found ...

Software Projects are Sailing Voyages

Software projects, at least the ones I have been involved in, feel more like ocean voyages in small yachts than anything else. Like any sailing voyage you will have an idea of where you want to get to, and if you are smart you will have checked the weather forecast and planned your route - even calculating what you average speed should be and when you will reach your destination.

3100906233_d9fa131d23_b

from: http://www.flickr.com/photos/14132971@N05/3100906233/ 

This is reasonably similar to the planning and preparation for a software project where a goal will be set, the estimated time to reach it mapped out and the current environment and conditions researched.

However, in sailing you can expect to find a number of things changing your plans. Perhaps the wind changes in unexpected ways, or your plan failed to take into account the wind shadow of a particularly large island, or the strength of local currents or tides. The helmsman may have neglected to keep his eye on where you were headed, being distracted by a pretty sight, or other crew may not have kept the sails properly trimmed or closed the bilge valve.

2791384256_cb84d28653_b

from: http://www.flickr.com/photos/flasporty/2791384256/ 

Mishaps can occur, such as equipment failure, collisions with other vessels or even crew sickness or injury. Your destination port might be closed, and you find that you need to make landfall either sooner or later than planned. Even the captain's intent may change as it becomes apparent that the passengers would prefer a leisurely cruise around the coast to a deep ocean adventure.

2691809503_bd8ec4d118_b

from: http://www.flickr.com/photos/wili/2691809503/ 

Very similar changes can be expected in software projects. Your tools vendor may bring out a new release that opens up new development options, business goals can change or be re-assessed, leading to changes in requirements. Staff changes can cause a change in vision, or highlight mistakes or negligence in the work previously performed. Individuals' performance may fall short or exceed expectations for their role, and unexpected leave or changes in priorities may derail timelines.

These problems occur on construction projects too, but they have a real, physical object they have been developing with building approvals already acquired so it is far more likely that they will simply continue to the end (even then, you might end up with a project like Sydney's World Square, where it was just a huge hole in the ground with some cement in it for years).

Hole-in-the-ground

from: City of Sydney 

With software it always seems easier to throw it all away and start again (and indeed this is sometimes the best course to take). Simply having a shared analogy with the customer that makes more sense than a building will help them understand that while we can give up the voyage now, we can't avoid the fact that we have sailed this far, and perhaps we should take that learning and make use of it.

Agile software development in this case can be seen as a series of small hops around the coast, rather than the usual grand exploratory voyage undertaken by many traditional software projects. Each hop allows us to review where we have gotten to, and lets passengers get off if they so desire.

Of course analogies have their limitations, and using them to model a complex system by comparing it to another simpler, or more familiar, system does not guarantee success. It just makes client meetings shorter (which by some definitions is success!).

Friday, November 07, 2008

My first Agile retrospective

I facilitated my first Agile retrospective today, going over a six month project with 12 people, mostly from Elcom's development team, along with one or two others from the systems admin, project management and implementation teams. We followed the format suggested by the Retrospective Goddesses in their Agile Retrospectives book. It was an interesting and rewarding experience, so I thought I'd document it here.

P1050666

Alan Lee, Rajesh Stephen and Chae Wan Kim do some group work.

This project had been a large and challenging one for us, whilst still a success for the ultimate client. It had sucked in most of the development team at one point and I was one of the few people not directly involved. When our management decided to run the retrospective they had envisaged nothing more than a brainstorming session and opportunity to vent some emotions.

We had purchased the Agile Retrospectives book some months ago and I suggested that our Technical Director run it using the book - except it had somehow gotten stuck in my bookshelf at home. Whilst bringing it in I flicked through it and noticed that the main facilitator role sounded exactly like the facilitator role I've been fulfilling in the Valiant Man courses I've been involved in this year. So I volunteered myself as the facilitator, negotiated to increase the time from one to two hours and promptly got stuck with all the preparation involved!

Working Agreement

As the person with the most experience of this sort of group work I suggested most of the agreements, but the group did come up with "Don't interrupt others" on their own. Going "topless" was particularly hard for the managers in the group - but I think it really helped them engage with the process.

P1050656

Goal

This was an interesting one, I had selected a reasonably neutral, positive (and useful) goal by focusing on one of the unusual characteristics of the project, our work with a partner organisation. But the team thought that added to this should be a focus on big projects as this was another defining characteristic of this particular project.

P1050658

I then got everyone to give us a quick one or two word description of their feeling or state at the end of this project. This was interesting as we got flat out positive and negative responses as well as mixed ones like, "stressed, happy." The range of emotions described was a reminder that not everyone sees things the same way.

Information Gathering

I chose the timeline method to draw out details from the group as that would allow us to vary the amount of time we spent on it, and for a six month long project would help people remember their experiences.

P1050660

Tracey Schneider, Gavin Silver, Balaji Sridharan and Chae Wan Kim place their cards

The group broke out into three groups of 4 and worked on identifying important events in the project timeline, categorised into client (blue), Elcom (green), people/team (pink) and technical (yellow). I had previously prepared the whiteboard by separating it into months, and including the number of days of developer time used in each one.

We ended up with a very interesting overview of the project that included many events that people had forgotten, or not even known about, as well as some that everyone was affected by (we stacked duplicate cards on top of each other). I mostly stood back and watched (that's me below, closest to the camera), and occasionally stepped in to help people understand how to handle duplicates (stacking), confusions with long-running events (put multiple cards across months at same level) and disagreements about what was an event (pretty much anything could be).

P1050663

Craig Bailey, Tracey Schneider and Gavin Silver check out the cards, watched by me.

Once each group had their cards up, I asked people to review the board, one group at a time, and discuss what others had identified, as well as looking for duplicates or events still missing. That only got us one or two extra events, but gave everyone a chance to see the total timeline up close.

I then told each participant that they had five positive (green) dots and five negative (red) dots to apply to cards on the timeline - but they could only put one dot on each card. They came up in pairs to dot the cards. I wanted to use colour coded dots, but we made do with red and green pens. You can see the result below:

P1050671 - Copy

People wanted to know how to identify what was positive, or negative, but I pointed out it wasn't supposed to be scientific, just a subjective review of their feelings about the events listed. That worried people less than I thought it might.

Once they had all finished I took over and went through the events that seemed the most interesting (lots of green dots, red dots, or a mix). I also pointed out events that were usually regarded as important, but that ended up having little significance to the team.

Some of the event cards turned out to be confusing, so I also queried what they meant and made sure I added text where necessary to help clarify their meaning. One of the events that had a real mix of dots, turned out to be interpreted two completely different ways, so we drew a line between the dots and added text to each side to explain further.

Identify Experiments and Actions

Unfortunately time was short for this retrospective, so I had planned a quick 5 Whys activity to help the team quickly find useful insights. They broke up into small groups/pairs (two groups of 3 and three pairs) and started asking why.

5 Whys has one participant ask the others in their group why an event was important, positive or negative. It was left pretty open for them to choose their favourite item from the timeline. Once they got the answer, then then asked why that was important/relevant. They would do this a total of 5 times, and only record the answers to the fourth and fifth questions. Every member of each group got a chance to do this at least once and we got some funny and insightful responses as a result (one one of the cards below the fifth answer was that the person was "speechless", so they drew an empty word bubble!).

P1050676

Next was the only awkward part of the retrospective (from my perspective). I had planned for people to do a vote on each item with one stroke per card, as many strokes as they wanted (hat-tip to ThoughtWorkers Lachlan Heasman/Jason Yip for that one!), but the criteria I wanted to use was "which item would you most want to work on". Indeed this is what I announced to the group, but I noticed when people actually came up to vote that the question/answer cards did not identify the experiment/action that could be taken to answer them.

We were over-time by this stage, so I made sure everyone knew what the problem was, but let them keep voting anyway. I then had to step in to organise the cards by stroke-votes (as shown above) and then run a quick discussion of what steps might be taken for each one.

P1050677The result was a set of 5 actions that were not really owned by the team. Coincidentally this was also when the more vocal and confident management types really shined, limiting input from others. To be fair (to myself and the team) this project highlighted some really basic issues in our project management, sales and quoting procedures - and these caused so much grief that it was a bit hard to see past them to other problems that might be owned by more of the team.

Summary

I'm really proud of how enthusiastic the development team were in approaching this. I did a Return On Time Invested poll at the end of the event and on a rating of 0-4 (0=No return, 2=Break even, 4=High return) most of the guys thought it was a 4, some thought it was 3 and a quarter of the group rated it as a 2 (one of those turned out to misunderstand what we were measuring and later said she would have voted 4 too!).

We should be making this a regular part of our Agile process and I will certainly be agitating for it occur more often. I'm sure that having more retrospectives will help identify some of the more fundamental issues that affect the development team and get us continually improving.

Wednesday, October 29, 2008

Salesforce.com's big bang Agile rollout

Whilst researching for next week's Scrum meetup in Sydney I came across interesting presentation by the Salesforce.com R&D guys as to how they transformed their internal development teams to Agile - all at one time.

Lachlan points out that the advantage of this approach is that it means there is no waterfall project running alongside the Agile ones and making them look bad (just because anything new will have its problems, and because Agile makes obvious existing dysfunctional behaviour).

ALT.NET Sydney - NHibernate and ASP.NET MVC

(Microsoft are definitely capitalists, just look at all the caps in that title!)

I went to the second ALT.NET Sydney meeting last night and really enjoyed myself. Richard Banks from ThoughtWorks is organising the event and pizzas, saucy chicken wings, beer and caffeinated beverages were kindly provided by ThoughtWorks (which looked like it had plenty of people working back).

After Richard hit off with the news and we ate/drank we had a joint presentation by Damian McLennan (NHibernate) and Ali Shafai (ASP.NET MVC). I think the point was to show each separately and then point out how easy NHibernate made the back-end in ASP.NET MVC - but we distracted Ali and had a more interesting general discussion about MVC.

I probably got more out of the NHibernate session as that was the piece I was least familiar with. We had a few people in the audience who have "been there, done that" which was great because it gave us some very interesting discussions.

See Richard's summary post on the group blog, or Damian's NHibernate post for more links.

Friday, July 04, 2008

LinkedIn's architecture

Via Oren Hurvitz's blog comes an interesting summary of two presentations that LinkedIn's engineers gave at the JavaOne 2008 conference. I've embedded the presentations below. The first goes into their architecture:

The second one deals more with their use of Java and agile practices:

It's great seeing details like this as it helps bring the rest of the industry up to date with how the largest websites in the world are actually handling the problems of performance and scalability. These are great for Microsofties to read too as the lessons learned apply just as well to .NET projects as Java ones.

Tuesday, April 22, 2008

Second ALT.NET conference

Ben Scheirman has a good couple of posts on the ALT.NET open conference that ran just after the Microsoft MVP Summit. Not being an MVP I wasn't anywhere close to going, so it is nice to see this sort of photo intensive report ... looks like fun and I'm hoping Code Camp Oz 2008 can be as productive for me.

Monday, March 10, 2008

It's the software that matters!

James Shore, creator of NUnitASP and author of The Art of Agile Development has recently announced that he's no longer supporting NUnitASP development, citing his increasing lack of interest in it and pointing interested parties to Selenium or Watir (or perhaps WatiN, although that has received some criticism from Scott Bellware, others are not so sure).

It's a shame we're not getting more choices in this area of testing, but James is right to focus on his own current projects. He has left it open for someone else to take over the project, although the way ASP.NET MVC will expose HTTPRequest as an interface might make it superfluous.

Back to the point of this post. James has a great blog with some very thoughtful articles. I found this great quote in It's the Software, Stupid! It summarises for me the problem with simply implementing Scrum and thinking that makes you Agile.
“your team is expected to self-organize and define its own practices”
“It's time we brought back the early emphasis on great engineering practices. If you're using Scrum or another agile method that doesn't include engineering practices, realize that your method is incomplete. Scrum, for example, intentionally creates an environment in which your team is expected to self-organize and define its own practices. If you aren't doing that--if you aren't talking about engineering practices, what's working, what's not, and how to improve--you're going to run into trouble someday. Probably someday soon.”

Tuesday, February 05, 2008

Interesting Links 2008-02-05

Madhav Rao has a great post summarising the unit testing tools currently available for .NET developers.

Chad Myers also has a great post explaining his development methodology, called I don't trust me.

I've been reading C.S. Lewis' The Abolition of Man, and have found a good version of it online thanks to Columbia University.

Finally, DevTopics has 101 great computer programming quotes (via Chad Myers, via DotNetKicks). Here are some of my favourites:
“The most amazing achievement of the computer software industry is its continuing cancellation of the steady and staggering gains made by the computer hardware industry.”
(Henry Petroski)

“Complexity kills. It sucks the life out of developers, it makes products difficult to plan, build and test, it introduces security challenges, and it causes end-user and administrator frustration.”
(Ray Ozzie)

“There are only two industries that refer to their customers as 'users'.”
(Edward Tufte)

“The trouble with programmers is that you can never tell what a programmer is doing until it's too late.”
(Seymour Cray)

“That's the thing about people who think they hate computers. What they really hate is lousy programmers.”
(Larry Niven)

“A hacker on a roll may be able to produce–in a period of a few months–something that a small development group (say, 7-8 people) would have a hard time getting together over a year. IBM used to report that certain programmers might be as much as 100 times as productive as other workers, or more.”
(Peter Seebach)

“The best programmers are not marginally better than merely good ones. They are an order-of-magnitude better, measured by whatever standard: conceptual creativity, speed, ingenuity of design, or problem-solving ability.”
(Randall E. Stross)

“Optimism is an occupational hazard of programming; feedback is the treatment.”
(Kent Beck)

“In software, we rarely have meaningful requirements. Even if we do, the only measure of success that matters is whether our solution solves the customer's shifting idea of what their problem is.”
(Jeff Atwood)

“Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are–by definition–not smart enough to debug it.”
(Brian Kernighan)

“As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.”
(Maurice Wilkes discovers debugging, 1949)

“I don't care if it works on your machine! We are not shipping your machine!”
(Vidiu Platon)

“Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.”
(Martin Golding)

Saturday, January 12, 2008

VB.NET is the illegal immigrant of Microsoft languages

I currently mostly code in VB.NET, developing ASP.NET web applications. I have in the past developed in Omnis, MS Access 2/95/2000, Lotus Notes/LotusScript, Java, VB6, VBA, PHP, JavaScript, C# and Ruby (on Rails). Some of those I did for fun, but most were because I was paid (and asked) to. I point this out not to boast, but in order to put some background around this particular rant.

[NOTE: Another accidental thing that denigrates VB is that we must put ".NET" at the end if we are talking about 'real' development, because VB as a term covers such a broad range of languages. This alone makes C# seem exclusive and more 'grown-up'. In fact C# is more properly referred to as Visual C#, or C#.NET.]

“Computer languages influence how you think about a problem, and how you think about communicating.”
The Pragmatic Programmer
Microsoft have a publicly declared agnostic view of which language a developer chooses to use within the .NET platform, so long as they use it. For this reason they have researched new languages for the CLR, and are introducing the DLR to enable IronPython, IronRuby etc.

Internally however Microsoft have a culture that prefers people using C#. In practice this means that the 'cheap' option is always to do code samples and demo code in C# and not VB (or some other .NET language), because it is simply more convenient to do so. When asked why a given example is not also in VB, this is the answer given (and the occasional plea for help). There is however a tendency to see VB developers as somehow inherently less capable that C# developers - perhaps this is a view that the 'best' work at Microsoft, or maybe it's a hangover from the "Basic" part of VB's name. Most probably it also has a lot to do identifying VB users with using a persona called 'Mort', whilst C# is Elvis and C++ Einstein (thanks usability guys).

A few months ago Bill McCarthy, a VB MVP, ranted about this issue in his post Is VB the n*gg*r of programming languages?. Leaving aside a controversial choice of words (for which he was smacked down), he had a very good point. My post's title is a reference to his point (with a more PC turn of phrase).

Bill points out that how to videos (for beginners?) are often coded in VB, but the SDKs have C# examples (for real developers?).

A more recent balancing view comes from Scott Hanselman, who asks Is rooting for Visual Basic like rooting for the Red Sox?. Leaving aside the unfortunately humorous connotations of 'rooting' to us Aussies, his point is that VB is making something of a comeback in popularity, but old-timers (like Bill?) still want to feel like underdogs. Reading the rest of Scott's blog shows that he's been exploring beyond the homely environs of C# and hanging out with people who are pragmatic enough to use whatever works.

Personally I think this last point is the one that makes the most sense. In The Pragmatic Programmer Andy Hunt and David Thomas said:
“Computer languages influence how you think about a problem, and how you think about communicating ... Designing a solution with Lisp in mind will produce different results than a solution based on C-style thinking, and vice versa. Conversely, and we think more importantly, the language of the problem domain may also suggest a programming solution.”

All of this is what has attracted me to the ALT.NET movement (see the first use of the term, the conference last year, it just had a leadership summit and a Yahoo! group). Martin Fowler weighed in and said this about ALT.NET:
“The alt.net mind-set is one that is very familiar to me. It has that mix of agile + object-orientation + patterns + TDD + DDD which is very much the school of software development that I favor. (Lacking a proper name for it, I'm inclined to call it the OOPSLA school of software development.) There is certainly a belief that there is a mainstream Microsoft orthodoxy at the moment, one that doesn't fit the OOSPLA school. And there's some frustration with that. But the point here is that it's not that the alt.net community thinks that the perceived mainstream Microsoft route should be erased, but that the Microsoft world is big enough for different approaches.”
To all this I say a big hooray! But ... it seems everyone who wants to do something with ALT.NET is a devoted fan of C# (or at least NOT a fan of VB). So every code sample, explanation or tool is aimed at C# and either requires much grokking to be used in VB, or in many cases, cannot be used with VB at all.

The answer I'm afraid is that I must use more C# if I want to become more agile, practice TDD or play with ASP.NET MVC. The fact that I get paid to work with VB.NET is not a good enough reason to stick with it. VB is the illegal immigrant of .NET languages, it snuck into .NET when it should have been told to stay away (perhaps forcibly deported) and the sooner any of us who use it realise that, the better. Shifting to a true .NET citizen like C# is clearly the only responsible thing to do - and using those Johnny-come-latelys like IronRuby or IronPython (or Boo) is a joke - they are only toy languages, not super-strong doers of great deeds like C# is.

Still, at least there is one hope on the horizon, with Silverlight assuming such importance in Microsoft's plans for the web, perhaps VBx will come to our rescue? With VB moving into the DLR perhaps VB developers can all become the new, new boys on the block, and start looking down on those poor C# types who are unable to duck-type or use Jasper. And my persona? Let's just call him Ben.

Monday, January 07, 2008

Partial classes and methods in VB.NET

For the current project I'm on we decided to spend a little time upgrading our code generation tool so that we could generate DAL classes more easily from our database tables. Part of this involved adding in extra loops to make the generated code bulletproof so we could rely on what was generated and not need to think twice about re-generating classes when we wanted to fix a bug or add a new piece of functionality.

To be clear, the generator would create seven files:
  1. Basic entity class (e.g. Group.vb)
  2. Collection class (GroupCollection.vb)
  3. A DAL class (e.g. GroupDAL.vb)
  4. Insert stored procedure
  5. Update stored procedure
  6. Delete stored procedure
  7. Select stored procedure to get a single record by primary key(s)
[Disclaimer: This is a legacy system, so field and variable names in sample code below are specified for me, and don't reflect my own thoughts on naming conventions.]

The first step was to add more smarts to the templates so that the generator could handle some difficult situations (multiple primary keys, non-identity single primary keys) and to add in some additional base functionality (collection sorting, object caching, OnValidate checking that the IList interface is not being abused).

For a good article on strongly typed Collection classes check out this one on Builder AU.

Next we looked at the range of customisation we already had in existing data layer classes and ways of separating this out so that we could add ways of getting data that meet specific business requirements without polluting our generated classes. The answer of course was to use partial classes.

“we nearly always need an additional partial DAL class to hold useful extra ways of retrieving data”
By declaring that our generated classes were partial public classes we could complete the rest of the class in another file that is not being auto-generated. Somehow I ended up christening these for the team, and where necessary we now have additional files (e.g. Group_Additional.vb, GroupCollection_Additional.vb, GroupDAL_Additional.vb). It turns out that we nearly always need an additional partial DAL class to hold useful extra ways of retrieving data (e.g. searches, find by foreign key, etc.), with regards to collections and entity classes it was less useful, but more on that later. We now had a way to bring in our legacy customisations, whilst still generating most of the code.

Our DAL classes now start like this:

    1 Imports System

    2 Imports System.Data.SqlClient

    3 Imports System.Data

    4 Imports System.Text

    5 Imports System.Collections

    6 Imports Elcom.Common.Data

    7 

    8 ''' <summary>

    9 ''' Data Access layer (DAL) class for Group entity objects.

   10 ''' </summary>

   11 ''' <remarks>

   12 ''' Auto-generated by Elcom Code Generator at 4/01/2008 4:01:03 PM.

   13 ''' </remarks>

   14 Partial Public Class GroupDAL

   15     Inherits BaseDAL

   16 

   17     ''' <summary>

   18     ''' Constructor method is declared and made private to ensure

   19     ''' class is a singleton.

   20     ''' </summary>

   21     Private  Sub New()

   22     End Sub

   23 

   24     ''' <summary>

   25     ''' Public 'Instance' method allows external objects to access

   26     ''' the current singleton instance.

   27     ''' </summary>

   28     ''' <returns>

   29     ''' Returns a reference to the single GroupDAL object

   30     ''' in memory.

   31     ''' </returns>

   32     Public Shared ReadOnly Property Instance() As GroupDAL

   33         Get

   34             If _instance Is Nothing Then

   35                 SyncLock GetType(GroupDAL)

   36                     If _instance Is Nothing Then

   37                         _instance = new GroupDAL()

   38                     End If

   39                 End SyncLock

   40             End If

   41             Instance = _instance

   42         End Get

   43     End Property


The DAL additional class looks like this:

    1 Imports System

    2 Imports System.Data.SqlClient

    3 Imports System.Data

    4 Imports System.Text

    5 Imports System.Collections

    6 Imports Elcom.Common.Data

    7 

    8 ''' <summary>

    9 ''' Data Access Layer class for Group

   10 ''' </summary>

   11 ''' <remarks>

   12 ''' Old code brought in as Additional partial class by Angus

   13 ''' on 4/01/2008 for CM 5.5.

   14 ''' </remarks>

   15 Partial Public Class GroupDAL

   16 



Unfortunately we were still left with a major issue, some of the customisation we had made involved making changes to the way we saved some data items (e.g. encrypting users' passwords before saving them to the database), or having additional read-only data items that were not instantiated in the database table but properly belonged to the entity. The partial class solution allowed us to add extra properties, but did not enable us to override the basic Save() method, so what were we to do?

One option was to implement an alternative Save() method that differed either via parameter signature or name, but that would leave the incorrect one hanging around. Another trade off would be to customise the generated class file and just live with the necessity to not generate that file (problematic and prone to human error).

The answer turned out to be partial methods. The VB team blogged about these last year:
“Partial methods enable code generators to write extremely flexible code by creating a lot of "hooks", where developers can provide their own custom functionality that integrates into the "boiler plate" code created by the generator. Because the hooks are optimized away if they aren't used, no performance penalty is introduced by defining them. This is really useful if generated code needs to be used in high performance scenarios. For example, partial methods are used by the DLINQ designer for exactly this purpose. Developers wishing to invoke custom code on their data objects when properties are set can do so without requiring all users of DLINQ to suffer performance problems.”
So we ended up adding in calls to partial methods in all of our property setting code:

   14 

   15     Public Property intGroupID() As Int32

   16         Get

   17             Return _intGroupID

   18         End Get

   19         Set (ByVal value As Int32)

   20             ' partial event method to allow insertion of

   21             ' behaviour using partial class

   22             OnSettingintGroupID(value)           

   23             _intGroupID  = value

   24         End Set

   25     End Property

   26 



We also ended up adding pre/post save calls to partial methods to allow us to intervene and tweak either data to be saved or to handle other behaviour post the save operation. An example of a typical save operation is below:

   70 

   71     ''' <summary>

   72     ''' Public method allows external objects to save changes to a Group object.

   73     ''' </summary>

   74     Public Sub Save(ByRef objGroup As [Group], ByVal blnForUpdate As Boolean)

   75         Dim parameters As ArrayListNew ArrayList()

   76         Dim sproc As String

   77 

   78         parameters.Add(New SqlParameter("@strName",objGroup.strName))

   79         parameters.Add(New SqlParameter("@intURLID",objGroup.intURLID))

   80         parameters.Add(New SqlParameter("@blnAllowAddressUpdateInEshop",objGroup.blnAllowAddressUpdateInEshop))

   81         parameters.Add(New SqlParameter("@blnShowAdminInTopMenu",objGroup.blnShowAdminInTopMenu))

   82         parameters.Add(New SqlParameter("@strLDAPGroupName",objGroup.strLDAPGroupName))

   83         parameters.Add(New SqlParameter("@strLDAPIdentifier",objGroup.strLDAPIdentifier))

   84         parameters.Add(New SqlParameter("@blnSystemsAdministrator",objGroup.blnSystemsAdministrator))

   85         parameters.Add(New SqlParameter("@blnRestrictAccessBaseFol",objGroup.blnRestrictAccessBaseFol))

   86         parameters.Add(New SqlParameter("@blnRestrictArticleOptAttr",objGroup.blnRestrictArticleOptAttr))

   87         parameters.Add(New SqlParameter("@blnRestrictFolderOptAttr",objGroup.blnRestrictFolderOptAttr))

   88         parameters.Add(New SqlParameter("@blnForceSelectTemplate",objGroup.blnForceSelectTemplate))

   89         parameters.Add(New SqlParameter("@blnRestrictChangeLayout",objGroup.blnRestrictChangeLayout))

   90         parameters.Add(New SqlParameter("@blnRestrictAddElements",objGroup.blnRestrictAddElements))

   91         parameters.Add(New SqlParameter("@blnRestrictCreateTempFromArt",objGroup.blnRestrictCreateTempFromArt))

   92         parameters.Add(New SqlParameter("@blnAllowEventStatusUpdate",objGroup.blnAllowEventStatusUpdate))

   93         parameters.Add(New SqlParameter("@blnAllowVenueLimitUpdate",objGroup.blnAllowVenueLimitUpdate))

   94         parameters.Add(New SqlParameter("@blnAllowAddAttendee",objGroup.blnAllowAddAttendee))

   95         parameters.Add(New SqlParameter("@blnAllowDeleteAttendee",objGroup.blnAllowDeleteAttendee))

   96         parameters.Add(New SqlParameter("@blnAllowOnbeHalfRegistration",objGroup.blnAllowOnbeHalfRegistration))

   97         parameters.Add(New SqlParameter("@blnAllowMoveEvent",objGroup.blnAllowMoveEvent))

   98 

   99         ' partial event method to allow insertion of behaviour using partial class

  100         PreSave(parameters, objGroup, blnForUpdate)

  101 

  102         If blnForUpdate And  objGroup.intGroupID > 0   Then

  103             parameters.Add(New SqlParameter("@intGroupID",objGroup.intGroupID)) 

  104             sproc = "proc_GroupUpdate"

  105             ExecuteNonQuery(ConnectionString, CommandType.StoredProcedure, sproc, parameters)

  106         Else

  107             sproc = "proc_GroupInsert"

  108 

  109             Dim intNewKey as Integer

  110             intNewKey = CType(ExecuteScalar(ConnectionString, CommandType.StoredProcedure, sproc, parameters), Integer)   

  111             objGroup.intGroupID = intNewKey

  112         End If

  113 

  114         ' partial event method to allow insertion of behaviour using partial class

  115         PostSave(objGroup, blnForUpdate)

  116 

  117         ' refresh the cached version of this object

  118         Dim cacheName As StringString.Format("Group:{0}", ""  + objGroup.intGroupID.ToString() + "" )

  119         RemoveFromCache(cacheName)

  120         AddToCache(cacheName, objGroup)       

  121     End Sub

  122 



The end result for us was a much more flexible system of generated code which meant that we can now fix problems with out data classes in the templates directly and then just re-generate the code and replace the files in our project folders and commit to our source code repository (it helps that we use Subversion via TortoiseSVN).



Note: Thanks to Guy Burstein for taking the time to migrate the Copy Source as HTML addin to VS2008, even though it seems to have problems with my my colour scheme. Mazal Tov with that new MS job too!