Category Archives: Project Management

Versioning is Just Too Complex

I’m currently trying to unlearn 10 years of CVCS and Subversion certainties to learn DVCS and Git. And although a lot of people see those as a huge leap forward, I can’t help thinking it’s still way too hard. Versioning is such a basic need, it shouldn’t require that much knowledge. Right now, whether it’s Subversion or Git (or Mercurial, or whatever), the problem is always the same for me: it’s just too low-level. Those systems are designed as if I needed to be a mechanic to drive my car. I don’t care about injectors, gearbox and all that stuff. All I want is to go faster or slower, turn left or right, and that’s about it. Now sure if I’m a mechanic, I can make my car perform much better, but if I just need to get from point A to point B, do you really think I will learn mechanics? Most people won’t. They will just walk there, or ride a bike. And that’s exactly what’s happening in too many companies: “manual versioning”, version numbers in file names, shared drives, even $harepoints, are just poor solutions to a real problem. We need versioning, but existing solutions have an unacceptable learning curve. They’re not smart enough, and they require us to be smart, even though that’s not our main purpose.

Even as a developer, sure I can understand such concepts, but I don’t want to learn what merge, pull, push, commit, branch or update mean. I want to start new projects, participate in existing projects, fix bugs, implement features, refactor my code. Whatever underlying system my company or team is using, whatever technologies I’m working with, my own workflow should not be influenced so much by the tools I use to keep track of my work and collaborate with my team. It should be transparent.

My point is: Git is a leap forward, but it’s a small one, and we need a giant one. We need a versioning system that implements workflows at a higher level, that abstracts away all those silly commands with similar names but different meanings. We need an abstraction layer that can work on top of existing lower-level tools like git, subversion and others, and that can be integrated seamlessly in development environments like IntelliJ Idea or Eclipse (sigh). Sure we will probably lose some power in the process, but we can leave it as an option.

In fact, we need to do with versioning what Maven has done to build lifecycle. Convention over configuration. What are the most common tasks we do on a daily basis:

  • start a new project
  • enter an existing project
  • start fixing a bug
  • start implementing a new feature
  • start refactoring some code
  • switch to another bugfix/feature/refactoring
  • complete a bugfix/feature/refactoring
  • share a bugfix/feature/refactoring with my team or with the world
  • what else?

And then let’s try to map those high-level tasks with lower-level command sequences. And let’s do it in such a way that I can easily change the underlying implementation, or reconfigure certain aspects of it. And why not add the possibility to define your own workflows for other things than traditional dev. In fact, to make things clearer, let’s just call git, subversion and other existing tools “versioning tools”, and let’s design a “collaboration framework”. Exactly like Maven is not a “build tool”, it’s a “build lifecycle framework”.

I’m just ranting out loud here, but what do you think?

What? You’re doing TDD!

h_4_ill_1022732_black-sheep-bisI’ve always been distrustful by default with the “main way of thinking”. Which doesn’t mean that I systematically reject the opinion of the majority. But given the strong tendency of human beings towards laziness and our natural tendency to sheep-like behaviour, the faster something gets popular, the more I question it.
Test-driven development has always fallen in this category for me. The problem it’s supposed to solve is real: software coding is a highly error-prone activity and we need to do everything we can to avoid shipping erroneous code. I totally agree with that. But then TDD is just one solution to this problem: to avoid shipping errors, let’s test and check our code in order to detect errors before shipping. But this solution leads to other problems:

  • tests are just more code where there can be other errors, and that need to be maintained. So what about false positives and negatives?
  • test writing has always looked like a very boring and chore-like activity to me, especially if it involves thinking about my implementation algorithm before I actually write the corresponding algorithm, which is often required when doing behavioral testing
  • I’m still not convinced that ensuring each part of the system is working is supposed to reassure me about the system as a whole.

And more importantly, there are other alternative solutions to the same original problems: for example, instead of waiting for tests to fail to fix mistakes, why not trying to avoid them in the first place? We have some pretty good tools for that: refactoring is error-prone, especially when you use a refactoring engine that is not reliable. Then stop using Eclipse, move to IntelliJ. You keep reproducing the same wrong patterns from some old language you’ve worked with for 10 years? How about static analysis? And instead of typing that much error-prone boilerplate code, why not use code generators or dynamic languages like Groovy? I’ve been using all of these tools for years, and I’m gonna say something bold here: I shipped some pretty decent software with not enough bugs to justify the  weight of TDD. A craftsman can also go a long way in improving his work by getting better tools, and training himself to use them. Actually that’s what really pisses me off with all the TDD religion thing: too much focus on techniques and processes, not enough on technologies and people. And I think I’m just more interested in the latter two.

Now most of the time, when I raise my objections about TDD to people who say “What? You’re not doing TDD!!!” with some sort of horrified look, they just answer me with disdain instead of actual arguments, which can mean one of 2 things:

  1. They didn’t look into it that much. They just followed the trend because it seems nice to say that you do TDD
  2. I didn’t look into it enough and some of them are actually right to treat me that way, because I don’t know what I’m talking about

I thought I had tried TDD enough in the past to avoid the second option, but this morning I attended an amazing talk by Robert C. Martin at Devoxx. In fact, there was another talk before by Ivar Jacobson. And the guy kept talking about software as an industry (comparable to manufacturing for example), about the fact that we have a big professionalism issue (which I totally agree with), and that in his view the best way to fix the problem is to add some theoretical foundations into the mix. Let’s say I didn’t agree at all with his view of where the problem lies, and with his inferiority mix up towards other industries. But on the other hand, I agreed a lot with Uncle Bob’s view of things… until he started talking about how silly it was not to do TDD these days. How can I agree with his conception of where the problem really is, and not with one of the main solutions he proposes. Maybe I’m missing something here. The best way to figure it out is to read in more details what he has to say. And given the incredible energy he showed in his talk this morning, his book is likely to be a lot more interesting than all the TDD manifestoes I’ve read in the past (or tried to read before falling asleep).

How Open Source can be a Game Changer

A few years back, I read a very interesting article (that I can’t find anymore) by John Newton, the CEO of Alfresco, explaining how Open Source changed their strategy. Of course, one of the main interests of having an Open Source version of your product is to expand your market by making it more accessible, thus creating opportunities for high-value services such as training, customization and so on. But beyond this shift in “how do we make money?”, he explained that it totally changed the nature of their salesforce. When you sell that kind of enterprise system, the best way to get potential customers to know and see your product is to invest a whole lot of money in marketing and sales. And you end up with a sales cycle of several months between the first prospection call and the actual sale… when it goes through. No wonder why license fees are so expensive for that kind of product, making it even harder to sell, increasing the length of the sales cycle, and here goes the vicious cycle.

Having at least an Open Source version of your product completely changes that because people who don’t have the checkbook but will eventually use your product can try it on their own. Maybe they’re working on personal projects, maybe even Open Source projects, and they can install and use your product for free. Then if they like it, they are more likely to go see their boss at work and recommend your product. In other words, you can replace a big chunk of your very expensive salesforce by free happy end-users and their recommendations. Hence not only does Open Source change how you make money, it makes it possible for you to save money where it’s not that necessary, and focus a lot more on your core business: building a great product.

But the way I see it, there’s another very important change with Open Source. As I said before, when your business model is based only on license fees, you focus most of your attention on your salesforce, which costs a lot of money, which makes your license fees so high that it is very likely to represent a big investment for your customers. So your salesmen end up attacking prospects at high levels of management. The problem is that the guys who have the checkbook also have very different concerns and priorities compared to the people who will eventually use your product. Let’s say you are building a software project tool suite with things like issue tracking, source control and so on. You try to sell your product to companies who do in-house software development, and the people who will eventually use your software are developers. But you don’t talk to those guys directly because they have almost no decision power, especially when it comes to spending several hundreds of thousands of euros for software tools. So you talk to their Program Manager or VP of Software Engineering or CTO of some sort. And all of a sudden, you realize that those people have concerns like keeping things under control whatever happens, minimizing risk (whatever that means), reporting, making sure that no developer does anything on their own, and so on. So of course, you will implement features to satisfy those needs because you want the big guys to be happy.

Now let’s go back to the Open Source alternative. If you hope that developers will use your software on their own project, find it cool and then go back to their boss and say “we should have that”, you’re not talking to the same people. It really makes things straight again because you’re back to consider end-users’ concerns first. Developers want tools that don’t get in their way, that allow them to save some time, not waste it, that are integrated in their development environment, and so on. My point is that an Open Source model does not only change where you make money, or how you balance your own investments. It can also change your whole product itself by changing whom you’re talking to.

Now of course, I’m opposing 2 strategies here: the traditional top-down approach that forces you to focus on upper-management priorities versus the Open Source bottom-up strategy in which end-users are the ones you have to convince. And I know that this kind of 2-way alternative is likely to create controversy because project managers will read this article and think “but we need reporting, we need control, we need power!”. Now of course you know that I don’t agree with this obsession for power and control as it is associated to the software engineering mirage. It’s like the big guys couldn’t stand the IT guys anymore, with their strange language and culture and so on. So they irrationally tried to isolate us from them with layers and layers of project management and control. But they lost a great deal of intelligence in the process and they’ve created a whole generation of frustrated developers who just execute what they have been asked to, even when decisions are made by people who don’t understand a damn thing about what software is and how powerful it can really be. And of course, big old-school software vendors have created opportunities out of this mess and we’re all forced to work with their products now. But hopefully, Open Source is already changing that, it is participating in a movement of IT empowerment, together with Agile methodologies, that will lead to more intelligent uses of technology and fill in the gap between business and IT.

Now to conclude, I’m not saying that management is totally useless. YES, we have a strange culture and a weird way to look at things. So YES, we need people to help us interface with business people and teach us how to do it ourselves. NO, the solution is certainly not to reduce our field of action to the minimum but YES, we need people who understand both the power that we have AND the big picture of what is necessary, and are able to make them coincide. And YES, this mission might require additional features in the tools we’re using, for things like reporting, prioritization and so on. But those features should never conflict with our concerns as end-users, they should never get in our way to building better software. But the good news is that there are very clever ways to do just that: have a look at Mylyn, or FogBugz. And of course, I myself am thinking about bringing my own contribution. So stay tuned…

Now I would love to try an experiment and use comments on this post as an informal survey. So if you are a software developer, what tools are you using at work for things like source control, issue tracking, documentation management, and so on? Do you like those tools and why?

Software Engineering Is NOT Dead!

There’s been a few reactions lately about the Tom DeMarco article. To be honnest with you, I didn’t know the guy but apparently he’s quite respected. And the reason why I didn’t know the guy might be related to the fact that he wrote a software-related book entitled “Controlling Software Projects: Management, Measurement and Estimation”. In other words, he’s the one (OK, one of the guys) responsible with all those project managers asking us for sharp estimates, negotiating them down with us, and committing on them without our agreement, leaving us with no choice but to produce crappy software and/or fail to meet the deadlines. For me, Tom DeMarco looks a lot like this Dr. Winston D. Royce dude, who first described the theory of Waterfall methodologies and forgot to write the following sentence in bold font: “the implementation described above is risky and invites failure“. No kidding!

My reaction is about the same to what DeMarco took 40 years to realize. Software Engineering is not dead, it has never lived! There has been a lot of controversy about that, some people arguing it’s more of an art, others saying it’s science, yet others that it’s more like craftsmanship. At the end of the day, the only fact that there is a controversy shows that it doesn’t fit in the hole we’re trying to stick it into. And yes, this blog is called Software Artist, but I have to tell you that it’s more as a reaction to those who are fiercely trying to “industrialize IT processes”. Oh man, that makes me angry! In the end, I have to say that the more I think about it, the more I like the idea of craftsmanship.

Because craftsmanship is all about finding the right balance between the techniques that you’ve learnt and the intuition that you have about what you’re doing. And finding this right balance requires a lot of practice. Note that I didn’t say “time”, I said “practice”. I like it also because for any craftsman, using the right tools for the job is fundamental, and once again, note that I said “the right tools”, not “the tools that everyone else is using in here” or “the tools we invested a lot of money on and we need to make up for”. And finally I like it because when a joiner tells you that he doesn’t know exactly when this wardrobe will be finished, you don’t bother him with that because the fact that it corresponds exactly to what you need, and the fact that it will last a very long time, largely makes up for this little uncertainty. And if he tells you he’s going to need another 4 days, you don’t negotiate, because if you know better how to do it faster, why in the world didn’t you do it yourself in the first place? Or why didn’t you go and buy one at IKEA?

Now after re-reading the above paragraphs, I realize that my statements might seem a little harsh and extremist (as usual). But I don’t want to sound negative here. So if you manage software people and you feel disturbed by this perspective to lose control, let me just tell you this. Software can be incredibly powerful, it can really create tremendous value in a very flexible way. And I’m not talking about allowing you to just save money. I’m talking about the possibility to develop your business and create value from scratch. But the more value you expect to create, the more you have to ask yourself whether you absolutely need to control every aspect of the software creation process. Remember, you’ve tried it before, and it most probably didn’t work as expected. But fortunately for you there’s a different approach. Empower your software team. Don’t lead them, but make their ride smoother. Don’t make them work FOR you, work WITH them. Don’t consider them as standard office workers. Instead, hire the best craftsmen you can find and trust them with your project. Make it theirs as well as yours. Surrender on this idea of controlling everything and you’ll be surprised at the result.

Software engineering is not dead, because it has never existed. Hence there is no need to mourn it. Let’s just roll up our sleeves because we have a few palaces to build for the years to come.

We don’t need an online IDE!

I’m on vacation this week, and it’s great. I usually try to take at least one week off in between missions because it’s the best time to do it. Nothing to worry anymore about the one before, no knowledge of the one ahead, my head is free to dream and wander around and I have plenty of time to move forward on personal projects.

Now beyond my work on betRway.com, I’ve been thinking a lot today about the idea of an online development environment. On my last project, I’ve seen a lot of the possibilities AND limitations of current RIA technology, and more specifically the most advanced one IMO, ie Flex. Combine that with some of the modularity offered by OSGi on the server-side, the productivity brought by things like Grails, and more importantly the promise of unification of all of those technologies under SpringSource’s sponsorship… and it’s starting to become more and more interesting.

So I googled “online IDE”, and I found this great article on DZone, which is a few months old but gives a lot of links to some of the services that are already offering both niche and generic online environments. I’ve browsed through all of these and I have a bad feeling of “déjà vu”. It’s like those mobile web sites that try to reproduce a trimmed down version of their web counterpart: it doesn’t really work, and worse, it doesn’t even use all the potential of mobile platforms. It’s like we’re trying to copy our desktop IDE onto the web.

But when you really think of it for a moment, the reason why we need version control systems for example, it’s because everyone is working on its own, and we need to synchronize at some point. But this constraint almost disappears if we’re all working online. And online development can have impacts of similar magnitude on things like continuous integration, deployment, monitoring. Some of these tasks don’t even make sense anymore. And how about several programmers working on the same artifact at the same time? And think about the computing power that could be available for code generation or compilation…

The way I see it, even the programming languages that we use today are not designed to work like that. Think of Java packages correlated with file system directories. Think of all these destructured text files that our IDE’s have to parse and index to make some sense out of them and allow us to navigate through them: what if text files are replaced by databases at some point?

I’m just thinking out loud here, but I’m wondering whether we shouldn’t forget about everything we know in terms of programming paradigms, and get back to the objective: producing working software collaboratively. What would be the best and simplest way to do that, knowing that we are starting to have all the tools we will need in order to move our workspace into the cloud?

As far as I’m concerned, I already know what I would like in such an environment:

  • More graphical programming, especially for high-level design, because visualizing is still the most efficient way to create and design IMO.
  • No more boilerplate tasks like “check out”, “check in”, “run tests” and things like that. Just “share my code in the public workspace” and “test on save”.
  • The ultimate reproducible build environment: everyone is using the very same frameworks and libraries at any time. No more environment variables or shell idiosyncrasies.
  • Smart client stuff, with offline mode, auto-synchronization and conflict resolution when back online.
  • Methodology directly integrated into the environment, not some small window on the real thing. Issue tracking, user stories, all of that at your fingertips.
  • Direct communication with your remote peers through advanced tools like whiteboard, webcam, code review.

In fact, we don’t need an online IDE, not in the sense we think of an IDE today on our desktops. We need more than that. We need a collaborative environment that uses the full potential of the cloud to help us produce better software more quickly.

What do you think? What kind of high-level functionality would you like to see in such an environment? How do you picture it?

Meta-Agility

I’ve been a professional software development consultant for 3 years now and I’ve seen agile methodologies become very popular in the company I work for. A lot of people have started using SCRUM on their customer projects and the approach has proven to be very successful. I think that the most surprising realization people have come to is that for years, if not decades, we’ve been stuck with the old way of doing it, thinking that software development could be planned and industrialized just like any manufacturing process. And a lot of software projects have failed, over and over again, before some people actually started to think outside of the box and came up with a more pragmatic and realistic approach.

Now, even though the methodologies themselves are very interesting, even though those methodologies still have plenty of room for improvement, I’m starting to see people trying to map agile methodologies on other things than just software development. People extract small chunks of SCRUM, like managing tasks through post-it notes, frequent synchronization meetings, specifying deliverables in user stories. But after playing a few months with these fun tools, they are starting to realize that they don’t work so well out of context. Because what makes SCRUM so powerful is the fact that it is a small but very consistent set of practices that all work together with common objectives to reach: the four principles of the Agile Manifesto. And it’s becoming obvious now that we cannot just take a few of these practices out of context and expect them to work for other purposes.

In fact, I have the feeling that people are more and more seduced by the meta-level of agility, the reflexion that led us to rethink our whole way of working to reach better results. And in my opinion, THIS is what we need to reproduce: the approach that led us to design the methodology, not the methodology itself.

Let’s try to analyze this generic approach:

First of all, let’s define our scope. For SCRUM, it was software projects with everyone working on the same spot at the same time. In Axen right now, it’s change management, more precisely how do we introduce a new business model into our company while keeping up the good work on the existing one.

Second, what’s wrong with our current way of working? For software projects it was that everything was linear (analysis, design, development, testing) and offered really poor visibility on how things were going. For our change management process, it’s the fact that all the team work is pretty much done in disconnected after-hour meetings, because everyone has a day job to keep the existing model running. The end result is that a lot of energy is wasted in pointless discussions and motivation is lost inbetween meetings, people working pretty much on their own.

To be more precise, agile methodology gurus have come up with a very interesting way to think of failures and solutions in a very constructive manner: what do we value over what? For change management, examples could be:

  • remote collaboration over individual work and synchronization meetings
  • small manageable objectives rather than big multi-year expectations everyone loses sight of
  • professional evolution over personal sacrifices in exchange for fun

At least that’s my view of what’s not ticking in our current way of doing it. I know that other people have different views and hopefully we’ll soon have a debate and come up with a consensus on the four or five values we need to encourage for everything to work.

Then when you know the values you have to put forward in order to reach your objectives on any project in the same context, you can start to think of the tools and practices you need to implement:

  • remote collaboration can be helped with collaboration software easing communication and issue tracking
  • user stories are good for software, but the actor/action/goal paradigm might not be the most adapted to the implementation of new business practices. We have to be creative and find a new way to define our “requirements”.
  • change management cannot be free, and investing money to free up time is not enough. Maybe some of this money needs to be dedicated to training participants on new techniques and knowledge that will improve their professional experience while developing the global expertise of the company.

And of course, once again, plenty of ideas can come up if we allow ourselves to think outside of the box and stop trying to make new stuff with spare parts from old stuff.

I will soon try to initiate this reflexion inside Axen and hopefully we’ll come of with a new change management methodology that embraces the constraints of this context rather than trying to hide them under the carpet. To be followed…

By the way, if you work for Axen, you know that the company takes great pride in involving consultants into its evolution. So what do you find motivating or not when working on those internal projects? If you don’t work for Axen then maybe you’ve worked on similar change management projects and you have your own experience on how to make it work while involving everyone in the process. I’m very interested in your feedback.

    The Fourth Commitment

    Following my article about customer satisfaction as a third commitment, I read many articles and thought a little bit about this fixed-price thing. Actually it’s quite funny because the software development industry is about 30 years old at least, and it seems we are just starting to wonder why fixed-price projects don’t work with software the way it works in other industries. And I really think it all started with the Agile Methodology hype because now we’re trying to be pragmatic, we’re trying to embrace the constraints of software development instead of hiding them under the carpet. And one of those constraints is this damn iron triangle.

    It’s funny to see how people hold on to it, including both customers and salespersons. It’s like they don’t dare to change it. It’s like they are afraid of changing this model. The problem is that each commitment in this triangle involves a substancial risk that no one is willing to take. But what I realized a few minutes ago after reading this very interesting article is that the reason why it’s really not working is because we’re definitely trying to force a cube into a triangle hole! There’s a commitment we’re constantly forgetting: quality! Whether you like it or not, quality is not something implicit, it’s not even related to the developers on the project. Every experienced developer knows that you can’t code with quality, you have to test it.

    My father has been working in a car gearbox factory for over thirty years and I’ve worked there during a month as a summer job when I was a student. And my father is a gearbox tester: a box comes in, it’s set up on the bench, my father goes through first, second, third, fourth, fifth and back down, he listens to the noise that the gearbox is making, and if there’s a problem, it goes back to repair. He does that when the box is finished and the proportion of bad boxes is pretty low, because quality is enforced all through the production line by very careful processes that involve as less intelligence as possible. That’s one or the reasons why robots replace human beings in those factories. Because robots don’t think of their kids or dream of their wife. They just do what they were told to.

    Do you really think that this kind of quality can be enforced on a software development project? Of course not! It’s far too intellectual and creative to allow such low-level considerations. And yet I’ve heard tens of times this stupid expression: “software process industrialization”. Come on!

    The truth is that the only way to enforce quality on a software project is to test it permanently. To test it automatically, functionnally. To test robustness, scalability, reliability, security, performance. And of course you have to review a lot to ensure that the code remains readable, maintainable and flexible. Software quality can be all of that, and enforcing all those aspects is very time-consuming. It takes some time and some special resources as well. You can’t just forget about that.

    So we’re left with four risks: scope, budget, resources… and quality, leading us to four commitments to force into our contracts, right? Wrong! Because there’s something that we’re forgetting: those 4 constraints are changing over time. So here we are, trying to force a irregular and mutating cube into a predetermined and unmodifiable triangle hole. And customers still wonder why it doesn’t work?