My Crusade for Agility: Part 5

In my previous 4 installments of My Crusade for Agility I've outlined my career as a professional developer and how and why I've been striving for an agile development environment. In Part 4, I discussed how I introduced our continuous integration server and how we were able to automate our testing, code coverage, metrics, database schema and data changes, deployments, and ticket tracking. Those were all positive moves and we were working in a much more automated, much easier, much more fun development environment.

Things running more smoothly than ever, but I still wanted more. I wanted a fully agile development environment, not just CI, and I still wasn't sure what that meant, but I knew that so far all of these changes were only benefiting developers. Managers and business analysts haven't really seen much benefit yet, and in order for this to stick, I was going to have to get them on board. So, I decided to approach some acquaintances (who I'd met at local user groups) who worked for agile shops in our area about giving myself and some of my co-workers tours of their facilities and give us a high level "tour" of what agile means to them and how they do it.

Everyone was excellent, they opened their doors to us, spent an hour (at least) answering our questions and showing us how they operate from day to day and filling our heads (well, mine at least) with lots of ideas about how to improve our development process. Everyone at my company was great also, they entertained my ideas and our managers took time out of their busy days to go on these visits with me. They all had great questions, to which they received great answers from our hosts. To be honest, I learned a lot more than I expected to (not that I had low expectations).

After having seen how these agile shops operate I was extremely excited to try to implement some of the things that they were doing. I wanted a big public display our our CI environment, big white boards that showed what everyone was working on, scheduled releases with a predefined set of new features and enhancements that would be implemented and I wanted to do TDD and pair programming. The unfortunate thing was that my team was currently halved, half on one end of the building on the second floor and the other half on the opposite end of the building on the first floor. One consistent theme from all of the agile shops that we saw was the physical location of the team were very close.

Luckily, there was just enough room for us to move and bring our entire team together in a much closer area. Now that we were all together as a team, maybe we could start being more agile. My manager got a giant white board and put it up in his office, and we started reviewing what we were working on, and planned to work on every morning in a short stand up meeting. We still had high cubicle walls up and some of us are in separate rooms, but we were all just around the corner from each other, instead of across the building.

Management has agreed to, in the near future, do some renovation and build us our own big open space where we can develop together without any walls separating us. We should have ample wall space for white boards and a good public place to display some information from our CI environment. There are a lot of exciting developments that are happening now, and are starting to happen, and plenty more that we've been doing that I haven't told you about, yet. Tune in next time, there is plenty more to tell.

Python Dependency Management

I decided to take a bit of a detour from My Crusade for Agility and write about something that has had a huge impact on my Python coding lately. When I first began professionally developing in Python I was in a place where the entire code base was proprietary. Since that time, a lot has changed. The first change was the introduction of jQuery, then came Django, and shortly after South. Then, finally, the greatest single thing since slicing lists, Buildout.

Buildout fits into a category of tools that a lot of developers could probably brush off. They may think, "I don't need Buildout I can use pip, or I can accomplish that with symlinks, or I can manage dependencies manually because they don't change all that often". I, personally, can't see myself ever writing another Python application without using Buildout.

Buildout offers a lot, and I have to give a huge shout out to Joe Kington who first introduced me to Buildout here, and also to Brandon Craig Rhodes who's PyAtl demo showed me some of Buildout's capabilities and really got me excited. Buildout is as simple as it is wonderful. Here is a really quick way to start taking advantage of Buildout's greatness right away.

There are 3 things you need to get started.

bootstrap.py
You can grab bootstrap.py from zope's svn repo. bootstrap.py is a script that will download and set up buildout so there is nothing for you to install.

buildout.cfg
buildout.cfg contains configuration for your buildout project. Here is an example of a simple buildout.cfg file. The buildout.cfg file contains parts, each part has a recipe. There are a lot of recipes available to choose from. The recipe is what buildout uses to know what each part does.

setup.py
setup.py contains the dependencies that your project requires. Here is an example of a sample setup.py file. You can also, optionally define command line scripts that run within your buildout project.

So now you've got everything you need to start using Buildout. I set up my source control just like Brandon did in his example. All of my application code was in a src directory with bootstrap.py, buildout.cfg and setup.py being the only other files in the root of my source tree. After doing that, run the bootstrap.py script. It will generate a few files and directories. Then run bin/buildout (one of the files generated by bootstrap.py). bin/buildout will do all of the heavy lifting. It will fetch the recipes that your buildout.cfg is using, and it will also fetch all of your application's dependencies. Once all of those things have been downloaded it will place them in a directory called eggs. The great thing about Buildout is that it doesn't install all of these things on your system Python. It puts everything in your eggs directory and that's it.

If your buildout.cfg is similar to this one you should end up with a few different things in the bin directory after having run bin/buildout. If you open one of those up and take a look, you should see right away how Buildout is able to download all of the dependencies that your application needs without needing to touch your system's Python. It is adding everything in the eggs directory to the beginning of your sys.path. The great thing about doing it that way is that if I decide that I want to use Django from SVN or a new release of Django or an older release of Django the only thing I need to do is update the version number in my setup.py file and re-run bin/buildout. My eggs directory will then include two different Django eggs and it will only add the one the matches my setup.py file to sys.path.

In conclusion, Buildout is great because it gives you an extremely simple, consistent and repeatable way to create your entire environment on any number of developer machines, virtual machines, test machines, production machines, or continuous integration machines and you don't have to worry about saying "DOH! I forgot to install module X on the production box".

My Crusade for Agility: Part 4

This is my 4th installment of My Crusade for Agility. In previous installments I've discussed my career up to the point where I'm aware of agile practices and I want to be in the agile space but I don't really know enough about agile to implement it myself. What I do know a little bit about is source control, continuous integration, automated tests and deployments. Luckily for me, my new employer already has the entire code base in Subversion. I had been toying around with TDD so there were some unit tests in our code base, but not many. I decided that if I could set up a continuous integration server and show the other developers how all of my tests automatically ran after every commit, they may begin to see some value in writing tests of their own.

I approached my lead about installing Hudson on our test server, to which he did not object. So I got it installed and spent more time than I care to remember trying to get Django's unit tests, Python's coverage, pylint, and clonedigger all running and working together and with Hudson's fancy graphs. I did get everything working though, and it was worth the extra effort. Now we had a way to test our code and check the quality of our code every time anyone would commit a change. The next thing that came was Trac integration, which was actually the suggestion of my lead. I felt like this was a huge win for my cause. Not only had someone else gone out and found a free open source alternative to our proprietary ticket system, but they also wanted to integrate it with our CI system. With Hudson's Trac integration we could link our commit messages directly to our tickets and vice versa.

The next thing that I wanted to implement was automated deployments. Unfortunately, this was not going to be very easy. Django has an excellent ORM, but it does not do anything to handle changes to existing tables, it really only handles the creation of new tables. To solve that problem, I found a project called South. South allowed us to create database migrations, so now we can easily apply any and all database (schema and data) changes quickly and easily using a Django management command.

Now that the database changes can be automated, I was ready to automate deployment. Previously, our deployment process was not very complex, but fairly manual. It consisted of SSHing to our test server and running a bash script that did an export from our Subversion repository. After running that script, we needed to restart WSGI (Apache's mod_wsgi - persistent Python interpreter) and ideally we needed to hit the website to get the start-up overhead out of the way. The first step that I took in making this process more automated was adding the running of our migrations, the wsgi restart, and a curl to our base url into the deploy script. That limited our deployment to SSHing to our test server and running a script. Then, it was a simple task to move that script into Hudson and have Hudson execute that script over SSH. Finally, having our build job trigger the deploy job in Hudson completed our deployment automation.

Now we were working in an environment where we could code, commit, see our tests, our coverage and other metrics, and then shortly after (with no further action on our parts after the commit) see our changes reflected on our test server. It doesn't get much better than that, does it? Oh, yes it does. We were still just getting started. Lots more to come, tune in for Part 5 coming soon to a web browser near you.

Process & Ceremony Over Results

Agile development is something that I've never really done professionally, but I really want to be in that space. Implementing agile practices to an organization that is not agile is not an easy thing to do. There will be resistance from every angle, BAs, QAs, developers, managers, etc. This morning, I had an experience that helped to clarify in my mind why agile techniques can be hard to adopt. My experience made me realize that by nature people favor process and ceremony over results. While most people are geared towards end results, intermediate results don't seem to matter much as long as the standard process and ceremonies are followed.

The experience that made the "bulb over my head" turn on was a very simple example how, without even thinking, I followed a process and ceremony with the end desired result in mind but ignoring the intermediate results. This morning, I walked into the bathroom and reached for the light switch. I moved my hand in the usual upward motion over the switch and made contact but didn't feel the typical "flip" and didn't hear the typical "click" of the switch being turned to the on position.

Following my usual ceremony and procedure, I tried again. Again, I made contact with the switch but the switch didn't flip and I didn't hear the comforting ceremonial "click" of the switch being flipped. It was at this point that I realized that while my intention of reaching the end result of having the lights turned on was completely distracted by my typical ceremony and process of flipping the light switch. In that same moment, I realized that the switch was already in the on position, and the lights were already on. I just needed to wake up and realize that the end result was already attained and I didn't need to follow my typical process twice, once or even at all.
Process and ceremony are hard things to bread once we get comfortable, but in some cases they overstay their welcome and usefulness. I could have walked into that room and gone about my business, but instead my typical ceremony got in my way and slowed me down.
Stay tuned for part one of a new blog post series that I'm going to start called "My Crusade for Agility".

My Crusade for Agility: Part 3

Welcome to the third installment of My Crusade for Agility. In the first two parts, I outlined the beginning of my career as a professional developer. I didn't really know what it meant to be agile, but I knew about some of the agile practices (like pair programming and tdd) and I really liked what I was seeing and hearing, but not yet doing. I wanted to be in a place where I could develop that way, I wanted to have more control over my own environment and be able to change it when it was not working. Part two left off with me deciding to accept an opportunity with a new employer. I felt confident that this new employer could provide me with the type of environment that I was seeking.

My new employer was a significantly smaller company, which was very appealing to me. Here I might have some say in what goes on, my opinions and ideas might be heard, and really listened to, and maybe even implemented. I knew going into it that the team that I was going to be working with was using Linux (as opposed to Windows like the rest of the company), MySQL instead of DB2, Apache instead of IIS and Python instead of Perl. Since the "corporate standard mold" had been broken, I was sure that they would be receptive to new ideas. I just need to build up some street cred and get past "the new guy" phase so I can start throwing some ideas into the ring.

Right off the bat, there were some things that concerned me. They were using a web framework that they had developed in house, which was less than ideal. Their proprietary framework was similar to the Java Servlet API, which is not a problem in and of itself. The problem was that there was not really any other organization or structure beyond that. There were HTML strings being pieced together, SQL strings being built, and business logic all happening in the same Python method. My first day on the job I was given a project, a new subsystem for this application. After digging through the existing code base and not seeing any indication of some sort of MVC structure I asked if I could write this new subsystem using the Django framework.
I thought that introducing a popular, robust, open source solution for something that they had historically done themselves could score some popularity points for me. I think, on my first day, I still had that "new guy smell" or something because nobody was really interested in letting me change everything they'd already done. So I played ball, I developed the new subsystem using their proprietary framework (with the addition of my own MVC layer on top of their framework). I did, however, introduce jQuery to do some ajax autocomplete in my subsystem, which was well received. I think the introduction of jQuery opened some minds to the use of open source frameworks.

The company agreed to send myself and another developer to PyCon and after the conference, and once jQuery had been accepted, and with my persistent inquiries into using Django, and the help of Cal Henderson's "Why I Hate Django" PyCon talk they finally caved in. (Apparently Cal's take on Django backfired, luckily for me, thanks Cal.) We took a few months to re-write the entire application using Django. I felt like this was a huge step forward. We now had an ORM, a full MVC, and a huge, active community helping us do our jobs. This was only the beginning, more, even greater changes were coming.
Check back soon for part 4 of My Crusade for Agility.