My Crusade for Agility: Part 7

It has been over a year since I started working at my current employer.  In my last blog post I published a large list of things that we had started doing in the past year. We are continuing to advance our technology choices and development process.

We've been playing with Fabric, which we will most likely use to automate our production deployment, we have also been messing around with mongo and mongoengine, which we hope will replace our use of the linux file system for document storage. We've also been discussing looking more into Celery to use to do some asynchronous processing. So there are a lot of exciting things on the horizon from a technology perspective.

We are still trying to figure out how to manage our workflow process. We've been writing story cards and using a white board to organize what is being worked on, but we haven't done any sprints or retrospectives. I'm still trying to decide what we're missing, if we're missing anything at all. I like the idea of having sprints so we are able to lay out a number of stories that we plan on implementing within a period of time, but I also like the continuous development that we are currently doing. Currently, when we finish one story we write another and get started on that one. The problem that I see with our current process is there is a lot of room for interruption. Without a fixed roadmap, like one we would get with sprints, it seems easy to fall into working on tasks that may satisfy an immediate need but detract from our product's long term goals.

Being able to satisfy immediate needs I think is a very desirable place to be in. If we could satisfy each customer's immediate needs we would do very well, and our customers would always be happy. The problem is that every customer's immediate needs are different. We have to be able to choose the tasks that make the most sense for our product and will benefit the largest number of customers. I feel like working in sprints gives us time to reflect on what our customers seem to be asking for collectively and decide what to deliver to them. Without sprints, I think that we focus more on greasing the squeaky wheel, which could easily result in adding features and customizations to our product that may only make sense for a single customer which may also cost more to implement than the value added to our product.

In other exciting news, our new floor plan has been completed and construction has begun. Our new area is going to be a large open room with plenty of wall space for white boards. I imagine we will have a table in the middle of the room with pairing stations, but we'll just have to wait and see how exactly that turns out.

I can't really express in words how proud I am of how far we've come in so little time. In my humble opinion, I couldn't ask for a better employer or better co-workers who share my passion for software development and for constantly improving our development process.
Stay tuned for more, we're always trying to make things better in some way or another, so I'm sure there will be plenty more to write about.

My Crusade for Agility: Part 6

This is the sixth installment of My Crusade for Agility, welcome. I've been writing about my adventures in implementing agile development practices at my place of employment. In the past eleven months (since I began working for my current employer) a lot has changed. We have implemented all of the following things:

• Co-Locate our team members
• Morning Stand Ups
• Managing our work loads using Story Cards and White Boards
• Added over a thousand Unit Tests (with 72% coverage)
• Started releasing more regularly
• Added a Hudson CI server that runs all tests, coverage, and other metrics with every commit
• Automated Deployments to our test server after successful builds
• Started using jQuery
• Replaced proprietary web framework with Django
• Handle Database schema changes using South migrations
• Automated dependency management with Buildout
• Replaced our proprietary ticket system with Trac
• Started developing on our own Ubuntu machines instead of WindowsXP
• Migrated from Subversion to Git
• Migrated from PyDev to PyCharm
• Started a "library" which consists of the following books
• Practices of an Agile Developer
• High Performance Websites
• Coaching Agile Teams
• Test Driven Development: By Example
• The Pragmatic Programmer: From Journeyman to Master

There are (at least) two more things that we have began doing, which are to me, the most exciting. The first of the two is Test Driven Development. TDD is something that I have been doing for the past few months, but I found myself falling back into old habits of writing production code first and writing the tests after or not writing a test at all. TDD takes a lot of discipline and it is unfortunately easy to not write tests when you are working alone. Now that I've been doing TDD for a little while, I am much more confident in my self-discipline to write tests before any code without falling into old habits.

The second new and exciting thing that we've started doing is Pair Programming. Pair programming is what really helped me to develop my self-discipline for TDD. With a second pair of eyes, and most importantly a second brain, working through code and making sure that no code is written without a failing test first, it is significantly easier to... well... do everything. TDD by yourself is hard, especially when you're just starting. TDD when you're pair programming, even if you're just starting with TDD, is much easier because instead of just having that little voice in your head that says, "it's ok to not write a test for this code", you've got a real audible voice right next to you saying, "Oh no you don't!".

Looking back 11 months; our code, our process, nearly everything is nearly unrecognizable. Everyone has welcomed these changes, and has seen success in the quality of work that has been done since implementing them. We still have plenty of work to do, but we have come a long way in a very short amount of time. There are still big plans for the future of our office, both our physical location and our workflow process, so stop back for Part 7.

My Crusade for Agility: Part 5

In my previous 4 installments of My Crusade for Agility I've outlined my career as a professional developer and how and why I've been striving for an agile development environment. In Part 4, I discussed how I introduced our continuous integration server and how we were able to automate our testing, code coverage, metrics, database schema and data changes, deployments, and ticket tracking. Those were all positive moves and we were working in a much more automated, much easier, much more fun development environment.

Things running more smoothly than ever, but I still wanted more. I wanted a fully agile development environment, not just CI, and I still wasn't sure what that meant, but I knew that so far all of these changes were only benefiting developers. Managers and business analysts haven't really seen much benefit yet, and in order for this to stick, I was going to have to get them on board. So, I decided to approach some acquaintances (who I'd met at local user groups) who worked for agile shops in our area about giving myself and some of my co-workers tours of their facilities and give us a high level "tour" of what agile means to them and how they do it.

Everyone was excellent, they opened their doors to us, spent an hour (at least) answering our questions and showing us how they operate from day to day and filling our heads (well, mine at least) with lots of ideas about how to improve our development process. Everyone at my company was great also, they entertained my ideas and our managers took time out of their busy days to go on these visits with me. They all had great questions, to which they received great answers from our hosts. To be honest, I learned a lot more than I expected to (not that I had low expectations).

After having seen how these agile shops operate I was extremely excited to try to implement some of the things that they were doing. I wanted a big public display our our CI environment, big white boards that showed what everyone was working on, scheduled releases with a predefined set of new features and enhancements that would be implemented and I wanted to do TDD and pair programming. The unfortunate thing was that my team was currently halved, half on one end of the building on the second floor and the other half on the opposite end of the building on the first floor. One consistent theme from all of the agile shops that we saw was the physical location of the team were very close.

Luckily, there was just enough room for us to move and bring our entire team together in a much closer area. Now that we were all together as a team, maybe we could start being more agile. My manager got a giant white board and put it up in his office, and we started reviewing what we were working on, and planned to work on every morning in a short stand up meeting. We still had high cubicle walls up and some of us are in separate rooms, but we were all just around the corner from each other, instead of across the building.

Management has agreed to, in the near future, do some renovation and build us our own big open space where we can develop together without any walls separating us. We should have ample wall space for white boards and a good public place to display some information from our CI environment. There are a lot of exciting developments that are happening now, and are starting to happen, and plenty more that we've been doing that I haven't told you about, yet. Tune in next time, there is plenty more to tell.

Python Dependency Management

I decided to take a bit of a detour from My Crusade for Agility and write about something that has had a huge impact on my Python coding lately. When I first began professionally developing in Python I was in a place where the entire code base was proprietary. Since that time, a lot has changed. The first change was the introduction of jQuery, then came Django, and shortly after South. Then, finally, the greatest single thing since slicing lists, Buildout.

Buildout fits into a category of tools that a lot of developers could probably brush off. They may think, "I don't need Buildout I can use pip, or I can accomplish that with symlinks, or I can manage dependencies manually because they don't change all that often". I, personally, can't see myself ever writing another Python application without using Buildout.

Buildout offers a lot, and I have to give a huge shout out to Joe Kington who first introduced me to Buildout here, and also to Brandon Craig Rhodes who's PyAtl demo showed me some of Buildout's capabilities and really got me excited. Buildout is as simple as it is wonderful. Here is a really quick way to start taking advantage of Buildout's greatness right away.

There are 3 things you need to get started.

bootstrap.py
You can grab bootstrap.py from zope's svn repo. bootstrap.py is a script that will download and set up buildout so there is nothing for you to install.

buildout.cfg
buildout.cfg contains configuration for your buildout project. Here is an example of a simple buildout.cfg file. The buildout.cfg file contains parts, each part has a recipe. There are a lot of recipes available to choose from. The recipe is what buildout uses to know what each part does.

setup.py
setup.py contains the dependencies that your project requires. Here is an example of a sample setup.py file. You can also, optionally define command line scripts that run within your buildout project.

So now you've got everything you need to start using Buildout. I set up my source control just like Brandon did in his example. All of my application code was in a src directory with bootstrap.py, buildout.cfg and setup.py being the only other files in the root of my source tree. After doing that, run the bootstrap.py script. It will generate a few files and directories. Then run bin/buildout (one of the files generated by bootstrap.py). bin/buildout will do all of the heavy lifting. It will fetch the recipes that your buildout.cfg is using, and it will also fetch all of your application's dependencies. Once all of those things have been downloaded it will place them in a directory called eggs. The great thing about Buildout is that it doesn't install all of these things on your system Python. It puts everything in your eggs directory and that's it.

If your buildout.cfg is similar to this one you should end up with a few different things in the bin directory after having run bin/buildout. If you open one of those up and take a look, you should see right away how Buildout is able to download all of the dependencies that your application needs without needing to touch your system's Python. It is adding everything in the eggs directory to the beginning of your sys.path. The great thing about doing it that way is that if I decide that I want to use Django from SVN or a new release of Django or an older release of Django the only thing I need to do is update the version number in my setup.py file and re-run bin/buildout. My eggs directory will then include two different Django eggs and it will only add the one the matches my setup.py file to sys.path.

In conclusion, Buildout is great because it gives you an extremely simple, consistent and repeatable way to create your entire environment on any number of developer machines, virtual machines, test machines, production machines, or continuous integration machines and you don't have to worry about saying "DOH! I forgot to install module X on the production box".

My Crusade for Agility: Part 4

This is my 4th installment of My Crusade for Agility. In previous installments I've discussed my career up to the point where I'm aware of agile practices and I want to be in the agile space but I don't really know enough about agile to implement it myself. What I do know a little bit about is source control, continuous integration, automated tests and deployments. Luckily for me, my new employer already has the entire code base in Subversion. I had been toying around with TDD so there were some unit tests in our code base, but not many. I decided that if I could set up a continuous integration server and show the other developers how all of my tests automatically ran after every commit, they may begin to see some value in writing tests of their own.

I approached my lead about installing Hudson on our test server, to which he did not object. So I got it installed and spent more time than I care to remember trying to get Django's unit tests, Python's coverage, pylint, and clonedigger all running and working together and with Hudson's fancy graphs. I did get everything working though, and it was worth the extra effort. Now we had a way to test our code and check the quality of our code every time anyone would commit a change. The next thing that came was Trac integration, which was actually the suggestion of my lead. I felt like this was a huge win for my cause. Not only had someone else gone out and found a free open source alternative to our proprietary ticket system, but they also wanted to integrate it with our CI system. With Hudson's Trac integration we could link our commit messages directly to our tickets and vice versa.

The next thing that I wanted to implement was automated deployments. Unfortunately, this was not going to be very easy. Django has an excellent ORM, but it does not do anything to handle changes to existing tables, it really only handles the creation of new tables. To solve that problem, I found a project called South. South allowed us to create database migrations, so now we can easily apply any and all database (schema and data) changes quickly and easily using a Django management command.

Now that the database changes can be automated, I was ready to automate deployment. Previously, our deployment process was not very complex, but fairly manual. It consisted of SSHing to our test server and running a bash script that did an export from our Subversion repository. After running that script, we needed to restart WSGI (Apache's mod_wsgi - persistent Python interpreter) and ideally we needed to hit the website to get the start-up overhead out of the way. The first step that I took in making this process more automated was adding the running of our migrations, the wsgi restart, and a curl to our base url into the deploy script. That limited our deployment to SSHing to our test server and running a script. Then, it was a simple task to move that script into Hudson and have Hudson execute that script over SSH. Finally, having our build job trigger the deploy job in Hudson completed our deployment automation.

Now we were working in an environment where we could code, commit, see our tests, our coverage and other metrics, and then shortly after (with no further action on our parts after the commit) see our changes reflected on our test server. It doesn't get much better than that, does it? Oh, yes it does. We were still just getting started. Lots more to come, tune in for Part 5 coming soon to a web browser near you.

Process & Ceremony Over Results

Agile development is something that I've never really done professionally, but I really want to be in that space. Implementing agile practices to an organization that is not agile is not an easy thing to do. There will be resistance from every angle, BAs, QAs, developers, managers, etc. This morning, I had an experience that helped to clarify in my mind why agile techniques can be hard to adopt. My experience made me realize that by nature people favor process and ceremony over results. While most people are geared towards end results, intermediate results don't seem to matter much as long as the standard process and ceremonies are followed.

The experience that made the "bulb over my head" turn on was a very simple example how, without even thinking, I followed a process and ceremony with the end desired result in mind but ignoring the intermediate results. This morning, I walked into the bathroom and reached for the light switch. I moved my hand in the usual upward motion over the switch and made contact but didn't feel the typical "flip" and didn't hear the typical "click" of the switch being turned to the on position.

Following my usual ceremony and procedure, I tried again. Again, I made contact with the switch but the switch didn't flip and I didn't hear the comforting ceremonial "click" of the switch being flipped. It was at this point that I realized that while my intention of reaching the end result of having the lights turned on was completely distracted by my typical ceremony and process of flipping the light switch. In that same moment, I realized that the switch was already in the on position, and the lights were already on. I just needed to wake up and realize that the end result was already attained and I didn't need to follow my typical process twice, once or even at all.
Process and ceremony are hard things to bread once we get comfortable, but in some cases they overstay their welcome and usefulness. I could have walked into that room and gone about my business, but instead my typical ceremony got in my way and slowed me down.
Stay tuned for part one of a new blog post series that I'm going to start called "My Crusade for Agility".

My Crusade for Agility: Part 3

Welcome to the third installment of My Crusade for Agility. In the first two parts, I outlined the beginning of my career as a professional developer. I didn't really know what it meant to be agile, but I knew about some of the agile practices (like pair programming and tdd) and I really liked what I was seeing and hearing, but not yet doing. I wanted to be in a place where I could develop that way, I wanted to have more control over my own environment and be able to change it when it was not working. Part two left off with me deciding to accept an opportunity with a new employer. I felt confident that this new employer could provide me with the type of environment that I was seeking.

My new employer was a significantly smaller company, which was very appealing to me. Here I might have some say in what goes on, my opinions and ideas might be heard, and really listened to, and maybe even implemented. I knew going into it that the team that I was going to be working with was using Linux (as opposed to Windows like the rest of the company), MySQL instead of DB2, Apache instead of IIS and Python instead of Perl. Since the "corporate standard mold" had been broken, I was sure that they would be receptive to new ideas. I just need to build up some street cred and get past "the new guy" phase so I can start throwing some ideas into the ring.

Right off the bat, there were some things that concerned me. They were using a web framework that they had developed in house, which was less than ideal. Their proprietary framework was similar to the Java Servlet API, which is not a problem in and of itself. The problem was that there was not really any other organization or structure beyond that. There were HTML strings being pieced together, SQL strings being built, and business logic all happening in the same Python method. My first day on the job I was given a project, a new subsystem for this application. After digging through the existing code base and not seeing any indication of some sort of MVC structure I asked if I could write this new subsystem using the Django framework.
I thought that introducing a popular, robust, open source solution for something that they had historically done themselves could score some popularity points for me. I think, on my first day, I still had that "new guy smell" or something because nobody was really interested in letting me change everything they'd already done. So I played ball, I developed the new subsystem using their proprietary framework (with the addition of my own MVC layer on top of their framework). I did, however, introduce jQuery to do some ajax autocomplete in my subsystem, which was well received. I think the introduction of jQuery opened some minds to the use of open source frameworks.

The company agreed to send myself and another developer to PyCon and after the conference, and once jQuery had been accepted, and with my persistent inquiries into using Django, and the help of Cal Henderson's "Why I Hate Django" PyCon talk they finally caved in. (Apparently Cal's take on Django backfired, luckily for me, thanks Cal.) We took a few months to re-write the entire application using Django. I felt like this was a huge step forward. We now had an ORM, a full MVC, and a huge, active community helping us do our jobs. This was only the beginning, more, even greater changes were coming.
Check back soon for part 4 of My Crusade for Agility.

My Crusade for Agility: Part 2

This is the second installment of My Crusade for Agility, if you missed it check out Part One first.  As a quick recap, I went through college, not agile, I worked for a few years as a COBOL developer, not agile, then I moved to Java and despite all the new shiny tools I realized...still not agile.

So what is "agile" and why was I looking everywhere for it? Good question. At the time, I didn't know what I was looking for, I just knew that something wasn't right. Something with the way I was doing my job. I felt like I was writing good code, but it seemed to always end up becoming a lot of work to maintain. I didn't have any tests of any kind, I was spending a majority of my time in meetings, I was putting in all kinds of extra hours trying to make up all of the time I had spent in meetings, code reviews were getting put off until "later" (which never came), I was being pulled in tons of different directions working on multiple projects at the same time, and there was an overwhelming amount of work to do. Agile is what I wanted, I just didn't know it existed at the time.

I tried to implement some more agile practices, but I didn't really know what I was doing enough to sell anyone on it. After a few more years as a Java developer another opportunity presented itself with another (much smaller) company and I took it. There were a lot of reasons for me to move to a new employer, among those: my prior employer had been a very large company. One of the cons on working for a large company is that change is hard to implement, there is a lot of red tape and politics involved. I had a plenty of good things going for me, I had my co-worker's and customer's trust, I had a fair amount of influence as far as technology related decisions, but I was constantly wondering what if...

What if I didn't have to answer to corporate? What if my team was all Agile all the time? What if my team could use the right technology for the job and not have to conform to the corporate standards? What if I could deploy my own code, or change my own database, or have root access to a server? What if I could control my own professional destiny?
I asked myself these questions and I didn't really have an answer, because I'd never really experienced any of them. Part of what appealed to me about my new employer is that my interviewer had mentioned how the project I would be working on was on a small team, who had their own server, who was using an operating system different than the rest of the company, and a different programming language, and a different database, and a different application server... I couldn't believe the freedom. When I inquired into why they had decided to go a different direction for this project than the rest of the company the answer boiled down to "because we felt like it". Major selling point.

That concludes Part 2 of My Crusade for Agility. Tune again next time for Part 3 to see how my transition from a very large to a much smaller employer changed everything more than I could have ever imagined.

My Crusade for Agility: Part 1

Over the past few years I have been slowly introduced to agile development practices. Between a combination of conferences (like Iowa Code Camp, No Fluff Just Stuff, and PyCon), local user groups (like Central Iowa Java Users Group, The Iowa Ruby Brigade, The Iowa Python Users Group, and Agile Iowa), books (like The Pragmatic Programmer: From Journeyman to Master, Practices of an Agile Developer and Test Driven Development: By Example), and the extremely generous nature of agile shops in the area (Iowa Student Loan, GeoLearning, and CDS Global) (thanks Tim, Brandon and Trent) I feel like I have a decent understanding of what some good agile development practices are. Now that I have a taste for what it means to be an agile developer working on an agile team, I crave it, I need it, but I don't currently have it. This, and what I hope to be a series of follow up posts, is my crusade from the least agile imaginable developer, to whatever awkward agile puberty that I am currently in, to where I hope to one day be.


I'll start at the very beginning of my career as developer. I went to Indian Hills Community College right out of high school and got an Associates of Applied Science in the field of Computer Programming/Analysis. Indian Hills was great, I loved it there, it was hard core "code code code" the entire time. I learned a ton very quickly and got a job as a COBOL developer months before graduation. I enjoyed the challenges of real world COBOL development. I was still learning every day, I was solving problems, writing code, and working with some pretty awesome people, so I loved it. At the same time, there was a void growing inside of me that could not be filled by hacking away on top of hacks of hacks that had been hacked on since before I was born. I knew there was a better way, a more developer friendly approach to development. So I began doing side projects using PHP, Perl, Python, Ruby, Java and anything else that I could play with for free. I blamed the technology (COBOL) for the inadequacies of my environment. After a few short years as a COBOL developer I saw an opportunity to move into the Java world, professionally, and took it.


Moving from the COBOL to Java was a breath of fresh air. I had all of these nice fancy GUI tools and I was working on code that was written within the last decade. I was so smitten with working on more current technology (and source control!) that I was blind to the fact that although the technology changed, I was really just doing the same thing. Patching hacks of hacks with more hacks. There were no tests. Instead there was "spot checking" then passing the buck to BAs and QAs to do all of the "serious testing". Huge requirements documents that had been handed off from one person to the next, re-interpreted and re-translated again and again obfuscating what the business reallyneeded the software to do and why. Battling constant scope creep, missed deadlines, cancelled and delayed projects and fixing and re-fixing the same bugs over and over I began to realize that something here was broken.


Once I realized this, I also realized that I could be working on the latest and greatest technologies but if the development process was the same I might as well be writing COBOL again. I didn't want to write COBOL again, I wanted to be a part of the latest and greatest technologies, but so far my lateral move from one legacy code base to another just delayed my inevitable realization that different technology isn't what I was really seeking.
I will let that conclude Part 1 of My Crusade for Agility. I've got loads more that I want to share, so tune in next time for Part 2.

Django on Jython

Last night I presented a high level look at running the Django web framework on Jython at the Central Iowa Java Users Group. I posted all of my code examples and slides on github. If you have any follow up questions, comments or corrections I'd like to hear from you, so drop me a line. Thanks to everyone who was able to make it, I hope you enjoyed my presentation.