Python Mixins Continued

This is the second part of my original Python Mixins post. I left off mentioning first class functions vs. mixins, mixins being harmful, and classification of bacteria and lichen.

Let's start off talking about first class functions vs. mixins. What does a first class function have in common with a mixin or how could one replace the other, you may ask. To answer that question lets take a look at the Jsonable mixin from the original post.


Now let's see how we can instead use a first class function to do the same thing.



Let's look at some pros and cons of the mixin approach vs. the first class function approach. I like the function approach. It is simple, it gets the job done, and it is reusable. So, what is wrong with the function approach? As it currently is, nothing. However, when things change we may run into an issue. The issue is extensibility. Let's say that the save_json and date_handler functions are part of a 3rd party library that my application is using. If my application runs into a special scenario that requires some customization this approach becomes more difficult. What happens when I need to use a different date handler? With these functions existing in a 3rd party library my application cannot control the date handling without modification to the 3rd party library.

The mixin approach to Jsonable allows the extensibility where the first class functions do not. In our mixin example, our Person class could implement it's own date_handler method to modify the original functionality of Jsonable without having to modify the 3rd party source. Granted, both the mixin and the first class function approach can both be written in an extendable way but the mixin approach will result in cleaner code. To modify the function approach to meet our needs for a different date format we could either:

1. Pass the date format we want to the save_json function
2. Pass the date handler function to the save_json function

In this scenario, adding one additional argument is probably fine, but if we do not control that code, we don't have that many options. Also, while using functions in place of mixins can be extensible, at some point a function will end up having far too many parameters. The function approach is a good one as long as the functions are very small both in size and complexity. The functions should do one thing, do it well, and have hooks that allow for extensibility while maintaining a balance of simplicity and functionality. I think of mixins as an object oriented alternative to this first class utility function approach.

Mixins need to follow similar guidelines to first class functions. They should do one thing, do it well and have hooks to allow for extensibility while maintaining that perfect balance of simplicity and functionality. Mixins, like anything, can be abused. This article goes into more detail and gives some specific examples of why Mixins should be considered harmful. I do not think that any single design pattern should be considered harmful, however any design pattern can be abused.

I'd like to go take a step back now and examine classes, inheritance, multiple inheritance and mixins.

Let's start with some definitions:

Class - A construct that defines a type of object, the state that object may have and the behavior that it may exhibit.

Inheritance - A mechanism in which classes can extend other classes to fine-tune the classification of the parent class by providing additional state and/or behavior. Typically a class can extend one and only one parent class.

Multiple Inheritance - A mechanism in which classes can extend multiple other classes to not only fine-tune the classification of the parent classes but also combine those classifications into a new single entity.

Mixins - A collection of behavior that can be plugged into classes for reuse.

My hope is that you already have at least a basic understanding of classes and inheritance. If not I would be surprised if you're still reading. Let's take a look at a valid use of multiple inheritance.


Lichen are composite organisms that consist of a fungus and a symbiotic photosynthetic partner (like green algae). This is actually an excellent example of something where single inheritance falls short and multiple inheritance shines. With multiple inheritance a class is a composite of other classes, just like how lichen is a composite of other organisms.

Inheritance is a very powerful tool, and multiple inheritance is an even more powerful tool. These tools have their uses, but I feel like "mixing in" functionality is not one of them. There are much different reasons and uses for inheriting functionality from a parent and mixing in functionality from elsewhere. Inheritance is a tool that does not necessarily fit all use cases. Bacteria is a good example where the parent / child relationship breaks down. While bacteria reproduce asexually and technically do have a single parent, they are also capable of evolving a number of different ways which makes it difficult to nail down an exact parent / child relationship which will always hold true.

I'm going to cut this off here, I still have more material that I would like to cover regarding mixins in general and more specifically in Python, but this post is plenty long at this point. I will be posting a follow up to hopefully conclude my thoughts on mixins.

Thanks for reading!

Iowa Code Camp 10

Today was Iowa Code Camp 10. It was a great event. It is amazing to me how this event is completely free to all of the attendees. There were over 370 registrations, which is awesome.

I gave a talk about Continuous Deployment, which was very well attended (thanks everybody). Unfortunately, I was cut off by the clock before I could wrap everything up. Hopefully, everyone who attended was able to get some helpful info.

Here is my code and slides:

https://github.com/mattjmorrison/icc2012-rolodex-service

Use Tools

I'd like to retell an old story that has been around for a very long time. I honestly do not remember where I first heard it, I wish I did so that I could give some credit where credit is due.

The story begins in a village cut off from the rest of the world. There is a fresh water stream two miles from the village. Every day the villagers make multiple trips to the stream to get fresh water. The villagers have used the same buckets for as long as anyone can remember. The buckets are carved from stone therefore very heavy, and even heavier once they are full of water.

One day a band of travelers set up camp near the stream and some of the villagers saw them. They watched in awe as the traveler children played in the stream, splashing each other with buckets of water. The villagers were taken aback. They had never seen or even considered the possibility that something as difficult as carrying water in buckets could be done by children so easily for entertainment.

Initially the villagers were angry. It was insulting to them that one of their most important and most difficult jobs was literally child's play to these travelers. One brave villager approached the travelers and explained their situation. The travelers explained that their buckets were made of wood, and therefore very light and easy to carry. As a friendly gesture, the travelers gave the brave villager a wooden bucket as a gift.

The brave villager returned home with the bucket and showed everyone how a wooden bucket would make everyone's lives better. All of the villagers were very impressed. They disassembled the wooden bucket to see how it was made and began building their own buckets. The next day, they took their new wooden buckets to the stream and realized that their homemade wooden buckets were leaky. They went back to using the heavy stone buckets when they saw how the wooden buckets failed.

After re-examining the pieces of the wooden bucket that they had disassembled, they realized that their wooden buckets didn't have anything to plug the gaps in the wood like the one from the travelers did. The villagers took their leaky wooden buckets and filled in all of the gaps with thick mud. The next day they filled their new, leak-proof buckets with water only to realize that the mud that was used to full the gaps aslo made the water dirty. So, the returned to using the stone buckets.

The brave villager returned to the travelers and explained why the wooden buckets would not work. The travelers offered to give the villagers all the wooden buckets they would need if they would promise to use them instead of disassembling them and trying to create their own. The brave villager agreed and returned home with the new wooden buckets.

From that day forward carrying water from the stream was a much more trivial chore than the colossal task that it had been in the past. Once the villagers had swallowed their pride and just used the free buckets that were built by professionals their lives were easier and they had more time and resources to spend on things that matter to them and not waste time on making buckets.

This story really resonates with me. It's a perfect analogy to so many things in the software industry. It shows how dangerous silos can be, the real cost of technical debt and not keeping (at least relatively) current, why NIH and the 'roll-your-own' mindset is dangerous, and the value of taking advantage of free open source tools. Of course, there are always exceptions. Sometimes a bucket is not the right tool for the job, sometimes the right tool does not exist and must be invented, and other times buckets need to be slightly modified to be able to do the job at hand. So, to wrap it up I just want to say +1 to open source and +1 to forking.

Language Exploration: Part 2

The tl;dr version - The fibonacci sequence written in a bunch of languages, each language unit tested in a library for said language.  Languages with built in REPLs and testing libraries are very nice.

Carl Sagan included audio clips of greetings in 55 human languages on the Voyager Golden Record. If there is intelligent life outside of our world that discovers this golden record, one can imagine that it would quite literally be a modern equivalent of the Rosetta Stone.

This language exploration experiment I suppose is a sort of Rosetta Stone in some ways. While it is interesting to see the same program in a bunch of different languages, it most likely is not the best reference or teaching tool because, let's be honest, I haven't created the best code in all languages. I've just created code in all languages. Seeing the syntax for a language is easily a Google search away, however, seeing how to unit test a given language sometimes is a bit more tricky.

Unit testing is something that I am very passionate about, and all code can be tested. Part 2 of my language exploration experiment is to add unit testing for each language (now 20) for a simple fibonacci sequence implementation. There are a few guidelines that I forced myself to follow, like in Part 1.

• Must only use a plain text editor to write tests and code (no fancy IDE's)
• Must be able to run tests via the command line
• Must use a testing framework in each respective language

That's it! One last quick note before we get into the details. Languages that ship with built in testing frameworks are extremely helpful because you don't have to go through that initial phase of trying to find a decent and popular testing framework, then figuring out how to install it. Also, languages that ship with a REPL are much appreciated. Alright, on to the details!

Python
By day, I write Python, so this was super easy for me. It's what I do day in and day out, write tests for python and write code to make those tests pass. Here is my code. I used the Python standard library's unittest module. This was really easy to get going. Nothing to set up, nothing to install, very straight forward.

Ruby
I also write a lot of Ruby, so this one was also not very challenging for me to throw together. I used Ruby's built in Test::Unit module. Like Python, there was nothing to set up and nothing to install, it was very easy to get up and running. Here is my code.

PHP
I spend some time writing PHP as well. phpunit is pretty straight forward, writing the tests was no problem. The biggest problem that I had here is just figuring out where to put phpunit and how to get in on the path so php knew about it. I eventually decided to use pear to install it and then everything pretty much worked. I ended up installing phpunit system wide, which is a turnoff for me (being used to virtualenv and rvm). Here is my code.

Node.js
I am a web developer, so I write a lot of JavaScript and use Jasmine for testing. Here is my code. There were a few hangups with getting everything running smoothly. You will see in fibonacci.js there is a line at the end that exports the fib function. The exports object in node.js is the object that is returned from the require statement. This was necessary because I wanted the fib function to be available to the object returned from requiring fibonacci.js. The other gotcha was making sure everything was on the path that node uses. This is the reason that fibonacci.js is inside a directory called node_modules, node.js looks in node_modules by default, without that the path would have to be manipulated.

Java
In a former life, I was a full time Java developer. Without an IDE, this was kind of a pain with the junit.jar dependency and class path garbage (as you can see here), but the tests were easy to write and easy to run once the classpath was getting set correctly. Here is my code.

XSLT
XSLT is another language that I used a lot in a past life. As far as unit testing XSLT... wow, just wow. I honestly wasn't sure that this one was going to be worth my time, but it was surprisingly easy. XSLTUnit is a proof-of-concept xslt testing framework that seems to work great right out of the box. It's very XSLTish, XMLish, verbose and pretty gross, but you know what? If I had to write some XSLT, I would use this to test my code. Here is my code.

Perl
Again, I wrote a ton of Perl in a former life, but I never really did any testing. Thankfully, this one was extremely simple and I was actually pleasantly surprised. For something as simple as the fibonacci sequence, Perl's Test::More is extremely low friction to get the tests written and working. The one weird thing is that you have to tell the test runner in advance how many tests you intend on running, which I didn't like. Luckily, I found that you can call a subroutine called `done_testing` and the test runner will just figure it out by itself. I have a feeling like this may not scale up so well when there are more complex things to test that require mocking and complex setup, since the tests consist of a series of assertions in a bare file with no obvious setup/teardown or test isolation mechanism. Here is my code.

Clojure
Clojure is probably one of the few really interesting languages that I really like, but will most likely never use. The built-in clojure.test framework was super easy. Here is a great introduction and here is my code.

CoffeeScript
I really like CoffeeScript. I used Jasmine (again) for the tests. After already running into the path and exports issues for the Node tests I had pretty much already figured out all of the strange gotchas. Using "console.log module.paths" really helps to figure out why things don't seem to be working. Here is my code.

C
Writing unit tests, no wait, let me back up, figuring out how to write unit tests, nope let me back up further, finding libraries to make unit testing easy in C is hard. I attempted and failed to use Check and then Unity before finally settling with CU. CU works very well, and the documentation is excellent, but it is extremely tedious and un-dry. It seems like every C unit testing library that I looked at for more than a few minutes had the same essential flaw... no automatic test discovery. I have to write every test and manually add each test to a test suite individually. I'm sure there must be a way to automate that process, I wouldn't be surprised if there is one that I just was not able to find. Maybe someday I will revisit this and discover what I was missing. Here is a great introduction tutorial to CU and here is my code. As you can see, there is a lot more than a source file and a test file. There are header files, a makefile, and the source of the CU framework. It all needs to be compiled and linked before running. It definitely requires some experience to become proficient in testing C.

C++
This one started out much like C... there were a lot of options of testing frameworks to choose from and no obvious choice. Thanks to this article I was able to take a look at some examples and make a decision. I decided on CPPUnitLite because it looked the easiest to get setup and running. It was pretty easy to get going, the tests were easy to write and easy to read. Here is my code.

Scala
ScalaTest is pretty simple, it's easy to use and well documented. This requires the same classpath set up work as Java, but it's not a very big deal. Here is my code.

Go
Writing tests was easy, compiling and linking with Makefiles, not so easy. Here is my code. The code I ended up with doesn't look too bad, getting there was a different story. The first hick-up that I ran into once I got past the compiling and linking issues was that only functions that start with an upper case letter will be exported. My original fibonacci.fib did not work until I renamed my fib function to Fib. After I figured that out, the other thing I noticed was that there weren't any assertions (that I found) in the built in testing library. I ended up writing my own assertion which works fine, but seems like it should be unnecessary.

C#
Keep in mind this is C#, on the command line, with no IDE, using mono... this one sucked. There were a few challenges here. One of which was getting nunit setup, it turns out that nunit ships with mono (of course I didn't know that until after a lot of waisted time trying to figure out how to install it), so most of my work ended up being in vein. I ended up just having to add a flag to `gmcs` (which is the mono C# compiler) to point to nunit. The next challenge was to find where the dll for nunit lives in mono's installation (which you can see here). Another problem I had here was trying to find good resources online with documentation that does not refer to some GUI wizard to create tests and compile and run them outside of an editor. Unfortunately I didn't really come up with any good resources, I found a lot of bits and pieces here and there (mostly on StackOverflow) and I was able to piece everything together.... along with the --help option on `gmcs` and `nunit-console`. The actual test code was easy, pretty similar to Java in almost every way. Here is my code.

Dart
This one took a bit of searching to find a testing framework, and finally I came across bullseye.dart. Our of the box everything just seemed to work. There was a little bit of weirdness in the test output... I was seeing something like this:

null
 0 is 0
 1 is 1
 ...

It took me a while to figure out what "null" was supposed to be, but finally I just randomly came across something in Bullseye's docs that had a "get description()" function defined. That did the trick and switched "null" to my test name. Unfortunately, I didn't find a way to "install" bullseye, so I ended up just dropping it in with my code, which is a smell that I would prefer to avoid. Here is my code.

Erlang
This was not too bad to get up and running. Eunit was straight forward and well documented. Here is my code. The biggest challenge that I had here was figuring out how to execute the tests. This post helped out with that a lot, and I ended up with this.

Haskell
Hunit was surprisingly easy to set up and very easy to write tests. However, there is no automatic test discovery that I could see, so you have to manually configure tests into a test suite which isn't ideal. Here is my code.

F#
I had a lot of trouble actually getting an F# testing framework to work so I just fell back on NUnit. I'm not really sure why I couldn't get FsUnit or FsTest to work, they seem very light weight and simple, but I could not get them to compile. NUnit seems to works fine though. Here is my code.

OCaml
Finding a unit test library was easy, OUnit. The only tricky bit with getting it installed was that I had to manually install a prerequisite first, findlib. Once that was installed it was easy sailing. Again, there is no automatic test discovery, which is a modern convenience that us humans just shouldn't have to be without. Here is my code.

tcl

It was nice that tcl has tcltest which ships with the language, that is always a plus. Here is my code. These tests were easy to write as long as you don't forget the "expr" part. It was a little bit weird at first because when all tests were passing there was no output from running the tests... I found that adding "cleanupTests" at the end of the test file will print the final test results.

Finally, so summarize my findings:

1. Languages that ship with some kind of testing framework are easier to test because you can avoid the searching for and installing 3rd party libraries. This is especially important when you are new to the language.
2. Test frameworks without automatic test discovery are excessively tedious and they leave the possibility that you have real tests that are not actually being executed when you might think they are.
3. IDEs are not necessary. 
4. IDEs should be used once it becomes tedious or error prone to not use them (I'm looking at you Java, Scala, C#, and F#).  

That's it. That is my experience learning to write unit tests for a simple fibonacci algorithm in 20 different programming languages. Feel free to peruse all the code on github, and let me know if there are improvements that can be made, or languages that should be added. I would be more than happy to accept pull requests.

Continuous Integration: Where working code goes to die

It all started on July 1st, 2012, our continuous integration server (Ubuntu running Jenkins) was maxing out the CPU due to the leap second bug and had to be restarted. After the restart, Jenkins tried to "catch up", I guess, and kicked off a whole bunch of builds. Much to my surprise, many of the builds failed. After sifting through the failures, I realized that most of them had not run in months which isn't surprising considering that the code in the corresponding repository also had not changed in months. But what was going on here? No code had changed, yet the build was somehow mysteriously broken.

Let's take a step back and review our development process. We do test driven development and use continuous integration. Let's start with a quick TDD recap.

Our mantra is Red, Green, Refactor...

Step 1: Write a failing test.
Step 2: Run it.
Step 3: Make sure it fails for the reason we expected.
Step 4: Write code to make it pass.
Step 5: Run the test.
Step 6: Refactor our code (to make it right vs. just making it work).
Step 7: Run the test.
Step 8: Go to Step 1

That happens until a feature is complete and ready to be delivered to a QA environment. At that time the code changes are pushed to some central location and a continuous integration server is notified of that change (either via polling or post-receive hook). Continuous integration then will run a barrage of tests, some static analysis, code complexity, coverage stats and generate a bunch of neat charts and graphs and give you all kinds of interesting information about your code.

Then, assuming that our tests all pass and that our code complexity and quality are within the allowed thresholds a deployment job is triggered and our code is delivered to the target QA environment. That's great! Code is changed, tested, measured and delivered. Process complete, success, everybody is happy and everybody wins!

Then what happens? Say you've gone through this process, your application is live and stable and being used every day. The users are satisfied and you have moved on to developing a different component. Your original repository is not changing, your CI environment is not running any tests or giving you any feedback about the status of your production code. It is still living breathing production code, but it is not being tested anymore. But why would it need to be tested when it isn't changing? I'll tell you why, just because your code is not changing does not mean the rest of the world stopped changing, also. What about:

• Newer versions of dependencies
• there could be bug fixes, deprecations or changes to the api
• Server Updates
• New OS versions, OS package updates
• Newer versions of your language
• Java, Python, Ruby, whatever
• Newer versions of Browsers
• that may or may not be compatible with your version of Selenium or your client side code

Running tests when code changes is not frequently enough, especially when code is not changing. I know that from now on our builds will not only be triggered by code changes but also run on a periodic schedule, probably daily. So that as soon as something changes (code or not) that will cause a build to break I will know about it. A build that has been broken for months but never run is just not fun to fix.

Testing Inheritance with Inheritance

Unit testing and test driven development are things that I am very passionate about. Writing good unit tests is not always as straight forward as you would hope and it takes practice to really get good at it. Even then, there are still things that are inherently hard to test.

Inheritance is one area where unit testing becomes difficult. The difficulty does not necessarily come from actually writing the tests themselves (though obviously, some of it may be) but more in figuring out which tests to write, and how to go about doing it.

Say, for instance, I have the following class


with the following tests


And I want to extend this class for second graders to include multiplication and division. This is where I run into a dilemma. What is my first test for the second grade math implementation? I could start a number of different ways:

1. I could re-test drive addition and subtraction and refactor by adding FirstGradeMath as a parent class
2. I could copy the tests from FirstGradeMathTests and paste them into SecondGradeMathTests
3. I could assert that SecondGradeMath is a subclass of FirstGradeMath
4. I could Inherit the FirstGradeMath Tests

I'd like to review the pros and cons of each of those options.

With the first option, re-test driving addition and subtraction then making SecondGradeMath a subclass of FirstGradeMath seems like a tedious waist of time. This approach will not scale. Can you imagine writing the tests for Algebra, Trigonometry, and Calculus when each time you have to start with addition? However, this is probably the most pure test driven approach. If you prefer purity over pragmatism this may be your preferred approach.

In the second option, copying all of the tests and pasting them is going to be painful. Now you have identical tests that are exactly duplicated. Hopefully, you have some sort of static analysis tool that will let you know that you are sinning if you choose this approach. If, at some point, there are additional tests added for some edge cases the addition tests you will have to remember to duplicate them for FirstGradeMath and SecondGradeMath. This has the same scaling issue as the previous approach. How many tests are you going to have to copy, and how many times? On the positive side, your tests for SecondGradeMath do not make any assumptions about the implementation and assert that addition and subtraction both work correctly in addition to multiplication and division. 

The third option takes a very different approach than the first two. This option makes the assumption that FirstGradeMath is a solid implementation that has been thoroughly tested and is as correct as possible. The problem with this approach is that both the addition and the subtraction methods depend on data that is set in the constructor of FirstGradeMath. If, for some reason, the SecondGradeMath implementation does something differently in the constructor, it could potentially break the implementations of addition and subtraction. This may be a viable solution if you are comfortable making the necessary assumptions. If you control both the parent and the child class implementations, maybe you don't need to make any assumptions. This approach definitely has the least amount of friction, but has the potential to be risky.

The final option is somewhat of a middle ground between all of the first three approaches. By creating new tests for SecondGradeMath that inherit from the FirstGradeMath Tests you are getting a similar effect of copying the original FirstGradeMath tests. So you are covered from a regression standpoint. At the same time your test code is mirroring the implementation, so since SecondGradeMath extends FirstGradeMath, SecondGradeMathTests extends FirstGradeMathTests. 

Lets take a look at what the code would look like for the initial SecondGradeMath tests and implementation with the addition of multiplication and division:



There is one major issue with the tests at this point. They violate one of the FIRST rules of unit testing. (Fast, Isolated, Repeatable, Self-Validating, Timely). These tests aren't isolated. Each test is dependent upon some instance variable called "sut" being set up prior to the test running, and each test is dependent upon that object's state. Maybe this is alright if the only thing that we will ever need to add, subtract, multiply or divide is 2 and 2, but that is not the case here.

Inheriting unit tests requires an abstraction, an instance of a *GradeMath, however is not the correct abstraction. If we remove the "sut" variable from the setUp methods and just instantiate FirstGradeMath in the FirstGradeMathTests and SecondGradeMath in the SecondGradeMathTests then inheriting the FirstGradeMath tests are not going to really test anything about SecondGradeMath. Our abstraction level needs to be higher than an instance, so let's move up one level to the class. Here are the revised tests:


This is much better. Now each test class tests exactly one implementation but is free to instantiate that implementation however is necessary for the test at hand. This provides the isolation that the previous tests were missing.

I prefer this method of testing when inheritance is in the picture over most others. In the end, the best approach is most likely different depending on the situation.

Python Dependency Management: Part 2

If you read my post about Python Dependency Management, I'm sorry. I've changed my opinion about Buildout. While Buildout is still a great tool and does an excellent job of isolating your Python environment, it falls short in a few different areas.

The first area is documentation. It is sparse. Buildout is a tool that does a lot of things, I would even say too much, but you would never guess it by looking at the docs. While I'm sure that there are plenty of people who use Buildout, when you run into a problem and you can't find the solution on StackOverflow, IRC is really your only chance at getting help. Most likely, the only help you will find online is initial setup tutorials for using Buildout. Finally, buildout has the concept of recipes, which is an awesome idea. You can define your own custom recipe to build your project or a dependency or whatever you want. If you decide to go that route, you're completely on your own. The API for building a recipe is completely undocumented and you're better off copying an existing recipe and hacking on it until it does what you want. (that's what I did and it sort of worked).

Another thing that is an annoyance with Buildout is the bin directory. When I am working on a project that uses buildout I have to remember that $PROJECT_ROOT/bin/python is where my dependencies are and system python doesn't know anything about my project's dependencies. Some sort of shell wrapper (like RVM or Virtualenv which I will talk about later) would greatly improve the developer experience.

Lastly, I don't like having to have a bootstrap.py script that I have to run to set up Buildout, then a bin/buildout script that I have to run that requires a buildout.ini file and a setup.py. That seems like too much configuration, and too much cruft in my repo to manage a single thing. Maybe this is a bit nit picky, but the alternative is much less intrusive to my code base.

The alternative that I prefer is Virtualenv. Virtualenv has some of it's own weirdness that you need to overcome. First step is installing Virtualenv. This is easy:


Yes, I did use sudo there, but that's the only time (well almost) that you will need to install anything in system python. But something that manages isolated python environments is just fine to install system wide.

The next thing is creating a virtulaenv which looks something like this:


Once you've created it, then you have to use it. That's the first bit of weirdness. To use a virtualenv you do something like this:


Once that is done your prompt will change to look something like this:


Now you are using a virtualenv, at any time to stop using that virtualenv you can do this:


The big problem that I have with this is that now instead of Buildout's 3 required files in my project I have a directory that contains, geeze what is in there, oh yeah, ALL OF PYTHON. Sure I could just add that virtualenv name to my .gitignore, but the virtualenv's name is arbitrary, the next time I clone my project and create a new virtualenv I could call it blueberry_pie and I'd have to add that to .gitignore also. The alternative is creating the virtualenv somewhere outside of my project. So where should it go? I don't know, maybe I'll just have a bunch of virtualenvs floating all over my file system wherever I need them.




That sound bite pretty much sums up my feelings on that topic. Before I even get into what having a virtualenv means and what it can do, lets talk about how we can fix all that rigmarole. The first step is installing Virtualenvwrapper:


You may have noticed that this installation requires you to modify your ~/.bashrc. There is a very good reason for that. Virtualenvwrapper needs a single directory on your file system that it will use to house all of your virtualenvs. YAY! Also, I used sudo again. This is the last time, I promise. (sudo is ok for pip, virtualenv, and virtualenvwrapper and that's it!). Now that we are all set up with virtualenvwrapper, how do we use it? Easy. First step, like before, is create a virtualenv like so:


Well, well, well, would you look at that. Our prompt looks like:


after only a single command! Also, to stop using a virtualenv, it is the same as before:


But wait, how do I get back to it? Do I have to cd into $WORKON_HOME and do the whole:


rigmarole again? I think not! To activate a virtualenv using virtualenvwrapper simply do this:


Another extremely useful thing to do with the "workon" command is to run it with no arguments. Doing this will show you a list of every virtualenv on your system that you can activate using the "workon" command. Pretty great right?

But what does it all mean? What does an active virtualenv buy you? I'll tell you what it buys you, FREEDOM! Now when you type "python" in your shell you are actually using the python command that lives inside the bin directory of your virtualenv, same with pip, and same with any other python module that you install into your virtualenv. You don't have to do anything special, just use python and pip like you normally do (which you Do do, right?). If you get into trouble and you've installed a bad version of a package or you've manually screwed something up. Just deactivate your virtualenv and create a new one. No harm done.

What if you're done with a virtualenv? Just delete it like so:


Virtualenvwrapper provides many very useful, very succinct commands that you should check out. Now that you have all the tools you need to create a completely isolated Python environment you can use PIP to manage the dependencies required by your project. One guideline that I personally try to adhere to is that I set up a deployable project slightly differently than I set up a project that will be used as a library.

When I'm developing a deployable project, like a web application, I keep all of my dependencies listed in a requirements.txt file that pip can install using the '-r' flag. When developing a library I typically just list the dependencies in the setup.py using install_requires. I have, on occasion, made setup.py just read a requirements.txt file and populate the install_requires option dynamically, which may be a better approach. Also, in either case, keeping a separate development.txt requirements file that includes any testing dependencies and also includes a '-r requirements.txt' line is a must. The latter approach for managing dependencies for libraries using a requirements file is required when doing this.

One last thing I will mention about creating libraries is that make sure that you either document how to run your tests OR hook your test runner into "setup.py test". If everyone did the latter, the world would be a better place.

As with Buildout and virtualenv, virtualenvwrapper also has it's shortcomings. What I really want to see is something similar to RVM from the Ruby space. RVM allows you to create an isolated Ruby environment also, but it goes above and beyond what virtualenv and virtualenvwrapper provides. With RVM you can install different versions of ruby, and easily switch between which version is "active", and create different gemsets (a gemset would be similar to a virtualenv) and switch between which gemsets are active and install gems into a single gemset.


Virtualenvwrapper is excellent and if you're writing Python, you should be using it. Hopefully, someday, virtualenvwrapper will get to the point where it can manage versions of Python as well as installed Python modules. When that day comes I will be on board, until then I'm more than happy with PIP, virtualenv and virtualenvwrapper to manage my Python environments.

My Crusade for Agility: Part 13

Today was a good day, we just got a shipment of 3 brand spanking new white boards. I'm super excited to get those things hung up on the walls and see how they look. I'll share pictures as soon as they are available.

This is the 13th write-up that I've done regarding My Crusade for Agility. It has been 5 months since Part 12 and there have been quite a few really great developments. In Part 11, I mentioned that we had a new hire starting. He has been working out great and we have another new hire starting on Monday. We decided that with 4 full time developers working at our 2 tables in the pit, things are starting to get a bit cramped. So we decided to tear down some cubicle walls and make our area larger. (Pictures to come soon).

I mentioned "Another Challenge" in Part 12, which is getting the rest of the development team up to speed on all of the technologies that we are using. It's a hard problem to solve. Uncle Bob videos and Gary Bernhardt's Destroy All Software videos have been helpful. Also regular code katas where we typically pair and use TDD to solve some problem have yielded positive results. It's difficult to work a full time job and within that time teach and learn a completely new technology, or I should say a completely new category of technology that includes dozens of libraries and multiple languages.

In other news, we have taken another big step towards becoming more agile as far as our products are concerned. Up until recently we have had a single monolithic application that does everything. While it is nice as a user to be able to go to one place and have everything at your fingertips, some users don't want everything. What if we, as a company, want the ability to sell a single feature of our application as it's own product. That is difficult to do when all the code is in one place.

We decided to create an entirely stand alone application for our latest feature and built a web service api for integration with our existing product. Now our new feature is still a feature of our current application, but is also a stand alone product that can be marketed and sold separately. The development of this feature went extremely smooth. Without a doubt, we are going to be doing more features in a similar fashion.

The last thing that I will mention that I believe does contribute to our agility is Chef and Capistrano. As part of our last new web service feature we also had to tackle deployment and server setup type tasks. Being developers we automated it. We have Chef recipes that set up our web servers, database servers and even our development machines. We can get some brand new hardware with only an OS and have it up and ready to serve requests or ready to do some hard core development in a matter of minutes.

We are in a good place, a great place and more exciting things come up all the time. I will post a follow up soon with some pictures of our newly expanded development pit and the new whiteboards once they've been hung. Thanks for reading, check back soon.

Pycon 2012 Tutorials

PyCon 2012 is the 3rd PyCon that I have attended. This year I honestly didn't know what to expect. After attending the first year my team and I abandoned our proprietary web framework and began using Django. Then after the second year we automated our server configuration and application deployment processes. This year I was not sure what the next big thing for us would be.

This year I attended 3 of the 4 tutorial sessions.

The first day, the only tutorial I attended was "The Real Time Web with Co-Routines" with John Anderson. The content in this session was great. We essentially walked through a real-time chat web application using backbone.js, handlebars.js, socketio, and gevent. Due to an unfortunate release schedule most of the examples were broken for the duration of the tutorial. Luckily we were still able to see some real time web using coroutines in a demo app that was a real time web based chat client. This was somewhat hands on, I think a lot less than it was intended due to the bad timing of the release bug.

Day two I attended both tutorial sessions, the first of which was "Optimize Performance and Scalability with Parallelism and Concurrency" with Bob Hancock. This was an excellent talk, not so much an on-hands tutorial like I expected, more of an extended lecture. Not that that is a bad thing, there was a ton of information and it was all very thorough and extremely interesting. There were tons of low level details about how system calls work on POSIX systems. To sum up (maybe oversimplify is a better description for what I'm about to do) the talk, Bob had a program that opened a 60 GB file and searched each record for a given string.

There were a few different approaches that he used, the first was brute force which consisted of opening the file, looping over the records and using Python's "in" operator to search the record for the string, the next swapped the "in" operator for a regex. Some other methods included generators, threads, multi processes, futures, coroutines, gevent, and non-blocking io. All of his examples are on this github repo.

For the second half of the final tutorial day I attended "Devops for Python: Doing more with less" by Noah Kantrowitz. This was a very hands-on tutorial about managing virtual servers (specifically on EC2) using Chef and Fabric. This was actually pretty amazing. I do have some experience using EC2, Chef and Fabric, but I was impressed with how easy Chef makes working with EC2. Using Chef's "knife" command, we had a new Ubuntu EC2 instance running with only 6 (local) commands. Then, creating a second identical Ubuntu EC2 instance was only one additional command.

In Noah's talk, he walked through some of the common Chef terminology, did a quick crash course in Ruby syntax (which is used by Chef), then a crash course in Fabric. Mixed in between the lecture crash course parts of this tutorial we did small labs. The first lab was to get an EC2 instance up and running, the second lab was to write a sample apache Chef recipe, and the final lab was to write a Fabric script to start "chef-client" on each both Ubuntu machines.

That is the quick recap of the tutorials. I've got some more write-ups in the queue about the rest of PyCon 2012 this year, so stay tuned for those.

February 2012 Pyowa Meeting

Tonight the Iowa Python User Group met at The IMT Group in West Des Moines from 6 until 8 pm CST. This month's meeting was a little bit different than what we have typically seen in past Pyowa get togethers. Rather than having a local guest speaker, Mike Driscoll, the founder of Pyowa, was able to get Steve Holden and Doug Hellmann to agree to join our group via Skype. I was extremely excited about this. Steve and Doug are both respected members of the Python community, great speakers and authors.

Before we got started, Doug sent me a message saying that he wanted to be able to see what Steve had to say. This presented a bit of a problem for me, not being a regular Skype user, I wasn't sure how to make this happen. After a few minutes of messing with things we decided to abandon Skype and use Google+ Hangout, which seemed to get the job done.

Steve Holden started off without any prepared material, but led some very good discussions about the Python community and the evolution and future of the Python language. He started out by mentioning that PyCon 2012 in Santa Clara, CA was sold out for the first time in PyCon history. He went on to mention 3 Python and 3 Django conferences that he was in the process of planning that will be much smaller, more intimate, single track conferences (capped at 300 attendees, I believe he said).

There was also some discussion around spreading the word about Python, which based on the news about PyCon 2012, it sounds like the word is getting out. I'm glad that I got my PyCon ticket when I did! A question was asked about finding good training resources for developers new to the Python language. Steve mentioned ShowMeDo.com as well as PythonAnywhere.com as good online teaching resources and Steve and Doug both mentioned Mark Lutz's Learning Python book. The Learning Python book is where I learned a majority of what I know about the core Python language.

Having Steve Holden attend the Pyowa user group was awesome. I couldn't have been happier with how it went. It was an honor and my pleasure to have him attend our meeting. Steve had another engagement at 7pm CST, so he had to leave and miss Doug's talk about Sphinx.

Doug Hellmann had a talk prepared for PyCon 2012 called Better Documentation Through Automation: Creating Sphinx Extensions which he said that since he is not going to be able to attend PyCon this year would be a Pyowa exclusive talk. Sphinx is something that I have personally just started to scratch the surface in with a few small open source projects for documentation. Doug's Sphinx talk was really great, I was hoping to attend it at PyCon this year, and since Doug won't be able to make it I'm glad I was able to see it anyway.

I probably will not do Doug's presentation justice in this very short recap, but I'll do my best. Doug is very knowledgeable, was very well prepared and his delivery was interesting and fun to listen to. He started out by describing reStructuredText as "not a markup language" and "not layout like HTML" but rather a programming language for building documents. He compared "sphinx-build" to "gcc" in that "sphinx-build" takes an ".rst" file and builds it into an ".html" file similar to how "gcc" takes a ".c" file and builds it into "a.out".

Doug began with a simple example of how a custom role can make your documentation source more DRY. In his example he built a custom role that he was able to use in his documentation to create a hyperlink to ticket numbers in his BitBucket.org ticketing system. This made the documentation much cleaner and removed the room for error where a ticket number has the potential to link to the wrong place.

Doug's next example was a custom directive which would allow sql to be embedded in your document and when the documentation is built, the sql would be replaced with a table of the results of executing the sql. This really opened my eyes to the extensibility of Sphinx and made me realize how much it has the potential to do beyond just generating html from reStructuredText.

Finally, Doug showed an example from sphinx-contrib in sphinx-contrib-spelling where there is a custom sphinx builder that, instead of just rendering html output, it actually creates a file that contains misspelled words, that words' file and line number, as well as suggested spelling corrections. Pretty amazing and impressive stuff! I was extremely happy to have a speaker of Doug Hellmann's caliber present for the Pyowa user group.

One negative thing about our remote presenter set up was, during Doug's presentation, he had quite a few slides in his deck that were code examples, and the picture quality of Google+ hangout is not ideal for reading code. Maybe using Google+ Hangouts (with extras) would be better with the Google document sharing feature. Besides that, we did have one hick-up where we were temporarily kicked out of our hangout, but we recovered fairly quickly and easily.

One thing is for sure, I thoroughly enjoyed Doug Hellmann's presentation and the discussions with Steve Holden. I think this type of meeting, with a remote presenter, is really going to work well for the Pyowa user group. This meeting in particular had 10 total attendees (in person), which is fairly average, 4 of which were new faces, which was really great. I am definitely going to look for future opportunities to bring in outside presenters to speak at Pyowa meetings.

I think this first trial run of remote presenters was an absolute success and I want to send out a huge "Thank You" to Mike Driscoll for getting Steve Holden and Doug Hellmann to agree to talk to us, another huge "Thank You" to Steve Holden and Doug Hellmann for agreeing to talk with us and giving us a few hours of their time, and one final huge "Thank You" for the Pyowa attendees who were able to make it to the meeting tonight.

Learning CodeIgniter: Part 1

I've been working on a project using PHP and Code Igniter recently. While I have used both PHP and CodeIgniter in the past, this is the first project of this scale that I've been involved with using said technologies.

When I first joined this project there was already an established codebase. PHP and CodeIgniter were the primary technologies with the addition of Ruckusing for database migrations, Capistrano for deployment and Mercurial hosted on BitBucket for source control. Not a bad sounding set-up.

This post is undoubtedly going to fall into the tl;dr category, but I want to spend some time digging into the Code Igniter framework and do some comparisons and hopefully air some of my grievances related to not only the Code Igniter framework, but also the PHP language in general. This is going to end up as a multi-part post (as indicated by the "Part 1" in the title).

The first thing that I want to point out about CodeIgniter is the installation process. It is extraordinarily simple to do, it essentially boils down to unzipping CodeIgniter to a directory. While this is extremely easy and entirely fool-proof, I cannot honestly imagine a worse way to install framework code. The implications of this installation method are that every CodeIgniter project is a fork of the CodeIgniter framework. Imagine upgrading from one version of CodeIgniter to another (or just take a look at the instructions). Compare this to a tool like pip for Python or gem for Ruby, it's not even a comparison. Running a single command vs. following a set of instructions depending on your current version and target version. It's ludicrous, and I don't understand how any self respecting developer can be alright with this.

Alright, enough about my installation rant, let's take a look at some functionality that CodeIgniter provides. Looking at the directory structure of a CodeIgniter project, it seems very similar to Rails. In fact, according to the CodeIgniter documentation it is very similar to Rails in that it offers an MVC architecture, provides clean urls, and utilized a "modified version" of the Active Record pattern. The major difference that I see when comparing CodeIgniter and PHP to Rails and Ruby is that with Rails I don't have to write very much code to get a lot done and with CodeIgniter, unfortunately, I do. Let's look at some code.

Here is a simple controller action for a Rails app:

Here is a similar controller action for a CodeIgniter app:

Obviously, there are some pretty significant differences here. Granted, the CodeIgniter source could be refactored to remove a majority of the noise, but right out of the box none of that is done for you, whereas in Rails, it has already been done. Also, there is a lot of extra noise added by PHP's syntax vs Ruby's in this example. PHP's



compared to Ruby's



and CodeIgniter's


compared to Rails'



Also, I've found that CodeIgniter's so-called "modified Active Record pattern" is really not anything resembling the Active Record pattern at all. Essentially, CodeIgniter's modified "Active Record" pattern is simply dependency injection of a database handle into a model object. In the Rails example:

This creates a new Tag and saves it, which is abundantly clear by looking at the code. In the CodeIgniter example:


we must load the model, map the data from the post data to a temporary location that we will eventually use to insert into the database, use the tag model's db attribute and call the insert method, give it the name of the table that we are inserting into and an associative array of the column names and data we want inserted. Phew, that reeks of unnecessary ceremony and is absolutely not anything that resembles the Active Record pattern. But again, this could all be refactored to look better it's just up to me to do it.

Interacting with CodeIgniter's "ORM" is pretty painful at best. It really amounts to some sort of awkward PHP SQL wrapper code. To be honest, I'd much rather just write sql then use a PHP API that mimic's SQL closely enough that I'm calling db->insert, db->update, db->join, db->from, db->select, db->where, etc.

I have two last things to mention regarding CodeIgniter's database integration. One is DataMapper and the other is Ruckusing. These two projects extend CodeIgniter in different ways. DataMapper seems to fill some of the gap left by the out-of-the-box CodeIgniter ORM, however I've had a lot of issues getting it to just work the way it appears that it should. Ruckusing, on the other hand, does a very good job of handling database migrations. While Ruckusing could use a few helpers (similar to those provided by Rails and Django that are able to generate migrations) the api is fairly clean and easy enough to use and write migrations by hand.

This is where I will leave Part 1. There is much more to cover in future installments including but not limited too hooks, routing, controllers and alternative php syntax.