Why I do Agile: The Swamp King's Dilemna

The King of the Swamp People is not happy. Living in a swamp is really unpleasant. There’s mosquitoes and flies, alligators are always a danger and there is no dry ground to put buildings . So the King decides something must be done.

Managing the Project in a Traditional Way

The King gathers his workforce of cartographers, explorers and engineers around and tells them that the Swamp People are moving to the mountain. Why the mountain? Because the King has heard it’s dry and there aren’t any mosquitoes.

The cartographers immediately research the route using their library and begin drawing detailed maps to be compiled into a book. Once completed, the book is handed over to the explorers who begin blazing the trail to the mountain, faithfully following the book. It’s not long before the explorers encounter their first obstacle: a river. The cartographers knew the river was there and prepared for it. They immediately send the engineers to build the bridge they've carefully planned to handle the Swamp People and their belongings. The explorers find a rope bridge about a mile down river and use it to cross the river.

While the bridge is being finished, the explorers encounter their next obstacle: a desert. This isn’t on any map in the book so they send for the cartographers. The cartographers insist that there is no desert and the explorers must have gone in the wrong direction. This naturally makes the explorers angry because going in the right direction is what they are exceptionally good at doing. So one of the cartographers is sent to put everyone back on track. When he gets to the desert, the cartographer is silent. Clearly the explorers were right. He tells the explorers to stay there until the cartographers can develop a new plan back in the swamp. On his way back, the cartographer tells the engineers to stop work since they may not need the bridge after all.

The cartographers meet and decide they should build boats and attempt to reach the mountain via the river. They know they should first create all new maps before the explorers or engineers do anything, but the desert has really set them back. They tell the explorers to go downstream and the engineers to build boats.

As the explorers make their way down river they discover a road paralleling the river. They know they should tell the engineers about this but feel it’s more important for them to continue. When they get to the mouth of the river they find that it empties into an ocean. They send for the cartographers who once again insist that they’ve done something wrong. But of course, when a cartographer arrives, they immediately know the explorers are right. The cartographer tells the explorers to stay where they are and then informs the engineers they won't need boats since there is a road.

The cartographers confer and decide to have the engineers build ships and the explorers find their way to the mountain (forget about creating any new maps for them). The engineers, having never built an ocean going vessel before, head back to the library to learn how. The explorers, meanwhile, follow the beach to get to the mountain.

Despite their outwardly calm appearance, the cartographers are worried. The plan they originally laid out isn’t working and the Swamp king has no idea there have been any problems. They inform a very unhappy King that getting to the mountain is going to take longer than they originally anticipated.

Construction of a shipyard finally begins. The engineers, understanding that they are stepping out of their field of expertise, are still confident they can build ships that will transport their people and belongings. Unfortunately, however, the explorers have reached their destination and the news isn’t good. Sure the mountain is dry, but there is no drinking water. There are no mosquitoes, but the wind is fierce. The mountain is simply inhospitable.

The King is furious at the news. He’s spent a huge amount of resources and has nothing to show for it. He banishes the senior members of his workforce from the swamp.

An Agile Approach

The Swamp King gathers everyone around and begin discussing where the Swamp People should move. Many ideas are put forth but most people think that the mountain is a likely place to live. Being a wise leader, the King acknowledges that even though the mountain seems a good idea now, it may not be the best solution for his people. His declares his initial goal is to gather as much information as possible as quickly as possible. The King begins by selecting a cartographer, an explorer and an engineer to help him identify and explore a likely route to the mountain. The group picks a route to explore and departs.

When the team encounters the river, they immediately head back to the swamp to share their findings. The cartographer shows a rough map of their route, the explorer talks about the terrain and what supplies a migrating people may need and the engineer shares his thoughts on how they might build a bridge if it becomes necessary.

Two more teams are formed using members of the original team and some new people. One team is tasked with heading upriver and the other exploring downriver. Before they leave, an outpost is constructed on the shore of the river where supplies can be kept and messages sent. This also serves as an experiment in living in a different environment than a swamp. The cartographer from the original team stays at the outpost to refine his map. Because of the time he spent with the explorer and engineer, he clearly defines sources of food, water, and building materials as well as obstacles.

Not long after departing, the down river team discovers a rope bridge and immediately sends a message back to the outpost. A new team is formed to explore across the river.

Meanwhile, the upriver team returns to report that there is a town several miles away with a bridge suitable for moving a large number of people and their belongings. Envoys are sent to the town to begin negotiating use of their bridge.

The downriver team returns to report that the river empties into an ocean. To get to the mountain using this route, ships will have to be built. The engineers are concerned that they have no experience with constructing sea-worthy vessels. When the cross river team returns and reports they’ve run into a desert, the King realizes it may not be possible to get to the mountain., However, he has seen that it is possible to move his people to the river. The outpost has proven to be self-sufficient; the river provides plenty of food and water and the surrounding trees provide lumber. Sure there are mosquitoes, but there isn’t nearly as many as back in the swamp. It may not be the perfect solution, but it would substantially improve his people’s quality of life. The teams all agree. New teams are created to begin building the new town and to prepare the Swamp People for the migration to the river.

Observations

• In the traditional model, the most important thing was that each group of people were asked to solve predefined problems. In the agile model, each group was asked to find problems and suggest possible solutions.
• The second model accomplished much more in less time than the first and used fewer resources. In roughly the same amount of time it took the first model to find the desert and build part of a bridge, the second model found the desert, the ocean and the town, built an outpost and sent envoys the town.
• It seems that the second model is waterfall since everyone is working toward building documentation, but that’s not really true. A map is simply an artifact of a team’s experiences during their travel. I would equate these missions with programming spikes.
• In the first model, the King banishes his most senior workers leaving behind an inexperienced workforce further hurting his chances of improving his people's situation.
• In the second model, innovation happened. The cartographer saw what was important to the explorer and the engineer and created a way of showing that on a map. The only way this could happen is if the cartographer was working side by side with the explorer and the engineer.
• The outpost in the second model was clearly intended to be a temporary structure. It was meant to help the teams complete their work and learn something of the area, not to serve as part of the migration of the Swamp People.
• The number of people involved in the migration initially was small for the second model: the King, the cartographer, the explorer and the engineer. When the scope of the project increased and more people were required, teams were reformed with a mixture of old and new workers.
• In the first model, the cartographers did not trust what the explorers told them. This was not true of the second model.
• The amount of waste in the Traditional model is significant. The engineers designed and started a bridge that would never be used, the cartographers drew maps of areas that were later proven to be completely inaccurate, both the explorers and the engineers spent a lot of downtime while they waited for the cartographers to plan a new strategy, the engineers built boats they would never use and then later began work on a shipyard that was never needed.
  

Note to Self 14: Blogger FTP Problems

Way back in January, Google announced that they will no longer be supporting FTP publishing. Unfortunately for me, I was apparently one of only 0.5% of all people who actually relied on this offering. I finally buckled down and solved the problem. Here’s what I learned:

• The migration tool they offer (see the yellow box at the top of your blogger dashboard) is not going to work for you if you want to use a custom domain. You’ll have to use a manual process.
• Though the process is not difficult, you’ll want to do a bit of reading before diving into this. Making a mistake could cause you more grief than you really want. Here’s an article that I found particularly helpful.
• If you have downloads or images on your site that you want to maintain, they won’t be part of the migration. You’ll have to setup a missing files host. A missing files host is just another place to look in the event your page contains broken links. Blogger requires that your missing file host is a subdomain. I’d recommend you read this article to learn more.
• In the Blogger Publishing Settings, I set my domain name to www.darrellhawley.com so that I could maintain my original urls while my missing file host was set to www2. I was concerned that this might cause my problems, but I was pleasantly surprised.
• Repointing my domains is something that I always dread and this was no different though it appears I had nothing to worry about. I pointed my www2 subdomain at my original host and my naked domain (i.e., darrellhawley.com as opposed www.darrellhawley.com) at the IP addresses provided by Blogger.

Note to Self 13 – Whole Lot of Django

• Testing Django can be somewhat challenging. Instead of simply running a unittest module, you need to run the manage.py script in your Django app passing it the “test” parameter. This sets up a test database for your model, runs your tests against that model and then destroys your database. In effect, it turns your unit tests into functional tests.
• Another item regarding testing that bothers me is the hoops you have to jump through in order to use a testing framework other than unittest. We want to use Nose for our project and getting it to run in the Django environment requires that we install a Django App (django-nose) in our Project, and then make a couple of changes in our project settings file. I admit it doesn’t sound too bad, but when you compare it to the ease of simply running “nosetest” the story is not nearly as compelling.
• Putting testing aside for the moment, the rest of Django has been a pretty good experience. Both the templating language and the models are simple to learn and start using right away. Even though Django favors configuration over convention, the amount of configuration necessary is relatively small and hasn’t been much of a burden.
• The main configuration hassle has been mapping urls. The process can be simple, but leaves the door open for something much more elaborate. This is one of those things where I wish they would looked closely at Rails and adopt some of their practices: use conventions until it configurations become necessary.
• Going back to the manage.py file for a moment, I should say there are some cool things that you can do here. For those Rails developers out there, some of the functionality of Rake is contained in manage.py though certainly not all of it. Though the most common use of manage.py is syncing the database with your model (syncdb) and starting your development server (runserver), I find the shell command to be extremely useful for test driving models I’ve created.
• Though not specifically Django related, I feel Nosy is worth mentioning. I’ve become addicted to this script (go here for the original post) written by Jeff Winkler a few years ago. Like many others, I’ve taken the original script and modified it suit the needs of my project. In short, Nosy is a script that runs Nose anytime a python file in a given directory changes. It’s essentially like having a mini-ci server running on your desktop. I’ve found it very freeing to simply write code and be instantly notified if something breaks. No switching windows to run a test.
• As I mentioned before, the testing story for Django is not ideal. But this doesn’t mean you can’t improve the situation. I’ve made sure that the code going into our views.py file (which interestingly enough is really a controller) as close to pure framework code as I can. Most of my complexity is stored in a separate module that can be easily tested with Nose. To allow for even better testing, I make sure that all of the methods in the extra module accept a model as a parameter making mocking the models possible.
• In summation, Django absolutely makes creating websites much easier. Though testing is challenging, you can improve upon it if you put some though into it. Would I recommend Django? If you’re already comfortable in Python, than absolutely. If not, I’m not so sure. Rails is a compelling option considering it’s preference of convention over configuration. Keep in mind the required configurations in Django are very obvious whereas “stepping off the Rails” in RoR takes a bit knowledge of the framework.

Django at the Ann Arbor Software Development Study Group

I was supposed to lead a jam last December at the Ann Arbor Software Development Study Group, but a client project postponed it. On Tuesday, February 2, we’ll try one more time. The topic is “Zero to Django: Writing Web Apps with Python Using the Django Framework”. By the end of the session, you will be collecting form data and redisplaying it on a webpage.

NOTE: This exercise is meant to get you collecting data quickly and I’ll be skipping a lot of explanations. Feel free to ask questions or research any areas that you may find particularly tricky.

On with the jam!

Pre-Requisites:

Though there is only one requirement for this session - installing Django - you'll want to get this done ahead of time. It's not difficult, but there are a few steps and it will take enough time that it could keep you from completing all the exercises. To install Django, go to http://docs.djangoproject.com/en/dev/topics/install for a detailed description of what you need to do. Before following that link, here's a couple of suggestions:

• Be sure to install Python 2.5 (Python 2.6 will probably be fine). Django will not run with the 3.x series of Python.
• You can skip installing Apache and mod-wsgi. This session is only about how to use the framework which can be done completely within the development environment provided by Django.
• You can also skip the section on getting a database running. We'll be using SQLite which is baked into Python 2.5 and later versions.
• Be sure to install an official release of Django. Under "Installing an  official release", click on "download page" in line item 1. Use the version under "Option 1: Get the latest official version".

The Exercise:

Create a folder where all of your Django apps can you live. I'm using Windows and I'm putting all of my Django projects in the C:\django directory.

1. From the command line, navigate to the directory you just created
2. from the command line, run "python django-admin.py startproject people". You may have to explicity give the path of the django-admin.py file which can be found in python25/Lib/site-packages/[django dir]. If you copied the file as suggested by the installation directions, just use that version of the file instead. Also, you may get a "permission denied" message. If that's the case, check your permissions on the folder.
3. If you successfully completed step 3, you should now have a "people" directory. On my machine, the path is c:\django\people.
4. Navigate to the directory mentioned in Step 4 and run "python manage.py runserver"
5. Open up a browser and navigate to "http://localhost:8000/people". You should see a message congratulating you on "your first Django-powered Page"

Now that you’ve created a page, let’s configure our site so that we can do something useful.

• Django Projects contain one or more Apps (OK, they don’t have to contain Apps, but if you want to connect to a database they do). To create an app, run “python manage.py startapp dataentry”. The “manage.py” file can be found in the root of the people directory you created earlier. On my machine, the path is C:\django\people\manage.py.
• In the “settings.py” file contained within the project root, configure the database. For this exercise we’ll use SQLite since it comes bundled with Python 2.5 and all later versions:

  DATABASE_ENGINE = 'sqlite3'
  DATABASE_NAME = 'peopledb'
  DATABASE_USER = ''
  DATABASE_PASSWORD = ''
  DATABASE_HOST = ''
  DATABASE_PORT = ''

• In the settings file, you need to add your app to the INSTALLED_APPS section. Your INSTALLED_APPS section should look like the following (note the “people.dataentry” on the last line):

  INSTALLED_APPS = (
      'django.contrib.auth',
      'django.contrib.contenttypes',
      'django.contrib.sessions',
      'django.contrib.admin', 
      people.dataentry',
  )

• Again in the settings file, add “"/templates",” to the TEMPLATE_DIRS setting.
  
  Don’t worry about the extra lines. Those go beyond the scope of this session. If they really bother you, you can remove them. Just be aware that you’ll need add the extra comma (,) at the end so Python understands that this is a single item tuple.

Now that the configuration is done, it’s time to setup your database. To do that, we’ll need to open up model.py file inside your application directory. On my machine, that’s “c:\django\people\dataentry\model.py”. Your model should like the following:

from django.db import models

class Person(models.Model):
    T_SHIRT_SIZE_CHOICES = (
        ("WS","Women's Small"),
        ("WM","Women's Medium"),
        ("WL","Women's Large"),
        ("S","Small"),
        ("M","Medium"),
        ("L","Large"),
        ("XL","X-Large"),
        ("2XL","2X-Large"),
        ("3XL","3X-Large"),
    )
    name = models.CharField(max_length=30)
    t_shirt_size = models.CharField(max_length=10,
        choices=T_SHIRT_SIZE_CHOICES)
    special_dietary_concerns = models.BooleanField(
        default=False)
    special_dietary_concerns_comments = models.TextField(null=True,
        blank=True)

    def __unicode__(self):
        return "<Person %s %s %s>" % (
            self.name,
            self.t_shirt_size,
            self.special_dietary_concerns)

Now that your model is in place, let’s make sure that it’s reflected in the database.

1. From the command line in the directory where the model.py file lives (“c:\django\people\dataentry\model.py” on my machine), run “python manage.py syncdb”. A series of tables will be created.
2. You will be prompted to create a superuser. Type “yes” and continue through the prompts.

That’s it! The Person class is now in the database. But how do you use it.? To test it out, run “python manage.py shell” from the command line. You’ll be placed into Python’s Interactive Console. Now run the following commands

1. from dataentry.models import Person
2. p = Person()
3. p.name = “Me”
4. p.t_shirt_size = “L”
5. p.save()
6. Person.objects.get(id=1)

That last command should have printed “<Person Me L False>” in the console. Assuming it did, how do we get this form on a webpage? Before we do that, let’s make a webpage first.

1. Create a new folder called “templates” in the “c:\django\people\dataentry” directory.
2. In the “c:\django\people\dataentry\templates” directory, create a new file called “template.html”
3. In the template.html file, paste the following snippet “<h1>{{ hello }}</h1>” . Save the file.

You’ve just created a very basic template. To use it, make sure that your views.py file looks like the following:

from django.shortcuts import render_to_response

def hello(request):
    hello = "Hello, World"
    return render_to_response("template.html", locals())

Open “c:\django\people\urls.py” and make sure the following code is in it:

from django.conf.urls.defaults import *
from people.dataentry import views

urlpatterns = patterns('',
    (r'^hello/$', views.hello)
)

All the pieces should be in place for you to actually view your webpage. From the command line, run “python manage.py runserver”. Open your favorite web browser and navigate to http://localhost:8000/hello. You should be rewarded with “Hello, World” in a very large font. That’s all well and good, but what about that model we created? Let’s put it to good use right now. Let’s create a list.html in our template directory that looks something like the following:

<html>
    <head>
        <title>{{ title }}</title>
    </head>
    <body>
        <table>
            <tr>
                <th>Name</th>
                <th>T Shirt Size</th>
                <th>Special Diet</th>
                <th>Special Diet Comments</th>
            </tr>
        {%  for person in people %}
            <tr>
                <td>{{ person.name }}</td>
                <td>{{ person.t_shirt_size }}</td>
                <td>{{ person.special_dietary_concerns }}</td>
                <td>{{ person.special_dietary_concerns_comments }}</td>
            </tr>
        {% endfor %}
    </body>
</html>

Now let’s create a new method in our view so that our views.py file looks like the following:

from django.shortcuts import render_to_response
from people.dataentry.models import Person

def hello(request):
    hello = "Hello, World"
    return render_to_response("template.html", locals())

def list_people(request):
    title = "List of People"
    people = Person.objects.all()
    return render_to_response("list.html", locals())

Switch to the urls.py file and wire a url to our new method. Our updated urls.py file should look like the following:

from django.conf.urls.defaults import *
from people.dataentry import views

urlpatterns = patterns('',
    (r'^hello/$', views.hello),
    (r'^list/$', views.list_people),
)

From your web browser, navigate to http://localhost:8000/list. You should see the person you entered in the Interactive Console. Cool! By now you should see a pattern on how to create a new page in Django: create a template, create a new method in the view and then edit the urls.py file. Let’s add a bit more complexity by creating a form. Create a new forms.py file in the “C:\django\people\dataentry” directory. Put the following code block in that new file:

from django import forms
from models import Person

class PersonForm(forms.ModelForm):
    class Meta(object):
        model = Person

Create a new template in the template folder called “form.html” and put the following html inside it:

<html>
    <head>
        <title>Adding a Person</title>
    </head>
    <body>
        <form method="post">
            <table>
                {{ form.as_table }}
            </table>
            <input type='submit'>
        </form>
    </body>
</html>

Switch over to your view.py file and modify it so that it looks like the following:

from django.shortcuts import render_to_response, HttpResponseRedirect
from people.dataentry.models import Person
from people.dataentry.forms import PersonForm

def hello(request):
    hello = "Hello, World"
    return render_to_response("template.html", locals())

def list_people(request):
    title = "List of People"
    people = Person.objects.all()
    return render_to_response("list.html", locals())

def add_person(request):
    title = "Add Person"
    if 'name' in request.POST:
        form = PersonForm(request.POST)
        if form.errors:
            errors = form.errors
        else:
            form.save()
            return HttpResponseRedirect("../list")
    else:
        form = PersonForm()
    return render_to_response("form.html", locals())

Now all you have to do is wire up the url in the urls.py file. Now when you successfully add a new person using the form you will be redirected back to the list of people where you will see your entry at the bottom of the list. 

IterHelper Released

I’ve mentioned IterHelper in my last few posts and I’m glad to say that I finally released a version of it that I feel is solid. I’ve run my unit tests on Python versions 2.5 and 2.6 and also on IronPython 2.0 and 2.6.

It doesn’t do anything earth-shattering at this point, but it does have some functions I find particularly useful such as a couple of filter methods and a number of “skip” and “take” methods. For a complete rundown of what it does, just run “help(IterHelper)”

I’ve used LaunchPad to manage my project and the experience has been good thus far. If you want to download it, you’ll need to do so from the trunk. I’m eventually moving to Python eggs, but until I work out the particulars, you’ll have to do include it in your project manually.