Darrell Hawley: Home Page

Django at the Ann Arbor Software Development Study Group

2010-01-24T13:21:00.001-08:00

I was supposed to lead a jam last December at the Ann Arbor Software Development Study Group, but a client project postponed it. On Tuesday, February 2, we’ll try one more time. The topic is “Zero to Django: Writing Web Apps with Python Using the Django Framework”. By the end of the session, you will be collecting form data and redisplaying it on a webpage.

NOTE: This exercise is meant to get you collecting data quickly and I’ll be skipping a lot of explanations. Feel free to ask questions or research any areas that you may find particularly tricky.

On with the jam!

Pre-Requisites:

Though there is only one requirement for this session - installing Django - you'll want to get this done ahead of time. It's not difficult, but there are a few steps and it will take enough time that it could keep you from completing all the exercises. To install Django, go to http://docs.djangoproject.com/en/dev/topics/install for a detailed description of what you need to do. Before following that link, here's a couple of suggestions:

Be sure to install Python 2.5 (Python 2.6 will probably be fine). Django will not run with the 3.x series of Python.
You can skip installing Apache and mod-wsgi. This session is only about how to use the framework which can be done completely within the development environment provided by Django.
You can also skip the section on getting a database running. We'll be using SQLite which is baked into Python 2.5 and later versions.
Be sure to install an official release of Django. Under "Installing an official release", click on "download page" in line item 1. Use the version under "Option 1: Get the latest official version".

The Exercise:

Create a folder where all of your Django apps can you live. I'm using Windows and I'm putting all of my Django projects in the C:\django directory.

From the command line, navigate to the directory you just created
from the command line, run "python django-admin.py startproject people". You may have to explicity give the path of the django-admin.py file which can be found in python25/Lib/site-packages/[django dir]. If you copied the file as suggested by the installation directions, just use that version of the file instead. Also, you may get a "permission denied" message. If that's the case, check your permissions on the folder.
If you successfully completed step 3, you should now have a "people" directory. On my machine, the path is c:\django\people.
Navigate to the directory mentioned in Step 4 and run "python manage.py runserver"
Open up a browser and navigate to "http://localhost:8000/people". You should see a message congratulating you on "your first Django-powered Page"

Now that you’ve created a page, let’s configure our site so that we can do something useful.

Django Projects contain one or more Apps (OK, they don’t have to contain Apps, but if you want to connect to a database they do). To create an app, run “python manage.py startapp dataentry”. The “manage.py” file can be found in the root of the people directory you created earlier. On my machine, the path is C:\django\people\manage.py.
In the “settings.py” file contained within the project root, configure the database. For this exercise we’ll use SQLite since it comes bundled with Python 2.5 and all later versions:

DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'peopledb'
DATABASE_USER = ''
DATABASE_PASSWORD = ''
DATABASE_HOST = ''
DATABASE_PORT = ''

In the settings file, you need to add your app to the INSTALLED_APPS section. Your INSTALLED_APPS section should look like the following (note the “people.dataentry” on the last line):
INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.admin',
    people.dataentry',
)
Again in the settings file, add “"/templates",” to the TEMPLATE_DIRS setting.

Don’t worry about the extra lines. Those go beyond the scope of this session. If they really bother you, you can remove them. Just be aware that you’ll need add the extra comma (,) at the end so Python understands that this is a single item tuple.

Now that the configuration is done, it’s time to setup your database. To do that, we’ll need to open up model.py file inside your application directory. On my machine, that’s “c:\django\people\dataentry\model.py”. Your model should like the following:

from django.db import models

class Person(models.Model):
    T_SHIRT_SIZE_CHOICES = (
        ("WS","Women's Small"),
        ("WM","Women's Medium"),
        ("WL","Women's Large"),
        ("S","Small"),
        ("M","Medium"),
        ("L","Large"),
        ("XL","X-Large"),
        ("2XL","2X-Large"),
        ("3XL","3X-Large"),
    )
    name = models.CharField(max_length=30)
    t_shirt_size = models.CharField(max_length=10,
        choices=T_SHIRT_SIZE_CHOICES)
    special_dietary_concerns = models.BooleanField(
        default=False)
    special_dietary_concerns_comments = models.TextField(null=True,
        blank=True)

    def __unicode__(self):
        return "<Person %s %s %s>" % (
            self.name,
            self.t_shirt_size,
            self.special_dietary_concerns)

Now that your model is in place, let’s make sure that it’s reflected in the database.

From the command line in the directory where the model.py file lives (“c:\django\people\dataentry\model.py” on my machine), run “python manage.py syncdb”. A series of tables will be created.
You will be prompted to create a superuser. Type “yes” and continue through the prompts.

That’s it! The Person class is now in the database. But how do you use it.? To test it out, run “python manage.py shell” from the command line. You’ll be placed into Python’s Interactive Console. Now run the following commands

from dataentry.models import Person
p = Person()
p.name = “Me”
p.t_shirt_size = “L”
p.save()
Person.objects.get(id=1)

That last command should have printed “<Person Me L False>” in the console. Assuming it did, how do we get this form on a webpage? Before we do that, let’s make a webpage first.

Create a new folder called “templates” in the “c:\django\people\dataentry” directory.
In the “c:\django\people\dataentry\templates” directory, create a new file called “template.html”
In the template.html file, paste the following snippet “<h1>{{ hello }}</h1>” . Save the file.

You’ve just created a very basic template. To use it, make sure that your views.py file looks like the following:

from django.shortcuts import render_to_response

def hello(request):
hello = "Hello, World"
return render_to_response("template.html", locals())

Open “c:\django\people\urls.py” and make sure the following code is in it:

from django.conf.urls.defaults import *
from people.dataentry import views

urlpatterns = patterns('',
(r'^hello/$', views.hello)
)

All the pieces should be in place for you to actually view your webpage. From the command line, run “python manage.py runserver”. Open your favorite web browser and navigate to http://localhost:8000/hello. You should be rewarded with “Hello, World” in a very large font. That’s all well and good, but what about that model we created? Let’s put it to good use right now. Let’s create a list.html in our template directory that looks something like the following:

<html>
    <head>
        <title>{{ title }}</title>
    </head>
    <body>
        <table>
            <tr>
                <th>Name</th>
                <th>T Shirt Size</th>
                <th>Special Diet</th>
                <th>Special Diet Comments</th>
            </tr>
        {% for person in people %}
            <tr>
                <td>{{ person.name }}</td>
                <td>{{ person.t_shirt_size }}</td>
                <td>{{ person.special_dietary_concerns }}</td>
                <td>{{ person.special_dietary_concerns_comments }}</td>
            </tr>
        {% endfor %}
    </body>
</html>

Now let’s create a new method in our view so that our views.py file looks like the following:

from django.shortcuts import render_to_response
from people.dataentry.models import Person

def hello(request):
    hello = "Hello, World"
    return render_to_response("template.html", locals())

def list_people(request):
    title = "List of People"
    people = Person.objects.all()
    return render_to_response("list.html", locals())

Switch to the urls.py file and wire a url to our new method. Our updated urls.py file should look like the following:

from django.conf.urls.defaults import *
from people.dataentry import views

urlpatterns = patterns('',
(r'^hello/$', views.hello),
(r'^list/$', views.list_people),
)

From your web browser, navigate to http://localhost:8000/list. You should see the person you entered in the Interactive Console. Cool! By now you should see a pattern on how to create a new page in Django: create a template, create a new method in the view and then edit the urls.py file. Let’s add a bit more complexity by creating a form. Create a new forms.py file in the “C:\django\people\dataentry” directory. Put the following code block in that new file:

from django import forms
from models import Person

class PersonForm(forms.ModelForm):
class Meta(object):
model = Person

Create a new template in the template folder called “form.html” and put the following html inside it:

<html>
    <head>
        <title>Adding a Person</title>
    </head>
    <body>
        <form method="post">
            <table>
                {{ form.as_table }}
            </table>
            <input type='submit'>
        </form>
    </body>
</html>

Switch over to your view.py file and modify it so that it looks like the following:

from django.shortcuts import render_to_response, HttpResponseRedirect
from people.dataentry.models import Person
from people.dataentry.forms import PersonForm

def hello(request):
    hello = "Hello, World"
    return render_to_response("template.html", locals())

def list_people(request):
    title = "List of People"
    people = Person.objects.all()
    return render_to_response("list.html", locals())

def add_person(request):
    title = "Add Person"
    if 'name' in request.POST:
        form = PersonForm(request.POST)
        if form.errors:
            errors = form.errors
        else:
            form.save()
            return HttpResponseRedirect("../list")
    else:
        form = PersonForm()
    return render_to_response("form.html", locals())

Now all you have to do is wire up the url in the urls.py file. Now when you successfully add a new person using the form you will be redirected back to the list of people where you will see your entry at the bottom of the list.

IterHelper Released

2009-12-29T09:53:00.001-08:00

I’ve mentioned IterHelper in my last few posts and I’m glad to say that I finally released a version of it that I feel is solid. I’ve run my unit tests on Python versions 2.5 and 2.6 and also on IronPython 2.0 and 2.6.

It doesn’t do anything earth-shattering at this point, but it does have some functions I find particularly useful such as a couple of filter methods and a number of “skip” and “take” methods. For a complete rundown of what it does, just run “help(IterHelper)”

I’ve used LaunchPad to manage my project and the experience has been good thus far. If you want to download it, you’ll need to do so from the trunk. I’m eventually moving to Python eggs, but until I work out the particulars, you’ll have to do include it in your project manually.

Euler Problem 1 Revisited

2009-11-29T17:22:00.001-08:00

The other day at the Software Development Study Group in Ann Arbor, Chris Marinos was demonstrating F#, a functional programming language targeting the .NET framework. Of course, this got me thinking about programming styles in Python. So what better way to compare and contrast programming styles than with Euler problems? As you probably don’t recall, the first Euler problem is:

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.

Find the sum of all the multiples of 3 or 5 below 1000.

The traditional approach to this might look something like the following:

def traditional_approach():
    total = 0
    for i in range(1,1000):
        if i % 5 == 0 or i % 3 == 0:
            total += i
    return total

It’s not a bad approach, but there are more concise approaches. Here is my original solution using a list comprehension:

isValid = lambda x: x % 5 == 0 or x % 3 == 0

def with_list_comprehension():
filteredValues = [x for x in range(1,1000) if isValid(x)]
return sum(filteredValues)

I like list comprehensions, but this feature does introduce a lot of noise characters. If you use list comprehensions, you probably like this solution, otherwise you’re looking for something else. The next approach uses the itertools module (and the isValid lambda from the previous example).

def with_itertools():
filteredValues = itertools.ifilter(isValid, range(1,1000))
return sum(filteredValues)

This approach removes a couple of noise characters and is structured in a way familiar to most developers. My last solution uses a module called IterHelper that I proposed in a previous post. As I mentioned in that post, IterHelper sits on top of the itertools module and implements some of the recipes from the official itertools documentation. It’s largely inspired by my desire to use LINQ from both Python and IronPython. Here’s my solution to Euler Problem 1 using IterHelper:

def with_iterhelper():
    return IterHelper(range(1,1000))\
        .where(isValid)\
        .sum()

Thanks to the Python continuation character, a backslash, I can put discreet units of functionality on their own line, making it easier for me to consume.

If you decide to download and try out IterHelper, be aware that I’m not even considering it alpha yet. It’s still very much in the R & D phase and I’m planning on changing some of the method names.

Note to Self 12 – Lessons Learned from Our Latest Project

2009-10-28T14:14:00.001-07:00

I know I’m supposed to be releasing my IterHelper tool right now, but I’ve been on a bit of death march lately and have had no time. Though the project hasn’t ended yet, there are some items that I need to store away for future use before they’re forgotten.

On small teams, non-development related responsibilities should be divided among the team. Then the single non-dev responsibility (hopefully it’s the only one) should always take precedence over development responsibilities. It sounds backwards, but no matter what my priorities, I’m always going to find a way to write code. The same isn’t true of project management.
I can’t say enough about having a non-developer - who I’ll refer to as the Customer Advocate – on the project team. Things we had problems making time for – client meetings and acceptance testing for instance – fell to the Customer Advocate which takes a lot of pressure off everyone else.
Story points are great, but in order to get value out of them you actually have to use them – obvious, I know. Estimating tasks and then tracking how many points were completed daily, turned an out of control project into a manageable one.
Speaking of scoring, I wish we would have re-scored each task after we completed it. It would have taken minimal effort while at the same time providing us with enough information to improve our estimates.
We started using Agile Zen at the beginning of our project, but eventually abandoned it in favor of physical notecards. This introduced a lot of challenges, but it did allow us to grow our process exactly the way we wanted it. The added benefit is we’re in a much better position to evaluate the various kanban applications for our next project. To be clear, Agile Zen is a great product it just wasn’t what we needed at the time.
We didn’t use Selenium or Watir to run acceptance tests. Even if we couldn’t run all the tests we wanted, it would have been a huge time saver. A costly mistake that won’t be repeated.

Does IronPython need LINQ?

2009-09-10T15:17:00.001-07:00

Before I get started, I think I need to clear up some misconceptions that the Python community might have about LINQ. From the posts I’ve read, Python developers have dismissed LINQ as only a series of language enhancements that allow for SQL-like querying of lists. Though it is true that both C# and VB.NET have made changes to their respective languages for working with lists, you can still use LINQ without the aforementioned language enhancements. Python developers would be better served to think of LINQ as wrappers around the map, filter and reduce functions. In fact, LINQ is not much different than the “recipes” found in the official itertools documentation.

So the question still stands: does IronPython need LINQ? In my last post, I described how to use LINQ from an IronPython application and implied that I didn’t think it was the best idea. Don’t get me wrong, I really enjoy writing LINQ statements, I just don’t think LINQ fits well with IronPython. Why not? The first problem I have with the IronPython/LINQ combination is put on prominent display in this code sample.

print Enumerable.Average[object](
list, Func[object, int](lambda x:x))

If you are a static language aficionado, you probably are wondering what the problem is. If you’re a Python developer, however, the fact that I’m explicitly naming the “object” and “int” types is sending up giant, red flags. Python folks don’t really care what type of object it is they’re using, they only want to know what the object can do and what it contains (i.e., duck-typing). It’s much easier – and more readable - to simply write a completely Python version of the Average function. Consider the following chunk of LINQ code from C#:

List<string> values = new List<string>("the quick brown fox jumped over the lazy dogs".Split());

double avgLengthOfLongWords = values
                .Select(x => x.Length)
                .Where(x => x > 3)
                .Average();

Console.WriteLine(avgLengthOfLongWords)

To convert that to IronPython, we’d be forced to write something like the following:

import clr
clr.AddReferenceToFileAndPath("c:\users\darrell\desktop\System.Core.dll")

import System.Linq
from System import Func
from System.Linq import Enumerable

stuff = "the quick brown fox jumped over the lazy dogs".split()

def Select(col, func):
    return Enumerable.Select[object, object](col, Func[object, object](func))
def Where(col, func):
    return Enumerable.Where[object](col, Func[object, bool](func))
def Average(col):
    return Enumerable.Average[object](
    col, Func[object, float](lambda x:x))

avgLengthOfLongWords = Average(Where(Select(stuff, lambda x:len(x)), lambda x:x > 3))
print avgLengthOfLongWords

Before I’m even able to write useable code, I need to add a reference to a dll, import a namespace and a couple of classes and then write some helper functions; all so that I can write LINQ statements without the benefits of chaining. That’s right – I have to nest my LINQ statements. No chaining. That’s because when we use IronPython, the Select and Where don’t return an IEnumerable or an IQueryable. Instead they returna WhereSelectEnumerableIterator.

But do you know what really rankles me about using LINQ in IronPython? It contributes absolutely nothing to the Python community. The only way you can use LINQ is if you are using IronPython, not CPython, not Jython. I want to engage the entire Python community, not just my corner of the world. If ever there was a case where implementing the principle trumps migrating the implementation, this is it.

Enter the itertools module. Itertools is a Python standard library module containing tools for working with iterators. It also will serve as the cornerstone for my implementation of LINQ. When you visit the official itertools documentation at python.org, you’re even presented with code “recipes” that could be described as LINQ-ish. The only problem is that the recipes only accomodate nesting of statements. Why is this a problem? Consider the differences in readability between nested and chained in the following example:

stuff = "the quick brown fox jumped over the lazy dogs".split()

getLength = lambda x:len(x)
isNotTooLong = lambda x:x<6
isNotTooShort = lambda x:x>3

nested = ifilter(isNotTooShort, ifilter(isNotTooLong, imap(getLength, stuff)))

chained = IterHelper(stuff).select(getLength).where(isNotTooLong).where(isNotTooShort)

As always is the case with nesting methods, you have to read the statement populating nested backwards. That is, the outermost function call is the last to be run and you work your way inwards from there to find the origin. This is not a big deal to most of us since this is something we had to get used to very early in our careers. But when you compare that to the statement populating chained, I know I can read it from left to right. My eyes aren’t jumping back and forth to understand what is happening in what should be a relatively simple statement.

So what does IterHelper look like?

class IterHelper(object):
    def __init__(self, someIterable):
        self.someIterable = someIterable
    def __iter__(self):
        return self.someIterable
    def next(self):
        return self.someIterable.next()
    def select(self, func):
        return IterHelper(itertools.imap(func, self.someIterable))
    def where(self, predicate):
        return IterHelper(itertools.ifilter(predicate, self.someIterable))

This extremely simplistic version of IterHelper is really only a wrapper around functions found in the itertools module. This is great news since any application written using Python 2.3 or later can make use of this pattern with little effort. There’s no special libraries AND everyone is invited regardless of your flavor of Python.

My plan is to release IterHelper as an open source project by sometime by mid-October. I have a well-tested, basic implementation now, but I still have questions regarding licensing, hosting and deployment. I will be doing a bit of research on these topics and be posting what I’ve learned in my Notes to Self series.

IronPython and LINQ: A Devlink Correction

2009-08-15T20:19:00.001-07:00

During my “Anatomy of an IronPython Application” presentation at Devlink, I mistakenly said that LINQ can be consumed from IronPython easily. I said this because I was thinking that IronPython 2.6 was built using .NET 3.5 when in fact it’s built with .NET 2.0. Fortunately, if you want to use LINQ, you still can because .NET 3.5 is really .NET 2.0 plus extensions; you just have a bit of extra work to do. First on the todo list is to copy the System.Core.dll for 3.5 to some other location (I was able to get to it from it’s home in the c:\windows\assembly directory) and then use the “clr.AddReferenceToFileAndPath” in your IP code to gain access. Then, with a simple import of the Enumerable or Queryable type, you magically have access to LINQ.

Still, the story doesn’t end there. You have to remember that IronPython does NOT support extension methods and since LINQ is pretty much a collection of extensions methods, there’s a problem. Why this design decision? Because in order to support extension methods, the IronPython team would have had to make changes to the core language which would have, of course, been rather un-Pythonic. This means that the code for consuming LINQ is going to be somewhat messy. The following snippet gives an example of consuming the Average function of the Enumerable object.

clr.AddReferenceToFileAndPath(r"C:\folder\System.Core.dll")
from System.Linq import Enumerable
from System import Func

list = [1,2,3,4,5]

print Enumerable.Average[object](
list, Func[object, int](lambda x:x))

I have a lot of concerns as soon as I see “object” and “int” in brackets. At this point I’m going to hold save those concerns until my next post. Instead I’m going to leave this as an example of consuming LINQ from IronPython.

I should point out that Harry Pierson’s post on IronPython and Linq-to-XML really got me started on this topic. I also have to give a shout-out to Chris Smith who was kind enough to help me out and relay a couple of CodeMash stories at the same time.

Speaking at the Lansing Day of .NET on Saturday

2009-07-31T07:31:00.001-07:00

If you’re going to be at the Lansing Day of .NET this Saturday, stop by and catch my Anatomy of an IronPython Application presentation. I’ve completely rewritten the talk so if you’ve seen it before, this new version will be covering completely different topics including testing and mocking. I’ll also be giving away a copy of IronPython in Action, courtesy of Manning Publications, at the end of my talk.

Note to Self 11 – Patterns in Python, Process is Everything, Selenium

2009-07-25T05:01:00.001-07:00

While studying a particular portion of IronPython in Action, I came across a footnote to a blog post entitled Patterns in Python. By the end of the post, I found myself mentally tweaking a lot of my code. Though quite old (the page was posted February of 2003), the content remains quite relevant.
I’m fortunate to be working on a project where everyone involved is committed to being agile (note the lack of a capital “A”). We’ve done enough things right that the items I would do differently are very minor. With that said, the imbalance in hours spent by each team member seems to be causing a problem…
Another interesting observation – the project that I feel very fortunate to be working on is not using the technology of my choice. It’s a Java project. In fact, almost every facet of the project requires some specific technology in which I almost know nothing, yet the development process makes the experience a joy.
Configuring my Selenium tests to work with Selenium RC and IE8 has not been an easy task. Some things I learned along the way…
- Easiest way to create a new Selenium test is to use the Selenium IDE plugin for FireFox. It records everything you click and do and allows you to convert the recording to C#, Java, Python, PHP or Ruby. From there, further customizations are a snap.
- Selenium RC allows you to run your Selenium tests in browsers other than FireFox, such as IE, Chrome and Safari.
- Before running your tests in one of these “other” browsers, you’ll need to start the Selenium Server. Selenium Server comes parceled with the Selenium RC download and can be started from the command line with “java -jar selenium-server.jar”.
- To run my Selenium tests with IE8, I had to use the *iexploreproxy option even though I’m not using a proxy. Otherwise I kept getting redirected to a phantom file supposedly hidden away in my user directory.

Book Review: IronPython in Action

2009-07-14T08:24:00.001-07:00

IronPython in Action (IPIA) gets the distinction of being the first book published on IronPython. As the first book, it has the responsibility of giving developers their first impression of IronPython. IPIA does exactly that by taking numerous shallow dives into various technologies and techniques. To be clear, this book is aimed at professional developers preferably with either .NET or Python experience. Pythonistas will find the book offers quick introductions to a number of Microsoft technologies such as Windows Forms, WPF and ASP.NET. .NET developers will get the chance to see technologies they’re already familiar with expressed in a completely new fashion.

IPIA avoids the trap of simply being a brochure of Microsoft technologies using “IronPython” in place of “C#” and “VB.NET”. Instead, the authors mix in chapters on development techniques ranging from the familiar - such as unit testing and mocking – to the more exotic such as metaprogramming. As .NET developers wind their way through the book, they’ll begin to appreciate the differences between the static and dynamic worlds – a crucial step toward wide ranging acceptance.

If you’re curious about what IronPython can do for you, IronPython in Action is well worth your time.

Note to Self 10 – JQuery Selectors and Custom Functions and a thought on JQuery

2009-06-28T09:01:00.001-07:00

After using Python, diving into jQuery has proven exceptionally difficult. Even though it solves the not-so-small problem of commonizing the DOM across multiple browsers, I find it very difficult to read. I know with time I’ll get used to it and maybe even find some beauty in my jQuery code, but it will never stop me from wishing for a cleaner syntax like that of Python or Ruby.

With that said, here’s a bit of what I learned about jQuery recently.

Selectors in jQuery are a syntax for selecting an html element. Selectors can return a single element or several (or none if nothing matching is found).
$(“#[elementID]”) – finds an element with an id attribute equal to “elementID”.
$(“[name=’elementName’]”) – gets all elements with a name attribute equal to “elementName”
$(“[name^=’element’]”) – gets all elements with a name attribute that begins with “element”
$(“[name$=’Name’]”) – gets all elements with a name attribute that ends with “Name”
$(“input:checkbox”) – gets all checkboxes
$(“#blah > input:checkbox”) – gets all checkboxes within a parent element of id “blah”
Creating a custom function in jQuery:

$.fn.CustomFunctionName({function()({
//code here
})

$.fn.exclaim = function(message){
alert(message)
}
$.fn.exclaim("hello, world")

Note to Self 9 – Exploring SharpDevelop and Going off Topic

2009-05-29T07:05:00.001-07:00

It’s been too long since I last blogged which typically means I have too many competing ideas to settle on just one. So welcome to Note to Self 9!

SharpDevelop is an open source IDE for the .NET platform. Is it new? Hardly. Based on the history of news releases from their website, It’s been around since December of 2000. Considering that the .NET platform was publicly announced only 6 months earlier, SharpDevelop seems only that much more mature.
If you’re comfortable with the basics of Visual Studio, SharpDevelop should feel like home to you. In fact, the best way to describe SharpDevelop is “Visual Studio without a lot of stuff I don’t want and a few thid-party items I do”.
SharpDevelop uses NUnit instead of MSTest. That’s right. No third party plugins to make your IDE work with NUnit. It simply works.
It’s time for me to confess. It’s all about IronPython. The reason I started investigating SharpDevelop is because of my frustration with Visual Studio’s lack of support for the language. Specifically, I want Intellisense when I write IronPython. To be sure, you only get Intellisense for “non-DLR code”, but surprisingly that doesn’t seem to be as much of a handicap as you might think.
Speaking of first class citizens, SharpDevelop comes stock with C#, Visual Basic, Boo (I can hear Jay Wren cheering now), IronPython and F# (now a number of SRT folks are cheering).
If you’re frightened of the consequences that may come from changing your development environment, don’t be. You can open one of your existing Visual Studio solutions with SharpDevelop and start coding. Switching back is as simple as opening up your now modified project/solution in Visual Studio.
For clarity, SharpDevelop is new to me. I haven’t used SharpDevelop for any projects yet so I can’t tell you about any of its nuances. I plan on putting it to the test, however, over the next several weeks. I’ll be posting about my experiences.
Now for the off-topic part. I’ve been listening to The History of Rome, a FANTASTIC podcast series on, well, the history of Rome. Mike Duncan, creator and narrator, is not only an engaging story teller, but also has the technical savvy to put together a professional podcast. Though it’s not the sort of podcast typically on a geek’s menu, I think there are enough parallels to some of the challenges we are facing in the software industry to make it worthwhile.

Anatomy of an IronPython Application at AADND

2009-04-07T20:00:00.001-07:00

I'm going to be talking IronPython at the Ann Arbor .NET Developers Group at SRT Solutions tomorrow night (April 8) at 6:00. I'll be showing you how you might introduce IronPython into your existing .NET environment through a lot of sample code and very few PowerPoint slides. I hope to see you there.

Anatomy of an IronPython Application

You've heard about IronPython and maybe have even taken the time to write the obligatory "Hello, World" application. Now what? The buzz around IronPython continues but there's still little guidance on tools and development processes. This presentation focuses on the basics of interacting with C# and VB.NET libraries using IronPython while introducing useful tools and techniques to get you started. Though experience with .NET is necessary, the samples and discussions are understandable to even beginning Python developers.

Writing an IronPython WCF Host

2009-03-15T06:19:00.001-07:00

This is a simple WCF host written in IronPython that consumes a library I wrote in C#.

import clr
clr.AddReferenceByPartialName("System.ServiceModel")
clr.AddReferenceByPartialName("CSharpClass")

from System.ServiceModel import ServiceHost, BasicHttpBinding
from System.ServiceModel.Description import ServiceMetadataBehavior
from CSharpClass import PersonService, IPersonService
from System import Array, Uri

uri = Uri("http://localhost:1234/people")
uriArray = Array[Uri]([uri]) # creates a generic array of type Uri
serviceHost = ServiceHost(PersonService, uriArray)

binding = BasicHttpBinding()

serviceHost.AddServiceEndpoint(IPersonService, binding, "http://localhost:1234/people")

behavior = ServiceMetadataBehavior()
behavior.HttpGetEnabled = True
serviceHost.Description.Behaviors.Add(behavior)

serviceHost.Open()

raw_input("Service has been started. Press Enter to exit...")

Running nose with IronPython

2009-03-06T11:28:00.001-08:00

I just spent the past week at the 2009 Microsoft MVP Summit where I got a chance to talk to the some of the crew behind the dynamic languages at Microsoft. When I mentioned that I had gotten nose, a Python unit testing framework, working with IronPython, I was strongly encouraged to blog about it. Without further adieu, here's my code.

import sys
sys.path.append(r"C:\Python25\Lib\site-packages\nose-0.10.4-py2.5.egg")
sys.path.append(r"C:\python25\lib")
sys.path.append(r"C:\Python25\Lib\email")
sys.path.append(r"C:\Python25\Scripts")
import nose
nose.main()

I dropped this script into a directory with a couple of bogus unit tests and ran it with Python 2.5.4. Results were consistent with my expectations - successful tests passed and failing tests threw exceptions. When I ran the same suite against IronPython 2.0.0, I still received the correct results but the script took 1 minute and 20 seconds to complete. Compare that to less than a second (estimated) for CPython. For the sake of completeness, I ran the same test with IronPython 2.0.1 and received the same result.

The root of the problem appears to be the large number of GeneratorExitExceptions being thrown. I'm not sure why they're being thrown, but I'll continue to investigate and report my findings. If you're interested in the gory details, here's the stack trace of the first exception:

at IronPython.Runtime.PythonGenerator.ThrowThrowable()
at IronPython.Runtime.PythonGenerator.CheckThrowable()
at IronPython.Runtime.PythonGenerator.
CheckThrowableAndReturnSendValue()
at IronPython.Runtime.Operations.PythonOps.
GeneratorCheckThrowableAndReturnSendValue(Object self)
at S$11.lambda_method$312(Closure , Int32& state, Object& current) in C:\python25\lib\types.py:line 52
at Microsoft.Scripting.Runtime.GeneratorEnumerator`1.
System.Collections.IEnumerator.MoveNext()
at IronPython.Runtime.PythonGenerator.MoveNextWorker()
at IronPython.Runtime.PythonGenerator.System.
Collections.IEnumerator.MoveNext()
at IronPython.Runtime.PythonGenerator.throw(Object type, Object value, Object traceback)
at IronPython.Runtime.PythonGenerator.throw(Object type)
at IronPython.Runtime.PythonGenerator.close()

IronPython Exception: [A]Person cannot be cast to [B]Person

2009-02-20T12:54:00.001-08:00

Had an interesting problem while using the Hosting API the other day. Here's the exception message.

[A]CSharpClass.Person cannot be cast to [B]CSharpClass.Person. Type A originates from 'CSharpClass, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in the context 'LoadNeither' at location '[project directory]\bin\Debug\CSharpClass.dll'. Type B originates from 'CSharpClass, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in the context 'Default' at location '[temp directory]\CSharpClass.DLL'.

I was struggling with this for a while until I found this link. In order to get my IronPython scripts to communicate with my CLR-based code, I was using the clr module's AddReference methods. These methods require you to point at the appropriate copy of the library you want to consume. I do stress "copy". Let's say you point your IronPython script at the bin\debug folder of a C# library project you're consuming in a console application. When you run the application, you'll get the exception message above. To make this example work, you'd have to point your IronPython script at the bin\Debug folder of your console application instead. Clearly, this is a maintenance nightmare. Instead, insert the following chunk of code into your hosting application (in the example described above, your console application) to avoid the problem all together.

Assembly assembly = typeof (CSharpClass.Person).Assembly;
runtime.LoadAssembly(assembly);

Note to Self 8 - IronPython2, Hosting, DLR

2009-02-04T19:03:00.001-08:00

The latest DLR code and documentation can be found at www.codeplex.com/dlr
For the examples below, here is the Python script I'm using

class Person(object):
    def __init__(self, name, phone):
        self.name = name
        self.phone = phone
    def TalkToCSharp(self):
        val = {"name":self.name, "phone":self.phone}
        return val

def GetPerson():
    person = Person("darrell", "555-4321")
    return person.TalkToCSharp()

def GetPersonWithParams(name, phone):
    person = Person(name, phone)
    return person.TalkToCSharp()

theNumberFive = 5
myPerson = GetPerson()

Simplest thing you can do is execute a Python script.

static void ExecutePythonScript(string path)
{
ScriptRuntime runtime = Python.CreateRuntime();
runtime.ExecuteFile(path);
}

Sometimes you need to get a variable after a script has run.

static void GetDoubleVariable(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    object x = scope.GetVariable("theNumberFive");
    Console.WriteLine(x);
}

What happens if you're trying to return a custom class defined in the script? The easiest - and IMHO the best - way to go about this is to convert the object to a dictionary before passing it to your C# code.

static void GetInstanceCustomClass(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    PythonDictionary dictionary =
        scope.GetVariable("myPerson")
        as PythonDictionary;
    foreach (var property in dictionary)
        Console.WriteLine(property.Key +
            " == " + property.Value);
}

Getting properties is definitely useful, but you'll probably want to call a function.

static void CallingAFunction(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    ObjectOperations op =
        scope.Engine.CreateOperations();
    ObjectHandle handle =
        scope.GetVariableHandle("GetPerson");
    var x = op.Call(handle, new object[0]);
    PythonDictionary dictionary =
        x.Unwrap() as PythonDictionary;
    foreach (var property in dictionary)
        Console.WriteLine(property.Key +
            " == " + property.Value);
}

Here's an example of passing parameters to a Python function.

static void CallingAFunctionWithParameters(string path)
{
    ScriptRuntime runtime = Python.CreateRuntime();
    ScriptScope scope = runtime.ExecuteFile(path);
    ObjectOperations op =
        scope.Engine.CreateOperations();
    ObjectHandle handle =
        scope.GetVariableHandle("GetPersonWithParams");
    var x = op.Call(handle, "Larry", "555-1234");
    PythonDictionary dictionary =
        x.Unwrap() as PythonDictionary;
    foreach (var property in dictionary)
        Console.WriteLine(property.Key +
            " == " + property.Value);
}

Note to Self 7 - Naming Python Modules, SciTE, IronPython and name

2009-01-29T08:14:00.001-08:00

Be careful naming your Python modules. If you name a Python file the same as an existing file in your Python install directory, you may get the following exception message: "'module' object has no attribute '[some-attribute-name]'".
I'm really loving SciTE. It's a dead simple code editor that offers syntax highlighting for an impressive number of languages and displays the results of the run in it's own output window.
More SciTE: If you decide to use SciTE, you'll probably want to customize how it works. Here's a nice quick reference to supplement the SciTE documentation to help guide you through the config files.
If you want to use SciTE as your IronPython editor, you'll need to modify the existing python.properties file. Select "Open python.properties" from the "Options" menu. Search for "command.go" and modify the value so that it points at your IronPython directory.
All Python modules have a special variable called "__name__". When the code in your module is executed with CPython, __name__ is set to the file name of your module minus the "py" extension ("mymodule" if the file name of your module is "mymodule.py", for example), except when the module is executed directly. If that's the case, __name__ is equal to "__main__". Unfortunately, this is not the case if you execute it with IronPython. The special variable __name__ will always be based on the file name. UPDATE: The above is true only when you compile your IronPython code. If you launch a *.py file with ipy.exe, the __name__ variable will be "__main__".

Note to Self 6 - IronPython, pyc.py and SciTE

2009-01-19T20:09:00.001-08:00

The pyc.py file can compile you IronPython code into .NET executables.
The pyc.py file is part of the sample code released with IronPython. It can be found here. Look for the "IronPython-2.0-Samples.zip" link.
Typing out "[IP directory]\ipy [code directory]pyc.py" explicitly everytime you want to use the tool is somewhat annoying. Consider creating a script to do this for you. Here's a simple batch file.

cd C:\data\FeClass
c:\fepython\ipy.exe pyc.py class.py /main:main.py

Note that you'll have to copy the pyc.py file to the same directory as your code for this to work.
IronPython Studio just isn't up to snuff. You're better off to use a text editor and a build script
I like SciTE as my IronPython editor even more so than Notepad++. I really like that I can run my Python or IronPython script from SciTE and the output the normally would be displayed in the command window is captured in SciTE.

SQL Alchemy ORM Basics

2009-01-11T20:25:00.001-08:00

In my previous post, Learning SQL Alchemy, I focused on the SQL-like aspect of SQL Alchemy. You'll need to review the sample code in that post before understanding the code below. On with the code.

class User(object): pass
mapper(User, user_table)

Believe it or not, we just created a class with an ID, Name and Password. Simply create an empty class and pass it to the the mapper along with the user_table we had created. How do we use it? First off we have to create a Session object. Note that when we call the sessionmaker, it returns a Session class which you use to create the session.

Session = sqlalchemy.orm.sessionmaker()
session = Session()

All have we to do now is create a new User object, set the appropriate fields and pass it to the "save" method.

user = User()
user.name = "Fred Flintstone"
user.password = "yaba daba doo"
session.save(user)

But how do we get our user back from the database?

query = session.query(User)

We can now iterate through the query like any list.

for user in query:
print user.id, user.name, user.password

You can see the new record we inserted along with the other records in the table.

2 angie password2
3 nate awesomeness
4 Fred Flintstone yaba daba doo

The important thing to note is that the query object is actually a list of the very same User type I defined at the beginning of this post. Of course we probably don't want everything in the table, so here's an example of getting back a single object.

for user in query.filter_by(name = "angie"):
print user.id, user.name, user.password

The result is just what we would expect.

2 angie password2

But let's say I want to change Angie's password.

user.password = "a much more secure passphrase"

for user in query.filter_by(name = "angie"):
print user.id, user.name, user.password

The result?

2 angie a much more secure passphrase

Learning SQL Alchemy

2009-01-07T17:58:00.001-08:00

I spent a portion of my holidays studying the basics of SQL Alchemy. My first lesson is that SQL Alchemy is NOT an ORM for Python but instead a SQL abstraction tool with an ORM built-into it. At first I was having trouble understanding what this really meant, but after seeing an example it sunk in. Let's start by building a table.

from sqlalchemy import Table, Column, Integer, String

meta = sqlalchemy.MetaData("sqlite:///")

user_table = Table('userTable', meta,
    Column('id', Integer, primary_key=True),
    Column('name', String, unique=True, nullable=False),
    Column('password', String, nullable=False))

meta.create_all()

First thing we need is a MetaData object which holds all of our database schema information as well as our engine for executing queries. By passing "sqlite:///" in the MetData constructor we create an engine pointing at an in-memory SQLite database. To create a table, we just declare an instance of the Table object and pass in the necessary parameters: table name, our MetData instance and whatever columns we need in the table. The constructor for the Column class not only takes the column name and type, but constraints as well. Then all we have to do is execute the statement by using the create_all() method of the MetaData class.

Did you note the user_table variable we created? That comes in really handy when we want to start manipulating our new table.

insert = user_table.insert()
insert.execute(name='darrell', password='password1')
insert.execute(name='angie', password='password2')
insert.execute(name='nate', password='password3')
insert.execute(name='drew', password='password4')

See how easy inserting data is? There are methods just like the insert method above for select, insert and delete. Let's take a look at the select method.

select = user_table.select()
results = select.execute()

for row in results:
print row.name, row.password

The result are just as you would expect.

darrell password1
angie password2
nate password3
drew password4

That's great, but how do we change Nate's password and then return only his record?

update = user_table.update(user_table.c.name=='nate')
update.execute(password='awesomeness')

select = user_table.select(user_table.c.name=='nate')
results = select.execute()

for row in results:
print row.name, row.password

It's very simple and readable code with one caveat: what is "user_table.c.name"? It's a convenience attribute containing a collection of all the columns in userTable. So what our update statement is saying in terms of SQL is "UPDATE userTable SET ????? WHERE name = 'nate'". The only thing we don't know at this point is what to change. That happens when we execute the statement on the next line. If you understand the update statement above, the select statement works exactly the same way. Our result is below.

nate awesomeness

We've seen select, update and insert, but what about delete?

delete = user_table.delete(user_table.c.name.like("d%"))
delete.execute()
select = user_table.select()
results = select.execute()

I couldn't help myself - I had to confuse the issue by using "LIKE". In my defense, now you know how to introduce a "LIKE" comparison into your code AND that the "c" attribute is more than just a collection of strings. Beyond that, the delete works exactly the way you would expect it to work: call the delete method by passing the constraining expression as a parameter and then execute the result.

Now I understand why SQL Alchemy is an SQL abstration tool. All I did in the examples above were execute very basic SQL statements; the ORM was never used. Why is this so powerful? Imagine being able to write the same code for a SQL Server database as you did for a MySQL server and having the confidence that it's going to work. That's the power of SQL Alchemy.

Congratulations 2009 Microsoft MVP!

2009-01-02T06:26:00.001-08:00

That's the subject line of an email I received yesterday. No one is more surprised than me. Considering who the other MVPs are, I'm honored to be among them. I'm not entirely sure what I did to deserve this, but I'm grateful all the same. Thank you, Microsoft, for noticing me and here's to a productive 2009. Cheers!

Note to Self 5 - Inherit and Override

2008-12-27T08:37:00.001-08:00

I've been reading Working Effectively with Legacy Code by Michael C. Feathers. It's a trove of useful information for turning your unreadable and unreliable code into the complete opposite.

Here's a useful technique for dealing with dependencies that have problematic methods for testing. Mark the offending method as virtual and inherit from the class. Now you can use this new derivative class in your test methods in place of the original. Here's an example in C#.

public class MyClass
{
    public MyClass()
    {
        Console.WriteLine("MyClass has been initialized");
    }

    public virtual void DifficultMethod()
    {
        Console.WriteLine(
                "Difficult method makes testing difficult.");
    }
}

public class MyClassDerived : MyClass
{
    public MyClassDerived() : base()
    {
    }

    public override void DifficultMethod()
    {
        Console.WriteLine("It's testable!");
    }

}

public class ClassToTest
{
    public ClassToTest(MyClass myclass)
    {
        Console.WriteLine("Initializing ClassToTest");
        myclass.DifficultMethod();
    }
}

The ClassToTest has a constructor that takes a MyClass parameter. Unfortunately, MyClass has the DifficultMethod which is a very...well, difficult method. By marking DifficultMethod as virtual we can override it in a derived class which can then be used to pass into the ClassToTest constructor. Handy.
Here's the same idea in Python

class MyClass(object):
    def __init__(self):
        print "MyClass Initialized"
    def difficult_method(self):
        print "this makes things hard"

class MyClassDerivative(MyClass):
    def __init__(self):
        MyClass.__init__(self)
    def difficult_method(self):
        print "MUCH better"

class ClassToTest(object):
    def __init__(self, myClass):
        print "ClassToTest has been initialized"
        myClass.difficult_method()

Using Lambdas to Avoid Gnarly IF Statements

2008-12-16T20:20:00.001-08:00

Do you have a function in your C# code somewhere that takes a parameter - the type doesn't matter - and then executes another function based on that parameter's value?

public void what_to_do(string param)
{
    if (param == "this")
        DoThis();
    else if (param == "that")
        DoThat();
    else
        DoSomethingElse();
}

Whenever I see "else if", my spider-sense starts tingling. Adding another "else if" is just too easy and the result can be cumbersome. I do use "else if" (and "switch" - it's just as vulnerable to this idea), but I don't always like it. The good news is that there is a fix - if you're willing to think "functionally". The bad news is it took me so long to piece it all together.

When did this epiphany come? At the latest MichiPUG meeting, while Mark Ramm was demonstrating a Python library for implementing generics, he displayed an interesting technique for avoiding the Python version of the above code. Here's my rendition of what he showed us:

whatToDoDictionary = {'this':do_this, 'that':do_that}

def what_to_do(param):
    if param in whatToDoDictionary:
        whatToDoDictionary[param]();
    else:
        do_something_else();

The key to this bit of Python is the values in the whatToDoDictionary. Instead of your standard string or integers, the values are functions. When we call the what_to_do method, we look to see if the parameter is present as a key in the dictionary. If it is, we simple execute the value portion of the key/value pair. Voilà! No more "else if"!

Can we do this in C#? Sure. Let's do it using lambdas.

Dictionary<string, Action> dict =

     new Dictionary<string, Action>();

public Lambda()
{
    dict.Add("this", () => do_this());
    dict.Add("that", () => do_that());
}

public void WhatToDo(string param)
{
    if (dict.ContainsKey(param))
        dict[param]();
    else
        do_something_else();
}

The concept is exactly the same. Build a dictionary of lambdas (functions in Python), see if the parameter passed to the WhatToDo(string) is in the dictionary and then execute the lambda if it is.

Note to Self 4 - Python Editors, Debuggers, stdout and stderr

2008-12-13T13:11:00.001-08:00

PyScripter has been IDE of choice for Python, but I've run into some problems with it. I would have continued to use it, but I noticed that the latest anyone did any work on it was October of 2006 - or at least that's what I thought. It turns out that early this year PyScripter is now maintained in Google Code and supports Python 2.6. You can download it here.
I've started using Notepad++ for editing my Python. Though it's only an editor, it uses syntax highlighting, supports macros and allows me to define hot keys to run external programs. This last point makes it possible for me to launch my code in Python with a simple keypress.
Want to know how to use Notepad++ to launch the file you're currently editing in Python? Type "[%Python directory%]\python.exe "$(FULL_CURRENT_PATH)" into the Notepad++ Run dialog and then save it to the keystroke combination of you choice.
If you took my previous bullet point to heart, you'll find that Python launches in a command window and then promptly closes after your tests run losing any output you may have had. The only way that I've been able to solve this is by pointing the stdout and stderr to a file. I would prefer to override the these when I launch Python, but instead I have to put some addtional code into my application. Bummer.
WinPDB is a debugging tool for Python. The only tool I'd used for debugging in the past is the one embedded into PyScripter. When I started using Notepad++ as my editor, I started trying to debug from the command line using the PDB module. No fun. That's when I came across WinPDB. It's worked well for the past few days that I've been using it, but it still isn't as nice as having your debugger integrated into your environment.

Note to Self 3 - CodeMash Screen Casting Options

2008-11-17T10:41:00.001-08:00

I've been learning a lot about Screencasting this week...

I've updated my podcasting links at http://delicious.com/dwhawley/podcasting with a couple of new items.
I reviewed a couple of USB devices to capture video and audio (USB 28000D DVD Maker and the Diamond VC500 One Touch Video Capture Device) and I wasn't happy with any of the reviews. This really seemed to be the simplest avenue but seems to be a dead-end.
The solution I'm searching for assumes that I'll only have laptops available. There are a lot of video capture cards out there for a desktop machine but I'm avoiding this one for now.
I like the VNC option the best, but a major problem with this is that we'll need to rely on speakers installing something on their machine. Jason Follas pointed this out to me and he's right. I think the only way to resolve this is ask the community. If they don't like the solution I'll know to keep looking. Besides, someone might have a really good idea.
Jing is cool. Jay Wren gave me quick demo earlier today and I was impressed with it's simplicity. One of the things that I really like about Jing is it's social spirit. I was looking at Camtasia, another offering by TechSmith , but I think Jing may be a better fit.