Technobabble

Time is an illusion, lunchtime doubly so

Do you have some spare CPU cycles and a little bandwidth on your server?

Then you should consider taking part in the NTP Pool project and become a part of the global effort to provide accurate time for millions of users.

There continues to be a large growth of users of the pool, and the number of servers doesn't seem to grow quite as fast.

It will only take you about 15 minutes to setup. Read more here: How do I join pool.ntp.org?

Oh yeah, and the quote in the headline of course is by Douglas Adams from h2g2.

Hacking Django forms for CSS flexibility

The default output of the Django forms (former newforms) module is not very CSS friendly. With a few simple adjustments, you can make your web designer colleague happy.

This patch will add three classes on the parent HTML element of the rendering of each form field (the tr, li or p tag depending on your rendering mode):

1. The type of the form field. (Examples: CharField, ModelChoiceField)
2. The type of the widget. (Examples: TextInput, SelectInput)
3. Is the form field optional or required: Optional or Required

Now a required DateField will render, using the as_table rendering, as:

<tr class="DateField TextInput Required">
  <th>
    <label for="id_date">Date</label>
  </th>
  <td>
    <input type="text" name="date" id="id_date" />
  </td>
</tr>

Example uses

A couple of example use cases where my patch will help you out:

• Special styling of required fields possible.
• Easier to add a date picker by JavaScript.
• Special styling of checkboxes (styling input elements to width: 100% also affects those).

Download the patch

Patch against forms/forms.py in Django 1.0: Download - View

How to patch your newly downloaded Django-1.0.tar.gz

For those of you not quite familiar with working with patches:

$ wget http://www.hacktheplanet.dk/export/HEAD/misc/forms.py.patch
$ wget http://www.djangoproject.com/download/1.0/tarball/
$ tar xvfz Django-1.0.tar.gz
$ patch -d Django-1.0/django/forms/ < forms.py.patch

Django and mod_wsgi: A perfect match!

mod_wsgi is an Apache module for serving WSGI-based Python web applications from the Apache HTTP server. Django, along with almost every other Python web framework today, comes bundled with a backend for acting like a WSGI application.

A couple of months ago I decided to try it out in spite of mod_python. Discovering and trying out mod_wsgi really suprised me. It can take a massive beating, and outperforms mod_python in every practical aspect.

The setup

You will need a short Python "bootstrap" script to create a WSGI-handler for your Django project. Here is an example (call it wsgi_handler.py and place it in the root directory of your Django project - the one with manage.py and settings.py):

import sys
import os

sys.path.append(os.path.dirname(os.path.abspath(__file__)) + '/..')
os.environ['DJANGO_SETTINGS_MODULE'] = 'projectname.settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

Finally set up your Apache virtualhost to use mod_wsgi:

<VirtualHost *>

  ServerName www.projectname.org
  ServerAlias *projectname.org  

  Alias /admin_media /usr/lib/python2.4/site-packages/django/contrib/admin/media

  <Location /admin_media>
    Order allow,deny
    Allow from all
  </Location>

  Alias /media /home/user/projectname/media

  <Location /media>
    Order allow,deny
    Allow from all
  </Location>

  WSGIScriptAlias / /home/user/projectname/wsgi_handler.py

  WSGIDaemonProcess projectname user=user group=user processes=1 threads=10
  WSGIProcessGroup projectname

</VirtualHost>

In the WSGIDaemonProcess line, you can easily manage the amount of system resources (measured in processes and threads) mod_wsgi should use. In my experience a single process with 10 threads will cover most small and medium loaded websites.

Why?

This is some of the reasons why you should ditch mod_python for mod_wsgi when hosting Django projects:

• Faster

  The load times of the websites now served with mod_wsgi really surprised me. Normally a page would be served within 150-300 ms. This was reduced to load times in the range of 40-80 ms.

  I also discovered that running mod_wsgi in embedded mode (as opposed to daemon mode) was not worth the effort. I didn't really see any difference between load times when using Django.

• Less memory usage

  Everyone hosting more than a couple of Django projects on a single Apache instance knows that Django projects squanders a bit with memory usage, and every single Apache child process will easily end up using 50 MB RAM.

  mod_wsgi dedicates a process (or multiple processes) to a single interpreter for a single Django project, and keeps the memory usage low in the "normal" Apache child processes. On a server with 8 small Django projects, I went from using ~1500 MB RAM on Apache child processes to using 150 MB.

• Secure

  When using mod_python your Python interpreter will be running as the user running the Apache webserver itself (on Debian systems, the user is called www-data). Typically this will allow you to peek around in places where you do not want your users peeking. This is due to the fact that www-data must have read access to every file you use in your application (including settings/configuration/media files).

  mod_wsgi addresses this problem by changing to a user id specified in the configuration file, and run your Python interpreter as another user than www-data, allowing you to lock down every project on your server to seperate user accounts.

These points cover mod_wsgi running in daemon mode.

Conclusion

mod_wsgi rocks!

So if you are thinking about moving your systems to, or just curious about, mod_wsgi, you should really get to it. I, for one, welcome our new mod_wsgi overlords! (sorry)

Futher reading

"How to use django with mod_wsgi" (from the Django wiki)

http://code.djangoproject.com/wiki/django_apache_and_mod_wsgi

"Integration With Django" (from the mod_wsgi wiki):

http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango

About Django and the importance of releases

My favorite Python web framework, Django, has not been updated for a long time. The most current release, 0.96, was released in March 2007. This is a very long time, when you're in the market of web frameworks.

This doesn't seem to bother a lot of people, as the common answer in the django community seems to be just to run the trunk version (development version).

I for one doesn't like that solution. And here are some of the reasons why.

Some of the problems with running a development version

• When a security release is made, I cannot just update, but need to merge the change in, in all of my installations. An update could maybe break my existing code with backward incompatible changes.
• It's easier to tell my co-workers that our projects will run 0.96, and not r6389 for one project and r7291 for another (+ a couple of security patches). That's okay if you are a single-person team working on a single project, but not when you have several people and projects.
• Developers are afraid to commit new things to trunk, because a lot of users will be disappointed when they eagerly update their repositories each morning just to find that backwards compatibility has been broken. A good example of this is ticket 3639:

  This patch will be committed to trunk eventually, don't worry. But I, personally, haven't done it yet because of the massive backwards-incompatibility it introduces, making timing important.

  No-one should ever be afraid to break backwards compatibility in the development version. This will just add more complexity to the job of making new releases by queuing up a lot of uncommitted patches.

Release early, release often

Many great people have preached this paradigm a lot. And for a reason. This is the best way for an open source project to succed. If you don't know what your users want, you will most likely fail.

Getting out releases will foster users to tell you about their experiences and if you are heading in the right direction with development with regards to what the users actually need.

Having more frequent releases would probably also spur more developers to contribute. Developers like active projects and getting their contributions released.

A suggestion

One of my biggest problems with the current (0.96) release is how buggy newforms is. This could easily be solved by making django.newforms (and maybe django.contrib.admin) into a separate project with a separate release schedule.

Then everyone wouldn't have to rewrite a lot of form handling code when the 1.0 release happens, but would now currently be using cleaned_data and ModelForms in spite of their current incarnations.

Conclusion

Despite my points in this post, I still think The Django framework is the best thing that has happened to the craft of building web applications since the invention of HTTP ;-)

Translate strings using Google Translate

Someone dared "me" to write a python interface for Google Translate. Here it is:

"""
translate.py

Translates strings using Google Translate

All input and output is in unicode.
"""

__all__ = ('source_languages', 'target_languages', 'translate')

import sys
import urllib2
import urllib

from BeautifulSoup import BeautifulSoup

opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'translate.py/0.1')]

# lookup supported languages

translate_page = opener.open("http://translate.google.com/translate_t")
translate_soup = BeautifulSoup(translate_page)

source_languages = {}
target_languages = {}

for language in translate_soup("select", id="old_sl")[0].childGenerator():
    if language['value'] != 'auto':
        source_languages[language['value']] = language.string

for language in translate_soup("select", id="old_tl")[0].childGenerator():
    if language['value'] != 'auto':
        target_languages[language['value']] = language.string

def translate(sl, tl, text):

    """ Translates a given text from source language (sl) to
        target language (tl) """

    assert sl in source_languages, "Unknown source language."
    assert tl in target_languages, "Unknown taret language."

    assert type(text) == type(u''), "Expects input to be unicode."

    # Do a POST to google
    
    # I suspect "ie" to be Input Encoding. 
    # I have no idea what "hl" is.

    translated_page = opener.open(
        "http://translate.google.com/translate_t?" + 
        urllib.urlencode({'sl': sl, 'tl': tl}),
        data=urllib.urlencode({'hl': 'en',
                               'ie': 'UTF8',
                               'text': text.encode('utf-8'),
                               'sl': sl, 'tl': tl})
    )
    
    translated_soup = BeautifulSoup(translated_page)

    return translated_soup('div', id='result_box')[0].string

Usage:

>>> import translate
>>> translate.translate('da', 'en', u'Goddag')
u'Good day'

Bash history aggregation

So, I wanted to join the fun:

$ history|awk '{a[$2]++} END{for(i in a){printf "%5d\t%s\n",a[i],i}}'|\
sort -rn|head
   87   python
   56   svn
   42   ssh
   38	cd
   33	rdesktop
   22	touch
   21	mplayer
   20	ls
   16	whois
   15	host

I guess it's obvious that I currently mostly work on python projects hosted on svn ;-)

What does yours look like?

Service announcement: Comments enabled again

Just a quick note to let everyone know that comments has been enabled again.

XML-RPC dispatching through Django test client

Django has a cleverly designed test client that creates a WSGIRequest and routes it through your views without the need of a web server. Furthermore it works on a temporary test database, thus any errors won't have any effect on your live database.

I'm currently working on an application written using Django. This application, amongst other features, has an XML-RPC interface. The tests for this interface was being done by spawning a local web server and carrying out the tests over the wire. However, it would be nice to exploit the Django test framework.

The solution is amazingly simple. I wrote a TestTransport class for xmlrpclib:

import cStringIO

from xmlrpclib import Transport

from django.test.client import Client

class TestTransport(Transport):

    """ Handles connections to XML-RPC server through Django test client."""

    def __init__(self, *args, **kwargs):

        self.client = Client()

    def request(self, host, handler, request_body, verbose=0):

        self.verbose = verbose

        response = self.client.post(handler,
                                    request_body,
                                    content_type="text/xml")

        res = cStringIO.StringIO(response.content)
        res.seek(0)

        return self.parse_response(res)

In my tests where I use the XML-RPC interface I initialize the ServerProxy object with:

server = ServerProxy('http://localhost/xmlrpc/', transport=TestTransport())

That's about it. Now calls trough the server object will be handled by the Django test client.

In my case I had a decorator around the view function handling XML-RPC requests checking the HTTP headers for authentication information and replying with a 401 error. I had to modify this to allow the test client access with no authentication (you may not need this, but I will show it as an example on how to know in your view that this is a test client request). Here is the relevant parts cut out:

def xmlrpc_authenticate(func):
...
    def inner(request, *args, **kwargs):
...
        if request.META.get('SERVER_NAME', '') == 'testserver':
            return func(request, *args, **kwargs)
...
    return inner

Introducing: PyBeat

I recently bought myself a NSLU2 and needed some software to stream music files from it to my laptop. The alternatives used too many resources and wasn't practical on a box with 32 MB ram and 266 MHz CPU.

So I needed something that wouldn't take up as much resources on the server - so I wrote my own server system. The "revolutionary" part of my system is, that the server is kept simple, and all the resource intensive work, such as directory listings and playlist generation, is left to the client.

Read more about PyBeat.

Exposing calendar events using iCalendar in Django

I recently wrote simple abstraction for exposing calendar events in Django as iCalendar feeds. It relies on vobject for managing the formatting of the calendar file, so you will need this if you want to try it out. It is available in Debian in the python-vobject package.

Save this as a file somewhere in your project:

import vobject

from django.http import HttpResponse

EVENT_ITEMS = (
    ('uid', 'uid'),
    ('dtstart', 'start'),
    ('dtend', 'end'),
    ('summary', 'summary'),
    ('location', 'location'),
    ('last_modified', 'last_modified'),
    ('created', 'created'),
)

class ICalendarFeed(object):

    def __call__(self, *args, **kwargs):

        cal = vobject.iCalendar()

        for item in self.items():

            event = cal.add('vevent')

            for vkey, key in EVENT_ITEMS:
                value = getattr(self, 'item_' + key)(item)
                if value:
                    event.add(vkey).value = value

        response = HttpResponse(cal.serialize())
        response['Content-Type'] = 'text/calendar'

        return response

    def items(self):
        return []

    def item_uid(self, item):
        pass

    def item_start(self, item):
        pass

    def item_end(self, item):
        pass

    def item_summary(self, item):
        return str(item)

    def item_location(self, item):
        pass

    def item_last_modified(self, item):
        pass

    def item_created(self, item):
        pass

Now we need to couple this abstraction with a Django queryset. I placed this in a feeds.py in my application:

from yourproject.path.icalendar import ICalendarFeed
from yourproject.yourapp.models import SomeEvent

class SomeEventCalendar(ICalendarFeed):

    def items(self):
        return SomeEvent.objects.all()

    def item_uid(self, item):
        return str(item.id)

    def item_start(self, item):
        return item.start

    def item_end(self, item):
        return item.end

Finally we just need to put it somewhere in the urls.py:

from yourproject.yourapp.feeds import SomeEventCalendar

[...]

    (r'^feeds/icalendar/someevent/$', SomeEventCalendar()),

[...]

That's it.

Implementation notes

• Start and end timestamps can be datetime.datetime objects (django.db.models.DateTimeField) as well as datetime.date objects (django.db.models.DateField). If they are datetime.date objects, the event will be exposed as an all-day event.
• (Obviously) The methods you can override are listed in the ICalendarFeed class.
• If you do not override the item_summary method, the __str__ representation of your model is used.

Managing local settings in Django

Sometimes it is nice to be able to configure specific Django settings for a single host and not get tons of conflicts the next time you do svn up. I personally solve this by exploiting that the Django settings.py is nothing but Python code. At the last line of the file I do a:

from local_settings import *

That is, way I do a relative import from local_settings.py and gets every global symbol mixed into the current namespace, allowing me to overwrite every option. An example could be to configure the global settings.py to use sqlite as a database backend for the project, but in the production environment overwrite the DATABASE_* options in the local_settings.py. This goes as well for caching - not many developers run a PostgreSQL and memcached on their laptop.

To make sure that local_settings.py never is committed to the repo (and maybe compromising database passwords), it is a good idea to add it to the Subversion property svn:ignore:

svn propset svn:ignore local_settings.py /path/to/your/project

Furthermore, I usually put up a local_settings.py.dist with a couple of commented out examples for the developers of what could be done here.

The Python Debugger

It cannot be said enough. The Python Debugger pdb is just a great tool. Say you have a line in a function from which you wish to inspect the environment. Just do a jump into the debugger and analyze:

import pdb; pdb.set_trace()

That's all you need. To get more familiar with the debugger, I suggest visiting the pdb docs at python.org.

Tall screenshots from Firefox

Looking through the Internet the other day for a tool to make "tall" screenshots of my browserwindow, I stumbled upon Pearl Crescent Page Saver:

Pearl Crescent Page Saver is an extension for Mozilla Firefox that lets you capture images of web pages. These images can be saved in PNG format or (with Firefox 2) in JPEG format. The entire page or just the visible portion may be captured. Options let you control whether images are captured at full size (which is the default) or scaled down to a smaller size. Page Saver uses the canvas feature that was introduced in Firefox 1.5.

The basic version is even free, and works perfectly. The screenshot at my WTForm page at djangosnippets was taken using this.

New WTForm release

I have just published a new version of WTForm, my Django newforms addon to allow for grid form layouts (using YUI) and extra classes for more specific and easier CSS styling.

There was a problem when using WTForm with form_for_model or form_for_instance resulting in not getting any fields in the generated form class. This should be fixed now.

The djangosnippets page for WTForm now also includes a link for a screenshot to see an example of a form using grids and some other neat CSS styling by my colleague Oscar.