Why use Twisted?

I've suspected that Unix sockets sometimes work differently to Win32 sockets. After considering updating my FibraNet networking code, I came across a document, which explains the differences between the two platforms.

The document provides a good argument for tackling the Twisted learning curve, as Twisted takes care of the listed incompatibilities and provides consistent behaviour for the programmer.

Game Programming Fun

Over the last few years, I've had a lot of fun building simple games with some rather clever and talented friends. I've done few things which can rival the feeling of achievement I received from building these tiny little pieces of entertainment.

This video shows the highlights of some of our efforts.

I haven't had much time for personal projects lately, but I figured starting something, and working away at it slowly is better than not doing anything at all. I might even make the effort a bit more public, a but more open, so others can pitch in if they get interested.

The first thing I'm going to decide... is whether to base my networking code on Twisted, or risk using my own home grown FibraNet. :-)

A memcached helper class.

This little class wraps the memcache functionality:

• It allows objects to expire after a default or custom age.
• It allows lazy evaluation of default values (think dict.setdefault, with real lazy behavior) by hiding the default value behind a lambda.

import memcache
import time


class Cache(object):
    def __init__(self, cache_addresses, default_timeout=30*60): 
        self.default_timeout = default_timeout
        self.cache = memcache.Client(cache_addresses)

    def get(self, key, default=lambda:None, timeout=None):
        obj_timestamp = self.cache.get(key)
        if obj_timestamp is None:
            obj = default()
            self.set(key, obj)
        else:
            obj,timestamp = obj_timestamp
            if timeout is None: timeout = self.default_timeout
            if time.time() - timestamp > timeout:
                self.cache.delete(key)
                obj = default()
                self.set(key, obj)
        return obj

    def set(self, key, obj):
        self.cache.set(key, (obj, time.time()))
         
>>> tc = Cache(['127.0.0.1:1979'])
>>> results = tc.get('some_big_list', lambda:build_some_big_list())

An advantage of using this class, is that if the memcached process dies, your process will continue to work as normal, as long as a default is provided to the .get method call.

An Aha! Moment

Something clicked in my brain this morning...

lambda is a really neat way to implement lazy evaluation.

cache = {}
def get_from_cache(key, lazy_else=lambda:None):
    if key not in cache:
       cache[key] = lazy_else()
    return cache[key]

>>> obj = get_from_cache('big_list', lambda:function_to_build_big_list())

This is an obviously contrived example (which is slightly broken), but I'm using something similar in a web app, to fetch results from costly SQL queries from memcached.

It simple, elegant, and a good argument for lambda's simple syntax.

Don't kill your HTTP cache-ability!

In the 7 years I've been actively involved in web development, I have never seen any of my peers bother implementing proper controls to allow web proxies, and browser caches to correctly cache dynamic content.

I took the time to do this for a recent project, and I partly credit the HTTP caching code for allowing the site to survive huge traffic surges driven by TechCrunch and BoingBoing articles. I believe correctly implemented HTTP caching for a dynamic site is one of the smartest things a developer can do to mitigate the effect of these surges, and make best use of the CPU cycles and bandwidth of a web server.

For a long time though, I struggled with strange caching bugs and I ended up having to turn off the caching mechanims for authenticated users. Not an optimal solution... then last week I came across a comment from mnot.net which spelt out my error.

Changing the content based on IP address or cookies really damages cacheability and the idea that a GET is idempotent, as I understand it.

This is exactly what I had been doing... dynamic customisation of page content based whether a user is authenticated... or not. ARgh!

Unfortunatly, most of the Python web frameworks I've worked with encourage and even demonstrate this technique in documentation and examples.

The solution to this problem is shown in the same comment.

Rather than separating a user identifier...

Cookie: userid={USER_ID}
/content/foo/

Why not try something like this for personalization...

/content/{USER_ID}/foo/

Thanks for the tip l.m.orchard!

Brain Impedance and ZODB

Over the weekend, I've done some more programming in a Pylons application with ZODB.

I've discovered that my brain is finding it hard to let go of all the relational constraints and paradigms which I have been working with for the last 11 years. I keep imagining that I need to create an index for this or that, so that I can look it up real-quick-like... But then I realize a sequential scan isn't going to be that costly... so I should just write the simplest-thing-that-works... and it does just work!

ZODB really makes prototyping a web app simple, and fast. I'm glad I took the time to learn how it works, its already becoming a valuable tool.

As far as Pylons goes... I'm liking it. When compared to TurboGears, it feels much more composed rather than integrated. This might be a good, or a bad thing, depending on your point of view.

My current challenge is getting the methods on a RESTResource to return data encoded in XML, HTML or JSON, depending on values in the HTTP Accept header.

I am no longer a Twit.

Though some may still argue otherwise.


The 'Delete My Account' account button is an amazing innovation.

Form Authentication and REST

I started playing with AuthKit inside a Pylons App last week. AuthKit works nicely, just like TurboGears identity management, but they both share a common problem when working with RESTful controllers.

When a 401 error is raised, the framework takes over and redirects to a login form. The login form then checks the validation, and redirects back to the original page, in effect converting a GET request into a POST request.

In your Pylons app, this could have the effect of calling your create method in your controller, rather than the index method.

I think the lesson is: only raise 401 on methods which are called by a POST request, or use standard HTTP Authentication systems.

Which is the lesser evil?