Ubisoft coming to Perth?

I went to the pulse expo this evening, where one of the speakers mentioned that Ubisoft are in town (Perth, Western Australia), looking to set up a studio.

Cool.

A chat server using fibra.

The following code is a very simple chat server implemented using cooperative threads in the fibra 0.01 framework. Explanation follows below.

import fibra
import fibra.plugins.network as network
import fibra.plugins.tasks as tasks


class Chatter(object):
    def __init__(self, address):
        self.address = address
        self.members = {}

    def listener(self):
        while True:
            conn = (yield network.listen(self.address))
            yield tasks.spawn(self.login(conn))

    def login(self, conn):
        handle = None
        while handle is None:
            yield network.send(conn, 'What is your handle?')
            handle = (yield network.receive(conn))
            if handle in self.members:
                yield network.send(conn, 'That handle is already taken.')
                handle = None
        self.members[handle] = conn
        yield tasks.spawn(self.chat(conn, handle))

    def lost_conn(self, conn, handle):
        if handle in self.members:
            self.members.pop(handle)
        yield self.broadcast('%s has left the chat.' % handle)

    def broadcast(self, text):
        for member, conn in self.members.items():
            try:
                yield network.send(conn, text)
            except network.NetworkError:
                yield tasks.on_finish(self.lost_conn(conn, member))

    def closed_socket(self, conn, handle):
        yield network.on_lost_connection(conn)
        yield self.lost_conn(conn, handle)

    def chat(self, conn, handle):
        yield tasks.on_finish(self.lost_conn(conn, handle))
        yield tasks.spawn(self.closed_socket(conn, handle))
        yield self.broadcast('%s has joined the chat.' % handle)
        data = ''
        while True:
            try:
                data = yield network.receive(conn)
            except network.NetworkError:
                break
            if data == '/quit': break
            yield self.broadcast('%s says: %s' % (handle, data))
        yield network.close(conn)

if __name__ == "__main__":
    chatter = Chatter(('localhost',1980))
    s = fibra.Schedule()
    s.register_plugin(network.NetworkPlugin())
    s.install(chatter.listener())
    while s.tick(): pass

The Chatter class has 6 methods, which are all python generators, which in the context of fibra, I call tasklets.

At the bottom of the code, a fibra Schedule is created, and the NetworkPlugin is registered. The NetworkPlugin allows tasklets to yield certain values which perform network related operations. The main tasklet, chatter.listener, is installed into the scheduler, then the schedule is continually ticked in a while loop. The while loop will finish when there are no more tasklets to run.

So, what does the listener method do? It creates a tasklet, and yields the network.listen object, which will return a new connection when someone connects to the address passed into the network.listen call. This is a non-blocking operation. When the connection object is returned, the listener method spawns a new tasklet (self.login) with the connection, then goes back to listening for another new connection. The self.login tasklet will continue to run concurrently while the listen method is waiting.

The login method sends a prompt to the new connection, asking for a handle to identify the user. If the handle has not already been used, it spawns a chat tasklet and then exits.

The first line of the chat tasklet schedules another tasklet (self.lost_conn) which will be run when the chat tasklet finishes. The second line spawns a tasklet (self.closed_socket) which waits for the socket to close unexpectedly. The chat tasklet then broadcasts a message to any users who are already logged in. It then loops, sending chat messages to all users as they are received. Finally, if the tasklet receives a '/quit' line, it breaks out of the loop and closes the socket and finishes. At this point, the self.lost_conn task is awakened and runs.

If you want to test this code yourself, start the server, and telnet to localhost 1980.

Why do sockets die?

I'm testing a network lib, which uses select for polling socket.

I'm running a stress test, which connects 100 sockets to a server socket (all in the same process), then echoes data back and forth as quickly as possible. If a socket dies, it gets removed. Each loop iteration I print time passed, and the number of active sockets left.

As it runs, I watch the number of sockets slowly decline, until I have a set of 15 sockets left, which seem to keep running happily. Why do the other 85 sockets die? They either raise ECONNRESET, EPIPE or ETIMEDOUT. I imagined sockets connected via localhost would be quite reliable...

Update: The same test between two different machines does _not_ show this same problem. So what's up with localhost?

Looking for Web Designer.

I'm looking for an XHTML/CSS expert. Must also have excellent graphics/design skills. Must have a portfolio I can view online, and be comfortable learning new technologies.

Send me an email (simonwittber at gmail dot com) if you are interested in full-time work, and can work in Perth, Western Australia or are willing to relocate.

On the way to work this morning...

TCP Networked Tasklets in Python

Fibra 3 introduces some networking features.

Generator based tasklets can now communicate with each other over TCP! The networking plugin uses Twisted to do its magic. What other sorts of plugins might be useful? I'm running out of ideas now. :-)

This is the code for a simple server which echoes everything it receives, and starts the conversation with a 'hi.' message:

import fibra
import fibra.plugins.sleep
import fibra.plugins.tasks as tasks
import fibra.plugins.network as network


def listener():
    conn = (yield network.ListenForNewConnection(1980))
    conn.send_string('hi.')
    yield echo(conn)
    yield watch(conn)

def watch(connection):
    yield network.WaitForLostConnection(connection)
    print connection, ' has been lost'

def echo(connection):
    while True:
        data = (yield network.WaitForData(connection))
        connection.send_string(data)

s = fibra.Schedule()
s.register_plugin(tasks.TaskPlugin())
s.register_plugin(network.NetworkPlugin())

s.install(listener())

while s.tick(): pass

This is the code for a simple client which echoes everything it receives:

import fibra
import fibra.plugins.sleep
import fibra.plugins.tasks as tasks
import fibra.plugins.network as network


def watch(connection):
    yield network.WaitForLostConnection(connection)
    print connection, ' has been lost'

def echo(connection):
    while True:
        data = (yield network.WaitForData(connection))
        connection.send_string(data)

def connector():
    conn = (yield network.ConnectToHost(('localhost',1980)))
    yield echo(conn)
    yield watch(conn)

s = fibra.Schedule()
s.register_plugin(tasks.TaskPlugin())
s.register_plugin(network.NetworkPlugin())

s.install(connector())

while s.tick(): pass

New Code Repositories and Docs.

I'm now using bzr instead of svn. I'm pushing my repositories to:

http://exactlysimilar.org/bzr/

I'm also auto publishing documentation to:

http://exactlysimilar.org/docs/

Cooperative + Preemptive Concurrency

I've just uploaded Fibra 2 to the cheeseshop. Fibra 2 includes the promised non-blocking plugin, which allows a generator based task to momentarily run in a seperate thread.

import fibra
import fibra.plugins.nonblock
import fibra.plugins.sleep
import time


def stuff():
    yield fibra.plugins.nonblock.WillBlock()
    print 'I am running inside a thread.'
    time.sleep(2)
    print 'I am still running inside a thread.'
    time.sleep(1)
    print 'I am exiting the  thread, going back into cooperative mode.'
    yield None
    for i in xrange(3):
        print 'I am running cooperatively too.'
        yield 1

def other_stuff():
    for i in xrange(5):
        print 'I am running cooperatively.'
        yield 1


s = fibra.Schedule()
s.register_plugin(fibra.plugins.sleep.SleepPlugin())
s.register_plugin(fibra.plugins.nonblock.NonBlockPlugin())
s.install(stuff())
s.install(other_stuff())
while s.tick(): pass

Bitten by Configuration Management

I've learnt a lesson re. configuration management.

When setting up new projects, esp. projects built on frameworks like TurboGears, you should keep your eggs handy, for all possible platforms.

an "easy_install TurboGears==1.0.1" in September will not necessarily download the same code in October. In particular, it seems RuleDispatch has changed, which exposes a new bug in my application... grrr.

Cool Unix Tools

Today I was faced with dumping a postgres database, compressing it, downloading it, uncompressing it and restoring it on my local dev machine. It's a painfully slow process, which I've always had to supervise.

Surely, there must be a better way to do this...

Some small research time later I discovered netcat, or nc for short. nc lets you create a pipe over a network. As its man page states, it's a TCP/IP swiss army knife.

This is how I ended up automating most of the process.

On my local machine:

nc -l -p 1979 | bunzip2 | psql database_name

On the remote machine:

pg_dump database_name | bzip2 | nc my_local_ip 1979

Voila! The database dumped its data, piped it through bzip2 then over the network, to my waiting nc process which received the data, then piped it through bunzip2 and then into psql.

Cool. Another demonstration showing that Unix is such a great environment.

Pyglet amazes me again.

Pyglet 1.0 beta2 was released to the world a few days ago. I svn updated my local copy of the repository and noticed a new soundspace folder in the examples directory.

Curious, I tried to run soundspace.py. Hmm seems I require an extra 'abvin' library... I try the example again...


Wow. Cool. Hang on... I can drag these widgets around, change their direction... and the audio changes too!

This is 3D positional audio! Pyglet rocks! This is a seriously cool, feature-full library. The more I look into Pyglet, the more I am pleasantly surprised. I need to find a project where I need to use these kinds of cool features!

Well done Alex.

Spatial Hashing

Often, when building a game, you need to test if objects are colliding. The objects could be spaceships, rocks, mouse pointers, laser beams... whatever. The simple approach is to iterate over all your objects, and test if they collide with a specific point.

If you do this using a linear algorithm, you'll quickly find that as you get more and more objects, your collision detection code will slow down at the same rate.

To get around this, you can test against a smaller set of objects, by using a spatial index. A spatial index (in this example, a Spatial Hash / Hash Map) stores all object positions, and can quickly tell you what objects _might_ be colliding in a certain area. You can then iterate through this smaller list, testing for exact collisions if needed. This is called a broad phase collision detection strategy.

from math import floor

class HashMap(object):
    """
    Hashmap is a a spatial index which can be used for a broad-phase
    collision detection strategy.
    """
    def __init__(self, cell_size):
        self.cell_size = cell_size
        self.grid = {}

    @classmethod
    def from_points(cls, cell_size, points):
        """
        Build a HashMap from a list of points.
        """
        hashmap = cls(cell_size)
        setdefault = hashmap.grid.setdefault
        key = hashmap.key
        for point in points:
            setdefault(key(point),[]).append(point)
        return hashmap

    def key(self, point):
        cell_size = self.cell_size
        return (
            int((floor(point[0]/cell_size))*cell_size),
            int((floor(point[1]/cell_size))*cell_size),
            int((floor(point[2]/cell_size))*cell_size)
        )

    def insert(self, point):
        """
        Insert point into the hashmap.
        """
        self.grid.setdefault(self.key(point), []).append(point)
   
    def query(self, point):
        """
        Return all objects in the cell specified by point.
        """
        return self.grid.setdefault(self.key(point), [])

The above class implements a spatial hash. A simple way of putting it is: "we store these points in a grid, and you can retrieve an entire grid cell with its points."

if __name__ == '__main__':

    from random import uniform
    from time import time

    NUM_POINTS = 100000
    new_point = lambda: (
        uniform(-100,100),uniform(-100,100),uniform(-100,100)
    )

    points = [new_point() for i in xrange(NUM_POINTS)]
    T = time()
    hashmap = HashMap.from_points(10, points)
    print 1.0 / (time() - T), '%d point builds per second.' % NUM_POINTS

    T = time()
    hashmap.query((0,0,0))
    print 1.0 / (time() - T), '%d point queries per second.' % NUM_POINTS

This example inserts 10000 points into the hashmap, using a cell size of 10. This means, when we query point (0,0,0), we retrieve all points in the cube defined by (0,0,0),(10,10,10).

On my machine, I can build a 10000 point hashmap 2.7 times per second, and query it 70000 times per second. This makes it great for colliding static points, but not so great for colliding moving points. I imagine the from_points method could be improved somewhat. Any suggestions?

XPS M1330 Review

The Dell XPS M1330 arrived today. I'll be using this machine in my new office in the Big Blue Room. :)

The screen is backlit by WLED's. This means it's a lot brighter than most Laptop LCD's, but it is still not as bright as my iMac LCD.

Ubuntu Gutsy installed perfectly (had to install using safe mode video, after that, the nvidia drivers worked fine). On the first boot, the battery monitor happily reported 6.5 hours of battery life left. Excellent! Everything works, including the multimedia card reader and the sound card.

The build quality feels rather sturdy, even though the machine is very light. The keyboard looks and feels cheap. The battery isn't a perfect fit, and has a very slight wobble.

Pystone reports 68000 pystones, glxgears reports 4800 fps. Not very accurate benchmarks, but at least you get an idea of what to expect.

Overall, I'm happy with it. The build quality is better than the usual Dell standard, and the performance is great, considering the price!

Ubuntu is killing your laptop.

Bronwen sent me a very interesting link the other day. I'm glad she did, because it turns out that Ubuntu has been killing my hard disk!

How? When your ubuntu laptop is running on battery, the disk heads are parked as part of the power saving strategy. When the disk needs to be accessed, the heads are unparked. Apparently this can only happen about 600000 times before a disk becomes likely to fail. This is all well and good, however the Feisty and Gusty releases of Ubuntu do this up to 4 times per minute, which is _bad_, considering that kind of frequency gives your hard disk a life expectancy of 104 days!

This problem only occurs when your laptop is running on battery. The way to solve it is (replace sda with your hard disk device):

sudo hdparm -B 255 /dev/sda

which turns off the aggressive power management features of your hard drive.

If you want to check how much life there is left in your hard disk:

sudo smartctl -d ata -a /dev/sda | grep Load_Cycle_Count

My laptop has clocked up 300000 cycles... which is amazing (in a bad way!), considering it is rarely unplugged.

Update: I've just discovered this is old news. I don't read slashdot anymore... :-) Shame there hasn't been a fix yet.

Update: According to an Ubuntu Dev, Ubuntu does not alter hard disk settings. So, it would appear that aggressive power management is not the problem. The problem is something is writing to the disk too frequently, which will unpark the disk heads. This is still an Ubuntu issue IMO.

Concurrency is Fun!

I find myself spending too much time building cool features for my scheduler. It's a real time and brain sink. :-)

I've just uploaded Fibra 1, which provides a plugin which lets tasklets spawn other tasklets, wait for other tasklets to complete, and spawn tasklets when the current tasklet terminates. This is very neat for building sequences of actions, and is much more natural than the way I used to do it.

I'd like to build functionality so that a tasklet can watch what another tasklet is producing, watch if it raises an exception etc. I've also got to replicate the non blocking yield magic I put into nanothreads.

New 'Office' for the Summer

This Summer, I'll be working from a few new locations, around and about the city.



This particular spot looks like a good candidate! Working outdoors will require a few changes. I'll need to swap my power hungry laptop for something more portable and long lived, and I'll need to sort out a mobile broadband solution. On top of that, I'll need to carefully plan what I would like to achieve each day, so that I don't get too distracted... and achieve nothing. :)

Building Games in Small Pieces - The Scheduler

I've just uploaded fibra to the cheeseshop. This is another small piece of code which I find very useful in developing simulations and games.

Fibra is the scheduler I used in ICCARUS to simulate concurrency. It uses Python generators as 'tasklets' which are iterated cooperatively. It does a job which is very similar to another library I've written, but is different in that it is very light weight, and uses a plugin system to provide extra functionality, such as sleeping, deferring execution, spawning into a real thread etc. To achieve this, it uses new Python 2.5 generator methods, so I decided to split it out of the older nanothreads module and create a new package.

Usually, I have a global scheduler available in the game, so any part of the code can defer a function call, or install a new tasklet. I usually iterate the scheduler just after I handle GUI events.

It's nothing new, its been done before, but I imagine with the right plugins, it could achieve much of what people want when they talk about concurrent Python.

This is a simple example:

import fibra
import fibra.plugins.sleep

def sleeper(x):
    while True:
        print x
        yield x

def normal():
    while True: yield None


s = fibra.Schedule()
#tell the schedule that Sleep, float and int objects should be
#handled by the SleepPlugin
s.register_plugin(fibra.plugins.sleep.SleepPlugin(), (fibra.plugins.sleep.Sleep, float, int))
#the SleepPlugin provides a new method which lets us defer
#as tasklets start for X seconds.
s.defer(4, sleeper(0.5))
#install a sleep tasks that will only iterate once ever 2 seconds
s.install(sleeper(2))
#install a normal task that will iterate on every tick
s.install(normal())
#iterate the scheduler
while True:
    s.tick()

REST in Pylons needs work.

RESTful controllers, in Pylons 0.9.6.1, are second class citizens.

The issue is, Routes does not support nested dispatching to a depth > 1. This means if you want to build a RESTful API which looks like this:

/users/{user_name}/collections/{collection_name}/items/{item_name}

you are bang out of luck. No dice. Can't do it. You are restricted to:

/users/{user_name}/collections/{collection_name}

which is an artificial restriction, which gets in the way. Blocks the whole road, in fact.

pylonshq.com states that, if you don't like the behavior of routes, you can 'plug in your favorite' dispatcher. This is much harder than it sounds, as the WSGIController classes you've probably already written likely depend on information coming from the routes layer. IMO this makes routes (in the context of Pylons) part of the application. It is not middleware.

It would be great if someone could please tell me I'm wrong, and show me the way!

Monetizing a Web Service?

I felt like doing something new last night, so I built a web service using the Pylons 0.9.6 and SQLAlchemy 0.4 libraries. It was really just an excuse to catch up on the latest releases of the libraries...

...so now I've got a cool little web service which provides online storage (like Amazon's S3) with a sequence/list style API. It has an XMLRPC and an RESTful API. It's useful and well-featured enough that I am going to try and monetize the service. I'm thinking of having free accounts with a fixed monthly quota. If a user pays a subscription, the quota increases. Pretty simple, and self sustaining. I hope! :)

The target audience is other developers who want to provide a synchronized database to their distributed applications. Think TODO lists, High score lists for games or even twitter-like chat applications.

I still need to design a human interface, write some docs etc. I'll need some beta testers too. Anyone interested?

BTW: SQLAlchemy 0.4 has added some very cool features. It is still the best ORM for Python. Everyone else is playing catch up.