Firemint.

I've just read a job advertisement for a game programer position at Firemint, the Australian developer of the smash hit iPhone game, Flight Control. They must be making bajillions of dollars, as they seem to have about 14 employees in their office, which is not bad for a new studio with no apparent external funding.

Anyhow, one of the lines on the list of job requirements was:

* Demonstrated understanding of how to translate a game design into code and fill in gaps as necessary

"Fill in gaps as necessary". LOL.

That's probably the #1 requirement for a game developer. Most of the time, there is no design, no spec. If there is a spec, it is probably wrong. I'm not sure if this is a good thing, or a bad thing. The engineer inside me says "Surely this is no good!@#!" but the late night coder in me says "This is reality. Cope with it."

Wing Commander Concept Art

I was digging around my folks shed the other day, and came across some ancient Wing Commander Concept Art which I collected during my mis^H^H^H well spent youth.



This is the Dorkathi. I don't remember this ship, however I guess it is some kind of Kilrathi Transport for... dorks?

BinaryWriter, BinaryReader for Python

I've released some code which I use to read and write binary data in Python, which can then be used with System.Collections.BinaryReader and BinaryWriter in .NET.

I primarily use this for communicating between Unity3D applications (Mono) and Stackless Python servers. It includes the classes as an optional C extension if you feel the need for speed.

The Sky Crawlers

I've just finished watching The Sky Crawlers. This is a brilliant yet depressing film. Don't watch it if you're looking for something cheerful. Do watch if you want to spend a day or two thinking about what it might mean.

I'll probably take a look at the game when it is released, sometime this January.

Perth - GameJam Capital of .au?

I just had a look at what the other GGJ sites in Australia are doing.

Currently, NSW has two locations with two Jammers in total, and Victoria has one location with two Jammers.

Perth has one location with... 14 JAMMERS! Yes, Perth is the GameJam capital of Australia. We have an awesome community of artists and developers. I am certain that we will kick collective ar$e in January!

C# 3.0, properties.

After discovering that my favourite game platform could potentially use C# 3.0, I decided to see what I might be missing out on while I'm stuck with C# 2.0. I found the relevant document on MSDN, and came across this curious snippet. It is an example of "Automatically Implemented Properties".

public Class Point {
   public int X { get; set; }
   public int Y { get; set; }
}

Please enlighten me C# experts, why would I want do this? Surely this is semantically equivalent to:

public Class Point {
   public int X;
   public int Y;
}

It seems all the first snippet does is add extra function call overhead and extra keyboard taps. What am I missing here?

C# 3.0 in Unity3D?

Yes, it seems it is possible.

From http://answers.unity3d.com/:

Download and install the latest version of Mono (2.4.3)

From C:\Program Files\Mono-2.4.2.3\bin, grab these files: mono.exe, mono.dll, libglib-2.0-0.dll, libgmodule-2.0-0.dll

From C:\Program Files\Mono-2.4.2.3\lib\mono\2.0, grab these files: gmcs.exe, gmcs.exe.config, mscorlib.dll

Copy all these files to Unity\Editor\Data\MonoCompiler.framework

I'm guessing that this works because the C# 3.0 compiler still compiles to bytecode which is compatible with the mono runtime used by Unity 2.6.

I might try this out later, and post instructions for OSX.

Animation Resources for Indie Game Devs

Mixamo offer a tool for building and modifying prebuilt animations. The site looks very easy to use. Watching a soldier release his inner ballerina is... quite entertaining.

I'm quite impressed with the results so far, I can see myself using this in future projects.

Symbols in Python

Wikipedia defines a Symbol (as used in Lisp):

A symbol in the programming language Lisp is a primitive data structure that has a name. Symbols can be used as identifiers. Symbols are unique in a namespace (called package in Common Lisp). Symbols can be tested for equality with the function EQ. Lisp programs can generate new symbols at runtime. When Lisp reads data that contains textual represented symbols, existing symbols are referenced. If a symbol is unknown, the Lisp reader creates a new symbol.

I'm not overly familiar with Lisp, however I have done some work with Scheme; and an explicit Symbol type is something that I sometimes miss in Python. Do you ever find yourself defining classes to be used as global constants, so that you can use the identity operator? Perhaps something like this:

class START: pass
class QUIT: pass

if something() is QUIT: exit()

Symbols offer a better alternative to this, so I decided to implement a Symbol type in Python, which offered some of the features of a Lisp Symbol.

import sys 

class Symbols(object):
    def __init__(self, name):
        self.name = name
        self._symbols = {}

    def __repr__(self):
        return "<SYM %s>"%(self.name)

    def __getattr__(self, name):
        s = self._symbols[name] = self.__class__(name)
        self.__dict__[name] = s 
        return s

    def __getitem__(self, name):
        return getattr(self, name)

s = sys.modules["symbols"] = Symbols("Root")

How is this module intended to be used? Try this:

>>> import symbols
>>> symbols.START is symbols.QUIT
False
>>> symbols.START is symbols.START
True
>>> 

The symbols module contains every possible Symbol, and can be used as simple set of constants... but it can also do more than that.

>>> symbols["START"] is symbols.START
True
>>>

Symbols can be accessed by a string name, which lets you use different characters in a Symbol name, and provides a convenient method for creating new Symbols at runtime.

The last requirement is for Symbols to be unique within their own namespace. Our symbols module can support this too.

>>> symbols.Foo.Bar.Foo is symbols.Foo
False
>>> symbols.Foo.Bar.Foo is symbols.Foo.Bar.Foo
True
>>>

This feature allows you to have an nested set of Symbols which retain their identity wherever they're used in your program.

Secret Unity3D feature?

This one flew under the radar... at least for me.

While working on the audio in our game... I noticed that audio sources increased in pitch as I moved toward them... yes, it's the doppler effect in action.

What an awesome engine. :-)

A Geek Lounge?

After chatting briefly with chrism (who is now back in Perth, w00T!) we lamented the lack of a good place for Geeks to meet up in Perth, and hack on game code, art, music whatever. Perhaps something like the Linux Caffe in Toronto.

Who owns a cafe that we can hijack? :-)

Any other ideas?

Go vs Stackless

Andrew Dalke compares Go and Stackless Python and gets some very interesting results.

I'd like to see answers to some of the questions he raises.

Why does Pike emphasize the performance of Go's goroutine creation and channel communication when it seems to be slower than Stackless and definitely is not an order of magnitude faster?

Why indeed? Please answer us Google Engineers!

Update:
Just for fun, I ran the benchmarks myself, and added a benchmark for Fibra, because I wrote it, and I know that it is pretty slow. :-)

Go...

wdgt:tmp simon$ time ./6.out 
100000

real 0m1.147s
user 0m0.646s
sys 0m0.491s

Stackless...

wdgt:tmp simon$ time python sgo.py 
100000

real 0m0.532s
user 0m0.448s
sys 0m0.080s

Fibra...

wdgt:tmp simon$ time python fgo.py 
100000

real 0m4.054s
user 0m3.883s
sys 0m0.153s

This is the Fibra benchmark code. It's almost the same as the Stackless example, except for the ugly "yield" statements.

import fibra
from optparse import OptionParser

parser = OptionParser()
parser.add_option("-n", type="int", dest="num_tasklets", help="how many", default=100000)

def f(left, right):
    x = yield right.pop()
    yield left.push(x+1)

def main():
    options, args = parser.parse_args()
    leftmost = fibra.Tube()
    left, right = None, leftmost
    for i in xrange(options.num_tasklets):
        left, right = right, fibra.Tube()
        schedule.install(f(left, right))
    yield right.push(0)
    x = yield leftmost.pop()
    print x

schedule = fibra.schedule()
schedule.install(main())
schedule.run()

The results are: Stackless fast, Go slow, Fibra... forget it.

Python 3, is it doomed?

Why would I ask that question?

Python 3 has been available for some time now, yet uptake is slow. There aren't a whole lot of packages in the package index shop for Python 3. It seems there just isn't a demand for Python 3 libraries.

Google have released a new language, even though they hired the BDFL, and started the Unladen Swallow project (which coincidentally, does not target Python 3). Why?

Finally, despite my best efforts to move a project across to Python 3 from Python 2.6, I've been sabotaged by undocumented differences in the behaviour of the select module. I've decided to move back to 2.6 for the time being, rather than change code which has worked for a long time. Who knows what else might come out of the woodwork to raise strange exceptions in the middle of the night?

I think Python 3 suffers from attempting to be compatible with Python 2. It should have been a clean break with major repairs and reconstruction, as the original Python 3000 was intended to be. Hang the expense, let's make this language _right_, and Python 2 be damned.

Is it too late to remedy this situation?

How PC games should be played...

Beer + Air Conditioning + High Back Recliner + 42 inches of 1920x1080 LCD.

Awesome.

GameRanger - Perth Game Company

I just found out about (via a random google search) GameRanger, a company based in Perth that builds software which seems similar to GameSpy. It appears that they are willing to work with developers too. Cool.

Unity3D 2.6 - now with builtin profiler.

I am very impressed with the new profiler that comes with Unity3D. It is very simple to use and is so easy to understand, that even I can use it!

In 20 literal minutes, I've been able to optimise my current project from 14 fps to 60 fps. Excellent! Here is a general performance tip I've come to appreciate:

- Don't use dynamic scaling of meshes to produce animation. It triggers a mesh rebuild and sucks your CPU cycles. If you really must change the scale property of a mesh, don't do it every frame!

A Huge GameJam Poster!

I'm upgrading GameJam.org to be more aesthetically pleasing to the eye... and to be more usable too!

While looking at the hundreds of screenshots and avatar images, I had a Great Idea. Make a big mosaic poster!

I've uploaded a HUGE image (58 MB) for anyone interested.

The GameJam.org changes include a reduction of features, to a core set of tools. All the extra fluff is being removed. The login system is also changing to OpenID. More on this soon.

Unity becomes free, Unreal follows lead.

Hot on the heels of the UT announcement of a free version of Unity 3D,
Unreal just released a free set of game tools.

I can't help but think that this is a knee jerk reaction, and really shows that Unreal is feeling very threatened by Unity 3D.

Forward thinking from Radiohead.

Radiohead is Braindead

Let's throttle the internet to stop file sharing. Great Idea. While we're at it, lets stop road accidents by limiting highways to one lane at 20kph. Excellent thinking.

What is right with Unity3D?

I'm pleased to see that my previous rant about problems with Unity3D is now irrelevant. All those problems are fixed. Excellent!

Even better, Unity Indie is now free. The new profiler will also be very useful! Unfortunately its too late to port my current project, so I'll have to wait a bit to try out the new features.

Lost Serial Numbers

I've paid for Homeworld 2 three times.

Each time, I've lost the installation media, or the manual (with the printed serial key).

Wanting to fire it up again, I found the CD, but not the serial key. Bummer. Is the only alternative to search for a crack, or someone else's serial?

I think this is the reason that platforms like Steam or Impulse are so great. You never lose your games, unless the provider goes bust, of course.

West Australian Game Developers

I've started a LinkedIn group for West Australian Game Developers.

If you are a professional, and you want to network, feel free to sign up.

What is wrong with Unity3D?

I've worked with Unity3D for quite a while now. I really like it. It saves so much time, with all the functionality I need pre-baked and ready to use. I also think the component based architecture is an absolute artwork.

So what is wrong with it?

The last year as seen me working with a team of Unity3D developers on a project. We have three developers, and two artists. This is where Unity3D fails. The Asset Server solution which Unity Technologies provide to allow for collaborative projects is really bad. I'm often prompted with change sets from the start of the project, which don't seem to harm anything but are still confusing. The inability to merge Scenes, and the Prefab workaround is simply heinous.

The final straw came today, when I was attempting to merge in a colleagues work before I committed my own. I was receiving a simple runtime error from one of his functions which I could not resolve. In desperation, I clicked reimport all, a time consuming process which reimports all the assets in a project, during which you cannot work on the project.

Problem solved. Arrrgh.

Unity3D is awesome for solo developers, not so great for teams. These kinds of strange issues need to be fixed before a team of any size should seriously consider using Unity3D.

No Power Operator in C#...

There is no power operator in C#.

What the...

This is the reasoning:

It would be possible to add a power operator to the language, but performing this operation is a fairly rare thing to do in most programs, and it doesn't seem justified to add an operator when calling Math.Pow() is simple.

Huh? A "rare thing to do"... as compared to the bit shifting operators for example? If this question is in the FAQ, surely it deserves a place in the language.

Ridiculous. Brain Dead. Stupid. Utterly Stupid.

The Python Ternary Sucks

I have just spent an hour trying to track down a weird bug in some Javascript interpolation code. The offending code looks like this:

var n = i+1?i<length:length;

Ternary expressions almost always have the structure:

condition value_if_true value_if_false

My Python brain got it wrong, as you can see, I used:

value_if_true condition value_if_false

in the Javascript code, which is the Python way of doing things. Eg:

i+1 if i < length else length

Stupid Python. In this case, The language syntax of Python should have followed the conventions used by most of the planet. "Practicality beats Purity" and all that.

Edit:To be clear, my point is that the Python Ternary expression is needlessly different to almost every other language and causes Brain Pollution.

</rant>

What is delicious?

One sheet of Puff Pastry.

A few scoops of Peanut Butter.

A few cubes of Dark Chocolate.

Combine inside a sandwich press.

10 minutes.

Delicious.

Dell Laptop Dead. Sad Face. Happy Face.

I work with OSX, Linux and Windows. Most of my Python development happens on Linux, and I just use Windows to run VM instances of Linux. :-)

Unfortunately, my favorite machine, a Dell XPS M1330 died yesterday due to a common fault with the video chipset, which causes the graphics chip to get really hot and unsolder itself. This manifests itself by causing glitchy screens, then eventually the screen won't work at all, it just fills up with random vertical lines and slowly turns white.

Much of my Python game dev happens on this machines, so I was pretty sad. But... After some research, I discovered that Dell had extended the warranty of my particular model especially to cover the fault! Awesome. After ten minutes on the phone, a Dell technician is coming to my office tomorrow to replace the faulty parts. I'm very happy now. It would be nice if Apple respected their customers in the same way.

SPW does stupid easy GUI screens.

I would like an easy way to write GUI screens for my games... one function to draw the GUI, handle GUI state, and respond to GUI events. No widget graphs, no parent child relationships, no callbacks. It can be done. The code below uses a new GUI module in the SPW library. Many more widgets are required, as at the moment there are only input boxes, buttons, toggles and menu widgets.

import pygame
from spw.gui import GUI, Context

class Window(object):
  def draw_gui(self):
    GUI.canvas.quad((0,0,320,240))
    text = GUI.input("textbox", (70,10,100,30))
    if GUI.toggle("show", (290,10,10,10)):
      if GUI.button("quit", "Quit", (190,30,100,100)):
        print text
        raise SystemExit
      options = "new", "open", "save"
      action = GUI.menu("file", "FILE", (10,10,50,30), options)
      if action is not None: print action

pygame.init()
GUI.init()
context = Context(pygame.display.set_mode((320,240)))
window = Window()
while True:
    e = pygame.event.wait()
    if e.type == pygame.QUIT: break
    with context as canvas:
        GUI.draw(window, canvas, e)

Javascript Game Tools.

V8 Javascript + OpenGL. Awesome.

V8 Javascript + Game Engine. More Awesome.

Simple Pygame Wrapper

I do a lot of prototyping. Simulations, user interfaces, algorithms etc. Often I'll use pygame to visualise the results, as it is a nice simple library.

Sometimes though, it's just a bit too low level. So I created SPW, which add a very light interface over the pygame drawing and blitting functions. This interface provides a canvas which handles scaling and translations, enabling you to easily scroll across a drawing, or zoom in and out. It also looks after dirty rectangle updates. This is what it looks like:

from spw import gui

with gui.Context() as canvas:
    canvas.translate((400,300))
    canvas.circle((0,0),50)
    canvas.translate((0,150))
    canvas.scale(0.5)
    canvas.circle((0,0), 50)

The context manager tracks the dirty rectangles, and does a screen.update when it exits. I've put it on google code, and it can also be installed using easy_install.

Terrain building is fun...

This is done in Unity3D. I've found that having a tablet (instead of a mouse) is essential for this sort of thing. It's much better for your wrists too.

Star Hammer Preview

Fellow Perth Game Dev Paul Turbett has released a trailer video featuring an early build of Operation Star Hammer.

Who is up for some PyWeek?

Pyweek is coming up. Who is up for some furious game / python coding action?

Here is a bunch of games I've written for past a few previous PyWeek events. For the last few events, I've been unable to finish due to RL commitments. I'm thinking about trying again...

Anyone feel like teaming up?

Stupid advice on Batteries

I'm tried of reading stupid, ignorant advice about rechargeable batteries. Eg:

...as a rule of thumb for ANY rechargeable device you should always charge it fully and drain it down until it dies at least once a month. this will cycle the battery and give you longer lasting batteries.

Don't follow this advice, it is wrong wrong wrong. In this age of ultra-portable, battery powered devices, you owe it to yourself to get educated, so don't take my word for it, and especially don't listen to advice like the above. Do some research yourself and get the facts. Having said that, I'll state the facts below, because I think most people won't bother to dig further! :-)

If you want to look after your battery powered device, first you need to find out what sort of batteries it uses. Most modern electronic devices use Lithium-ion batteries, so I'm going to talk about them. Note, this advice applies only to Lithium-ion batteries. These are the sorts of batteries you will find in your cell phone, laptop, iPod or Tesla Roadster.

1. Your battery has a fixed shelf life.

This has a few implications. Firstly, nothing you can do will extend the battery beyond it's shelf life. It will die, even if it is never used. This leads me on to the second point... don't buy a spare battery. When you need it, it will probably be dead, beyond resurrection. Generally, a well-cared-for battery will last five years, losing about 20% capacity per year.

2. Your battery will continue to discharge slowly, even if it is turned off.

Why? It's related to the chemical reactions happening inside the battery. However, the most likely reason is that there is special circuit in your device which monitors the battery voltage. Why do you need such a circuit? Read on.

3. Your battery will be damaged if it is discharged below a certain threshold, or overcharged.

For this reason, most devices have special circuits which measure the voltage of the battery, and turn the device off if the voltage drops too low. As already discussed, if you completely discharge the battery, it will continue to discharge even when it is turned off. This means the battery may in fact discharge to a dangerously low level, where it will become irreparably damaged. It will drastically reduce the life of your battery. Fortunately, the same circuit protects against overcharging, so you will probably never have to worry about that.

The moral of the story. Don't deliberately try to completely discharge a battery. Don't let your device sit around with a flat battery for days on end. Don't buy a spare until you need it.

Python + Wiimote + 30 tonnes of Steel

We're almost done with our Giant Robot Project. Realising we had only a few hours before the robots would be deconstructed and sent away, we decided... that we must add Wiimote control to our system! Yes, 15 tonne steel arms, giant claws and 200 bar of hydraulic pressure... all controlled with a flick of the wrist.

With loads of help from my partner in crime, Dan Adams and the Python cwiid module we were up and running in about 2 hours. We used tilt left and right to move the slew, and lean back and forward to control the main boom and the two smaller jibs.

Huge thanks to Transmin for letting us use their robots! I'm willing to bet that these robots are the largest Wiimote controlled devices on the planet.

Anyone care to dispute? :-)

Holy Cow, Galcon runs in Flash!

Phil Hassey, noted Python/Pygame enthusiast has ported his Award Winning Galcon game to Flash! Awesome!

I'd recommend limiting yourself to 1 match per day, else you'll be joining me at Galcon Anonymous meetings.

Stackless vs GIL: It's a draw, cont.

I decided to double check my assumptions about Stackless and the GIL on a multicore machine, using the same example provided by David Beazley in his slides, which highlights the GIL contention issue. It's probably a better example than my previous test.

import stackless
import time
import threading

def count(n): 
    while n > 0:  
        n -= 1 

T = time.clock()
count(100000000) 
count(100000000) 
print time.clock() - T

>>> 17.37

A normal sequential execution takes about 17 seconds.

T = time.clock()
t1 = threading.Thread(target=count,args=(100000000,)) 
t1.start() 
t2 = threading.Thread(target=count,args=(100000000,)) 
t2.start() 
t1.join(); t2.join()
print time.clock() - T

>>> 53.22

This confirms David's experiment. It takes much longer due to threads fighting over the GIL.

T = time.clock()
stackless.tasklet(count)(100000000)
stackless.tasklet(count)(100000000)
while stackless.getruncount() > 1:
    task = stackless.run(100)
    if task:
        task.insert()
print time.clock() - T

>>> 25.34

Twice as fast as the threaded solution, and roughly 50% slower than than the sequential solution. Cool.

We can get much better results by increasing the granularity of the scheduler. If we change the tick count to 1000, the Stackless solution takes 17.71 seconds, and the threaded solution takes 22.25 seconds. This is interesting behavior, which I can't yet explain.

Stackless vs GIL: Benchmarks.

Following on from my last post, I decided I should check my assertion that "Stackless will outperform CPython, even with CPU bound tasks." I'm not saying the GIL is bad, I'm just pointing out how the same behavior can be achieved with Stackless. If these tasks called some random C function which blocked, but still yielded the GIL, it is likely that CPython would come out on top.

import time
from threading import Thread
import stackless

def factorial(c):
    T = 1
    for i in xrange(1, c):
        T *= i
    return T

#Benchmark stackless using 500 tasklets    
T = time.clock()
for i in xrange(500):
    stackless.tasklet(factorial)(1024)
while stackless.getruncount() > 1:
    task = stackless.run(100)
    if task:
        task.insert()
print time.clock() - T

#Benchmark OS threads using 500 threads
T = time.clock()
threads = []
for i in xrange(500):
    thread = Thread(target=factorial, args=(1024,))
    thread.start()
    threads.append(thread)
for thread in threads: thread.join()
print time.clock() - T

>>> 0.5
>>> 0.77

On my dual core machine, Stackless performs around 30%-40% faster than regular threads. This is usually not a suprise, we all know that IO bound threads always come with a penalty in CPython. However, these Stackless tasklets are being pre-empted in the same way that the GIL works. This is something my Fibra framework and other similar frameworks which are based on generators, can never achieve.

Very Interesting!

Update: Fixed factorial funcion and timings.

Stackless vs GIL: It's a draw.

After thinking some more about David Beazley's research into the GIL, I realized the gravity of his comment that the GIL provides "Basically a kind of "cooperative" multitasking". I've spent 10 minutes knocking up some code in Stackless that demonstrates something similar to what David describes. I doubt many people knew Stackless has this capability.

import stackless

#A contrived CPU bound task
def factorial(c):
    T = 1 
    for i in xrange(1, c):
        T *= i
    return T

#create two tasklets
stackless.tasklet(factorial)(512)
stackless.tasklet(factorial)(1024)

#used to track of how many task switches happen
switches = {}

#while there is more than the main task running...
while stackless.getruncount() > 1:
    #run the schedule for 100 ticks
    task = stackless.run(100)
    #if we have a pre-empted task
    if task:
        #increment it's switch counter
        C = switches.setdefault(task, 0)
        switches[task] += 1
        #insert it at the end of the schedule
        task.insert()

print switches.values()

>>> [13, 26]

What is happening here?

Firstly, notice the factorial function. It is a contrived example of a CPU bound task that does not yield to the Stackless scheduler (by calling stackless.schedule) anywhere in it's body.

Next, two tasklets are installed into the schedule. One computes the factorial of 512, the other computes the factorial of 1024. If we call stackless.run() at this point, both tasks would run to completion, one after the other. This is because the factorial function never yields control back to the Stackless scheduler.

However, in this example, when we call stackless.run, we pass it an argument of 100. This asks Stackless to run each tasklet for a maximum of 100 'ticks'. If the tasklet exceeds this, it is interrupted and returned to main tasklet. This forces our factorial tasklet to yield control back to our main loop, at which point we can decide what to do with it. In this case, the best thing to do is insert at the back of the run queue to give other tasklets a chance to execute.

In the example above, the tasklets were forced to switch 13 and 26 times respectively. These few lines of code show how Stackless can almost achieve the same kind of threading that CPython provides, without using OS threads. I think that is pretty neat.

Why do I say "almost"? The Stackless solution does not provide everything the GIL does. For example, if a tasklet makes some C library call which blocks for a long time, all tasklets will block and wait, whereas in CPython other threads will get some CPU time. Still, the task switching in Stackless is much faster than OS thread switches, so if you're using mostly Python code, Stackless will outperform CPython, even with CPU bound tasks.

CPython Threading is Fundamentally Broken.

Most Python coders know that the Global Interpreter Lock (GIL) in CPython effectively serializes execution of threaded programs, even on systems with multiple CPU cores.

Despite this, Python threads are still useful in some situations, especially when working with blocking IO. Or so we thought.

The GIL problem is much, much worse than I expected.

David Beazley has published some slides which summarize his research into the GIL, which demonstrate how the implementation is fundamentally broken. The slides are clear and easy to understand and it is definitely worth taking the time to read and digest.

To summarize, the implementation of the GIL causes threads (on multicore machines) to compete with each other as they attempt to acquire the GIL. The scary part is, as you add more cores and more threads, the problem will grow exponentially, causing a massive degradation in performance when compared to serial execution. This isn't the only issue. Read the slides if you ever intend to use threads in CPython.

Is Python's select.poll unreliable?

I've been building a DHT using Stackless Python, and a nonblocking IO layer which provides IO event notification using epoll, poll, or select, depending on which module is available on the operating system.

I'm developing on OSX, so my module cannot use epoll, and downgrades to the regular select.poll object. This is the first time I've used a select.poll object instead of select.

I'm coming up against some nasty problems. Every now and then, select.poll does weird things. Eg, returning a file no which I have not registered, which might be 1 or some random number. Sometimes it even returns negative numbers. Recently, in some circumstances it does not return a write event, when clearly it should be. And of course, these bugs are intermittent and hard to reproduce.

So... I switched the IO layer to use select.select instead of select.poll. Voila, everything works perfectly.

Is select.poll known to be buggy? It's hard to find lots of example python code which uses it, so I wonder if it is very well tested across a range of platforms.

Working with Subversive Games.

I'm now working with Subversive Games, helping establish the Perth Studio, and at the same time delivering an awesome project to a large client! Subversive Games helped sponsor the Global Game Jam here in Perth.

We've started a blog, so we can talk about some of our game tech, and perhaps reveal some of our Python and Unity 3D tips and tricks. I'm currently preparing a demo to show how we've created hundreds of kilometers of train tracks across a vast landscape.

Stay tuned!

A Room Heater, in Python.

It is about 8 degrees C this morning. So cold, especially when last week we had high twenties.

To help solve the problem, a friend suggested I write a heater application to help warm me up. Not one to refuse a challenge, and always eager to show that Python can solve all your problems, I came up with a Room Heater, in Python, in 15 lines of code.

import time
import sys

try:
    D = int(sys.argv[1])
except IndexError:
    D = 10
except ValueError:
    print "Usage: %s MINUTES"%sys.argv[0]
    raise SystemExit

print "Starting Heater..."
start = time.time()
while (time.time() - start) < (D*60): pass  
print "Heater is stopping."

It works very well, especially if you are using a laptop, on your lap of course! It takes about one minute or so for your CPU to get to hot enough to be felt, and if you are using a Mac Book Pro, like me, you'll really feel the heat!

Cool.

Spinning globe, in Javascript.

http://www.willmcgugan.com/js-globe/

The Game Incubator

I'm assessing the feasibility of starting a 'Game Incubator' in Perth, Western Australia. The idea primarily being to service independent developers and artists, and assist with startups. It will be a place where people can rent a desk for a month or two, network with other professionals, and share the camaraderie of game development.

If you're local to Perth, and interested, please take the survey.

Cobra vs Python

I've been investigating different .net languages for use with Mono. I've spent some time with Boo, and have decided I don't like it. It feels immature due to poor syntax error reporting, and silently changes the scope of variables when there happens to be an inadvertent mix of tabs and spaces in your source code. Yuck.

Then I came across Cobra. The language comparison with Python stung a little, I guess because it had some valid points.

Cobra vs Python

There is definitely a lot of flame bait in there, but the point about being "self hosted" resonates with me.

It is late...

... And I am still playing Guitar Hero.

The Saga of Duke Nukem is over...

I've just read that 3D Realms is shutting down. 3D Realms have been working on Duke Nukem Forever for over 12 years.

Now that the story is over, I want to see a postmortem and timeline. What really went wrong with Duke Nukem Forever?

iPhone Developer Survey

Adam from T=Machine is looking for statistics on iPhone developers.

I'm interested too, so lets all take the survey!

GameJam coming up...

The next GameJam is scheduled to start Friday May 1.

The guys at OneTwenty are running a mixer event so Jammers can meet, greet and make plans. Please RSVP if you can make it!

GHF

So, I've discovered a new medical condition, which causes numbness and tingling in your fingers.

I've called it GHF...

...

...

...Guitar Hero Fingers.

Heh.

PyCap - Another Python Game Framework

I've been spending some time with a client, helping maintain a game written in PyCap. PyCap is The PopCap Game Framework + Python hooks.

It is quite fun to work with. The API is simple and stays out of the way, and it's fast! It also takes care of building the binary distributable, which has some appeal for game development in Python. There is also builds for Linux and Mac. It looks neat, but I imagine it could be hard to debug.

1:41 AM

Still coding. A cooperative scheduler. Argh. So much more that I should be doing...

Well, I think Fibra is 50% faster now. I think. I can't remember the numbers I was benchmarking against... :-(

I've re-instated the network stuff too. Much better API now.

I really should be fixing the gamejam site... or something.

<plonk>

Zzzzzz...

yield from - redundant syntax?

Greg Ewing is putting together a proposal to adding a "yield from" keyword to Python.

The Spam Server code he provides as a supporting example can already be achieved using the Fibra scheduler, without the new syntax. Notice that Fibra allows exceptions to propagate from an inner task to the outer task.

Fibra also allows an inner task to return values to its parent task, by yielding a "Return(value)" instance. This is what the line_recv task uses to continually yield lines from a socket to the parent task, while maintaining its own internal state.

Perhaps the "yield from" syntax is redundant. Fibra doesn't need it, and Fibra fulfilles the main use-case behind the PEP.

The code below is a direct translation of the Spam Server example, using Fibra.

import socket
import fibra
import fibra.net

port = 4200

def handler(transport):
    try:
        while True:
            try:
                line = yield transport.recv_line()
                n = parse_request(line)
                yield transport.send_line("100 SPAM FOLLOWS")
                for i in xrange(n):
                    yield transport.send_line("spam glorious spam")
            except BadRequest:
                yield transport.send_line("400 WE ONLY SERVE SPAM")
    except socket.error:
        pass 
    
class BadRequest(Exception):
    pass
  
def parse_request(line):
    tokens = line.split()
    if len(tokens) != 2 or tokens[0] != "SPAM":
      raise BadRequest
    try:
      n = int(tokens[1])
    except ValueError:
      raise BadRequest
    if n < 1:
      raise BadRequest
    return n

schedule = fibra.schedule()
schedule.install(fibra.net.listen(('localhost', port), handler))
schedule.run()

Update: modified handler to use a new transport class.

An AppStore Journey

Jack Nutting, of Scribattle fame, has documented his journey on the Apple Appstore. Make sure you check out the slides. At one point, it seems that Scribattle netted ~$300 per day.

I seem to gather from the slides, that Jack now believes that advertising supported, free games are the way to go if you're looking for a greater success in the dollar department. Interesting...

Travel back in time to ID Software, 1993...

John Romero just posted some a film of ID Software's office as it was in 1993.

Wow, look at all that old tech.

Something Rotten at Apple...?

Are you an iPhone developer? Do you want to be an iPhone developer?

Here is some bad news. The Apple App Store refund policy could cost you more than you think.

If someone buys your app, and decides within 90 days that they want a refund... Apple will refund the full amount to the user, and ask you, the developer to pay that full amount back to Apple, including the commission Apple took from the original purchase.

On the surface, this looks rather evil. However, it could help stem the tide of trash apps which are flooding the App store.

Ladies and Gentlemen, prep your engines.

The next GameJam is starting on May 1st, 2009.

Registrations will open soon. Site upgrades are imminent.

I need a Scuzzie.

Scuzzies sound incredibly useful.

I can't find anyone that sells them though. :-(

Windows has found a problem...

I found this to be rather amusing.

Unity now runs on Windows.

Unity 2.5 has been released, and it runs on Windows. w00T!

Mimicking Exceptions in Fibra

When a fibra schedule is running, a task can yield a sub task and pause until the sub task completes and returns a value to the parent task.

This works well, however, when an exception occurs in the sub task, the parent task does not receive the exception. This is a real pain, and breaks my normal thinking about exception handling. When using normal function calls, the exception should always bubble up the stack until it is handled. Fibra should provide the same behavior for sub tasks.

Well, now it does. Snippet below.

import fibra

def sub_task(x):
    if x < 5:  
        yield fibra.Return(x**x)
    else:
        raise ValueError("x must be < 5")

def main_task():
    # launch a sub task
    # wait for it to finish and collect the result.
    x = yield sub_task(4)
    print x
    # the sub task will raise an exception here...
    # yet the exception will bubble up to the parent task, 
    # just like a regular function call.
    try:
        x = yield sub_task(5)
    except ValueError:
        print "Oops, an exception occured."

schedule = fibra.schedule()
schedule.debug = True
schedule.install(main_task())
schedule.run()

>>> 256
>>> Oops, an exception occured.

These features will arrive with Fibra 0.0.11, which is coming Real Soon Now.

Scalability of Stackless, Fibra and Kamaelia

After my last post, I decided to benchmark the scaling properties of Stackless, Kamaelia, Fibra using the same hackysack algorithm.

Left axis is milliseconds.
Bottom axis is number of tasks * 100.
Green line is Kamaelia.
Blue line is Fibra.
Red Line is Stackless.

---

These are the results, using Python 2.6.1 to run Fibra and Kamaelia, and Stackless 2.6.1 to run the Stackless test:

---

These are the results when using Stackless 2.6.1 to run all the tests:

---

It's quite interesting to see that Fibra copes with 600000 tasks better than 500000 tasks in both sets of results. Strange.

Benchmarking Stackless, Kamaelia and Fibra

In the cooperative threading world, as far as Python goes, there are a few choices. Stackless is the most well known, I guess. Since reading this post, I've been itching to see how Fibra compares to Kamaelia and Stackless. Now that I've implemented 'channels' or as I like to call them, 'tubes' in Fibra, I can implement the Stackless hackysack benchmark, and finally get a direct comparison of task switching speed between Fibra, Kamaelia and Stackless.

Fibra results:

python timeit.py -n 3 -s "import hackysack" "hackysack.runit(10000,1000, dbg=0)" 
3 loops, best of 3: 227 msec per loop

Kamaelia Results: Using this code.

python timeit.py -n 3 -s "import hackysack" "hackysack.runit(10000,1000)
3 loops, best of 3: 2.41 sec per loop

Stackless results: Using this code.

python timeit.py -n 3 -s "import hackysack" "hackysack.runit(10000,1000, dbg=0)" 
3 loops, best of 3: 31.5 msec per loop

These benchmarks show that Stackless is 7x faster than Fibra, and Fibra is 10x faster than Kamaelia.

This is the code I used to benchmark Fibra.

import fibra
import random
import sys

scheduler = fibra.schedule()

class hackysacker:
    counter = 0
    def __init__(self,name,circle):
        self.name = name
        self.circle = circle
        circle.append(self)
        self.messageQueue = fibra.Tube()
        scheduler.install(self.messageLoop())

    def incrementCounter(self):
        hackysacker.counter += 1
        if hackysacker.counter >= turns:
            while self.circle:
                hs = self.circle.pop()
                if hs is not self:
                    return hs.messageQueue.push('exit')
            sys.exit()

    def messageLoop(self):
        while 1:
            message = yield self.messageQueue.pop()
            if message == "exit":
                debugPrint("%s is going home" % self.name)
                return
            debugPrint("%s got hackeysack from %s" % (self.name, message.name))
            kickTo = self.circle[random.randint(0,len(self.circle)-1)]
            debugPrint("%s kicking hackeysack to %s" % (self.name, kickTo.name))
            yield self.incrementCounter()
            yield kickTo.messageQueue.push(self)

def debugPrint(x):
    if debug:
        print x

debug=1
hackysackers=5
turns = 5

def runit(hs=10000,ts=1000,dbg=1):
    global hackysackers,turns,debug
    hackysackers = hs
    turns = ts
    debug = dbg

    hackysacker.counter= 0
    circle = []
    one = hackysacker('1',circle)

    for i in range(hackysackers):
        hackysacker(`i`,circle)

    def main():
        yield one.messageQueue.push(one)

    scheduler.install(main())
    scheduler.run()

if __name__ == "__main__":
    runit()

Fibra grows Tubes.

Tubes are like pipes, but... different. Anyhow, Fibra 0.0.9 includes some new Tube classes for communicating amongst tasks.

import fibra

def a(tube):
    print 'Pushing "a" into tube.'
    tube.push('a')
    yield  None

def b(tube):
    while True:
        x = yield tube.pop()
        print 'Received "%s" from tube.'%x

t = fibra.Tube()
schedule = fibra.schedule()
schedule.install(a(t))
schedule.install(b(t))
schedule.run()

How to Increase App Store Sales

Are you releasing an iPhone Game?

Jack show's us why you need to release a free demo version as well.

JanKenPon RockPaperScissors

This game is so cool. It's web based, and multiplayer. Awesome.

It made me laugh for a least 10 minutes, as I played along with office colleagues.

Beast!

Phil Hassey is a machine. He has released another iPhone game, called Beast. I love the retro-glowy line art style.

Thanks Phil, you've inspired me to work harder on my own game!

Scribattle Released

My friend Jack Nutting has released his first iPhone game.

I have to say, this is the first iPhone game I've not regretted paying money for. Most of the games in the Top 25 list don't hold my attention for very long at all.

If you have an iPhone, you can get it over here.

GGJ Perth Team Video

Boing Boing are putting together a video compiled from GGJ footage from around the globe. You can see a hello from some of our Perth team in the preview.

Global Game Jam - Final Games

Here are some screenshots and links to the final games produced at the Global Game Jam in Perth.

Little Big Forts

Under One Roof

Zeppelin Escape

Little Shop of Farters

Brown Cloud

GGJ is almost over...

Global Game Jam 09 is almost over, and here in Perth, many games are nearing completion.


This is a dev screenshot from the game "Under One Roof". It's written in Python and uses Pygame. Jack Casey and Brad Power did the programming, Simon Boxer was the artist. I'll post download links when they are finished.

Global Game Jam, ready to launch!

After much prep-work, the Perth Global Game Jam event is ready to launch!

We've got 25 registrations, plenty of sponsors, lots of loot, free broadband, an excellent venue and even Free Pizza and Soda! Woohoo!

Lot of participants are very keen to try out the Unity 2.5 beta (which now runs on windows), however I'm also hoping for a good turnout of faithful Pygame/Pyglet/Python coders, who will show these 'Unity' shmumps up. :-) There is a good mix of artists, modellers and coders turning up. Personally, I doubt I'll have time for much of anything, however I do hope I get get a few tunes together for teams that require music.

Giant, Python Powered Robots.

These are the robots I've been working on for the last 12 months. They each weigh about 11 tonnes and have a 17 meter reach.

The control system is written in Python, with small sections of C which run in hard-real-time to guarantee safety. The robots work cooperatively, semi-autonomously, with drive-by-wire style assistance when under manual control.

Update: added a close-up of the business end. This claw weighs just over 1 tonne, and gets hurled around at up to 3.5 meters per second.

New, New Gig.

Because I am clearly not busy enough...

I've started working with Subversive Games on a top secret serious game.

We're using my favorite technologies, including Python. I'll post more when I can.

New Gig

I've scored a new gig, teaching game dev with Python! It's all happening at a local tech college. I'm going to resurrect one of my 2D frameworks and see how the students cope with that. I'll update it all to Python 2.6 and integrate it with Pyglet, then probably put it up somewhere. Another Game Framework FTW! :-)

Global Game Jam in Perth

The preparations for the Global Game Jam in Perth are going smoothly.

We've almost nailed down a venue, and we now have a website for the local Perth GGJ. There are now 15 registrations, which makes for 4-5 teams at the event, which is quite encouraging. There is also building interest from various sponsors, which is great as I'll hopefully be able to supply some decent loot to participants.

There is quite a few Python coders coming along to the event, so I expect to see some neat games that actually get finished within the two days! :-)

The Embedded GUI, continued...

After 3 days of mixing it up with Pyglet, I gave up. I had resolved to do things the pygletty way, and I really wanted to use pyglet's shiny batched graphics, but in the end it defeated me. There was simply too many hoops to jump through to draw lines, and draw text and then detect picks with the mouse... etc. If I had more time, this would be the superior solution, but I simply didn't have time to invest in new development.

The next thing to try was wxPython. I went further with wx, but when I realised how little control I had over the look (or even just sizing) of buttons, I became a little discouraged. Then, when wx started hiding exceptions from me with what seemed to be a sys.exit(-1), I gave up. I doubt I'll be going back to wx, its just not... fun.

Getting somewhat desperate, I contemplated putting a BaseHTTPServer in my embedded code, and building the interface using HTML and jQuery. Half a day later, I realized this was a very bad idea. Without a nice framework like Pylons, it would be a nightmare.

Defeated and deflated, with 80% of my time budget spent... I was at a loss. So, I turned back to the old and faithful pygame. It is so much simpler to understand bytes, surfaces, sprites. And hey, If I want to draw text on a sprite, I can. Oh and it has some neat vector drawing operations. A few hours later, I had a custom GUI system, working just the way I wanted. Forms, Buttons, Text Boxes, Labels and Toggle buttons.

I also seem to prefer he way pygame deals with HID events, as compared to pyglet. Pygame gets out of the way and lets me handle things the way I like. :-)

I might actually get this done. Only 8 hrs left....