The model.py file for chrss, was starting to get a bit big, so it seemed like a good time to do this.

First I moved model.py into model/__init__.py. Then I moved all of the model code itself into separate files (three as it happens) and imported them into model/__init__.py as indicated in the Turbogears docs:

from chrss.model.cms import *
from chrss.model.chess import *
from chrss.model.base import *

However that wasn’t enough, as the __connection__ module level variable for SQLObject wasn’t set and Turbogears couldn’t connect to the DB. So I added this to model/__init__.py (before the other imports):

from turbogears.database import PackageHub

hub = PackageHub("chrss")

and then in each file containing models added the following:

from chrss.model import hub
__connection__ = hub

The main trick was to get the import order correct. model/__init__.py must declare the hub variable, before importing the other files, so that they can access it when they are imported. It’s a bit of a cyclical dependency, which is maybe not ideal, but it’s only used in a limited way.

UPDATE. It turns out that you also need to update the sqlobject.txt file in the .egg-info directory of your project. Otherwise the various tg-admin sql * commands don’t work (as it can’t find the SQLObject classes). Basically you have to list every sub-package of the newly split model package. e.g. change:

db_module=chrss.model

to:

db_module=chrss.model,chrss.model.base,chrss.model.chess,chrss.model.cms

So to start off with here’s the scaled down scan I used for the clock face and hands:

I cut out the face and hands and put them into separate files (see .tar.gz below) for using in the Actionscript. All images were 24bit PNGs with variable alpha transparency, so that the edges look nice and smooth on the Chumby screen (which is relatively low resolution).

The Actionscript (ChumbyClock.as) is very simple and looks like this:

class ChumbyClock extends MovieClip {
	var hourHand:MovieClip;
	var minHand:MovieClip;
	var secHand:MovieClip;

	function onLoad() {
		this.hourHand = loadHand('hour_hand');
		this.minHand  = loadHand('minute_hand');
		this.secHand  = loadHand('second_hand');
	}

	function loadHand(hand_id:String) {
		var clip = this.createEmptyMovieClip("clip", this.getNextHighestDepth());
		clip._x = 160;
		clip._y = 120;

		// load image an shift it to be in 12 o'clock position
		// with 0,0 as middle of bottom of image
		var hand = clip.attachMovie(hand_id, 'hand', clip.getNextHighestDepth());
		hand._x = -hand._width/2;
		hand._y = -hand._height + (hand._width/2);
		return clip;
	}

	function toDegrees(val:Number, range:Number) {
		return (360*(val % range))/range;
	}

	function onEnterFrame() {
		var now:Date = new Date();

		var sec:Number = now.getSeconds();
		secHand._rotation = toDegrees(sec, 60);

		var min:Number = now.getMinutes();
		minHand._rotation = toDegrees(min, 60);

		// let hour hand move a little bit with min hand
		var hourWithMin:Number = now.getHours() + min/60.0;
		hourHand._rotation = toDegrees(hourWithMin, 12);
	}

}

All this does is create three movies which contain the hand images, placed so they are in the 12 O’Clock position. At each timestep (onEnterFrame) the current time is queried and the hand rotations are updated. The second and minute hands are just rotated in proportion with the relevant values. The hour hand is also slightly rotated with the current minute. As we get nearer the end of the hour, the hour hand moves nearer to the next hour. Much as you’d expected for a real clock. That’s it. I did play around with having smoother animation for the hands, but it looked a bit eery.

To tie the Actionscript and images together there is an xml file (app.xml) used by swfmill:

<?xml version="1.0" encoding="utf-8" ?>
<movie width="320" height="240" framerate="12">
  <background color="#ffffff"/>
  <clip import="build/classes.swf" />

  <frame>
    <library>
      <clip id="face"        import="images/hand_drawn/face.png" />
      <clip id="hour_hand"   import="images/hand_drawn/hour_hand.png"  />
      <clip id="minute_hand" import="images/hand_drawn/minute_hand.png"/>
      <clip id="second_hand" import="images/hand_drawn/second_hand.png"/>
      <clip id="app" class="ChumbyClock" />
    </library>

    <place id="face" name="clockFace" depth="0" />
    <place id="app" name="myApp" depth="1" />
  </frame>
</movie>

This will allow us to embed the PNGs in the generated SWF file and reference them from the Actionscript. In addition the clock face image is placed statically behind the main movie of the hands.

These files are then run through a build script to generate the SWF:

~/bin/mtasc-1.12-osx/mtasc -swf build/classes.swf -header 320:240:12 ChumbyClock.as
~/bin/swfmill-0.2.12-macosx/swfmill simple app.xml chumbyclock.swf

Here’s the generated SWF:

To test this on the Chumby I put the SWF on a memory stick along with this debugchumby script:

#!/bin/sh
chumbyflashplayer.x -i /mnt/usb/chumbyclock.swf

Then when the Chumby is rebooted with the memory stick in the clock displays and all is good.

So there you go that’s a simple clock widget for the Chumby - using open source tools.

Download Widget Source Code and Images.

Seeing as Wargames turned 25 years old this week, it seemed like a good time to do a drawing of David Lightman (Mr Broderick) just as he logs into WOPR for the first time. It’s a classic moment in the film and shows a slightly more realistic view on hacking/cracking. None of this “re-routing the encryption codes” nonsense. Good old-fashioned sleuthing and intuition instead.

After letting it sit for a while Ian and Kyran also started using it on ShowMeDo. That was a couple of months ago. So now that I can be fairly certain it works it seemed like time to share it with the world.

So first off here’s a few features/comments:

• It’s shamelessly based on code from Django (the caching backends at least)
• It features an “anti-dogpiling” mechanism to try and make sure only one thread triggers a cache refresh
• Multiple backends supported:
• dummy - does no caching (for testing/development use)
• simple - just uses a dictionary (for testing/development use)
• localmem - thread-safe cache for use in process
• file - uses file-system to cache data (this is what’s used with chrss and ShowMeDo)
• memcache - uses memcached for caching (it should work, but not massively tested at the moment)

Now for some example usage:

from turbogears import expose, controllers
from cache import cache_result

class MyController(controllers.Controller):

    @expose(content_type="text/plain")
    @cache_result()
    def cache_some_text(self):
        ''' no template so it's pretty straightforward - expose just has to come first '''
        return 'this will be cached'

    @expose(template="my_template)
    @cache_result()
    def cache_data_only(self):
        ''' with a template we can just cache the data and not rendered html '''
        return dict(value='this dictionary will be cached')

    @expose()
    @cache_result()
    @expose(template="my_template)
    def cache_html(self):
        '''
        or we can cache the rendered html, but we have to use an outer expose()
        to make the method public
        '''
        return dict(value='will be cached with the html')

To see how you can use the @cache_result() you are probably best looking
at the source (there’s a fairly detailed comment explaining it). The following parameters can
be passed in:

• key_fn - function used to derive a key to store the data in (default uses current url and user identity)
• version_fn - can be used to control how a cached value expires (defaults to a function that returns the same value everytime)
• timeout_seconds - how many seconds until the value start to expire

The default key function can be controlled via the config parameter cache.decorator.anon_only. If set to True (the default) it will only look in the cache for data when users are not logged in. Otherwise when users are logged in it will use a key just for them. The default is handy if you just want to avoid problems with a flood of anonymous users (e.g. from Slashdot/Digg etc).

The version function can be used to force expire cached values. The value of the version function is compared to the value stored in the cache and if different this triggers a cache refresh. For example if the version function was based on the number of comments in a blog post, then whenever a comment was added to the blog post the cache would get refreshed. This avoids having to wait for the cache to expire.

timeout_seconds specifies how many seconds before a value expires. It defaults to the value set in cache.decorator.timeout_seconds in your config file (or 1800 seconds if not set there).

anti-dogpiling

So first I’ll explain what I mean by “dogpiling” with respect to cache expiry.

The standard way to use a cache is to do something like:

value = cache.get('key', None)
if value is None:
    value = recompute_cached_value()
    cache['key'] = value
return value

Now this is fine normally. When the cached value expires the next request will simply call recompute_cached_value() and the cache will be updated for future requests.

The trouble arises when recompute_cached_value() takes a long time to run and you have have a lot of other requests running at the same time. If a request is still recalculating the value and another request comes along, then that will also attempt to recalculate the value. This will in turn probably slow down the calculation going on, making it more likely that the next request to arrive will also trigger a recalculation and so on. Very quickly you can end up with tens/hundreds/thousands of request all attempting to recalculate the cached value and you have lost most of the advantage of caching in the first place.

So to handle this situation more gracefully this caching decorator employs a two stage expiry.

First there is a hard cut off expiry that works like normal. This is set to occur later than the other expiry time and is the value that would be fed to memcache or equivalent.

The second expiry time set is the one normally used. Basically when we store/retrieve the cached data we also have access to this expiry time (and the version). If we see that we need to recalculate the value (due to the expiry time being in the past or the version being different), then we attempt to grab a lock to recalculate the value. If we don’t grab the lock, we assume another thread is doing the recalculation and rather than wait around we simply serve up the old (stale) data. This should mean that one thread (potentially per-process) will end up doing the recalculation rather than several.

This also means that we don’t have to remove a value from the cache to force a refresh (which might cause dogpiling). Instead we can update whatever value we use in our version function, to trigger a graceful refresh.

conclusion

So that’s a basic intro to this caching decorator. It’s quite a handy quick way of adding some caching to your turbogears app. You’ll need to see how well it works for you. I’m providing it “as is” and making no claims about anything. Feel free to incorporate it into your code and modify as you see fit. Just let me know if you have any issues or feedback.

bonus decorator

The cache code also includes a simple decorator to control the Expires header sent out with a response:

def expires(seconds=0):
    '''set expire headers for client-size caching'''
    def decorator(fn):
        def decorated(*arg,**kw):
            cherrypy.response.headers['Expires']=formatdate(_current_time()+seconds)
            return fn(*arg,**kw)
        return decorated
    return decorator

It’s handy for getting the client to cache some data for us too. I use it on some of the PIL generated images served up via my app.

source code

Download turbogears caching decorator

The source for the decorator(s) includes a simple test suite (to be run using nose).

This is my first foray into flash development, so for the moment I’m just cribbing from other sources. I’m hoping that as Actionscript is a cousin of Javascript it won’t be too tricky to pick up. Though of course much of the learning will be related to the various APIs.

To get started I ran over to Open Source Flash and read the article on setting up a flash project without using the Flash IDE (seeing as I don’t want to buy Flash - for now at least). I then followed through from there to the MTASC site and looked at the example tutorial and realised that may be all I needed to get a simple “Hello World” widget working.

So here’s the Actionscript I used (saved in Tuto.as):

class Tuto {

    static var app : Tuto;

    function Tuto() {
        _root.createTextField("tf",0,0,0,320,240);
        // write some text into it
        _root.tf.text = "Hello\nChumby!";

        // make the text appear bigger on the screen
        var format = new TextFormat();
        format.size = 64;
        _root.tf.setTextFormat(format);
    }

    // entry point
    static function main(mc) {
        app = new Tuto();
    }
}

Which is based on the code in the MTASC tutorial.

Then with MTASC downloaded and installed in ~/bin/mtasc-1.12-osx/ on my Powerbook, I compiled the code using:

~/bin/mtasc-1.12-osx/mtasc -swf tuto.swf -main -header 320:240:12 Tuto.as

Which creates the SWF file tuto.swf.

Note the -header 320:240:12 option. This is to set the size (320×240) and framerate (12fps) of the resulting SWF. These are the settings as required for the Chumby.

With that done I then tried out tuto.swf in my web-browser to make sure it worked. Next I uploaded the widget via chumby.com (which meant I had to create an 80×60 pixel icon too). I made sure that the widget was set to private when uploading (seeing as only I wanted to see the widget) and then added it to my “development” channel.

Then I selected the “development” channel on my chumby to see the results:

I did run into a little trouble getting the widget to refresh after uploading. It seemed to work sometimes, but in the end I started sshd on the chumby so I could stop and start the control panel (after connecting via ssh):

chumby:~# stop_control_panel
chumby:~# start_control_panel

Which seemed to solve the trick. I’ll have to investigate other means of deploying to the chumby when I’m properly developing, but overall it wasn’t too much effort going via the chumby site.

It’s got that geek gadget factor working for it very well. Lots of options to configure, but at the same time it’s pretty easy to use. Lots more exploring to do, but even just the fact that it’ll play Radio Paradise out of the box is enough for me…

See more chumby photos

Anyway, I thought I’d revisit the brush pen last night.  So I did a few sketches with it:

The sketches include the mentats, Piter de Vries and Thufir Hawat, as well as a certain girl sleeping.  Very pleasant working with the brush-pen, as it’s nice and quick and the results look quite striking.  Think I’ll be doing a few more of these.

Just a reminder that MASS is on for another two weekends (17th and 24th of May). The preview video above should give you an idea of what to expect and even briefly features my piece in situ.

A little background

PsiWin was the bit of software that Psion provided to communicate/sync/convert files between a regular PC and their (now defunct) PDAs - such as the 5mx, Revo and netBook.  You connected your Psion to your PC with a serial connection (RS232) and could then treat the Psion like a disk on your PC.  You could navigate the Psion’s file system just like any disk and transfer files.  In addition you could use PsiWin to open and convert Psion’s native/custom files in normal Windows applications.

I myself owned a netBook and really enjoyed having it. It was a really small laptop that I could carry around with me easily. It even came travelling with me and had enough power to let me noodle around writing compilers and all sorts on the road. In retrospect it was a bit pricey, but it was still a wonderful machine. I’ve been eyeing up the EEE PC in the hope that it’ll make a good replacement for my netBook. Though I’m probably going to wait for the bigger screened version to come out, as then it’ll at least have the same screen size. Shame there won’t be a touch screen on it…

There is still quite a thriving Psion community out there - well over five years since the last Psion rolled off of the production line.  People loved their Psions and rightly so.  These were fantastic machines, that unlike other “PDAs” were fully functioning computers and weren’t totally reliant on having a PC to act as a mothership.  In fact saying “people loved their Psions” is probably the wrong tense - people still love their Psions and continue to use them.   Even now I’ll still have the occasional mail about using JPL and looking at the sourceforge site shows people are still downloading it.

Version 1.0

So the one main problem I had was that I had an iMac, running MacOS 9 and PsiWin would only work on a PC.  This was around 2000.  There was a Mac version called MacConnect, but it only handled file transfer and came on a 3.5″ disk.  My iMac didn’t have a 3.5″ disk drive and didn’t feel like having to buy MacConnect (when PsiWin was free with the Psion) _and_ have to buy a USB disk drive.  So for quite a while I persevered with using the Comms app on the Psion to transfer files directly over a USB serial port adapter, as well as transferring files via compact flash (CF) disks.

Then I stumbled on some reversed engineered documentation for the “Psion Link Protocol” (PLP).  Without that documentation I would never have started writing JPL.  Even with this documentation it was quite a lot of hard work getting everything going.  I did finally get my head around hexadecimal notation.  Not bad for someone who’d done AI and no regular CS at uni.

Version 1.0 of JPL more or less worked, but was a bit flaky.  It’d crash and bring down the VM and sometimes even the mac itself!  It also suffered from being a bit of a hack job - I wrote enough to get it working, but didn’t really go back and rewrite things and refactor.

Still I put it online and had a few people downloading it and then started getting mail about it.  There’s a lot to be said for having people actually using your software - even if it is a bit flaky and needs some work!

However at about that point I took a year to go travelling in Australia, so development on JPL ground to a halt.  I did have the netBook in tow though and had got a java compiler working on it.  So with that and a gsm modem vaguely kept my eye on things.

Version 2.0

I started coding version 2 of JPL in a bar in Ko Pha Ngan on the netBook itself.  I couldn’t run the code of course, but I was able to get a lot of the structure down and compile the code.  In fact the code I wrote then still lived on, though greatly modified, into the version of JPL 2 I finally released.

I created myself an API for the Psion Link Protocol to code the GUI application against.  I was able to create input and output streams to write to individual files on the psion, so I could treat it like it’s own file system.  It took quite a bit of effort to make that happen, but it greatly simplified writing the overall application. I was effectively writing my own library for accessing the psion and also writing an application to use that library.

Of course I never quite achieved the level of separation I’d have liked.  In retrospect I wish I’d actually treated JPL as two projects.  I could have then released the library for other developers to use and then coded the GUI like I was a user of the library.  Hopefully this would have meant the core library would be quite small and focussed.

Another thing I wish I had done was to have actually made use of unit-testing in JPL.  At the time I had no real experience of unit-testing, but since then have come to realise how useful automated tests can be.  It would of required a fair bit of thought in some places though.  Programming in Java would have required me to carefully consider how to mock-up a serial port and so on - no monkey patching to get you out of jail in Java!

At the time though Java was a pretty good choice for me.  The language is receiving quite a lot of flak nowadays, but I’d spent two years previously chained to a C++ compiler.  Java by comparison was markedly more productive.  Plus it meant that I could get JPL running on the Mac, Windows and Linux.  The much vaunted “write once, run anywhere” paid off in my case.  Which is not to say I could just ignore platform differences, but it meant that the app was essentially functional on the three major platforms with little extra work.

As my main development platform was a mac I spent extra time exploiting some of the extra features of the mac’s java runtime (MRJ).  This meant that JPL was able to better act like a regular mac application.  By using java’s reflection api I only had to create one version of the code.  JPL would detect whether certain classes were available and if so expose extra features to the user.

The real-world

JPL was for the most part a hobby project.  I did get donated a Revo and a 5mx for testing and a little bit of funding at one point.  This helped get at least one more release of JPL out, but in the end the real-world got in the way.  In particular I started my MSc.  Though I was able to spend some time working on JPL during this period I couldn’t really spend too long on it.  Taking out a career development loan to fund your further education does kind of sort out your priorities like that!

I also found that I was using my netBook less and less, so JPL itself was not as useful to me.  So like many open source projects, once I lost interest the incentive disappeared and work ground to a halt.  Which is not to say anything would have been different if I’d been running JPL as a commercial project.  I’m not sure there would have been enough users to have supported the cost of extra development.

Still it was an interesting project to have worked on.  It represented my first real “large” project outside of work and academia.  Since then I’ve always felt a need to have something I’m working on in my spare time.  Something to hone my skills with.  JPL was very useful for keeping my knowledge of Java up-to-date.  Now that I’m mostly coding in Python at home, having little side-projects is a great way to get some extra experience.  The trouble with most day-jobs of course being that you actually have clients and deadlines - so exploratory coding often goes out the window.

There’s nothing quite like trying to create something useful on your own time.  Writing software from beginning to end by yourself can give you insights that you might otherwise never get.  I’d thoroughly recommend it.