With this Google Summer of Code’s firm pencils down deadline approaching rapidly, I thought I’d write a status report for what is mostly going to be the ‘official’ result of my work. I won’t drop dead right after the deadline passes or this posting is published, but this is just one of the formalities I have to take care of, the sooner the better.

During the course of this Google Summer of Code, I’ve been extensively working on the OS X support of the Kivy framework. As I have explained previously, Kivy’s set of features is based on the concept of providers, meaning that every functionality required by the framework is encapsulated in a provider that defines an abstract interface to the user (i.e. developers) of the framework. Each such interface then has a number of specialized implementations using (in almost all cases) a third-party or system supplied software library that was designed to do work in the field that the respective provider operates in.

Now my task of this GSoC was titled “Tight Mac OS X Integration”, which means that we wanted to reduce the number of external libraries (not system libraries) that we were using in Kivy, as those have to be compiled for and bundled with our OS X executable. Not only does this add to the overall memory requirements (storage wise), it also requires extra steps to be taken when it comes to maintenance and deployment.

In concrete terms, during this GSoC I implemented the following providers:

• Window (based on SDL; this was an internal decision tito and I made as this would be reusable on iOS as well)
• Image (based on Apple’s Core Graphics and Quartz APIs; I actually provided two versions of this, see below)
• Text (based on Apple’s Quartz APIs; I too provided two implementations for this)
• Audio (based on Apple’s Cocoa APIs)
• Video (based on Apple’s QTKit APIs)

Now for the image and text providers, I wrote two versions, respectively: One Python based version that uses PyObjC (available by default on OS X) and one ObjC based version to which I bridge using Cython. The advantage of the PyObjC versions are that they’re just single Python files that can be run on any recent Mac without inquiring any bundling or presence of additional tools, or even compile-link cycles. The drawback is that I cannot use them on iOS, as PyObjC is way too bloated for what I need, not well maintained and not functional on iOS anyway. That’s why I have a second version of those providers that actually works on iOS, as can be witnessed in my last blog posting. I will branch these off into the iOS support branch and bring them in back later into master when Kivy support for iOS is officially supported.

So what have we got now? Seven new provider implementations for 5 different core tasks. One of which (Window, SDL) will actually be reusable on all our supported platforms, not just OS X. The other four use native system APIs that already exist on any mac, so there is absolutely no memory footprint (storage wise) added and as soon as this hits master, we will see a dramatic reduction of size and bloat of the Kivy installer (and it will make my life as the OS X maintainer a whole lot easier). In numbers (and this is a back-of-the-envelope calculation), we’ll probably be reducing the size from 100 MB (uncompressed) to about 10 MB (uncompressed). They also benefit from the functionality that OS X inherently provides, such as audio and video codecs (the list of which can be added to by the user by installing things like Perian).

Since midterm, that means we’ve got audio and video providers, a new audio example, text and image PyObjC-based providers, significant visual improvements in terms of text rendering and under the hood changes for text and image display. This is a before/after shot for text rendering. There have also been some other artifacts around the characters that I’ve also gotten rid of. This and a couple fixes from tito now also make it work in the text input widget.

Here’s a video showing the video provider in action (The stutter comes from the screen capture. That video displays smoothly on my Mac).

This was the first time I’ve worked with Objective C or Apple’s APIs, so naturally a lot of work went into researching the different APIs, learning how Objective C works and how I can make use of it (I will write another posting to describe an alternative approach to using Objective C from Python that I came up with). I have a great sense of personal accomplishment in terms of teaching myself and therefore learning new things in this area and this is one of the major things I really like about Google’s Summer of Code project.

As a side note, I recently moved to the US for the remainder of the year to write my master’s thesis and I had to take care of a huge amount of (paper)work for that (this is kind of a pioneer project). So there’s some glitches remaining that I’ll certainly be looking into as soon as I get the time and then merge all of this new code into master and make it available to the user.

For instance, amongst other things, the text display isn’t a pixel perfect match with the other platforms yet and the video provider has problems with larger files that gstreamer handles properly (but at least it handles some other formats that gstreamer doesn’t, I just don’t want to break existing apps that rely on gstreamer supported formats to work at this point).

Anyway I’ll continue to dig into these remaining tasks before and after the deadline and I’d like to take a moment to thank a few people for what has been a terrific Google Summer of Code (probably my last as a student, unfortunately): Paweł Sołyga for being my mentor, Christian Moore for taking care this project could exist under the NUIGroup organisation umbrella and Mathieu Virbel for all the discussions and the help.

I recently had the opportunity to do some research with the goal of being able to run Python on any iOS device (iPhone, iPad, iPod touch). The idea is to only write some Python code (and nothing else) and deploy that to different platforms without changing it (e.g. Windows, Linux, Mac OS X, Android, iOS).

If you’re interested, here’s a preview/draft document that at a very high and easy to understand level very roughly summarizes what had to be done.

Now I’m not saying that this is THE way to develop cross-platform software, especially for devices such as tablets. The goal just was to see whether or not it’s technically possible and feasible to write applications for iOS using Python only. Fortunately, it seems possible and actually the programs run pretty snappy. They also use the GPU for rendering using OpenGL ES 2.0. Also, there was no jailbreak necessary.

Consider this work in progress. There’s still many things on the TODO list, I just wanted to share the early results with you and let you know that it is in fact possible. The code is on github and I’m using the kivy framework. I’m looking for opportunities to present this in much more depth in a journal or at a conference. If you know of any opportunities, please send me a mail (address in the PDF).

Update: I mentioned the code to be on github, but didn’t provide any actual links as I was in a hurry when I wrote the blog post. Here are the links: Python for iOS repo (compiles Python 2.7 for ARM, based off of cobbal’s repo): https://github.com/dennda/python-for-iphone Kivy iOS support branch: https://github.com/tito/kivy/tree/ios-support Objective C test app that embeds Python and runs a Kivy example: https://github.com/dennda/python-for-iphone-test You will also need SDL 1.3.

And last but not least I’d like to repeat what I wrote in the PDF and thank my friend Mathieu Virbel (from the kivy team) for all the help. I especially enjoyed the hack session we had at UDS.

I am very happy that I am allowed to participate in yet another Google Summer of Code in 2011. This will be my fourth consecutive and probably last GSoC – at least as a student.

I am participating on behalf of the Kivy project for the second time now and my objective is to improve the integration of Kivy on Mac OS X.

Let me explain what that means: In order for Kivy to function with its full set of features, which includes things like audio and video playback, window creation with OpenGL content, image display, spelling correction and more, we need to make use of some external software libraries that are dedicated to handle the respective tasks, such as video decoding, etc. Implementing a cross-platform video decoder for example is much work and beyond the scope of our project, which is why we use the work of other brilliant minds instead of reinventing the wheel.

That said, there are a lot of different implementations for the kind of things we require. Since our goal is not to restrict the user to a certain set of those solutions, we abstract from those third party APIs and provide the necessary functionality through a common interface; both cross-platform and cross-API. This approach also enables us to more easily port Kivy to new platforms, such as Android, as we don’t have to modify the code that is using the abstractions, but just add a new implementation (which we refer to as a ‘provider’) for the new platform.

Taking video as an example, we currently rely on gstreamer for the implementation. While gstreamer has proven to be a good choice on Linux (as it’s often included in Gnome based distributions anyway) it’s not an ideal choice from my point of view for other platforms like Mac OS X. This is because it adds additional packaging and maintenance overhead and also significantly increases the size of the Kivy distribution, as we have to compile and ship gstreamer with Kivy for the Mac (which is not easy anyway) because it doesn’t exist there by default. This is especially true because there is a native API to do the same things you’d use gstreamer for on the Mac. That API is always available on the Mac and using it would mean that we can still provide video playback support without shipping gstreamer. In fact, this is not only true for video, but for all of Kivy’s so-called ‘core providers’.

Long story short, the goal of this GSoC is to use as many native Mac APIs (as is reasonable) instead of external libraries.

Here is a list of Kivy’s core providers:

• Window
• OpenGL
• Image
• Text
• Audio
• Video
• Spelling
• Clipboard
• Camera

OpenGL already works and so does Spelling because I had already taken care of those previously.

The first things I tackled during this GSoC were the window and image providers. Obviously you need a window to display the other things and have some feedback while developing, so this was a pretty obvious starting point. The original plan was to base the new window provider on Cocoa’s APIs directly, but while looking into that Mathieu pointed out that it might be preferable at the start to base it on the development version of SDL instead (which is also what pygame uses) as they already have this abstraction for every platform under the sun, so other platforms might benefit as well. Also, SDL is pretty easy to compile and package. So after some hacking, what do we have right now? There is a working SDL window provider. You can use it to open a Window and display OpenGL content. Still missing is handling of certain SDL events such as proper mouse/touch integration. Wile you can already draw a line with it, it’s not finished the way it should be yet. That is a TODO item.

Next up was the image provider. Why? Simply because Kivy loads a default texture for use in its GL operations, so we figured instead of patching that out temporarily I might as well implement the image provider directly. So what we have right now is a working image provider based on CoreGraphics that should be able to handle all image formats that the CG APIs natively handle. There are only two things missing. Firstly it doesn’t seem to be handling alpha channels properly yet and secondly it currently reports a fake list of supported formats. These are TODO items.

After finishing what I pointed out before, next up would be text and video providers.

Starting tomorrow (May 30th) I will be in the US for a couple of days to visit the university where I’ll hopefully be doing my Master’s thesis. Unfortunately I had to sign a form that I will not engage in any GSoC related coding activities while I am in the US (you have to do that if you’re non-US based), so my next report will have to be a little delayed.

This is what it currently looks like with the pictures example:

The code is available here and right now very much work in progress.

Google’s Summer of Code 2010 comes to an end for me today. It has been a great time working on awesome projects like PyMT and Movid. My task was to enhance PyMT’s text input methods. One of the joys of this task was that it allowed me to work on a relatively wide scope of things. Here’s a brief list of what I worked on:

• I added a new spelling provider to PyMT that abstracts from individual spellchecking libraries. That means you can use your favorite spellchecking library, which is important considering that PyMT is cross-platform.
• I added two actual spelling providers that implement this protocol: One enchant spelling provider (usable after installing enchant) and one provider using OS X’s native AppKit spellcheckers (so you get that out of the box on OS X).
• Mathieu once wrote a basic virtual keyboard with spelling suggestions which I adapted, cleaned and merged.
• PyMT obviously already had some text input widgets, which I improved (e.g. MTTextArea).
• I began working on a version of MTTextInput with added spellchecking (like OO.org with red lines drawn for incorrectly spelled words), but that needs some more love.
• One of the more concrete objectives of my task was a Swype-like keyboard for PyMT. I created a prototype for that, see the video below.
• Another concrete objective was a split keyboard (split into two parts, one half for the left, one for the right hand) that adjusts to your hand’s properties (e.g. size). To achieve this, a substantial amount of changes was needed to our vision tracking application (Movid):
• For the keyboard to adjust to the user’s hands, a handtracking algorithm was needed that I implemented for Movid. It detects the fingertips of the hand as well as the hand’s center. These are just seen as a certain type of ‘blobs’ internally.
• These blobs need to be tracked over a sequence of frames from the camera. Additionally, we also want to find simple touches (without all the hand information). For that, I added and integrated BlobFinder and BlobTracker modules that obey a common format so they’re easily interchangable.
• When your camera senses a blob on the touch surface, the application needs to perform a mapping to get the blob into screen coordinates. We do that using a calibration module, which I had started before SoC. I finished it and merged it back into our master branch.
• As an extra feature, I added a PyMT module that you can use to calibrate your tracker from within your client application, eliminating the need to switch applications. I also added a Flash GUI for the calibration so that you can easily do it on any remote computer via our web interface.
• To actually send the handtracking to the client application, Mathieu added a TUIO2 module to Movid. I started a PyMT input provider for TUIO2. Both of which is work in progress, but I believe we’re the first project to adapt TUIO2 (there’s not even a reference implementation yet).
• The result of that can be seen in the second video below. Also, make sure to read the vimeo description.
• Other than that we now also provide portable binary packages for PyMT 0.5 for both Windows and OSX. I created the OSX package, so it’s no longer a major pain to install. You just download and run it.
• And, of course, many more fixes!

Some of that is already in PyMT 0.5. All of the Movid stuff will be in the first release. In future releases we shall see much improved versions of these prototypes and hopefully even context aware word suggestions.

Here are the two promised videos, if you’re reading this through a planet, please go directly to my blog.

Prototype WipeToType Keyboard for PyMT from Christopher Denter on Vimeo.

Ergonomic multitouch keyboard prototype from Christopher Denter on Vimeo.

Thanks to all the people who made this possible. Thanks Google, Christian, Pawel, Mathieu and Thomas, for being (a) fantastic mentor(s). It has been a great pleasure and privilege to work with you in GSoC 2010 and I sure will continue to work on both projects.

Many things happened in the last few weeks. I just want to quickly outline them in case you’ve been wondering what I’ve been doing.

PyMT Spelling

Starting with PyMT 0.5 (to be released in August), we added support for spelling correction and word suggestions. This is based on my GSoC work. The code has been polished and integrated into the master branch, which will soon lead to the 0.5 release. I’m also currently working on a text input widget that indicates incorrectly spelled words as you type (just as OpenOffice would). This much is working. In a next step, I plan to add a feature that lets you select from a list of suggestions for the word that you just tapped.

WipeToType

One of the tasks of my GSoC proposal is the implementation of a Swype-like keyboard. What this means is that you just wipe over the keys that make up the word you want to type and it automatically determines which word you intended to enter. It is clearly far beyond the scope of a single, multitouch-oriented GSoC proposal to implement something as clever as a Swype clone (especially since this also requires A LOT of backend code for the intelligence). However, something remotely similar and usable should be doable and is what I’m looking for. A while back I started something like this and quickly sketched a modified version of the virtual keyboard.

This still needs much more love, but keeping in mind that I did this in a really short amount of time I think I can say that we’re getting somewhere.

FITG in Lille

Thomas, Mathieu and I have had the chance to meet in Lille at the FITG conference and present PyMT in a talk and several workshops. This was a great opportunity and in fact, it was the biggest real-life meeting of core developers and users so far.

The conference itself was a great success, both for the organizers and us. We had many people come to us and ask questions concerning PyMT and Movid. After our talk (which I think was well-received) we decided to give an additional workshop so that people interested could play with PyMT and get help from us. The room was pretty crowded and people were standing. The workshop presented a basic PyMT overview and first steps in a ‘hello world’ fashion (At least I think that’s what Mathieu was talking about. He spoke french and Thomas and I were answering questions or translating things in English). In the evening we gathered all the people that were still at the conference and went to a nice little restaurant to chat.

The next day we gave two more workshops. The idea to do the first one came up while we had breakfast. We decided to implement a simple version of the game at linerider.com with PyMT. When we arrived an hour later, we just picked a python 2D physics library that was easy to install and started live and from scratch, without any actual code having been written (or even thought about) beforehand. Luckily it all turned out well. After almost exactly one hour (in which Mathieu helped people in the audience, Thomas pointed to and explaining stuff at the projection and I coded and talked) we had finished what we were looking for in just 60 lines of unoptimized python code. The last workshop was about advanced OpenGL. Mathieu presented some of his insights that he had gathered while optimizing PyMT’s performance (great advances have been done here, by the way).

I stayed four days in total and it was absolutely worth it. Lille is a wonderful city and the conference was fantastic. The venue itself was just mind-blowing to begin with. We had a lot of fun together and obviously worked on PyMT as well. It’s even more fun if we’re in the same room! Sincere thanks to everyone involved in making these days as awesome as they were!

Reworked TextArea

For our talk in Lille, we used a very nice presentation tool (PreseMT) which is, obviously, written with PyMT. While using it for my own bachelor’s colloquium (I’m officially a Bachelor of Science now, by the way) I noticed that entering text suffered from severe limitations of the text input widget. Given that I had no multitouch-capable device around to enter text, I did it all with my hardware keyboard. I added to PyMT’s TextArea widget the ability to resize automatically depending on the text that was entered (which is what you want in PreseMT). Furthermore, the widget now properly reacts to several special keys like the arrow keys, home, end, del, pgup and pgdown.

Portable PyMT

It is no secret that installing PyMT on OSX is a major pain. This is not our fault, though. The problem simply is that installing almost anything in non-app format involves a non-trivial compilation process using MacPorts and the like. Unfortunately, one of our dependencies (gstreamer) is not easily installed this way.

Since we really don’t want our users to go through all of this, we decided to distribute portable versions of PyMT for OSX, Linux and Windows. I did the OSX version and hope to be able to finish it soon so that it can be reviewed. With it, you just download a zip file, unzip it and go. It contains everything that is needed to run PyMT.

In the course of this, I also fixed the compilation of our OpenGL-dependant cython modules for OSX.

Conclusion

Hopefully you will see a wonderful PyMT release next month. We’ve added many new features, improvements and fixed a lot of bugs. Some of my GSoC work will also go into it. In terms of GSoC, I will finish the aforementioned spelling-aware text input widget. I also intend to improve the quality of the results of the WipeToType keyboard and implement the things left on my GSoC proposal.

The NUIGroup Google Summer of Code students (I was lucky enough to become one of them for PyMT this year) are asked to summarize their weekly activities in blog format. Given that the first week has passed I figured I should just quickly outline what I have been working on up to now.

My proposal aims at developing more advanced text input methods for PyMT.

Work on PyMT

Some of the ideas I will realize draw heavily upon spelling correction and suggestion. It is therefore necessary that PyMT can interact with a spelling backend. Given that PyMT should be kept modular, I first implemented an abstract new core provider for spelling suggestions to become independent of a specific library. I then realized two concrete implementations of this provider:

• An enchant spelling backend. This uses the enchant spelling library which can itself be used with different kinds of dictionaries.
• A spelling backend based on OSX’s AppKit spell checker.

After the foundation was laid out I adapted a virtual keyboard with spelling support that Mathieu once developed to the new API and added it to the code base. All of this is not yet finished and needs some more love before I can merge it back into the master branch. You can check the branch I’m currently working on here.

Work on Movid

While spellchecking is important for some of my upcoming widgets, some other text input approaches make use of additional information provided by the tracking application. For example, one idea I had was to split the keyboard in half and dedicate one half to each hand. The halves would then automatically orient themselves following the respective hand’s position and orientation. Theoretically, further information such as properties of the user’s hands (length of fingers, etc.) could be taken into account to lay out the keyboards. For this I obviously need some kind of hand and fingertip tracking. Luckily I implemented that for Movid already:

However, since Movid is still not ready for end users due to a missing calibration utility and a proper (generic!) blob tracker (which means I can’t use it yet either), I continued my work on both of those. Again, both of which are not finished, but I can see the light at the end of the tunnel (or rather, the light below my fingers):

I hope that we can finish all of this and push out a first version of Movid for end users soon. And obviously, I want to test my text input widgets on my multitouch table and not in the mouse simulator.

This concludes my work for week one. If you have any questions or are interested in PyMT or Movid, feel free to join our IRC channel at #pymt and #movid on irc.freenode.net.

During 2008′s Google Summer of Code I introduced a new storage abstraction layer to the MoinMoin wiki engine. This allows you to run moin on a variety of backends (filesystem, mercurial, etc.).

This year (2009) I participated again. My application contained three major areas of work:

• Reintroducing ACLs for the storage branch.
• Adding fairly advanced ‘routing’ configuration capabilities for your storage backend(s).
• Adding a SQLAlchemy backend which, in theory, should support a variety of RDBMS.

The first two objectives were finished quite well. Two different pieces of middleware were introduced for both, the ACLs and the routing. You can now use several different backends in a simple or optionally fairly complex way. You could just use one single filesystem backend or potentially hundreds of different backends, each with different ACLs applied, and mount each one into its own namespace (you may be familiar with this concept from UNIX, where you can mount discs and such into arbitrary places of the filesystem tree).

In addition to that, ACLs have been refactored. ‘revert’ and ‘delete’ were removed (obviously you will still be able to perform the corresponding actions from the UI), ‘create’ and ‘destroy’ were added.

As for the SQLAlchemy backend, we do have a somewhat working version. Unfortunately it still suffers from severe performance problems. Originally I wanted to fix that after SoC, but real-life (bachelor’s thesis, etc.) caught me, so I will have to postpone that. If you have some SQLAlchemy experience and want to help out here (or with something else), you are very welcome to join us (#moin-dev on irc.freenode.net).

For an overview of what else has been done during GSoC 2009 on the storage side of things, take a look at the following resources:

• http://moinmo.in/Storage2009
• http://hg.moinmo.in/moin/2.0-storage/

Please keep in mind that this is still fairly alpha. Especially the UI is for developers and geeks only and will be redone properly (with Jinja2) before the release (help needed here as well).

Thanks to the whole moin crew and especially my mentor Thomas Waldmann for the nice and fun collaboration!

(Oh and – hopefully – hello planet python!)