At the end of 2010 I went to Paris (France) for an internship at a local company that produces multitouch hardware. I chose the job myself because I thought it would be interesting. What I implemented was markerless object recognition & improved tracking for the computer vision framework we’re developing (movid.org).

The video below shows the results of the prototype. The program is able to recognize objects based on their shape and size and does not need additional fiducial markers. It also takes into account object rotation (as long as you don’t have a perfectly circular object), even for square objects. You get an angle between 0 and 360 degrees.

The demo runs on an LLP setup. Only lasers, no diffuse illumination or similar approaches added.

The program that you actually see is just a visualization of the recognition and tracking, written in PyMT.

The quality of the video suffers from the fact that we had only 10 minutes to capture it before the table was transferred to an exhibition. The calibration was just quick and dirty, which is why I had to press that button to register an object (on the bottom) with a mouse instead of touching it.

What you see is a WIP project prototype. The code can be found in the GitHub repository on the master branch.

Markerless Object Recognition & Tracking (Movid) from Christopher Denter on Vimeo.

The NUIGroup Google Summer of Code students (I was lucky enough to become one of them for PyMT this year) are asked to summarize their weekly activities in blog format. Given that the first week has passed I figured I should just quickly outline what I have been working on up to now.

My proposal aims at developing more advanced text input methods for PyMT.

Work on PyMT

Some of the ideas I will realize draw heavily upon spelling correction and suggestion. It is therefore necessary that PyMT can interact with a spelling backend. Given that PyMT should be kept modular, I first implemented an abstract new core provider for spelling suggestions to become independent of a specific library. I then realized two concrete implementations of this provider:

• An enchant spelling backend. This uses the enchant spelling library which can itself be used with different kinds of dictionaries.
• A spelling backend based on OSX’s AppKit spell checker.

After the foundation was laid out I adapted a virtual keyboard with spelling support that Mathieu once developed to the new API and added it to the code base. All of this is not yet finished and needs some more love before I can merge it back into the master branch. You can check the branch I’m currently working on here.

Work on Movid

While spellchecking is important for some of my upcoming widgets, some other text input approaches make use of additional information provided by the tracking application. For example, one idea I had was to split the keyboard in half and dedicate one half to each hand. The halves would then automatically orient themselves following the respective hand’s position and orientation. Theoretically, further information such as properties of the user’s hands (length of fingers, etc.) could be taken into account to lay out the keyboards. For this I obviously need some kind of hand and fingertip tracking. Luckily I implemented that for Movid already:

However, since Movid is still not ready for end users due to a missing calibration utility and a proper (generic!) blob tracker (which means I can’t use it yet either), I continued my work on both of those. Again, both of which are not finished, but I can see the light at the end of the tunnel (or rather, the light below my fingers):

I hope that we can finish all of this and push out a first version of Movid for end users soon. And obviously, I want to test my text input widgets on my multitouch table and not in the mouse simulator.

This concludes my work for week one. If you have any questions or are interested in PyMT or Movid, feel free to join our IRC channel at #pymt and #movid on irc.freenode.net.