augmented reality – Joaquim Rocha

Controlling GNOME with Leap

August 9, 2013December 31, 2013

When I explained how the Leap Motion device could be used on Fedora 19, I mentioned how I had one of those early prototypes. Well, Leap Motion was extremely kind and sent me an actual device as a thank you for starting the thread asking for Linux support. Now that GUADEC is over and I am spending my vacation in Portugal, I had a little time to play with my fancy new device and wrote a relatively small script to control GNOME with it. I call it the über original name of: Leap GNOME Controller!

For those who don’t care about technical details, here’s the video showing what can be done with Leap, GNOME and this script. Technical details follow below the video:

The two videos that compose the one above were recorded with an HD camera and GNOME Shell’s screencast recorder. I tried to sync them the best I could but a certain delay can be noticed, especially at the end of the video.

The code

Leap Motion provides a “close source” shared library and a high-level API with the respective documentation for the many bindings it has. To code it quickly, I used the Python bindings and Xlib to fake the input events.

Leap Motion’s APIs make it really easy for one to simulate a touch-screen. It even offers a “screen tap” gesture that should be the obvious choice when mapping a finger touch/tap to a mouse click. However, this didn’t work very well. The problem is that if we are tracking one finger to control the mouse movement, when performing the “screen tap” gesture, the finger (and mouse) will of course move. Making it as frustrating as seen on ArsTechnica hands-on video.

I came up with a solution for this by dropping the “screen tap” gesture and using the “key tap” instead. The “key tap” is a simple, quick down-and-up finger movement, like pressing a key. This is much more precise and easier for a user to do than the “screen tap”. Of course that when the finger moves for performing the gesture, the mouse pointer would move as well, so I came up with a little trick to work around this: when the mouse pointer doesn’t move more than a couple of pixels for half a second, it will stop and only move again if the user makes it move for more than 150 pixels. This allows for the user to stop the pointer with precision where it needs to be and perform the gesture without making the pointer move.

Future

The Leap device offers a lot of possibilities for adding many gestures. Ideally they should be implemented per application but being able to control the shell is already pretty useful, so it would be wonderful to fine-tune the current gestures and add new ones. I also wish the library’s source code were open because I ran into small issues and I wish I could take a look at the source code, instead of trying to fix it based on the theories of what might be wrong.

I haven’t explored the AirSpace appstore yet so I don’t know if it is worth adding (or possible to add) this script there but I will check it out.

Have fun with Leap and GNOME!

Leap Motion on Fedora 19

July 22, 2013July 22, 2013

I was one of the fortunate developers who got one of the first Leap Motion development units. Unfortunately the SDK was only available for Windows and Mac OS so, since I couldn’t use it, I started a topic on Leap’s developer forum to ask for a Linux SDK. The number of users joining the petition kept increasing until Leap released an SDK (closed source binaries) for Linux about 4 months ago… when I was moving to a different country and had left my, then useless, Leap device in my parents’ attic 🙂 (thanks to my brother for sending it to me)

The SDK for Linux is actually a deb package intended for Ubuntu and, although there is also a tarball with the compiled shared libs and examples, as far as I could check, only the deb package provides the demon and other useful tools. Since I am running Fedora, that meant that I had to allocate a bit more of my time to explore this wonderful device.

The device will officially launch tomorrow so I thought I could write a little set of steps for Fedora (and maybe other distros) users to play with it. The only dependency that I think is necessary to install is mesa-libGLU.x86_64 for running the visualizer.

I cooked a little video showing how it works before the boring sections below:

Install the Leap SDK

First we need to download and extract the deb package (at the time of writing, the SDK was no longer available to download until they release it, hopefully in less then a day).
If you are a rebel and have dpkg installed on your Fedora like I do, go ahead and do:
$ dpkg -x Leap-0.8.0-x64.deb Leap-0.8.0
or, if you don’t want to use dpkg:
$ mkdir Leap-0.8.0 && ar p Leap-0.8.0-x64.deb data.tar.gz | tar zx

Inside the Leap-0.8.0 folder there will two folders: usr and lib .

Copy (as root) the rules file to the udev rules directory:
# cp lib/udev/rules.d/25-com-leapmotion-leap.rules /etc/udev/rules.d/

Then we need to add our user to the “plugdev” group. If it doesn’t exist, create it (as root):
# groupadd plugdev
# usermod -a -G plugdev YOUR_USER_NAME

After this step it’s probably a good idea to restart your session…

As mentioned above, the deb ships a few binaries, found inside the usr/bin folder. It contains the leapd service, a Visualizer, a Recalibrate tool, a LeapControlPanel and a ScreenLocator (this one segfaults on my machine and I didn’t care why). I prefer to put these and other files inside my user’s .local folder:
$ cp bin/* ~/.local/bin/

Notice that, except for leapd, all other binaries have an upper case name and, most of them, a rather generic one which some users might want to change (e.g. to leap-visualizer, leap-calibrator, etc.) before copying them. I will not change them in this example.

There are also some configuration files that need to be copied from usr/share/:
$ cp -R usr/share/Leap ~/.local/share/

At this point we can already play with the Leap Motion device by running the leapd service and using e.g. the visualizer (it might be a good idea to actually recalibrate the device before using it, with the Recalibrate tool):
$ leapd &
$ Visualizer

Development

All is done except for the development libraries. If you want to develop you own stuff using Leap, you should have libLeap.so in your libraries’ path. This .so can be found in usr/lib/Leap, inside the extracted deb, or you can download the Leap Developer Kit which brings are some bindings like Python (LeapPython.so) and Java (libLeapJava.so).
Of course you can copy these libs directly into /usr/lib64/ but I prefer to have them in .local/lib/ and setting the LD_LIBRARY_PATH and, for Python, the PYTHONPATH, variables.
To copy the libs from the folder extracted from the deb package:
$ cp usr/lib/Leap/* ~/.local/lib/
To copy the libs from the Leap Developer Kit tarball (using a 64 bit architecture):
$ cp LeapSDK/lib/x64/* ~/.local/lib/

After copying the .so files:
$ export LD_LIBRARY_PATH=~/.local/lib
$ export PYTHONPATH=~/.local/lib

Have fun with Leap Motion on Fedora!

Skeltrack 0.1.14 released

June 27, 2013

The first release of Skeltrack in 2013 is out!

This is also the first version I release without being associated with Igalia but the company agreed that I keep maintaining the project. That’s one of the good aspects of working in Free Software as your daily job: you can continue working on them even if you’re no longer in the company where it was first developed.

This new version is not bloated with new features so “why such a wait?” you might ask. Well, the reason is that there have been a few changes in my life since last year and, more importantly, after I left Igalia, I didn’t have a Kinect and thus, working on Skeltrack was a bit difficult… But finally a few weeks ago I bought one so now I am able to keep hacking.

Faster Tracking

The greatest thing in this release is a big improvement in the performance of Skeltrack when tracking joints. You see, when I originally developed it, I was more concerned with getting it to actually track the skeleton joints than doing it quickly 🙂
Luckily, Iago López (kudos to him) did a neat work in fixing some misuses of GList but more importantly, re-implementing Dijkstra using a priority queue. If you remember your algorithms courses well, with this change you can make Dijkstra go from quadratic to linearithmic time and that’s a BIG improvement.
Here is the difference between the old code and the new expressed in a plot:

The values used to generate the plot were the time it took skeltrack_skeleton_track_joints_sync to execute. For each of the versions, this function was called 25 times for each of the 550 depth frames and the final values in the plot are the arithmetic average of the results per frame. They were executed in my i5 2.40 GHz laptop without any applications or the desktop running (besides system’s services).
As you can see, there’s a great difference between the speed of the old code and the new. I can’t wait to try Skeltrack on weaker hardware now 😀

(BTW, why 0.1.14 and not 0.1.12? Because of the “release curse” which made me aware of a simple but important bug fix right after pushing 0.1.12…)

Other Things

Besides some small bug fixing in the library, the Skeltrack Kinect example was ported to Clutter 1.12 to keep up with the work done upstream.

Another new thing is that now Skeltrack has a webpage. I recently found out about GitHub’s automatically generated web pages and I thought this would be better than having people visit GitHub’s repo page so there you go: http://joaquimrocha.github.io/Skeltrack/
The page is a bit ugly and minimalist but we’ll improve it.

The documentation can now be found here and bugs can be filed in GitHub as usual.