it is time for the next demo of my project, as I reached the beta status. (feature complete) I think you can see quite clearly now where this is going and which kind of interactions will be possible using this technique. The concrete features are described in the annotations.
If you use more advanced tracking methods and add some physics to this, you could easily port numpty physics to this kind of interaction or create an easy to use level editor. In case you missed my last video, here is the link.
It will still take some weeks until this hits maemo-extras as there are still some bugs left and I still want to get rid of keyboard use for interaction.