Although I really haven’t been looking, I don’t think I’ve seen any iPhone applications that use multi-touch for 3D navigation, so I thought I’d see if I could come up with a simple mapping suitable for navigation in E15, effectively turning the iPhone into a device for remote navigation of 3D space. My initial set of mappings used the pinching gesture to navigate in the z direction, and parallel two-fingered gestures for rotation about the x and y axes.


After sketching these up, I wrote an iPhone application that detects the relevant gestures, and communicates with a running E15 instance on another machine, sending this gesture information over the network. I added E15 support to handle this incoming data, as well as built some infrastructure for invoking user-defined Python callbacks in response to actions received from the iPhone (e.g. tapping and shaking). Below is a short movie of demonstrating this functionality working in a 3D E15 context to interact with an image collection obtained from YouTube video thumbnails. It also shows E15 playing live video from YouTube, something I’ve never quite gotten around to showing in anything other than static images on my Flickr account and E15:Web.


Read more here.

I’ve been struggling for the past many months about how to most appropriately build an interface between Python and the DOM provided by the embedded E15 browser. As Tak and I prepare to show some E15 stuff at the upcoming ICWSM 2008 in Seattle, I began to feel some pressure to fix the integration once and for all. The end result (at least for the time being) is a Python function: jscripteval(). This allows us to evaluate arbitrary chunks of Javascript in the context of the embedded E15 browser in a manner similar to the Greasemonkey Firefox plugin. We can now write Javascript to inspect and modify elements of the DOM, and even return these values back to the Python context.

This kind of easy DOM access allows us to augment the traditional browsing experience in strange new ways. As a simple example, we can write some Javascript to return all of the src attributes of the current page, and load them as individual images into the E15 context:

from staticwriter import *
from random import random

jscript = “””function pyeval(s) {
    res = document.getElementsByTagName(s)
    var myArray = [];
    for (var i = 0; i < res.length; i++)
        myArray[i] = res[i].src;
    return myArray;
}
“””
envcolor(1, 1, 1)
images = jscripteval( jscript, “pyeval”, [”img”])
for img in images:
    imgload(img, img, random(), random(), random(), 0, 0, 0)

In addition to this usage, jscripteval() can be used in another way. Users can specify a URL pattern that, if matched, will trigger the invocation of the specified Javascript, and return the value through the invocation of a specified Python callback function. In this way, we can write some Javascript that extracts out all of the message headers from my Facebook message inbox, and places that text next to each user icon.