Although I really haven’t been looking, I don’t think I’ve seen any iPhone applications that use multi-touch for 3D navigation, so I thought I’d see if I could come up with a simple mapping suitable for navigation in E15, effectively turning the iPhone into a device for remote navigation of 3D space. My initial set of mappings used the pinching gesture to navigate in the z direction, and parallel two-fingered gestures for rotation about the x and y axes.


After sketching these up, I wrote an iPhone application that detects the relevant gestures, and communicates with a running E15 instance on another machine, sending this gesture information over the network. I added E15 support to handle this incoming data, as well as built some infrastructure for invoking user-defined Python callbacks in response to actions received from the iPhone (e.g. tapping and shaking). Below is a short movie of demonstrating this functionality working in a 3D E15 context to interact with an image collection obtained from YouTube video thumbnails. It also shows E15 playing live video from YouTube, something I’ve never quite gotten around to showing in anything other than static images on my Flickr account and E15:Web.


Read more here.

We’ve just started handing out E15 binaries to a select group of individuals to help us start hammering out the bugs. If you’re interested in being a beta tester, email buza at mit dot edu with your system specs (OS X 10.5 required), and why you’re interested in giving it a try.

Go!

I’ve been struggling for the past many months about how to most appropriately build an interface between Python and the DOM provided by the embedded E15 browser. As Tak and I prepare to show some E15 stuff at the upcoming ICWSM 2008 in Seattle, I began to feel some pressure to fix the integration once and for all. The end result (at least for the time being) is a Python function: jscripteval(). This allows us to evaluate arbitrary chunks of Javascript in the context of the embedded E15 browser in a manner similar to the Greasemonkey Firefox plugin. We can now write Javascript to inspect and modify elements of the DOM, and even return these values back to the Python context.

This kind of easy DOM access allows us to augment the traditional browsing experience in strange new ways. As a simple example, we can write some Javascript to return all of the src attributes of the current page, and load them as individual images into the E15 context:

from staticwriter import *
from random import random

jscript = “””function pyeval(s) {
    res = document.getElementsByTagName(s)
    var myArray = [];
    for (var i = 0; i < res.length; i++)
        myArray[i] = res[i].src;
    return myArray;
}
“””
envcolor(1, 1, 1)
images = jscripteval( jscript, “pyeval”, [”img”])
for img in images:
    imgload(img, img, random(), random(), random(), 0, 0, 0)

In addition to this usage, jscripteval() can be used in another way. Users can specify a URL pattern that, if matched, will trigger the invocation of the specified Javascript, and return the value through the invocation of a specified Python callback function. In this way, we can write some Javascript that extracts out all of the message headers from my Facebook message inbox, and places that text next to each user icon.

Well, it wasn’t easy, but now we’ve included support for the procedural loading and playing of YouTube videos directly within E15. Through the use of the poorly documented Google Data APIs, E15 users can access all kinds of interesting YouTube video metadata to aid their visualizations. The new E15 interface with the new console. Now with YouTube.

This past week, Tak was nice enough to give the E15 interface a full overhaul, and he’s done an incredible job. Everything is much cleaner, and this has allowed us to continue pushing forward.

During this time, I finished up some more advanced E15 additions, providing users with the ability to modify the way they interact with web content. I think the easiest way to demonstrate these new additions is through a detailed example, so here we go….

Suppose I’m an E15 user. I use it to write scripts and make nice visualizations. Suppose I have collected a few examples that I like, and want to make a nice little web page that contains some images that show what these scripts look like, and have those images link to the scripts themselves. As a demonstration, I took four existing scripts, and created a web page that shows them as follows:

The images in the page above link to the scripts themselves. That is, the html looks like the following:

<a href=”http://plw.media.mit.edu/balls.py”> <img src=”balls.png”> </a>

When loaded in the context of a standard web browser, clicking on one of the images will show the textual form of the python script, which isn’t particularly interesting. What we would really like to do, is be able to click on one of those images, and have the script load directly into E15. Load, and start running immediately. The question is, then, how the heck would you do something like that?

While there isn’t really any particularly elegant way of supporting this functionality when using a standard browser, we can now do some nice things within E15 itself. What end user can do now, is write Python code not only to create visualizations and graphics, but code to create native OS X interfaces. With this functionality, end users can write scripts to create OS X windows that contain a web browser, for example. In addition, users can define what happens when certain browser-specific events happen (e.g. clicking on a link). For the most part, this boils down to the user writing a small piece of handling code to do something special when links are clicked on in the context of the custom browser the user has written. So, the following script will allow us to detect whether or not the link we clicked on was a Python script, and if so, try to load it into E15:

def webView_didFinishLoadForFrame_(self,webview,frame):
    url = …
    if url.hasSuffix_(”.py”):
        e15_loadScript(url)

The result?

Running this custom handling code in the context of our custom E15 browser allows us to talk to the web in ways never before possible. Giving creative individuals the ability to do with web content what they want, as opposed to what they have to, stands to be a paradigm changing idea for the future of creative media, and is an integral part of E15.

This semester, I’ve been using E15 to visualize data from Facebook. Whether you like Facebook or not, it is one place that can offer a lot of social data related to you, and the data is unique for individual user. Thanks to E15 using Python, we can easily interface with Facebook using PyFacebook. Anyway, the Facebook visualization is still not done, but it will be posted when it is. This is a video I made for a class that shows the initial Facebook visualization.

There were numerous features that did not exist in E15 which had to be implemented for the Facebook visualization. Kyle already mentioned the threading issue. There’s now a method called settimeout and cleartimeout which can execute methods with a delay, much like JavaScript’s window.setTimeout() and clearTimeout(). Also, E15 did not have any facilities to move to an object in the environment nor a way to move the object’s coordinates once inserted.

Now both of these features are implemented. There is an elem_moveto() method which can animate any element into a new location in space. The camera can animate to fit an element into the viewing space by calling camera_goto().

Another feature attempted was using the accumulation buffer to create a trailing effect when elements moved. Given that E15’s native view size is 1600×900, when the accumulation buffer was enabled, the frame rate dropped to a point of uselessness. It does look cool though…

With the recent browser integration, E15 is now able to support custom user scripting. Because E15 has a Python interpreter at its core, page content can be searched, and browsing automated. As an example, we have integrated a spidering script into E15:In this example, a Python script directs the procedural traversal of a set of links originating from a single Flickr page. The end result is a space of pages that can then be visually explored. And if visual exploration is not enough, text-based filtering of results can reveal pages that would have otherwise been hard to find in a large set of pages:

When we give the E15 demo to sponsors here at the Media Lab, a question we’re often asked is whether or not the pages we’ve loaded are still “active”. Unfortunately, E15 was not built for this type of support. We’re trying to enable experiences and interactions with web content beyond the web browser, not to simply re-create the browser experience within a 3D environment. However, there is an important relationship between the browser and web content, and E15 needs to have the ability to talk directly to the browser. As a result, we’ve recently integrated a full-fledged web browser into E15:

With a browser, it becomes trivial to leverage the existing features of E15 to visually represent browser interactions. As a simple first demo, we can load a page into the embedded E15 browser, navigate through a set of pages, and construct a visual history of pages traversed. Show below is the result of the traversal of 9 links starting from google:

A lot of times people tend to forget about the importance of type support. Some have poor font rendering, but most don’t support things like kerning or ligatures. Not supporting kerning is just unacceptable. We made sure we have great type support in E15, and we do. E15 can render TrueType, PostScript and OpenType, all with kerning, ligatures and for OpenType, unicode support. The font rendering is also nice thanks to CoreText and Quartz. Here are some screen captures.

While adding support to text, we discovered that our mipmaps weren’t working. We noticed a lot of scintillation effects from the texts drawn in the environment. We are using NPOT textures, and after a couple days of going back and forth about whether NPOT mipmaps were possible or not, we finally figured it all out. As long as ARB_texture_non_power_of_two extension is supported, NPOT mipmaps will work just like POT. The trick is to use GL_TEXTURE_2D and treat NPOT just like POT.The other problem which hopefully is fixed with the official release of Leopard is that our ATI X1900 does not support hardware mipmap generation. Instead, we have to call gluBuild2DMipmaps(). The NVIDIA 7300 however does support it.

We’ve been using E15 to experiment with different layouts of web content in a 3D environment. The Python scripting interface makes it possible for us to have access to a wide variety of datasets using existing libraries to grab content from blogs, Flickr, Facebook, Wikipedia, and just about any other site with an API or RSS feed. What to do with the web content is a much more challenging problem than actually acquiring the content.

In some early sketches of web content layouts, I had arranged webpages around the perimeter of a circle. While searching for more interesting equations to use for laying out pages, I found an awesome Wikipedia page called List of Curves with over 50 different types of mathematical curves. The problem with this page, however, is that it has no pictures of the curves. I want to see what all the curves look like, but I’m not about to click through every single page, one at a time.

So, I wrote a Python script that uses the MediaWiki API for Wikipedia to find all the articles linked from the List of Curves article, finds the images linked from those articles, and finally renders all the images in E15 with the relevant article titles:

If you’re curious about how these pages were laid out, I used a rose curve with 8 petals — pictures of this curve are in the middle of the screenshot below:

One more screenshot:

I’ll be working with the same data more this week, so check back soon for more images, and probably a video or two!