Zoom-Zoom-Zoom — Get Multi-Touch Zooming Support on your T-Mobile G1 TODAY

I posted recently to show that working multi-touch input is available on the T-Mobile G1 phone. Now the necessary changes to the Android software stack are finally in good shape, and the software is easily installable on your own phone. (“Easily” being a relative term of course — you have to re-flash your phone’s firmware, and that may void the warranty…)

Keep reading for links to instructions for installing multi-touch support on the G1, as well as source code and screenshots for several multi-touch demo applications

Please donate to support continued development of awesome features for Android!

Obligatory video

This video shows the multi-touch demo apps that are described below.

EDIT: Yes, these demos are a little slow/clunky if you’re used to the iPhone — but that’s not the point. The demo apps (and the hacked system apps, Browser and MapView) can be made a lot more responsive through the addition of:

• OpenGL acceleration
• kinetic zooming / inertia (so that “fling” is supported) — should be trivial to add to the multitouch controller, I just didn’t do it
• and, in the case of maps and the browser, the addition of an API that is designed for partial zoom factors: the current API for both just suppports “zoom in”/”zoom out” not “zoom to 1.053x current zoom”, and starts the builtin jerky animation to get there, jumping in/out by a big step each time the animation has finished.

This is a proof of concept, people — of course it will be optimized over time — the fact it’s not done yet doesn’t mean it can’t be or won’t be. Apple’s engineers were paid mega moola to implement their bling, I was not. It’s actually not that bad if you try it yourself — and here’s the real point: you can get working MT on your G1 TODAY and it was never designed for that. Try it out and you’ll agree that MT zooming in the browser is a freakish lot better than the zoom button solution that Google implemented in the 1.0 release.

Approach

I moved my original multi-touch code back into the kernel, because it turns out that currently it’s a lot easier to patch the kernel and get a working Android system than it is to patch the Android java stack and get a working system. (The Android java stack that made it into the G1 was branched and polished long before the source code was released publicly, and the source code in git usually doesn’t run without problems due to being in a state of flux.) You can find the kernel patch to the synaptics touchpad driver here. Many thanks to zinx for helping to polish the kernel patch and figure out the best way to get mutitouch info into userspace.

I also patched the Android browser to support multi-touch scaling, source/diffs are linked below. The patched version also includes support implemented by JesusFreke for autorotating web pages based on phone orientation (you turn the phone on its side without even sliding out the keyboard, and the web page you’re viewing rotates) — you have to manually enable this in the Preferences to get it working though. See a list of known issues with the browser below.

Note that this approach to multi-touch is completely backwards- and forwards-compatible: you can write an app that implements extra features if multi-touch is supported on the phone. (The technical details: currently there is no way for an app to detect whether or not the phone has this multi-touch patch until the user initiates a multi-touch gesture, but at that point the size field of a MotionEvent in the onTouchEvent() method will have a value greater than 1.0. You can read multi-touch information by parsing this value, as shown in MultiTouchController.java in the demo apps below. Unpatched G1s will never generate a size value greater than 1.0, so you know that a size value less than or equal to 1.0 is a single-touch event regardless of whether or not the phone has been patched.)

EDIT: I received the following comment on this blog post, and wanted to highlight it, along with my response: “So let’s add this kernel patch to the mainline linux kernel, then google has no choice but to include it (I’m sure goog devs would love this). So google pushes this kernel patch with a OTA update, and everyone can use multitouch. I’m sure google would then try to delete all multitouch apps off the marketplace or they fight it out w/ apple over their bogus lawsuit.” My response: “Won’t happen, this is just a hack. The mainline kernel doesn’t need modifying, it already generates multitouch events, the event pipeline in the Android Java stack just drops the info for the second touchpoint. I hacked the kernel driver to hide info about the second touch point in the first touch point’s size field, so that the Java stack doesn’t need modifying. However ultimately the correct solution is to fix the Java stack to pick up the multitouch info and pass it on to userspace. This will require quite a lot of re-engineering though.”

Getting Multi-Touch on Your Phone

Unless you want to build and install your own patched Android kernel, the easiest way to get multi-touch on your phone is to get root access on your phone and then install JesusFreke’s replacement Android firmware image v1.41, available here:

–> Rooting your phone and installing JFv1.41 firmware images <–

The JFv1.41 images include the multi-touch patches to the kernel and browser to support multi-touch zooming, as well as numerous other useful bits and pieces (such as busybox, ext2 support, easy “su” root access, a data partition backup utility, and more).

If you want more detailed directions, you can find them here and here.

IMPORTANT NOTE: Flashing your phone carries a non-zero risk of voiding your warranty (although it is not clear whether or not this is true) and/or of bricking your phone (meaning you could render it unusable). If you re-flash your phone’s firmware, you take full responsibility for any damage that may result. Generally if you are careful, you won’t run into any major problems — but note especially that all user-data will be wiped from your phone when you first re-flash back to RC29 to get root access. Any data that is synchronized with the Google mothership *shouldn’t* be lost due to the fact that it’s all primarily stored on Google’s system, but other non-synchronized things *will* be lost e.g. your SMS message history, your Market-installed applications and their data, and probably other things. There are a few backup utilities on the Android Market, e.g. one to back up SMS messages. Once you have root and have patched your phone to include the engineering bootloader, then you can use the nandroid backup utility to backup files so that this does not happen in future.

Note also that OTA (over-the-air) updates from T-Mobile are generally disabled when you install community-generated alternative flash images like the ones above. This is to prevent T-Mobile re-flashing your phone with an image that takes away root access from you again. This means that when T-Mobile releases an update, you will need to wait a few days for someone in the community to produce an updated system image that includes the update. Generally (and increasingly, as the public source code repository becomes more stable), community images will be far more up-to-date anyway than what T-Mobile pushes out to you over the air, but there is of course a risk that something might break. JesusFreke has images based off of both T-Mobile official firmware and development (ADP1.1) firmware.

MULTI-TOUCH DEMO APPS

I have written several multi-touch demo apps for the G1. In most cases these are not intended for widespread use but to demonstrate to developers how to make use of the new multi-touch information that is generated by the kernel patch. Each app contains a copy of a class MultiTouchController.java that I wrote to simplify writing MT apps. Note that kinetic panning or zooming has not yet been implemented in the controller, but it should be simple enough to add.

Looks identical to the regular Android Browser -- but much easier to zoom on...

Multi-Touch Browser

This is a patched version of the standard Android WebKit browser that supports multi-touch zooming. JesusFreke has kindly included it in his v1.41 image as described above, and you can get the source/patches here.

Features

• Multi-Touch zooming using “pinch” gesture like on the iPhone. Fully working and ready to use for daily browsing.
• Some pages are not zoomable (even in the single-touch browser), and all pages have an upper and lower zoom limit. With multi-touch zooming, you are shown a new message when you cannot zoom into or out of a given page, in order to give the user more feedback as to why the pinch gesture isn’t working.
• Also contains a patch by JesusFreke to support autorotate of the browser display based on phone orientation. You have to manually enable this in the Prefs.

Issues

• When you zoom (even in the single-touch browser), the Android Browser changes the text wrap width of all major blocks of text on a page to fit the width of the screen, causing a reflow of the HTML page a second or two after zooming. With multi-touch, this causes the browser page to jump around as you are zooming — mostly noticeable on smaller pages that don’t take long to reflow, e.g. google.com “Classic”. EDIT: Note that there is an option “Auto-fit pages” in the Browser settings that you can turn off to disable reflow. This makes multitouch browsing much more responsive, because the browser is not constantly trying to reflow and doesn’t suddenly reset your view position when the reflow has completed, but of course if the text line width is greater than the screen width then you will need to scroll back and forth to read. It’s a tradeoff…
• There is a bug in the current release of the (single-touch) Android Browser where the embedded WebView remembers its zoom setting across page views, but the BrowserActivity does not. Therefore the BrowserActivity may be thinking it is displaying a page at a different zoom than it actually is. If you zoom all the way into a zoomable web page like say cnn.com, and then browse to a non-zoomable page like google.com “Mobile”, then the embedded WebView will still be zoomed all the way in, but the BrowserActivity will tell you you can’t zoom out on the current page. This bug seems to be present whether you use multi-touch zooming or the zoom buttons, but it is a lot easier to run into on the multi-touch browser, because zooming is a much more pleasant experience so you’re likely to do it more 🙂 Somebody should look into this bug, it’s probably not hard to fix. If you find a fix please send it to me…
• You need to keep your fingers at least an inch or so apart for the zooming to work well, due to the “snapping” problem that was shown in my first video, and in the part of the video in this post that shows the MTVisualizer application.

A rather boring screenshot of the MTMapsDemo application -- no fun unless you're actually using it 🙂

MTMapsDemo

This is a very basic implementation of multi-touch zooming on Google Maps. It basically just wraps a WebView and displays your current location on the map, which you can then scroll around and zoom into/out of. You can get the source here or an .apk (compiled binary) that you can install on a patched phone here.

Features

• Display maps showing your current GPS location, jump to current location, pan, multi-touch zoom.

Issues

• No directions, no searching of the map, and nothing else either — the main reason being, Google has chosen not to release their Google Maps app as open source for reasons that are beyond me, so you’re left with having to re-invent the wheel for every map application, and you have to work through with a WebView through an API that is very limited (at least from a navigation perspective).
• There are issues with the map jumping wildly and/or scrolling locking up if you zoom out too far. This is due to inaccuracies in the inverse-Mercator projection calculations. Google doesn’t give you much of an API to work with in MapView, unfortunately, or much example code — so it’s hard to get a maps app working as well as their one does. EDIT: it is almost impossible to browse Europe or New Zealand for this reason.  I’m sure there’s a way to do this right using the API, it’s just complicated because the API is very small and limited, and converting between the different coordinate systems used (degrees, microdegrees, pixels, zoomed pixels, pixels-at-the-equator — as well as Mercator projection vs lat/long) makes this quite hard to do.  Somebody please look into it, this is just a demo 🙂
• Additionally, I think there is an issue with the view jumping back to the GPS position sometimes when you’re zooming. Shouldn’t be hard to fix but this is only a demo of what can be done, it’s not intended for use as a real replacement for Maps… Some interested party should look into this, I am unlikely to keep developing this app myself. I hope somebody will pick this up and make a full-featured Google Maps replacement…

The photo sorter demo application

MTPhotoSortrDemo

This is a basic photo management application that demonstrates how to use the MultiTouchController to handle dragging and stretching multiple different objects. You can get the source here or the .apk here.

Features

• Allows you to move photos around in a stack, reorder them, and stretch/scale them.

Issues

• Limited to the built-in images. A simple improvement would be to load photos from the SD card.
• Somewhat slow because it uses Skia, it would be better to code this up as an OpenGL app.
• May occasionally run out of memory and crash — Android apps are limited to using 16MB on the G1, so image-manipulating programs have to be very careful about how they use memory.  EDIT: Happens especially when you slide open the keyboard.

The MT Visualizer app. Touch points are at the endpoints of one of the diagonal lines.

MTVisualizer

This is an app that allows you to visualize the multi-touch information, so you can see the limitations of the G1’s multi-touch screen (that it is a 2x1D device not a true 2D device, as I described in my previous post). You can get the source here or the .apk here.  NOTE: For this particular demo, because the .apk file is very small, if installing directly on the phone you must download the .apk over wifi, not over 3G, because a bug in the browser causes the file to get truncated a couple of kb short.  The app installer in that case will say that it wants to replace “Android System”.  Some info is here.

Features

• Displays row and column of each touch point (vertical/horizontal lines), touch radius for multi-touch (big red circle), angle between touch points (with diagonal lines), midpoint between touch points (blue lines), pressure (yellow circle centered around midpoint).

Issues

• This app shows that multi-touch on the G1 is limited in several ways — you get some weird snapping/popping effects etc. when touch points get close in one or other axis.  As a result, the G1’s screen is a much better device for multi-touch scaling than it is for multi-touch rotation or more general multi-touch input, because the distance between the two touch-points in multi-touch can be measured a lot more reliably than the actual position of the points.

Official support in Android?

There is no official statement from Google about planned multi-touch support in Android, although there is a driver in the kernel tree for an iPhone-like screen. In my opinion, it is unlikely that the G1 will ever get official support for multi-touch from Google in its lifetime, because it was never designed as a multi-touch device, and there are some limitations to multi-touch on this device. Google will also not accept upstream this specific workaround solution to hack multi-touch onto the G1, understanably so, as no multi-touch API has yet been proposed or agreed on. See this thread for more information.

Credits

Thanks to RyeBrye for his original work on multi-touch, JesusFreke for his awesome work on producing souped-up replacement images and for incorporating my changes into his latest release, zinx for helping polish the kernel patch, and DarkriftX and Disconnect for general awesomeness and help with documentation, testing and other aspects of this release.

Links

Shameless plug: If you like this work then please feel free to use this link for amazon.com purchases.

Also if you don’t have a G1 phone yet, you can buy one using this link, or if you already have a G1, you can buy accessories here.

Conclusion

That’s it! This is all the work I ever expect to do on multi-touch for the G1, because I have more interesting ideas for apps that I want to spend my time working on. Watch this space. I hope other programmers will pick up multi-touch and run with it. Since this multi-touch solution is backwards and forwards compatible, there is no reason people can’t start adding additional multi-touch capabilities to apps that they release to the Market, starting today. When an official multi-touch API is available, apps can target that instead.

Have fun.

—
Luke Hutchison

Advertisement