Pronoid Android

By which I mean, my Android app is now in alpha test: I just uploaded it onto my HTC Magic, and whaddaya know, it runs, it doesn't crash, it actually does stuff. Indeed, I just updated WikiTravel via its interface. (Slightly buggily, but hey, alpha test.)

Mind you, the Android app is a fairly trivial piece of programming next to the AppEngine middleware that does the actual parsing and updating of WikiTravel. Which is the way it should be. Resources are scarce on a smartphone, and both processing and battery power are relatively meagre; I expect the design pattern of choice is going to be "get as much of the work done as you can on your server farm, then communicate simple low-bandwidth stuff to the phone."

Ultimately, the phone isn't much more than a user interface. In fact, now that I think about it, my whole app follows the classic Model-View-Controller design: WikiTravel is the model, the smartphone is the view, and my AppEngine service is the controller. Huh. Plus ça change, plus c'est la même chose.

Anyway, a list of tips, tricks, and annoyances since last we met:

• Let's start off with the annoyances. Character encoding. This has never been my strong suit, and I all too often wind up trial-and-erroring a solution. What doesn't help here is that I've got URL encoding (eg "?"->"%3F" in an HTTP request), HTML encoding (eg & -> & in a web page) and actual character encoding (ASCII? Unicode? UTF-8?) and while I grok the first two, I always have a mental block with the latter. I seem to have brute-forced something like a solution. We'll see. I expect it will rear its ugly head again.

  
  Python, weirdly, for all its text-processing power, doesn't come with much in the way of built-in HTML escaping and unescaping. "cgi.escape" will do much, but, annoyingly, not everything in the way of escaping. As for unescaping, fuhgedaboudit. Fortunately, I found this neat little solution somewhere on the Internets:
  

  
  #HTML unescape
  #taken from the Internet
  class Unescaper:
      entity_re = re.compile(r'&(#?[A-Za-z0-9]+?);')
  
      def replace_entities(self,match):
          try:
              ent = match.group(1)
              if ent[0] == "#":
                  if ent[1] == 'x' or ent[1] == 'X':
                      return unichr(int(ent[2:], 16))
                  else:
                      return unichr(int(ent[1:], 10))
              return unichr(name2codepoint[ent])
          except:
              return match.group()
  
      def html_unescape(self,data):
          return self.entity_re.sub(self.replace_entities, data)
  
  

  
  
• While I'm complaining, as a Java coder, I'm used to all objects having a "toString" representation, and thus, when logging or debugging, being able to put
  
  

  "this is"+anObject+" and this is "+anOther

  
  
  without worrying about syntax. But Python, otherwise a far superior text-processing lanaguage, won't let you do this: you have to put "str()" or "repr()" around objects you want to include in a string. For no apparent reason. Sigh.
  
• The good news is, the Java SDK now includes a perfectly acceptable HTML parser. I needed to parse attributes in a tag, didn't want to include a whole new JAR in my project for just that purpose, and really didn't want to write that code myself (it'd be like reinventing the wheel in Detroit.) org.xml.sax to the rescue:
  

  
   class ListingHandler extends DefaultHandler {
    public void startElement(String namespaceUri, String localName, String qualifiedName,
                  Attributes attributes) throws SAXException {
        for (int i=0; i<attributes.getLength(); i++) {
         String field = attributes.getLocalName(i);
         String value = attributes.getValue(i);
  

  
  and you don't even have to fuss with XML namespaces if you don't want to (and here, I don't.)
  
• GAEUnit continues to be awesome. Android unit testing continues to be much clumsier. Yet another reason for AppEngine to do most of the tricky work.
  
• Optimization. Always a thorny issue. As a wise programmer once taught me:
  
  • The first rule of optimization: don't do it.
    
  • The second rule of optimization (For Experts Only): don't do it yet.

  
  The android SDK comes with three excellent documents - Designing for Performance, Designing for Responsiveness, Designing for Seamlessness - that describe best practices. On the one hand, I'm not following them as closely as I could; the ExpandableListView at the heart of my UI is filled with homegrown and relatively expensive ViewEntry objects, rather than arrays as they suggest. On the other, it makes the code a lot easier to read, and ... don't optimize yet. Thus far the app seems fast and responsive enough. Says me. We'll see what others think.
  

• HTTP GETs and POSTs. In case you'd like an example of how to do that fairly efficiently from Android, here ya go:
  

  
      public static HttpResponse DoHttpPost(String target, ArrayList params)
       throws IOException
      {
       try {
        HttpPost httpost = new HttpPost(target);
     UrlEncodedFormEntity entity = new UrlEncodedFormEntity
      (params, HTTP.DEFAULT_CONTENT_CHARSET);
     httpost.setEntity(entity);
   
     //configure our request
        HttpParams my_httpParams = new BasicHttpParams();
     HttpConnectionParams.setConnectionTimeout(my_httpParams, CONNECT_TIMEOUT);
     HttpConnectionParams.setSoTimeout(my_httpParams, SOCKET_TIMEOUT);
     HttpClient httpclient = new DefaultHttpClient(my_httpParams);  //get http client with given params
        httpclient.getParams().setParameter("http.useragent", Settings.AppName);
   
     //upload set, let's try it
        Log.i("com.rezendi.wtw.Util", "Preparing to post "+params+" to "+target);
     HttpResponse response = httpclient.execute(httpost);
     Log.i("com.rezendi.wtw.Util", "Sent POST, got " + response.getStatusLine());
   
     //clear out the form data
     entity.consumeContent();
     return response;
       }
       catch (IOException ex)
       {
        Log.e("com.rezendi.wtw.Util", "Error posting "+ params +" to "+target, ex);
        throw (ex);
       }
      }
      
      public static String Encode(String toEncode) throws UnsupportedEncodingException
      {
       return URLEncoder.encode(toEncode, "UTF-8");
      }
  
      public static String DoHttpGet(String queryUri) throws IOException
      {
          HttpParams my_httpParams = new BasicHttpParams();
    HttpConnectionParams.setConnectionTimeout(my_httpParams, Util.CONNECT_TIMEOUT);
    HttpConnectionParams.setSoTimeout(my_httpParams, Util.SOCKET_TIMEOUT);
    HttpClient httpclient = new DefaultHttpClient(my_httpParams);  //get http client with given params
       HttpGet httpget = new HttpGet(queryUri);
  
       try {
        Log.i("com.rezendi.wtw.Util","Opening HTTP GET connection to "+queryUri);
        HttpResponse response = httpclient.execute(httpget);
        InputStream is = response.getEntity().getContent();
   
           BufferedReader reader = new BufferedReader(new InputStreamReader(is), BUFFER_SIZE);
           StringBuilder sb = new StringBuilder();
           String line = null;
           while ((line = reader.readLine()) != null) {
            sb.append(line);
           }
           
           is.close();
           response.getEntity().consumeContent();
           String results = sb.toString();
           return results;
       }
          catch (IOException ex) {
        Log.e("com.rezendi.wtw.Util", "Error GETting from "+queryUri, ex);
        throw (ex);
          }
      }
  

  
  
  

  I recommend that you put these in one central Util class, as I did. Mind you, these are static methods on Util, and not (yet) synchronized ... meaning not thread-safe. At the moment I don't think that's an issue; all my HTTP connections are very-low-bandwidth and resolve in seconds, so I can't imagine two such calls realistically colliding unless the phone user has very fast fingers indeed. But famous last words, right? Live by the thread, die by the thread.

  
  
• In both the Android and AppEngine cases, I started off by writing the tutorials that come with the SDK, and expanded the app from there. In both cases I'm not sure this was such a good idea. I dunno, maybe it's just my OO background, but if I'm writing a Notepad application, I'd like to have an object model with a Note object. But maybe that's not really the Android way. Performance, responsiveness, seamlessness, and all that.
  

  I did like the way they calved all direct DB access off to a DbHelper class, and I've expanded that. Here's a hint; have all your DbHelpers inherit from a common ancestor. Yeah, I know, favour composition over inheritance, but still, here it will save you a lot of time, and give you a central repository for database creation scripts and such. Also, I'm still not exactly sure when you're supposed to close your database objects. In "onPause()"? Immediately after accessing them? Don't bother and let the Activity superclass handle it? It seems unclear.
  

• I'm not sure of the best way to handle error handling in Android. I like to have a common superclass for all my UI objects and inherit error handling from it, but that ain't gonna happen here, as my Activities inherit from various different places on Android's Activity tree. I guess I'll build some sort of ErrorHandler and pass exceptions to it where I think they're most likely to occur? Kind of annoying that there's no single piece of code you can write to catch all exceptions before they hit Android's own error-handling, but I guess that's the price you pay for their heavily compository app system.
  
• Moving back to AppEngine: the memcache service is awesome. Speeds up performance and cuts down on resource consumption immensely. However, for your own good, create a "ClearCache" web handler, for debug purposes. I've twice now spent fifteen minutes wondering why a problem wasn't fixed before realizing that the erroneous data was still in the cache and had not been replaced.
  
• Python's HTMLParser is very useful. However, it seems to have weird problems with ampersands. I think I've now end-run around those problems, in a clumsy kludgy way, but I fear they too may yet crop up again.
  
• This project has included just enough JavaScript (maybe 100 lines) that I wonder if I should have used JQuery, and just little enough that I conclude that I probably shouldn't. Next time, maybe.
  
• I do believe that's all for now. See all y'all in beta, if not before.
  

Labels: Android, GET, HTTP, POST

So I've been doing lots of little stuff of late. I wrote my own Preferences edit screen, only to find that Android comes with an automagic one (sigh, although, I also wish this had been more obvious in the documentation.) I explored most of the mysteries of ExpandableListView, which is a pain to use, but would be an even bigger pain not to use.

And I've built the basic spine of my application - an Android app talking to an AppEngine service, which takes a few well-specified requests and feeds back a few tightly structured (and low-bandwidth) responses. The smartphone app never talks to WikiTravel directly; instead it goes through AppEngine, which can a) do all the heavy lifting, b) log and track what's going on, c) cache data more intelligently, d) provide another layer of control and indirection.

It's actually working pretty well; I'm updating WikiTravel successfully. (Well, from the emulator. I'm going to buy a real phone next week, and I expect to unearth more problems with the app when I do...) The UI is sinfully ugly, and right now it can only show you the well-formatted kind of WikiTravel listings to update, and there are a bunch of niggling things to fix and hardcoded strings to move to R.string and kludges to klean up, but by and large I'm pretty pleased with the progress.

However. I have found the first thing about the Android-AppEngine stack about which I have serious reservations. I'm talking about AppEngine's "GQL" data storage.

Now, on the one hand, this object database is quite cool. You can create, save, and write simple SQL-like queries for objects very easily. It reminds me, pardon me while I date myself, of the good old days of working with GemStone for Smalltalk, a wonderfully elegant (but ahead of its time, and hence cripplingly slow) object database for the world's purest OO language.

But if memory serves you could do a lot more with Gemstone than GQL. Especially when it comes to queries. Now, I'm biased from years of experience with SQL databases - I already know how to make them do what I want, whereas with GQL I have to work it out from a fairly thin foundation - but even so, there is much to be desired.

For instance: you can only have one inequality operator (eg "height < 5") in a query. Which means that "height > 3 AND height < 5" is actually impossible to perform. But wait, there's more; if there's an order by clause as well, it must refer to the field that uses the inequality operator. So "height < 5 ORDER BY emailAddress" is similarly impossible. Oh yes, and fetching more than 1000 rows at a time is equally impossible. As is counting all the rows that match a given WHERE clause. The official workaround for that last is to write a Sharded Counter, which, I think you'll agree, is orders of magnitude more complex than writing "SELECT COUNT(*)".

In general, getting around these restrictions means restructuring your object model heavily ... if you can get around them at all. OK, you denormalize databases for performance, and SQL has its quirks and blind spots too, but jeez, this is way past that: it's more like getting a car and being told "Listen, the steering wheel only turns right, so you have to make three right turns every time you want to go left."

I'm sure they have their reasons. Scaling being one. And for an app like mine, where the data-storage requirements are pretty simple and straightforward, this all works fine. But if I was building something seriously data-intensive? I would think not just twice but thrice before using AppEngine, at least until it supports a more fully-featured datastore.

Labels: AppEngine, GQL

So I've finished the first crude version of my middleware and uploaded it to AppEngine. Looks like I'm going to have to upload the indexes by hand, though - I ran it through its paces before uploading it, but got a NeedIndexError: no matching index found when I tried to run it on appspot.com. Oh well. No biggie. I'll do that tomorrow and then go back to my Android app.

But mostly I wanted to tell you about a fun little app called zipme. AppEngine doesn't let you directly examine the source of the files you've uploaded. However, someone named "manatlan" wrote "zipme", a single python file that you add to your root directory so that you can subsequently download the entirety of your source code, zipped, from AppEngine. See here. (It's configured so you have to be logged in as admin, in case you don't wanna show your source to the world...)

eta: spoke too soon - the indexes are now up n' running. However, JavaScript form handling is not. Well, this is why you deploy early and deploy often, so that you don't get bit by it at the last minute. Goin' on a bug hunt, brb...

etaa: in case you're curious, I realized after five minutes that the culprit was neither AppEngine deployment nor my code: it was the NoScript in my browser.

Labels: AppEngine, python, zipme