Wednesday, September 21, 2011

 

Image sizing and scraping with JQuery and Rails


I know, I know, long time no type. Not my fault, I swear! First I went travelling, then I had a buncha work to do, then I went to San Francisco TechCrunch Disrupt, all of which occupied my time. Well, I suppose, on reflection, all those things were my fault, but, um ... look, I'm back now, OK?

Back with a li'l discussion of image sizing and scraping with Rails and JQuery, for a project I can't really talk about yet. Suffice to say that sometimes I want to calculate the size of various remote images - quite a lot of them, in fact, making performance important - and sometimes I want to scrape all the images from a set of web pages.

I thought the first task was going to be tricky. And maybe it is. But fortunately, someone has solved it for me, via the totally awesome FastImage Ruby gem, for which praise should be heaped upon one sdsykes. It works exactly like advertised, which is to say, like this:


  def self.picture_info_for(url)
    return nil if url.blank?
    begin
      size = FastImage.size(url)
      return size  #[width, height]
    rescue
      logger.info "Error getting info for picture at "+url.to_s
      return Array[0,0]  #this makes sense for my app, but maybe not yours
    end
  end


So, hurrah! This meant the scraping bit was actually tricker. Sure, I could have done it all in Rails, but it's user-facing, and I didn't want the user to have to wait for a bunch of potentially sequential http requests to complete without seeing any results. I could have done it all on the client side, but parsing HTML with Javascript, even with JQuery, sounded painful and fraught with difficulties, compared to using the dead-easy Hpricot gem. So I came up with a compromise I quite like:

1. On the client side: (written using HAML, which I mostly adore)
- @image_urls.each_with_index do |url,idx|
  = link_to url, url
  %div{:id => 'page_'+idx.to_s}     

%script
  $(function() {
  - @image_urls.each_with_index do |url,idx|
    $.ajax({
    url: '/stories/scrape_images?url=#{CGI::escape(url)}',
    success: function(msg){ $('#page_#{idx}').html(msg); },
    error: function(msg){ $('#page_#idx}').html(msg); }
    });
  });

2. On the server, to first create and then respond to that client page:
  def popup_scraped_images
    start_time = Time.now
    @image_urls = []
    @seed = Seed.find(params[:seed_id])
    @seed.active_signals.each do |signal|
      next if signal.main_url.blank?
      @image_urls << signal.main_url
    end
  end

  def scrape_images
    url = params[:url]
    slash = url =~ /[^\/]\/[^\/]/
    host = slash.nil? ? "" : url[0,slash+1]
    html = ''

    page = HTTParty.get(url, :timeout => 5)
    Hpricot(page).search("//img").each do |element|
      img_src = element.attributes["src"]
      img_src = host+img_src if img_src.match(/^\//)
      html += '<img src="'+img_src+'" />' if img_src.match(/^http/)
    end
  end


Hopefully how they all interact is self-explanatory. Et voila - semi-asynchronous Rails/JQuery image scraping, handled on the server side for easy caching if need be later on.

Labels: , , , , , , ,


Monday, May 16, 2011

 

Looting Android's log files

Hello, my faithful droogsreaders. Been awhile since I've posted here, because I've been super-busy, mostly writing code. But I'm off travelling for a couple of months shortly (Ethiopia, France, Ladakh) so I thought I'd leave you with a little Android/Bluetooth tidbit before I go:

In theory, any Android 2.1-or-later device can open serial port protocols to any Bluetooth device. In practice, though, a bunch of handset manufacturers have, for reasons known to them alone, crippled their Bluetooth stack so that they can't Discover (a word with a loaded meaning in Bluetooth) any device with a generic (ie set to 0) Bluetooth class.

How annoying, no? However, it turns out that - even on those devices - you can pair with and connect to those devices, if you know their address. "But Jon," you say, "how can I get their address, if the Discovery process has been crippled?"

Well. It turns out that the Qualcomm drivers in question, when they ignore the generic Bluetooth devices, do write to the Android log file the fact that they are ignoring it. So all you need to do is mine your log files for the address. How do you do that? So glad you asked.

First, add to your AndroidManifest:

<uses-permission android:name="android.permission.READ_LOGS" />


Second, use this method:

private Set checkLogs() {
HashSet devices = new HashSet();
if (mAdapter==null)
return devices;

//check the logs
Log.i(""+this, "Log check...");
try {
Process process = Runtime.getRuntime().exec("logcat -d *:e");
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
String deviceAddress=null;
while ((line = bufferedReader.readLine()) != null) {
if (line.indexOf("0x00 - skip it") > 0) {
int end = line.indexOf("] class");
int start = line.indexOf("Device [");
if (start>=0)
start=start+"Device [".length();
deviceAddress = line.substring(start, end);
Log.i(""+this, "Found "+deviceAddress+" in log");
BluetoothDevice device = mAdapter.getRemoteDevice(deviceAddress) ;
devices.add(device);
//String deviceName = device.getName()==null ? device.getAddress() : device.getName();
//Util.DoToast(Main.this, "Discovered device "+deviceName);
}
}
bufferedReader.close();
process.destroy();
return devices;
}
catch(IOException ex) {
Log.e(""+this, "Error checking logs - "+ex);
ex.printStackTrace();
return null;
}
}


...and finally, just connect to the device explicitly:


btSocket = device.createRfcommSocketToServiceRecord(GatewayManagerConversation.uuid);
btSocket.connect();


...and that should trigger the pairing process, and after pairing, you can connect to the device (which will be in your local cache of paired devices, as normal.)

Hope that helps!

Labels: , ,


Monday, January 3, 2011

 

Two great maps that taste great together

I spent yesterday adding OSMDroid support to iTravelFree; here's a little precis of what I learned about letting users swap back and forth between Google Maps and OpenStreetMaps within the same Android Activity.

Why? you might ask: well, two reasons. One, OpenStreetMaps is a pretty cool project. Two, OSMDroid lets you access OpenStreetMaps even while offline, which is pretty awesome for a travel app, and while Google Maps App 5.0 has some offline support, it not you decides which tiles get cached. Mind you, creating and downloading OSM maps to Android is non-trivial, though the next iTravelFree release will make it much easier.

Also, it's just kinda interesting swapping back and forth between GMaps and OSM and seeing the differences between the two.

Anyway: how to do it? With a basic Decorator pattern, ie we roll our own MapView object (in my case, ITRMapView) which wraps both a Google MapView and and OpenStreetMapView, and forwards method calls appropriately depending on its state.

Arguably the best implementation would be via a common ITRMapViewImplementation interface and a concrete class for every map type. (If there were more than two map types, this would no longer really be arguable.) But I implemented it quick-and-dirty, with a bunch of if statements, which does at least have the serendipitous effect of being easy to use as an example.

Anyhoo, without further ado, here's the code:

/**
* Wrapper class which lets a MapActivity switch back and forth between Google Maps and OpenStreetMaps.
*/
public class ITRMapView {
private MapActivity mapActivity;
private BalloonLayout balloon;

private ITRGMapView gmap;
public ITRGMapView getGMap()
{
if (gmap==null) {
gmap = new ITRGMapView(mapActivity, "my_api_key");
gmap.setClickable(true);
gmap.setMapActivity(mapActivity);
gmap.setBuiltInZoomControls(true);
gmapLocation = new com.google.android.maps.MyLocationOverlay(mapActivity, gmap);
balloon = BalloonLayout.GetBalloonFor(mapActivity);
}
return gmap;
}
private com.google.android.maps.MyLocationOverlay gmapLocation;
private ListingsOverlay gmapListingsOverlay;

private OpenStreetMapView osm;
public OpenStreetMapView getOSMMap() { return osm; }
private org.andnav.osm.views.overlay.MyLocationOverlay osmLocation;
private OpenStreetMapViewItemizedOverlay<OpenStreetMapViewOverlayItem> osmListingsOverlay;

private boolean osmActive=false;
public boolean isOSMActive() { return osmActive; }
public void setOSMActive(boolean doOSM) { osmActive=doOSM; }

public ITRMapView(MapActivity activity) {
mapActivity=activity;
}

public void setOSMMap(OpenStreetMapView theOSMMap) {
osm=theOSMMap;
osmLocation = new org.andnav.osm.views.overlay.MyLocationOverlay(mapActivity, osm);
int centerLat = gmap==null ? 0 : gmap.getMapCenter().getLatitudeE6();
int centerLon = gmap==null ? 0 : gmap.getMapCenter().getLongitudeE6();
int zoom = gmap==null ? 10 : gmap.getZoomLevel();
org.andnav.osm.util.GeoPoint center = new org.andnav.osm.util.GeoPoint(centerLat, centerLon);
osm.getController().setCenter(center);
osm.getController().setZoom(zoom);
osm.setUpFor(mapActivity);
}

public void onResume() {
if (gmapLocation!=null) {
gmapLocation.enableMyLocation();
gmapLocation.enableCompass();
}
if (osmLocation!=null) {
osmLocation.enableMyLocation();
osmLocation.enableCompass();
}
}
public void onPause() {
if (gmapLocation!=null) {
gmapLocation.disableMyLocation();
gmapLocation.disableCompass();
}
if (osmLocation!=null) {
osmLocation.disableMyLocation();
osmLocation.disableCompass();
}
}

public void centerAndZoom(int centerLat, int centerLon, int spanLat, int spanLon) {
Log.i(""+this, "Center and zoom: "+centerLat+" "+centerLon+" "+spanLat+" "+spanLon+" osm "+osmActive);
if (osmActive) {
org.andnav.osm.util.GeoPoint center = new org.andnav.osm.util.GeoPoint(centerLat, centerLon);
OpenStreetMapViewController controller = osm.getController();
controller.setCenter(center);

//quick-and-dirty, because zoomToSpan doesn't work
Integer maxSpan = Math.max(spanLat, spanLon);
Double ln = Math.log(maxSpan.doubleValue());
Integer zoomLevel = 20-ln.intValue();
if (zoomLevel>5)
zoomLevel+=2;
if (zoomLevel>18)
zoomLevel=18;
controller.setZoom(zoomLevel);
}
else {
com.google.android.maps.GeoPoint center = new com.google.android.maps.GeoPoint(centerLat, centerLon);
MapController controller = gmap.getController();
controller.setCenter(center);
controller.zoomToSpan(spanLat, spanLon);
}
}

public int getMapCenterLatE6() {
if (osmActive) {
int lat = osm.getMapCenterLatitudeE6();
//Need to clamp to be consistent with GMaps
if (lat<-180000000)
lat+=360000000;
if (lat>180000000)
lat-=360000000;
return lat;
}
return gmap.getMapCenter().getLatitudeE6();
}
public int getMapCenterLonE6() {
if (osmActive) {
int lon = osm.getMapCenterLongitudeE6();
//Need to clamp to be consistent with GMaps
if (lon<-180000000)
lon+=360000000;
if (lon>180000000)
lon-=360000000;
return lon;
}
return gmap.getMapCenter().getLongitudeE6();
}

public void fillMap(ProgressIndicator indicator, HashSet<MappableItem> mCurrentSubset) {
if (osmActive)
fillOSMMap(indicator, mCurrentSubset);
else
fillGMap(indicator, mCurrentSubset);
}
synchronized void fillOSMMap(ProgressIndicator indicator, HashSet<MappableItem> mCurrentSubset) {
if (mCurrentSubset==null)
return;

ArrayList<OpenStreetMapViewOverlayItem> items = new ArrayList<OpenStreetMapViewOverlayItem>();
MappableItem[] toMap = mCurrentSubset.toArray(new MappableItem[mCurrentSubset.size()]);
int listEnd=Math.min(toMap.length, Settings.GetMapMax());
for (int i=0; i<listEnd; i++) {
OpenStreetMapViewOverlayItem newItem = new OpenStreetMapViewOverlayItem(
toMap[i].getIDString(),
toMap[i].getMapDetailText(),
new GeoPoint(toMap[i].getLatitudeE6(), toMap[i].getLongitudeE6()));
newItem.setMarker(toMap[i].getMarker(mapActivity));
items.add(newItem);
if (indicator!=null)
indicator.showProgress(i, listEnd);
}

osmListingsOverlay = new OpenStreetMapViewItemizedOverlay<OpenStreetMapViewOverlayItem>(
mapActivity,items,
new OpenStreetMapViewItemizedOverlay.OnItemGestureListener<OpenStreetMapViewOverlayItem>(){
@Override
public boolean onItemSingleTapUp(int index, OpenStreetMapViewOverlayItem item) {
RelativeLayout rl = (RelativeLayout) osm.getParent();
rl.removeView(balloon);
balloon.disablePointer();
balloon.setUpTapFor(mapActivity, item.getTitle(), item.getSnippet());
RelativeLayout.LayoutParams params = new RelativeLayout.LayoutParams(200, 100);
params.addRule(RelativeLayout.ALIGN_PARENT_TOP);
params.addRule(RelativeLayout.CENTER_HORIZONTAL);
rl.addView(balloon, params);
return true;
}

@Override
public boolean onItemLongPress(int index, OpenStreetMapViewOverlayItem item) {
// Toast.makeText(mapActivity, "Item '" + item.mTitle + "' (index=" + index + ") got long pressed", Toast.LENGTH_LONG).show();
return false;
}
});
}
synchronized void fillGMap(ProgressIndicator indicator, HashSet<MappableItem> mCurrentSubset) {
if (mCurrentSubset==null)
return;

Drawable defaultMarker = mapActivity.getResources().getDrawable(R.drawable.map_fave);
gmapListingsOverlay = new ListingsOverlay(mapActivity, gmap, defaultMarker);
MappableItem[] toMap = mCurrentSubset.toArray(new MappableItem[mCurrentSubset.size()]);
int listEnd=Math.min(toMap.length, Settings.GetMapMax());
for (int i=0; i<listEnd; i++) {
Drawable marker = toMap[i].getMarker(mapActivity);
ListingOverlayItem overlay = toMap[i].getListingOverlayItem();
if (overlay!=null)
gmapListingsOverlay.addItem(overlay, marker);
if (indicator!=null)
indicator.showProgress(i, listEnd);
}
gmapListingsOverlay.doPopulate();
}

public void showListingsOverlay() {
if (osmActive) {
List<OpenStreetMapViewOverlay> mapOverlays = osm.getOverlays();
if (!mapOverlays.contains(osmListingsOverlay))
mapOverlays.add(osmListingsOverlay);
if (!mapOverlays.contains(osmLocation))
mapOverlays.add(osmLocation);
}
else {
List<Overlay> mapOverlays = gmap.getOverlays();
if (!mapOverlays.contains(gmapListingsOverlay))
mapOverlays.add(gmapListingsOverlay);
if (!mapOverlays.contains(gmapLocation))
mapOverlays.add(gmapLocation);
}
postInvalidate();
}

public void postInvalidate() {
if (osmActive)
osm.postInvalidate();
else
gmap.postInvalidate();
}

public int getLatitudeSpan() {
if (osmActive)
return osm.getLatitudeSpanE6();
return gmap.getLatitudeSpan();
}
public int getLongitudeSpan() {
if (osmActive)
return osm.getLongitudeSpanE6();
return gmap.getLongitudeSpan();
}
public int getZoomLevel() {
if (osmActive)
return osm.getZoomLevel();
return gmap.getZoomLevel();
}

}


And how is it used? Well, the MapActivity includes code like this:

private void switchMapMode() {
int lastLat = mapView.getMapCenterLatE6();
int lastLon = mapView.getMapCenterLonE6();
int lastLatSpan = mapView.getLatitudeSpan();
int lastLonSpan = mapView.getLatitudeSpan();
boolean willHandleCenterZoom = lastLat!=0 && lastLon!=0 && lastLatSpan!=0 && lastLonSpan!=0;

if (mapView.isOSMActive())
useGMap(!willHandleCenterZoom);
else
useOSMMap(!willHandleCenterZoom);

if (willHandleCenterZoom)
mapView.centerAndZoom(lastLat, lastLon, lastLatSpan, lastLonSpan);
}
private void useGMap(boolean doCenterZoom) {
mapView.setOSMActive(false);
setContentView(R.layout.map);
//remove map view from old parent, if any
ITRGMapView gmap = mapView.getGMap();
RelativeLayout rl = (RelativeLayout) gmap.getParent();
if (rl!=null)
rl.removeView(gmap);
//add map view to new parent
rl = (RelativeLayout) findViewById(R.id.mapLayout);
rl.addView(mapView.getGMap());
setUpMap(doCenterZoom);
}
private void useOSMMap(boolean doCenterZoom) {
mapView.setOSMActive(true);
setContentView(R.layout.map_osm);
mapView.setOSMMap((ITROSMMapView)findViewById(R.id.mapview_osm));
setUpMap(doCenterZoom);
new FillMapTask().execute();
}
private void setUpMap(boolean doCenterZoom) {
registerForContextMenu(findViewById(R.id.tagButton));
registerForContextMenu(findViewById(R.id.mapButton));
setUpToolbar();
if (doCenterZoom && mCurrentLat!=0 && mCurrentLong!=0 && mCurrentLatSpan!=0 && mCurrentLongSpan!=0)
mapView.centerAndZoom(mCurrentLat, mCurrentLong, mCurrentLatSpan, mCurrentLongSpan);
else if (doCenterZoom && mapView.isOSMActive() && mCurrentLat==0 && mCurrentLong==0)
mapView.centerAndZoom(51000000, 0, 1000000, 1000000);
}


Of some interest, I suppose, are the BalloonLayout and ListingsOverlay classes mentioned above, so:


public class BalloonLayout extends LinearLayout {
private boolean pointerEnabled=true;
public void disablePointer() { pointerEnabled=false; }
public void enablePointer() {pointerEnabled=true; }

public BalloonLayout(Context context) {
super(context);
}

public BalloonLayout(Context context, AttributeSet set) {
super(context, set);
}

@Override
protected void dispatchDraw(Canvas canvas) {

Paint panelPaint = new Paint();
panelPaint.setARGB(0, 0, 0, 0);
RectF baloonRect = new RectF();
baloonRect.set(0,0, getMeasuredWidth(), 2*(getMeasuredHeight()/3));
panelPaint.setARGB(230, 255, 255, 255);
canvas.drawRoundRect(baloonRect, 10, 10, panelPaint);

if (pointerEnabled) {
Path baloonTip = new Path();
baloonTip.moveTo(5*(getMeasuredWidth()/8), 2*(getMeasuredHeight()/3));
baloonTip.lineTo(getMeasuredWidth()/2, getMeasuredHeight());
baloonTip.lineTo(3*(getMeasuredWidth()/4), 2*(getMeasuredHeight()/3));
canvas.drawPath(baloonTip, panelPaint);
}

super.dispatchDraw(canvas);
}

public void setUpTapFor(final Activity activity, String title, String snippet)
{
setVisibility(View.VISIBLE);
TextView titleText = (TextView) findViewById(R.id.detailText);
titleText.setText(snippet);

ImageButton detailButton = (ImageButton) findViewById(R.id.detailGo);
try {
final Long id = Long.valueOf(title);
detailButton.setOnClickListener(new View.OnClickListener() {
public void onClick(View view) {
Intent intent = new Intent(activity, ListingActivity.class);
intent.putExtra(ListingActivity.KEY_ID, id);
activity.startActivityForResult(intent, 0);
//not pretty!
if (activity instanceof ITRMapActivity)
((ITRMapActivity)activity).editedListingID=id;
}
});
}
catch (NumberFormatException ex) {
ex.printStackTrace();
}

public static BalloonLayout GetBalloonFor(Activity activity) {
LayoutInflater layoutInflater = (LayoutInflater) activity.getSystemService(Context.LAYOUT_INFLATER_SERVICE);
final BalloonLayout balloon = (BalloonLayout) layoutInflater.inflate(R.layout.map_balloon, null);
RelativeLayout.LayoutParams layoutParams = new RelativeLayout.LayoutParams(200,100);
layoutParams.addRule(RelativeLayout.CENTER_VERTICAL);
layoutParams.addRule(RelativeLayout.CENTER_HORIZONTAL);
balloon.setLayoutParams(layoutParams);

ImageButton closeButton = (ImageButton) balloon.findViewById(R.id.detailClose);
closeButton.setOnClickListener(new View.OnClickListener() {
public void onClick(View view) {
balloon.setVisibility(View.GONE);
}
});
return balloon;
}
}


although I have to work the kinks out of placing the balloon on the OSM map still. As for the ListingsOverlay,


public class ListingsOverlay extends ItemizedOverlay<ListingOverlayItem> {
private MapActivity activity;
private ITRGMapView mapView;
private BalloonLayout balloon;
private ArrayList<ListingOverlayItem> items=new ArrayList<ListingOverlayItem>();
public ArrayList<ListingOverlayItem> getItems() { return items; }

public ListingsOverlay(MapActivity anActivity, ITRGMapView aMapView, android.graphics.drawable.Drawable defaultMarker) {
super(defaultMarker);
activity=anActivity;
mapView=aMapView;
balloon = BalloonLayout.GetBalloonFor(activity);
}

public void addItem(ListingOverlayItem item, Drawable marker) {
if (item!=null && marker!=null && !items.contains(item)) {
super.boundCenterBottom(marker);
item.setMarker(marker);
items.add(item);
}
}

public void doPopulate() {
populate();
setLastFocusedIndex(-1);
}

@Override
protected ListingOverlayItem createItem(int i) {
return(items.get(i));
}

@Override
public int size() {
return(items.size());
}

@Override
protected boolean onTap(int i) {
mapView.removeView(balloon);
balloon.enablePointer();
ListingOverlayItem item = items.get(i);
balloon.setUpTapFor(activity, item.getTitle(), item.getSnippet());
MapView.LayoutParams params = new MapView.LayoutParams(200, 100, item.getPoint(), MapView.LayoutParams.BOTTOM_CENTER);
mapView.addView(balloon, params);
return true;
}

}



I hereby BSD 2.0-release all the above code, if anyone wants to use it - enjoy!

Labels: , , , , ,


Saturday, November 20, 2010

 

HttpHelper for all, redux

Yeah, I know I said I wouldn't be posting much, but I've cleaned up and updated my HttpHelper class for one of my paid projects so that it supports cookies, form/multipart file uploads, non-ASCII characters, etc., so I thought I'd release it into the wild for y'all.

Apache License 2.0, as before.

The interface should be pretty self-evident from the .h file:


//
// HttpHelper.h
//
// Created by Jon Evans on 04/09/09.
// Copyright 2009 Jon Evans. Apache 2.0 license.
//

#import <Foundation/Foundation.h>

@interface HttpHelper : NSObject {
}

//Convenience methods
+(NSURLRequestCachePolicy)getCachePolicyFor:(NSString *)urlString;

//Cookie handling
+(void) useCookies:(BOOL)use;
+(NSArray*) latestCookies;
+(void) getCookiesFrom:(NSHTTPURLResponse*)response forURL:(NSURL*)url;

//HTTP Posts
+(NSURLRequest*) buildPostRequestForURL:(NSString *)urlString withData:(NSDictionary*) data;
+(NSURLRequest *) buildRequestWithPostKeys:(NSArray *) postKeys postValues:(NSArray *) postValues urlString:(NSString *)urlString;
+(BOOL) doPost:(NSURLRequest *)request forCaller:(id)caller onSuccess:(SEL)successSelector onFailure:(SEL)failSelector;

//HTTP Gets
+(NSURLRequest *)getURLRequestFor:(NSString *)urlString;
+(BOOL) doGet:(NSString *)urlString forCaller:(id)caller onSuccess:(SEL)successSelector onFailure:(SEL)failSelector;

//Synchronized accessors
+(NSString *) doSynchronizedGet:(NSString *)urlString;
+(NSData *) doSynchronizedDataGet:(NSString *)urlString;
+(NSString *) doSynchronizedPostTo:(NSString *)urlString withRequest:(NSURLRequest*)request;
+(NSString *) doSynchronizedPostTo:(NSString *)urlString withData:(NSDictionary*)data;
+(NSString *) doSynchronizedPostTo:(NSString *)urlString withKeys:(NSArray*)postKeys andValues:(NSArray*)postValues;

//Form/multipart (for file upload)
+(NSString *) doSynchronizedMultipartPostTo:(NSString*)urlString withData:(NSDictionary*)data fileKeys:(NSArray*)fileKeys;
+(NSString *) doSynchronizedMultipartPostTo:(NSString*)urlString withKeys:(NSArray*)keys andValues:(NSArray*)values fileKeys:(NSArray*)fileKeys;

@end


...and here's where the magic happens:


//
// HttpHelper.m
//
// Created by Jon Evans on 04/09/09.
// Copyright 2009 Jon Evans. Apache 2.0 license.
//

#import "HttpHelper.h"


@implementation HttpHelper

static HttpHelper *singleton=nil;
static NSArray *latestCookies=nil;
static BOOL useCookies=NO;


#pragma mark -
#pragma mark Singleton methods

+(HttpHelper *) getInstance {
if (singleton==nil)
singleton = [[[HttpHelper alloc] init] autorelease];
return singleton;
}

+(id)allocWithZone:(NSZone *)zone {
if (singleton == nil) {
singleton = [super allocWithZone:zone];
return singleton;
}
return nil;
}

-(id)copyWithZone:(NSZone *)zone {
return self;
}

-(id)retain {
return self;
}

-(NSUInteger)retainCount {
return UINT_MAX;
}

-(void)release {
//pass
}

-(id)autorelease {
return self;
}

- (void)dealloc {
[super dealloc];
}


#pragma mark -
#pragma mark Convenience methods

+(NSURLRequestCachePolicy)getCachePolicyFor:(NSString *)urlString {
//Hived out to a separate method because we might fine-tune this later.
return NSURLRequestUseProtocolCachePolicy;
// return NSURLRequestReloadIgnoringCacheData;
}


#pragma mark -
#pragma mark Cookie handling

+(void) useCookies:(BOOL)use
{
useCookies=use;
}

+(void) getCookiesFrom:(NSHTTPURLResponse*)response forURL:(NSURL*)url {
if ([response class]==[NSHTTPURLResponse class])
latestCookies = [NSHTTPCookie cookiesWithResponseHeaderFields:[response allHeaderFields] forURL:url];
}

+(NSArray*) latestCookies {
return latestCookies;
}



#pragma mark -
#pragma mark HTTP Posts

+(NSURLRequest*) buildPostRequestForURL:(NSString *)urlString withData:(NSDictionary*) data
{
NSArray *keys = [data allKeys];
NSMutableArray *values = [NSMutableArray arrayWithCapacity:[keys count]];
for (NSString *key in keys) {
NSString *value = [data objectForKey:key];
[values addObject:(value ? [NSString stringWithFormat:@"%@",value] : @"")];
}
return [self buildRequestWithPostKeys:keys postValues:values urlString:urlString];
}

+(NSURLRequest*) buildRequestWithPostKeys:(NSArray *) postKeys postValues:(NSArray *) postValues urlString:(NSString *)urlString {
NSURL * url = [NSURL URLWithString:urlString];
NSURLRequestCachePolicy policy = NSURLRequestReloadIgnoringCacheData; // we never want a cache of a post response
NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:url cachePolicy:policy timeoutInterval:30.0];
[request setHTTPMethod:@"POST"];

if (useCookies && latestCookies!=nil && [latestCookies count]>0) {
NSDictionary * headers = [NSHTTPCookie requestHeaderFieldsWithCookies:latestCookies];
[request setAllHTTPHeaderFields:headers];
}

NSMutableData *paramData = [NSMutableData dataWithCapacity:8192];

NSMutableString *params=[[NSMutableString alloc] initWithCapacity:1024];
for (int i=0; i<[postValues count]; i++) {
NSString *param = i>0 ? @"&" : @"";
param = [param stringByAppendingFormat:@"%@=",[postKeys objectAtIndex:i]];
[paramData appendData:[param dataUsingEncoding:NSUTF8StringEncoding]];

NSObject* valueToPost = [postValues objectAtIndex:i];
NSString *stringToPost = [valueToPost class]==[NSString class] ? (NSString*)valueToPost : [NSString stringWithFormat:@"%@",valueToPost];
stringToPost = [stringToPost stringByReplacingOccurrencesOfString:@"&" withString:@"%26"];
[paramData appendData:[stringToPost dataUsingEncoding:NSUTF8StringEncoding]];
}

NSString *msgLength = [NSString stringWithFormat:@"%d", [paramData length]];
[request addValue: msgLength forHTTPHeaderField:@"Content-Length"];
[request setHTTPBody: paramData];
[params release];

return request;
}

+(BOOL) doPost:(NSURLRequest *)request forCaller:(id)caller onSuccess:(SEL)onSuccess onFailure:(SEL)onFailure {

NSArray *keys = [NSArray arrayWithObjects:@"request", @"caller", @"onSuccess", @"onFailure", nil];
NSArray *values = [NSArray arrayWithObjects:request, caller,
[NSValue valueWithBytes:&onSuccess objCType:@encode(SEL)],
[NSValue valueWithBytes:&onFailure objCType:@encode(SEL)],
nil];
NSDictionary *args = [NSDictionary dictionaryWithObjects:values forKeys: keys];
[NSThread detachNewThreadSelector:@selector(doHttp:) toTarget:[self getInstance] withObject:args];
return TRUE;
}


#pragma mark -
#pragma mark HTTP Gets

+(NSURLRequest *)getURLRequestFor:(NSString *)urlString {
NSURL *url = [NSURL URLWithString:urlString];
NSURLRequestCachePolicy policy = [self getCachePolicyFor:urlString];
NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:url cachePolicy:policy timeoutInterval:30.0];
[request setHTTPMethod:@"GET"];

if (useCookies && latestCookies!=nil && [latestCookies count]>0) {
NSDictionary * headers = [NSHTTPCookie requestHeaderFieldsWithCookies:latestCookies];
[request setAllHTTPHeaderFields:headers];
}
return request;
}

+(BOOL) doGet:(NSString *)urlString forCaller:(id)caller onSuccess:(SEL)onSuccess onFailure:(SEL)onFailure {
NSURLRequest *request = [self getURLRequestFor:urlString];
NSArray *keys = [NSArray arrayWithObjects:@"request", @"caller", @"onSuccess", @"onFailure", nil];
NSArray *values = [NSArray arrayWithObjects:request, caller,
[NSValue valueWithBytes:&onSuccess objCType:@encode(SEL)],
[NSValue valueWithBytes:&onFailure objCType:@encode(SEL)],
nil];
NSDictionary *args = [NSDictionary dictionaryWithObjects:values forKeys: keys];
NSThread* uploadThread = [[NSThread alloc] initWithTarget:[self getInstance] selector:@selector(doHttp:) object:args];
[uploadThread start];
[uploadThread release];
return TRUE;
}


#pragma mark -
#pragma mark Internals

-(BOOL) doHttp:(NSDictionary *)args
{
@synchronized (self) {
//autorelease pool
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];

NSURLRequest *request = [args objectForKey:@"request"];
NSObject *caller = [args objectForKey:@"caller"];

SEL onSuccess;
[[args objectForKey:@"onSuccess"] getValue:&onSuccess];
SEL onFailure;
[[args objectForKey:@"onFailure"] getValue:&onFailure];

NSError *error=nil;
NSHTTPURLResponse *response=nil;
NSData *returnData = [NSURLConnection sendSynchronousRequest: request returningResponse: &response error: &error ];
NSString * responseString = [[[NSString alloc] initWithData: returnData encoding: NSUTF8StringEncoding] autorelease];

NSString *errorString=nil;

if (error)
errorString = [NSString stringWithFormat:@"Error %@ '%@': %@",NSLocalizedString(@"loading page",nil), [request URL], [error localizedDescription]];
else if (response==nil || returnData==nil)
errorString = [NSString stringWithFormat:@"Error - %@ %@",NSLocalizedString(@"No response from URL",nil), [request URL]];
else if (response.statusCode!=200)
errorString = [NSString stringWithFormat:@"Error - %@ %i %@ %@",NSLocalizedString(@"HTTP Error",nil), response.statusCode, NSLocalizedString(@"from URL",nil), [request URL]];

if (errorString && [errorString length]>0) {
NSLog(@"iTravel Error ",errorString);
[caller performSelectorOnMainThread: onFailure withObject:errorString waitUntilDone:NO];
}
else {
if (useCookies)
[HttpHelper getCookiesFrom:response forURL:[request URL]];
[caller performSelectorOnMainThread: onSuccess withObject:responseString waitUntilDone:YES];
}
[pool release];
}
return TRUE;
}


#pragma mark -
#pragma mark Synchronized methods

+(NSString *) doSynchronizedGet:(NSString *)urlString {
NSData *returnData = [HttpHelper doSynchronizedDataGet:urlString];
if (!returnData)
return [NSString stringWithFormat:@"Error: %@ %@",NSLocalizedString(@"No data received from server",nil), urlString];

NSString * responseString = [[[NSString alloc] initWithData: returnData encoding: NSUTF8StringEncoding] autorelease];
return responseString;
}

+(NSData *) doSynchronizedDataGet:(NSString *)urlString {
NSURLRequest *request = [self getURLRequestFor:urlString];
NSHTTPURLResponse *response=nil;
NSError *error=nil;
NSData *returnData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];

NSString *errorString=@"";
if (error)
errorString = [NSString stringWithFormat:@"Error %@ '%@': %@",NSLocalizedString(@"loading page",nil), urlString, [error localizedDescription]];
else if (response==nil || returnData==nil)
errorString = [NSString stringWithFormat:@"Error - %@ %@",NSLocalizedString(@"No response from URL",nil), urlString];
else if (response.statusCode!=200)
errorString = [NSString stringWithFormat:@"Error - %@ %i %@ %@",NSLocalizedString(@"HTTP Error",nil), response.statusCode, NSLocalizedString(@"from URL",nil), urlString];

if ([errorString length]>0)
return [errorString dataUsingEncoding:NSUTF8StringEncoding];;

if (useCookies)
[HttpHelper getCookiesFrom:response forURL:[request URL]];

return returnData;
}

+(NSString *) doSynchronizedPostTo:(NSString *)urlString withData:(NSDictionary*)data {
NSURLRequest *request = [self buildPostRequestForURL:urlString withData:data];
return [self doSynchronizedPostTo:urlString withRequest:request];
}
+(NSString *) doSynchronizedPostTo:(NSString *)urlString withKeys:(NSArray*)postKeys andValues:(NSArray*)postValues {
NSURLRequest *request = [self buildRequestWithPostKeys:postKeys postValues:postValues urlString:urlString];
return [self doSynchronizedPostTo:urlString withRequest:request];
}
+(NSString *) doSynchronizedPostTo:(NSString *)urlString withRequest:(NSURLRequest*)request {
NSHTTPURLResponse *response=nil;
NSError *error=nil;
NSData *returnData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];

NSString *errorString=@"";
if (error)
errorString = [NSString stringWithFormat:@"Error - %@ '%@': %@",NSLocalizedString(@"HTTP Error posting to URL",nil), urlString, [error localizedDescription]];
else if (response==nil || returnData==nil)
errorString = [NSString stringWithFormat:@"Error - %@ %@",NSLocalizedString(@"No response from URL",nil), urlString];
else if (response.statusCode!=200)
errorString = [NSString stringWithFormat:@"Error - %@ %i %@ %@",NSLocalizedString(@"HTTP Error",nil), response.statusCode, NSLocalizedString(@"from URL",nil), urlString];

if ([errorString length]>0)
return errorString;

if (useCookies)
[HttpHelper getCookiesFrom:response forURL:[request URL]];

NSString * responseString = [[[NSString alloc] initWithData: returnData encoding: NSUTF8StringEncoding] autorelease];

if ([[responseString lowercaseString] hasPrefix:@"error"])
errorString = [NSString stringWithFormat:@"Error %@ %@ %@ %@",NSLocalizedString(@"message",nil), responseString, NSLocalizedString(@"from URL",nil), urlString];
else if ([[responseString lowercaseString] hasPrefix:@"fail"])
errorString = [NSString stringWithFormat:@"Error - %@ '%@' %@ %@",NSLocalizedString(@"Failure message",nil), responseString, NSLocalizedString(@"when posting to URL",nil), urlString];

if ([errorString length]>0)
return errorString;

return responseString;
}

+(NSString*) doSynchronizedMultipartPostTo:(NSString*)urlString withData:(NSDictionary*)data fileKeys:(NSArray*)fileKeys
{
NSArray *keys = [data allKeys];
NSMutableArray *values = [NSMutableArray arrayWithCapacity:[keys count]];
for (NSString *key in keys) {
NSString *value = [data objectForKey:key];
[values addObject:(value ? value : @"")];
}
return [self doSynchronizedMultipartPostTo:urlString withKeys:keys andValues:values fileKeys:fileKeys];
}

+(NSString *) doSynchronizedMultipartPostTo:(NSString*)urlString withKeys:(NSArray*)keys andValues:(NSArray*)values fileKeys:(NSArray*)fileKeys
{
NSURL * url = [NSURL URLWithString:urlString];
NSURLRequestCachePolicy policy = NSURLRequestReloadIgnoringCacheData; // we never want a cache of a post response
NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:url cachePolicy:policy timeoutInterval:30.0];
[request setHTTPMethod:@"POST"];

NSString *boundary = [[NSProcessInfo processInfo] globallyUniqueString];
[request setValue: [NSString stringWithFormat:@"multipart/form-data; boundary=%@", boundary] forHTTPHeaderField:@"Content-Type"];

NSMutableData *postBody = [NSMutableData dataWithCapacity:65536];
for (int i=0; i<[keys count]; i++) {
NSString *key = [keys objectAtIndex:i];
NSObject *value = [values objectAtIndex:i];

if (value && fileKeys && [fileKeys containsObject:key]) {
[postBody appendData: [[NSString stringWithFormat:@"--%@\r\n", boundary] dataUsingEncoding:NSUTF8StringEncoding]];
if ([value isKindOfClass:[NSString class]]) {
NSString *valueString = (NSString*)value;
if ([valueString length]>0) {
NSRange slashRange = [valueString rangeOfString:@"/" options:NSBackwardsSearch];
NSString *fileName = slashRange.location!=NSNotFound && ![valueString hasSuffix:@"/"]
? [valueString substringFromIndex:slashRange.location+1]
: valueString;
[postBody appendData: [[NSString stringWithFormat: @"Content-Disposition: form-data; name=\"%@\"; filename=\"%@\"\r\n", key, fileName] dataUsingEncoding:NSUTF8StringEncoding]];
[postBody appendData: [[NSString stringWithFormat: @"Content-Type: application/octet-stream\r\n\r\n"] dataUsingEncoding:NSUTF8StringEncoding]];
[postBody appendData: [NSData dataWithContentsOfMappedFile:valueString]];
}
}
else if ([value isKindOfClass:[NSURL class]]) {
NSURL *fileURL = (NSURL*)value;
[postBody appendData: [[NSString stringWithFormat: @"Content-Disposition: form-data; name=\"%@\"; filename=\"%@\"\r\n", key, [fileURL lastPathComponent]] dataUsingEncoding:NSUTF8StringEncoding]];
[postBody appendData: [[NSString stringWithFormat: @"Content-Type: application/octet-stream\r\n\r\n"] dataUsingEncoding:NSUTF8StringEncoding]];
[postBody appendData: [NSData dataWithContentsOfURL:fileURL]];
}
else if ([value isKindOfClass:[NSData class]]) {
[postBody appendData: [[NSString stringWithFormat: @"Content-Disposition: form-data; name=\"%@\"; filename=\"%@\"\r\n", key, key] dataUsingEncoding:NSUTF8StringEncoding]];
[postBody appendData: [[NSString stringWithFormat: @"Content-Type: application/octet-stream\r\n\r\n"] dataUsingEncoding:NSUTF8StringEncoding]];
[postBody appendData: (NSData*)value];
}
}
else {
[postBody appendData:[[NSString stringWithFormat: @"--%@\r\n", boundary] dataUsingEncoding:NSUTF8StringEncoding]];
[postBody appendData:[[NSString stringWithFormat: @"Content-Disposition: form-data; name=\"%@\"\r\n",key] dataUsingEncoding:NSUTF8StringEncoding]];
[postBody appendData: [[NSString stringWithFormat: @"Content-Type: text/plain;charset=utf-8\r\n\r\n"] dataUsingEncoding:NSUTF8StringEncoding]];
NSString *valueString = [NSString stringWithFormat: @"%@", value];
[postBody appendData:[valueString dataUsingEncoding:NSUTF8StringEncoding]];
}
[postBody appendData:[@"\r\n" dataUsingEncoding:NSUTF8StringEncoding]];
}
[postBody appendData:[[NSString stringWithFormat:@"--%@--\r\n", boundary] dataUsingEncoding:NSUTF8StringEncoding]];

NSString *msgLength = [NSString stringWithFormat:@"%d", [postBody length]];
[request addValue: msgLength forHTTPHeaderField:@"Content-Length"];
[request setHTTPBody:postBody];

NSHTTPURLResponse *response=nil;
NSError *error=nil;
NSData *returnData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];

NSString *errorString=@"";
if (error)
errorString = [NSString stringWithFormat:@"Error - %@ '%@': %@",NSLocalizedString(@"HTTP Error posting to URL",nil), urlString, [error localizedDescription]];
else if (response==nil || returnData==nil)
errorString = [NSString stringWithFormat:@"Error - %@ %@",NSLocalizedString(@"No response from URL",nil), urlString];
else if (response.statusCode!=200)
errorString = [NSString stringWithFormat:@"Error - %@ %i %@ %@",NSLocalizedString(@"HTTP Error",nil), response.statusCode, NSLocalizedString(@"from URL",nil), urlString];

if ([errorString length]>0)
return errorString;

NSString * responseString = [[[NSString alloc] initWithData: returnData encoding: NSUTF8StringEncoding] autorelease];

if ([[responseString lowercaseString] hasPrefix:@"error"])
errorString = [NSString stringWithFormat:@"Error %@ %@ %@ %@",NSLocalizedString(@"message",nil), responseString, NSLocalizedString(@"from URL",nil), urlString];
else if ([[responseString lowercaseString] hasPrefix:@"fail"])
errorString = [NSString stringWithFormat:@"Error - %@ '%@' %@ %@",NSLocalizedString(@"Failure message",nil), responseString, NSLocalizedString(@"when posting to URL",nil), urlString];

if ([errorString length]>0)
return errorString;

if (useCookies)
[HttpHelper getCookiesFrom:response forURL:[request URL]];
return responseString;
}


@end

Friday, November 5, 2010

 

Pronoid Android sleeps

As you may have noticed, if your eyes for Nothing are as keen as Alice's, there hasn't been a whole lot of activity 'round here of late; that's because I've been up to my nose in paying app work, most of which I can't write about in public for obvious non-disclosure-agreement reasons. But I hope to find some pet-project time come the new year, so check back in then -

Jon

Sunday, September 19, 2010

 

Of Zips and the Internet

So of late I've been working on adding the ability to download a complete snapshot of Wikitravel to my iTravel app. And I've finally got it working, for both Android and iPhone - but there was many a glitch along the way.

First off: downloading. Now that part's actually pretty easy. You know what's tricky? Working out whether you can download. Both Android and iPhone bury their "hey, am I connected to the Internet, and if so, how?" API facilities deep in their documentation.

To be fair, Android isn't so bad:

public static boolean CheckInternetConnection(Context context) {
if (context==null)
return false;
ConnectivityManager connec = (ConnectivityManager) context.getSystemService(Context.CONNECTIVITY_SERVICE);
return ( connec.getNetworkInfo(ConnectivityManager.TYPE_MOBILE).isConnectedOrConnecting() ||
connec.getNetworkInfo(ConnectivityManager.TYPE_WIFI).isConnectedOrConnecting() );
}


...straightforward enough, just not as well-documented as it could be.

The iPhone, though. Well. Sheesh. The way you actually do it is through seriously byzantine and messy C code. Because most people don't want to have to deal with that, Apple provides a reference implementation - but do they tell you about this anywhere in their documentation? No, they do not. Thank heavens for Stack Overflow, which pointed me to Reachability.

But wait, there's less: the old version has a very simple interface:

+(BOOL) serverReachable {
Reachability *reachability = [Reachability reachabilityWithHostName:@"http://itravelapp.appspot.com/"];
return [reachability currentReachabilityStatus]!=NotReachable;
}


but the new version requires you to set up a whole asynchronous notification framework. OK, so it probably works better, and it's not too onerous:


-(void) initiateReachability {
// check for internet connection
[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(checkNetworkStatus:) name:kReachabilityChangedNotification object:nil];

internetReachable = [[Reachability reachabilityForInternetConnection] retain];
[internetReachable startNotifier];

// check if a pathway to my server exists
serverReachable = [[Reachability reachabilityWithHostName: @"itravelapp.appspot.com"] retain];
[serverReachable startNotifier];
}

+(void) initiateReachability {
[[self getInstance] initiateReachability];
}

- (void) checkNetworkStatus:(NSNotification *)notice
{
// called after network status changes

NetworkStatus internetStatus = [internetReachable currentReachabilityStatus];
internetActive = internetStatus != NotReachable;

NetworkStatus serverStatus = [serverReachable currentReachabilityStatus];
serverActive = serverStatus != NotReachable;
}

-(BOOL) serverReachable {
return serverActive;
}

+(BOOL) serverReachable {
return [[self getInstance] serverReachable];
}


But still, compare and contrast to the Android version above. Oh, Apple. Whatever were you thinking?

OK. So you've got the Internet. You've downloaded your zip files. (Yes, files, plural; one of my early problems is that both Androids and iPhones choke and die on 100MB zip files, due to their memory constraints. But five 20MB zip files, handled consecutively, no problem.)

Now how do you open them?

In Android, it's easy enough: java.util.zip.ZipFile and java.util.zip.ZipEntry. (In fact, it's easy - and fast - enough that I don't even unpack those files; I just access data from within them on the fly.) On the iPhone, though, although the libz.1.2.3.dylib zlib framework is provided, actually working with it requires some serious low-level C chops, and mine are rusty beyond belief.

But that's OK! Because others' aren't, and we live in an open-source world. I just went out and downloaded Karl Mostowski's truly excellent ZipKit framework, and used it. I had some trouble using it as a static library... so I gave up and just included his source in my project. And it worked like a charm. A diamond-studded, streamlined, memory-miserly charm. Thanks, Karl!

Here's an example of it in action:

-(void) doUnpacking {
[Util showActivity];
[self startProgressView];
self.title=@"Unpacking ...";
NSThread* unpackThread = [[NSThread alloc] initWithTarget:self selector:@selector(launchUnpacking) object:nil];
[unpackThread start];
[unpackThread release];
}

-(void) launchUnpacking {
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];

int zipFiles = [[Settings getBulkZipFiles] intValue];
if (zipFiles > 0)
[Settings setBulkUnpackedFiles:0];

for (int i=0; i < zipFiles; i++) {
NSString *titleString = [NSString stringWithFormat:@"Unpacking %d of %d", i+1, zipFiles];
[self performSelectorOnMainThread:@selector(setTitle:) withObject:titleString waitUntilDone:YES];
NSAutoreleasePool *pool2 = [[NSAutoreleasePool alloc] init];
NSString *fileName = [NSString stringWithFormat:@"/iTravelDump%d.zip",i];
NSString *dumpPath = [[Util getBulkDownloadPath] stringByAppendingPathComponent:fileName];
ZKFileArchive *archive = [ZKFileArchive archiveWithArchivePath:dumpPath];
archive.delegate = self;
totalFiles = [archive.centralDirectory count];
currentFile=0;
[archive inflateToDirectory:[[Util getBulkDownloadPath] stringByAppendingPathComponent:@"temp"] usingResourceFork:NO];
archive=nil;
[pool2 release];
[Settings setBulkUnpackedFiles:[Settings getBulkUnpackedFiles]+totalFiles];
}

//done!
[self performSelectorOnMainThread:@selector(unpackingFinished) withObject:nil waitUntilDone:NO];
[pool release];
}

//ZKArchive delegate
- (void) onZKArchive:(ZKArchive *) archive willUnzipPath:(NSString *) path {
if (progressView) {
float currentLength = (float)++currentFile;
float totalSize = (float)totalFiles;
float fraction = (float)currentLength/totalSize;
[progressView setProgress:fraction];
}
}

-(void)unpackingFinished {
[Util stopShowingActivity];
[self stopProgressView];
[Settings setLastBulkUnpack:[NSDate date]];
[Settings save];
self.title=@"Unpacking Complete";
[self loadHTML];
}

Labels: , , , , ,


Thursday, August 12, 2010

 

App Localizer 2.0, now with more iPhone

Quick update: the new App Localizer now handles both Android and iPhone strings files.

Labels: , , , , , , ,


This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]