Wednesday, September 21, 2011
Image sizing and scraping with JQuery and Rails
I know, I know, long time no type. Not my fault, I swear! First I went travelling, then I had a buncha work to do, then I went to San Francisco TechCrunch Disrupt, all of which occupied my time. Well, I suppose, on reflection, all those things were my fault, but, um ... look, I'm back now, OK?
Back with a li'l discussion of image sizing and scraping with Rails and JQuery, for a project I can't really talk about yet. Suffice to say that sometimes I want to calculate the size of various remote images - quite a lot of them, in fact, making performance important - and sometimes I want to scrape all the images from a set of web pages.
I thought the first task was going to be tricky. And maybe it is. But fortunately, someone has solved it for me, via the totally awesome FastImage Ruby gem, for which praise should be heaped upon one sdsykes. It works exactly like advertised, which is to say, like this:
def self.picture_info_for(url)
return nil if url.blank?
begin
size = FastImage.size(url)
return size #[width, height]
rescue
logger.info "Error getting info for picture at "+url.to_s
return Array[0,0] #this makes sense for my app, but maybe not yours
end
end
So, hurrah! This meant the scraping bit was actually tricker. Sure, I could have done it all in Rails, but it's user-facing, and I didn't want the user to have to wait for a bunch of potentially sequential http requests to complete without seeing any results. I could have done it all on the client side, but parsing HTML with Javascript, even with JQuery, sounded painful and fraught with difficulties, compared to using the dead-easy Hpricot gem. So I came up with a compromise I quite like:
1. On the client side: (written using HAML, which I mostly adore)
1. On the client side: (written using HAML, which I mostly adore)
- @image_urls.each_with_index do |url,idx|
= link_to url, url
%div{:id => 'page_'+idx.to_s}
%script
$(function() {
- @image_urls.each_with_index do |url,idx|
$.ajax({
url: '/stories/scrape_images?url=#{CGI::escape(url)}',
success: function(msg){ $('#page_#{idx}').html(msg); },
error: function(msg){ $('#page_#idx}').html(msg); }
});
});
2. On the server, to first create and then respond to that client page:
def popup_scraped_images
start_time = Time.now
@image_urls = []
@seed = Seed.find(params[:seed_id])
@seed.active_signals.each do |signal|
next if signal.main_url.blank?
@image_urls << signal.main_url
end
end
def scrape_images
url = params[:url]
slash = url =~ /[^\/]\/[^\/]/
host = slash.nil? ? "" : url[0,slash+1]
html = ''
page = HTTParty.get(url, :timeout => 5)
Hpricot(page).search("//img").each do |element|
img_src = element.attributes["src"]
img_src = host+img_src if img_src.match(/^\//)
html += '<img src="'+img_src+'" />' if img_src.match(/^http/)
end
end
Hopefully how they all interact is self-explanatory. Et voila - semi-asynchronous Rails/JQuery image scraping, handled on the server side for easy caching if need be later on.
Labels: FastImage, Hpricot, Images, JQuery, Rails, Ruby, scaling, sizing
Monday, November 30, 2009
How to store images larger than 1 megabyte in Google App Engine
Over the summer, Google App Engine raised its limits for web requests and responses from 1MB to 10MB, but kept the maximum size of any single database element at 1MB. If you try to exceed this, you'll get a MemoryError. You can find a fair amount of grief and woe and gnashing of teeth and wearing of sackcloth and ashes about this online.
Which is kind of surprising, because it's not that hard to break files up into chunks and store those chunks in the database separately. Here's what I did today for my current project, which stores data - including photos - uploaded from smartphones:
First, we have to receive the uploaded image. Our uploads are two-phase - first data, then a photo - for various reasons. The data upload includes the image's file name; the photo upload is a basic form/multipart POST with exactly one argument (the filename) and its value (the file).
So, in "main.py":
and in "ec.py":
Pretty basic stuff: we chop the image up at each 1,000,000-byte mark, and put each chunk into its own ImageChunk DB object.
Then, when we need to retrieve the image, in 'main.py':
and in 'ec.py':
Since db.Blob is a subtype of str, that's all you have to do. I don't understand why some people are so upset about this: it's mildly annoying that I had to write the above, but hardly crippling. At least with JPEGs, which is what we use. (But I don't see why any other file type would be more difficult; they're ultimately all just a bunch of bytes). Could hardly be easier ... well, until App Engine rolls out their large file service.
(eta, Dec 14: which came out today! Meaning you can now disregard all the above and just use the new Blobstore instead.)
(eta, Dec 16: mmm, maybe not. Looked at the Blobstore in detail today, and it's really best suited for browser projects, not app or web-service stuff. The API for the blobs is very limited, and you can only access them via one-time-only URLs that App Engine puts in your HTML. You could scrape that, granted, but that's a pain in the ass, no less inelegant than the image-chunking solution above. It's experimental and subject to change, too. I think I'll hold out until its API improves.)
Which is kind of surprising, because it's not that hard to break files up into chunks and store those chunks in the database separately. Here's what I did today for my current project, which stores data - including photos - uploaded from smartphones:
First, we have to receive the uploaded image. Our uploads are two-phase - first data, then a photo - for various reasons. The data upload includes the image's file name; the photo upload is a basic form/multipart POST with exactly one argument (the filename) and its value (the file).
So, in "main.py":
class SaveImage(webapp.RequestHandler):
def post(self):
entryHandler=ec.EntryHandler()
for arg in self.request.arguments():
file = self.request.get(arg)
response = entryHandler.saveImage(arg,file)
self.response.out.write(response)
and in "ec.py":
class ImageChunk(db.Model):
entryRef = db.ReferenceProperty(Entry)
chunkIndex = db.IntegerProperty()
chunk = db.BlobProperty()
class EntryHandler:
def saveImage(self, fileName, file):
results = Entry.all().filter("photoPath =", fileName).fetch(1)
if len(results)==0:
logging.warning("Error - could not find the entry associated with image name "+fileName)
return "Failed"
else:
MaxBTSize=1000000
entry = results[0]
marker=0
chunks=[]
while marker*MaxBTSize<len(file):
if MaxBTSize*(marker+1)>len(file):
chunk = ImageChunk(entryRef=entry, chunkIndex=marker, chunk=db.Blob(file[MaxBTSize*marker:]))
else:
chunk = ImageChunk(entryRef=entry, chunkIndex=marker, chunk=db.Blob(file[MaxBTSize*marker:MaxBTSize*(marker+1)]))
chunk.put()
marker+=1
logging.info("Successfully received image "+fileName)
return "Successfully received image "+fileName
Pretty basic stuff: we chop the image up at each 1,000,000-byte mark, and put each chunk into its own ImageChunk DB object.
Then, when we need to retrieve the image, in 'main.py':
class ShowImageWithKey(webapp.RequestHandler):
def get(self):
key = self.request.get('entryKey')
entryHandler = ec.EntryHandler()
image = entryHandler.getImageByEntryKey(key)
if image is not None:
self.response.headers['Content-Type'] = 'image/jpeg'
self.response.out.write(image)
and in 'ec.py':
def getImageByEntryKey(self, key):
chunks = db.GqlQuery("SELECT * FROM ImageChunk WHERE entryRef = :1 ORDER BY chunkIndex", key).fetch(100)
if len(chunks)==0:
return None
image=""
for chunkRow in chunks:
image+=chunkRow.chunk
return image
Since db.Blob is a subtype of str, that's all you have to do. I don't understand why some people are so upset about this: it's mildly annoying that I had to write the above, but hardly crippling. At least with JPEGs, which is what we use. (But I don't see why any other file type would be more difficult; they're ultimately all just a bunch of bytes). Could hardly be easier ... well, until App Engine rolls out their large file service.
(eta, Dec 14: which came out today! Meaning you can now disregard all the above and just use the new Blobstore instead.)
(eta, Dec 16: mmm, maybe not. Looked at the Blobstore in detail today, and it's really best suited for browser projects, not app or web-service stuff. The API for the blobs is very limited, and you can only access them via one-time-only URLs that App Engine puts in your HTML. You could scrape that, granted, but that's a pain in the ass, no less inelegant than the image-chunking solution above. It's experimental and subject to change, too. I think I'll hold out until its API improves.)
Labels: AppEngine, BigTable, chunking, chunks, Images, JPEG, JPG, limit, MemoryError, python, size
Subscribe to Posts [Atom]