Monday, November 30, 2009
How to store images larger than 1 megabyte in Google App Engine
Which is kind of surprising, because it's not that hard to break files up into chunks and store those chunks in the database separately. Here's what I did today for my current project, which stores data - including photos - uploaded from smartphones:
First, we have to receive the uploaded image. Our uploads are two-phase - first data, then a photo - for various reasons. The data upload includes the image's file name; the photo upload is a basic form/multipart POST with exactly one argument (the filename) and its value (the file).
So, in "main.py":
for arg in self.request.arguments():
file = self.request.get(arg)
response = entryHandler.saveImage(arg,file)
and in "ec.py":
entryRef = db.ReferenceProperty(Entry)
chunkIndex = db.IntegerProperty()
chunk = db.BlobProperty()
def saveImage(self, fileName, file):
results = Entry.all().filter("photoPath =", fileName).fetch(1)
logging.warning("Error - could not find the entry associated with image name "+fileName)
entry = results
chunk = ImageChunk(entryRef=entry, chunkIndex=marker, chunk=db.Blob(file[MaxBTSize*marker:]))
chunk = ImageChunk(entryRef=entry, chunkIndex=marker, chunk=db.Blob(file[MaxBTSize*marker:MaxBTSize*(marker+1)]))
logging.info("Successfully received image "+fileName)
return "Successfully received image "+fileName
Pretty basic stuff: we chop the image up at each 1,000,000-byte mark, and put each chunk into its own ImageChunk DB object.
Then, when we need to retrieve the image, in 'main.py':
key = self.request.get('entryKey')
entryHandler = ec.EntryHandler()
image = entryHandler.getImageByEntryKey(key)
if image is not None:
self.response.headers['Content-Type'] = 'image/jpeg'
and in 'ec.py':
def getImageByEntryKey(self, key):
chunks = db.GqlQuery("SELECT * FROM ImageChunk WHERE entryRef = :1 ORDER BY chunkIndex", key).fetch(100)
for chunkRow in chunks:
Since db.Blob is a subtype of str, that's all you have to do. I don't understand why some people are so upset about this: it's mildly annoying that I had to write the above, but hardly crippling. At least with JPEGs, which is what we use. (But I don't see why any other file type would be more difficult; they're ultimately all just a bunch of bytes). Could hardly be easier ... well, until App Engine rolls out their large file service.
(eta, Dec 14: which came out today! Meaning you can now disregard all the above and just use the new Blobstore instead.)
(eta, Dec 16: mmm, maybe not. Looked at the Blobstore in detail today, and it's really best suited for browser projects, not app or web-service stuff. The API for the blobs is very limited, and you can only access them via one-time-only URLs that App Engine puts in your HTML. You could scrape that, granted, but that's a pain in the ass, no less inelegant than the image-chunking solution above. It's experimental and subject to change, too. I think I'll hold out until its API improves.)
But how do you make thumbs of 1MB+ files? The Image class can only take 1MB files and blobstore values?
I really like your implementation, but can't be loading 5MB images in a gallery everytime I want to display some thumbs.
Any help would be greatly appreciated.
Subscribe to Post Comments [Atom]
Subscribe to Posts [Atom]