Monday, November 30, 2009
How to store images larger than 1 megabyte in Google App Engine
Over the summer, Google App Engine raised its limits for web requests and responses from 1MB to 10MB, but kept the maximum size of any single database element at 1MB. If you try to exceed this, you'll get a MemoryError. You can find a fair amount of grief and woe and gnashing of teeth and wearing of sackcloth and ashes about this online.
Which is kind of surprising, because it's not that hard to break files up into chunks and store those chunks in the database separately. Here's what I did today for my current project, which stores data - including photos - uploaded from smartphones:
First, we have to receive the uploaded image. Our uploads are two-phase - first data, then a photo - for various reasons. The data upload includes the image's file name; the photo upload is a basic form/multipart POST with exactly one argument (the filename) and its value (the file).
So, in "main.py":
and in "ec.py":
Pretty basic stuff: we chop the image up at each 1,000,000-byte mark, and put each chunk into its own ImageChunk DB object.
Then, when we need to retrieve the image, in 'main.py':
and in 'ec.py':
Since db.Blob is a subtype of str, that's all you have to do. I don't understand why some people are so upset about this: it's mildly annoying that I had to write the above, but hardly crippling. At least with JPEGs, which is what we use. (But I don't see why any other file type would be more difficult; they're ultimately all just a bunch of bytes). Could hardly be easier ... well, until App Engine rolls out their large file service.
(eta, Dec 14: which came out today! Meaning you can now disregard all the above and just use the new Blobstore instead.)
(eta, Dec 16: mmm, maybe not. Looked at the Blobstore in detail today, and it's really best suited for browser projects, not app or web-service stuff. The API for the blobs is very limited, and you can only access them via one-time-only URLs that App Engine puts in your HTML. You could scrape that, granted, but that's a pain in the ass, no less inelegant than the image-chunking solution above. It's experimental and subject to change, too. I think I'll hold out until its API improves.)
Which is kind of surprising, because it's not that hard to break files up into chunks and store those chunks in the database separately. Here's what I did today for my current project, which stores data - including photos - uploaded from smartphones:
First, we have to receive the uploaded image. Our uploads are two-phase - first data, then a photo - for various reasons. The data upload includes the image's file name; the photo upload is a basic form/multipart POST with exactly one argument (the filename) and its value (the file).
So, in "main.py":
class SaveImage(webapp.RequestHandler):
def post(self):
entryHandler=ec.EntryHandler()
for arg in self.request.arguments():
file = self.request.get(arg)
response = entryHandler.saveImage(arg,file)
self.response.out.write(response)
and in "ec.py":
class ImageChunk(db.Model):
entryRef = db.ReferenceProperty(Entry)
chunkIndex = db.IntegerProperty()
chunk = db.BlobProperty()
class EntryHandler:
def saveImage(self, fileName, file):
results = Entry.all().filter("photoPath =", fileName).fetch(1)
if len(results)==0:
logging.warning("Error - could not find the entry associated with image name "+fileName)
return "Failed"
else:
MaxBTSize=1000000
entry = results[0]
marker=0
chunks=[]
while marker*MaxBTSize<len(file):
if MaxBTSize*(marker+1)>len(file):
chunk = ImageChunk(entryRef=entry, chunkIndex=marker, chunk=db.Blob(file[MaxBTSize*marker:]))
else:
chunk = ImageChunk(entryRef=entry, chunkIndex=marker, chunk=db.Blob(file[MaxBTSize*marker:MaxBTSize*(marker+1)]))
chunk.put()
marker+=1
logging.info("Successfully received image "+fileName)
return "Successfully received image "+fileName
Pretty basic stuff: we chop the image up at each 1,000,000-byte mark, and put each chunk into its own ImageChunk DB object.
Then, when we need to retrieve the image, in 'main.py':
class ShowImageWithKey(webapp.RequestHandler):
def get(self):
key = self.request.get('entryKey')
entryHandler = ec.EntryHandler()
image = entryHandler.getImageByEntryKey(key)
if image is not None:
self.response.headers['Content-Type'] = 'image/jpeg'
self.response.out.write(image)
and in 'ec.py':
def getImageByEntryKey(self, key):
chunks = db.GqlQuery("SELECT * FROM ImageChunk WHERE entryRef = :1 ORDER BY chunkIndex", key).fetch(100)
if len(chunks)==0:
return None
image=""
for chunkRow in chunks:
image+=chunkRow.chunk
return image
Since db.Blob is a subtype of str, that's all you have to do. I don't understand why some people are so upset about this: it's mildly annoying that I had to write the above, but hardly crippling. At least with JPEGs, which is what we use. (But I don't see why any other file type would be more difficult; they're ultimately all just a bunch of bytes). Could hardly be easier ... well, until App Engine rolls out their large file service.
(eta, Dec 14: which came out today! Meaning you can now disregard all the above and just use the new Blobstore instead.)
(eta, Dec 16: mmm, maybe not. Looked at the Blobstore in detail today, and it's really best suited for browser projects, not app or web-service stuff. The API for the blobs is very limited, and you can only access them via one-time-only URLs that App Engine puts in your HTML. You could scrape that, granted, but that's a pain in the ass, no less inelegant than the image-chunking solution above. It's experimental and subject to change, too. I think I'll hold out until its API improves.)
Labels: AppEngine, BigTable, chunking, chunks, Images, JPEG, JPG, limit, MemoryError, python, size
Comments:
Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!data analytics course in nagpur
I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.<a href="https://360digitmg.com/india/cloud-computing-course-in-jaipur>cloud computing course in jaipur</a>
<< Home
This has been really helpful in getting around an annoying issue with Blobstore.
But how do you make thumbs of 1MB+ files? The Image class can only take 1MB files and blobstore values?
I really like your implementation, but can't be loading 5MB images in a gallery everytime I want to display some thumbs.
Any help would be greatly appreciated.
But how do you make thumbs of 1MB+ files? The Image class can only take 1MB files and blobstore values?
I really like your implementation, but can't be loading 5MB images in a gallery everytime I want to display some thumbs.
Any help would be greatly appreciated.
Thanks for sharing this informative content.,
Turient is an All-in-one platform for all our teaching needs. If Teaching is your passion ,enabling is ours
Read the Informative blog - 11 Free Teaching Tools for Online Teachers
11 Free Teaching Tools for Online Teachers
Free Teaching Tools
Turient is an All-in-one platform for all our teaching needs. If Teaching is your passion ,enabling is ours
Read the Informative blog - 11 Free Teaching Tools for Online Teachers
11 Free Teaching Tools for Online Teachers
Free Teaching Tools
Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!data analytics course in nagpur
I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.<a href="https://360digitmg.com/india/cloud-computing-course-in-jaipur>cloud computing course in jaipur</a>
It is very rare these days to find sites that provide information someone is watching for. I am glad to see that your site share valued information that can help to many readers. nice one and keep writing! Feel free to visit my website; 배트맨토토
This is a great inspiring blog.You have shared really very helpful information thank you.
Data Scientist Course in Amritsar
Data Scientist Course in Amritsar
I’m gone to inform my little brother, that he should also
visit this blog on regular basis to take updated from newest gossip. 메이저사이트
visit this blog on regular basis to take updated from newest gossip. 메이저사이트
Very shortly this site will be famous amid all blogging and site-building people, due to it’s nice articles or reviews. 온라인경마
I savour, lead to I found exactly what I used to be having a look for. You’ve ended my four day long hunt! God Bless you man. Have a nice day. Bye 사설토토
Thank you for sharing blog comments on this site.This site provides the information about larger than 1mbyte store in the google engines.I got the valuable information from this site.keep updates more things.
contract dispute litigation
contract dispute litigation
Storing images larger than 1 megabyte in Google App Engine necessitates a strategic approach, balancing efficient data management with system limitations. By employing Cloud Storage, developers can efficiently handle larger image files, leveraging its scalable infrastructure and cost-effective storage solutions. Utilizing Cloud Storage's resumable uploads, developers can break down larger images into manageable chunks, facilitating smoother uploads while circumventing size constraints. Additionally, optimizing image formats and employing compression techniques like WebP can help reduce file sizes without compromising quality, enhancing storage efficiency. Although managing larger images in Google App Engine presents challenges, leveraging Cloud Storage's capabilities and employing optimization strategies empowers developers to efficiently handle and store substantial image files within the platform's constraints.
Trucking accident attorney
hampton virginia personal injury lawyers
Trucking accident attorney
hampton virginia personal injury lawyers
Storing images larger than 1 megabyte in Google App Engine demands a thoughtful and strategic approach, considering the platform's constraints. Leveraging Google Cloud Storage is a prudent choice, allowing seamless storage and retrieval of large files. By breaking down images into smaller, manageable chunks, developers can efficiently handle data transfer limitations inherent in App Engine. Adopting Google Cloud Storage's resumable uploads ensures reliability and integrity during the storage process, providing a robust solution for handling larger image files.
truck accident lawyer virginia
criminal law firm washington dc
truck accident lawyer virginia
criminal law firm washington dc
Storing images larger than 1 megabyte in Google App Engine (GAE) requires careful consideration and implementation of efficient storage strategies. One approach is to utilize Google Cloud Storage (GCS), a scalable and cost-effective solution for storing large files, including images. By integrating GCS with GAE, developers can seamlessly upload, store, and retrieve images of any size, leveraging GCS's robust infrastructure and performance capabilities.
estate lawyer charlottesville va
estate lawyer charlottesville va
Subscribe to Post Comments [Atom]
<< Home
Subscribe to Posts [Atom]
Post a Comment