Monday, November 30, 2009


How to store images larger than 1 megabyte in Google App Engine

Over the summer, Google App Engine raised its limits for web requests and responses from 1MB to 10MB, but kept the maximum size of any single database element at 1MB. If you try to exceed this, you'll get a MemoryError. You can find a fair amount of grief and woe and gnashing of teeth and wearing of sackcloth and ashes about this online.

Which is kind of surprising, because it's not that hard to break files up into chunks and store those chunks in the database separately. Here's what I did today for my current project, which stores data - including photos - uploaded from smartphones:

First, we have to receive the uploaded image. Our uploads are two-phase - first data, then a photo - for various reasons. The data upload includes the image's file name; the photo upload is a basic form/multipart POST with exactly one argument (the filename) and its value (the file).

So, in "":

class SaveImage(webapp.RequestHandler):
def post(self):
for arg in self.request.arguments():
file = self.request.get(arg)
response = entryHandler.saveImage(arg,file)

and in "":

class ImageChunk(db.Model):
entryRef = db.ReferenceProperty(Entry)
chunkIndex = db.IntegerProperty()
chunk = db.BlobProperty()

class EntryHandler:
def saveImage(self, fileName, file):
results = Entry.all().filter("photoPath =", fileName).fetch(1)
if len(results)==0:
logging.warning("Error - could not find the entry associated with image name "+fileName)
return "Failed"
entry = results[0]
while marker*MaxBTSize<len(file):
if MaxBTSize*(marker+1)>len(file):
chunk = ImageChunk(entryRef=entry, chunkIndex=marker, chunk=db.Blob(file[MaxBTSize*marker:]))
chunk = ImageChunk(entryRef=entry, chunkIndex=marker, chunk=db.Blob(file[MaxBTSize*marker:MaxBTSize*(marker+1)]))
marker+=1"Successfully received image "+fileName)
return "Successfully received image "+fileName

Pretty basic stuff: we chop the image up at each 1,000,000-byte mark, and put each chunk into its own ImageChunk DB object.

Then, when we need to retrieve the image, in '':

class ShowImageWithKey(webapp.RequestHandler):
def get(self):
key = self.request.get('entryKey')
entryHandler = ec.EntryHandler()
image = entryHandler.getImageByEntryKey(key)
if image is not None:
self.response.headers['Content-Type'] = 'image/jpeg'

and in '':

def getImageByEntryKey(self, key):
chunks = db.GqlQuery("SELECT * FROM ImageChunk WHERE entryRef = :1 ORDER BY chunkIndex", key).fetch(100)
if len(chunks)==0:
return None

for chunkRow in chunks:
return image

Since db.Blob is a subtype of str, that's all you have to do. I don't understand why some people are so upset about this: it's mildly annoying that I had to write the above, but hardly crippling. At least with JPEGs, which is what we use. (But I don't see why any other file type would be more difficult; they're ultimately all just a bunch of bytes). Could hardly be easier ... well, until App Engine rolls out their large file service.

(eta, Dec 14: which came out today! Meaning you can now disregard all the above and just use the new Blobstore instead.)

(eta, Dec 16: mmm, maybe not. Looked at the Blobstore in detail today, and it's really best suited for browser projects, not app or web-service stuff. The API for the blobs is very limited, and you can only access them via one-time-only URLs that App Engine puts in your HTML. You could scrape that, granted, but that's a pain in the ass, no less inelegant than the image-chunking solution above. It's experimental and subject to change, too. I think I'll hold out until its API improves.)

Labels: , , , , , , , , , ,

This has been really helpful in getting around an annoying issue with Blobstore.

But how do you make thumbs of 1MB+ files? The Image class can only take 1MB files and blobstore values?

I really like your implementation, but can't be loading 5MB images in a gallery everytime I want to display some thumbs.

Any help would be greatly appreciated.
Amazing blog with unique information found valuable and enjoyed reading this one. Keep posting. Thanks for sharing.
Data Science Training in Hyderabad
I finally found a great article here. I just added your blog to my bookmarking sites looking forward for next blog thank you.
Data Science Course in Bangalore
I want to thank you for your efforts in writing this article. I look forward to the same best job from you in the future.

Business Analytics Course in Bangalore
I really enjoy every part and have bookmarked you to see the new things you post. Well done for this excellent article. Please keep this work of the same quality.
Artificial Intelligence course in Chennai
I really enjoy reading all of your blogs. It is very helpful and very informative and I really learned a lot from it. Definitely a great article
Data Science Course Bangalore
Thank a lot. You have done excellent job. I enjoyed your blog . Nice efforts
Cyber Security Course in Bangalore
Nice Blog and i would like to thank for the efforts you have made in writing this post, hoping the same best work from you in the future as well. Thanks for sharing. Great websites!
Tableau Training in Bangalore
Such a very useful article and very interesting to read this article, i would like to thank you for the efforts you had made for writing this awesome article. Thank you!
Python Training in Bangalore
Awesome article. I enjoyed reading your articles. this can be really a good scan for me. wanting forward to reading new articles. maintain the nice work!
Data Science Courses in Bangalore

I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
Data Science Course in Chennai
All of these posts were incredible perfect. It would be great if you’ll post more updates and your website is really cool and this is a great inspiring article.
Artificial Intelligence course in Chennai
You have completed certain reliable points there. I did some research on the subject and found that almost everyone will agree with your blog.

Data Science Training in Bangalore

I have voiced some of the posts on your website now, and I really like your blogging style. I added it to my list of favorite blogging sites and will be back soon ...

Digital Marketing Training in Bangalore

I found Habit to be a transparent site, a social hub that is a conglomerate of buyers and sellers willing to offer digital advice online at a decent cost.

Artificial Intelligence Training in Bangalore
The Extraordinary blog went amazed by the content that they have developed in a very descriptive manner. This type of content surely ensures the participants explore themselves. Hope you deliver the same near the future as well. Gratitude to the blogger for the efforts.

Machine Learning Course in Bangalore
Thanks for sharing this informative content.,
Turient is an All-in-one platform for all our teaching needs. If Teaching is your passion ,enabling is ours
Read the Informative blog - 11 Free Teaching Tools for Online Teachers
11 Free Teaching Tools for Online Teachers
Free Teaching Tools

I am more curious to take an interest in some of them. I hope you will provide more information on these topics in your next articles.

Machine Learning Course in Bangalore
I was browsing the internet for information and found your blog. I am impressed with the information you have on this blog.
MLOps Course
It is late to find this act. At least one should be familiar with the fact that such events exist. I agree with your blog and will come back to inspect it further in the future, so keep your performance going.

Data Scientist Training in Bangalore

Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!data analytics course in nagpur


I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.<a href=">cloud computing course in jaipur</a>
Really this article is truly one of the best in article history and am a collector of old "items" and sometimes read new items if i find them interesting which is one that I found quite fascinating and should be part of my collection. Very good work!
Data Scientist Course in Gurgaon
If you are looking for Illinois license plate sticker renewals online, you have to go to the right place. We have the fastest Illinois license plate sticker renewals in the state.

Data Science Course in Nagpur
Very great post which I really enjoy reading this and it is not everyday that I have the possibility to see something like this. Thank You.
Best Online Data Science Courses
I think this is a really good article. You make this information interesting and engaging. Thanks for sharing.
Data Science Course in India
This is really nice which is really cool blog and you have really helped a lot of people who visit the blog and give them useful information.
Data Science Training in Noida
There is obviously a lot to know about this. I think you made some good points in Features also. Great job!
Mlops Course
We are really grateful for your blog post. You will find a lot of approaches after visiting your post. Great work Data Science Training in Vadodara
Just a shine from you here and have never expected anything less from you and have not disappointed me at all which i guess you will continue the quality work. Great post.
Data Science Training in Gurgaon
Very informative Blog! There is so much information here that can help thank you for sharing.
Data Analytics Training in Bangalore
Informative Post. The information you have posted is very useful and sites you have referred was good. Thanks for sharing.
Data Science Course with Placement

I have read your article; it is very informative and helpful for me. I admire the valuable information you offer in your articles. Thanks for posting it. Feel free to visit my website; 토토사이트

It is very rare these days to find sites that provide information someone is watching for. I am glad to see that your site share valued information that can help to many readers. nice one and keep writing! Feel free to visit my website; 배트맨토토

Well done for this excellent article. and really enjoyed reading this article today it might be one of the best articles I have read so far and please keep this work of the same quality.
Data Analytics Course in Noida
This is a great inspiring blog.You have shared really very helpful information thank you.
Data Scientist Course in Amritsar
The blog and data is excellent and informative as well your work is very good and I appreciate well hopping for some more informative posts.
Business Analytics Course in Gurgaon
A good blog always contains new and exciting information that helps readers to get useful information I am sure this is it.
Data Science Course in Lucknow
It's an honor to visit this blog by chance.
I'd appreciate it if you could visit my blog.
This is my blog.

Amazing knowledge and I love to share this kind of information with my friends and hope they like it, why I am doing it.

Data Scientist Course in Nagpur
I’m gone to inform my little brother, that he should also
visit this blog on regular basis to take updated from newest gossip. 메이저사이트

Very shortly this site will be famous amid all blogging and site-building people, due to it’s nice articles or reviews. 온라인경마

I savour, lead to I found exactly what I used to be having a look for. You’ve ended my four day long hunt! God Bless you man. Have a nice day. Bye 사설토토

I have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.

Great information.Thanks for sharing the article
Data science course in hyderabad
Data science training in hyderabad
I have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.

Data science course in hyderabad
Data science training in hyderabad
I'm so happy to finally find a post with what I want. 메이저토토사이트 You have inspired me a lot. If you are satisfied, please visit my website and leave your feedback.

That's a great blog. There are a lot of good writings. I really appreciate people like you.


Post a Comment

Subscribe to Post Comments [Atom]

<< Home

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]