s3 slowdown, 63 queries for 7 thumbnails

[fabiocaccamo]

As you suggested, I already set THUMBNAIL_FORCE_OVERWRITE = True

[mariocesar]

@fabiocaccamo please try to use the latest version first: pip install --pre sorl-thumbnail

[fabiocaccamo]

@mariocesar the issue persists also using the latest version.

Now, loading just 1 image (so 1 thumbnail) I can see that this query is executed 6 times:

SELECT `thumbnail_kvstore`.`key`, `thumbnail_kvstore`.`value` FROM `thumbnail_kvstore` WHERE `thumbnail_kvstore`.`key` = 'sorl-thumbnail||image||65a8dc7d91b1badadf0df5c762c95ecb'

and this query is duplicated 3 times:

INSERT INTO `thumbnail_kvstore` (`key`, `value`) VALUES ('sorl-thumbnail||image||52f291ff36262f49b8bb589942f99750', '{\"storage\": \"sorl.thumbnail.images.UrlStorage\", \"name\": \"{{ S3_URL }}\", \"size\": [1000, 1500]}')

Thanks for the quick reply.

[mariocesar]

Probably related to cache, clean your cache keys

What cache backend are you using?

[fabiocaccamo]

Cleaning cache keys didn't solved the problem, I don't think that the number of queries depends by the cache backend.

[jjdelc]

For me the slowdown using s3 is not the db queries but the file transfer time:

• Fetch original image (network time) - usually a large file, phone high quality a couple of MB at least
• Generate the thumbnail (CPU time), can consume high memory as well
• Transfer the generated thumbnail to s3 cache bucket (network time)

This all happens during template rendering time, and once per new thumbnail to generate which causes the responses to take a lot of time to the point of timing out. I don't have an easy solution, what I did is generate the thumbnails async after flie upload using celery so they're ready on the next request.

All this cycle is WAY faster when running just on local filesystem, it manifests when using s3.

The DB hits haven't been much of a slowdown for me.

[fabiocaccamo]

Frankly I don't know the cause of the slowdown, but I solved it switching to VersatileImageField.

[SalahAdDin]

👍