Image processing performance

[celsojr-websters]

I have been working with Twill CMS for about two years and we are starting to experience issues related to the performance of the image service.

We are currently using Glide. But the problem is that users are uploading very large images to make them clearer. And this requires a lot of server resources, both CPU and memory, to process these images before rendering them.

I remember that in WordPress projects we had no performance problems, even when trying to upload very large images. I think this is because WordPress is not processing all the images in real time. But instead, it saves cropped and resized images to the file system. I mean it will crop and resize images in one go and never bother the server again.

We are already using a high-performance VM from Amazon and also caching resources with Cloudflare, but it is still not enough.

Has anyone here faced a similar problem? Do you know if it's possible to have an image service similar to what we have in WordPress with no or minimal impact on the frontend?

P.S.: I know there were performance improvements in Twill CMS in version 3+. But the Twill CMS version 2.5.3 is good enough for us. I believe the problem is just the imaging service now.

Ref.: https://twillcms.com/docs/media-library/image-rendering-service.html

[ifox]

Hi @celsojr-websters, this is not a performance problem with Twill itself as you correctly identified. Glide is a dynamic image rendering service, so what you're observing is expected. Can you clarify what very large images means in actual filesize/resolution though?

Which Glide driver are you using? GD? Imagick? Ultimately PHP is not the most efficient for image rendering, and that's why Twill also supports third party services for that. But you can also scale with Glide by dedicating resources for it instead of consuming server capacity that you need to actually serve other requests. Glide is also supposed to only render once, and if you have Cloudflare on top, your origin shouldn't need to render images that often. It is also possible to warm your Glide cache if your various query parameters applied to image URLs are known upfront.

I totally understand what you are saying about WordPress and many other CMSs that don't have dynamic rendering but rather are generating a bunch of predefined thumbnails at upload time. Twill allows contextual cropping, so this approach isn't necessarily ideal. Also, responsive images implementation sometimes do need the dynamism to avoid having to regenerate all the thumbnails just to change something minor on the frontend.

I'm still open to considering a new service that would generate predefined resolutions if that can help.

[ifox]

Please share actual numbers when reporting something like this. "High" and "large" can mean very different scale for different people.

[celsojr-websters]

Yeah, sure. I fully understand your concern because these are, indeed, two different things. But I believe our biggest issues are the large images. And by large, I mean large in megabytes.

https://thehealthyhome.me/img/a679a55b-c2cc-44c0-adec-7618132c6ad6/home-page-v2-1.jpg

For example, this background image we use in the home page is large in size, but not in megabytes. I mean, this is a low resolution image. It will be easily processed be Glide and also won't warm our server.

https://thehealthyhome.me/en/ae/all-carousel-testing-2

But, as you can see on this other page, we also have large images in megabytes in high resolution with more than 1Mb which is not good for page loading performance. That's where Glide comes in to help, but it hasn't helped much, as we've increased our storage in numbers with multiple images and have increased our page views numbers too.

I have heard that the Croppa does what I believe we need, but it has some limitations. For example, we need to some how convert the images into webp or some more modern format before rendering. Also, the URL schema that Croppa uses is limited and would have a huge impact for us to change our links in the frontend.

We are using Glide with GD and it was very helpful for a long time, but we're in a place where it seems to not scale anymore. I also love the dynamic rendering approach, but when it comes to a website with multiple request per seconds I don't think this is the most efficient solution.

We are internally discussing a better solution for the image service and we want to make this change as smooth as possible for our client/users.

[ifox]

I think you are wrongly accusing Glide when the problem is with your implementation.

On your homepage, you are correctly applying the w parameter to that image URL in your CSS background-image property.
However, on the page you shared, there is a syntax error in your HTML response, so the w parameter is not applied, resulting in the original image being returned (which is 4096px wide).

See the ') at the end of the image URL, it ends up being sent to Glide as w: 1400'):

The resulting image is 331kb vs 72k when w=1400 is correctly applied: https://thehealthyhome.me/img/a0e2ccdb-7d51-4607-ac32-40ea3d2b243c/exp-seamless-moving-experience-with-premium-movers-in-dubai.jpg?fm=webp&q=85&fit=max&crop=4096%2C2731%2C0%2C0&w=1400

[ifox]

Once the requested size is generated once by Glide, I can see sub 40ms responses from Cloudflare, so there is no reason for it to slow down your origin server from there, unless you are adding new images and requesting new sizes all the time.

[celsojr-websters]

I must to confess that I'm also having a hard time believing that the cache isn't working as it should. But I started blaming Glide after trying to put together a quick setup in pure WSL2, without Docker, to experiment with the performance. Where it didn't work and I accidentally noticed an "improvement" in performance on the website in general despite some images being rendered in their original size (before the browser cache came into action).

Do you believe that the CMS backend could have any impact on the server in some way? I mean, our backend is also heavily accessed because we have a team dedicated to creating content for this website and they may be experimenting with different image sizes and widths (megabytes and pixel wide images respectively).

[celsojr-websters]

I think you are wrongly accusing Glide when the problem is with your implementation.

On your homepage, you are correctly applying the w parameter to that image URL in your CSS background-image property. However, on the page you shared, there is a syntax error in your HTML response, so the w parameter is not applied, resulting in the original image being returned (which is 4096px wide).

See the ') at the end of the image URL, it ends up being sent to Glide as w: 1400'):

The resulting image is 331kb vs 72k when w=1400 is correctly applied: https://thehealthyhome.me/img/a0e2ccdb-7d51-4607-ac32-40ea3d2b243c/exp-seamless-moving-experience-with-premium-movers-in-dubai.jpg?fm=webp&q=85&fit=max&crop=4096%2C2731%2C0%2C0&w=1400

Ah, ha ha, good insight! Thank you very much for pointing this out. But, as you can see from the name of the slug, this is a test page that contains blocks in a work-in-progress state. We will correct this error you just pointed out as soon as possible. Twill makes it very easy to create custom block components as per client request. It's very flexible for this type of scenario.

[celsojr-websters]

I'm reading this just now

The simplest way to accomplish this is to save the cached images in the same place, and with the exact same URL as the requested images. This way, when the first request comes in, the cached image won't exist, so the request will be routed to PHP and handled by Glide. However, all subsequent requests will hit the cached image directly, without hitting PHP. This should greatly improve the performance of loading cached images.

Brainstorming on Glide 2.0

And

the ability to store images directly using Glide, profiting from the ability to resize on storage, which decrease file size and improve runtime performance

Glide 2.0 or future development

But, despite those are old issues never fixed, it seems they agree with me. And by this, of course, I'm not saying you disagree. We will probably scale our VMs this time, but hopefully the future is gonna be brighter. Also, if we come up with some sort of custom solution, I will try to come back and share. Some problems will only rise when storage is full of high resolution images, so I don't think it will be a big problem for everybody.

As you said before, It's true that PHP is not the most recommended option when it comes to performance for image rendering nowadays, I would go with C/C++ or Rust. But I must admit it still scales very well in most scenarios, at least in the beginning, with no or low cost.