[Bug]: Map loading very long after upgrading to LWC 3.7

[SecondGIS]

What is the bug? (in English)

Hi there,
I was running Lizmap Web Client 3.6.12 with a few maps based on PostgreSQL / PostGIS layers, which was very fine.
Since I got a message in QGis Lizmap plugin saying version 3.6 is deprecated I decided to upgrade to LizMap Web Client version 3.7.10 (upgrading lizmap_server plugin too to version 2.10.0).
Since then my maps are incredibly long to load : 30 seconds at least, when it was only a few seconds with previous version I haven't modified my maps or any other software component in between.
When I load one of maps, I got the LizMap logo for 1 / 2s, then the empty map frame is displayed, with the legend panel taking, as said, like 30s to be displayed, and only then my map layer are visible.

Steps to reproduce the issue

1 - Publish a map with PostGis layers with LWC 3.6 and make sure that it is displayed correctly and fast enough in your web browser ;
2 - Upgrade Lizmap Web Client to version 3.7.10 (and lizmap_server plugin to version 2.10.0 or 2.9.4 as well)
3 - Try to load the same map in your web browser

Versions, safeguards, check summary etc

Versions :

• Lizmap Web Client : 3.7.10
• Lizmap plugin : 4.3.24
• QGIS Desktop : 3.28.15
• QGIS Server : 3.28.13
• Py-QGIS-Server : 1.8.8
• QGIS Server plugin atlasprint : 3.3.2
• QGIS Server plugin lizmap_server : 2.10.0
• QGIS Server plugin wfsOutputExtension : 1.8.0

List of safeguards :

• Mode : normal
• Allow parent folder : no
• Prevent other drive : yes
• Prevent PG service : yes
• Prevent PG Auth DB : yes
• Force PG user&pass : yes
• Prevent ECW : yes

Check Lizmap plugin

• I have done the step just before in the Lizmap QGIS desktop plugin before opening this ticket. Otherwise, my ticket is not considered valid and might get closed.

Operating system

ubuntu 22.04

Browsers

Firefox

Browsers version

Firefox 130.0, Chromium 128.0

Relevant log output

No response

[Gustry]

Unfortunately, without more info, you will need to debug where the bottleneck is. It really depends on your project.

Please open your "Developper tools" (F12), to see HTTP requests and check which ones are too long.

Maybe linked to #4521 about the legend, or maybe another bottleneck.

[SecondGIS]

You can follow further debugging and discussion about this issue at #4521.

[github-actions]

This issue is missing some feedbacks. 👻
Please have a look to the discussion, thanks. 🦎

[r9zzai]

Anyone can confirm that "getProjectConfig" request increased loading time since lwc 3.7.x (tested multiple projects, but one with 5 sec for 4 local file layers)? We had similar expierence (but not related to given link i think - getlegendimage request) and staid on lwc 3.6.x because of that.

Tested:
Lwc 3.8.3, qgis 3.38.3
Lwc 3.8.3 qgis 3.34.3
Lwc 3.6.x tested up to 3.7.x qgis 3.34.3

Curr. Setting:
Lwc 3.6.14 qgis 3.28.15

[github-actions]

This issue is missing some feedbacks. 👻
Please have a look to the discussion, thanks. 🦎

[SecondGIS]

Today I could confirm this issue is not solved :
I built a lizmap container using official docker image on my laptop (Ubuntu 22.04) specifying lizmap version 3.6.12 with qgis-server 3.28, loaded a test map and could verify that is was displayed very fast in Firefox ;
Then I built a new lizmap container, using lizmap 3.7.10 with qgis-server 3.34, and same map was very long to load ;
Then again, I built one last lizmap container, not specifying any version (so I would get last versions of both lizmap and qgis-server, 3.8.3 / 3.34), and again my map was very long to load, with Firefox web console showing getLegendGraphic to mainly cause the latency issue.
Also, I ran this command docker logs -f --since 1m lizmap-docker-compose-map-1 > log.txt 2>&1 before loading my map in Firefox : log with lizmap 3.6.12 is 80Ko when log version 3.8.3 is 220Ko, showing many more requests are done with newer version.
Thanks for your help

[Gustry]

Hi @SecondGIS
Thanks for the debug.
Can you try with QGIS Server 3.40 ? A bugfix was made in this version.

[gioman]

Can you try with QGIS Server 3.40 ? A bugfix was made in this version.

I did my own tests with 3.7.* and QGIS Server 3.40 and I can't see any major different compared to using QGIS Server LTR: it is still quite slow, and the slowness comes from GetLegend. In another tickets there was a discussion about possible mitigations of this problem, did anything got considered to be implemented?

[Gustry]

In another tickets there was a discussion about possible mitigations of this problem, did anything got considered to be implemented?

One bugfix landed in QGIS 3.40 only. qgis/QGIS#58937
It's just one piece of the puzzle, not the full solution.

[josemvm]

i'm on qgis server 3.34 and i have noticed a huge difference between lwc 3.7 and lwc 3.8, particularly in a project with many layers (mostly raster type/group as layer)

[Gustry]

i have noticed a huge difference between lwc 3.7 and lwc 3.8

On which side are you talking. Increased or decreased the loading time ?

[josemvm]

@Gustry very increased from 3.7 to 3.8. i can provide an url if you want
thanks

[gioman]

One bugfix landed in QGIS 3.40 only. qgis/QGIS#58937
It's just one piece of the puzzle, not the full solution.

On the LMWC side of things, is there any chance/plan to allow users to choose to use legends requested as PNG as it was until LMWC 3.6?

[mdouchin]

@gioman we could indeed add an option to let the user choose if the legend should be PNG (no possibility to toggle symbology classes) or JSON

We should really tackle this issue about performance degradation and figure out what changes between 3.7 & 3.8 provoked it
Diff is big though...
release_3_7...release_3_8

I would love to have a "DEBUG" behaviour allowing to time each PHP method and see what is going on...

[gioman]

@gioman we could indeed add an option to let the user choose if the legend should be PNG (no possibility to toggle symbology classes) or JSON

yeah that is clear of course, but it seems there are a lot of users that are affected that I guess would not mind to have the other 3.7/8 goodness but still use legends in raster format, or... there is a ticket (closed not sure why) with other suggestions, like have the legend JSON file generated by the LM plugin and uploaded together with the .CFG file, or have it generated only on first load of the project then stored on disk and read from there.

[SecondGIS]

Hi @SecondGIS Thanks for the debug. Can you try with QGIS Server 3.40 ? A bugfix was made in this version.

I just tried QGis server 3.40, it seems at least as slow as 3.34 (if not slower)... Every request is very long, not only GetLegendGraphic :/

[josemvm]

maybe also due to a lizmap cache issue.
the slowdown occurs mainly when all the caches are empty, that's when the project is loaded for the first time.
then the next times everything seems to work normally.

[gioman]

maybe also due to a lizmap cache issue.
the slowdown occurs mainly when all the caches are empty, that's when the project is loaded for the first time.
then the next times everything seems to work normally.

@josemvm I believe I set caching of projects capabilities the right way in my virtual hosts configs, and indeed it takes a first load of the project (after a web server restart) to get cached and load much faster, still after the first phase (when the "loading.svg/gif" is shown) it takes longer to generate legends... and until it finishes users cannot do anything.

[SecondGIS]

On my local setup, GetLegendGraphic request is not cached and still very long even after first loading... And anyway in my opinion this is not a caching issue, requesting JSON elements one by one would still take ages even if JSON file was cached

[gioman]

On my local setup, GetLegendGraphic request is not cached and still very long even after first loading... And anyway in my opinion this is not a caching issue, requesting JSON elements one by one would still take ages even if JSON file was cached

@SecondGIS I don't think that the getlegend requests outputs are cached. What is cached in the the capabilities of the projects see https://docs.qgis.org/3.34/en/docs/server_manual/config.html#environment-variables "QGIS_SERVER_CAPABILITIES_CACHE_SIZE". You noticed that because of first load you see the "loading.svs/gid" animation for a few instants/seconds. but on subsequent loads it goes away very fast. But after that now it takes much longer to get to the point where users can see maps because it has to wait legend requests to be done (even for layers that are not toggled on load?). That is my understanding, but I can be wrong.

[SecondGIS]

We should really tackle this issue about performance degradation and figure out what changes between 3.7 & 3.8 provoked it

@mdouchin from my own experience issue appeared between 3.6 and 3.7, as well as other issues like #4938