Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

idr-testing logs analysis from testing on 10th April #692

Open
will-moore opened this issue Apr 10, 2024 · 0 comments
Open

idr-testing logs analysis from testing on 10th April #692

will-moore opened this issue Apr 10, 2024 · 0 comments

Comments

@will-moore
Copy link
Member

will-moore commented Apr 10, 2024 •
edited
Loading

TLDR: ERROR in logs seem to be coming from memo file generation (which was cancelled but still running during web testing), rather than from web testing itself....

Start by scanning logs on all 5 servers for ERROR statements from 9am today:
NB: logs are timestamped with GMT (not BST), so we need to grep for timestamps for an hour earlier '2024-04-10 08...'

for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do echo $server && ssh $server "grep ERROR /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep '2024-04-10 08'"; done

omeroreadwrite
2024-04-10 08:25:44,777 ERROR [        ome.services.util.ServiceHandler] (.Server-19) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 7625339
2024-04-10 08:41:10,171 ERROR [        ome.services.util.ServiceHandler] (l.Server-0) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 7753026
2024-04-10 08:44:52,401 ERROR [        ome.services.util.ServiceHandler] (.Server-23) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 7404650
2024-04-10 08:57:04,335 ERROR [        ome.services.util.ServiceHandler] (.Server-12) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 7905672
omeroreadonly-1
omeroreadonly-2
2024-04-10 08:02:30,423 ERROR [        ome.services.util.ServiceHandler] (.Server-12) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8642795
2024-04-10 08:06:20,880 ERROR [        ome.services.util.ServiceHandler] (l.Server-7) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8516806
2024-04-10 08:07:12,305 ERROR [        ome.services.util.ServiceHandler] (.Server-14) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8421699
2024-04-10 08:09:47,515 ERROR [        ome.services.util.ServiceHandler] (.Server-16) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8625053
2024-04-10 08:11:25,710 ERROR [        ome.services.util.ServiceHandler] (.Server-18) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8535615
2024-04-10 08:11:43,066 ERROR [        ome.services.util.ServiceHandler] (l.Server-8) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8596496
2024-04-10 08:12:37,982 ERROR [        ome.services.util.ServiceHandler] (l.Server-6) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8332341
2024-04-10 08:21:45,104 ERROR [        ome.services.util.ServiceHandler] (l.Server-2) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8401669
omeroreadonly-3
2024-04-10 08:58:26,437 ERROR [        ome.services.util.ServiceHandler] (.Server-31) Method interface ome.services.util.Executor$Work.doWork invocation took 1443967
2024-04-10 08:58:26,438 ERROR [        ome.services.util.ServiceHandler] (.Server-31) Method interface omeis.providers.re.RenderingEngine.load invocation took 1443984
omeroreadonly-4

Since idr-testing web tends to use omeroreadonly-3 and omeroreadonly-4 lets check omeroreadonly-3:

ssh omeroreadonly-3

[wmoore@test120-omeroreadonly-3 ~]$ grep -B 10 -A 2 "2024-04-10 08:58:26,437" /opt/omero/server/OMERO.server/var/log/Blitz-0.log
2024-04-10 08:58:21,090 INFO  [                      omero.cmd.SessionI] (.Server-30) Removed servant from adapter: 984148d6-7e5b-469c-b9c2-e3f4b2c4d867omero.api.IConfig
2024-04-10 08:58:25,635 INFO  [ ome.services.blitz.fire.SessionManagerI] (.Server-45) Found session locally: fc82fd8f-f877-449c-9f1c-a6eeac4a3069
2024-04-10 08:58:25,636 INFO  [ ome.services.blitz.fire.SessionManagerI] (.Server-45) Rejoining session ServiceFactoryI(session-cc243189-5076-426a-9d5a-a03e0a312e36/fc82fd8f-f877-449c-9f1c-a6eeac4a3069) (agent=OMERO.web)
2024-04-10 08:58:25,642 INFO  [o.services.sessions.SessionContext$Count] (l.Server-8) -Reference count: fc82fd8f-f877-449c-9f1c-a6eeac4a3069=0
2024-04-10 08:58:25,642 INFO  [                      omero.cmd.SessionI] (l.Server-8) cleanupSelf(ServiceFactoryI(session-cc243189-5076-426a-9d5a-a03e0a312e36/fc82fd8f-f877-449c-9f1c-a6eeac4a3069)).
2024-04-10 08:58:26,432 DEBUG [                   loci.formats.Memoizer] (.Server-31) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/18-54-45.119_mkngff/a78bd2cc-f574-47d9-ae83-e3df322efdda.zarr/OME/.METADATA.ome.xml.bfmemo7903852454255395295
2024-04-10 08:58:26,432 DEBUG [                   loci.formats.Memoizer] (.Server-31) start[1712739504492] time[1939] tag[loci.formats.Memoizer.saveMemo]
2024-04-10 08:58:26,435 DEBUG [                   loci.formats.Memoizer] (.Server-31) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/18-54-45.119_mkngff/a78bd2cc-f574-47d9-ae83-e3df322efdda.zarr/OME/.METADATA.ome.xml.bfmemo (357780 bytes)
2024-04-10 08:58:26,435 DEBUG [                   loci.formats.Memoizer] (.Server-31) start[1712738062477] time[1443958] tag[loci.formats.Memoizer.setId]
2024-04-10 08:58:26,435 INFO  [                ome.io.nio.PixelsService] (.Server-31) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-54-45.119_mkngff/a78bd2cc-f574-47d9-ae83-e3df322efdda.zarr/OME/METADATA.ome.xml Series: 0
2024-04-10 08:58:26,437 INFO  [                 org.perf4j.TimingLogger] (.Server-31) start[1712738062469] time[1443967] tag[omero.call.success.ome.services.RenderingBean$12.doWork]
2024-04-10 08:58:26,437 INFO  [        ome.services.util.ServiceHandler] (.Server-31)  Rslt:	ome.io.bioformats.BfPixelBuffer@1671e72d
2024-04-10 08:58:26,437 ERROR [        ome.services.util.ServiceHandler] (.Server-31) Method interface ome.services.util.Executor$Work.doWork invocation took 1443967

from idr0011 - ScreenB:

less /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-54-45.119_mkngff/a78bd2cc-f574-47d9-ae83-e3df322efdda.zarr/OME/METADATA.ome.xml
...
Name="/uod/idr/filesets/idr0011-thorpe-Dad4/20150826-peter_thorpe/T34 x TS/Plate1-TS/Plate1-TS-Red-B"

Using the same approach we can see...

On readonlyomero-2, the last error is at 2024-04-10 08:21:45,104 (before we started web testing) coming from the memo file saved for idr0013: plate LT0064_25.

On omeroreadwrite the error at 2024-04-10 08:41:10,171 (GMT) during testing (9:40 BST) is "memo file saved" from idr0013: plate LT0065_05.
2024-04-10 08:44:52,401 is from idr0013 plate LT0065_06.
2024-04-10 08:57:04,335 is from idr0013 plate LT0066_02

All these happened during testing, using a different server (omeroreadonly) but the same Database.

Checking an hour later, we see that memo file generation was still ongoing...

[wmoore@test120-proxy ~]$ for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do echo $server && ssh $server "grep ERROR /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep '2024-04-10 09'"; done
omeroreadwrite

2024-04-10 09:01:27,393 ERROR [        ome.services.util.ServiceHandler] (.Server-22) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8237225
2024-04-10 09:09:47,201 ERROR [        ome.services.util.ServiceHandler] (l.Server-6) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8357950
2024-04-10 09:16:02,991 ERROR [        ome.services.util.ServiceHandler] (l.Server-9) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8531641
2024-04-10 09:21:19,547 ERROR [        ome.services.util.ServiceHandler] (.Server-10) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8734811
2024-04-10 09:22:39,273 ERROR [        ome.services.util.ServiceHandler] (.Server-14) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8577378
2024-04-10 09:24:30,872 ERROR [        ome.services.util.ServiceHandler] (.Server-24) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8822350
omeroreadonly-1
2024-04-10 09:39:57,966 ERROR [        ome.services.util.ServiceHandler] (.Server-26) Method interface ome.services.util.Executor$Work.doWork invocation took 1645799
2024-04-10 09:39:57,968 ERROR [        ome.services.util.ServiceHandler] (.Server-26) Method interface omeis.providers.re.RenderingEngine.load invocation took 1645815
2024-04-10 09:45:58,915 ERROR [        ome.services.util.ServiceHandler] (.Server-21) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8039111
2024-04-10 09:46:39,153 ERROR [        ome.services.util.ServiceHandler] (.Server-15) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 7571553
2024-04-10 09:47:48,514 ERROR [        ome.services.util.ServiceHandler] (.Server-24) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8129785
omeroreadonly-2
omeroreadonly-3
2024-04-10 09:13:40,629 ERROR [        ome.services.util.ServiceHandler] (.Server-48) Method interface ome.services.util.Executor$Work.doWork invocation took 1861115
2024-04-10 09:13:40,630 ERROR [        ome.services.util.ServiceHandler] (.Server-48) Method interface omeis.providers.re.RenderingEngine.load invocation took 1861135
2024-04-10 09:13:40,667 ERROR [        ome.services.util.ServiceHandler] (2-thread-5) Method interface ome.services.util.Executor$Work.doWork invocation took 22879
2024-04-10 09:13:40,679 ERROR [        ome.services.util.ServiceHandler] (-thread-71) Method interface ome.services.util.Executor$Work.doWork invocation took 80847
2024-04-10 09:13:40,680 ERROR [        ome.services.util.ServiceHandler] (.Server-22) Method interface ome.api.IAdmin.getEventContext invocation took 80852
2024-04-10 09:13:40,691 ERROR [        ome.services.util.ServiceHandler] (.Server-26) Method interface ome.api.IConfig.getConfigValues invocation took 49924
2024-04-10 09:13:40,696 ERROR [        ome.services.util.ServiceHandler] (-thread-70) Method interface ome.services.util.Executor$Work.doWork invocation took 80863
2024-04-10 09:13:40,697 ERROR [        ome.services.util.ServiceHandler] (.Server-30) Method interface ome.api.IAdmin.getEventContext invocation took 80869
2024-04-10 09:13:40,698 ERROR [        ome.services.util.ServiceHandler] (-thread-74) Method interface ome.services.util.Executor$Work.doWork invocation took 80864
2024-04-10 09:13:40,699 ERROR [        ome.services.util.ServiceHandler] (.Server-14) Method interface ome.api.IAdmin.getEventContext invocation took 80869
2024-04-10 09:13:40,703 ERROR [        ome.services.util.ServiceHandler] (-thread-75) Method interface ome.services.util.Executor$Work.doWork invocation took 80868
2024-04-10 09:13:40,704 ERROR [        ome.services.util.ServiceHandler] (.Server-42) Method interface ome.api.IAdmin.getEventContext invocation took 80875
2024-04-10 09:13:40,708 ERROR [        ome.services.util.ServiceHandler] (l.Server-6) Method interface ome.services.util.Executor$Work.doWork invocation took 44333
2024-04-10 09:13:40,709 ERROR [        ome.services.util.ServiceHandler] (.Server-31) Method interface ome.api.IConfig.getConfigValues invocation took 79937
2024-04-10 09:13:40,709 ERROR [        ome.services.util.ServiceHandler] (.Server-29) Method interface ome.services.util.Executor$Work.doWork invocation took 44281
2024-04-10 09:13:40,715 ERROR [        ome.services.util.ServiceHandler] (l.Server-9) Method interface ome.services.util.Executor$Work.doWork invocation took 40783
2024-04-10 09:13:40,715 ERROR [        ome.services.util.ServiceHandler] (.Server-25) Method interface ome.services.util.Executor$Work.doWork invocation took 40782
2024-04-10 09:13:40,715 ERROR [        ome.services.util.ServiceHandler] (.Server-46) Method interface ome.services.util.Executor$Work.doWork invocation took 44250
2024-04-10 09:13:40,716 ERROR [        ome.services.util.ServiceHandler] (.Server-24) Method interface ome.services.util.Executor$Work.doWork invocation took 40775
2024-04-10 09:13:40,729 ERROR [        ome.services.util.ServiceHandler] (.Server-27) Method interface ome.api.IConfig.getConfigValues invocation took 34965
2024-04-10 09:13:40,744 ERROR [        ome.services.util.ServiceHandler] (.Server-52) Method interface ome.services.util.Executor$Work.doWork invocation took 40802
2024-04-10 09:13:41,783 ERROR [        ome.services.util.ServiceHandler] (.Server-38) Method interface ome.api.IConfig.getConfigValues invocation took 66015
2024-04-10 09:40:41,938 ERROR [        ome.services.util.ServiceHandler] (.Server-44) Method interface ome.services.util.Executor$Work.doWork invocation took 1809920
2024-04-10 09:40:41,940 ERROR [        ome.services.util.ServiceHandler] (.Server-44) Method interface omeis.providers.re.RenderingEngine.load invocation took 1809936
2024-04-10 09:43:23,162 ERROR [        ome.services.util.ServiceHandler] (.Server-33) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8133881
2024-04-10 09:47:00,892 ERROR [        ome.services.util.ServiceHandler] (.Server-16) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 7794919
omeroreadonly-4
2024-04-10 09:37:05,549 ERROR [        ome.services.util.ServiceHandler] (.Server-92) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 7485398
2024-04-10 09:37:33,544 ERROR [        ome.services.util.ServiceHandler] (.Server-86) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 7707792
2024-04-10 09:37:57,370 ERROR [        ome.services.util.ServiceHandler] (.Server-73) Method interface ome.services.util.Executor$Work.doWork invocation took 1585274
2024-04-10 09:37:57,372 ERROR [        ome.services.util.ServiceHandler] (.Server-73) Method interface omeis.providers.re.RenderingEngine.load invocation took 1585286

Most of those are coming from omeroreadonly-3
First of these is associated with creating PixelBuffer which waited on the completion of memo file generation

[wmoore@test120-proxy ~]$ ssh omeroreadonly-3

[wmoore@test120-omeroreadonly-3 ~]$ grep -A 5 -B 5 "2024-04-10 09:13:40,629" /opt/omero/server/OMERO.server/var/log/Blitz-0.log
2024-04-10 09:13:40,571 DEBUG [                   loci.formats.Memoizer] (.Server-48) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/20-51-59.292_mkngff/f63bc331-42b0-4e55-abd3-abf4de843026.zarr/OME/.METADATA.ome.xml.bfmemo1881140284604001071
2024-04-10 09:13:40,572 DEBUG [                   loci.formats.Memoizer] (.Server-48) start[1712740415493] time[5078] tag[loci.formats.Memoizer.saveMemo]
2024-04-10 09:13:40,627 DEBUG [                   loci.formats.Memoizer] (.Server-48) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/20-51-59.292_mkngff/f63bc331-42b0-4e55-abd3-abf4de843026.zarr/OME/.METADATA.ome.xml.bfmemo (399027 bytes)
2024-04-10 09:13:40,628 DEBUG [                   loci.formats.Memoizer] (.Server-48) start[1712738559523] time[1861105] tag[loci.formats.Memoizer.setId]
2024-04-10 09:13:40,628 INFO  [                ome.io.nio.PixelsService] (.Server-48) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/20-51-59.292_mkngff/f63bc331-42b0-4e55-abd3-abf4de843026.zarr/OME/METADATA.ome.xml Series: 11
2024-04-10 09:13:40,629 INFO  [                 org.perf4j.TimingLogger] (.Server-48) start[1712738559514] time[1861115] tag[omero.call.success.ome.services.RenderingBean$12.doWork]
2024-04-10 09:13:40,629 INFO  [        ome.services.util.ServiceHandler] (.Server-48)  Rslt:	ome.io.bioformats.BfPixelBuffer@218d70be
2024-04-10 09:13:40,629 ERROR [        ome.services.util.ServiceHandler] (.Server-48) Method interface ome.services.util.Executor$Work.doWork invocation took 1861115

This is from idr0011:

less /data/OMERO/ManagedRepository/demo_2/2016-07/28/20-51-59.292_mkngff/f63bc331-42b0-4e55-abd3-abf4de843026.zarr/OME/METADATA.ome.xml
...
 Name="/uod/idr/filesets/idr0011-thorpe-Dad4/20150826-peter_thorpe/T34 x TS/Plate3-TS/Plate3-TS-Blue-B"

It looks like all the other ERROR from 09:13:40 were other services that were held-up by that one completing.

Errors at 09:40:41 were the same plate from idr0011:

[wmoore@test120-omeroreadonly-3 ~]$ grep -A 5 -B 5 "2024-04-10 09:40:41,938" /opt/omero/server/OMERO.server/var/log/Blitz-0.log
2024-04-10 09:40:41,789 DEBUG [                   loci.formats.Memoizer] (.Server-44) start[1712742032139] time[9650] tag[loci.formats.Memoizer.saveMemo]
2024-04-10 09:40:41,935 DEBUG [                   loci.formats.Memoizer] (.Server-44) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/20-51-59.292_mkngff/f63bc331-42b0-4e55-abd3-abf4de843026.zarr/OME/.METADATA.ome.xml.bfmemo (399013 bytes)
2024-04-10 09:40:41,935 DEBUG [                   loci.formats.Memoizer] (.Server-44) start[1712740232034] time[1809901] tag[loci.formats.Memoizer.setId]
2024-04-10 09:40:41,936 INFO  [                ome.io.nio.PixelsService] (.Server-44) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/20-51-59.292_mkngff/f63bc331-42b0-4e55-abd3-abf4de843026.zarr/OME/METADATA.ome.xml Series: 25
2024-04-10 09:40:41,937 INFO  [                 org.perf4j.TimingLogger] (.Server-44) start[1712740232017] time[1809920] tag[omero.call.success.ome.services.RenderingBean$12.doWork]
2024-04-10 09:40:41,938 INFO  [        ome.services.util.ServiceHandler] (.Server-44)  Rslt:	ome.io.bioformats.BfPixelBuffer@5f163467
2024-04-10 09:40:41,938 ERROR [        ome.services.util.ServiceHandler] (.Server-44) Method interface ome.services.util.Executor$Work.doWork invocation took 1809920

NB: we have a repeated generation of memo file for this Fileset, saved at the same location but with a difference of 14 bytes in size:

2024-04-10 09:13:40,627 DEBUG [                   loci.formats.Memoizer] (.Server-48) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/20-51-59.292_mkngff/f63bc331-42b0-4e55-abd3-abf4de843026.zarr/OME/.METADATA.ome.xml.bfmemo (399027 bytes)

2024-04-10 09:40:41,935 DEBUG [                   loci.formats.Memoizer] (.Server-44) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/20-51-59.292_mkngff/f63bc331-42b0-4e55-abd3-abf4de843026.zarr/OME/.METADATA.ome.xml.bfmemo (399013 bytes)

Other ERRORs also associated with memo file generation:

[wmoore@test120-omeroreadonly-3 ~]$ grep -A 5 -B 5 "2024-04-10 09:43:23,162" /opt/omero/server/OMERO.server/var/log/Blitz-0.log
2024-04-10 09:43:22,951 DEBUG [                   loci.formats.Memoizer] (.Server-33) start[1712742197527] time[5424] tag[loci.formats.Memoizer.saveMemo]
2024-04-10 09:43:23,159 DEBUG [                   loci.formats.Memoizer] (.Server-33) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-05/04/03-27-54.535_mkngff/7493b87d-b6f6-48ce-a19d-3741eb11a57f.zarr/OME/.METADATA.ome.xml.bfmemo (420748 bytes)
2024-04-10 09:43:23,159 DEBUG [                   loci.formats.Memoizer] (.Server-33) start[1712734069303] time[8133856] tag[loci.formats.Memoizer.setId]
2024-04-10 09:43:23,160 INFO  [                ome.io.nio.PixelsService] (.Server-33) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-05/04/03-27-54.535_mkngff/7493b87d-b6f6-48ce-a19d-3741eb11a57f.zarr/OME/METADATA.ome.xml Series: 0
2024-04-10 09:43:23,161 INFO  [                 org.perf4j.TimingLogger] (.Server-33) start[1712734069280] time[8133881] tag[omero.call.success.ome.services.RawPixelsBeanReadOnly.setPixelsId]
2024-04-10 09:43:23,162 INFO  [        ome.services.util.ServiceHandler] (.Server-33)  Rslt:	null
2024-04-10 09:43:23,162 ERROR [        ome.services.util.ServiceHandler] (.Server-33) Method interface ome.api.RawPixelsStore.setPixelsId invocation took 8133881

This is for idr0013, plate Name="LT0068_43" from less /data/OMERO/ManagedRepository/demo_2/2016-05/04/03-27-54.535_mkngff/7493b87d-b6f6-48ce-a19d-3741eb11a57f.zarr/OME/METADATA.ome.xml

Since we weren't looking at idr0013 during testing, this is likely coming from the parallel memo file generation that was only cancelled as testing started around 9:35.

This is confirmed by memo generation logs... (times here are GMT - 1 hour out). At 8:32 (9:32 BST) we see Killed by signal 15. in stderr.

[wmoore@test120-proxy ~]$ ls -alh /tmp/ngff_cache_20240409_ngff/1/Image\:1557553
total 20K
drwxrwxr-x.   2 wmoore wmoore   45 Apr 10 07:27 .
drwxrwxr-x. 258 wmoore wmoore 8.0K Apr 10 08:25 ..
-rw-rw-r--.   1 wmoore wmoore    3 Apr 10 07:27 seq
-rw-rw-r--.   1 wmoore wmoore   22 Apr 10 08:32 stderr
-rw-rw-r--.   1 wmoore wmoore    0 Apr 10 07:27 stdout
[wmoore@test120-proxy ~]$ cat /tmp/ngff_cache_20240409_ngff/1/Image\:1557553/stdout 
[wmoore@test120-proxy ~]$ cat /tmp/ngff_cache_20240409_ngff/1/Image\:1557553/stderr
Killed by signal 15.

But the memo file generation continued on the server until 09:43:23 (all during testing time).
8133856 ms (memo saved time) is 135 minutes, corresponding to start at 7:27 -> 9:43 (GMT).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@will-moore