Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upload to BioStudies #659

Open
will-moore opened this issue May 18, 2023 · 14 comments
Open

Upload to BioStudies #659

will-moore opened this issue May 18, 2023 · 14 comments
Assignees
@will-moore

Comments

@will-moore
Copy link
Member

will-moore commented May 18, 2023 •
edited
Loading

Notes for upload to BioStudies workflow...

  • Create a zip per Fileset
  • Upload to BioStudies via FTP
  • Group Files into a Submission by listing in a csv File, and submit that
  • Zip contents will be uploaded to s3 and archived to Fire

See https://www.ebi.ac.uk/biostudies/submissions/help#help-files
and https://www.ebi.ac.uk/bioimage-archive/submit/
@will-moore will-moore converted this from a draft issue May 18, 2023
@will-moore
Copy link
Member Author

Wanted to test with idr0054 data on pilot-zarr2-dev...
Having logged-in and got FTP details from https://www.ebi.ac.uk/biostudies/submissions/files...

$ ssh pilot-zarr2-dev

$ sftp [email protected]
write: Connection reset by peer
Couldn't read packet: Connection reset by peer

Trying from local machine:

$ sftp [email protected]
kex_exchange_identification: read: Connection reset by peer
Connection reset by 193.62.193.142 port 22
Connection closed

Will try a different client...

@will-moore
Copy link
Member Author

will-moore commented May 18, 2023 •
edited
Loading

Let's just try with Cyberduck and FileZilla from local machine...
Grab the data...

$ rsync -rvP --progress pilot-zarr2-dev:/data/ngff/idr0054/ .
$ zip -r "Tonsil 1.ome.zarr.zip" "Tonsil 1.ome.zarr"
etc.

I can login with Cyberduck but can't work out how to use Cyberduck to upload to my "secret directory" on the server.

Manually uploaded zips via the web UI

https://www.ebi.ac.uk/biostudies/submissions/

Submitted -> https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD704

EDIT: Tried FileZilla and I can see how to upload files to the secret directory. I see the files there that I uploaded via the web UI.

@will-moore
Copy link
Member Author

From Matthew:

"Corresponding S3 URIs in our bucket for each NGFF fileset:"

Tonsil 2.ome.zarr.zip, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr/0
Tonsil 1.ome.zarr.zip, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr/0
Tonsil 3.ome.zarr.zip, https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr/0

As discussed in IDR meeting today, I'll try to create a single goofys mount and look at how to use those mappings to create correct symlinks to the data on idr0125-pilot.

@will-moore
Copy link
Member Author

On idr0125-pilot:

$ sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

$ ls /bia-integrator-data/S-BIAD704
36cb5355-5134-4bdc-bde6-4e693055a8f9  3b4a8721-1a28-4bc4-8443-9b6e145efbe9  5583fe0a-bbe6-4408-ab96-756e8e96af55

@will-moore
Copy link
Member Author

Created:
With IDR/idr-utils@2ce54a1

# /tmp/idr0054_fileset_mappings.csv

Tonsil 2.ome.zarr,S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr
Tonsil 1.ome.zarr,S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr
Tonsil 3.ome.zarr,S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr

$ python idr-utils/scripts/managed_repo_symlinks.py Dataset:5601 /bia-integrator-data --fileset-mappings /tmp/idr0054_fileset_mappings.csv --report

fileset_dirs {'Tonsil 2.ome.zarr': 'S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr', 'Tonsil 1.ome.zarr': 'S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr', 'Tonsil 3.ome.zarr': 'S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr'}

Fileset: 5287119 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/
Render Image 5025551
fs_contents ['Tonsil 1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr to /bia-integrator-data/S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr

Fileset: 5287120 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/
Render Image 5025552
fs_contents ['Tonsil 2.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/Tonsil 2.ome.zarr to /bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr

Fileset: 5287121 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/
Render Image 5025553
fs_contents ['Tonsil 3.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/Tonsil 3.ome.zarr to /bia-integrator-data/S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr

@will-moore
Copy link
Member Author

will-moore commented May 29, 2023 •
edited
Loading

Viewing Image after symlink update above fails,

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
full stack trace 
Traceback (most recent call last):

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/decorators.py", line 538, in wrapped
    retval = f(request, *args, **kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/decorators.py", line 597, in wrapper
    context = f(request, *args, **kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/webclient/views.py", line 1777, in load_metadata_preview
    rdefId = manager.image.getRenderingDefId()

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 7944, in wrapped
    if not self._prepareRenderingEngine() \

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 8220, in _prepareRenderingEngine
    self._re = self._prepareRE(rdid=rdid)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 8199, in _prepareRE
    re.load(ctx)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 4859, in __call__
    return self.handle_exception(e, *args, **kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/webclient/webclient_gateway.py", line 2109, in handle_exception
    super(OmeroWebSafeCallWrapper, self).handle_exception(e, *args, **kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 4856, in __call__
    return self.f(*args, **kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero_api_RenderingEngine_ice.py", line 1192, in load
    return _M_omero.api.RenderingEngine._op_load.invoke(self, ((), _ctx))

omero.ResourceError: exception ::omero::ResourceError
{
    serverStackTrace = ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
	at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:907)
	at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
	at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
	at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
	at jdk.internal.reflect.GeneratedMethodAccessor345.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy92.doWork(Unknown Source)
	at ome.services.util.Executor$Impl.execute(Executor.java:447)
	at ome.services.util.Executor$Impl.execute(Executor.java:392)
	at ome.services.RenderingBean.getPixelBuffer(RenderingBean.java:2202)
	at ome.services.RenderingBean.load(RenderingBean.java:417)
	at jdk.internal.reflect.GeneratedMethodAccessor1572.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy136.load(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor1572.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy136.load(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor1711.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172)
	at ome.services.throttling.Callback.run(Callback.java:56)
	at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
	at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140)
	at ome.services.blitz.impl.RenderingEngineI.load_async(RenderingEngineI.java:316)
	at jdk.internal.reflect.GeneratedMethodAccessor1710.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at omero.cmd.CallContext.invoke(CallContext.java:85)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy138.load_async(Unknown Source)
	at omero.api._RenderingEngineTie.load_async(_RenderingEngineTie.java:248)
	at omero.api._RenderingEngineDisp.___load(_RenderingEngineDisp.java:1223)
	at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:2405)
	at IceInternal.Incoming.invoke(Incoming.java:221)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
	at Ice.ConnectionI.message(ConnectionI.java:1056)
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.base/java.lang.Thread.run(Thread.java:829)

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
}

<WSGIRequest: GET '/webclient/metadata_preview/image/5025551/?_=1685379192144'>

but the symlinks look OK:

$ ls -alh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/
total 0
drwxrwsr-x. 2 omero-server omero-server  31 May 29 13:00 .
drwxrwsr-x. 3 omero-server omero-server  50 May 18 09:50 ..
lrwxrwxrwx. 1 omero-server omero-server 109 May 29 13:00 Tonsil 1.ome.zarr -> /bia-integrator-data/S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr

$ cat /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil\ 1.ome.zarr/OME/METADATA.ome.xml 

<?xml version="1.0" encoding="UTF-8"?><OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2016-06" xmlns:xsi="http://www.w3.o....

This was referenced May 29, 2023
Open
Closed
@will-moore will-moore self-assigned this May 29, 2023
@will-moore
Copy link
Member Author

Reverted the symlinks to previous values and that fixed the rendering in webclient:
NB: had to comment out # preview_image(conn, fileset_id, args) in the script since that crashed with the ResourceError above.

$ python idr-utils/scripts/managed_repo_symlinks.py Dataset:5601 /idr0054/zarr/ --report

Fileset: 5287119 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/
fileset_dirs {}
fs_contents ['Tonsil 1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr to /idr0054/zarr/Tonsil 1.ome.zarr

Fileset: 5287120 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/
fileset_dirs {}
fs_contents ['Tonsil 2.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/Tonsil 2.ome.zarr to /idr0054/zarr/Tonsil 2.ome.zarr

Fileset: 5287121 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/
fileset_dirs {}
fs_contents ['Tonsil 3.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/Tonsil 3.ome.zarr to /idr0054/zarr/Tonsil 3.ome.zarr

@will-moore
Copy link
Member Author

I don't see any differences between what's in those locations:

$ ls -alh /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil\ 1.ome.zarr/
total 17K
drwxr-xr-x. 2 root root 4.0K May 18 09:06 .
drwxr-xr-x. 2 root root 4.0K May 18 09:18 ..
drwxr-xr-x. 2 root root 4.0K May 18 09:18 0
drwxr-xr-x. 2 root root 4.0K May 18 09:18 OME
-rw-r--r--. 1 root root   33 May 18 09:06 .zattrs
-rw-r--r--. 1 root root   23 May 18 09:06 .zgroup

$ ls -alh /bia-integrator-data/S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr
total 17K
drwxr-xr-x. 2 root root 4.0K May 19 16:50 .
drwxr-xr-x. 2 root root 4.0K May 29 11:11 ..
drwxr-xr-x. 2 root root 4.0K May 29 11:11 0
drwxr-xr-x. 2 root root 4.0K May 19 16:50 OME
-rw-r--r--. 1 root root   33 May 19 16:49 .zattrs
-rw-r--r--. 1 root root   23 May 19 16:49 .zgroup

Checked a bunch of other dirs too - couldn't see any differences.

@will-moore
Copy link
Member Author

will-moore commented Jun 5, 2023 •
edited
Loading

The Aspera instructions on BioStudies:

The Aspera ascp command line client is distributed as part of the Aspera CLI high-performance transfer browser plug-in which can be downloaded from here
You will need to execute the downloaded file and switch to the CLI folder, e.g.,

$ sh ~/Documents/ibm-aspera-cli-3.9.6.1467.159c5b1-mac-10.11-64-release.sh
$ cd ~/Applications/Aspera\ CLI/bin
$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d <directory to upload> [email protected]:<secret directory>

where - -P33001 and [email protected] defines port, user and server for Aspera connection. - is the path of the directory to be uploaded. - looks like xx/xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-xxxx and is shown above.

The download link there contains OSX but even if I remove that I only see OSX options to download. This will make it hard to install on the CentOS7 machines where we want to do the conversions etc.

@jburel pointed me at https://github.com/Euro-BioImaging/BatchConvert/
They upload to BIA via Aspera. The ascp cli tool is installed via conda:

https://github.com/Euro-BioImaging/BatchConvert/blob/c3a89791416babd2681b718ba0f6a043f9249d7a/dependencies/Dockerfile#L26

I followed that manually...

conda create --yes --name ascp2bia python=3.9
conda install -n ascp2bia -c hcc aspera-cli

  aspera-cli         hcc/osx-64::aspera-cli-3.9.6-hbec0bfc_0 None
You must agree to the license before using aspera-cli, http://www-03.ibm.com/software/sla/sladb.nsf/displaylis/2ABD3328689EBBFE8525830C007A5F38.

I copied the file at https://github.com/Euro-BioImaging/BatchConvert/blob/c3a89791416babd2681b718ba0f6a043f9249d7a/configs/asperaweb_id_dsa.openssh
and saved it locally. This is normally included in the download of Aspera above.

$ conda activate ascp2bia
$ which ascp
/Users/wmoore/opt/anaconda3/envs/ascp2bia/bin/ascp

$ ascp -P33001 -i ~/.ssh/asperaweb_id_dsa.openssh -d Desktop/images/lif [email protected]:**/*******************
Killed: 9

This might be failing because I haven't agreed to the license. Tried following http://www-03.ibm.com/software/sla/sladb.nsf/displaylis/2ABD3328689EBBFE8525830C007A5F38. but that redirects to License Information documents https://www.ibm.com/support/customer/csol/terms/#license-search with no clear info on what I need to do. Just looks like documents for reference, not any license agreement form?

@will-moore
Copy link
Member Author

will-moore commented Jun 5, 2023 •
edited
Loading

Signed up for IBM account and logged-in, but that didn't change what I saw at links above.
Downloaded according to instructions from BioStudies...

$ cd Documents/Aspera/
$ sh ibm-aspera-cli-3.9.6.1467.159c5b1-mac-10.11-64-release.sh

Installing IBM Aspera CLI

Installation into /Users/wmoore/Applications/Aspera CLI successful

Optional installation steps:

  To include aspera in your PATH, run this command (or add it to .bash_profile):
    export PATH=/Users/wmoore/Applications/Aspera\ CLI/bin:$PATH

  To install the man page, run the following command:
    export MANPATH=/Users/wmoore/Applications/Aspera\ CLI/share/man:$MANPATH

$ which ascp
/Users/wmoore/Applications/Aspera CLI/bin/ascp

$ cd ~/Applications/Aspera\ CLI/bin
$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d ~/Desktop/images/lif/ [email protected]:xx/xxxxxxxxxxxxxxxxxxxxxxx
20191025 Test FRET 585. 423, 426.lif                                                                                                                         100%   47MB  175Mb/s    00:02    
Completed: 48157K bytes transferred in 3 seconds
 (130222K bits/sec), in 1 file, 1 directory.

Success!

@will-moore
Copy link
Member Author

I see that you can choose the OS in the BioStudies page to get the correct link for e.g. Linux:

On idr0125-pilot:

$ wget https://ak-delivery04-mul.dhe.ibm.com/sar/CMA/OSA/08q6g/0/ibm-aspera-cli-3.9.6.1467.159c5b1-linux-64-release.sh

Installing IBM Aspera CLI

Installation into /home/wmoore/.aspera/cli successful

Optional installation steps:

  To include aspera in your PATH, run this command (or add it to .bash_profile):
    export PATH=/home/wmoore/.aspera/cli/bin:$PATH

  To install the man page, run the following command:
    export MANPATH=/home/wmoore/.aspera/cli/share/man:$MANPATH

$ cd /.aspera/cli/bin
$ ./ascp -P3..... # worked as above!

@will-moore
Copy link
Member Author

For idr0054 on pilot-idr0125, to debug symlinking to BioStudies s3, I updated the symlinks as above:

managed_repo_symlinks.py 
$ python idr-utils/scripts/managed_repo_symlinks.py Dataset:5601 /bia-integrator-data --fileset-mappings /tmp/idr0054_fileset_mappings.csv --report

Fileset: 5287119 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/
Render Image 5025551
fileset_dirs {'Tonsil 2.ome.zarr': 'S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr', 'Tonsil 1.ome.zarr': 'S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr', 'Tonsil 3.ome.zarr': 'S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr'}
fs_contents ['Tonsil 1.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr to /bia-integrator-data/S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr

Fileset: 5287120 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/
Render Image 5025552
fileset_dirs {'Tonsil 2.ome.zarr': 'S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr', 'Tonsil 1.ome.zarr': 'S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr', 'Tonsil 3.ome.zarr': 'S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr'}
fs_contents ['Tonsil 2.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628/Tonsil 2.ome.zarr to /bia-integrator-data/S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr

Fileset: 5287121 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/
Render Image 5025553
fileset_dirs {'Tonsil 2.ome.zarr': 'S-BIAD704/36cb5355-5134-4bdc-bde6-4e693055a8f9/36cb5355-5134-4bdc-bde6-4e693055a8f9.zarr', 'Tonsil 1.ome.zarr': 'S-BIAD704/5583fe0a-bbe6-4408-ab96-756e8e96af55/5583fe0a-bbe6-4408-ab96-756e8e96af55.zarr', 'Tonsil 3.ome.zarr': 'S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr'}
fs_contents ['Tonsil 3.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-05/18/09-51-13.702/Tonsil 3.ome.zarr to /bia-integrator-data/S-BIAD704/3b4a8721-1a28-4bc4-8443-9b6e145efbe9/3b4a8721-1a28-4bc4-8443-9b6e145efbe9.zarr

$ tail -f /opt/omero/server/OMERO.server/var/log/Blitz-0.log

Resource Error Blitz.log 
2023-06-05 13:08:35,767 INFO  [                 org.perf4j.TimingLogger] (l.Server-0) start[1685970515763] time[3] tag[omero.call.success.ome.services.RenderingBean$7.doWork]
2023-06-05 13:08:35,767 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Rslt:	(ome.model.enums.RenderingModel:rgb:Id_1, ome.model.enums.RenderingModel:greyscale:Id_2)
2023-06-05 13:08:35,767 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Executor.doWork -- ome.services.RenderingBean.getPixelBuffer[]
2023-06-05 13:08:35,767 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Args:	[null, InternalSF@441482912]
2023-06-05 13:08:35,768 INFO  [    ome.security.basic.BasicEventContext] (l.Server-0)  cctx:	group=3
2023-06-05 13:08:35,769 INFO  [         ome.security.basic.EventHandler] (l.Server-0)  Auth:	user=52,group=3,event=null(User),sess=a24936e0-1b10-40c9-b1a9-a95101791d8e
2023-06-05 13:08:35,770 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-0) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
2023-06-05 13:08:35,984 DEBUG [                   loci.formats.Memoizer] (l.Server-0) memo(lastModified=1684404160866) older than real(lastModified=1684515006000)
2023-06-05 13:08:36,038 INFO  [                loci.formats.ImageReader] (l.Server-0) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
2023-06-05 13:08:36,267 DEBUG [                   loci.formats.Memoizer] (l.Server-0) start[1685970515772] time[494] tag[loci.formats.Memoizer.setId]
2023-06-05 13:08:36,268 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-0) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
2023-06-05 13:08:36,269 ERROR [                ome.io.nio.PixelsService] (l.Server-0) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1
	at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
	at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
	at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
	at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
	at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
	at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
	at jdk.internal.reflect.GeneratedMethodAccessor345.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy92.doWork(Unknown Source)
	at ome.services.util.Executor$Impl.execute(Executor.java:447)
	at ome.services.util.Executor$Impl.execute(Executor.java:392)
	at ome.services.RenderingBean.getPixelBuffer(RenderingBean.java:2202)
	at ome.services.RenderingBean.load(RenderingBean.java:417)
	at jdk.internal.reflect.GeneratedMethodAccessor1572.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy136.load(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor1572.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy136.load(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor1711.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172)
	at ome.services.throttling.Callback.run(Callback.java:56)
	at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
	at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140)
	at ome.services.blitz.impl.RenderingEngineI.load_async(RenderingEngineI.java:316)
	at jdk.internal.reflect.GeneratedMethodAccessor1710.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at omero.cmd.CallContext.invoke(CallContext.java:85)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy138.load_async(Unknown Source)
	at omero.api._RenderingEngineTie.load_async(_RenderingEngineTie.java:248)
	at omero.api._RenderingEngineDisp.___load(_RenderingEngineDisp.java:1223)
	at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:2405)
	at IceInternal.Incoming.invoke(Incoming.java:221)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
	at Ice.ConnectionI.message(ConnectionI.java:1056)
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: Invalid index: -1
	at loci.formats.FormatReader.coreIndexToSeries(FormatReader.java:1328)
	at loci.formats.FormatReader.getSeriesCount(FormatReader.java:963)
	at loci.formats.MetadataTools.populatePixels(MetadataTools.java:137)
	at loci.formats.MetadataTools.populatePixels(MetadataTools.java:116)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:289)
	at loci.formats.FormatReader.setId(FormatReader.java:1443)
	at loci.formats.ImageReader.setId(ImageReader.java:849)
	at ome.io.nio.PixelsService$3.setId(PixelsService.java:869)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.Memoizer.setId(Memoizer.java:690)
	at ome.io.bioformats.BfPixelsWrapper.<init>(BfPixelsWrapper.java:52)
	at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73)
	... 82 common frames omitted
2023-06-05 13:08:36,271 INFO  [                 org.perf4j.TimingLogger] (l.Server-0) start[1685970515767] time[503] tag[omero.call.exception]
2023-06-05 13:08:36,271 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Excp:	ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
2023-06-05 13:08:36,272 INFO  [                 org.perf4j.TimingLogger] (l.Server-0) start[1685970515759] time[512] tag[omero.call.exception]
2023-06-05 13:08:36,272 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Excp:	ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
2023-06-05 13:08:36,279 INFO  [        ome.services.util.ServiceHandler] (l.Server-9)  Meth:	interface ome.api.StatefulServiceInterface.close
2023-06-05 13:08:36,279 INFO  [        ome.services.util.ServiceHandler] (l.Server-9)  Args:	()
2023-06-05 13:08:36,279 INFO  [                 org.perf4j.TimingLogger] (l.Server-9) start[1685970516279] time[0] tag[omero.call.success.ome.services.RenderingBean.close]
2023-06-05 13:08:36,279 INFO  [        ome.services.util.ServiceHandler] (l.Server-9)  Rslt:	null
2023-06-05 13:08:36,279 INFO  [                      omero.cmd.SessionI] (l.Server-9) Unregistered servant:a24936e0-1b10-40c9-b1a9-a95101791d8e/5f20a8a7-a2ec-4982-82a3-f720bcc27a3comero.api.RenderingEngine(omero.api._RenderingEngineTie@88be2b14)
2023-06-05 13:08:36,281 INFO  [o.services.sessions.SessionContext$Count] (l.Server-6) -Reference count: a24936e0-1b10-40c9-b1a9-a95101791d8e=0
2023-06-05 13:08:36,281 INFO  [                      omero.cmd.SessionI] (l.Server-6) cleanupSelf(ServiceFactoryI(session-33debeb7-69e1-44ef-95c6-fde7f78d243e/a24936e0-1b10-40c9-b1a9-a95101791d8e)).
2023-06-05 13:08:36,281 INFO  [                      omero.cmd.SessionI] (l.Server-6) Unregistered servant:a24936e0-1b10-40c9-b1a9-a95101791d8e/33debeb7-69e1-44ef-95c6-fde7f78d243eomero.api.IContainer(omero.api._IContainerTie@8a31d1da)

cc @dgault @sbesson Any ideas what might be causing this error? java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1 at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)

@will-moore
Copy link
Member Author

will-moore commented Jun 16, 2023 •
edited
Loading

Meeting with Matthew today:

  • Need to create an IDR account for us to upload to BioStudies and submit. Use IDR e-mail account, which one?
  • Upload zips to a directory within the "secret directory" on BioStudies. E.g. /idr0010/ - Can continue to add to this dir until submission is complete (doesn't need to all go in one FTP command).
  • Need to list each .zarr.zip in the studies tsv for submission.
  • We'll get a mapping from each image.zarr to uuid.zarr on s3, which we can use to generate correct symlinks.
  • Need to decide on consistent naming strategy for Submissions. E.g. idr0054 NGFF ....
  • Create links of type: "Image data resource" in links section of Submission to Study in IDR, e.g. idr.openmicroscopy.org/webclient?show=project-1 so these can be searched for
  • Our EBI s3 buckets should be deleted once not needed.
  • bioformats2raw images in idr0054 are 'greyscale' in omero settings?
  • Need to list/check which studies have existing submissions in BioStudies - link to them from IDR submission
  • EBI will be able to delete my idr0054 submission - replace with new one from IDR account

@will-moore
Copy link
Member Author

Discussion with Josh and Matthew on slack about "zip zarrs": https://openmicroscopy.slack.com/archives/C01ABANKH8V/p1687170670769479
Safer to stick with zipping the image.zarr directory itself rather than trying to create a zip zarr by zipping the contents (less well tested at EBI).

@will-moore will-moore moved this from Zip and upload to BioStudies to convert all data to NGFF in NGFF conversion Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@will-moore will-moore

Labels
None yet
Projects
NGFF conversion
Status: convert all data to NGFF
Milestone
No milestone
Development

No branches or pull requests
1 participant
@will-moore