From 36cb2934c0b6194b9c3b9c7e4255c5a4fe35a6ba Mon Sep 17 00:00:00 2001 From: William Moore Date: Wed, 17 Jan 2024 06:54:38 +0000 Subject: [PATCH] Add first 2 notes of 2024 --- 2024/2024-01-09.md | 71 ++++++++++++++++++++++++++++++++++++++++++++ 2024/2024-01-16.md | 74 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+) create mode 100644 2024/2024-01-09.md create mode 100644 2024/2024-01-16.md diff --git a/2024/2024-01-09.md b/2024/2024-01-09.md new file mode 100644 index 0000000..2cd504f --- /dev/null +++ b/2024/2024-01-09.md @@ -0,0 +1,71 @@ +--- +tags: Tuesday +--- + +# 2024-01-09 + +Attending: J-M, Petr, Seb, Chris, Jason, Andreas, Kyle, Kevin, Melissa, David, Khaled, June, Tom, Erin, Dom + +Start: 2:00 pm UK + +## Accepting minutes from [last meeting](https://hackmd.io/team/ome?nav=overview) + +## Project Status + +(2-3 minutes each) + +- IDR + - NGFF upgrade: + - Will: pending re-processing of a handful of NGFF Plates by BioStudies, I have hosted these on our own uk1s3 buckets and [used these from idr-next](https://github.com/IDR/idr-metadata/issues/675#issuecomment-1878828866) so all NGFF data should now be valid. + - Will: I have [run check_pixels.py](https://github.com/IDR/idr-utils/pull/55#issuecomment-1879859634) on 1 image per Plate or 2 images per Dataset across ALL data on idr-next. I see various Exceptions with `getPlane()` but the only mismatching pixels detected were from idr0009. Seb: "The problem is that this data was originally loaded with a version of the ScanR reader that represents missing wells as black images while the default behavior in Bio-Formats is to represent these as sparse plates. When this was backported to OME Bio-Formats, an option was introduced to support both behaviors - see https://bio-formats.readthedocs.io/en/latest/formats/options.html +I think the action is to update all plates in this study to set scanr.skip_missing_wells to false , delete & regenerate the memo files and retest". Will: [tried this](https://github.com/IDR/idr-metadata/issues/675#issuecomment-1881406444) but no joy. + - Also Petr found that idr0001 images fail (except for the *first* Acquisition of each Plate), so we either need to convert to NGFF or reopen [Flex: merge IDR changes](https://github.com/ome/bioformats/pull/3537). Will: converted one Plate but [has issues with Field count per Well](https://github.com/IDR/idr-metadata/issues/683#issuecomment-1882669352). Needs more testing etc. Conversion of 192 Plates might take 6 days with bioformats2raw (don't know with omero-cli-zarr) and total 9TB. We probably don't want to block release for this (TBD)? + - J-M: to be discussed tomorrow after standup and make decisions. + - Seb: storage in good place. Working on upgrading IDR to Rocky Linux 9. Plan for Q1 + - Jason: Systems came back after power outage ahead of time. TOO: J-M add links to mirror in our doc for future reference. + + +- OMERO + - Issue with publication of ansible workflow to ansible galaxy: see [discussion](https://github.com/ome/ansible-role-postgresql-client/pull/13) + - rename repository or + - re-activate own GHA with workaround + + - Upgrading [omero-test-infra](https://github.com/ome/omero-test-infra/pull/66) so we can release applications like omero-figure + - Upgrade page for migrating systems + - OMERO.web looking at open PRs and cleaning up backlog. + +- Bio-Formats + - Melissa: PRs review next week for release timeline. Backlog is "acceptable" state. + - David: Reviewing state to plan release timeline for ZarrReader to include 0.5 spec + +- NGFF + - cf. ZarrReader timeline + +- GerBI + - Tom: Working KVP scripts. Presentation next week of the current status. No progress on Web + +- Glencoe + - Work on OMERO.tables + - Documentation + - API changes: dependencies issues using OMERO.py (client vs server ones). + - Review requirements in OMERO.web. Better minimum for dependencies + - Seb: CentOS 7/RHEL 7 upgrade + - Seb: Python 3.12 support coming, Python 3.8 EOL in 9 months + - Erin: Upcoming Harvard tissue workshop end of January: OME.zarr (public data) with Webnossos+OMERO plus. + - presentation after workshop + +- Community + - Continuing efforts with UGM team + - no issue reported so far with the migration + - More tests to be done by Dewa. + - Grants submitted for working with HCS data (Wellcome), IDR at PSC (NIH), in progress for IDR maintenance (NIH). + - Contact from Erick Ratamero and Ken Ho-- alternative hosts for OME meeting. + - looking at options + +## AOB + +(5 min. max; tech. Discussion should be highlighted to relevant people and rescheduled) + +## Main Topic + +(20-25 minutes plus 15 minutes questions max) diff --git a/2024/2024-01-16.md b/2024/2024-01-16.md new file mode 100644 index 0000000..49a6152 --- /dev/null +++ b/2024/2024-01-16.md @@ -0,0 +1,74 @@ +--- +tags: Tuesday +--- + +# 2024-01-16 + +Attending: Josh, Will, Dom, David Gault, Chris, Frances, Andreas, Khaled, David Stirling, June, Melissa, Jena Wendt (Münster), Tom, Seb, Kyle, Kevin, Maadhav, Petr, Jean-Marie, Jason + +Start: 2:00 pm UK + +## Accepting minutes from [last meeting](https://hackmd.io/team/ome?nav=overview) + +## Project Status + +(2-3 minutes each) + +- IDR: + - Next release: idr0138 (NGFF) and idr0154 + - idr0138 imported over holidays, ROIs now created - rendering settings needed + - idr0154 import will be quick + - NGFF release: "stress testing" this morning. Some images felt like they are loading a bit slower than on IDR, but difficult to quantify. Performance seemed to improve with more browsing (caching??). + - idr0011 Plate failing, due to renaming/moving on BIA s3. + - Various other plates resubmitted to BIA are now available (we have hosted these on our custom s3 buckets in the meantime for idr-next). Now looking at updating these to use BIA data... + - Running mkngff again to update NGFF Fileset will add `_mkngff` to templatePrefix, e.g. `demo_2/2016-07/28/21-44-02.006_mkngff_mkngff`. + - Some images didn't load, but appeared OK when re-loaded. + +- OMERO + - upcoming release omero-py: 3.8 Python is the minimum version. + - Fixes test-infra (Pillow version issue) + +- Bio-Formats (David) + - PR reviews - + - Aim for 1st week of Feb for next 7.2.0 release (or 2 weeks later if date is missed) + - Nico Quick czi reader progressing + - ZarrReader moving towards NGFF v0.5 spec support + - will keep changes up to date with spec changes + +- NGFF + - No word on transforms. Time limit/cutoff? + - implementations progressing, python, JavaScript, etc + - We should look at ome-ngff-validator (schemas now available) + - [RFC](https://forum.image.sc/t/ngff-rfc-process-proposal-draft-now-available/90181) proposal opened. Next step: identifer reviewers. + - Additionally [ZEP0](https://zarr.dev/zeps/active/ZEP0000.html) is also undergoing review. Hopefully get this signed off by spring. + - Starting [Office hours](https://forum.image.sc/t/ome-files-office-hours-wednesdays-at-alternative-times/90850) to restart the website meetings + - Will: currently open issues? Josh: ping me URLs or we can meet at a different time? + - Upcoming questions around implementations (e.g., ilastik) + +- Glencoe Chris + - Django 5.0 released. More async features, some things deprecated. Requires python 3.10. + - omero-web cleanup needed, old apis, interactions etc. + - Seb: python 2 compatibility code cleanup, 'future' etc. + - character encoding in NGFF, reserved chars, unicode etc. How to handle during conversion? + +- Community + - Josh: "OME data compatibility", next Wednesday (the 24th), 10AM EST/3PM UK/4PM EU + - Ping Josh if you need invite/more info? + +- GerBI (Tom) + - Key-Value pair scripts... See https://github.com/German-BioImaging/omero-scripts/tree/xtnd_support_kvpairs + - Editing exisitng KV scripts + - Users start KV workflow in webclient, but doesn't work for SPW Wells. + - Single script to handle CSV -> P/D/I S/P/A/W and vice-versa (export KVP to csv) + - Also need Delete KVP script + - Namespace update "editable" vv protected from client editing + - Also option to add Tags instead of KVP + - Jens: Q: using Tags - should we try to use existing Tags? - input option for users to choose + +## AOB + +(5 min. max; tech. Discussion should be highlighted to relevant people and rescheduled) + +## Main Topic + +(20-25 minutes plus 15 minutes questions max)