Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable chunk processing #63

Merged
merged 9 commits into from
Sep 3, 2024
Merged

Enable chunk processing #63

merged 9 commits into from
Sep 3, 2024

Conversation

KedoKudo
Copy link
Contributor

Description of Pull Request

This PR introduces the following changes:

  • The command line tool Sophiread now process an input tpx3 file chunk by chunk.
  • The write to HDF5 is now appending, allow multiple chunks from the same tpx3 file accumulated in the same HDF5 archive.
  • Track timing signals (TDC, GDC) for each chunk, useful for runtime diagnostics.

EWM item: Defect 7012

Known issue: the chunking is now done by size, not packet, so we are losing the packet that got split between the chunks. This will affect the total number of hits and neutrons slightly, and using the largest chunk size allowed by the system ram can help reduce its negative impact.

Purpose of work

The raw tpx3 file collected with VENUS tpx3 detector is a lot larger than those from HFIR, and the system is not capable of store all bytes in its ram for processing, hence the chunked processing is necessary.

Summary of work

N/A

Additional detail of work

N/A

Testing instructions

  • Build the feature branch within the conda dev environment.
  • Run unit tests with ctest -V --output-on-failure
  • Run the test to process with default chunk size (5G), i.e. ./Sophiread -i data/HV2700_1500_500_THLrel_274_sophy_chopper_60Hz_4.1mm_aperture_siemen_star_120s_000000.tpx3 -v
image
  • Run the test to process with 10 MB chunk size, i.e. ./Sophiread -i data/HV2700_1500_500_THLrel_274_sophy_chopper_60Hz_4.1mm_aperture_siemen_star_120s_000000.tpx3 -H hits.h5 -E neutrons.h5 -c 10 -v
image
  • Run the test to output tof images with ./Sophiread -i data/HV2700_1500_500_THLrel_274_sophy_chopper_60Hz_4.1mm_aperture_siemen_star_120s_000000.tpx3 -c 10 -T tiff_out -v
image

@KedoKudo KedoKudo self-assigned this Aug 30, 2024
@KedoKudo KedoKudo merged commit 02818a4 into next Sep 3, 2024
2 checks passed
@KedoKudo KedoKudo deleted the troubleshoot_commission branch September 3, 2024 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants