dataforge: selfcap dataset (exoego-self-collected) - #113
Open
pablo-rerun wants to merge 4 commits into
Open
Conversation
…cond dataset Nine synchronized streams per episode on one episode-relative video_time axis: four exo iPhones (loose intrinsics, empty extrinsics — logged as world-anchored rigs with no transform until a calibration/SLAM layer localizes them), the OAK ego rig (rgb + stereo with real brown_conrady distortion via PinholeWithDistortion, dev IMU gyro/accel columnar), and the Quest rig with its head pose as a temporal world_T_rig (so this base carries a live 3D view and owns the root ViewCoordinates, Y-up). All streams ride the shared re-encode PTS grid: the ego per-frame csvs cannot retime the ego mp4s (the cut step resampled 24 fps content onto the exo grid, ~50% show_existing_frame duplicates), so they are surfaced as native-rate AnyValues instead, with a TODO for a de-dup retime lens. Unreadable episodes (NAS mode-000 sync artifact, 1083 of 4168) skip with a warning; quest/body/hand poses, magnetometer, and the found-but- unvalidated quest/ego extrinsics are deliberate TODOs. Also: the convert verb's batch loop now survives per-sequence failures (prints FAILED, continues, exits non-zero at the end) — the robocap s10 lesson.
…ego quest) Blueprints cannot select entities by the rigs' kind AnyValues, so the exo/ego grouping travels through the pane list: the layout now reads as an Exo 2x2 grid over labeled OAK and Quest rows beside the 3D scene, instead of one undifferentiated camera grid. Data contract unchanged (flat rig_NN paths + kind AnyValues per exoego:v2); path-level exo/ego grouping is an exoego:v3 question, deliberately not taken here.
…t, exo bottom row) Layout only, schema untouched: dominant 3D scene with the ego views (OAK + Quest) in a two-column grid beside it, exo cameras as a bottom row, IMU strip beneath — mirroring simplecv's proven view_exoego arrangement. The 3D pane grows into its space once a calibration/SLAM layer localizes the exo rigs.
…st metadata, discover() Review (adversarial, full-stack) found selfcap had copy-mutated robocap primitives, silently dropping behavior three times: the exoego:v2 rig-node key set forked (name/kind vs reference/num_cameras), the spec'd static transform on the imu node was lost, and the blueprint + num_cameras could describe cameras that were never logged. Fixes: - logging_toolkit.py: log_rig_node (one rig-node contract; robocap's reference corrected to imu_00 — the rig origin is dev0's IMU frame, not cam_00), log_video_stream (one Mp4Reader tap, shift_ns for the boot-clock retime, invariants documented once), ImuChannel + log_imu (imu-node transform impossible to forget). exoego_schema.md legitimizes optional rig-level name/kind. - selfcap plan-then-log: EpisodePlan built before the recording opens, so panes == logged cameras == num_cameras by construction (a quest camera without intrinsics now honestly yields 8). robocap computes video epochs once instead of twice per segment. - writing.atomic_write extracted; register.py's .rbl now writes atomically (a live catalog server may hold the old file open). - Dataset name is a ClassVar single source of truth; dataset_key's isinstance-scan deleted. - discover() returns (identity, source) pairs; _locate and the redundant readability re-checks deleted — convert never inverts an identity back into a path. - send_capture_properties + paths.BASE_LAYER replace four copies of the property ceremony and the layer literal. README updated to reality. 40 tests green; both corpora regenerated and re-registered; pixel-verified.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #112. The second dataforge dataset:
main/cutof the self-collected exo+ego corpus (41 days, 692 sessions, 4,168 episodes — 3,085 currently readable pending a NAS chmod).Per episode, one base rrd with nine synchronized streams on an episode-relative
video_time:rig_00..03, plain Pinhole (upstream intrinsics are placeholders:available: false, fx=fy=700), no transforms (extrinsics empty upstream; a calibration/SLAM layer localizes them later).rig_04rgb + stereo with realbrown_conradydistortion (PinholeWithDistortion), IMU gyro/accel columnar.rig_05with its head pose as temporalworld_T_rig(base has a live 3D view; base owns root ViewCoordinates, Y-up). Body/hand poses deferred as TODOs.Key finding baked into the design: the cut step re-encoded every stream onto one shared PTS grid (ego 24 fps content resampled, ~50% duplicate frames), so the ego per-frame csvs cannot retime the mp4s — all streams trust PTS, and the csvs surface as
num_native_frames/native_duration_nsmetadata with a de-dup-lens TODO.Also fixes the convert verb's batch loop to survive per-sequence failures (the robocap s10 lesson).
Validated: 3 episodes (different days, frame-rate regimes 25/50/59.7 fps, and exo device sets) converted and registered into a
selfcapcatalog dataset; pixel-verified from the catalog URI — all 9 panes decoding, quest frustum animating under its head pose, IMU plots live. Gates green (37 tests).