Stream RRDP files #127
Labels
Comments
ydahhrk
added a commit
that referenced
this issue
Oct 8, 2024
For #82. It seems I'm finally done making dramatic wide-reaching changes to the codebase. There's still plenty to add and test, but I would like to start pushing atomic commits from now on. This is a squashed version of development brach "issue82". It includes a few merges with main. ``` cache/ rsync/ # rsync refresh 0/ # rsync module: rsync://a.b.c/mod1 rpp1/ # Repository Publication Point 1 d.mft d.crl d1.roa rpp2/ e.mft e.crl ta.cer # Trust Anchor 1/ # rsync module: rsync://a.b.c/mod2 ... 2/ # rsync module: rsync://x.y.z/mod1 ... https/ # HTTPS refresh 0 # https://a.b.c/repo/ta.cer 1 # https://x.y.z/repo/ta.cer rrdp/ # RRDP refresh 0/ # https://m.n.o/notification.xml 0 # rsync://m.n.o/mod1/rpp1/a.mft 1 # rsync://m.n.o/mod1/rpp1/b.crl 2 # rsync://m.n.o/mod1/rpp2/c.mft 3 # rsync://m.n.o/mod1/rpp2/d.cer 1/ # https://p.q.r/notification.xml ... fallback/ # Committed RPPs 0/ # Fallback of rsync://a.b.c/mod1/rpp1 0 # Hard link: cache/rsync/0/rpp1/d.mft 1 # Hard link: cache/rsync/0/rpp1/d.crl 2 # rsync://a.b.c/mod1/rpp1/d2.roa # (Unique because of refresh) 1 # Hard link: cache/https/0 2/ # Fallback of m.n.o's rsync://m.n.o/mod1/rpp1 0 # Hard link: cache/rrdp/0/0 1 # Hard link: cache/rrdp/0/1 index.json # URL/path mappings and some metadata ``` - `cache/rsync`, `cache/https` and `cache/rrdp` contain "refreshes" (the exact latest files according to the servers). RRDP withdraws are honored, and rsyncs run without --compare-dest. - "Refresh" files marked as valid are backed up in `cache/fallback` at the end of each validation cycle. - Validation first tests fallback+refresh. (If a file exists in both, refresh wins.) If that fails, it retries with fallback only. - The index is not a tree; everything is caged in numbered directories and indexed by exact URL, to prevent file overriding by URL hacking. There's also a `cache/tmp` directory, where Fort temporarily dumps notifications, snapshots and deltas. This directory will be removed once #127 is fixed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
"Download file into cache, use it, then delete it".
The storage step is completely unnecessary.
Fort should be downloading the files into memory in chunks, and decoding them in place.
Applies to RRDP Notifications, Snapshots and Deltas.
The text was updated successfully, but these errors were encountered: