httpreader: random access over an HTTP resource by hdonnay · Pull Request #1953 · quay/claircore

hdonnay · 2026-07-17T18:41:07Z

This adds a package (internal/httpreader) that implements io.ReaderAt over HTTP requests. It incorporates a lot of tricks I know from a past life dealing with RFC7233 non- and selectively- compliant servers.

Then, the fetcher package gains the capability to use an httpreader.Reader when it notices that a layer is uncompressed.

Future work may involve the tarfs layer doing transparent caching and being able to handle compressed layers directly.

hdonnay · 2026-07-17T18:42:07Z

Conflicts with #1948

BradLugo

I think there's enough to address for now. I'll do another deep dive in the next review

BradLugo · 2026-07-27T23:06:31Z

+			for s.Scan() {
+				fs := strings.FieldsFunc(s.Text(), func(r rune) bool { return r == ' ' || r == '=' })
+				if fs[1] == "size" {
+					// This is *not* the stringified octal madness of a real header.
+					*p, err = strconv.ParseInt(fs[2], 10, 64)
+					if err != nil {
+						return nil, parseErr("bad block at %d: weird PAX header: bad size: %v", off, err)
+					}
+				}
+			}


I've never really looked at the pax spec before, but after briefly reading about it, it seems like we'd at least need to break after finding size so we don't hit one of the extended headers, which might cause a an error: https://pubs.opengroup.org/onlinepubs/009695399/utilities/pax.html#tag_04_100_13_03

I'm not sure what you mean. This is handling the extended headers.

It seems that we should parse size first, then read size bytes, instead of using Scan(). Otherwise, we might read junk (I think), like too few bytes because we reached the limit of Scan() or because we hit a newline, etc

This is using a SectionReader to effectively "fence" the Scanner to just the size bytes of the extended header records. The Scanner is nice because it takes care of short reads as long as the line fits in the buffer, which is 4KiB by default.

BradLugo

More questions

BradLugo · 2026-08-18T14:23:04Z

+				// Without this, the global size would only take effect after
+				// the next regular file.
+				xSize = gSize
+			}


I think there might be a really subtle issue here. Since we aren't resetting cur for tar.TypeXGlobalHeader anymore, I think we would skip the header we're actually looking for (after the global header): https://github.com/golang/go/blob/98e6631a5aa274aadc7f5d8c390057fca50368fb/src/archive/tar/reader.go#L106-L114

I don't see it. Assume a tar stream like:

global (size = 0)

extended (size = n)

reg "A"

file data

reg "B"

EOF

Then we should emit two sections:

[ global, extended, reg "A" (size = n) ]

[ reg "B" (size = 0) ]

Will the the stdlib tar decoder freak out because of the global size? Maybe, but I don't see how we skip a "regular" header.

I think the alternative is to ignore the global headers and say they don't make sense for setting the size, which is all our code cares about.

According to the findSegments function doc:

Each returned segment describes a region that is not a complete tar file, but can have exactly one file read from it.

Which, after finding the usage of findSegments, in my own words, seems to be: "each returned segment will have exactly one header that describes the file":

claircore/pkg/tarfs/tarfs.go

Line 114 in 6b23c3f

i.h, err = rd.Next()

i.e., in the case of [ global, extended, reg "A" (size = n) ], I think only global will be read.

No, the stdlib tar.Reader handles the extra headers transparently

I think there's an exception for TypeXGlobalHeader (see link above)

Yeah, okay, after carefully re-reading the go package documentation and spelunking with a debugger, you're right.

The go stdlib package does not handle global headers, so we don't have to either. That'll make this simpler.

BradLugo · 2026-08-18T15:11:35Z

+		if err != nil {
+			return err
+		}


Should we fall back to spooling to disk instead?

I think any failure of the inspect routine means the spooling would fail, also.

I'm not so sure, e.g., I can see some weirdness happening where a server/proxy/WAF returns a non-200 when it receives a request with the Range header or something. In any case, after thinking about it for a bit, should we move the inspect after we check the cache anyway?

Reorganized it so the inspect and fetchFileForCache calls both happen only on the "cache miss" path.

This package does an io.ReaderAt over an HTTP resource. It includes a novel way to determine the end of a resource for partially-compliant servers. Using wholly-compliant servers is recommended. Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I08d6671217535ac897a2ed5c9cd20aa06a6a6964

Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I162632409abf01efcec4633713c9f0d06a6a6964

Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I8bd87cb66de93dad080c1d6d8aac34a96a6a6964

Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I6c2d6cfae9f6e742056c157e38b8b6156a6a6964

This handles the "easy" case of simply proxying reads for uncompressed tar archives to range requests. Future improvements would move the "spooling" out of this package and into the `fs.FS` implementation. Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I9d200dd841954054df0b187b9fd160a56a6a6964

hdonnay requested review from a team as code owners July 17, 2026 18:41

hdonnay requested a review from crozzy July 17, 2026 18:41

hdonnay force-pushed the hack/httpreader branch 6 times, most recently from 9c3ce05 to 719faf9 Compare July 22, 2026 16:41

hdonnay force-pushed the hack/httpreader branch 2 times, most recently from 7635a0a to 01c8352 Compare July 27, 2026 20:06

BradLugo self-requested a review July 27, 2026 20:12

BradLugo requested changes Jul 27, 2026

View reviewed changes

hdonnay force-pushed the hack/httpreader branch 2 times, most recently from 28024a7 to 29a4bdc Compare July 28, 2026 18:38

hdonnay requested a review from BradLugo July 28, 2026 18:44

hdonnay force-pushed the hack/httpreader branch from 29a4bdc to 8617e50 Compare July 29, 2026 16:23

hdonnay force-pushed the hack/httpreader branch 2 times, most recently from 06bd4c0 to 1196816 Compare August 13, 2026 20:21

BradLugo reviewed Aug 18, 2026

View reviewed changes

hdonnay force-pushed the hack/httpreader branch from 1196816 to ea62b17 Compare August 18, 2026 21:36

hdonnay requested a review from BradLugo August 18, 2026 21:36

hdonnay force-pushed the hack/httpreader branch 4 times, most recently from a28f949 to 2fdeec5 Compare August 20, 2026 17:35

hdonnay added 3 commits August 20, 2026 13:36

zreader: expose DetectCompression function

27da67b

Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I162632409abf01efcec4633713c9f0d06a6a6964

tarfs: add test for pax size

dead254

Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I8bd87cb66de93dad080c1d6d8aac34a96a6a6964

hdonnay added 2 commits August 20, 2026 13:36

tarfs: implement pax size extension

4c7ba57

Signed-off-by: Hank Donnay <hdonnay@redhat.com> Change-Id: I6c2d6cfae9f6e742056c157e38b8b6156a6a6964

hdonnay force-pushed the hack/httpreader branch from 2fdeec5 to ba37694 Compare August 20, 2026 18:36

Uh oh!

Conversation

hdonnay commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hdonnay commented Jul 17, 2026

Uh oh!

BradLugo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BradLugo Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BradLugo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

hdonnay commented Jul 17, 2026 •

edited

Loading

BradLugo Aug 18, 2026 •

edited

Loading