SourceDateEpoch= is very invasive

[septatrix]

We are currently working on getting some of our projects reproducible, one of which uses mkosi. While the source date epoch settings seems to work (at an initial glance) it is a quite rough/overkill solution. We have a few places in our software where file dates are checked which we can do without for now and should ideally be replaced by some better mechanism but nevertheless setting the mtime of every file is still quite drastic.

Ideally, only the mtime of files generated by mkosi should be changed (e.g. sd-firstboot, sd-tmpfiles, Ssh= config file etc.), and even that should be configurable. This way if someone already has a reproducible build process they can utilize that in their build script and not have it ruined by mkosi.

I created this as a discussion as I think this is not something directly actionable and a good design should be though out first. Should this be opt-in or opt-out, should it be configurable for everything (all files), only files generated by mkosi internally (sd-firstboot etc.), separate for each mkosi.* script and so on...?

[DaanDeMeyer]

This would require us to diff the image for each command to figure out the added files, seems like a potentially expensive operation. I really think there's no way to do this but to apply it to everything.

[septatrix]

I was thinking of running all tools in an overlay. This way all generated and modified files should land in upperdir, which we could iterate over, set their mtime and then merge them into the lowerdir. Sadly this does not seem to be supported by bubblewrap yet but there is at least a PR containers/bubblewrap#547 (or it might be possible to reuse the logic from Overlay=).

[DaanDeMeyer]

This still seems rather excessive to me.

cc @malt3 @katexochen you both might be interested in this as well

[malt3]

I wonder if clamping the mtime to SOURCE_DATE_EPOCH would be enough for you. By this I mean: leave any mtimes older than SOURCE_DATE_EPOCH untouched and set any mtimes that are newer than SOURCE_DATE_EPOCH to SOURCE_DATE_EPOCH.
This is feasible to implement without too much work. wdyt?

[septatrix]

I wonder if clamping the mtime to SOURCE_DATE_EPOCH would be enough for you.

Sound like a great idea. A quick search to see if there is someone else out there doing it this way even reveals that we would be in good company by adopting this approach: Fedora has accepted a change proposal to do exactly this for their RPM packages in version 38. There do not seem to be any bugs linked towards that proposal so it seems to work ;)

[septatrix]

As we have now a good idea for an alternative approach (clamp mtime of newer files only) which can be acted upon I created a new issue from this discussion: #2635