Minimum image replication count

[phillebaba]

Describe the problem to be solved

Currently Spegel will only replicate an image to a node when it is required by a Pod. This works fine as Spegel is a best effort cache and the image can be fetched from the original registry if no longer available. It is a bit more challenging in air-gapped or edge environments where it may be not possible or slow. In these situations making sure that an image is always replicated a certain amount of times would reduce the risk of an image being stored on a single node that is then removed from the cluster.

Proposed solution to the problem

In a perfect world we would allow the user to specify which images should be replicated and how many times they should be replicated. I have not really thought about all of the technical details and there is a large risk that we will re implement parts of IPFS by mistake.

The nodes are already communicating with each other so creating consensus on image replication should not be impossible. Depending on how image pulling is implemented however it may require more write privileges than we would like.

[bittrance]

How would this work, though? Is there an easy way for instances to reach consensus on who should replicate a newly discovered layer?

Alternatively, the spegel instance that receives the request from the client could look up each layer requested to see that it has enough advertised peers. If not, it picks one peer at random and uses a spegel-specific API to advise it to pull the layer. The advisee would presumably look up the layer in the DHT (rather than trusting the advising peer) and pull from an advertising peer. The advisee would then in turn perform a lookup and if there are too few advertised peers, it would in turn advise another random peer, and so on.

By advising one peer at a time, we won't magnify pull bursts as badly. Of course, this results in more DHT lookups which may turn out to be a worse problem. If so, it may be better for the adviser to advise sufficient instances at once.

Advising this way would be best effort, but assuming the layer is popular, we will eventually reach the desired replication count.

In theory, such an advise API could also be used to implement more opinionated replication strategies, such as advising instances residing in edge locations to perform proactive pulls.