FRC: Retrieval Checking Requirements

[bajtos]

When we set out to build Spark, a protocol for testing whether payload of Filecoin deals can be retrieved back, we designed it based on how Boost worked at that time (mid-2023). Soon after FIL+ allocator compliance started to use Spark retrieval success score (Spark RSR) in mid-2024, we learned that Venus Droplet, an alternative miner software, is implemented slightly differently and requires tweaks to support Spark. Things evolved quite a bit since then. We need to overhaul most of the Spark protocol to support Direct Data Onboarding deals. We will need all miner software projects (Boost, Curio, Venus) to accommodate the new requirements imposed by the upcoming Spark v2 release.

This FRC has the following goals:

1. Document the retrieval process based on IPFS/IPLD.
2. Specify what Spark needs from miner software.
3. Collaborate with the community to tweak the requirements to work well for all parties involved.
4. Let this spec and the building blocks like IPNI Reverse Index empower other builders to design & implement their own retrieval-checking networks as alternatives to Spark.

Discussion

#1086

Progress

• Simple Summary
• Abstract
• Change Motivation
• Specification
• Design Rationale
• Backwards Compatibility
• Test Cases
• Security Considerations
• Incentive Considerations
• Product Considerations
• Implementation
• TODO

[bajtos]

Tagging @steven004 @LexLuthr @magik6k @masih @willscott @juliangruber @patrickwoodhead for visibility.

[bajtos]

@steven004 Does Venus (or Venus Droplet?) support payload retrieval over IPFS Trustless HTTP Gateway protocol?

Does this require any manual configuration or does it work out of the box?

If manual configuration is required, could you please point me to the relevant documentation?

[bajtos]

I tried to find the answer in Droplet's code, it seems it advertises Graphsync retrievals only.

https://github.com/ipfs-force-community/droplet/blob/f4bffcde1b689119688fa1e64c19387ea0f51d73/indexprovider/index_provider.go#L418-L430

[bajtos]

Droplet docs on configuring HTTP retrievals:

• https://github.com/ipfs-force-community/droplet/blob/master/docs/zh/模拟官方机器人HTTP方式检索.md#开启-http-检索

• Translation to EN: https://github-com.translate.goog/ipfs-force-community/droplet/blob/master/docs/zh/模拟官方机器人HTTP方式检索.md?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp#开启-http-检索

[LexLuthr]

This requirement cannot be fulfilled in Curio. We no longer have a concept of minerID <> Unique peerID binding. IPNI must be extended to support other keys types like worker key to sign ads.

[bajtos]

I am aware of that; see the note in the text below this paragraphs.

> [!NOTE]
> This is open to extensions in the future, we can support more than one form of linking
> index-provides to filecoin-miners. See e.g. [ipni/spec#33](https://github.com/ipni/specs/issues/33).

From my point of view, I prefer not to block progress on this FRC until the Curio team figures out how to extend IPNI to support other key types. Instead, I'd like this FRC to document the solution that works with Boost & Venus now and then enhance it with the new mechanism Curio needs once that new solution is agreed on.

[lanzafame]

I am probably mistaken here but Droplet (Venus' Boost) supports multiple minerIDs being associated with a single PeerID (see docs), does that mean if I am using Droplet, I need to limit myself to a 1:1 relationship to meet this requirement?

[bajtos]

Great call, @lanzafame! I am still learning more about how Venus Droplet work and what features they offer.

Based on the docs you linked to, I believe you can have multiple minerIDs associated with a single Droplet PeerID and still meet this requirement.

In Spark, we need the PeerID returned by Filecoin.StateMinerInfo to match the PeerID used in IPNI advertisements. Spark does not check whether that PeerID is unique or shared by multiple miners.