merge (#5)

* refactor: downloader

* test: add test for concurrent progress reporting

* chore: cleanup

* refactor: rename some things for clarity

* feat: limit concurrent dials per download

* chore: cleanup

* refactor: respect progress sender IDs to allow reusing the senders in subsequent operations

* chore: fmt & clippy

* cleanup: shared download progress

* feat: handle tags in the downloader

* fix: visibility

* fixup

* fixup

* chore: clippy

* fixup

* refactor: make export a seperate operation from download

* fixup after merge

* refactor: remove id mapping

* refactor: move BlobId up to the progress events

* fix tests

* cleanup

* fix: invariants

* fixes and cleanups

* fix: nodes_have should only add to queued downloads

* chore: fmt

* chore: clippy!

* fix: do not queue downloads with no providers

* feat: set more than one node on download requests (n0-computer#2128)

## Description

Based on n0-computer#2085 

The downloader already has the feature to try multiple nodes for a
single hash or hashseq. With n0-computer#2085 we expose the downloader over the RPC
protocol, by adding a `DownloadMode { Queued, Direct }` enum to the
`BlobDownloadRequest`.
This PR modifies the `BlobDownloadRequest` to include a list of provider
nodes for a hash instead of just a single node. For queued requests that
go to the downloader, the nodes are just passed to the downloader. For
direct requests, the nodes are tried in-order, until one succeeds.

## Notes & open questions

<!-- Any notes, remarks or open questions you have to make about the PR.
-->

## Change checklist

- [ ] Self-review.
- [ ] Documentation updates if relevant.
- [ ] Tests if relevant.

* fix: address review by divma

* fix: remove now obsolete into_send method from ProgressSender

* chore: remove unused code

* fix: use Display for DownloadKind

* fix: first round of changes after review

* docs: downloader next_step

* more review

* more review

* fix: only remove hashes from provider map if they are really not needed

* refactor: move TagSet into util

* refactor: store intents in request info

* cleanup

* refactor: no allocation in next_step and simpler code

* fix: test after merge

* refactor: better method

* fix(iroh-bytes): do not log redundant file delete error (n0-computer#2199)

## Description

fix(iroh-bytes): do not log when a file can not be deleted...

because it is not there. this makes the delete op idempotent, and also
helps in case the file was not there in the first place.

Fixes n0-computer#2142

## Notes & open questions

<!-- Any notes, remarks or open questions you have to make about the PR.
-->

## Change checklist

- [ ] Self-review.
- [ ] Documentation updates if relevant.
- [ ] Tests if relevant.

* fix(iroh-dns-server): fix bug in pkarr name parsing (n0-computer#2200)

## Description

The order of labels when parsing a DNS query as pkarr name was reversed.
This means all queries for second-level subdomains under a pkarr pubkey
failed.

* First commit is the actual fix. 
* Second commit adds a test for various record name and type variations
* Third and forth commit improve the debug and info logging

## Notes & open questions

Was discovered by @rklaehn in n0-computer#2188 and fixes the wrong behavior
observed there.

## Change checklist

- [x] Self-review.
- [x] Documentation updates if relevant.
- [x] Tests if relevant.

---------

Co-authored-by: Franz Heinzmann (Frando) <[email protected]>
Co-authored-by: Rüdiger Klaehn <[email protected]>

Author: 3 people

Cargo.lock

@@ -7,7 +7,7 @@ use bao_tree::blake3;
 use postcard::experimental::max_size::MaxSize;
 use serde::{de, Deserialize, Deserializer, Serialize, Serializer};
 
-use crate::base32::{parse_array_hex_or_base32, HexOrBase32ParseError};
+use crate::base32::{self, parse_array_hex_or_base32, HexOrBase32ParseError};
 
 /// Hash type used throughout.
 #[derive(PartialEq, Eq, Copy, Clone, Hash)]
@@ -54,6 +54,12 @@ impl Hash {
     pub fn to_hex(&self) -> String {
         self.0.to_hex().to_string()
     }
+
+    /// Convert to a base32 string limited to the first 10 bytes for a friendly string
+    /// representation of the hash.
+    pub fn fmt_short(&self) -> String {
+        base32::fmt_short(self.as_bytes())
+    }
 }
 
 impl AsRef<[u8]> for Hash {
@@ -173,7 +179,18 @@ impl MaxSize for Hash {
 
 /// A format identifier
 #[derive(
-    Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize, Default, Debug, MaxSize,
+    Clone,
+    Copy,
+    PartialEq,
+    Eq,
+    PartialOrd,
+    Ord,
+    Serialize,
+    Deserialize,
+    Default,
+    Debug,
+    MaxSize,
+    Hash,
 )]
 pub enum BlobFormat {
     /// Raw blob
@@ -205,7 +222,7 @@ impl BlobFormat {
 }
 
 /// A hash and format pair
-#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, MaxSize)]
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, MaxSize, Hash)]
 pub struct HashAndFormat {
     /// The hash
     pub hash: Hash,
@@ -289,6 +306,11 @@ mod redb_support {
 }
 
 impl HashAndFormat {
+    /// Create a new hash and format pair.
+    pub fn new(hash: Hash, format: BlobFormat) -> Self {
+        Self { hash, format }
+    }
+
     /// Create a new hash and format pair, using the default (raw) format.
     pub fn raw(hash: Hash) -> Self {
         Self {

iroh-base/src/hash.rs

@@ -25,6 +25,7 @@ flume = "0.11"
 futures = "0.3.25"
 futures-buffered = "0.2.4"
 genawaiter = { version = "0.99.1", features = ["futures03"] }
+hashlink = { version = "0.9.0", optional = true }
 hex = "0.4.3"
 iroh-base = { version = "0.14.0", features = ["redb"], path = "../iroh-base" }
 iroh-io = { version = "0.6.0", features = ["stats"] }
@@ -51,6 +52,7 @@ tracing-futures = "0.2.5"
 
 [dev-dependencies]
 http-body = "0.4.5"
+iroh-bytes = { path = ".", features = ["downloader"] }
 iroh-test = { path = "../iroh-test" }
 proptest = "1.0.0"
 serde_json = "1.0.107"
@@ -63,8 +65,8 @@ tempfile = "3.10.0"
 
 [features]
 default = ["fs-store"]
+downloader = ["iroh-net", "parking_lot", "tokio-util/time", "hashlink"]
 fs-store = ["reflink-copy", "redb", "redb_v1", "tempfile"]
-downloader = ["iroh-net", "parking_lot", "tokio-util/time"]
 metrics = ["iroh-metrics"]
 
 [[example]]