Merge pull request #231 from alan-turing-institute/222-request-max-retries

thobson88 · web-flow · commit 1edde143e924 · 2025-03-10T16:29:55.000Z
Docs: add hint for setting requestMaxRetries parameter
diff --git a/docs/http-server.md b/docs/http-server.md
@@ -35,7 +35,7 @@ Under the section headed `[http]`, add or edit the following configuration param
 
 !!! example "Example HTTP server configuration"
 
-    After completing the above steps, the `[http]` section of `trustchain_config.toml` should look similar the following example:
+    After completing the above steps, the `[http]` section of `trustchain_config.toml` should look similar to the following example:
     ```bash
     [http]
     root_event_time = 1697213008
diff --git a/docs/ion.md b/docs/ion.md
@@ -569,6 +569,24 @@ The final configuration step is to set the `bitcoinWalletOrImportString` paramet
 
     On Testnet, a key will be automatically generated when ION runs for the first time which can be used for the `bitcoinWalletOrImportString` parameter, so you don't need to do anything in this step.
 
+!!! tip "Tip: Set the `requestMaxRetries` configuration parameter"
+
+    This step is optional but is strongly recommended because it may significantly speed up the synchronisation process which takes place when ION runs for the first time.
+
+    When ION requests information from the local Bitcoin node it may have to retry several times before receiving a response. This is particularly common during its initial synchronisation, when many requests are made at high frequency.
+
+    After several failed requests ION will stop trying and the synchronisation process will restart, forfeiting the progress already made. By default this will happen after only three failed attempts, but this can be increased by setting the `requestMaxRetries` config parameter.
+
+    Run the following command to increase the maximum number of retry attempts:
+    === "Linux"
+        ```console
+        $ N=$(grep -n '\"port\"' $ION_BITCOIN_CONFIG_FILE_PATH | cut -d':' -f1); sed -i "$((N+1))"'i\'$'\n''  "requestMaxRetries": 6,'$'\n' $ION_BITCOIN_CONFIG_FILE_PATH
+        ```
+    === "macOS"
+        ```console
+        $ N=$(grep -n '\"port\"' $ION_BITCOIN_CONFIG_FILE_PATH | cut -d':' -f1); sed -i '' "$((N+1))"'i\'$'\n''  "requestMaxRetries": 6,'$'\n' $ION_BITCOIN_CONFIG_FILE_PATH
+        ```
+
 ### Build ION
 
 Change directory into the ION repository:
@@ -886,7 +904,7 @@ Then use this command to list the receiving addresses for this wallet (with thei
 $ bitcoin-cli -rpcwallet="sidetreeDefaultWallet" listreceivedbyaddress 1 true
 ```
 
-To fund your wallet, send Bitcoins to the **first** receive address in this list.
+To fund your wallet, send Bitcoins to the **first address** in this list.
 
 === "Mainnet"
 
diff --git a/docs/usage.md b/docs/usage.md
@@ -225,38 +225,148 @@ In fact, four private key were generated by the CLI when the DID was created. Al
 
     Instructions on how to fund your Bitcoin wallet are available [here](ion.md#funding-your-bitcoin-wallet).
 
+Currently the Trustchain CLI does not include a command for publishing DIDs. This will be added in a future version. In the meantime, DIDs can be published by running a script from the command line.
 
-When you are ready to publish your DID, execute the `publish.sh` script by running the following command:
+When you are ready to publish one or more DIDs, execute the `publish.sh` script by running the following command:
 ```console
 $ "$TRUSTCHAIN_REPO"/scripts/publish.sh
 ```
 
-This script will attempt to publish all of the DID operations that are found in the `$TRUSTCHAIN_DATA/operations/` directory.
+This script will attempt to publish all of the DID operations that are found in the `$TRUSTCHAIN_DATA/operations/` directory. The output from this command should include the following message, indicating it was successful:
+```
+* We are completely uploaded and fine
+```
 
-Support for publishing via the Trustchain CLI will be added in a future version.
+After the `publish.sh` script has run, there will be some delay before the newly-published DID can be resolved. This is due to i) the ION publication mechanism, which supports batching of DID operations to reduce transaction fees, and ii) the Bitcoin network processing time. For more details, see the information panels below.
 
-!!! tip "Tip: Batching DID operations"
+=== "Mainnet"
 
-    To save time and reduce transaction fees, multiple DID operations can be batched into a single Bitcoin transaction. In fact, ION supports batching of up to 10,000 operations per transaction.
+    ??? info "ION DID publication mechanism"
 
-    To perform batching, simply repeat the create operation as many times as you like before running the `publish.sh` script. Then run the script once to publish all operations in a single batch.
+        The `publish.sh` script takes all of the DID operations in `$TRUSTCHAIN_DATA/operations/` and dispatches them to the local ION node for publishing. They are then placed in the collection of `queued-operations` inside ION's Mongo database.
 
-    The only exception to this rule is that **the root DID must not be published in a batched transaction**. It must be the unique DID operation associated with the transaction in which it is published. The reason for this condition is to enable fast and efficient scanning of the Bitcoin blockchain to identify potential root DID operations.
+        To view the contents of this database, open the MongoDB shell with this command:
+        ```console
+        $ mongo --shell
+        ```
+        Then run the following MongoDB commands (omitting the `>` prompt character) to select the database:
+        ```console
+        > ion-mainnet-core
+        ```
+        and check how many queued DID operations exist:
+        ```console
+        > db["queued-operations"].count()
+        ```
+        The output from this command will usually be zero, indicating that there are no queued operations. Immediately after running the `publish.sh` script, the number of queued operations will increase to one (or more, if there were multiple files inside `$TRUSTCHAIN_DATA/operations/` when the script was executed).
+
+        Periodically, ION will check if there are any queued operations and, if any exist, it will batch them together and publish them in a single Bitcoin operation. The frequency with which this check is performed can be controlled by setting the `batchingIntervalInSeconds` parameter, found in the ION core config file. To view this file, run:
+        ```console
+        $ less $ION_CORE_CONFIG_FILE_PATH
+        ```
+
+        The default batching interval is 600 seconds. This can be reduced by changing the value of the `batchingIntervalInSeconds` parameter and restarting ION.
+
+    ??? info "Bitcoin network processing time"
+
+        When a new (or updated) DID document is published, it will take some time for the Bitcoin network to [process](https://bitcoin.org/en/how-it-works#processing) the relevant transaction so that it becomes visible to all other network participants.
+
+        Only after this processing has finished will it be possible to resolve the DID using the Trustchain CLI `resolve` command.
+
+        Typically, the processing time will be between 10 and 60 minutes, but it might be longer depending on factors such as the level of congestion on the Bitcoin network and the size of the fee inserted in the relevant transaction.
+
+    !!! tip "Tip: Identifying your DID transaction"
+
+        While you are waiting for your DID to be published, you can track its progress by observing the Bitcoin wallet address used to publish the DID operation. Run this command to list your Bitcoin addresses:
+        ```console
+        $ bitcoin-cli -rpcwallet="sidetreeDefaultWallet" listreceivedbyaddress 1 true
+        ```
+        Then copy the **first address** in the list and paste it into the search bar at [blockstream.info](https://blockstream.info/).
+
+        This search will return information about your Bitcoin address, including the number of confirmed transactions that have taken place and the unspent amount in the address (i.e. its current balance). Below the summary information will be a list showing every transaction associated with this address, *including any unconfirmed transactions*.
+
+        Assuming the ION publication mechanism has executed (see the panel above), the first transaction in the list will be the new one that was created by that process.
+
+        Click on the first transaction ID (this is a long string of hexadecimal characters that uniquely identifies the transactions). This takes you to a new page with details about that particular transaction.
+
+        Check the "status" of the transaction. If it is marked as "Unconfirmed", this indicates that it has not yet been processed by the Bitcoin network. In that case it will not yet be possible to resolve the new DID.
+
+        By refreshing this page, you can check its progress. When the transaction has been processed its status will change to "$n$ Confirmations", where $n$ is the number of Bitcoin blocks mined since the one containing this transaction.
+
+        As soon as the transaction has one or more confirmations, it should be possible to resolve the newly-published DID.
+
+        **If your are publishing a root DID**, make a note of the transaction ID so you can easily find it later. You should also make a note of the transactions's timestamp (i.e. the exact date & time that it was confirmed). The timestamp can be found on the same page as the transaction status on [blockstream.info](https://blockstream.info/).
+
+=== "Testnet"
+
+    ??? info "ION DID publication mechanism"
+
+        The `publish.sh` script takes all of the DID operations in `$TRUSTCHAIN_DATA/operations/` and dispatches them to the local ION node for publishing. They are then placed in the collection of `queued-operations` inside ION's Mongo database.
+
+        To view the contents of this database, open the MongoDB shell with this command:
+        ```console
+        $ mongo --shell
+        ```
+        Then run the following MongoDB commands (omitting the `>` prompt character) to select the database:
+        ```console
+        > ion-testnet-core
+        ```
+        and check how many queued DID operations exist:
+        ```console
+        > db["queued-operations"].count()
+        ```
+        The output from this command will usually be zero, indicating that there are no queued operations. Immediately after running the `publish.sh` script, the number of queued operations will increase to one (or more, if there were multiple files inside `$TRUSTCHAIN_DATA/operations/` when the script was executed).
+
+        Periodically, ION will check if there are any queued operations and, if any exist, it will batch them together and publish them in a single Bitcoin operation. The frequency with which this check is performed can be controlled by setting the `batchingIntervalInSeconds` parameter, found in the ION core config file. To view this file, run:
+        ```console
+        $ less $ION_CORE_CONFIG_FILE_PATH
+        ```
+
+        The default batching interval is 600 seconds. This can be reduced by changing the value of the `batchingIntervalInSeconds` parameter and restarting ION.
+
+    ??? info "Bitcoin network processing time"
+
+        When a new (or updated) DID document is published, it will take some time for the Bitcoin network to [process](https://bitcoin.org/en/how-it-works#processing) the relevant transaction so that it becomes visible to all other network participants.
+
+        Only after this processing has finished will it be possible to resolve the DID using the Trustchain CLI `resolve` command.
 
-After running the `publish.sh` script, wait for the transaction to be [processed](https://bitcoin.org/en/how-it-works#processing) by the Bitcoin network, then check that it was successfully published by attempting to resolve the DID (or DIDs, if more than one was published) [using the CLI](#did-resolution).
+        Typically, the processing time will be between 10 and 60 minutes, but it might be longer depending on factors such as the level of congestion on the Bitcoin network and the size of the fee inserted in the relevant transaction.
+
+    !!! tip "Tip: Identifying your DID transaction"
+
+        While you are waiting for your DID to be published, you can track its progress by observing the Bitcoin wallet address used to publish the DID operation. Run this command to list your Bitcoin addresses:
+        ```console
+        $ bitcoin-cli -rpcwallet="sidetreeDefaultWallet" listreceivedbyaddress 1 true
+        ```
+        Then copy the **first address** in the list and paste it into the search bar at [blockstream.info](https://blockstream.info/testnet/).
+
+        This search will return information about your Bitcoin address, including the number of confirmed transactions that have taken place and the unspent amount in the address (i.e. its current balance). Below the summary information will be a list showing every transaction associated with this address, *including any unconfirmed transactions*.
+
+        Assuming the ION publication mechanism has executed (see the panel above), the first transaction in the list will be the new one that was created by that process.
+
+        Click on the first transaction ID (this is a long string of hexadecimal characters that uniquely identifies the transactions). This takes you to a new page with details about that particular transaction.
+
+        Check the "status" of the transaction. If it is marked as "Unconfirmed", this indicates that it has not yet been processed by the Bitcoin network. In that case it will not yet be possible to resolve the new DID.
+
+        By refreshing this page, you can check its progress. When the transaction has been processed its status will change to "$n$ Confirmations", where $n$ is the number of Bitcoin blocks mined since the one containing this transaction.
+
+        As soon as the transaction has one or more confirmations, it should be possible to resolve the newly-published DID.
+
+        **If your are publishing a root DID**, make a note of the transaction ID so you can easily find it later. You should also make a note of the transactions's timestamp (i.e. the exact date & time that it was confirmed). The timestamp can be found on the same page as the transaction status on [blockstream.info](https://blockstream.info/testnet/).
+
+After running the `publish.sh` script, wait for the transaction to be processed, then check that it was successfully published by attempting to resolve the DID (or DIDs, if more than one was published) [using the CLI](#did-resolution).
 
 Once you have confirmed that the DID(s) can be resolved, you can clean up the `.trustchain/operations/` folder by running this command to move all operations files to the `sent/` subdirectory:
 ```console
 $ mv $TRUSTCHAIN_DATA/operations/*.json* $TRUSTCHAIN_DATA/operations/sent/./
 ```
 
-!!! info "Network processing time"
+!!! tip "Tip: Batching DID operations"
 
-    When a new (or updated) DID document is published, it will take some time for the Bitcoin network to process the relevant transaction so that it becomes visible to all other network participants.
+    To save time and reduce transaction fees, multiple DID operations can be batched into a single Bitcoin transaction. In fact, ION supports batching of up to 10,000 operations per transaction.
 
-    Only after this processing has finished will it be possible to resolve the DID using the Trustchain CLI `resolve` command.
+    To perform batching, simply repeat the create operation as many times as you like before running the `publish.sh` script. Then run the script once to publish all operations in a single batch.
 
-    Typically, the processing time will be between 10 and 60 minutes, but it might be longer depending on factors such as the level of congestion on the Bitcoin network and the size of the fee inserted in the relevant transaction.
+    The only exception to this rule is that **the root DID must not be published in a batched transaction**. It must be the unique DID operation associated with the transaction in which it is published. The reason for this condition is to enable fast and efficient scanning of the Bitcoin blockchain to identify potential root DID operations.
 
 ## Downstream DID Issuance
 
diff --git a/scripts/publish.sh b/scripts/publish.sh
@@ -9,7 +9,7 @@
 # Returns:
 #   OK / Bad Request
 
-for file in $(find "$HOME/.trustchain/operations" -name "*.json" -depth 1); do
+for file in $(find "$HOME/.trustchain/operations" -maxdepth 1 -name "*.json"); do
     echo $file
     curl --tr-encoding -X POST -v -# -o "$file.out.json" -T $file -H "Content-Type: application/json; charset=utf-8" http://localhost:3000/operations
 done
