Testing Avalon Proxy model with Besu 404 Client Error: Not Found for url: http://rpc.node1.avalon.local:8555

[justalittlenoob]

I am trying to test proxy model with besu according to this doc. There is an error when i try to run ./eth_generic_client.py -b ethereum --workload_id "echo-result" -o --in_data "Hello" in avalon-shell container. Any advice would be appreciated.

Here are some information may be helpful.

vi /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
http_proxy=http://abc-proxy.abc.com:914
https_proxy=http://abc-proxy.abc.com:914
ftp_proxy=http://abc-proxy.abc.com:914
no_proxy=10.96.0.0/12,10.67.117.0/24,10.244.0.0/16,127.0.0.1,localhost,node1.avalon.local,rpc.node1.avalon.local,node2.avalon.local, rpc.node2.avalon.local

docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d3556bde412c avalon-blockchain-connector-eth-dev "bash -c '\n avalon_…" 23 minutes ago Up 23 minutes avalon-blockchain-connector-eth

e4c0dfc4387c avalon-shell-dev "bash -c 'tail -f /d…" 24 minutes ago Up 24 minutes avalon-shell

cc6238a6b6e2 avalon-enclave-manager-dev "bash -c '\n enclave…" 24 minutes ago Up 24 minutes 5555/tcp avalon-enclave-manager

f406125301fe avalon-listener-dev "bash -c '\n avalon_…" 24 minutes ago Up 24 minutes 1947/tcp, 5555/tcp avalon-listener

893ed7ebb9b0 avalon-lmdb-dev "bash -c '\n kv_stor…" 24 minutes ago Up 24 minutes 9090/tcp avalon-lmdb

eb8890062d4d truffle-envt:latest "docker-entrypoint.s…" 59 minutes ago Exited (0) 56 minutes ago truffle-envt-besu-setup

628cfd271c56 pegasyseng/ethsigner:0.4.0 "/opt/ethsigner/bin/…" 59 minutes ago Up 59 minutes 8545/tcp, 0.0.0.0:23001->8555/tcp besu_rpc.node2.avalon.local_1

c765e9e6c2c2 hyperledger/besu:1.4.3 "besu --data-path=/v…" 59 minutes ago Up 59 minutes 8547/tcp, 30303/tcp, 0.0.0.0:23011->8545/tcp, 0.0.0.0:23002->8546/tcp besu_node2.avalon.local_1

9681e68d4374 pegasyseng/ethsigner:0.4.0 "/opt/ethsigner/bin/…" 59 minutes ago Up 59 minutes 8545/tcp, 0.0.0.0:22001->8555/tcp besu_rpc.node1.avalon.local_1

1df85be814df hyperledger/besu:1.4.3 "besu --data-path=/v…" 59 minutes ago Up 59 minutes 8547/tcp, 30303/tcp, 0.0.0.0:22011->8545/tcp, 0.0.0.0:22002->8546/tcp besu_node1.avalon.local_1

/project/avalon/examples/apps/generic_client# ./eth_generic_client.py -b ethereum --workload_id "echo-result" -o --in_data "Hello"
[18:58:34 INFO main] ******* Hyperledger Avalon Generic client *******
[18:58:34 INFO solcx] Using solc version v0.5.15
[18:58:37 WARNING STDERR] Traceback (most recent call last):
[18:58:37 WARNING STDERR] File "./eth_generic_client.py", line 577, in
[18:58:37 WARNING STDERR] Main()
[18:58:37 WARNING STDERR] File "./eth_generic_client.py", line 477, in Main
[18:58:37 WARNING STDERR] worker_id, config)
[18:58:37 WARNING STDERR] File "./eth_generic_client.py", line 337, in _get_first_active_worker
[18:58:37 WARNING STDERR] jrpc_req_id
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/avalon_sdk/connector/blockchains/ethereum/ethereum_worker_registry.py", line 101, in worker_lookup
[18:58:37 WARNING STDERR] worker_type.value, org_id, application_id).call()
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/contract.py", line 959, in call
[18:58:37 WARNING STDERR] **self.kwargs
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/contract.py", line 1493, in call_contract_function
[18:58:37 WARNING STDERR] return_data = web3.eth.call(call_transaction, block_identifier=block_id)
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/eth_utils/functional.py", line 45, in inner
[18:58:37 WARNING STDERR] return callback(fn(*args, **kwargs))
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/eth.py", line 430, in call
[18:58:37 WARNING STDERR] [transaction, block_identifier],
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/manager.py", line 149, in request_blocking
[18:58:37 WARNING STDERR] response = self._make_request(method, params)
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/manager.py", line 128, in _make_request
[18:58:37 WARNING STDERR] return request_func(method, params)
[18:58:37 WARNING STDERR] File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 75, in apply_formatters
[18:58:37 WARNING STDERR] response = make_request(method, params)
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/middleware/gas_price_strategy.py", line 34, in middleware
[18:58:37 WARNING STDERR] return make_request(method, params)
[18:58:37 WARNING STDERR] File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 73, in apply_formatters
[18:58:37 WARNING STDERR] response = make_request(method, formatted_params)
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/middleware/attrdict.py", line 33, in middleware
[18:58:37 WARNING STDERR] response = make_request(method, params)
[18:58:37 WARNING STDERR] File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 73, in apply_formatters
[18:58:37 WARNING STDERR] response = make_request(method, formatted_params)
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/middleware/normalize_errors.py", line 25, in middleware
[18:58:37 WARNING STDERR] result = make_request(method, params)
[18:58:37 WARNING STDERR] File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 73, in apply_formatters
[18:58:37 WARNING STDERR] response = make_request(method, formatted_params)
[18:58:37 WARNING STDERR] File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 73, in apply_formatters
[18:58:37 WARNING STDERR] response = make_request(method, formatted_params)
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/middleware/exception_retry_request.py", line 105, in middleware
[18:58:37 WARNING STDERR] return make_request(method, params)
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/providers/rpc.py", line 83, in make_request
[18:58:37 WARNING STDERR] **self.get_request_kwargs()
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/web3/_utils/request.py", line 35, in make_post_request
[18:58:37 WARNING STDERR] response.raise_for_status()
[18:58:37 WARNING STDERR] File "/usr/local/lib/python3.6/dist-packages/requests/models.py", line 940, in raise_for_status
[18:58:37 WARNING STDERR] raise HTTPError(http_error_msg, response=self)
[18:58:37 WARNING STDERR] requests.exceptions
[18:58:37 WARNING STDERR] .
[18:58:37 WARNING STDERR] HTTPError
[18:58:37 WARNING STDERR] :
[18:58:37 WARNING STDERR] 404 Client Error: Not Found for url: http://rpc.node1.avalon.local:8555/

[rranjan3]

If you are behind a proxy, you might need to set the proxy inside docker containers as well.
Try echo $no_proxy from within shell container. If that does not work, you need to ensure the no_proxy is set within your container environment.

One way to do this would be to update ~/.docker/config.json. Refer - https://docs.docker.com/network/proxy/#configure-the-docker-client

[justalittlenoob]

@rranjan3
Thank you very much, your advice indeed help. After adding no_proxy in the shell container, it works.

Besides, i have the following confuse.

1. Is this(https://github.com/hyperledger/avalon/blob/master/docs/TestingBesuProxyModel.rst) running in HW mode? If not, how to change it to HW mode?
2. How to know whether avalon runs in HW mode or SIM mode?

[danintel]

The instructions for running Besu are the same for HW mode or SIM mode. The instructions for installing Avalon in HW mode or SIM mode are 1. Install prerequisites (including Intel SGX) https://github.com/hyperledger/avalon/blob/master/PREREQUISITES.md 2. Install Avalon (including setting SGX_MODE=HW ) https://github.com/hyperledger/avalon/blob/master/BUILD.md You know you are in Intel SGX Hardware Mode (SGX_MODE=HW) when you see messages similar to this in the Avalon Enclave Manager output at startup: INFO avalon_enclave_manager.ias_client] IAS settings: INFO avalon_enclave_manager.ias_client] SPID: BA5FDA588BE8A63B230955D38FA6B675 INFO avalon_enclave_manager.ias_client] IAS ApiKey: 8730f36c63ad49528dd3e7cf318997ed Dan — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#541 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH26UROYLCSAVZEYJCA7Q2LRYBOSXANCNFSM4OAQD76Q>.

[rranjan3]

Adding to what @danintel suggested.
Once you are done with the SGX specific setup and you want to run the enclave-manager in SGX HW mode with dockers,
For Step 7 in the docs, you could use the below command -

docker-compose -f docker-compose.yaml -f docker-compose-sgx.yaml -f docker-compose-eth-besu.yaml up -d --build

[justalittlenoob]

@danintel Hi Dan, your advice taught me a lot. Thank you very much.
@rranjan3 I changed my steps according to you advice and a new problem(no workers found) appeared in HW mode. Do you have any suggestions?

sgx sdk/psw: https://github.com/intel/linux-sgx master branch(proxy type = manual and aesm proxy = http://abc-proxy.abc.com:914 are added to /etc/aesmd.conf )
sgx driver: https://github.com/intel/linux-sgx-driver master branch
full log: eth_hw_log_1.txt

docker-compose -f docker-compose.yaml -f docker-compose-sgx.yaml -f docker-compose-eth-besu.yaml up --build
avalon-blockchain-connector-eth | 2020-06-24 17:54:01,750 - INFO - Blockchain to be used as proxy : ethereum
avalon-blockchain-connector-eth | 2020-06-24 17:54:01,751 - INFO - About to start Ethereum connector service
avalon-blockchain-connector-eth | 2020-06-24 17:54:01,754 - INFO - Using solc version v0.5.15
avalon-blockchain-connector-eth | 2020-06-24 17:54:01,824 - INFO - Using solc version v0.5.15
avalon-blockchain-connector-eth | 2020-06-24 17:54:01,924 - INFO - Using solc version v0.5.15
avalon-blockchain-connector-eth | 2020-06-24 17:54:01,966 - INFO - Ethereum Connector service started
avalon-blockchain-connector-eth | 2020-06-24 17:54:01,990 - INFO - Worker lookup response from blockchain: {
avalon-blockchain-connector-eth | "result": {
avalon-blockchain-connector-eth | "totalCount": 0,
avalon-blockchain-connector-eth | "lookupTag": 0,
avalon-blockchain-connector-eth | "ids": []
avalon-blockchain-connector-eth | }
avalon-blockchain-connector-eth | }
avalon-blockchain-connector-eth |
avalon-blockchain-connector-eth | 2020-06-24 17:54:01,990 - ERROR - No workers found in ethereum blockchain
...
avalon-blockchain-connector-eth | 2020-06-24 17:54:01,997 - INFO -
avalon-blockchain-connector-eth | Worker lookup response from kv storage : {
avalon-blockchain-connector-eth | "result": {
avalon-blockchain-connector-eth | "totalCount": 0,
avalon-blockchain-connector-eth | "lookupTag": "",
avalon-blockchain-connector-eth | "ids": []
avalon-blockchain-connector-eth | },
avalon-blockchain-connector-eth | "id": 73207,
avalon-blockchain-connector-eth | "jsonrpc": "2.0"
avalon-blockchain-connector-eth | }
avalon-blockchain-connector-eth |
avalon-blockchain-connector-eth | 2020-06-24 17:54:01,997 - ERROR - No workers found in kv storage
avalon-blockchain-connector-eth | 2020-06-24 17:54:02,013 - INFO - Started listener for events from blockchain
avalon-blockchain-connector-eth | 2020-06-24 17:54:02,016 - INFO - Started handler to handle events
...
avalon-enclave-manager | [17:54:03 WARNING avalon_enclave_manager.base_enclave_info] failure GROUP_OUT_OF_DATE (update your BIOS/microcode!!!) keep going
...
avalon-enclave-manager | [17:54:03 INFO avalon_enclave_manager.work_order_kv_delegate] No workorder entries found in wo-processing table, skipping Cleanup
avalon-enclave-manager | [17:54:03 INFO avalon_enclave_manager.singleton.singleton_enclave_manager] --------------- Boot time flow Complete ----------------
avalon-enclave-manager | [17:54:03 INFO avalon_enclave_manager.singleton.singleton_enclave_manager] ZMQ Port hosted by Enclave
avalon-enclave-manager | [17:54:03 INFO avalon_enclave_manager.singleton.singleton_enclave_manager] Enclave Manager waiting for next request

./eth_generic_client.py -b ethereum --workload_id "echo-result" -o --in_data "Hello"
[17:58:12 INFO main] ******* Hyperledger Avalon Generic client *******
[17:58:12 INFO solcx] Using solc version v0.5.15
[17:58:12 INFO main]
Worker lookup response: {
"result": {
"totalCount": 0,
"lookupTag": 0,
"ids": []
}
}

[17:58:12 ERROR main] No workers found
[17:58:12 ERROR main] Failed to lookup worker
[17:58:12 ERROR main] Cannot proceed without a valid worker

[rranjan3]

@justalittlenoob It must be a race condition we need to fix. The HW mode would take more time for bootupby when the blockchain connector would have started. This would lead to a situation where the connector does not find a worker in the KV storage and hence nothing gets synced up in the proxy blockchain.

Try the command below with a sleep induced (for now) in the blockchain connector container at -
https://github.com/hyperledger/avalon/blob/master/docker-compose-eth-besu.yaml#L34

      bash -c "
        sleep 30
        avalon_ethereum_connector -u http://avalon-listener:1947
        tail -f /dev/null
        "

If you have the images already built, you could ignore the trialing --build.

[justalittlenoob]

@rranjan3
Thank you very much for your patience. The walk around method make the ERROR - No workers found in kv storage gone. But the No workers found problem in avalon-shell still exist.

After running docker-compose -f docker-compose.yaml -f docker-compose-sgx.yaml -f docker-compose-eth-besu.yaml up -d --build, there will be the following warnings and errors.(full log: eth_hw_log_2.txt)

avalon-enclave-manager | [14:25:14 WARNING STDERR] /usr/lib/python3/dist-packages/urllib3/connectionpool.py:860: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
avalon-enclave-manager | [14:25:14 WARNING STDERR] InsecureRequestWarning)
avalon-enclave-manager | [14:25:16 WARNING avalon_enclave_manager.base_enclave_info] failure GROUP_OUT_OF_DATE (update your BIOS/microcode!!!) keep going

avalon-enclave-manager | [14:25:16 INFO avalon_enclave_manager.work_order_kv_delegate] No workorder entries found in wo-processing table, skipping Cleanup
avalon-blockchain-connector-eth | 2020-06-29 14:26:14,061 - ERROR - No workers found in ethereum blockchain

root@bb185bc97ec3:/project/avalon/examples/apps/generic_client# ./eth_generic_client.py -b ethereum --workload_id "echo-result" -o --in_data "Hello"

[15:54:52 INFO main] ******* Hyperledger Avalon Generic client *******
[15:54:52 INFO solcx] Using solc version v0.5.15
[15:54:52 INFO main]
Worker lookup response: {
"result": {
"totalCount": 0,
"lookupTag": 0,
"ids": []
}
}
[15:54:52 ERROR main] No workers found
[15:54:52 ERROR main] Failed to lookup worker
[15:54:52 ERROR main] Cannot proceed without a valid worker