Test the network - crash the ordering nodes

[koakh]

Hello Horea

first thanks for your awesome repo and videos, and other cool stuff posted, I learn a lot with it

I have a problem in my hlf raft network, and try with your repo and seems the same problem occurs,
I don't know if the problem is because of my limited knowledge about raft, it seems that we need first ordeder, and this don't make sense to me

let me explain the problem

if I stop any ordeder like orderer2, orderer3, orderer4 and orderer5, even two at the same time,
everything works as expected, same on my production network,
but when I stop first orderer network stops, and never responds, until I start orderer again and wait

Horea repo: node server logs

stop ordeder

$ docker container stop orderer.example.com

invoke transaction

::1 - - [25/Oct/2020:21:49:56 +0000] "OPTIONS /createCar HTTP/1.1" 204 0 "http://localhost:4200/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"
{ make: 'Bmw', model: '5', color: 'CAR12', owner: 'Pini' }
starting to queryAllCars
Wallet path: /media/mario/storage/Documents/Development/HyperLedger/ibmRaftFabricSample-HoreaPorutiu/web-app/server/wallet
Transaction has been evaluated, result is: [{"Key":"CAR0","Record":{"color":"blue","make":"Toyota","model":"Prius","owner":"Tomoko","docType":"car"}},{"Key":"CAR1","Record":{"color":"red","make":"Ford","model":"Mustang","owner":"Brad","docType":"car"}},{"Key":"CAR10","Record":{"color":"CAR10","docType":"car","make":"WV","model":"Golf","owner":"Koakh"}},{"Key":"CAR11","Record":{"color":"CAR11","docType":"car","make":"Audi","model":"A3","owner":"Pelo"}},{"Key":"CAR12","Record":{"color":"CAR11","docType":"car","make":"Audi","model":"A4","owner":"Jomu"}},{"Key":"CAR2","Record":{"color":"green","make":"Hyundai","model":"Tucson","owner":"Jin Soo","docType":"car"}},{"Key":"CAR3","Record":{"color":"yellow","make":"Volkswagen","model":"Passat","owner":"Max","docType":"car"}},{"Key":"CAR4","Record":{"color":"black","make":"Tesla","model":"S","owner":"Adriana","docType":"car"}},{"Key":"CAR5","Record":{"color":"purple","make":"Peugeot","model":"205","owner":"Michel","docType":"car"}},{"Key":"CAR6","Record":{"color":"white","make":"Chery","model":"S22L","owner":"Aarav","docType":"car"}},{"Key":"CAR7","Record":{"color":"violet","make":"Fiat","model":"Punto","owner":"Pari","docType":"car"}},{"Key":"CAR8","Record":{"color":"indigo","make":"Tata","model":"Nano","owner":"Valeria","docType":"car"}},{"Key":"CAR9","Record":{"color":"brown","make":"Holden","model":"Barina","owner":"Shotaro","docType":"car"}}]
<Buffer 5b 7b 22 4b 65 79 22 3a 22 43 41 52 30 22 2c 22 52 65 63 6f 72 64 22 3a 7b 22 63 6f 6c 6f 72 22 3a 22 62 6c 75 65 22 2c 22 6d 61 6b 65 22 3a 22 54 6f ... >
Wallet path: /media/mario/storage/Documents/Development/HyperLedger/ibmRaftFabricSample-HoreaPorutiu/web-app/server/wallet
we here in createCar
2020-10-25T21:49:59.228Z - error: [Remote.js]: Error: Failed to connect before the deadline URL:grpcs://localhost:8051
2020-10-25T21:49:59.228Z - warn: [Transaction]: _validatePeerResponses: Received error response from peer: message=Failed to connect before the deadline URL:grpcs://localhost:8051, stack=Error: Failed to connect before the deadline URL:grpcs://localhost:8051
    at checkState (/media/mario/storage/Documents/Development/HyperLedger/ibmRaftFabricSample-HoreaPorutiu/web-app/server/node_modules/grpc/src/client.js:720:16), connectFailed=true
2020-10-25T21:49:59.229Z - warn: [Transaction]: _validatePeerResponses: Received error response from peer: message=cannot retrieve package for chaincode fabcar/1.0, error open /var/hyperledger/production/chaincodes/fabcar.1.0: no such file or directory, stack=Error: cannot retrieve package for chaincode fabcar/1.0, error open /var/hyperledger/production/chaincodes/fabcar.1.0: no such file or directory
    at self._endorserClient.processProposal (/media/mario/storage/Documents/Development/HyperLedger/ibmRaftFabricSample-HoreaPorutiu/web-app/server/node_modules/fabric-client/lib/Peer.js:140:36)
    at Object.onReceiveStatus (/media/mario/storage/Documents/Development/HyperLedger/ibmRaftFabricSample-HoreaPorutiu/web-app/server/node_modules/grpc/src/client_interceptors.js:1191:9)
    at InterceptingListener._callNext (/media/mario/storage/Documents/Development/HyperLedger/ibmRaftFabricSample-HoreaPorutiu/web-app/server/node_modules/grpc/src/client_interceptors.js:564:42)
    at InterceptingListener.onReceiveStatus (/media/mario/storage/Documents/Development/HyperLedger/ibmRaftFabricSample-HoreaPorutiu/web-app/server/node_modules/grpc/src/client_interceptors.js:614:8)
    at callback (/media/mario/storage/Documents/Development/HyperLedger/ibmRaftFabricSample-HoreaPorutiu/web-app/server/node_modules/grpc/src/client_interceptors.js:841:24), status=500, , url=grpcs://localhost:10051, name=peer1.org2.example.com, name=peer1.org2.example.com, request-timeout=120001, grpc.ssl_target_name_override=peer1.org2.example.com, grpc.default_authority=peer1.org2.example.com, grpc.max_receive_message_length=-1, grpc.max_send_message_length=-1, isProposalResponse=true
2020-10-25T21:49:59.293Z - info: [TransactionEventHandler]: _onError: received error from peer localhost:8051: Error: 14 UNAVAILABLE: Connect Failed
2020-10-25T21:50:02.273Z - error: [Remote.js]: Error: Failed to connect before the deadline URL:grpcs://localhost:7050
2020-10-25T21:50:02.274Z - error: [Orderer.js]: Orderer grpcs://localhost:7050 has an error Error: Failed to connect before the deadline URL:grpcs://localhost:7050 
Failed to submit transaction: Error: Failed to connect before the deadline URL:grpcs://localhost:7050
::1 - - [25/Oct/2020:21:50:02 +0000] "POST /createCar HTTP/1.1" 200 76 "http://localhost:4200/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"
starting to queryAllCars
Wallet path: /media/mario/storage/Documents/Development/HyperLedger/ibmRaftFabricSample-HoreaPorutiu/web-app/server/wallet
Transaction has been evaluated, result is: [{"Key":"CAR0","Record":{"color":"blue","make":"Toyota","model":"Prius","owner":"Tomoko","docType":"car"}},{"Key":"CAR1","Record":{"color":"red","make":"Ford","model":"Mustang","owner":"Brad","docType":"car"}},{"Key":"CAR10","Record":{"color":"CAR10","docType":"car","make":"WV","model":"Golf","owner":"Koakh"}},{"Key":"CAR11","Record":{"color":"CAR11","docType":"car","make":"Audi","model":"A3","owner":"Pelo"}},{"Key":"CAR12","Record":{"color":"CAR11","docType":"car","make":"Audi","model":"A4","owner":"Jomu"}},{"Key":"CAR2","Record":{"color":"green","make":"Hyundai","model":"Tucson","owner":"Jin Soo","docType":"car"}},{"Key":"CAR3","Record":{"color":"yellow","make":"Volkswagen","model":"Passat","owner":"Max","docType":"car"}},{"Key":"CAR4","Record":{"color":"black","make":"Tesla","model":"S","owner":"Adriana","docType":"car"}},{"Key":"CAR5","Record":{"color":"purple","make":"Peugeot","model":"205","owner":"Michel","docType":"car"}},{"Key":"CAR6","Record":{"color":"white","make":"Chery","model":"S22L","owner":"Aarav","docType":"car"}},{"Key":"CAR7","Record":{"color":"violet","make":"Fiat","model":"Punto","owner":"Pari","docType":"car"}},{"Key":"CAR8","Record":{"color":"indigo","make":"Tata","model":"Nano","owner":"Valeria","docType":"car"}},{"Key":"CAR9","Record":{"color":"brown","make":"Holden","model":"Barina","owner":"Shotaro","docType":"car"}}]
::1 - - [25/Oct/2020:21:50:02 +0000] "GET /queryAllCars HTTP/1.1" 304 - "http://localhost:4200/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"

my production network

2020-10-24 20:28:49.466 UTC [grpc] createTransport -> DEBU 0f3 grpc: addrConn.createTransport failed to connect to {orderer1.example.com:7050 0  <nil>}. Err :connection error: desc = "transport: error while dialing: dial tcp 192.168.1.61:7050: connect: no route to host". Reconnecting...
2020-10-24 20:28:49.467 UTC [grpc] HandleSubConnStateChange -> DEBU 0f4 pickfirstBalancer: HandleSubConnStateChange: 0xc0003af7e0, TRANSIENT_FAILURE

we can see that network always try to connect to ordeder and fails

orderer1.example.com:7050: Err :connection error: desc = "transport: error while dialing: dial tcp 192.168.1.61:7050

this only occur in both cases it I crash first ordeder/ordeder1,
if this is normal, the network will not be resillent,
it depends that ordeder/ordeder1 to be online, and if fail everything fails too

and sorry to ask I try google but I can't find answear for the problem

thank you very, very much

[koakh]

sorry for the typo in ordeder...