Deploying any Armada Chart fails with "Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed"

[moffzilla]

If this is a BUG REPORT, please:
I fail to deploy any of the Armada yamls found in the repo with the error:

2017-08-03 22:18:07.200 162 ERROR armada [-] <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>: _Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>

Regardless of the Armada yamls I try I get the same error when using latest or Master or v0.6.0 Docker Armada versions,
When checking Tiller by Armada command or Kubectl , it shows active
here's the outcome from the simple.yaml test
(I attach the file with the debug-logging enabled )
armada-simple-debug-fail.txt

_docker exec armada armada apply examples/simple.yaml
found extension EntryPoint.parse('pika = oslo_messaging._drivers.impl_pika:PikaDriver')
found extension EntryPoint.parse('kafka = oslo_messaging._drivers.impl_kafka:KafkaDriver')
found extension EntryPoint.parse('kombu = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
found extension EntryPoint.parse('rabbit = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
found extension EntryPoint.parse('fake = oslo_messaging._drivers.impl_fake:FakeDriver')
found extension EntryPoint.parse('zmq = oslo_messaging._drivers.impl_zmq:ZmqDriver')
found extension EntryPoint.parse('amqp = oslo_messaging._drivers.impl_amqp1:ProtonDriver')
found extension EntryPoint.parse('log = oslo_messaging.notify._impl_log:LogDriver')
found extension EntryPoint.parse('messagingv2 = oslo_messaging.notify.messaging:MessagingV2Driver')
found extension EntryPoint.parse('noop = oslo_messaging.notify._impl_noop:NoOpDriver')
found extension EntryPoint.parse('routing = oslo_messaging.notify._impl_routing:RoutingDriver')
found extension EntryPoint.parse('test = oslo_messaging.notify._impl_test:TestDriver')
found extension EntryPoint.parse('messaging = oslo_messaging.notify.messaging:MessagingDriver')
2017-08-03 22:17:47.144 162 INFO armada.handlers.armada [-] Performing Pre-Flight Operations
2017-08-03 22:18:07.200 162 ERROR armada [-] <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>: _Rendezvous: <Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>

Deployment Environment?:
Kubernetes: 1.7.0
Docker: 17.06.0-ce
Helm v2.5.0
Ubuntu Xenial
All hosts are deployed in aws/us-east

Kubernetes 1.7.0
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T22:55:19Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Helm v2.5.0
helm version
Client: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}

docker version
Client:
Version: 17.06.0-ce
API version: 1.30
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:23:31 2017
OS/Arch: linux/amd64

Server:
Version: 17.06.0-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:19:04 2017
OS/Arch: linux/amd64
Experimental: false

Armada Status seems OK:

docker exec armada armada tiller --status
found extension EntryPoint.parse('pika = oslo_messaging._drivers.impl_pika:PikaDriver')
found extension EntryPoint.parse('kafka = oslo_messaging._drivers.impl_kafka:KafkaDriver')
found extension EntryPoint.parse('kombu = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
found extension EntryPoint.parse('rabbit = oslo_messaging._drivers.impl_rabbit:RabbitDriver')
found extension EntryPoint.parse('fake = oslo_messaging._drivers.impl_fake:FakeDriver')
found extension EntryPoint.parse('zmq = oslo_messaging._drivers.impl_zmq:ZmqDriver')
found extension EntryPoint.parse('amqp = oslo_messaging._drivers.impl_amqp1:ProtonDriver')
found extension EntryPoint.parse('log = oslo_messaging.notify._impl_log:LogDriver')
found extension EntryPoint.parse('messagingv2 = oslo_messaging.notify.messaging:MessagingV2Driver')
found extension EntryPoint.parse('noop = oslo_messaging.notify._impl_noop:NoOpDriver')
found extension EntryPoint.parse('routing = oslo_messaging.notify._impl_routing:RoutingDriver')
found extension EntryPoint.parse('test = oslo_messaging.notify._impl_test:TestDriver')
found extension EntryPoint.parse('messaging = oslo_messaging.notify.messaging:MessagingDriver')
2017-08-03 22:10:49.130 100 INFO armada.cli.tiller [-] Tiller is Active: True

kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
heapster-v1.4.0-2118284389-fl6kw 4/4 Running 0 3h
kube-dns-3097350089-n2rcm 3/3 Running 0 3h
kubernetes-dashboard-1265873680-xhbs4 1/1 Running 0 3h
monitoring-influxdb-grafana-v4-8bbdt 2/2 Running 0 3h
tiller-deploy-3235729489-j84j1 1/1 Running 0 3h

Release Tag or Master: Latest

Expected Behavior: Armada able to deploy the repo sample yamls

What Actually Happened: Any Armada yaml fails with the same error
"ERROR armada [-] <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>: _Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>"

How to Reproduce the Issue (as minimally as possible):

docker run -d --net host -p 8000:8000 --name armada -v /home/ubuntu/.kube/config:/armada/.kube/config -v /home/ubuntu/armada/examples:/armada/examples quay.io/attcomdev/armada:latest

docker exec armada armada apply examples/armada-aio.yaml --debug-logging
or
docker run -d --net host -p 8000:8000 --name armada -v /home/ubuntu/.kube/config:/armada/.kube/config -v /home/ubuntu/armada/examples:/armada/examples quay.io/attcomdev/armada:latest

Any Additional Comments:
Is there any ports I need to make sure they are open among the hosts besides the 8000?

[alanmeadows]

Thanks for the report. Are there any relevant log lines from the tiller pod itself regarding the connection attempt?

kubectl get logs tiller..

[moffzilla]

Hi Alan

Thanks for your reply,
FYI, My K8 deployment had originally 3 workers but I have rebuild it with just one however same behavior
The Tiller logs doesn't provide much information after it is initialized
I enabled log verbosity at the Docker Armada but not much information there
( I attach the output )

armada-simple-debug-fail.txt
armada08042017.txt

kubectl logs -f tiller-deploy-3235729489-zz6ds -n kube-system
[main] 2017/08/04 14:27:43 Starting Tiller v2.5.0 (tls=false)
[main] 2017/08/04 14:27:43 GRPC listening on :44134
[main] 2017/08/04 14:27:43 Probes listening on :44135
[main] 2017/08/04 14:27:43 Storage driver is ConfigMap

Or from the Pod:

Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message

---

30s 30s 1 default-scheduler Normal Scheduled Successfully assigned tiller-deploy-3235729489-mbw4t to ip-172-31-60-242
29s 29s 1 kubelet, ip-172-31-60-242 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-3shbz"
29s 29s 1 kubelet, ip-172-31-60-242 spec.containers{tiller} Normal Pulled Container image "gcr.io/kubernetes-helm/tiller:v2.5.0" already present on machine
29s 29s 1 kubelet, ip-172-31-60-242 spec.containers{tiller} Normal Created Created container
29s 29s 1 kubelet, ip-172-31-60-242 spec.containers{tiller} Normal Started Started container

[SharpEdgeMarshall]

Same issue here, there is no solution?