Document common troubleshooting steps

[cfryanr]

e.g. how to turn up istio-proxy's log level

For a sidecar proxy:
kubectl exec $applicationPod -c istio-proxy -- curl -X POST http://localhost:15000/logging?level=trace

For the ingress proxy:
kubectl exec -n istio-system $istioIngressgatewayPod -- curl -X POST http://localhost:15000/logging?level=trace

To turn the log level down again, repeat the same command, but substitute debug for trace.

e.g. how to enable GRPC logging on the authservice container

      - name: authservice
        image: <AUTHSERVICE_IMAGE_REPLACE_ME>
        imagePullPolicy: Always
        ports:
          - containerPort: 10003
        volumeMounts:
          - name: bookinfo-authservice-configmap-volume
            mountPath: /etc/authservice
        env: # ADDING THESE TWO ENV VARS TO ENABLE VERBOSE LOGGING
          - name: GRPC_VERBOSITY
            value: DEBUG
          - name: GRPC_TRACE
            value: all

and the interesting part of the authservice container logs will then look like this:

I0404 17:58:06.696553229       1 tcp_posix.cc:611]           TCP:0x558ab1a7cb60 got_read: "No Error"
I0404 17:58:06.696557156       1 tcp_posix.cc:602]           TCP:0x558ab1a7cb60 do_read
I0404 17:58:06.696577421       1 tcp_posix.cc:413]           TCP:0x558ab1a7cb60 call_cb 0x558ab1a7d360 0x558aaf5f5a4b:0x558ab1a7d1a0
I0404 17:58:06.696581557       1 tcp_posix.cc:416]           READ 0x558ab1a7cb60 (peer=ipv4:127.0.0.1:60780) error="No Error"
D0404 17:58:06.696709307       1 tcp_posix.cc:422]           DATA: 00 00 60 01 04 00 00 00 13 83 04 9e 60 b5 77 3f a5 d0 5b 3b 98 85 5c 76 a6 75 fb 89 78 6d 4c e7 b0 de c6 93 1e a6 2f 4e 52 75 e0 86 df de dd 7f 01 96 64 41 00 c8 d9 5e 0c ac c9 1b 80 92 50 8a 56 c0 d1 40 50 12 52 b5 7f 01 8c 04 85 92 52 88 25 13 ef 3c f4 92 7f 7f 01 8b 00 89 2c a2 76 51 b2 17 48 e0 df dc db da d9 00 06 45 00 01 00 00 00 13 00 00 00 06 40 0a bd 0c 0a 13 0a 11 0a 0f 12 09 31 30 2e 31 32 2e 30 2e 38 18 fe bd 03 12 13 0a 11 0a 0f 12 0a 31 30 2e 31 32 2e 33 2e 34 34 18 90 3f 22 8e 0c 12 8b 0c 0a 13 34 31 31 37 32 38 33 39 36 33 30 34 34 33 34 34 34 31 34 12 03 47 45 54 1a 0e 0a 07 3a 6d 65 74 68 6f 64 12 03 47 45 54 1a 24 0a 0f 61 63 63 65 70 74 2d 65 6e 63 6f 64 69 6e 67 12 11 67 7a 69 70 2c 20 64 65 66 6c 61 74 65 2c 20 62 72 1a 60 0a 05 3a 70 61 74 68 12 57 2f 74 6f 64 6f 73 2f 6f 61 75 74 68 2f 63 61 6c 6c 62 61 63 6b 3f 63 6f 64 65 3d 77 79 4b 37 39 37 30 48 6a 4e 26 73 74 61 74 65 3d 64 61 55 72 43 4f 6f 6e 49 62 55 70 4a 50 37 52 47 69 54 53 7a 4b 33 51 74 30 54 6f 63 31 34 5f 6a 4e 67 65 6c 46 4f 4a 62 69 77 1a 32 0a 07 72 65 66 65 72 65 72 12 27 68 74 74 70 73 3a 2f 2f 64 65 6d 6f 2e 6c 6f 67 69 6e 2e 72 75 6e 2e 70 69 76 6f 74 61 6c 2e 69 6f 2f 6c 6f 67 69 6e 1a 1a 0a 11 78 2d 66 6f 72 77 61 72 64 65 64 2d 70 72 6f 74 6f 12 05 68 74 74 70 73 1a 18 0a 0f 61 63 63 65 70 74 2d 6c 61 6e 67 75 61 67 65 12 05 65 6e 2d 75 73 1a 1f 0a 0b 78 2d 62 33 2d 73 70 61 6e 69 64 12 10 64 66 31 32 66 35 30 34 65 30 65 30 63 66 65 34 1a cf 01 0a 06 63 6f 6f 6b 69 65 12 c4 01 5f 5f 48 6f 73 74 2d 61 75 74 68 63 6f 64 65 2d 73 61 6d 70 6c 65 2d 61 70 70 2d 61 75 74 68 73 65 72 76 69 63 65 2d 73 65 73 73 69 6f 6e 2d 69 64 2d 63 6f 6f 6b 69 65 3d 31 4f 4f 36 76 6e 5f 33 34 48 66 58 52 34 68 55 6a 36 69 46 51 74 67 44 46 57 38 30 2d 4e 5a 35 2d 65 2d 70 75 32 5a 77 55 36 43 6f 46 47 53 6c 44 69 69 6d 49 4e 45 41 69 4f 47 70 58 65 46 37 35 57 54 62 78 79 30 31 30 48 77 48 6b 6b 38 6f 77 55 33 4c 71 67 3b 20 61 75 74 68 63 6f 64 65 2d 73 65 73 73 69 6f 6e 2d 61 66 66 69 6e 69 74 79 2d 63 6f 6f 6b 69 65 3d 22 62 64 35 31 32 32 61 36 32 64 32 32 62 33 32 31 22 1a 1b 0a 0a 3a 61 75 74 68 6f 72 69 74 79 12 0d 33 35 2e 32 33 39 2e 34 39 2e 31 31 36 1a 11 0a 0c 78 2d 62 33 2d 73 61 6d 70 6c 65 64 12 01 31 1a cf 03 0a 12 78 2d 69 73 74 69 6f 2d 61 74 74 72 69 62 75 74 65 73 12 b8 03 43 6b 73 4b 47 47 52 6c 63 33 52 70 62 6d 46 30 61 57 39 75 4c 6e 4e 6c 63 6e 5a 70 59 32 55 75 61 47 39 7a 64 42 49 76 45 69 31 68 64 58 52 6f 59 32 39 6b 5a 53 31 7a 59 57 31 77 62 47 55 74 59 58 42 77 4c 6d 52 6c 5a 6d 46 31 62 48 51 75 63 33 5a 6a 4c 6d 4e 73 64 58 4e 30 5a 58 49 75 62 47 39 6a 59 57 77 4b 53 51 6f 58 5a 47 56 7a 64 47 6c 75 59 58 52 70 62 32 34 75 63 32 56 79 64 6d 6c 6a 5a 53 35 31 61 57 51 53 4c 68 49 73 61 58 4e 30 61 57 38 36 4c 79 39 6b 5a 57 5a 68 64 57 78 30 4c 33 4e 6c 63 6e 5a 70 59 32 56 7a 4c 32 46 31 64 47 68 6a 62 32 52 6c 4c 58 4e 68 62 58 42 73 5a 53 31 68 63 48 41 4b 4d 51 6f 59 5a 47 56 7a 64 47 6c 75 59 58 52 70 62 32 34 75 63 32 56 79 64 6d 6c 6a 5a 53 35 75 59 57 31 6c 45 68 55 53 45 32 46 31 64 47 68 6a 62 32 52 6c 4c 58 4e 68 62 58 42 73 5a 53 31 68 63 48 41 4b 4b 67 6f 64 5a 47 56 7a 64 47 6c 75 59 58 52 70 62 32 34 75 63 32 56 79 64 6d 6c 6a 5a 53 35 75 59 57 31 6c 63 33 42 68 59 32 55 53 43 52 49 48 5a 47 56 6d 59 58 56 73 64 41 70 50 43 67 70 7a 62 33 56 79 59 32 55 75 64 57 6c 6b 45 6b 45 53 50 32 74 31 59 6d 56 79 62 6d 56 30 5a 58 4d 36 4c 79 39 70 63 33 52 70 62 79 31 70 62 6d 64 79 5a 58 4e 7a 5a 32 46 30 5a 58 64 68 65 53 30 31 5a 47 4e 6d 5a 6a 6b 32 59 6d 4a 6d 4c 57 34 35 62 6d 31 77 4c 6d 6c 7a 64 47 6c 76 4c 58 4e 35 63 33 52 6c 62 51 3d 3d 1a 34 0a 0c 78 2d 72 65 71 75 65 73 74 2d 69 64 12 24 32 37 39 65 39 66 37 63 2d 32 39 36 66 2d 39 37 36 32 2d 39 35 38 30 2d 34 30 33 63 32 66 33 66 35 34 61 33 1a 1e 0a 0f 78 2d 66 6f 72 77 61 72 64 65 64 2d 66 6f 72 12 0b 31 30 2e 31 32 38 2e 30 2e 34 33 1a 49 0a 06 61 63 63 65 70 74 12 3f 74 65 78 74 2f 68 74 6d 6c 2c 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 68 74 6d 6c 2b 78 6d 6c 2c 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 6d 6c 3b 71 3d 30 2e 39 2c 2a 2f 2a 3b 71 3d 30 2e 38 1a 83 01 0a 0a 75 73 65 72 2d 61 67 65 6e 74 12 75 4d 6f 7a 69 6c 6c 61 2f 35 2e 30 20 28 4d 61 63 69 6e 74 6f 73 68 3b 20 49 6e 74 65 6c 20 4d 61 63 20 4f 53 20 58 20 31 30 5f 31 35 5f 34 29 20 41 70 70 6c 65 57 65 62 4b 69 74 2f 36 30 35 2e 31 2e 31 35 20 28 4b 48 54 4d 4c 2c 20 6c 69 6b 65 20 47 65 63 6b 6f 29 20 56 65 72 73 69 6f 6e 2f 31 33 2e 31 20 53 61 66 61 72 69 2f 36 30 35 2e 31 2e 31 35 1a 30 0a 0c 78 2d 62 33 2d 74 72 61 63 65 69 64 12 20 33 32 31 30 61 63 35 65 38 31 66 33 64 62 36 30 64 66 31 32 66 35 30 34 65 30 65 30 63 66 65 34 1a 18 0a 10 78 2d 65 6e 76 6f 79 2d 69 6e 74 65 72 6e 61 6c 12 04 74 72 75 65 1a 13 0a 0e 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 12 01 30 22 57 2f 74 6f 64 6f 73 2f 6f 61 75 74 68 2f 63 61 6c 6c 62 61 63 6b 3f 63 6f 64 65 3d 77 79 4b 37 39 37 30 48 6a 4e 26 73 74 61 74 65 3d 64 61 55 72 43 4f 6f 6e 49 62 55 70 4a 50 37 52 47 69 54 53 7a 4b 33 51 74 30 54 6f 63 31 34 5f 6a 4e 67 65 6c 46 4f 4a 62 69 77 2a 0d 33 35 2e 32 33 39 2e 34 39 2e 31 31 36 52 08 48 54 54 50 2f 31 2e 31 5a 00 '..`.........`.w?..[;..\v.u..xmL......./NRu........dA...^[email protected].%..<........,.vQ..H........E..........@...........10.12.0.8............10.12.3.44..?".......4117283963044344414..GET....:method..GET.$..accept-encoding..gzip, deflate, br.`..:path.W/todos/oauth/callback?code=wyK7970HjN&state=daUrCOonIbUpJP7RGiTSzK3Qt0Toc14_jNgelFOJbiw.2..referer.'https://demo.login.run.pivotal.io/login....x-forwarded-proto..https....accept-language..en-us....x-b3-spanid..df12f504e0e0cfe4.....cookie...__Host-authcode-sample-app-authservice-session-id-cookie=1OO6vn_34HfXR4hUj6iFQtgDFW80-NZ5-e-pu2ZwU6CoFGSlDiimINEAiOGpXeF75WTbxy010HwHkk8owU3Lqg; authcode-session-affinity-cookie="bd5122a62d22b321"....:authority..35.239.49.116....x-b3-sampled..1.....x-istio-attributes...CksKGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBIvEi1hdXRoY29kZS1zYW1wbGUtYXBwLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwKSQoXZGVzdGluYXRpb24uc2VydmljZS51aWQSLhIsaXN0aW86Ly9kZWZhdWx0L3NlcnZpY2VzL2F1dGhjb2RlLXNhbXBsZS1hcHAKMQoYZGVzdGluYXRpb24uc2VydmljZS5uYW1lEhUSE2F1dGhjb2RlLXNhbXBsZS1hcHAKKgodZGVzdGluYXRpb24uc2VydmljZS5uYW1lc3BhY2USCRIHZGVmYXVsdApPCgpzb3VyY2UudWlkEkESP2t1YmVybmV0ZXM6Ly9pc3Rpby1pbmdyZXNzZ2F0ZXdheS01ZGNmZjk2YmJmLW45bm1wLmlzdGlvLXN5c3RlbQ==.4..x-request-id.$279e9f7c-296f-9762-9580-403c2f3f54a3....x-forwarded-for..10.128.0.43.I..accept.?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.....user-agent.uMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15.0..x-b3-traceid. 3210ac5e81f3db60df12f504e0e0cfe4....x-envoy-internal..true....content-length..0"W/todos/oauth/callback?code=wyK7970HjN&state=daUrCOonIbUpJP7RGiTSzK3Qt0Toc14_jNgelFOJbiw*.35.239.49.116R.HTTP/1.1Z.'

e.g. how to compile with debug symbols and run in the gdb debugger locally, which is good for debugging crashes that don't happen in unit tests by getting stack traces

# First change Makefile to add --compilation_mode=dbg to the end of the command on the bazel-bin/src/main/auth_server target, then:
make docker-from-scratch
docker run -it -u 0 -v /Users/pivotal/workspace/authservice/run/config:/etc/authservice --entrypoint /bin/bash authservice:pivotal
apt-get update
apt-get install gdb -y
gdb /app/auth_server
run
info threads
bt