RFE: trace TLS in container #65

vincentmli · 2022-05-30T17:26:14Z

Hi,

for example I have a netshoot pod running in kubernetes, when I run curl

kubectl exec -it netshoot-hostnetwork  -- curl -k -v https://10.1.34.88/

I want to trace the TLS connection from the curl from the netshoot-hostnetwork pod, the curl in netshoot-hostnetwork pod has libssl below in the pod namespace

kubectl exec -it netshoot-hostnetwork  -- ldd /usr/bin/curl
	/lib/ld-musl-x86_64.so.1 (0x7f932eda0000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x7f932ece5000)
	libz.so.1 => /lib/libz.so.1 (0x7f932eccb000)
	libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f932eda0000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x7f932eca6000)
	libssl.so.1.1 => /lib/libssl.so.1.1 (0x7f932ec25000)
	libcrypto.so.1.1 => /lib/libcrypto.so.1.1 (0x7f932e9a4000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x7f932e998000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x7f932e975000)

is this possible? or in general, could we improve ecapture to capture container TLS traffic?

The text was updated successfully, but these errors were encountered:

vincentmli · 2022-05-30T18:05:04Z

I could do this:

find the netshoot docker ID

docker ps | grep -w netshoot
b37ffd7a8341   nicolaka/netshoot           "/bin/sleep 3600"        50 minutes ago      Up 50 minutes                k8s_netshoot_netshoot-hostnetwork_default_751666e9-5a23-4da1-952f-379288b47f97_0

docker inspect the ID

docker inspect b37ffd7a8341  | grep '"MergedDir"'
                "MergedDir": "/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged",

find libssl

find /var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged -name "libssl*"

/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1
/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/usr/lib/libssl.so.1.1

use ecapture with the correct libssl path

ecapture tls --libssl="/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1" --hex
2022/05/30 14:02:27 pid info :2825032
2022/05/30 14:02:27 start to run EBPFProbeOPENSSL module
2022/05/30 14:02:27 start to run EBPFProbeGNUTLS module
2022/05/30 14:02:27 HOOK type:2, binrayPath:/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1
2022/05/30 14:02:27 libPthread so Path:/lib64/libpthread.so.0
2022/05/30 14:02:27 target all process. 
2022/05/30 14:02:27 start to run EBPFProbeNSPR module
2022/05/30 14:02:27 stat /usr/lib/libnspr4.so: no such file or directory
2022/05/30 14:02:27 HOOK type:2, binrayPath:/lib64/libgnutls.so.30
2022/05/30 14:02:27 target all process.

execute the curl from netshoot pod

kubectl exec -it netshoot-hostnetwork  -- curl -k -v https://10.1.34.88/
*   Trying 10.1.34.88:443...
* Connected to 10.1.34.88 (10.1.34.88) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=US; ST=WA; L=Seattle; O=MyCompany; OU=IT; CN=localhost.localdomain; [email protected]
*  start date: Oct  1 19:29:04 2020 GMT
*  expire date: Sep 29 19:29:04 2030 GMT
*  issuer: C=US; ST=WA; L=Seattle; O=MyCompany; OU=IT; CN=localhost.localdomain; [email protected]
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET / HTTP/1.1
> Host: 10.1.34.88
> User-Agent: curl/7.83.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: BigIP
* HTTP/1.0 connection set to keep alive
< Connection: Keep-Alive
< Content-Length: 10
< 
* Connection #0 to host 10.1.34.88 left intact
 IT WORKS

the ecapture output:

2022/05/30 14:03:34 PID:2825502, Comm:curl, TID:2825502, Version:TLS1_2_VERSION, Send 74 bytes to [ADDR_NOT_FOUND], Payload:
GET / HTTP/1.1
Host: 10.1.34.88
User-Agent: curl/7.83.1
Accept: */*


2022/05/30 14:03:34 PID:2825502, Comm:curl, TID:2825502, Version:TLS1_2_VERSION, Recived 88 bytes from [ADDR_NOT_FOUND], Payload:
HTTP/1.0 200 OK
Server: BigIP
Connection: Keep-Alive
Content-Length: 10

 IT WORKS

cfc4n · 2022-05-31T04:09:26Z

yes, you are right. eh....Is your problem solved?

vincentmli · 2022-05-31T04:11:18Z

yes it is resolved, thanks for this great project :)

BurlyLuo · 2022-05-31T08:43:52Z

yes it is resolved, thanks for this great project :)

Thanks the test.
Can we have a sample to use this tool into the container?

cfc4n · 2022-05-31T10:59:34Z

@BurlyLuo That is a good idea. can you send a PR for it? or an article?

vincentmli · 2022-05-31T14:03:41Z

hm, this brings me thoughts about multi tenant cloud environment, that someone could run ecapture as container and gain privilege on the node and sniff other tenant containers TLS connection, is that possible ?

cfc4n · 2022-05-31T14:15:05Z

Of course, if the container is authorized with the SYS_ADMIN permission, then it can obtain the communication plaintext of all networks on this host.

So, That is an other topic about eBPF security on runtime.

ref: https://github.com/ehids/ebpf-slide/blob/master/security/us-21-With-Friends-Like-EBPF-Who-Needs-Enemies.pdf
demo: https://www.youtube.com/watch?v=zgsHc8apFGs
posts on chinese: https://www.cnxct.com/evil-use-ebpf-and-how-to-detect-ebpf-rootkit-in-linux/?f=ecapture-github

vincentmli · 2022-05-31T14:33:43Z

I wonder if https://github.com/cilium/tetragon is able to detect this scenario, I assume your ehids could detect this too

cfc4n · 2022-06-01T04:12:26Z

tetragon is the learning objective of ehids.

vincentmli · 2022-06-01T15:28:10Z

I am not familiar with docker, can we create a docker file for ecapture to run ecapture in container/pod ?

cfc4n · 2022-06-01T15:49:45Z

I think It is not necessary. via: #23

vincentmli · 2022-06-01T15:58:18Z

ok, good, just one more question, is it possible to give --libssl argument multiple libssl path so ecapture could capture multiple TLS connections for multiple processes/commands ? I guess we can run multiple ecapture commands to achieve that, just wonder :)

cfc4n · 2022-06-01T16:34:24Z

1, eCapture can capture all process TLS paintext who use the same libssl.so default. and can use pid arg to filter.
2, Can run multiple eCapture with every libssl path to achieve that.
3, Of course, an eCapture process can hook multiple ssl/tls SO file, but I don't think this is a necessary requirement.

vincentmli · 2022-06-01T16:40:30Z

ah, right I got 1), I am thinking in k8s pod/container scenario that each pod/container might have their own libssl copy, yes 2) can achieve that.

vincentmli · 2022-06-01T17:34:34Z

by the way, I recorded a short video playing with ecapture https://youtu.be/Au1YeB0nz3g

cfc4n · 2022-06-01T17:49:49Z

Good job. I'll create User Manual WIKI tomorrow with your video, thanks. 😊

vincentmli · 2022-06-03T14:59:41Z

Hi @cfc4n sorry to bother you again, I added the ecapture binary in netshoot pod like vincentmli/netshoot@72633ba so I can run ecapture from netshoot pod in k8s, the netshoot pod yaml file has privilege permission like below

apiVersion: v1
kind: Pod
metadata:
 name: netshoot-ecap
spec:
 hostPID: true
 hostNetwork: true
 nodeSelector:
    dedicated: master
 containers:
   - name: netshoot-ecap
     image: vli39/netshoot:ecap
     command:
       - /bin/sleep
       - "3600"
     volumeMounts:
       - mountPath: /mnt
         name: host-slash
     securityContext:
       privileged: true
 volumes:
   - name: host-slash
     hostPath:
       path: /
       type: ''

you can see I mounted the node root / in netshoot pod /mnt to access the host libssl, but it errors out with "lstat /etc/ld.so.conf: no such file or directory", I wonder why ecapture is trying to check host /etc/ld.so.conf, ecapture is statically build, I suspect the host libssl requiring host /etc/ld.so.conf, but I can't mount host /etc to netshoot pod /etc, wonder if you have better idea :)

kubectl exec -it netshoot-ecap -- /bin/bash
bash-5.1# ls -l /mnt/lib64/libssl*
lrwxrwxrwx    1 root     root            16 Mar 30  2021 /mnt/lib64/libssl.so -> libssl.so.1.1.1g
lrwxrwxrwx    1 root     root            16 Mar 30  2021 /mnt/lib64/libssl.so.1.1 -> libssl.so.1.1.1g
-rwxr-xr-x    1 root     root        615576 Mar 30  2021 /mnt/lib64/libssl.so.1.1.1g
-rwxr-xr-x    1 root     root        394632 Dec 18  2020 /mnt/lib64/libssl3.so

bash-5.1# ecapture tls --libssl="/mnt/lib64/libssl.so.1.1.1g" --hex
2022/06/03 14:51:29 pid info :1391071
2022/06/03 14:51:29 start to run EBPFProbeOPENSSL module
2022/06/03 14:51:29 lstat /etc/ld.so.conf: no such file or directory
2022/06/03 14:51:29 invalid argument

cfc4n · 2022-06-04T14:18:42Z

eh, eCapture will find pthread.so to capture IP ADDRESS default , and you can set pthread.so argument with pthread falg.

vincentmli · 2022-06-05T04:16:16Z

ecapture is statically built

ldd /usr/local/bin/ecapture
	not a dynamic executable

but are you saying eCapture dynamically linked with pthread.so? how do I specify ecapture with pthread flag? sorry I am not following you :)

BurlyLuo · 2022-06-05T04:37:42Z

eh, eCapture will find pthread.so to capture IP ADDRESS default , and you can set pthread.so argument with pthread falg.

bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libssl.so.1.1" --pthread="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libpthread.so.0" 
2022/06/05 04:36:54 pid info :10668
2022/06/05 04:36:54 start to run EBPFProbeOPENSSL module
2022/06/05 04:36:54 start to run EBPFProbeGNUTLS module
2022/06/05 04:36:54 invalid argument
bash-5.1#

bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libssl.so.1.1" --pthread="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libpthread.so.0"  -h
NAME:
        tls - alias name:openssl , use to capture tls/ssl text content without CA cert.

USAGE:
        ecapture tls [flags]

DESCRIPTION:
        use eBPF uprobe to capture process event data, not used libpcap.
        Can used to trace, debug, database audit, security event aduit etc.

OPTIONS:
      --curl=""         curl or wget file path, use to dectet openssl.so path, default:/usr/bin/curl
      --firefox=""      firefox file path, default: /usr/lib/firefox/firefox.
      --gnutls=""       libgnutls.so file path, will automatically find it from curl default.
  -h, --help[=false]    help for tls
      --libssl=""       libssl.so file path, will automatically find it from curl default.
      --nspr=""         libnspr44.so file path, will automatically find it from curl default.
      --pthread=""      libpthread.so file path, use to hook connect to capture socket FD.will automatically find it from curl.
      --wget=""         wget file path, default: /usr/bin/wget.

GLOBAL OPTIONS:
  -d, --debug[=false]   enable debug logging
      --hex[=false]     print byte strings as hex encoded strings
  -p, --pid=0           if pid is 0 then we target all pids

bash-5.1#

cfc4n · 2022-06-05T08:45:51Z

ecapture is statically built
ldd /usr/local/bin/ecapture
	not a dynamic executable
but are you saying eCapture dynamically linked with pthread.so? how do I specify ecapture with pthread flag? sorry I am not following you :)

sure, ecapture is statically built . but eCapture hook connect()function to capture IP ADDRESS ,sometimes connect function was define inpthread.so. so need to search the path.

https://github.com/ehids/ecapture/blob/ed5ebf13bb12873292625d550ffa902c96124388/kern/openssl_kern.c#L285-L290

https://github.com/ehids/ecapture/blob/ed5ebf13bb12873292625d550ffa902c96124388/user/probe_openssl.go#L157-L161

cfc4n · 2022-06-05T08:53:41Z

sometimes , defined in /lib64/libc.so.6 , you can use ldd which curl|grep libc.so|awk '{print $3}'|xargs objdump -T |grep connect to confirm it.

[root@VM-16-13-centos bin]# ldd `which curl`|grep libc.so|awk '{print $3}'|xargs objdump -T |grep connect
0000000000112f90  w   DF .text	000000000000009b  GLIBC_2.2.5 __connect
0000000000112f90  w   DF .text	000000000000009b  GLIBC_2.2.5 connect

@BurlyLuo @vincentmli
It will skip directory search if lib path args was set.

vincentmli · 2022-06-05T15:16:23Z

ah, sorry I missed the output libPthread so Path:/lib64/libpthread.so.0 when run ecapture normally

 ecapture tls  --libssl=/lib64/libssl.so.1.1.1g
2022/06/05 10:53:59 pid info :3232826
2022/06/05 10:53:59 start to run EBPFProbeOPENSSL module
2022/06/05 10:53:59 start to run EBPFProbeGNUTLS module
2022/06/05 10:53:59 stat /usr/lib/libgnutls.so.30: no such file or directory
2022/06/05 10:53:59 start to run EBPFProbeNSPR module
2022/06/05 10:53:59 stat /usr/lib/libnspr4.so: no such file or directory
2022/06/05 10:53:59 HOOK type:2, binrayPath:/lib64/libssl.so.1.1.1g
2022/06/05 10:53:59 libPthread so Path:/lib64/libpthread.so.0 <====MISSED THIS 
2022/06/05 10:53:59 target all process.

but still error when I run ecapture in container with --pthread=/mnt/lib64/libpthread.so.0, it looks to me when putting ecapture in container, for some reason, ecapture could not find the correct pthread lib even with --pthread argument, and still try to locate /etc/ld.so.conf

kubectl exec -it netshoot-ecap -- /bin/bash
bash-5.1# 

bash-5.1# ls -l /mnt/lib64/libpth*
-rwxr-xr-x    1 root     root        320504 Jul 20  2020 /mnt/lib64/libpthread-2.28.so
lrwxrwxrwx    1 root     root            27 Jul 20  2020 /mnt/lib64/libpthread.so -> ../../lib64/libpthread.so.0
lrwxrwxrwx    1 root     root            18 Jul 20  2020 /mnt/lib64/libpthread.so.0 -> libpthread-2.28.so

bash-5.1# ecapture tls --libssl=/mnt/lib64/libssl.so.1.1.1g --pthread=/mnt/lib64/libpthread.so.0
2022/06/05 15:01:40 pid info :3237812
2022/06/05 15:01:40 start to run EBPFProbeOPENSSL module
2022/06/05 15:01:40 start to run EBPFProbeGNUTLS module
2022/06/05 15:01:40 lstat /etc/ld.so.conf: no such file or directory
2022/06/05 15:01:40 invalid argument

vincentmli · 2022-06-05T15:34:25Z

tried create /etc/ld.so.conf with mounted host / in container, still can't find connect I guess

bash-5.1# echo "/mnt/lib64" > /etc/ld.so.conf
bash-5.1# cat /etc/ld.so.conf
/mnt/lib64

bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libssl.so.1.0.2k"  --hex
2022/06/05 15:32:04 pid info :3257576
2022/06/05 15:32:04 start to run EBPFProbeOPENSSL module
2022/06/05 15:32:04 start to run EBPFProbeGNUTLS module
2022/06/05 15:32:04 invalid argument

bash-5.1# ecapture tls --pthread="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libpthread.so.0" --libssl="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libssl.so.1.0.2k"  --hex
2022/06/05 15:29:01 pid info :3255607
2022/06/05 15:29:01 start to run EBPFProbeOPENSSL module
2022/06/05 15:29:01 start to run EBPFProbeGNUTLS module
2022/06/05 15:29:01 invalid argument

cfc4n · 2022-06-05T15:59:30Z

may be it's a bug. can you send a new issue for it?

vincentmli · 2022-06-05T16:12:26Z

sure, will do, appreciate the help!

vincentmli · 2022-06-05T16:53:32Z

opened #69

vincentmli · 2022-06-06T15:25:14Z

close this since question is answered by @cfc4n

cfc4n added the good first issue Good for newcomers label May 31, 2022

cfc4n pinned this issue May 31, 2022

cfc4n changed the title ~~RFE: trace TLS in container?~~ RFE: trace TLS in container May 31, 2022

vincentmli closed this as completed Jun 6, 2022

cfc4n mentioned this issue Sep 21, 2023

isContainerCgroup() for linux doesn't work in certain scenarios #395

Closed

cfc4n mentioned this issue Oct 14, 2023

Got "couldn't init manager error" when launching ecapture in docker container. #403

Closed

blaisewang mentioned this issue Dec 12, 2023

是否能够支持Docker中https流量解密 #441

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFE: trace TLS in container #65

RFE: trace TLS in container #65

vincentmli commented May 30, 2022 •

edited

vincentmli commented May 30, 2022

cfc4n commented May 31, 2022

vincentmli commented May 31, 2022

BurlyLuo commented May 31, 2022 •

edited

cfc4n commented May 31, 2022

vincentmli commented May 31, 2022

cfc4n commented May 31, 2022

vincentmli commented May 31, 2022

cfc4n commented Jun 1, 2022

vincentmli commented Jun 1, 2022

cfc4n commented Jun 1, 2022

vincentmli commented Jun 1, 2022

cfc4n commented Jun 1, 2022

vincentmli commented Jun 1, 2022

vincentmli commented Jun 1, 2022

cfc4n commented Jun 1, 2022 •

edited

vincentmli commented Jun 3, 2022 •

edited

cfc4n commented Jun 4, 2022

vincentmli commented Jun 5, 2022

BurlyLuo commented Jun 5, 2022 •

edited

cfc4n commented Jun 5, 2022 •

edited

cfc4n commented Jun 5, 2022 •

edited

vincentmli commented Jun 5, 2022

vincentmli commented Jun 5, 2022

cfc4n commented Jun 5, 2022

vincentmli commented Jun 5, 2022

vincentmli commented Jun 5, 2022

vincentmli commented Jun 6, 2022

RFE: trace TLS in container #65

RFE: trace TLS in container #65

Comments

vincentmli commented May 30, 2022 • edited

vincentmli commented May 30, 2022

cfc4n commented May 31, 2022

vincentmli commented May 31, 2022

BurlyLuo commented May 31, 2022 • edited

cfc4n commented May 31, 2022

vincentmli commented May 31, 2022

cfc4n commented May 31, 2022

vincentmli commented May 31, 2022

cfc4n commented Jun 1, 2022

vincentmli commented Jun 1, 2022

cfc4n commented Jun 1, 2022

vincentmli commented Jun 1, 2022

cfc4n commented Jun 1, 2022

vincentmli commented Jun 1, 2022

vincentmli commented Jun 1, 2022

cfc4n commented Jun 1, 2022 • edited

vincentmli commented Jun 3, 2022 • edited

cfc4n commented Jun 4, 2022

vincentmli commented Jun 5, 2022

BurlyLuo commented Jun 5, 2022 • edited

cfc4n commented Jun 5, 2022 • edited

cfc4n commented Jun 5, 2022 • edited

vincentmli commented Jun 5, 2022

vincentmli commented Jun 5, 2022

cfc4n commented Jun 5, 2022

vincentmli commented Jun 5, 2022

vincentmli commented Jun 5, 2022

vincentmli commented Jun 6, 2022

vincentmli commented May 30, 2022 •

edited

BurlyLuo commented May 31, 2022 •

edited

cfc4n commented Jun 1, 2022 •

edited

vincentmli commented Jun 3, 2022 •

edited

BurlyLuo commented Jun 5, 2022 •

edited

cfc4n commented Jun 5, 2022 •

edited

cfc4n commented Jun 5, 2022 •

edited