Memory leak

[erfan-khadem]

There is / has been a memory leak issue since at least version 1.8.9. I have tested the latest release, and it is leaking at a rate of about 1.7 GB for every 100 GB of traffic proxied, during a time frame of 24 hours and with 50 active users. I have not done experiments to correlate these numbers with each other, but it seems like the amount of leaked memory is proportional to the served traffic. There is also no ceiling to this, it just grows until systemd-oomd kills it. I have tried using vless only, vless+vmess and vless+trojan, and avoided the newly added transports but to no avail. I would be more than happy to experiment more and provide the team with more information. I am running xray behind haproxy and that behind cloudflare CDN.

OS: Ubuntu 22.04, all packages up-to-date.
Architecture: Armv8, Neoverse-N1 at Hetzner. Also generic x86 platforms.
Memory: 4GB
Kernel: Generic Ubuntu 5.15.0-xxx
Logging: Disabled

[erfan-khadem]

heap.zip
Here is the result of heap profiling using pprof. memory usage was about 800MB when I took this profile.

[GFW-knocker]

@er888kh
what kind of transmission do you use?
i know that grpc has memory leak in 1.8.0+ above
but ws is rather fine in my experience

[50infivedays]

same problem , in my experience , version 1.8.4 is fine , and after that version , xray will leak

[RPRX]

same problem , in my experience , version 1.8.4 is fine , and after that version , xray will leak

麻烦逐个测试 v1.8.4 至 v1.8.6 之间的 commits，看一下哪个 commit 出现该问题

@er888kh 你那边也是这样吗？

[yuhan6665]

heap.zip Here is the result of heap profiling using pprof. memory usage was about 800MB when I took this profile.

From this pprof, the biggest is at readv reader with 220MB. Although I'm not sure if this is indeed a leak. Can you profile a bigger usage? (Or reduce the buffer size to isolate the issue)

[GektorUA]

Same issue on MT7981A (ARM64, cortex A53) with 512MB RAM, have try 1.8.4 and 1.8.10, both consume all RAM on device after first speedtest running and hangs, switch to sing-box 1.8.10 without change anything on server side (xray-core 1.8.10) and have no issues with memory consumption (~180 MB RAM after 10 and more tests).

P.S.
buffer-size is set to 4

[amirhosss]

Same issue on MT7981A (ARM64, cortex A53) with 512MB RAM, have try 1.8.4 and 1.8.10, both consume all RAM on device after first speedtest running and hangs

Same problem when running speedtest with chain-proxy config and using Xray-Core

[taoabc]

Same problem

[shakibamoshiri]

my VM faced an OS crash yesterday because of OOM.

here is the screenshot of the console after the crash

never did profiling for xray , but could it be for memory leaks ?
protocol: vless-grpc
number of users: around 30/40
number of concurrent users : 10/15
OS : Debian 11 x64
physical memory: 1G
swap : off
xray : Xray 1.8.7 (Xray, Penetrates Everything.) 3f0bc13 (go1.21.5 linux/amd64)

total-vm: 2558796KB
anon-rss: 561376KB
file-rss: 0KB

which protocol has the least memory leak ?

did not have full console access and so was not able to recover the OS; I had to rebuild it and no further logs

[taoabc]

my VM faced an OS crash yesterday because of OOM.

here is the screenshot of the console after the crash

never did profiling for xray , but could it be for memory leaks ? protocol: vless-grpc number of users: around 30/40 number of concurrent users : 10/15 OS : Debian 11 x64 physical memory: 1G swap : off xray : Xray 1.8.7 (Xray, Penetrates Everything.) 3f0bc13 (go1.21.5 linux/amd64)

total-vm: 2558796KB anon-rss: 561376KB file-rss: 0KB

which protocol has the least memory leak ?

did not have full console access and so was not able to recover the OS; I had to rebuild it and no further logs

I updated to 1.8.11, seems it's been mitigated. While it's easy to repro when I use 1.8.10