Bug: High network load

[Roman-laufenderWahnsinn]

We have had server crashes because of network overloads.

We play with around 50 players, and in the crash situation, we had 10 active Blackhawks in the air.
After investigation, we found that the hatchet H-60 was the reason. We checked the network load with the Black Hawk and found it really ramping up when the pilot entered the BH.

It is in German, as we are a German clan.
Basically, orange is MH 60, and red is AH 60 DAP with one player. The spike was with 10 players as pilots. Every other module was way softer on the network.

[FireStone1029]

would you be able to provide some additional information?

would it be possible to get the graph in english?

what is the graph in measuring exactly?

did all the pilots have all the tac map open at the same time?

did you spawn in the birds in the mission before hand or was there a lag spike every time you spawned in a new type of bird?

[Holgerhollebolle]

would you be able to provide some additional information?

would it be possible to get the graph in english?

what is the graph in measuring exactly?

did all the pilots have all the tac map open at the same time?

did you spawn in the birds in the mission before hand or was there a lag spike every time you spawned in a new type of bird?

Hi FireStone, I am Holger (Technical Department of Titan Platoon). I made the Tests with Roman together. Roman will reply shortly.
Here is my oberservations: So what we figured out is that as soon as a Player enters the MH 60 or AH 60 DAP the Traffic goes up extremely. Not even the Engine of the Birds need to engage. If a player enters the cockpit the Traffic happens (Not the case with Cargo spaces).
The Graph above basically shows the Spike we had with 10 Players in 10 Helicopters (on the ground). The other part of the Graph shows what the usual Traffic is where we tested several other Helicopter and other Stuff. But the Spike as soon as 10 Player enter MH 60 is pretty big. And this test was only with 10 Players. The more Players the bigger the Spike. We usually play with 40+ Players where the traffic peaks at 16.000 Packages per second which brings our server down (And we have the best Server config you can whish for).
We fight this issue since half a year now until we figured out it is caused by the MH 60 and AH 60 DAP. It was a long journey of frustration and server crashes :-(.

In the Graph we meassured packages per second. The Traffic in KB per second is behaving acordingly.
Maybe Roman will tell you more about it...but what we figured out is the MH 60 sends tons of variables From the Server to all Players. Roman did log these Variables into his Log and the log became several hundreds of Megabyte big in a very short time.
My Guess is that all the Instruments data of the MH 60 is sycronized to all Players. But thats only a guess.
If that is the case a fix could be to send this Data only to the Pilot instead. A Player on the ground does not need to know the altitude or speed of a Heli. :-).

[Roman-laufenderWahnsinn]

Sorry for the late reply, there we go:

Here the english translation as best I could.
Basicly, only the Blue Line is important, as it is the overall server packet (!) load. We are talking about packet count (left column).
The packes rise dramatically when the pilot enters the helicopter, and it is per client, meaning they double with 5 to 10 players connected.

After that I started to investigate the code of the MH60 and add to every variable, which gets transferred over the network by the mod, a report and I did count between 229 and 246 variable updates per BH with a pilot. Those get send to every client. This data should, at least mostly, only be relevant to the copilot.
Obviously the players do need rotor rpm, so not every value is wrong, but a lot of those are not really needed on every client:
"vtx_uh60_sfmplus_engOilPSI" for example?

[Roman-laufenderWahnsinn]

To the question about the tac map: all had the default screens open; I was the pilot every time, and only on the 10-player test we had more pilots than me. But in my rtp the engine variables got really spammed.

[Aaren882]

were you using Stable Version?
if so maybe you can try the DEV Version ,I did some optimizations including Network traffic

[Roman-laufenderWahnsinn]

Is there a way to check the version? We are not using the normal stable, I know this much but I am not sure it is the dev. Or maybe not the last.

[Riverman02]

There's two versions of it on the workshop, a Stable and a Dev Build. If you are using some sort of altered version of the workshop mods then it'd be hard for us to troubleshoot because we don't know what you've changed.

[Roman-laufenderWahnsinn]

We only changed stuff with the ctab to jvmf, I checked and we did not change anything in the code affecting network, We added our custom pictures to jvmf. I logged only setVariable with brodcast true and there was a ton from ie engine.
I will speak with my clan if we are up to try the dev version and redoo the network test.
Edit: were do I find a dev branch here? There are a lot and I did not spot a dev branch.

[Roman-laufenderWahnsinn]

We are using this version:
https://github.com/Project-Hatchet/H-60/tree/bugfix/AI_Engagement

[Riverman02]

The Dev Branch is on the workshop