We should rethink our current logs

[J0sueTM]

While testing during development of the task #361, I noted that a single user usage of moclojer raises around 300 logs every 5 minutes or so. Most of these logs were debug info or had duplicated or unnecessary information. These include:

• request data, which contained way more user identifying info than necessary
• debug logs that were probably necessary in dev mode, but not in prod env

@Felipe-gsilva @avelino @matheusfrancisco I ask you guys to give a look at the logs in the opensearch dashboard. It's under moclojer-api-test-logs index.

What do you guys think? This should probably be taken seriously before launch. If it's already overwhelming for a single local user, imagine with hundreds, or even thousands of simultaneous users.

[Felipe-gsilva]

there are at least 4/5 fields we could just remove from every log

let me also ask, when will this parent uuid be filled? because it is set to nil in every log we got until now

[J0sueTM]

Wtf????? For some reason it went down to 9 hits lmao. Maybe it parsed them and returned only the unique ones? I'm still figuring it out

[J0sueTM]

@Felipe-gsilva The index manager says there are really 300 documents, my suggestion is that opensearch somehow really condensed duplicated documents, or showed only those where all data isn't null... I'm just shooting in the dark at this point.

[Felipe-gsilva]

well, if it auto fix our problem, we should see a way to configure this compression or whatever this is

[J0sueTM]

@Felipe-gsilva pelo que percebi, ele so guarda os traces, entao ele deve estar entendendo alguma coisa. O restante das mensagens estao sim sendo guardadas, mas quando fazemos query, so estamos pegando os logs de trace.

[J0sueTM]

@Felipe-gsilva Entendi o que esta acontecendo:

O OpenSearch tenta automaticamente agrupar logs que tem um elemento em comum. Por algum motivo ele chutou que o elemento em comum seria o parent/uid, e o tempo em comum seria o run-nsecs, o que deixava entao so os traces aparecerem, pois so eles tem os 2 valores nao nulos.

Alterei pra msg_ como o elemento em comum, e o inst como tempo em padrao. Agora voltou os 300 logs.

[J0sueTM]

Num sample de 300 logs, praticamente 1 terco sao logs de execucao de queries SQL. Como estamos utilizando de traces pra identificar as funcoes de DB, podemos desativar os logs por meio do next.jdbc

[Felipe-gsilva]

looks fine to have up to 180 as a normal flow of data, since the opensearch will already compress them 🔥 🚀