itemDataProcessing Out of memory

[ddennerline3]

Is your feature request related to a problem? Please describe.
When trying to import a large file, an out of memory condition is triggered. When reading through the code to find out which function may have caused the issue, there are no log message in https://github.com/ludeknovy/jtl-reporter-be/blob/master/src/server/controllers/item/shared/item-data-processing.ts#L40 to signal which part of the report import did complete. I have tried to increase memory to 24GB, but this doesn’t seem to be working. It’s not clear if the memory issue is the TS database or application server.

Describe the solution you'd like
Add log messages to itemDataProcessing() to help identify which parts of the critical pipeline didn't succeed due to exception.

Additional context
2024-12-10T13:13:21.463352447Z stdout F {"level":"error","message":"Processing of item 4d8da859-1c59-41eb-94b0-8d1fbeb256d3 failed DataProcessingException: Error while processing dataId: 4d8da859-1c59-41eb-94b0-8d1fbeb256d3 for item: 4d8da859-1c59-41eb-94b0-8d1fbeb256d3, error: error: Out of memory","timestamp":"2024-12-10 13:13:21"}

2024-12-10T13:13:21.463602841Z stdout F {"level":"error","message":"Error occurred during samples aggregation in database: Error while processing dataId: 4d8da859-1c59-41eb-94b0-8d1fbeb256d3 for item: 4d8da859-1c59-41eb-94b0-8d1fbeb256d3, error: error: Out of memory, item_id: 4d8da859-1c59-41eb-94b0-8d1fbeb256d3","timestamp":"2024-12-10 13:13:21"}

[ludeknovy]

Hi @ddennerline3 this definitely looks like postgres/timescaledb error. Please check the database logs.

[ludeknovy]

And my bet is this query is causing the problem: https://github.com/ludeknovy/jtl-reporter-be/blob/e91426510817dfa8bf8dcfa99353aeac43bd6d19/src/server/controllers/item/shared/item-data-processing.ts#L56

[ludeknovy]

ludeknovy/jtl-reporter-be#353

[ddennerline3]

@ludeknovy Would it be possible to change the winston logger creation to support environment variable log level configuration please?

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info', <<<<
  format: winston.format.json(),
  transports: [new winston.transports.Console()],
});

Our JTL reporter is cloud native deployment (Kubernetes) and changing sys logging levels is not something that is normally done.

[ludeknovy]

@ddennerline3 yes, but I guess the default would be debug.

[ludeknovy]

ludeknovy/jtl-reporter-be#354