Dependency Track API Timeouts

[dn281090pdi]

We are experiencing intermittent timeouts when interacting with the Dependency Track API. This occurs approximately 10-20 times per day, but not consistently.

Environment:

All components are hosted in AWS.
Dependency Track is deployed in AWS EKS with 4 CPU and 16GB RAM. No relevant errors are present in the Dependency Track logs.
The database is hosted on an EC2 t4g.medium instance. No relevant errors are present in the database logs.
Dependency Track version: v4.12.1
Database type: PostgreSQL
Observations:

We have Grafana dashboards set up for monitoring Dependency Track. While there are no critical issues evident, we do see some intermittent gaps in the following metrics:

Database Operations
Database Transactions
Average Connection Usage Duration

Problem Reproduction:

We have observed timeouts specifically on PUT requests to the /api/v1/bom endpoint. The requests time out after 300+ seconds.

Troubleshooting Steps Taken:

Verified Dependency Track and database logs for errors.
Checked network connectivity between the EKS cluster and the database instance.
Monitored resource utilization (CPU, memory, I/O) on both the EKS nodes and the database instance.
Adjusted the following Dependency Track configuration parameters (increasing and decreasing values):
ALPINE_DATABASE_POOL_MIN_IDLE
ALPINE_DATABASE_POOL_MAX_SIZE
Increased resources for the Dependency Track pod + 2 CPU and +8GB RAM.
Unfortunately, none of these steps have resolved the timeout issues.

We would appreciate any guidance on localizing and resolving these timeout issues. Could you please provide insights on potential causes and further troubleshooting steps?

Thank you for you time.

[nscuro]

If you're using the sample Grafana dashboard we distribute, be aware that some of the graphs will not show a value if the value is 0 (e.g., no operations happened). The corresponding PromQL queries have a > 0 at the end. As per the screenshot it doesn't look like your instance is particularly busy.

What operations / requests specifically are timing out for you?

Edit: Sorry, you wrote it already. It's PUT /api/v1/bom.

For that endpoint, can you try migrating to the POST endpoint, which uses multipart/form-data instead? You can find examples here: https://docs.dependencytrack.org/usage/cicd/

You could also try to temporarily disable BOM validation in the settings, to see if that helps. BOM validation is the most expensive action performed by the BOM upload endpoints.

[nscuro]

It would also be interesting after what time your requests experience timeouts. IIRC, DT itself will time out requests after 30sec. If yours happen any earlier than that, you might want to check any reverse proxy or load balancer configs you have.

[dn281090pdi]

We are experiencing timeouts specifically on PUT requests to the /api/v1/bom endpoint. Here's an example of the request body:
{"projectName":"TEST1 ","projectVersion":"master","bom":"${data}"}
While we have observed timeouts on this endpoint, it is possible that other API services are also affected.

[dn281090pdi]

OK, thank you for the suggestions. We will try the following:

Use multipart/form-data for uploading the BOM.
Disable BOM validation in Dependency Track.
Review and verify our load balancer configurations.

We will update you on the results of these troubleshooting steps.