[Python] Enhancements to Queue instrumentation

[antonpirker]

We want to improve the data we collect for task queues.
We will start with enhancing our Celery integration. This should work with redis, and rabbitmq, and as a stretch goal amazon sqs as brokers. (Kafka should follow later.)

How spans are right now:

• Producer creates <span op="queue.submit.celery"> when task is put into queue. (start_timestamp is when putting into queue is started and end_timestamp is when the item is in the queue.)
• Consumer creates <transaction op="queue.task.celery"> when task is fetched and processed. (start_timestamp is when the item was received from the queue, end_timestamp is when the processing of the task has finished. )
  • Transaction has a status:
    • ok in case of success
    • aborted if Celery raises Retry, Ignore or Reject celery exceptions
    • internal_error if an unhandled exception was raised during execution of the task

New data we want to add to queue spans:

(for naming see: https://opentelemetry.io/docs/specs/semconv/attributes-registry/messaging/)

• retry count (can be found in args of existing _wrap_tracer, the attached PR)
• byte size of message (messaging.message.body.size)
• byte size of message plus metadata (messaging.message.envelope.size)
• success/failure status - already there
• host (server.address) - (can be found in args of existing _wrap_tracer, the attached PR)
• queue or topic (messaging.destination.name) - not sure where to get this.
• message id (messaging.message.id) - should also be in args of the comments in the attached PR)
• consumer id (unique id of the consumer) (messaging.client_id) - dont know yet where to get.
• what broker is used in celery. (messaging.system)

See this PR for some points where to get the data: #2962

I also added a sample project that uses RabbitMQ with Celery: https://github.com/antonpirker/testing-sentry/tree/main/test-celery-queues-module

Mock of how it will look in the product:

Tasks

Beta Give feedback

1. Queue instrumentation enhancements: Celery integration #2959
   8 of 8
   Enhancement Feature: Performance Integration: Celery +3
   

[antonpirker]

This issue can be closed because the Celery integration was updated to emit spans that are eligible for the new Insights Queues module. (see mentioned PRs above)