3.2.1

Bug Fixes

• Library Injection
  • Fix for release script causing lib injection OCI images to not get published.

3.2.1rc2

No notes.

3.2.1rc1

Bug Fixes

• Library Injection
  • Fix for release script causing lib injection OCI images to not get published.

3.2.0

Known Issues

This release is missing the OCI images due to issues with the release pipeline. If you need an OCI image with the features introduced in the 3.2 release line, please use v3.2.1 instead.

New Features

• ASM: This introduces auto instrumentation of signup events for Django.

Upgrade Notes

• ASM: To align across products, we're replacing DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED with DD_APM_TRACING_ENABLED. Feature set remains the same.
• Bumps libdatadog dependency to v16.0.3.

Bug Fixes

• celery: When multiple broker URLs are provided as a list, use the first broker URL from a list to avoid parsing errors.
• CI Visibility: This fix resolves an issue where ddtrace's own sys.monitoring coverage tool in Python 3.12+ would block other sys.monitoring tools such as pytest-cov from being used.
• tracing: Changes the name of the span event generated by a record_exception call as it was not following OTEL semantics.
• Fixes an issue where Profiling native threads would respect the musl libc default stack size, which could cause stack overflows in certain configurations.
• profiling: The memory profiler has a guard to avoid re-entering its code if there are allocations during sampling. This guard was meant to be thread-local, but was not correctly declared as such. This doesn't immediately cause problems because the profiler uses try-locks to protect access to its data structures, and re-entering the code will fail to acquire the locks. But this bug could be a source of deadlocks or data corruption if the code changes substantially in the future. This fix makes the guard thread-local as originally intended.

3.2.0rc1

Upgrade Notes

• Bumps libdatadog dependency to v16.0.3.

• ASM

  • To align across products, we're replacing DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED with DD_APM_TRACING_ENABLED. Feature set remains the same.

New Features

• ASM
  • Introduces auto instrumentation of signup events for Django.

Bug Fixes

• CI Visibility

  • Resolves an issue where ddtrace's own sys.monitoring coverage tool in Python 3.12+ would block other sys.monitoring tools such as pytest-cov from being used.

• Profiling

  • Fixes an issue where Profiling native threads would respect the musl libc default stack size, which could cause stack overflows in certain configurations.
  • The memory profiler has a guard to avoid re-entering its code if there are allocations during sampling. This guard was meant to be thread-local, but was not correctly declared as such. This doesn't immediately cause problems because the profiler uses try-locks to protect access to its data structures, and re-entering the code will fail to acquire the locks. But this bug could be a source of deadlocks or data corruption if the code changes substantially in the future. This fix makes the guard thread-local as originally intended.

• Tracing

  • celery: When multiple broker URLs are provided as a list, use the first broker URL from a list to avoid parsing errors.
  • Changes the name of the span event generated by a record_exception call as it was not following OTEL semantics.

2.21.3

Bug Fixes

• Tracing
  • configurations: Resolves an issue where DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctly

3.1.0

New Features

• Dynamic Instrumentation

  • Symbol Database is now enabled by default.

• LLM Observability

  • Properly labels Deepseek for the model provider and span name for LLM Observability spans for calls to Deepseek models using the OpenAI SDK.
  • Introduces an improved automated context management system for LLM Observability-specific spans. Also modifies LLMObs.export_span(), LLMObs.inject_distributed_headers(), LLMObs.annotate() to default to the current active LLM Observability-specific span if span is not provided.
  • Introduces automatic distributed tracing support for LLM Observability traces involving the concurrent.futures.thread module.
  • Introduces metadata argument to LLMObs.submit_evaluation_for, which enables users to attach arbitrary structured metadata to custom evaluations.

• Tracing

  • api gateway: Adds preview support to create inferred AWS API Gateway spans when certain AWS Gateway headers are passed into the request. Requires DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED=true to enable. For enablement instructions, contact support.
  • graphql: Add span events on GraphQL error spans, allowing multiple errors to be surfaced on the span.
  • graphql: Adds user-provided extensions to graphql error span events through the DD_TRACE_GRAPHQL_ERROR_EXTENSIONS environment variable. This is disabled by default; when set it allows users to capture their extensions.
  • Introduces a record_exception method that adds an exception to a Span as a span event. Refer to Span.record_exception for more details.

Bug Fixes

• ASM

  • Resolves an issue where blocking mechanism could partially fail with a 500 error on fastapi with python>=3.11 with a custom middleware.

• Code Security (IAST)

  • In some applications, unique cookie names are used per request, leading to a high number of unique cookie vulnerabilities. This fix improves detection accuracy by identifying the vulnerability at the set_cookies function and adding more context. Warning: This change updates the vulnerability hash, meaning that if it was previously silenced in the web dashboard, it may reappear as a new vulnerability.

• LLM Observability

  • botocore: Moves bedrock token usage metrics on APM spans from the meta field under bedrock.usage.{prompt/completion}_tokens to the metrics field under bedrock.response.usage.{prompt/completion}_tokens.
  • Resolves an issue where the Gemini and Vertex AI integrations were not properly submitting spans for streamed responses in the case that the generator is not fully consumed.
  • Resolves an issue with correlating traces between APM and LLM Observability where trace IDs were always being converted to hexadecimal values which broke APM links for 64-bit trace IDs.
  • Fixes non-LLM message inputs and outputs to be rendered as json rather than Python strings.

• Profiling

  • Fixes a crash in memory allocation profiler when tracking a large number of live allocations, caused by unsigned integer wraparound which lead to invalid memory access
  • Fixes an issue where the profiler unnecessarily outputs log messages when it was unable to record a lock acquire event.

• Single-Step Instrumentation

  • Removes boto3 from the minimum versions list to avoid blocking SSI on boto3 version.
  • Resolves an issue in which incorrect runtime version constraints prevented injection into processes using Python 3.13.1+

• Tracing

  • asyncio: Fixes an issue where the name of a coroutine was being overridden by a ddtrace function.
  • baggage: Fixes an issue where ot-baggage headers in wsgi apps were not properly recognized due to the http_ header prefix. This ensures consistent baggage propagation in wsgi compatible web frameworks.
  • configurations: This fix resolves an issue where DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctly
  • configurations: Resolves an issue where accessing ddtrace.config.<undefined_attribute> would create an IntegrationConfig object instead of raising an AttributeError.
  • internal: Require bytecode>=0.15.1 for Python 3.12 to avoid known bug.
  • internal: Fix ddtrace internal logger initialization mutating an unlocked shared resource.
  • library: Updates the exception raised when ddtrace.patch(...) is called with an invalid argument.
  • pymongo: Changes the mongodb.query span tag to json so it can be parsed by the agent.
  • runtime_metrics: Ensures only the global service name is tagged on runtime metrics, instead of every service name found in the process.
  • span: Fixes issue where spans weren't being handled correctly and were not being sent when using a custom Exception class that raises an exception in __str__.
  • Captures the execution of async functions traced via DD_TRACE_METHODS, not just coroutine creation. This change increases span durations, which may affect latency-based metrics.
  • Resolves ModuleNotFoundError errors raised when a OpenTracing tracer is initialized with a AsyncioScopeManager.

3.1.0rc1

New Features

• Dynamic Instrumentation

  • Symbol Database is now enabled by default.

• LLM Observability

  • Properly labels Deepseek for the model provider and span name for LLM Observability spans for calls to Deepseek models using the OpenAI SDK.
  • Introduces an improved automated context management system for LLM Observability-specific spans. Also modifies LLMObs.export_span(), LLMObs.inject_distributed_headers(), LLMObs.annotate() to default to the current active LLM Observability-specific span if span is not provided.
  • Introduces automatic distributed tracing support for LLM Observability traces involving the concurrent.futures.thread module.
  • Introduces metadata argument to LLMObs.submit_evaluation_for, which enables users to attach arbitrary structured metadata to custom evaluations.

• Tracing

  • api gateway: Adds preview support to create inferred AWS API Gateway spans when certain AWS Gateway headers are passed into the request. Requires DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED=true to enable. For enablement instructions, contact support.
  • graphql: Add span events on GraphQL error spans, allowing multiple errors to be surfaced on the span.
  • graphql: Adds user-provided extensions to graphql error span events through the DD_TRACE_GRAPHQL_ERROR_EXTENSIONS environment variable. This is disabled by default; when set it allows users to capture their extensions.
  • Introduces a record_exception method that adds an exception to a Span as a span event. Refer to Span.record_exception for more details.

Bug Fixes

• ASM

  • Resolves an issue where blocking mechanism could partially fail with a 500 error on fastapi with python>=3.11 with a custom middleware.

• Code Security (IAST)

  • In some applications, unique cookie names are used per request, leading to a high number of unique cookie vulnerabilities. This fix improves detection accuracy by identifying the vulnerability at the set_cookies function and adding more context. Warning: This change updates the vulnerability hash, meaning that if it was previously silenced in the web dashboard, it may reappear as a new vulnerability.

• LLM Observability

  • botocore: Moves bedrock token usage metrics on APM spans from the meta field under bedrock.usage.{prompt/completion}_tokens to the metrics field under bedrock.response.usage.{prompt/completion}_tokens.
  • Resolves an issue where the Gemini and Vertex AI integrations were not properly submitting spans for streamed responses in the case that the generator is not fully consumed.
  • Resolves an issue with correlating traces between APM and LLM Observability where trace IDs were always being converted to hexadecimal values which broke APM links for 64-bit trace IDs.
  • Fixes non-LLM message inputs and outputs to be rendered as json rather than Python strings.

• Profiling

  • Fixes a crash in memory allocation profiler when tracking a large number of live allocations, caused by unsigned integer wraparound which lead to invalid memory access
  • Fixes an issue where the profiler unnecessarily outputs log messages when it was unable to record a lock acquire event.

• Single-Step Instrumentation

  • Removes boto3 from the minimum versions list to avoid blocking SSI on boto3 version.
  • Resolves an issue in which incorrect runtime version constraints prevented injection into processes using Python 3.13.1+

• Tracing

  • asyncio: Fixes an issue where the name of a coroutine was being overridden by a ddtrace function.
  • baggage: Fixes an issue where ot-baggage headers in wsgi apps were not properly recognized due to the http_ header prefix. This ensures consistent baggage propagation in wsgi compatible web frameworks.
  • configurations: This fix resolves an issue where DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctly
  • configurations: Resolves an issue where accessing ddtrace.config.<undefined_attribute> would create an IntegrationConfig object instead of raising an AttributeError.
  • internal: Require bytecode>=0.15.1 for Python 3.12 to avoid known bug.
  • internal: Fix ddtrace internal logger initialization mutating an unlocked shared resource.
  • library: Updates the exception raised when ddtrace.patch(...) is called with an invalid argument.
  • pymongo: Changes the mongodb.query span tag to json so it can be parsed by the agent.
  • runtime_metrics: Ensures only the global service name is tagged on runtime metrics, instead of every service name found in the process.
  • span: Fixes issue where spans weren't being handled correctly and were not being sent when using a custom Exception class that raises an exception in __str__.
  • Captures the execution of async functions traced via DD_TRACE_METHODS, not just coroutine creation. This change increases span durations, which may affect latency-based metrics.
  • Resolves ModuleNotFoundError errors raised when a OpenTracing tracer is initialized with a AsyncioScopeManager.

2.21.2

Bug Fixes

Internal:

• Fix performance overhead of Python distribution parsing for internal telemetry.

ASM:

• This fix resolves an issue where blocking mechanism could partially fail with a 500 error on fastapi with python>=3.11 with a custom middleware.

Library Injection:

• Address partial library copy failures causing failed application startups in k8s.
• Removes boto3 from the minimum versions list to avoid blocking SSI on boto3 version.

2.20.2

Bug Fixes

• ASM

  • Ensures that no module from ASM are loaded when ASM is disabled or unavailable.
  • Resolves an issue where IAST modules could be loaded, even if disabled, which could create an ImportError exception on Windows.

• Code Security

  • SCA: Ensures that no module from IAST are loaded when IAST is disabled or unavailable.
  • Runtime Code Analysis (IAST): Avoids imports of IAST native module when IAST is not enabled.

• Tracing

  • internal: Fixes performance overhead of Python distribution parsing for internal telemetry.