3.0.0rc1

Upgrade Notes

• CI Visibility

  • Official release of the new version of the pytest plugin, introducing the following features:

    • Auto Test Retries
    • Early Flake Detection
    • Improved coverage collection for Test Impact Analysis (formerly Intelligent Test Runner), using an internal collection method instead of coverage.py, with improved dependency discovery.

    NOTE: this new version of the plugin introduces breaking changes:

    • module, suite, and test names are now parsed from the item.nodeid attribute
    • test names now include the class for class-based tests
    • Test skipping by Test Impact Analysis (formerly Intelligent Test Runner) is now done at the suite level, instead of at the test level

    A beta version of the plugin had been available since v2.18.0, and could be enabled via the DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable. The new version is now the default, and the environment variable is not used anymore.

• LLM Observability

  • Support for the deprecated environment variable DD_LLMOBS_APP_NAME has been removed.
  • Removes the deprecated parameters argument from LLMObs annotate(). Use metadata instead.
  • openai: Drops tracing and LLM Observability support for OpenAI 0.x versions. To maintain support, upgrade to openai>=1.0.
  • langchain: removed tracing support for langchain.chains.base.Chains. To maintain support, migrate to using Langchain's LCEL instead.
  • langchain: Drops tracing and LLM Observability support for Langchain v0.0.x versions.

• Profiling

  • Bumps libdatadog dependency to v15.0.0.

• Tracing

  • Removes deprecated interfaces from all integrations.
  • propagation: Setting DD_TRACE_PROPAGATION_STYLE to b3 single header is no longer supported. Use b3 instead.
  • aws_lambda: Drops support for aws_lambda<=6.105.0.

Deprecation Notes

• ASM

  • Deprecated env var DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING is now removed. Please use the new env var DD_APPSEC_AUTO_USER_INSTRUMENTATION_MODE instead.

• LLM Observability

  • The \_DD_LLMOBS_EVALUATORS environment variable is deprecated and will be removed in ddtrace 3.0.0.
    As an alternative to \_DD_LLMOBS_EVALUATORS, you can use DD_LLMOBS_EVALUATORS instead. To migrate, replace \_DD_LLMOBS_EVALUATORS with DD_LLMOBS_EVALUATORS.
  • The \_DD_LLMOBS_EVALUATOR_SAMPLING_RULES environment variable is deprecated and will be removed in ddtrace 3.0.0.
    As an alternative to \_DD_LLMOBS_EVALUATOR_SAMPLING_RULES, you can use DD_LLMOBS_EVALUATOR_SAMPLING_RULES instead. To migrate, replace \_DD_LLMOBS_EVALUATOR_SAMPLING_RULES with DD_LLMOBS_EVALUATOR_SAMPLING_RULES.

• Tracing

  • Moves ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.
  • Deprecates the following constants in ddtrace.constants module:
    • ANALYTICS_SAMPLE_RATE_KEY
    • SAMPLE_RATE_METRIC_KEY
    • SAMPLING_PRIORITY_KEY
    • SAMPLING_AGENT_DECISION
    • SAMPLING_RULE_DECISION
    • SAMPLING_LIMIT_DECISION
    • ORIGIN_KEY
    • USER_ID_KEY
    • HOSTNAME_KEY
    • RUNTIME_FAMILY
    • BASE_SERVICE_KEY
    • SPAN_MEASURED_KEY
    • KEEP_SPANS_RATE_KEY
    • MULTIPLE_IP_HEADERS
    • CONFIG_ENDPOINT_ENV
    • CONFIG_ENDPOINT_RETRIES_ENV
    • CONFIG_ENDPOINT_TIMEOUT_ENV
  • Internalizes the ddtrace.settings.config module and deprecates the following ddtrace.config attributes:
    • http, use DD_TRACE_HEADER_TAGS environment variable instead.
    • http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.
    • trace_headers, this attribute is internal to the tracer.
    • header_is_traced, this attribute is internal to the tracer.
    • convert_rc_trace_sampling_rules, this attribute is internal to the tracer.
    • enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.
    • get_from, use ddtrace.trace.Pin to set instance level configurations.
  • Removes deprecated attributes from ddtrace.config.

New Features

• Removes code and tests related to Python 3.7, breaking the library's compatibility with this Python version.

• ASM

  • Introduces use of meta struct for appsec event reports. This will improve performances. ASM will now require at least agent version 7.35.0 to work as expected.

• Code Security (IAST)

  • Introduces Code Security support to PyMySQL and MySQLDB integrations.
  • XSS detection for Django applications, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.
  • Implements the detection of the Stacktrace-Leak vulnerability for Django, Flask and FastAPI.

• LLM Observability

  • openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.
  • Introduces an integration with the RAGAS evaluation framework to continuously monitor
    the performance of context-augmented LLM generations in production.
    The integration supports evaluating LLM inferences with the following RAGAS metrics:
    • Faithfulness: measures if the LLM response is faithful to the provided context.
    • Answer Relevancy: measures how relevant the LLM response is to the user input.
    • Context Precision: measures how effectively the context is used in the generated response.
      To learn more, see the LLM Observability evaluations guide.

Bug Fixes

• ASM

  • Resolves an issue where IAST modules could be loaded, even if disabled, which could create an ImportError exception on Windows.

• CI Visibility

  • Fixes an issue where Auto Test Retries with pytest would always consider retries of tests defined inside unittest classes to be successful.

• Code security (IAST)

  • Resolves an issue where the usage of callonce decorator could trigger an import loop

• LLM Observability

  • Resolves an issue where successive calls to LLMObs.annotate() on the same span caused overwriting of metrics, metadata, and prompt dictionaries. Now, calling LLMObs.annotate() will merge new metrics/metadata/prompt values with any existing ones.
  • Resolves an issue where spans were being enqueued to an inactive evaluator runner which caused noisy logs related to the evaluator runner buffer being full.
  • Resolves an issue where explicitly only using LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.
  • Resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors.
  • Resolves an issue where extracting token metadata from openai streamed chat completion token chunks caused an IndexError.
  • openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.
  • vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.

• Profiling

  • Fixes an issue on arm64 macOS where the profiler was not able to find native extension modules to enable the following features: DD_PROFILING_TIMELINE_ENABLED and DD_PROFILING_STACK_V2_ENABLED.
  • Fixes SystemError from the memory profiler returning NULL when collecting events

Other Changes

• Tracing
  • Ensures the ddtrace library does not use deprecated APIs internally. Deprecation warnings should only be logged when the user's code is using deprecated APIs.
  • cassandra , cherrypy, flask_cache, starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.
  • Removes the deprecated tracing modules and constants from the ddtrace package.

2.20.1

Deprecation Notes

• Tracing
  • Moves ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.
  • Deprecates the following constants in ddtrace.constants module:
    • ANALYTICS_SAMPLE_RATE_KEY
    • SAMPLE_RATE_METRIC_KEY
    • SAMPLING_PRIORITY_KEY
    • SAMPLING_AGENT_DECISION
    • SAMPLING_RULE_DECISION
    • SAMPLING_LIMIT_DECISION
    • ORIGIN_KEY
    • USER_ID_KEY
    • HOSTNAME_KEY
    • RUNTIME_FAMILY
    • BASE_SERVICE_KEY
    • SPAN_MEASURED_KEY
    • KEEP_SPANS_RATE_KEY
    • MULTIPLE_IP_HEADERS
    • CONFIG_ENDPOINT_ENV
    • CONFIG_ENDPOINT_RETRIES_ENV
    • CONFIG_ENDPOINT_TIMEOUT_ENV
  • Internalizes the ddtrace.settings.config module and deprecates the following ddtrace.config attributes:
    • http, use DD_TRACE_HEADER_TAGS environment variable instead.
    • http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.
    • trace_headers, this attribute is internal to the tracer.
    • header_is_traced, this attribute is internal to the tracer.
    • convert_rc_trace_sampling_rules, this attribute is internal to the tracer.
    • enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.
    • get_from, use ddtrace.trace.Pin to set instance level configurations.

New Features

• Tracing
  • openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.

Bug Fixes

• CI Visibility
  • Fixes an issue where Auto Test Retries with pytest would always consider retries of tests defined inside unittest classes to be successful.
• LLM Observability
  • Resolves an issue where explicitly only using LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.
  • This fix resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors.
  • This fix resolves an issue where extracting token metadata from openai streamed chat completion token chunks caused an IndexError.
• Tracing
  • openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.
  • vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.
• Profiling
  • Fix SystemError from the memory profiler returning NULL when collecting events

Other Changes

• Tracing
  • Ensures the ddtrace library does not use deprecated APIs internally. Deprecation warnings should only be logged when the user's code is using deprecated APIs.
  • cassandra, cherrypy ,flask_cache, starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.

2.19.3

Bug Fixes

• CI Visibility: fixes an issue where Auto Test Retries with pytest would always consider retries of tests defined inside unittest classes to be successful.
• profiling: fix SystemError from the memory profiler returning NULL when collecting events

2.19.2

Bug Fixes

• Tracing
  • celery: Fixes an issue where celery.apply spans from Celery prerun got closed too soon leading to span tags being missing.
  • openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.
  • openai: Ensures the OpenAI integration is compatible with Python versions 3.12 and 3.13.
  • vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.
• LLM Observability
  • This fix resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors.
• Lib Injection
  • Fixes incorrect telemetry data payload format.

2.20.0

Upgrade Notes

• Tracing
  • Validates Python 3.13 support for the ddtrace-run entrypoint.
  • Validates Python 3.13 support for the following integrations:
    • aiomysql
    • aiopg
    • asyncpg
    • avro
    • botocore
    • confluent-kafka
    • django
    • falcon
    • fastapi
    • grpcio
    • mysqldb
    • protobuf
    • pyodbc
    • sqlalchemy

Deprecation Notes

• CI Visibility

  • Moves the implementational details of the pytest, pytest_benchmark, pytest_bdd, and unittest integrations from ddtrace.contrib.<integration> to ddtrace.contrib.internal.<integration>.

• Tracing

  • Deprecates ddtrace.filters.FilterRequestsOnUrl. Spans should be filtered/sampled using DD_TRACE_SAMPLING_RULES configuration.
  • Deprecates the use of multiple tracer instances in the same process. The global tracer (ddtrace.tracer) `should be used instead.
  • Deprecates support for configuring samplers via a programmatic API. In v3.0.0 samplers will only be configurable via environment variables or remote configuration.
  • Ensures most tracing configurations are only set on application start up. This is done by deprecating the following parameters in ddtrace.configure(...) function. These parameters will be removed in ddtrace>=3.0.0: - enabled - hostname - port - uds_path - https - sampler - settings - priority_sampling - settings - dogstatsd_url - writer - partial_flush_enabled - partial_flush_min_spans - api_version - compute_stats_enabled - wrap_executor
  • Deprecates ddtrace.pin module and moves the Pin class to ddtrace.trace package. In v3.0.0 the ddtrace/pin.py will be removed.
  • Deprecates ddtrace.filters module and moves the TraceFilter and FilterRequestsOnUrl classes to ddtrace.trace package. In v3.0.0 the ddtrace/filters.py will be removed.
  • Deprecates all attributes in ddtrace.contrib.trace_utils_async and ddtrace.contrib.redis_utils. Replaces ddtrace.contrib.trace_utils_async.with_traced_module(...) with ddtrace.contrib.trace_utils.with_traced_module_async(...). Moves public attributes defined in ddtrace.contrib.redis_utils.* to ddtrace.contrib.trace_utils.
  • Deprecates the ability to use multiple tracer instances with ddtrace.Pin. In v3.0.0 pin objects will only use the global tracer.
  • Ensures the implementation details of ddtrace integrations are internal to ddtrace library. In ddtrace>=3.0.0 integrations should only be enabled and configured via ddtrace.patch(..), import ddtrace.auto or the ddtrace-run command. Unpatching integrations or getting the version of an integration is no longer supported.
  • rq: Ensures the implementation details of the rq integration are internal to ddtrace library. In ddtrace>=3.0.0 this integration should only be enabled and configured via ddtrace.patch(..), import ddtrace.auto or the ddtrace-run command

New Features

• ASM

  • Introduces full support for Automated user lifecycle tracking for login events (success and failure)
  • Introduces the support for command injection for Exploit Prevention. With previous support of shell injection with os.system, this provides automatic instrumentation for subprocess module functions and os.spawn* functions, ensuring monitoring and blocking for Exploit Prevention on those endpoints.

• Code Security

  • Adds support for Header Injection vulnerability sink point.
  • Code Injection vulnerability detection, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.

• LLM Observability

  • openai: Introduces automatic extraction of token usage from streamed chat completions. Unless stream_options: {"include_usage": False} is explicitly set on your streamed chat completion request, the OpenAI integration will add stream_options: {"include_usage": True} to your request and automatically extract the token usage chunk from the streamed response.

• Profiling

  • Stack V2 is enabled by default. It is the new stack sampler implementation for CPython 3.8+. It enhances the performance, accuracy, and reliability of Python CPU profiling. This feature activates our new stack sampling, collection and export system.
    The following are known issues and missing features from Stack V2
    • Services using gunicorn with Stack V2 results in performance degradation
    • Support for gevent is lacking
  • Exception sampling is missing If you find these as a blocker for enabling Stack V2 for your services, you can turn it off via setting DD_PROFILING_STACK_V2_ENABLED=0. If you find any other issue, then please proceed to escalate using appropriate support channels or file an issue on the repository.

• Tracing

  • Introduces the environment variable DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT to control the behavior of the extraction of distributed tracing headers. The values, continue (default), ignore, and restart, are supported. The default value is continue which has no change from the current behavior of always propagating valid headers. ignore ignores all incoming headers, never propagating the incoming trace information and restart turns the first extracted propagation style into a span link and propagates baggage if extracted.

Bug Fixes

• Code Security

  • Adds more modules to the IAST patching denylist to improve startup time

• Exception Replay

  • Includes missing nonlocal variables in snapshot log messages.

• Lib Injection/SSI

  • Removes python-json-logger from library compatibility check.
  • Fixes incorrect telemetry data payload format.

• LLM Observability

  • Resolves an issue where enabling LLM Observability in agentless mode would result in traces also being sent to the agent proxy endpoint.
  • Resolves an issue where configuring custom trace filters/processors onto the tracer would disable LLM Observability.
    Note that if LLM Observability is enabled in agentless mode, writing APM traces must be explicitly disabled by setting DD_TRACE_ENABLED=0.
  • botocore: Resolves formatting errors in the bedrock integration when parsing request model IDs, which can now accept AWS ARNs.

• Profiling

  • Fixes an issue where the memory allocation profiler can cause a segmentation fault due to data races when accessing its own global data structures from multiple threads.
  • Fixes a bug where profiling mutexes were not cleared on fork in the child process. This could cause deadlocks in certain configurations.
  • Resolves a data race issue accessing lock's acquired time, leading to an AttributeError: _Profiled_ThreadingLock object has no attribute self_acquired_at
  • Resolves an issue where lock release would have been captured with a wrong acquire timestamp
  • Removes a system call from the memory allocation profiler, used to detect forks, which ran on every allocation and resulted in a significant slowdown.

• Tracing

  • Resolves an issue where baggage header extraction was case sensitive and didn't accept the header prepended with HTTP. Now the baggage header will be extracted regardless of casing and the HTTP format.
  • Resolves an issue where the core instrumentation could raise an uncaught exception.
  • Improves error handling for exceptions raised during the startup of ddtrace integrations. This reduces the likelihood of the ddtrace library raising unhandled exceptions.
  • Fixes an issue where the baggage header was not being propagated when the baggage header was the only header extracted. With this fix, the baggage header is now propagated when it is the only header extracted.
  • asgi: Resolves an issue parsing response cookies in FastAPI and awsgi
  • asyncio: Resolves an issue where asyncio event loops fail to register when ddtrace-run/import ddtrace.auto is used and gevent is installed.
  • celery: Fixes an issue where celery.apply spans from Celery prerun got closed too soon leading to span tags being missing.
  • django: Fixes issue where django cache is represented as a django service rather than the third party service.

Other Changes

• Lib Injection/SSI
  • Reduces size of OCI image size to improve k8s lib-injection pull and startup times.

2.20.0rc1

Upgrade Notes

• Tracing
  • Validates Python 3.13 support for the ddtrace-run entrypoint.
  • Validates Python 3.13 support for the following integrations:
    • aiomysql
    • aiopg
    • asyncpg
    • avro
    • botocore
    • confluent-kafka
    • django
    • falcon
    • fastapi
    • grpcio
    • mysqldb
    • protobuf
    • pyodbc
    • sqlalchemy

Deprecation Notes

• CI Visibility

  • Moves the implementational details of the pytest, pytest_benchmark, pytest_bdd, and unittest integrations from ddtrace.contrib.<integration> to ddtrace.contrib.internal.<integration>.

• Tracing

  • Deprecates ddtrace.filters.FilterRequestsOnUrl. Spans should be filtered/sampled using DD_TRACE_SAMPLING_RULES configuration.
  • Deprecates the use of multiple tracer instances in the same process. The global tracer (ddtrace.tracer) `should be used instead.
  • Deprecates support for configuring samplers via a programmatic API. In v3.0.0 samplers will only be configurable via environment variables or remote configuration.
  • Ensures most tracing configurations are only set on application start up. This is done by deprecating the following parameters in ddtrace.configure(...) function. These parameters will be removed in ddtrace>=3.0.0: - enabled - hostname - port - uds_path - https - sampler - settings - priority_sampling - settings - dogstatsd_url - writer - partial_flush_enabled - partial_flush_min_spans - api_version - compute_stats_enabled - wrap_executor
  • Deprecates ddtrace.pin module and moves the Pin class to ddtrace.trace package. In v3.0.0 the ddtrace/pin.py will be removed.
  • Deprecates ddtrace.filters module and moves the TraceFilter and FilterRequestsOnUrl classes to ddtrace.trace package. In v3.0.0 the ddtrace/filters.py will be removed.
  • Deprecates all attributes in ddtrace.contrib.trace_utils_async and ddtrace.contrib.redis_utils. Replaces ddtrace.contrib.trace_utils_async.with_traced_module(...) with ddtrace.contrib.trace_utils.with_traced_module_async(...). Moves public attributes defined in ddtrace.contrib.redis_utils.* to ddtrace.contrib.trace_utils.
  • Deprecates the ability to use multiple tracer instances with ddtrace.Pin. In v3.0.0 pin objects will only use the global tracer.
  • Ensures the implementation details of ddtrace integrations are internal to ddtrace library. In ddtrace>=3.0.0 integrations should only be enabled and configured via ddtrace.patch(..), import ddtrace.auto or the ddtrace-run command. Unpatching integrations or getting the version of an integration is no longer supported.
  • rq: Ensures the implementation details of the rq integration are internal to ddtrace library. In ddtrace>=3.0.0 this integration should only be enabled and configured via ddtrace.patch(..), import ddtrace.auto or the ddtrace-run command

New Features

• ASM

  • Introduces full support for Automated user lifecycle tracking for login events (success and failure)
  • Introduces the support for command injection for Exploit Prevention. With previous support of shell injection with os.system, this provides automatic instrumentation for subprocess module functions and os.spawn* functions, ensuring monitoring and blocking for Exploit Prevention on those endpoints.

• Code Security

  • Adds support for Header Injection vulnerability sink point.
  • Code Injection vulnerability detection, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.

• LLM Observability

  • openai: Introduces automatic extraction of token usage from streamed chat completions. Unless stream_options: {"include_usage": False} is explicitly set on your streamed chat completion request, the OpenAI integration will add stream_options: {"include_usage": True} to your request and automatically extract the token usage chunk from the streamed response.

• Profiling

  • Stack V2 is enabled by default. It is the new stack sampler implementation for CPython 3.8+. It enhances the performance, accuracy, and reliability of Python CPU profiling. This feature activates our new stack sampling, collection and export system.
    The following are known issues and missing features from Stack V2
    • Services using gunicorn with Stack V2 results in performance degradation
    • Support for gevent is lacking
  • Exception sampling is missing If you find these as a blocker for enabling Stack V2 for your services, you can turn it off via setting DD_PROFILING_STACK_V2_ENABLED=0. If you find any other issue, then please proceed to escalate using appropriate support channels or file an issue on the repository.

• Tracing

  • Introduces the environment variable DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT to control the behavior of the extraction of distributed tracing headers. The values, continue (default), ignore, and restart, are supported. The default value is continue which has no change from the current behavior of always propagating valid headers. ignore ignores all incoming headers, never propagating the incoming trace information and restart turns the first extracted propagation style into a span link and propagates baggage if extracted.

Bug Fixes

• Code Security

  • Adds more modules to the IAST patching denylist to improve startup time

• Exception Replay

  • Includes missing nonlocal variables in snapshot log messages.

• Lib Injection/SSI

  • Removes python-json-logger from library compatibility check.
  • Fixes incorrect telemetry data payload format.

• LLM Observability

  • Resolves an issue where enabling LLM Observability in agentless mode would result in traces also being sent to the agent proxy endpoint.
  • Resolves an issue where configuring custom trace filters/processors onto the tracer would disable LLM Observability.
    Note that if LLM Observability is enabled in agentless mode, writing APM traces must be explicitly disabled by setting DD_TRACE_ENABLED=0.
  • botocore: Resolves formatting errors in the bedrock integration when parsing request model IDs, which can now accept AWS ARNs.

• Profiling

  • Fixes an issue where the memory allocation profiler can cause a segmentation fault due to data races when accessing its own global data structures from multiple threads.
  • Fixes a bug where profiling mutexes were not cleared on fork in the child process. This could cause deadlocks in certain configurations.
  • Resolves a data race issue accessing lock's acquired time, leading to an AttributeError: _Profiled_ThreadingLock object has no attribute self_acquired_at
  • Resolves an issue where lock release would have been captured with a wrong acquire timestamp
  • Removes a system call from the memory allocation profiler, used to detect forks, which ran on every allocation and resulted in a significant slowdown.

• Tracing

  • Resolves an issue where baggage header extraction was case sensitive and didn't accept the header prepended with HTTP. Now the baggage header will be extracted regardless of casing and the HTTP format.
  • Resolves an issue where the core instrumentation could raise an uncaught exception.
  • Improves error handling for exceptions raised during the startup of ddtrace integrations. This reduces the likelihood of the ddtrace library raising unhandled exceptions.
  • Fixes an issue where the baggage header was not being propagated when the baggage header was the only header extracted. With this fix, the baggage header is now propagated when it is the only header extracted.
  • asgi: Resolves an issue parsing response cookies in FastAPI and awsgi
  • asyncio: Resolves an issue where asyncio event loops fail to register when ddtrace-run/import ddtrace.auto is used and gevent is installed.
  • celery: Fixes an issue where celery.apply spans from Celery prerun got closed too soon leading to span tags being missing.
  • django: Fixes issue where django cache is represented as a django service rather than the third party service.

Other Changes

• Lib Injection/SSI
  • Reduces size of OCI image size to improve k8s lib-injection pull and startup times.

2.19.1

Bug Fixes

• Profiling

  • Fixes an issue where the memory allocation profiler can cause a segmentation fault due to data races when accessing its own global data structures from multiple threads.
  • Fixes a bug where profiling mutexes were not cleared on fork in the child process. This could cause deadlocks in certain configurations.
  • Removes a system call from the memory allocation profiler, used to detect forks, which ran on every allocation and resulted in a significant slowdown.

• Tracing

  • django: Fixes issue where django cache is represented as a django service rather than the third party service.
  • botocore: Resolves formatting errors in the bedrock integration when parsing request model IDs, which can now accept AWS ARNs.

2.17.5

Bug Fixes

• Tracing
  • celery: Fixes an issue where celery.apply spans from Celery pre-run got closed too soon leading to span tags being missing.

2.19.0

New Features

• ASM

  • Introduces "Standalone SCA billing", opting out for APM billing and applying to only SCA. Enable this by setting these two environment variables: DD_APPSEC_SCA_ENABLED and DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED

• Code Security

  • Introduces stack trace reports for Code Security.

• Profiling

  • Adds an experimental integration with the PyTorch profiler which can be enabled by setting DD_PROFILING_PYTORCH_ENABLED=true. This feature instruments the PyTorch profiler API (https://pytorch.org/docs/stable/_modules/torch/profiler/profiler.html) so that GPU profiling data can be sent to Datadog for visualization. This feature supports torch version >= 1.8.1.

• Tracing

  • azure_functions: Introduces support for Azure Functions.

Upgrade Notes

• Makes the library compatible with Python 3.13

Bug Fixes

• ASM

  • Resolves an issue where AppSec was using a patched request and builtins functions, creating telemetry errors.

• Code Security

  • Adds more modules to the IAST patching denylist to improve startup time

• Lib-Injection

  • Fixes missing lib-injection telemetry for common abort scenarios.

• LLM Observability

  • Resolves an issue where LLMObs.enable() ignored global patch configurations, specifically
    the DD_TRACE_<INTEGRATION>_ENABLED and DD_PATCH_MODULES environment variables.

• Telemetry

  • library: Resolves deadlocks that could occur when sending instrumentation telemetry data after an unhandled exception is raised.

• Tracing

  • ASGI: This fix resolves an issue parsing response cookies in FastAPI and awsgi
  • asyncio: Resolves an issue where asyncio event loops fail to register when ddtrace-run/import ddtrace.auto is used and gevent is installed.
  • datastreams: Logs at warning level for Kinesis errors that break the Data Streams Monitoring map.

2.19.0rc2

Bug Fixes

• Code Security

  • Adds more modules to the IAST patching denylist to improve startup time

• Tracing

  • ASGI: Resolves an issue parsing response cookies in FastAPI and awsgi
  • asyncio: Resolves an issue where asyncio event loops fail to register when ddtrace-run/import ddtrace.auto is used and gevent is installed.