Release Notes

v0.51.0

Bug Fixes

  • dogpile.cache: handle both kwargs and args in the wrapper functions (using

    only kwargs would result in an IndexError.

0.51.0rc4

Bug Fixes

  • opentracer: don’t override default tracing config for the ENABLED,

    AGENT_HOSTNAME, AGENT_HTTPS or AGENT_PORT settings.

v0.51.0rc3

Deprecation Notes

  • Removed the collect_metrics argument from Tracer.configure. See the release notes for v0.49.0 for the migration instructions.

v0.51.0rc2

New Features

  • Added support for aipg~=0.16.0.

  • Add MariaDB integration.

  • The profiler now exports active tasks for CPU and wall time profiles.

Upgrade Notes

  • botocore: Update trace propagation format for directly invoked Lambda functions. This breaks compatibility with Lambda functions instrumented with datadog-lambda-python < v41 or datadog-lambda-js < v3.57.0. Please upgrade datadog-lambda-* in invoked lambda functions, or engage legacy compatibility mode in one of two ways: - ddtrace.config.botocore.invoke_with_legacy_context = True - DD_BOTOCORE_INVOKE_WITH_LEGACY_CONTEXT=true

Bug Fixes

  • Fixed JSON encoding errors in the pytest plugin for parameterized tests with dictionary parameters with tuple keys. The pytest plugin now always JSON encodes the string representations of test parameters.

  • Fixed JSON encoding errors in the pytest plugin for parameterized tests with complex Python object parameters. The pytest plugin now defaults to encoding the string representations of non-JSON serializable test parameters.

  • Fix a possible NoneType error in the WSGI middleware start_response method.

v0.51.0rc1

New Features

  • Added support for jinja2~=3.0.0.

  • The pytest integration now uses the name of the repository being tested as the default test service name.

Upgrade Notes

  • The legacy Django configuration method (deprecated in 0.34) has been removed.

Deprecation Notes

  • monkey.patch_module is deprecated.

  • monkey.get_patch_module is deprecated.

Bug Fixes

  • Fixes an issue with enabling the runtime worker introduced in v0.49.0 where no runtime metrics were sent to the agent.

  • Fix pymongo 3.12.0+ spans not being generated.

v0.50.0rc5

New Features

  • The Python heap profiler can now be enabled by setting the DD_PROFILING_HEAP_ENABLED environment variable to 1.

Other Notes

  • The botocore integration excludes AWS endpoint call parameters that have a name ending with Body from the set of span tags.

v0.50.0rc4

New Features

  • The pytest plugin now includes support for automatically tagging spans with parameters in parameterized tests.

v0.50.0rc3

New Features

  • Add new environment variables to configure the internal trace writer.

    DD_TRACE_WRITER_MAX_BUFFER_SIZE, DD_TRACE_WRITER_INTERVAL_SECONDS, DD_TRACE_WRITER_MAX_PAYLOAD_SIZE_BYTES

  • The exception profiler now gathers and exports the traces and spans information.

Deprecation Notes

  • The deprecated dbapi2 configuration has been removed. The integration-specific configuration should be used instead. Look at the v0.48.0 release notes for migration instructions.

Bug Fixes

  • Fixed the handling of the Django template name tag causing type errors.

  • Fixes an issue when trying to manually start the runtime metrics worker:

    AttributeError: module 'ddtrace.internal.runtime' has no attribute 'runtime_metrics'
    
  • sanic: update instrumentation to support version 21.

  • Performance of the Celery integration has been improved.

  • Fix runtime-id and system.pid tags not being set on distributed traces.

v0.50.0rc2

New Features

  • Add new DD_TRACE_AGENT_TIMEOUT_SECONDS to override the default connection timeout used when sending data to the trace agent. The default is 2.0 seconds.

  • The CI tagging for the pytest plugin now includes OS and Python Runtime metadata including system architecture, platform, version, and Python runtime name and version.

Deprecation Notes

  • The reuse of a tracer that has been shut down has been deprecated. A new tracer should be created for generating new traces.

Bug Fixes

  • The shutdown task is re-registered when a tracer is reused after it has been shut down.

  • Fixed the optional argument of Span.finish to Optional[float] instead of Optional[int].

Other Notes

  • The pytest plugin now includes git metadata tags including author name and email as well as commit message from CI provider environments.

  • The profiler won’t be ignoring its own resource usage anymore and will report it in the profiles.

v0.50.0rc1

Prelude

Major changes to context management. See the upgrade section for the specifics. Note that only advanced users of the library should be affected by these changes. For the details please refer to the Context section of the docs: https://ddtrace.readthedocs.io/en/v0.50.0/advanced_usage.html

New Features

  • Add DD_CALL_BASIC_CONFIG={true,false} environment variable to control whether ddtrace calls logging.basicConfig. By default when using ddtrace-run or running in debug mode logging.basicConfig is called to ensure there is always a root handler. This has compatibility issues for some logging configurations. DD_CALL_BASIC_CONFIG=false can be used to skip calling logging.basicConfig. The default value is true to maintain existing behavior.

  • agent: support URL with a base path

  • Automated context management should now work in all asynchronous frameworks that use contextvars.

  • tracer.start_span() now accepts an activate argument (default False) to allow manual context management.

  • tracer.current_trace_context() has been added to be used to access the trace context of the active trace.

  • A warning has been added to alert when gevent monkey patching is done after ddtrace has been imported.

  • Add support for Flask 2

  • Added retry logic to the tracer to mitigate potential networking issues, like timeouts or dropped connections.

Upgrade Notes

  • ddtrace.contrib.asyncio - AsyncioContextProvider can now return and activate None, Span or Context objects.

  • ddtrace.contrib.gevent - GeventContextProvider can now return and activate None, Span or Context objects.

  • ddtrace.contrib.tornado - TracerStackContext can now return and activate None, Span or Context objects.

  • ddtrace.context.Context no longer maintains the active/current span state.

    get_current_root_span() has been removed. Use tracer.current_root_span() instead. get_current_span() has been removed. Use tracer.current_span() instead. add_span() has been removed. To activate a span in an execution use tracer.context_provider.activate() instead. close_span() has been removed. To deactivate a span in an execution use tracer.context_provider.activate() instead.

  • ddtrace.provider.BaseContextProvider active() now returns None, Span or Context objects. activate() now accepts None, Span or Context objects.

  • ddtrace.span.Span - Span.context will now return a Context

  • ddtrace.tracer.Tracer tracer.get_call_context() will now return a one-off Context reference. This is to maintain backwards compatibility with the API but the functionality differs slightly. tracer.start_span() passing a span.context for child_of no longer adds the strong _parent reference to the new span.

  • Support for MySQL-python has been removed.

  • Support for psycopg < 2.7 has been removed.

Bug Fixes

  • The OpenTracing tracer.start_span method no longer activates spans.

  • Datadog active spans will no longer take precedence over OpenTracing active spans.

  • django: fix a bug where multiple database backends would not be instrumented.

  • django: fix a bug when postgres query is composable sql object.

  • Fix double patching of pymongo client topology.

v0.49.0rc1

Prelude

Several deprecations have been made to Context as we prepare to move active span management out of this class.

New Features

  • The futures integration is now enabled by default.

  • requests: add global config support. This enables the requests service name to be configured with ddtrace.config.requests['service'] or the DD_REQUESTS_SERVICE environment variable.

Upgrade Notes

  • Support for aiohttp previous to 2.0 has been removed.

  • Support for deprecated DD_PROFILING_API_URL environment variable has been removed. Use DD_SITE instead.

  • Support for deprecated DD_PROFILING_API_KEY environment variable has been removed. Use DD_API_KEY instead.

  • Profiling support for agentless mode must now be explicitly enabled.

  • The ddtrace pytest plugin can now label spans from test cases marked xfail with the tag “pytest.result”

    and the reason for being marked xfail under the tag “pytest.xfail.reason”.

  • Removed ddtrace.ext.AppTypes and its usages in the tracer library.

  • requests: spans will no longer inherit the service name from the parent.

  • The return value of Span.pprint() has been changed to a single line in the tracer debug logs rather than the previous custom multiline format.

  • Spans are now processed per tracer instance. Formerly spans were stored per-Context which could be shared between tracer instances. Note that context management is not affected. Tracers will still share active spans.

  • Spans from asynchronous executions (asyncio, gevent, tornado) will now be processed and flushed together. Formerly the spans were handled per-task.

  • tracer.write() will no longer have filters applied to the spans passed to it.

  • The function ddtrace.utils.merge_dicts has been removed.

Deprecation Notes

  • Context.clone is deprecated. It will not be required in 0.50.

  • Context.add_span is deprecated and will be removed in 0.50.

  • Context.add_span is deprecated and will be removed in 0.50.

  • Context.close_span is deprecated and will be removed in 0.50.

  • Context.get_current_span is deprecated and will be removed in 0.50 please use Tracer.current_span instead.

  • Context.get_current_root_span is deprecated and will be removed in 0.50 please use Tracer.current_root_span instead.

  • Deprecate the configuration of the analytics through the generic dbapi2 configuration. This should now be configured via integration configurations, for example:

    # Before
    export DD_TRACE_DBAPI2_ANALYTICS_ENABLED=1
    
    # After
    export DD_TRACE_SQLITE3_ANALYTICS_ENABLED=1
    
  • ddtrace.compat has been deprecated and will be removed from the public API in ddtrace version 1.0.0.

  • Deprecate ddtrace.config.dbapi2 as default for TracedCursor and TracedConnection as well as DD_DBAPI2_TRACE_FETCH_METHODS. Use IntegrationConfig and DD_<INTEGRATION>_TRACE_FETCH_METHODS specific to each dbapi-compliant library. For example:

    # Before
    config.dbapi2.trace_fetch_methods = True
    
    # After
    config.psycopg2.trace_fetch_methods = True
    
  • The use of ddtrace.encoding has been deprecated and will be removed in version 1.0.0.

  • The ddtrace.http module has been deprecated and will be removed in version 1.0.0, with the ddtrace.http.headers module now merged into ddtrace.trace_utils.

  • The collect_metrics argument of the tracer.configure method has been deprecated. Runtime metrics should be enabled only via the DD_RUNTIME_METRICS_ENABLED environment variable.

Bug Fixes

  • Fix broken builds for Python 2.7 on windows where <stdint.h> was not available. This change also ensures we build and publish cp27-win wheels.

  • CGroup file parsing was fixed to correctly parse container ID with preceding characters.

  • grpc: handle None values for span tags.

  • grpc: handle no package in fully qualified method

  • grpc: Add done callback in streaming response to avoid unfinished spans if a StopIteration is never raised, as is found in the Google Cloud libraries.

  • grpc: handle IPv6 addresses and no port in target.

  • Fix DD_LOGS_INJECTION incompatibility when using a logging.StrFormatStyle (logging.Formatter(fmt, style="{")) log formatter.

  • Fixed a bug that prevented the right integration name to be used when trying to patch a module on import that is already loaded.

  • Fix urllib3 patching not properly activating the integration.

  • gRPC client spans are now marked as measured by default.

  • Fixes issue of unfinished spans when response is not a grpc.Future but has the same interface, as is the case with the base future class in google-api-core.

  • In certain circumstances, the profiles generated in a uWSGI application could have been empty. This is now fixed and the profiler records correctly the generated events.

  • The default agent timeout for profiling has been restored from 2 to 10 seconds to avoid too many profiles from being dropped.

  • Fix issue with missing traces when using pymemcache.client.hash.HashClient.

  • Added missing pymongo integration configuration, which allows overriding the service name for all the emitted spans.

Other Notes

  • Added environment variable DD_BOTTLE_DISTRIBUTED_TRACING to enable distributed tracing for bottle.

  • The attrs library has been unvendored and is now required as a normal Python dependency with a minimum version requirement of 19.2.0.

  • The six library has been removed from vendor and the system-wide version is being used. It requires version 1.12.0 or later.

  • Documentation on how to use Gunicorn with the gevent worker class has been added.

  • Added environment variable DD_FALCON_DISTRIBUTED_TRACING to enable distributed tracing for falcon.

  • When extracting context information from HTTP headers, a new context is created when the trace ID is either 0 or not available within the headers.

  • Added environment variable DD_PYLONS_DISTRIBUTED_TRACING to enable distributed tracing for pylons.

  • Update pymemcache test suite to test latest versions.

  • Added config.pyramid.distributed_tracing setting to integration config for pyramid.

  • The ddtrace.payload submodule has been removed.

  • Added environment variable DD_TORNADO_DISTRIBUTED_TRACING to enable distributed tracing for tornado.

v0.48.0rc2

New Features

  • Adds support for the Datadog Lambda Extension. The tracer will send traces to the extension by default if it is present.

  • Add support for space-separated DD_TAGS.

  • urllib3: add urllib3 integration

v0.48.0rc1

New Features

  • dogpile.cache: is now automatically instrumented by default.

  • pylons: now supports all the standard http tagging including query string, custom error codes, and request/response headers.

  • The ddtrace pytest plugin can now call ddtrace.patch_all via the --ddtrace-patch-all option.

  • Span now accepts a on_finish argument used for specifying functions to call when a span finishes.

Upgrade Notes

  • The deprecated dogstatsd_host and dogstatsd_port arguments to tracer.configure() have been removed.

  • Support for gevent 1.0 has been removed and gevent >= 1.1 is required.

  • flask: deprecated configuration option extra_error_codes has been removed.

  • The deprecated pyddprofile wrapper has been removed. Use ddtrace-run with DD_PROFILING_ENABLED=1 set instead.

  • A ValueError will now be raised on tracer initialization if the Agent URL specified to the initializer or with the environment variable DD_TRACE_AGENT_URL is malformed.

  • uWSGI is no longer supported with ddtrace-run due to a limitation of how tracer initialization occurs. See the updated instructions for enabling tracing in the library uWSGI documentation.

Bug Fixes

  • The Records parameter to Firehose endpoint calls is being excluded from the tags to avoid generating traces with a large payload.

  • Tracer: fix configuring tracer with dogstatsd url.

  • elasticsearch: patch versioned elasticsearch modules (elasticsearch1, …, elasticsearch7).

  • botocore: Do not assume that ResponseMeta exists in the results.

  • django: handle erroneous middleware gracefully.

  • The tracer now captures the task ID from the cgroups file for Fargate >= 1.4.0 and reports it to the agent as the Datadog-Container-ID tag.

  • Fix a bug when tracing a mako DefTemplate or any Template that does not have a filename property.

  • flask: fix a bug when the query string would contain non-Unicode characters

  • Fix a formatting issue on error exporting profiles.

  • A workaround is provided for the problem with uWSGI worker processes failing to respawn. This can occur when using ddtrace-run for automatic instrumentation and configuration or manual instrumentation and configuration without the necessary uWSGI options. The problem is caused by how the tracer can end up starting threads in the master process before uWSGI forks to initialize the workers processes. To avoid this, we have provided updated instructions for enabling tracing in the library uWSGI documentation.

Other Notes

  • The logic behind the header extraction for distributed tracing has been improved.

  • The default connection timeout for the profiling agent has now been reduced from 10 to 2 seconds to match the tracer behavior.

  • The tracemalloc memory profiler, which was disabled by default, has been removed.

  • Query strings are stripped out from URLs by default when setting URL metadata on a span. This change affects all integrations that store HTTP metadata, like aiohttp, falcon, requests, urllib3.

v0.47.0rc1

New Features

  • cherrypy: introduce TraceMiddleware for the CherryPy web framework.

  • django: tag root spans as measured.

  • elasticsearch: add support for version 7.

  • fastapi: add integration.

  • Introduce support for the DD_SERVICE_MAPPING environment variable to allow remapping service names on emitted spans.

  • httplib: distributed tracing is now enabled by default.

  • The profiler now supports most operation mode from uWSGI without much configuration. It will automatically plug itself in post fork hooks when multiprocess mode is used.

  • wsgi: add tracing middleware.

Upgrade Notes

  • elasticsearch: removed get_traced_transport method and ddtrace.contrib.elasticsearch.transport module.

  • The profiler now automatically sets up uWSGI compatibility in auto mode or with profile_children=True. Make sure that you don’t have custom code instrumenting the profiler in those cases.

  • The Tracer class properties DEFAULT_HOSTNAME, DEFAULT_PORT, DEFAULT_DOGSTATSD_PORT, DEFAULT_DOGSTATSD_URL, DEFAULT_AGENT_URL have been removed.

Bug Fixes

  • Resolves an issue in Django tracing where, if query_string is not present in request.META, a KeyError is raised, causing the request to 500

  • Deprecate the DD_LOGGING_RATE_LIMIT variable in favor of the standard DD_TRACE_LOGGING_RATE for configuring the logging rate limit.

  • sampler: removed bug causing sample_rate of 0 to be reset to 1, and raise ValueError instead of logging.

  • starlette: unpatch calls correctly.

  • flask: fix memory leak of sampled out traces.

  • Fix CPU time and wall time profiling not ignoring the profiler tasks with gevent.

Other Notes

  • The default maximum CPU time used for the stack profiler (CPU time, wall time and exceptions profiling) has been decreased from 2% to 1%.

v0.46.0rc1

New Features

  • aiohttp: store request and response headers.

  • bottle: store request and response headers.

  • flask: store response headers.

  • molten: store request headers.

  • pyramid: store request and response headers.

  • flask: store request and response headers when auto-instrumented.

  • The ddtrace.profiling.auto module will warn users if gevent monkey patching is done after the profiler is auto-instrumented.

  • The Profiler object can now be passed tags with the tags keyword argument.

Upgrade Notes

  • The profiler will only load tags from the DD_TAGS environment variable once at start.

Deprecation Notes

Bug Fixes

  • dbapi: avoid type error with potential non-compliance in db libraries when setting tag for row count.

  • django: add legacy resource format of {handler}.

  • grpc: fix wrapper for streaming response to support libraries that call an internal method directly.

  • sanic: use path parameter names instead of parameter values for the resource.

  • The profiler won’t deadlock on fork when gevent monkey patch is enabled.

  • requests: fix TracedSession when patches are not applied.

v0.45.0rc1

Prelude

Build and deploy Python 3.9 wheels for releases

New Features

  • botocore: added distributed_tracing configuration setting which is enabled by default.

  • The ddtrace-run command now supports the following arguments:

    -h, –help -d, –debug enable debug mode (disabled by default) -i, –info print library info useful for debugging -p, –profiling enable profiling (disabled by default) -v, –version show program’s version number and exit

    It now also has friendlier error messages when used incorrectly.

  • Add functionality to call gevent.monkey.patch_all() with ddtrace-run by setting the environment variable DD_GEVENT_PATCH_ALL=true. This ensures that gevent patching is done as early as possible in the application.

  • botocore: inject distributed tracing data to ClientContext to trace lambda invocations.

  • botocore: inject tracing data to MessageAttributes.

  • The profiler now tracks the running gevent Greenlet and store it as part of the CPU and wall time profiling information.

  • The profiler is now able to upload profiles to the Datadog Agent by using a Unix Domain Socket.

  • It is now possible to pass a url parameter to the Profiler to specify the Datadog agent location.

  • The new memory profiler for Python is now enabled by default. This improves the profiler memory consumption and performance impact. It can still be disabled by setting DD_PROFILING_MEMALLOC=0 as an environment variable.

  • The profiler now uses the tracer configuration is no configuration is provided.

  • pytest integration. This enables the pytest runner to trace test executions.

Upgrade Notes

  • Context.get() has been removed and the functionality has been rolled into Context.close_span().

  • Tracer.record() has similarly been removed as it is no longer useful with Context.get() removed.

  • The deprecated compatibility module ddtrace.profile has been removed.

  • The profiler now uses the tracer configuration is no configuration is provided.

Deprecation Notes

  • The pyddprofile wrapper is deprecated. Use DD_PROFILING_ENABLED=true ddtrace-run instead.

  • The profiler does not catch uncaught exception anymore.

Bug Fixes

  • core: always reset the current_span in the context.

  • django: Http404 exceptions will no longer be flagged as errors

  • django: add safe guards for building http.url span tag.

  • aiobotocore: set span error for 5xx status codes.

  • elasticsearch: set span error for 5xx status codes.

  • django, DRF, ASGI: fix span type for web request spans.

  • Fixes span id tagging in lock profiling.

  • Fix UDS upload for profiling not using the correct path.

  • Fixed an issue in profiling exporting profiles twice when forking.

  • core: fix race condition in TracerTagCollector.

Other Notes

  • Start-up logs are now disabled by default. To enable start-up logs use DD_TRACE_STARTUP_LOGS=true or DD_TRACE_DEBUG=true.

v0.44.0

Prelude

Add support for Python 3.9

New Features

  • Store request headers in Flask integration.

  • pyodbc integration. This enables the pyodbc library to trace queries.

  • starlette integration resource aggregation This aggregates endpoints to the starlette application resource that was accessed. It occurs by default but it is configurable through config.starlette[“aggregate_resources”].

  • The profiler now captures the traces information with the lock profiling.

  • The Profiler instances now restart automatically in child process when the main program is forked. This only works for Python ≥ 3.7.

Bug Fixes

  • dbapi: add support for connection context manager usage

  • django: check view before instrumenting MRO.

  • core: use loose types when encoding.

  • Patch pynamodb on import to prevent patching conflicts with gevent.

  • tornado: handle when the current span is None in log_exception().

Prior Releases

Release notes prior to v0.44.0 can be found in CHANGELOG.md in the root of the repository.