v2.1.7#
Bug Fixes#
Vulnerability Management for Code-level (IAST): This fix addresses AST patching issues where custom functions or methods could be replaced by aspects with differing argument numbers, causing runtime errors as a result. Furthermore, it addresses a case during patching where the module is inadvertently passed as the first argument to the aspect.
v2.1.6#
Bug Fixes#
ASM: add support for psycopg2 adapt mechanism to LazyTaintList, preventing a ProgrammingError when using psycopg2 with IAST.
langchain: This fix resolves an issue with tagging pydantic SecretStr type api keys.
lib injection: Fix permissions error raised when non-root users copy single step instrumentation files.
This fix resolves an issue where type-checking would fail on valid application code due to a type hint on
Span.set_tags
that was too restrictive.
v2.1.5#
Upgrade Notes#
The
wrapt
andpsutil
packages are vendored to help users avoid building these packages if wheels were not available for a given platform. This reverses a change released in v2.0.0.
Bug Fixes#
CI: fixes an issue which prevented the library from filtering user credentials for SSH Git repository URLs
Use a unique default service name across all the products provided by the library when one is not given via the configuration interface.
CI Visibility: fixes
unittest
data not being initialized properly
Vulnerability Management for Code-level (IAST): This fix resolves an issue where SimpleJSON encoder would throw an exception while encoding a tainted dict or list.
v2.1.4#
Bug Fixes#
Application Security Management (ASM): fix a body read error when
Transfer-Encoding: chunked
header is sent
v2.1.3#
Bug Fixes#
IAST: This fix resolves an issue where JSON encoder would throw an exception while encoding a tainted dict or list.
v2.1.2#
Known Issues#
ASM: fix a body read problem on some corner case where passing empty content length makes wsgi.input.read() blocks.
Bug Fixes#
CI Visibility: fixes an issue where class-based test methods with the same name across classes would be considered duplicates, and cause one (or more) tests to be dropped from results, by adding
--ddtrace-include-class-name
as an optional flag (defaulting to false) to prepend the class name to the test name.
dynamic instrumentation: fix an issue that caused the probe instrumentation error details from being reported for visualization in the UI.
dynamic instrumentation: fix an issue that caused function probes on the same module to fail to instrument and be reported in the
ERROR
status in the UI if the module was not yet imported.
sampling: This fix reverts a refactor which affected how the tracer handled the trace-agent’s recommended trace sampling rates, leading to an unintended increase in traces sampled.
remote config: Add git metadata to configuration requests to ensure Source Code Integration (SCI) works as expected with services that require it.
CI Visibility: fixes an issue where just importing unittest enabled CIVisibility and potentially caused unexpected logs and API requests
Vulnerability Management for Code-level (IAST): Fix potential string id collisions that could cause false positives with non tainted objects being marked as tainted.
v2.1.1#
Bug Fixes#
CI Visibility: fixes distributed tracing when using the CI Visibility integration
ASM: This fix resolves an issue where an f-string expression would not be formatted properly causing a segfault if IAST is enabled.
langchain: This fix resolves an import error with patching langchain versions newer than 0.0.300 due to langchain dropping support for wrapping or importing directly from root.
lib-injection: Update package files to not be world-writable.
data_streams: This fix resolves an issue where including exit signal handling for message queues caused imports of ddtrace in a thread to fail.
v2.1.0#
Upgrade Notes#
pymemcache: The
memcached.query
span tag will no longer be set by the integration. This command includes keys that can potentially contain sensitive information. If you require this span tag, you can retain the existing functionality by settingDD_TRACE_MEMCACHED_COMMAND_ENABLED=true
. This span tag can be redacted usingDD_APM_REPLACE_TAGS
in your Agent configuration.
New Features#
CI Visibility: adds full test suite level visibility for unittest
ASM: Add support for automatic user login events in Flask when using flask_login.
tracer: This introduces collection of inferred service names. The agent version v7.46.0 contains a new field “extra_services” in the remote config client, that allows clients to list any additional services that are used within tracer spans. Knowing all service names used by a tracer instance help the UI give better feedback to the user.
- tracer: Adds support for
DD_TRACE_METHODS
. This feature enables the specification of custom methods to be instrumented by the tracer when using
ddtrace-run
. See the configuration documentation for more information: https://ddtrace.readthedocs.io/en/v2.1.0/configuration.html.
- tracer: Adds support for
openai: This introduces official tracing support for Azure OpenAI services, specifically for completions, chat completions, and embeddings using models deployed on Azure OpenAI endpoints.
Vulnerability Management for Code-level (IAST): Weak randomness vulnerability detection.
pymemcache: add
DD_TRACE_MEMCACHED_COMMAND_ENABLED
environment variable for configuring the collection of memcached commands. This feature is disabled by default.
wsgi: This change introduces the keyword argument app_is_iterator to the DDWSGIMiddleware constructor. It’s provided as a workaround for an issue where the Datadog WSGI middleware would fail to handle WSGI apps that are not their own iterators. This condition can arise when a Django app attempts to send its “request finished” signal, in which case it may cause connection leaks. Standard methods of distinguishing an iterable from its iterator, such as checking for the presence of iter and next dunder methods, don’t work in this case for unknown reasons. Instead of adding brittle special-case detection logic to the middleware, this new argument allows users to indicate when this is the case.
Bug Fixes#
dbm: this fixes an issue with attributing metrics to incorrect services when peer.service is enabled
This fix resolves an issue where the library failed to install due to dependency conflicts caused by restrictive version specifiers on the psutil and Wrapt libraries.
tracing: Encoding traces in the v05 format has a known issue for applications generating spans at a high frequency, causing approximately 1/10000000 spans to be misencoded. This change sets
DD_TRACE_API_VERSION
tov04
by default.
DSM: fix off-by-one metric issue and error where statistics weren’t calculated when the core API was used.
tracing: Fix an issue with some integrations, such as OpenAI, that caused an exception on start-up when using gevent.
IAST: fix executed sink telemetry metric as it is not really linked to vulnerability report.
CI Visibility: Fixes incorrect Git version extraction
kafka: Fixes
ValueError
raised whenConsumer.commit(offsets=...)
is called.
Resolves issues with the import machinery that might have caused the garbage collector to clean up weak references in an unexpected order.
openai: This fix resolves an issue where chat completion requests with function calls led to failing to tag null message content fields in the chat completion response.
profiling: Fixed an issue with data encoding where non-string objects might be interned on Python 3.11
profiling: Load the protobuf module only if needed to avoid interfering with the module state for applications that also make use of it. The protobuf module is used in file and classic Python HTTP export. It is not needed for the libdatadog-based exporter.
Fix an issue that could have caused some tracing integrations to create invalid references to objects in Python frames, ultimately causing profiling tools to potentially induce a segmentation fault.
Fix an issue that could have caused some tracing integrations to leave the interpreter in an inconsistent state, resulting in the profiler incurring a segmentation fault.
redis: Resolves
UnboundLocalError
raised when a traced redis command raises an exception.
tracing: Increases the maximum payload size and buffer size from 8MB to 20MB. Also decreases the maximum number of spans in trace chunks when
DD_TRACE_PARTIAL_FLUSH_ENABLED=True
. This ensures large traces are correctly encoded and submitted. This should decrease the occurrence of “failed to send traces” error logs.
graphql: Resolves
AttributeError
raised while parsing graphql Documents where AST Location is None.
lib-injection: changes the log output to opt-in. Logging to stderr could interfere with applications. Logs can still be sent to stderr using DD_TRACE_DEBUG=1.
Other Changes#
remote config: change log levels of messages about agent down or request config to debug level.
v2.0.0rc1#
Prelude#
The Datadog APM Python team is happy to announce the release of v2.0.0 of ddtrace. This release drops support for Python 2.7, 3.5, and 3.6. This release adds support for Python 3.12.
Important
If you are on version of Python not supported by v2, we will continue to maintain the ddtrace v1 with bug fixes.
Note
Before upgrading to v2.0.0, we recommend users install ddtrace~=1.20.0
and enable deprecation warnings. All removals to the library interface and
environment variables in v2 were deprecated in the 1.x release line.
Note
The changes to environment variables apply only to the configuration of the ddtrace library and not the Datadog Agent.
Upgrading summary#
Functionality changes#
The default logging configuration functionality of ddtrace has been changed to avoid conflicting with application logging configurations. DD_CALL_BASIC_CONFIG
has been removed and the ddtrace logger will log to stdout by default, or a log file as specified using DD_TRACE_LOG_FILE
.
Setting the environment variable DD_TRACE_PROPAGATION_STYLE='b3'
, which previously enabled b3multi
now enables b3 single header
. b3 single header
still works but is deprecated for b3
. Simplified: b3
used to enable b3multi
, but now enables b3 single header
to better align with Opentelemetry’s terms.
Removed deprecated environment variables#
These environment variables have been removed. In all cases the same functionality is provided by other environment variables and replacements are provided as recommended actions for upgrading.
Removed deprecated library interfaces#
These methods and module attributes have been removed. Where the same functionality is provided by a different public method or module attribute, a recommended action is provided for upgrading. In a few limited cases, because the interface was no longer used or had been moved to the internal interface, it was removed and so no action is provided for upgrading.
Module |
Method/Attribute |
Note |
---|---|---|
|
|
|
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
|
|
|
||
|
|
|
|
||
|
||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Upgrade Notes#
DD_GEVENT_PATCH_ALL
is removed. There is no special configuration necessary to make ddtrace work with gevent if using ddtrace-run.
DD_AWS_TAG_ALL_PARAMS
is removed. The boto/botocore/aiobotocore integrations no longer collect all API parameters by default.
DD_REMOTECONFIG_POLL_SECONDS
is removed. Use the environment variableDD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS
instead.
APPSEC_ENABLED
,APPSEC_JSON
,APPSEC_EVENT_RULE_VERSION
,APPSEC_EVENT_RULE_ERRORS
,APPSEC_EVENT_RULE_LOADED
,APPSEC_EVENT_RULE_ERROR_COUNT
,APPSEC_WAF_DURATION
,APPSEC_WAF_DURATION_EXT
,APPSEC_WAF_TIMEOUTS
,APPSEC_WAF_VERSION
,APPSEC_ORIGIN_VALUE
,APPSEC_BLOCKED
,IAST_JSON
,IAST_ENABLED
,IAST_CONTEXT_KEY
are removed. This should not affect existing code as these deprecated ASM constants were meant for private use only.
ddtrace.contrib.starlette.get_resource
,ddtrace.contrib.starlette.span_modifier
, andddtrace.contrib.fastapi.span_modifier
are removed. The starlette and fastapi integrations now provide the full route and not just the mounted route for sub-applications.
ddtrace.contrib.starlette.config['aggregate_resources']
andddtrace.contrib.fastapi.config['aggregate_resources']
are removed. The starlette and fastapi integrations no longer have the option toaggregate_resources
, as it now occurs by default.
ddtrace.contrib.grpc.constants.GRPC_PORT_KEY
is removed. Useddtrace.ext.net.TARGET_PORT
instead.
ddtrace.ext.cassandra.ROW_COUNT
is removed. Useddtrace.ext.db.ROWCOUNT
instead.
ddtrace.ext.mongo.ROW_COUNT
is removed. Useddtrace.ext.db.ROWCOUNT
instead.
ddtrace.ext.sql.ROW_COUNT
is removed. Useddtrace.ext.db.ROWCOUNT
instead.
ddtrace.filters.TraceCiVisibilityFilter
is removed.
ddtrace.tracer.DD_LOG_FORMAT
is removed. As an alternative, please follow the log injection formatting as provided in the log injection docs.
DD_CALL_BASIC_CONFIG
is removed. There is no special configuration necessary to replaceDD_CALL_BASIC_CONFIG
. The ddtrace logger will log to stdout by default or additionally to a file specified byDD_TRACE_LOG_FILE
.
DD_TRACE_OBFUSCATION_QUERY_STRING_PATTERN
is removed. UseDD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP
instead.
New Features#
Adds support for Python 3.12.
Known Issues#
aiohttp: Python 3.12 is not supported.
aiohttp-jinja: Python 3.12 is not supported.
aiobotocore: Python 3.12 is not supported.
asm: IAST for Python 3.12 is not supported.
flask-caching: Python 3.12 is not supported.
openai/langchain: Python 3.12 is not supported.
opentelemetry-api: Python 3.12 is not supported.
opentracing: Python 3.12 is not supported.
pyramid: Python 3.12 is not supported.
pynamodb: Python 3.12 is not supported.
redis/redis-py-cluster: Python 3.12 is not supported.
v1.20.0#
Prelude#
Vulnerability Management for Code-level (IAST) is now available in private beta. Use the environment variable DD_IAST_ENABLED=True
to enable this feature.
New Features#
ASM: This introduces support for custom blocking actions of type redirect_request.
data_streams: Adds public api
set_produce_checkpoint
andset_consume_checkpoint
Bug Fixes#
kafka: Resolves an issue where traced kafka connections were assigned a default timeout of 1 second. The default timeout in Consumer.poll(…) should be None.
openai: This fix resolves an issue where errors during streamed requests resulted in unfinished spans.
v1.19.0rc1#
New Features#
Adds the db.row_count tag to redis and other redis-like integrations. The tag represents the number of returned results.
CI Visibility: adds test level visibility for unittest
ASM: Adds detection of insecure cookie vulnerabilities on responses.
ASM: This introduces trusted IPs capabilities in the tracer, to allow specific IPs not to be blocked by ASM but still be monitored.
ASM: This introduces a new capability to configure the blocking response of ASM. Users can change the default blocking response behavior or create new custom actions. Configuration of a custom blocking page or payload can still be provided by using DD_APPSEC_HTTP_BLOCKED_TEMPLATE_JSON and DD_APPSEC_HTTP_BLOCKED_TEMPLATE_HTML to change the static files used for the response body. The action block, that can be defined in the static rule file or via remote configuration, allows now to create new custom blocking actions with any status code for the response.
The aiopg and aiomysql integrations no longer set the sql.query tag on query spans. This tag duplicated the value captured by the span resource. Users who want to send this query unobfuscated can use the tracer API to set tags on the query span.
data_streams: Starts tracking Kafka lag in seconds.
kafka: Adds support for the Kafka serializing producer and deserializing consumer.
profiling: allow individual collectors to be disabled.
tracing: This change introduces the
allow_false
keyword argument toBaseSampler.sample()
, which defaults toTrue
.allow_false
controls the function’s return value. Ifallow_false
isFalse
, the function will always returnTrue
regardless of the sampling decision it made. This is useful whensample
is called only for its side effects, which can include setting span tags.
Known Issues#
There are known issues configuring python’s builtin multiprocessing library when ddtrace is installed. To use the multiprocessing library with ddtrace ensure
DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE
is set toTrue
.
When running setup.py extensions with the CMake parameter “-j”, it could potentially raise an out-of-memory error. If someone wants to expedite the ddtrace installation, they should manually set the “CMAKE_BUILD_PARALLEL_LEVEL” environment variable.
Bug Fixes#
ASM: avoid potentially unneeded import of the IAST native module.
ASM: avoid potentially unneeded import of the IAST native module if setup doesn’t build extensions correctly.
data_streams: This fix resolves an issue where data stream context propagation would not propagate via SNS if raw message delivery was enabled.
dynamic instrumentation: function duration measurements are now reported in milliseconds to match the expectation from the UI.
dynamic instrumentation: fixed an issue that prevented line probes from being injected in some finally blocks.
dynamic instrumentation: Fixed the programmatic API to ensure that the dynamic instrumentation service is fully enabled when
Dynamic Instrumentation.enable()
is called.
dynamic instrumentation: fixed a bug that might have caused probe status to fail to update correctly.
django: This fix resolves an issue where ‘span.resource’ would not include the endpoint when a Handler was interrupted, such as in the case of gunicorn worker timeouts.
CI Visibility: fixes an issue where the Intelligent Test Runner would not work when in EVP proxy mode due to missing
X-Datadog-NeedsAppKey
header.
CI Visibility: revert to using DD_CIVISIBILITY_ITR_ENABLED (instead of _DISABLED) to conform with other tracers.
profiling: fixed a bug that prevented profiles from being correctly correlated to traces in gevent-based applications, thus causing code hotspot and end point data to be missing from the UI.
docs: Fix undefined variable reference in otel documentation
CI Visibility: fixes that Python 2.7 test results were not visible in UI due to improperly msgpack-ed data
ASM: This fix resolves an issue where track_user_signup_event and track_custom_event where not correctly tagging the span. This could lead to the loss of some events in the sampling.
appsec: Fixes an issue where ddtrace.appsec is imported and assumed to be available in all deployments of ddtrace
- lib-inject: This fix resolves an issue where
libdl.so.2: cannot open shared object file: No such file or directory
errors occurred when the injection image started.
- lib-inject: This fix resolves an issue where
lib-injection: Resolves permissions errors raised when ddtrace packages are copied from the InitContainer to the shared volume.
mariadb: This fix resolves an issue where MariaDB connection information objects not including the user or port caused exceptions to be raised.
appsec: This fix resolves an issue in which the library attempted to finalize twice a context object used by the Application Security Management product.
propagation: Prevent propagating unsupported non-ascii
origin
header values.
pymongo: This upgrades the PyMongo integration to work with PyMongo versions 4.5.0 and above by choosing the root function of the integration on the basis of the PyMongo version.
tracing: This fix resolves an issue where the _dd.p.dm and _dd.*_psr tags were applied to spans in ways that did not match their intended semantics, increasing the potential for metrics-counting bugs.
ASM: This fix resolves issue where user information was only set in root span. Now span for user information can be selected.
sqlalchemy: sqlalchemy rollbacks could previously cause intermittent deadlocks in some cases. To fix this
DD_TRACE_SPAN_AGGREGATOR_RLOCK
was introduced in 1.16.2 with the default asFalse
. We are now changing the default toTrue
.
Other Changes#
Adds a get_version method to each integration and updates the basic template for developing an integration to include this method. The get_version method returns the integration’s package distribution version and is to be included in the APM Telemetry integrations payload.
Add a ddtrace_iast_flask_patch function defined in ddtrace.appsec.iast to ensure that the main Flask app.py file is patched for IAST propagation. This function should be called before the app.run() call. You only need this if you have set DD_IAST_ENABLED=1. Only the main file needs to call this functions, other imported modules are automatically patched.
docs: Fixes formatting in ddtrace docs.
ASM: Improve default value of regex for query string obfuscation. Rename env var
DD_TRACE_OBFUSCATION_QUERY_STRING_PATTERN
toDD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP
.
v1.18.0rc1#
Prelude#
Data Streams Monitoring (DSM) has added support for AWS Kinesis
Breaking change for CI Visibility: test.suite
and test.full_name
are changed, so any visualization or monitor that uses these fields is potentially affected.
Deprecation Notes#
DD_CALL_BASIC_CONFIG
will be removed in the upcoming 2.0.0 release. As an alternative toDD_CALL_BASIC_CONFIG
, you can calllogging.basicConfig()
to configure logging in your application.
DD_LOG_FORMAT
is deprecated and will be removed in 2.0.0. As an alternative, please follow the log injection formatting as provided in the log injection docs.
New Features#
CI Visibility: added tracing support for pytest-benchmark
ASM: The vulnerability report now includes a feature to scrub potentially sensitive information. This scrubbing process looks for common patterns, and it can be further expanded using environment variables such as
DD_IAST_REDACTION_NAME_PATTERN
andDD_IAST_REDACTION_VALUE_PATTERN
. See the docs for more information.
DSM: Adds DSM support for AWS Kinesis. For information about DSM, see the official documentation. This change requires users to use botocore version 1.26.30 or later and update calls to Kinesis’ PutRecord, PutRecords, and GetRecords calls with the StreamARN argument.
pytest: This change introduces an option to the pytest plugin to disable ddtrace:
--no-ddtrace
CI visibility: Adds support for tracking repository URLs via the BITBUCKET_GIT_HTTP_ORIGIN environment variable
CI visibility: Adds CodeFresh integration
CI Visibility: Beta release of
pytest
support for the Intelligent Test Runner .
- openai:
tiktoken
has been introduced as an optional package dependency to calculate the number of tokens used in a prompt for a streamed completion or streamed chat completion. To enable this feature, install
ddtrace[openai]
ortiktoken
. Iftiktoken
is not installed, the prompt token count will be continue to be estimated instead.
- openai:
Allows the use of a new backend for storing and exporting profiling data. This feature can be enabled for now by setting the DD_PROFILING_EXPORT_LIBDD_ENABLED environment variable to true. This should improve performance while decreasing memory overhead.
Known Issues#
sqlalchemy: sqlalchemy rollbacks can intermittently cause deadlocks in some cases. If experiencing this issue, set
DD_TRACE_SPAN_AGGREGATOR_RLOCK=True
. After testing and feedback we intend to make True the default value.
Bug Fixes#
CI Visibility: fixes an issue where the CIVisibility client would raise an exception if it was started in agentless mode without the DD_API_KEY set
core: This fix moves
cmake
frominstall_requires
tosetup_requires
.
data_streams: This change fixes a bug in the Kafka & SQS integrations in which the Data Streams product code incorrect set timestamps for statistics. This led to all points being submitted for the same timestamp (the start of the application).
dynamic instrumentation: handle null literal in conditions and expressions.
dynamic instrumentation: fixed a bug that prevented span decoration probes from being received and instrumented.
dynamic instrumentation: ensure that probes that fail to be instrumented because of invalid conditions/expressions are reported with status
ERROR
in the UI.
CI Visibility: This fix solves an issue where the git unshallow command wasn’t called
tracing: Ensures health metrics are tagged with the correct values.
CI Visibility: This fix resolves an issue where test skipping was not working properly.
- langchain: This fix resolves an issue where chat messages and embedding arguments
passed in as keyword arguments were not parsed correctly and resulted in an
ArgumentError
.
- langchain: This fix resolves an issue where
langchain.embeddings.HuggingFaceEmbeddings
embedding methods, and
langchain.vectorstores.Milvus.similarity_search
were patched twice due to a nested class hierarchy inlangchain
.
- langchain: This fix resolves an issue where
profiling: prevent deadlocks while recording events of different type.
pytest: This fix resolves an issue where test modules could be non-existent, causing errors in the CI Visibility product.
kafka: Resolves
UnicodeDecodeError
raised when kafka messages key contain characters that are not supported by UTF-8 encoding.
lib-injection: Adds support for non-root run applications in containers.
This fix resolves an issue causing span tags used by the Datadog backend not to be inherited by spans that exist in a different process from their parents.
Other Changes#
tracing: Previously the maximum size of a span tag was set to the full size of trace writer buffer (via DD_TRACE_WRITER_BUFFER_SIZE_BYTES). With this change the maximum size of span tags will be set to 10% of the size of the writer’s buffer. This should decrease the frequency of encoding errors due to large span tags.
v1.17.0rc1#
Prelude#
Datadog has added support for automatically creating login success or failure events when a configured Django authentication backend is used. This will automatically fill the following tags in these cases:
appsec.events.users.login.success.track
appsec.events.users.login.failure.track
appsec.events.users.login.success.[email|login|username]
appsec.events.users.login.failure.usr.exists
New Features#
ASM: Add support for automatic user login events in Django.
- langchain: Adds integration with support for metrics, logs, and traces from LangChain requests.
See the docs for more information.
redis: Add support for Async RedisCluster.
Bug Fixes#
core: This fix removes the inclusion of our
benchmarks/
directory in theddtrace
wheels.
internal: call
_fixupChildren
when retrievingDDLogger
profiling: Fixed a regression whereby the profile exporter would not handle known request errors and asks the user to report an issue instead.
profiling: Handles a race condition, which would occasionally throw an error, which would read
"RuntimeError: the memalloc module was not started."
CI visibility: fix version and step arguments gathering to enable plugin compatibility with pytest-bdd 6.1.x
Fixed a bug that caused applications using gevent and cassandra to fail to start with the ddtrace-run command.
tracing: This fix resolves a
google.protobuf
import error when module unloading.
wsgi: This fix resolves an issues when trying to parse the
environ
propertyHTTPS
as an HTTP header.
Pin
cython<3
due to an incompatibility withcython==3.0.0
and typing annotations in profiling code.
telemetry: resolves issue with sending unnecessary duplicate logs
v1.16.0rc1#
Prelude#
Application Security Management (ASM) has added support for tracing subprocess executions.
Exception Debugging allows capturing debug information from exceptions attached to traces. The information about local variables and function arguments is displayed in the Error Tracking UI and augments the traceback data already collected.
New Features#
ASM: vulnerabilities related to insecure request cookies will be reported when
DD_APPSEC_ENABLED
is set totrue
.
- ASM: add support for tracing subprocess executions (like os.system, os.spawn, subprocess.Popen and others) and adding
information to a span names command_execution with the new type system. Currently we add the cmd.exec or cmd.shell tags to store the full command line (cmd.shell will be used when the command is run under a shell like with os.system or Popen with shell=True), cmd.exit_code to hold the return code when available, component which will hold the Python module used and the span resource will hold the binary used. This feature requires ASM to be activated using the DD_APPSEC_ENABLED=True configuration environment variable.
botocore: Introduces environment variable
DD_BOTOCORE_INSTRUMENT_INTERNALS
that opts into tracing certain internal functionality.
botocore: Added message attributes to Amazon Simple Queue Service spans to support data streams monitoring.
exception debugging: Introduced the Exception Debugging feature that allows capturing debug information from exceptions attached to traces. This new feature can be enabled via the DD_EXCEPTION_DEBUGGING_ENABLED` environment variable.
- openai: Adds support for metrics, logs, and traces for the models, edits, images, audio, files, fine-tunes, and
moderations endpoints. See the docs for more information.
- CI Visibility: Updates how pytest modules and test suites are reported. Modules names are now set to the fully qualified name, whereas test suites will be set to the file name.
Before this change: {“module”: “tests”, “suite”:”my_module/tests/test_suite.py”} After this change: {“module”: “my_module.tests”, “suite”: “test_suite.py”}
core: Apply
DD_TAGS
to runtime metrics.
kafka: Adds messaging.kafka.bootstrap.servers tag for the confluent-kafka producer configuration value found in metadata.broker.list or bootstrap.servers
tracing: This reports the GRPC package name (optional) and service name in a single rpc.service tag
Bug Fixes#
botocore: This fix resolves an issue where ddtrace attempted to parse as URLs SQS QueueUrl attributes that were not well-formed URLs.
psycopg: Resolves
TypeError
raised when an async cursor object is traced. This fix ensures exc_type, exc_val, and exc_tb are passed down to the wrapped object on __aexit__.
Fixed an issue that prevented the library from working as expected when a combination of gevent and asyncio-based frameworks that rely on the functionalities of the ssl module is used.
openai: Fixes the issue with
ImportError
ofTypedDict
fromtyping
module in Python 3.7.
openai: This fix resolves an issue where embeddings inputs were always tagged regardless of the configured prompt-completion sample rate.
pytest: This fix resolves an issue where failures and non-skipped tests were not propagated properly when
unittest.TestCase
classes were used.
Fixes an issue where harvesting runtime metrics on certain managed environments, such as Google Cloud Run, would cause ddtrace to throw an exception.
graphql:
graphql.execute
spans are now marked as measured.
tracing: This fix resolves an issue where negative trace ID values were allowed to propagate via Datadog distributed tracing HTTP headers.
openai: Resolves some inconsistencies in logs generated by the image and audio endpoints, including filenames, prompts, and not logging raw binary image data.
pymemcache: This fix resolves an issue where overriding span attributes on
HashClient
failed whenuse_pooling
was set.
This fix resolves an issue causing MyPy linting to fail on files that import ddtrace.
The 1.15.0 version has a bug that arises when Remote Config receives both kinds of actions (removing target file configurations and loading new target file configurations) simultaneously, as the load action overrides the remove action. This error occurs if someone creates and removes Dynamic Instrumentation Probes rapidly, within a time interval shorter than the Remote Config interval (5s). To fix this issue, this update appends all new configurations and configurations to remove, and dispatches them at the end of the RC request.
v1.15.0rc1#
New Features#
pyramid: Adds http.route tag to
pyramid.request
spans.
data_streams: Add data streams core integration and instrument the confluent Kafka library with it. For more information, check out the docs, https://docs.datadoghq.com/data_streams/
dynamic instrumentation: Added support for span decoration probes.
Bug Fixes#
ASM: This fix resolves an issue where the WAF rule file specified by DD_APPSEC_RULES was wrongly updated and modified by remote config.
celery: Resolves an issue where hostname tags were not set in spans generated by
celery>4.0
.
django: Resolves an issue where the resource name of django.request span did not contain the full name of a view when
DD_DJANGO_USE_HANDLER_RESOURCE_FORMAT=True
. This issue impactsdjango>=4.0
.
CI Visibility: This fix resolves the compatibility for Gitlab 16.0 deprecated urls
openai: Resolves an issue where using an array of tokens or an array of token arrays for the Embeddings endpoint caused an AttributeError.
profiling: Fixed an issue with gunicorn and gevent workers that occasionally caused an
AttributeError
exception to be raised on profiler start-up.
psycopg: Fixes
ValueError
raised when dsn connection strings are parsed. This was fixed in ddtrace v1.9.0 and was re-introduced in v1.13.0.
gunicorn: This fix ensures ddtrace threads do not block the master process from spawning workers when
DD_TRACE_DEBUG=true
. This issue impacts gunicorn applications using gevent andpython<=3.6
.
v1.14.0rc1#
Prelude#
profiling: Code provenance is a feature that enhances the “My code” experience in the Datadog UI by allowing the tracer to report packaging metadata about installed source files. This information is used to distinguish between user and third-party code.
New Features#
aws: Adds span tags for consistency with tags collected by Datadog for AWS metrics and logs.
botocore: Adds the ability to control which botocore submodules will be patched.
ASM: Send WAF metrics over telemetry
pytest: This introduces test suite and module level visibility for the pytest integration. Pytest test traces will now include test session, test module, test suite, and test spans, which correlate to pytest session, pytest package, pytest module, and pytest test functions respectively.
redis: Introducing redis command span tag max length configuration for aioredis, aredis, redis, rediscluster, and yaaredis integrations.
profiling: Code provenance is enabled by default.
- OpenAI: Add integration with support for metrics, logs and traces from
OpenAI requests. See the docs for more information.
Bug Fixes#
dependencies: Resolves an issue where ddtrace installs an incompatible version of cattrs when Python 3.6 is used.
tracing: Resolves an issue where
DD_TRACE_<INTEGRATION>_ENABLED=False
could not be used to disable the following integrations whenddtrace-run
was used: flask, django, bottle, falcon, and pyramid.
asgi: Ensures
error.message
anderror.stack
tags are set when an exception is raised in a route.
appsec: Fixes an encoding error when we are unable to cleanup the AppSec request context associated with a span.
ASM: Fixes encoding error when using AppSec and a trace is partial flushed.
CI Visibility: This fix resolves an issue where the tracer was doing extra requests if the
DD_CIVISIBILITY_ITR_ENABLED
env var was not set.
CI Visibility: This fix resolves an issue where the API call would fail because it is reporting a null service name
bootstrap: fixed an issue with the behavior of
ddtrace.auto
that could have caused incompatibilities with frameworks such asgevent
when used as a programmatic alternative to theddtrace-run
command.
django: Fixed a bug that prevented a Django application from starting with celery and gevent workers if
DJANGO_SETTINGS_MODULE
was not explicitly set.
tracing: Fixes a cryptic encoding exception message when a span tag is not a string.
ASM: fix extract_body for Django such that users of Django Rest Framework can still use custom parsers.
flask: Remove patching for Flask hooks
app.before_first_request
andbp.before_app_first_request
if Flask version >= 2.3.0.
gevent: Fix a bug that caused traceback objects to fail to pickle when using gevent.
OpenAI: Resolved an issue where OpenAI API keys set in individual requests rather than as an environment variable caused an error in the integration.
profiler: Fixed a bug that caused segmentation faults in applications that use protobuf as a runtime dependency.
redis: Resolves an issue where the aioredis/aredis/yaaredis integrations cross-imported a helper method from the redis integration, which triggered redis patching before the redis integration was fully loaded.
wsgi: Resolves an issue where accessing the
__len__
attribute on traced wsgi middlewares raised a TypeError
django: Adds catch to guard against a ValueError, AttributeError, or NotImplementedError from being thrown when evaluating a django cache result for
db.row_count
tag.
- lib-injection: Ensure local package is installed. Previously the package
could still be pulled from the internet causing application slowdowns.
kafka: Fixes
TypeError
raised when arbitrary keyword arguments are passed toconfluent_kafka.Consumer
profiler: Fix support for latest versions of protobuf.
psycopg: Resolves an issue where an AttributeError is raised when
psycopg.AsyncConnection
is traced.
sanic: Resolves
sanic_routing.exceptions.InvalidUsage
error raised when gevent is installed orDD_UNLOAD_MODULES_FROM_SITECUSTOMIZE
is set to True.
elasticsearch: This fix resolves an issue where the tracer would throw an error when patching unsupported versions of elasticsearch (> 8.0). Patching is now skipped if an unsupported version is detected.
Other Changes#
span: Increases the traceback limit in
error.stack
tags from 20 to 30
aws_lambda: Logs warnings and exceptions on cold start only.
v1.13.0rc1#
New Features#
psycopg: This release adds support for the new psycopg3 package. This new integration has all the same tracing functionality as the previous psycopg2-binary package, with added support for new methods including async connection and async cursor classes. The release also adds support for using Django>=4.2 with psycopg3 integrated tracing.
Bug Fixes#
algoliasearch: This fix resolves an issue where non-text search query arguments caused Type Errors when being added as tags.
ASM: fix calling set_user without a created span raising a ValueError.
django: Adds fix for bug where Django cache return object throws an error if it does not implement
__bool__()
.
kafka: Previously instantiating a subclass of kafka’s Producer/Consumer classes would result in attribute errors due to patching the Producer/Consumer classes with an ObjectProxy. This fix resolves this issue by making the traced classes directly inherit from kafka’s base Producer/Consumer classes.
profiling: Fixed a regression in the memory collector that caused it to fail to cleanly re-initialize after a fork, causing error messages to be logged.
logging: Ensure that the logging module can report thread information, such as thread names, correctly when a framework like gevent is used that requires modules cleanup.
ASM: This fix resolves an issue where path parameters for the Flask framework were handled at response time instead of at request time for suspicious request blocking. This close a known issue opened in 1.10.0.
- lib-injection: Switch installation to install from included wheels. Prior,
the wheels were merged together which caused conflicts between versions of dependencies based on Python version.
tracer: Handle exceptions besides
ImportError
when integrations are loaded.
Other Changes#
ASM: Add information about Application Security config values on ddtrace-run –info.
otel: Fixes code formatting in api docs
v1.12.0#
New Features#
tracing: Adds support for 128 bit trace ids for b3 and w3c distributing tracing headers.
pytest: Adds the
DD_CIVISIBILITY_AGENTLESS_ENABLED
environment variable to configure theCIVisibility
service to use an agent-less test reportingCIVisibilityWriter
. Note that theCIVisibility
service will use regular agent reporting by default.
sci: Extracts and sends git metadata from environment variables
DD_GIT_REPOSITORY_URL
,DD_GIT_COMMIT_SHA
, or from the python package specified in theDD_MAIN_PACKAGE
. This feature can be disabled by settingDD_TRACE_GIT_METADATA_ENABLED=False
.
otel: Adds support for the OpenTelemetry Tracing API. Please refer to the
docs
for more details.
Bug Fixes#
tracing: Ensure datadog headers propagate 128 bit trace ids when
DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=False
aws_lambda: Fix AttributeError raised when
ddtrace.patch_all()
, orddtrace.patch(aws_lambda=True)
, is set on user handler.
aws_lambda: Fix AttributeError raised when extracting context from arguments.
aws_lambda: Fix AttributeError raised when callable handlers are traced.
dynamic instrumentation: Fixed an issue with expressions in metric probes that prevented them from being evaluated.
Prevent exceptions when autoreloading modules that directly or indirectly import ddtrace with the iPython autoreload extension.
profiling: Corrects accounting of wall and CPU time for gevent tasks within the main Python thread.
profiling: Fixed an issue with the memory collector where a segmentation fault could occur during shutdown.
lib-injection: The ddtrace package is now provided via the Docker image rather than relying on a run-time
pip install
. This solves issues like containers blocking network requests, installation overhead during application startup, permissions issues with the install.
v1.11.0#
Deprecation Notes#
ASM: Several deprecated ASM constants that were added to the public API will be removed. This should not affect existing code as they were meant for private use only.
New Features#
tracing: Adds support for 128 bit trace ids. To generate and propagate 128 bit trace ids using Datadog distributed tracing headers set the following configuration:
DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=True
. Support for B3 and W3C distributed tracing headers will be added in a future change.
aiohttp: Add missing component meta tag to aiohttp server spans.
redis: Adds tracing support for redis.cluster.RedisCluster.
celery: Adds automatic tracing of the
celery.beat
scheduling service to thecelery
integration.
kafka: Adds instrumentation support for
confluent-kafka>=1.7
. See theconfluent-kafka<https://ddtrace.readthedocs.io/en/stable/integrations.html#kafka>
documentation for more information.
dynamic instrumentation: introduced support for dynamic span probes.
Adds source code integration with setuptools build metadata. This enables traces and profiles to be automatically tagged with git metadata to track deployments in Datadog.
Bug Fixes#
tracing: This fix resolves an issue where making a sampling decision before the
env
span tag had been set caused sample rate data from the Datadog Agent to be ignored.
ASM: make
track_custom_event()
also setappsec.events.<custom_event>.track
which was missing.
django: Fixes an issue where
http.route
was only set ifuse_handler_resource_format
anduse_legacy_resource_format
were set toFalse
.
tracing: This fix resolves an issue where a very long string as a span attribute would cause that span not to be delivered. It replaces string span attributes larger than DD_TRACE_WRITER_BUFFER_SIZE_BYTES (which as of this version defaults to 8388608) with a small string containing debug information and not containing any of the original attribute string.
ASM: Resolves installation issues with compiling native code on Windows and unknown platforms.
aws_lambda: Fixes a
RecursionError
which is raised when aws lambda signal handlers are wrapped infinitely. This caused lambdas to crash on startup.
botocore: Fix TypeError raised by injecting trace context into Kinesis messages.
dynamic instrumentation: Fix a bug where the dynamic instrumentation would stop injecting function probes after the first failed one.
dynamic instrumentation: This change fixes a bug whereby probes that have been disabled/removed from the front-end would not be removed by the client library.
futures: Resolves an issue that prevents tasks from being submitted to a thread pool executor when gevent is used (e.g. as a worker class for gunicorn or celery).
propagation: This fix resolves an issue where previously W3C tracestate propagation could not handle whitespace. With this fix whitespace is now removed for incoming and outgoing requests.
httplib: Fixes an issue with patching of http client upon import
Ensure DD_REMOTE_CONFIGURATION_ENABLED environment variable disables remote config if set to False
Other Changes#
aws_lambda: Updates how DD_APM_FLUSH_DEADLINE_MILLISECONDS is used. Previously, we would set the deadline as the environment variable value, if set. Now, when the remaining time in an AWS Lambda invocation is less than DD_APM_FLUSH_DEADLINE_MILLISECONDS, the tracer will attempt to submit the current active spans and all finished spans. the value in the environment variable is used to subtract from the deadline. The default is still 100ms.
v1.10.0rc2#
Bug Fixes#
ASM: Solve some corner cases where a Flask blocking request would fail because headers would be already sent.
ASM: Solve the content-type not always being correct in blocking responses.
ASM: Ensure the blocking responses have the following tags: http.url, http.query_string, http.useragent, http.method, http.response.headers.content-type and http.response.headers.content-length.
ASM: fix memory leaks and memory corruption in the interface between ASM and the WAF library
psycopg2: Fixes a bug with DSN parsing integration.
Other Changes#
remote_config: Change the level of remote config startup logs to debug.
v1.9.1#
Deprecation Notes#
gevent:
DD_GEVENT_PATCH_ALL
is deprecated and will be removed in the next major version. Gevent compatibility is now automatic and does not require extra configuration when running withddtrace-run
. If not usingddtrace-run
, please importddtrace.auto
before callinggevent.monkey.patch_all()
.
Bug Fixes#
aws_lambda: Resolves an exception not being handled, which occurs when no root span is found before a lambda times out.
gevent: This fix resolves an incompatibility between ddtrace and gevent that caused threads to hang in certain configurations, for example the profiler running in a gunicorn application’s gevent worker process.
Other Changes#
- ASM: The list of headers for retrieving the IP when Application Security Management is enabled or the
DD_TRACE_CLIENT_IP_ENABLED environment variable is set has been updated. “Via” has been removed as it rarely contains IP data and some common vendor headers have been added. You can also set the environment variable DD_TRACE_CLIENT_IP_HEADER to always retrieve the IP from the header specified as the value.
v1.10.0rc1#
Prelude#
Application Security Management (ASM) has added Django support for blocking malicious users using one click within Datadog.
Note
One click blocking for ASM is currently in beta.
Deprecation Notes#
dbapi:
ddtrace.ext.mongo.ROWS
is deprecated. Useddtrace.ext.db.ROWCOUNT
instead.
New Features#
starlette: Add http.route tag to
starlette.request
spans.
fastapi: Add http.route tag to
fastapi.request
spans.
ASM: Add support for one click blocking of user ids with the Django framework using Remote Configuration Management.
ASM: This introduces the “suspicious request blocking” feature for Django and Flask.
Known Issues#
ASM: There is a known issue with the flask support for any rule blocking on
server.request.path_params
. The request will be correctly blocked but the client application will be receiving and processing the suspicious request. Possible workaround: useserver.request.uri.raw
instead, if you want the request to be blocked before entering the flask application.
Bug Fixes#
dbapi: The dbapi integration no longer assumes that a cursor object will have a rowcount as not all database drivers implement rowcount.
dbm: Support sql queries with the type
byte
.
- elasticsearch: Omit large
elasticsearch.body
tag values that are greater than 25000 characters to prevent traces from being too large to send.
- elasticsearch: Omit large
aws_lambda: This fix resolves an issue where existing signals were wrapped multiple times.
profiling: Handles a race condition on process shutdown that would cause an error about a module not being started to occasionally appear in the logs.
Fix for KeyError exceptions when when ASM_FEATURES (1-click activation) disabled all ASM products. This could cause 1-click activation to work incorrectly in some cases.
v1.9.0rc1#
Prelude#
Application Security Management (ASM) has added Django support for blocking malicious IPs using one click within Datadog.
Note
One click blocking for ASM is currently in beta.
Application Security Management (ASM) has added Flask support for blocking malicious IPs using one click within Datadog.
Note
One click blocking for ASM is currently in beta.
Deprecation Notes#
grpc: Deprecates
ddtrace.contrib.grpc.constants.GRPC_PORT_KEY
. Useddtrace.ext.net.TARGET_PORT
instead.
dbapi:
ddtrace.ext.sql.ROWS
is deprecated. Useddtrace.ext.db.ROWCOUNT
instead.
cassandra:
ddtrace.ext.cassandra.ROW_COUNT
is deprecated. Useddtrace.ext.db.ROWCOUNT
instead.
New Features#
Enable traces to be sent before an impending timeout for
datadog_lambda>=4.66.0
. UseDD_APM_FLUSH_DEADLINE
to override the default flush deadline. The default is the AWS Lambda function configured timeout limit.
debugger: Add dynamic log probes to that generate a log message and optionally capture local variables, return value and exceptions
tracing: Add support for enabling collecting of HTTP request client IP addresses as the
http.client_ip
span tag. You can set theDD_TRACE_CLIENT_IP_ENABLED
environment variable totrue
to enable. This feature is disabled by default.
ASM: add support for one click blocking of IPs with the Django framework using Remote Configuration Management.
- ASM: add support for one click blocking of IPs with the Flask framework using
Remote Configuration Management.
ASM: also fetch loopback IPs if client IP fetching is enabled (either via ASM or DD_TRACE_CLIENT_IP_ENABLED).
ASM: Enable ability to remotely activate and configure ASM features. To enable, check the Python Security page in your account. Note that this is a beta feature.
profiling: Collects endpoint invocation counts.
dynamic instrumentation: Python 3.11 is now supported.
graphene: Adds support for Python 3.11.
graphql: Adds support for Python 3.11.
httpx: Add support for
httpx<0.14.0,>=0.9.0
.
- tracer/span: Add
Span.finish_with_ancestors
method to enable the abrupt finishing of a trace in cases where the trace or application must be immediately terminated.
- tracer/span: Add
Known Issues#
remote config: There is a known issue with remote configuration management (RCM) when paired with gevent which can cause child processes to deadlock. If you are experiencing issues, we recommend disabling RCM with
DD_REMOTE_CONFIGURATION_ENABLED=false
. Note, this will disable one click activation for ASM.
gunicorn: ddtrace-run does not work with gunicorn. To instrument a gunicorn application, follow the instructions here.
Bug Fixes#
fastapi: Previously, custom fastapi middlewares configured after application startup were not traced. This fix ensures that all fastapi middlewares are captured in the fastapi.request span.
tracing: Pads trace_id and span_ids in b3 headers to have a minimum length of 16.
Fix full stacktrace being sent to the log on remote config connection errors.
httpx: Only patch
httpx.AsyncClient
forhttpx>=0.11.0
.
tracing: This fix resolves an issue with the encoding of traces when using the v0.5 API version with the Python optimization option flag
-O
or thePYTHONOPTIMIZE
environment variable.
pylons: This fix resolves an issue where
str.decode
could cause critical unicode decode errors when ASM is enabled. ASM is disabled by default.
gevent: This fix resolves incompatibility under 3.8>=Python<=3.10 between
ddtrace-run
and applications that depend ongevent
, for examplegunicorn
servers. It accomplishes this by keeping copies that have not been monkey patched bygevent
of most modules used byddtrace
. This “module cloning” logic can be controlled by the environment variableDD_UNLOAD_MODULES_FROM_SITECUSTOMIZE
. Valid values for this variable are “1”, “0”, and “auto”. “1” tellsddtrace
to run its module cloning logic unconditionally, “0” tells it never to run that logic, and “auto” tells it to run module cloning logic only ifgevent
is accessible from the application’s runtime. The default value is “0”.
- lib-injection: Use package versions published to PyPI to install the
library. Formerly the published image was installing the package from source using the tagged commit SHA which resulted in slow and potentially failing installs.
profiler: Handles potential
AttributeErrors
which would arise while collecting frames during stack unwinding in Python 3.11.
remote config: ensure proper validation of responses from the agent.
v1.8.0rc1#
Upgrade Notes#
ASM: libddwaf upgraded to version 1.6.1 using a new library loading mechanism
profiling: upgrades the profiler to support the
v2.4
backend API for profile uploads, using a new request format.
Deprecation Notes#
DD_REMOTECONFIG_POLL_SECONDS
environment variable is deprecated and will be removed in v2.0. Please useDD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS
instead.
New Features#
CI Visibility: Add support for CI provider buddy.works
The component tag has been added for all auto-instrumented spans. The value of the component tag is equal to the name of the integration that produced the span.
tracing: Adds support for IPv6 agent hostnames for DD_AGENT_HOST.
elasticsearch: Update
elasticsearch
integration to add support foropensearch-py
. See the elasticsearch documentation for more information.
ASM: one click activation enabled by default using Remote Configuration Management (RCM). Set
DD_REMOTE_CONFIGURATION_ENABLED=false
to disable this feature.
- ASM: New Application Security Events Tracking API, starting with the functions
track_user_login_success_event
and track_user_login_failure_event
for tracking user logins (it will also internally callset_user
) andtrack_custom_event
for any custom events. You can find these functions in theddtrace.appsec.trace_utils
module. Calling these functions will create new tags under theappsec.events
namespace (appsec.events.user.login
for logins) allowing you to track these events with Datadog. In the future this will be used to provide protection against account takeover attacks (ATO). Public documentation will be online soon.
- ASM: New Application Security Events Tracking API, starting with the functions
celery: Enhances context tags containing dictionaries so that their contents are sent as individual tags (issue #4771).
tornado: Support custom error codes: https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#custom-error-codes.
CI Visibility: Support reliably linking tests to the pipeline that executed them.
Known Issues#
profiling: There is currently a known performance regression issue with the profiler’s code provenance feature. Note that this feature is disabled by default and will only be enabled if
DD_PROFILING_ENABLE_CODE_PROVENANCE
is set to true.
Bug Fixes#
This fix improves a cryptic error message encountered during some
pip install ddtrace
runs under pip versions <18.
dynamic instrumentation: remove unnecessary log line from application start up
This fix removes unintended url parts in the
http.url
tag.
botocore: Before this change, the botocore integration stripped newlines from the JSON string encoded in the data blob of Amazon Kinesis records. This change includes a terminating newline if it is present in the decoded data.
profiling: This fix resolves an issue in Python 3.11 where a PyFrameObject strong reference count was not properly decremented in the stack collector.
telemetry: This fix resolves an issue when we try to fetch
platform.libc_ver()
on an unsupported system.
Fix for ValueError when
@
is not present in network location but other part of the url.
Other Changes#
profiler: CPU overhead reduction.
v1.7.0rc1#
Prelude#
Initial library support has been added for Python 3.11.
Note
Continuous Profiler and Dynamic Instrumentation are not yet compatible and must be disabled in order to use the library with Python 3.11. Support for them will be added in a future release. To track the status, see the Support Python 3.11 issue on GitHub.
Upgrade Notes#
The default propagation style configuration changes to
DD_TRACE_PROPAGATION_STYLE=tracecontext,datadog
. To only support Datadog propagation and retain the existing default behavior, setDD_TRACE_PROPAGATION_STYLE=datadog
.
tracer: support for Datadog Agent v5 has been dropped. Datadog Agent v5 is no longer supported since ddtrace==1.0.0. See https://ddtrace.readthedocs.io/en/v1.0.0/versioning.html#release-support for the version support.
Python 3.11: Continuous Profiler and Dynamic Instrumentation must be disabled as they do not current support Python 3.11.
The configured styles in
DD_TRACE_PROPAGATION_STYLE_EXTRACT
are now evaluated in order to specification. To keep the previous fixed evaluation order, set:DD_TRACE_PROPAGATION_STYLE_EXTRACT=datadog,b3,b3 single header
.
tracing: upgrades the default trace API version to
v0.5
for non-Windows systems. Thev0.5
trace API version generates smaller payloads, thus increasing the throughput to the Datadog agent especially with larger traces.
tracing: configuring the
v0.5
trace API version on Windows machines will raise aRuntimeError
due to known compatibility issues. Please see https://github.com/DataDog/dd-trace-py/issues/4829 for more details.
Deprecation Notes#
propagation: Configuration of propagation style with
DD_TRACE_PROPAGATION_STYLE=b3
is deprecated and will be removed in version 2.0.0. Please use the newly addedDD_TRACE_PROPAGATION_STYLE=b3multi
instead.
aws: The boto, botocore and aiobotocore integrations no longer include all API parameters by default. To retain the deprecated behavior, set the environment variable
DD_AWS_TAG_ALL_PARAMS=1
. The deprecated behavior and environment variable will be removed in v2.0.0.
New Features#
django: add configuration option to allow a resource format like {method} {handler}.{url_name} in projects with Django <2.2.0
django: Adds the
DD_DJANGO_INCLUDE_USER_NAME
option to toggle whether the integration sets thedjango.user.name
tag.
Added environment variable
DD_TRACE_PROPAGATION_STYLE
to configure both injection and extraction propagation styles. The configured styles can be overridden with environment variablesDD_TRACE_PROPAGATION_STYLE_INJECT
andDD_TRACE_PROPAGATION_STYLE_EXTRACT
.
tracing: This introduces
none
as a supported propagator for trace context extraction and injection. Whennone
is the only propagator listed, the corresponding trace context operation is disabled. If there are other propagators in the inject or extract list, the none propagator has no effect. For exampleDD_TRACE_PROPAGATION_STYLE=none
ASM: now http.client_ip and network.client.ip will only be collected if ASM is enabled.
tracing: Adds support for W3C Trace Context propagation style for distributed tracing. The
traceparent
andtracestate
HTTP headers are enabled by default for all incoming and outgoing HTTP request headers. The Datadog propagation style continue to be enabled by default.
flask: Adds support for streamed responses. Note that two additional spans:
flask.application
andflask.response
will be generated.
profiling: Adds support for Python 3.11.
tracer: added support for Python 3.11.
Bug Fixes#
ASGI: response headers are correctly processed instead of ignored
Fix issue with
attrs
andcontextlib2
version constraints for Python 2.7.
CGroup file parsing was fixed to correctly parse container UUID for PCF containers.
ASM: Do not raise exceptions when failing to parse XML request body.
ASM: fix a body read problem on some corner case where don’t passing the content length makes wsgi.input.read() blocks.
aws: We are reducing the number of API parameters that the boto, botocore and aiobotocore integrations collect as span tags by default. This change limits span tags to a narrow set of parameters for specific AWS APIs using standard tag names. To opt out of the new default behavior and collect no API parameters, set the environment variable
DD_AWS_TAG_NO_PARAMS=1
. To retain the deprecated behavior and collect all API parameters, set the environment variableDD_AWS_TAG_ALL_PARAMS=1
.
tracing: make
ddtrace.context.Context
serializable which fixes distributed tracing across processes.
django: avoid
SynchronousOnlyOperation
when failing to retrieve user information.
Remove
forbiddenfruit
as dependency and rollbackwrapt
changes whereforbiddenfruit
was called. IAST: Patch builtins only when IAST is enabled.
httpx: Fixes an incompatibility from
httpx==0.23.1
when theURL.raw
property is not available.
Fix error in patching functions.
forbiddenfruit
package has conflicts with some libraries such asasynctest
. This conflict raisesAttributeError
exception. See issue #4484.
tracer: This fix resolves an issue where the rate limiter used for span and trace sampling rules did not reset the time since last call properly if the rate limiter already had max tokens. This fix resets the time since last call always, which leads to more accurate rate limiting.
Ensure that worker threads that run on start-up are recreated at the right time after fork on Python < 3.7.
tracing: This fix resolves an issue where the
DD_SERVICE_MAPPING
mapped service names were not used when updating span metadata with theDD_VERSION
set version string.
wsgi: This fix resolves an issue where
BaseException
raised in a WSGI application caused spans to not be submitted.
library injection: Pin the library version in the library injection image. Prior, the latest version of
ddtrace
would always be installed, regardless of the image version.
Fix error in the agent response payload when the user disabled ASM in a dashboard using 1-click Remote Configuration.
flask: add support for flask v2.3. Remove deprecated usages of
flask._app_ctx_stack
andflask._request_ctx_stack
.
The specification of
DD_TRACE_PROPAGATION_STYLE_EXTRACT
now respects the configured styles evaluation order. The evaluation order had previously been fixed and so the configured order was ignored.
tracing: Ensures that encoding errors due to wrong span tag types will be logged. Previously, if non-text span tags were set, this resulted in v0.5 encoding errors to be output to
stderr
instead of to a logger.
Other Changes#
Kubernetes library injection: run commands as non-root user.
tracing: The value of
ddtrace.constants.PID
has been changed fromsystem.pid
toprocess_id
. All spans will now use the metric tag ofprocess_id
instead.
tracing: The exception logged for writing errors no longer includes a long, unhelpful stack trace. The message now also includes the number of traces dropped and the number of retries attempted.
v1.6.0rc1#
Prelude#
Application Security Management (ASM) has added support for preventing attacks by blocking malicious IPs using one click within Datadog.
Note
One click activation for ASM is currently in beta.
Dynamic instrumentation allows instrumenting a running service dynamically to extract runtime information that could be useful for, e.g., debugging purposes, or to add extra metrics without having to make code changes and re-deploy the service. See https://ddtrace.readthedocs.io/en/stable/configuration.html for more details.
Upgrade Notes#
Pin [attrs](https://pypi.org/project/attrs/) dependency to version
>=20
due to incompatibility with [cattrs](https://pypi.org/project/cattrs/) version22.1.0
.
Use
Span.set_tag_str()
instead ofSpan.set_tag()
when the tag value is a text type as a performance optimizations in manual instrumentation.
New Features#
ASM: add support for one click activation using Remote Configuration Management (RCM). Set
DD_REMOTE_CONFIGURATION_ENABLED=true
to enable this feature.
ASM: ip address collection will be enabled if not explicitly disabled and appsec is enabled.
tracing: HTTP query string tagged by default in http.url tag (sensitive query strings will be obfuscated).
Django: set usr.id tag by default if request.user is authenticated.
Introduced the public interface for the dynamic instrumentation service. See https://ddtrace.readthedocs.io/en/stable/configuration.html for more details.
Add
Span.set_tag_str()
as an alternative to the overloaded functionality ofSpan.set_tag()
when the value can be coerced to unicode text.
Enable telemetry collection when tracing is enabled.
Bug Fixes#
ASM: only report actor.ip on attack.
aioredis: added exception handling for CancelledError in the aioredis integration.
CI Visibility: fixed AppVeyor integration not extracting the full commit message.
Add iterable methods on TracedCursor. Previously these were not present and would cause iterable usage of cursors in DB API integrations to fail.
Fix parsing of the
DD_TAGS
environment variable value to include support for values with colons (e.g. URLs). Also fixed the parsing of invalid tags that begin with a space (e.g.DD_TAGS=" key:val"
will now produce a tag with labelkey
, instead of `` key``, and valueval
).
opentracing: don’t raise an exception when distributed tracing headers are not present when attempting to extract.
sqlite3: fix error when using
connection.backup
method.
Change dependency from
backport_ipaddress` to ``ipaddress
. Only installipaddress
for Python < 3.7.
gevent: disable gevent after fork hook which could result in a performance regression.
profiling: restart automatically on all Python versions.
profiling: fixes an issue with Gunicorn child processes not storing profiling events.
wsgi: when using more than one nested wsgi traced middleware in the same app ensure wsgi spans have the correct parenting.
Other Changes#
tracing: add http.route tag to root span for Flask framework.
v1.5.0rc1#
New Features#
graphene: add support for
graphene>=2
. See the graphql documentation for more information.
Add support for aiobotocore 1.x and 2.x.
ASM: add user information to traces.
ASM: collect http client_ip.
ASM: configure the sensitive data obfuscator.
ASM: Detect attacks on Pylons body.
ASM: propagate user id.
ASM: Support In-App WAF metrics report.
Collect user agent in normalized span tag
http.useragent
.
ASM: Detect attacks on XML body (for Django, Pylons and Flask).
Adds support for Lambda profiling, which can be enabled by starting the profiler outside of the handler (on cold start).
profiler: collect and export the class name for the wall time, CPU time and lock profiles, when available.
add DD_PYMONGO_SERVICE configuration
ASM: Redact sensitive query strings if sent in http.url.
redis: track the connection client_name.
rediscluster: add service name configuration with
DD_REDISCLUSTER_SERVICE
snowflake: add snowflake query id tag to
sql.query
span
Bug Fixes#
aiohttp_jinja2: use
app_key
to look up templates.
ASM: (flask) avoid json decode error while parsing request body.
ASM: fix Python 2 error reading WAF rules.
ASM: reset wsgi input after reading.
tracing: fix handling of unicode
_dd.origin
tag for Python 2
tracing: fix nested web frameworks re-extracting and activating HTTP context propagation headers.
requests: fix split-by-domain service name when multiple
@
signs are present in the url
profiling: internal use of RLock needs to ensure original threading locks are used rather than gevent threading lock. Because of an indirection in the initialization of the original RLock, we end up getting an underlying gevent lock. We work around this behavior with gevent by creating a patched RLock for use internally.
profiler: Remove lock for data structure linking threads to spans to avoid deadlocks with the trade-off of correctness of spans linked to threads by stack profiler at a given point in time.
profiling: fix a possible deadlock due to spans being activated unexpectedly.
v1.4.0rc1#
New Features#
graphql: add tracing for
graphql-core>2
. See the graphql documentation for more information.
ASM: Detect attacks on Django body.
ASM: Detect attacks on Flask request cookies
ASM: Detect attacks on Django request cookies
ASM: Detect attacks on Pylons HTTP query.
ASM: Detect attacks on Pylons request cookies
ASM: detect attacks on Pylons path parameters.
ASM: Report HTTP method on Pylons framework
ASM: Collect raw uri for Pylons framework.
AppSec: collect response headers
ASM: Detect attacks on Flask body.
ASM: Detect attacks on path parameters
The profiler now supports Windows.
The profiler now supports code provenance reporting. This can be enabled by using the
enable_code_provenance=True
argument to the profiler or by setting the environment variableDD_PROFILING_ENABLE_CODE_PROVENANCE
totrue
.
Bug Fixes#
flask: add support for
flask>=2.2.0
Fixed the environment variable used for log file size bytes to be
DD_TRACE_LOG_FILE_SIZE_BYTES
as documented.
jinja2: fix handling of template names which are not strings.
Fixed support for pytest-bdd 6.
Fixes cases where a pytest test parameter object string representation includes the
id()
of the object, causing the test fingerprint to constantly change across executions.
wsgi: ignore GeneratorExit Exception in wsgi.response spans
wsgi: ensures resource and http tags are always set on wsgi.request spans.
Other Changes#
- profiler: don’t initialize the
AsyncioLockCollector
unless asyncio is available. This prevents noisy logs messages from being emitted in Python 2.
- profiler: don’t initialize the
docs: Added troubleshooting section for missing error details in the root span of a trace.
v1.3.0rc1#
New Features#
internal: Add support for Datadog trace tag propagation
django: added
DD_DJANGO_INSTRUMENT_TEMPLATES=false
to allow tracing of Django template rendering.
internal: Add sampling mechanism trace tag
Add environment variables to write
ddtrace
logs to a file withDD_TRACE_LOG_FILE
,DD_TRACE_LOG_FILE_LEVEL
, andDD_TRACE_FILE_SIZE_BYTES
Adds pytest-bdd integration to show more details in CI Visibility product.
Bug Fixes#
starlette: Add back removed
aggregate_resources
feature.
fastapi: Add back removed
aggregate_resources
feature.
aiomysql: fix
AttributeError: __aenter__
when using cursors as context managers.
asgi, starlette, fastapi: Exclude background tasks duration from web request spans.
asgi: set the
http.url
tag using the hostname in the request header before defaulting to the hostname of the asgi server.
mypy: Avoid parsing redis asyncio files when type checking Python 2
starlette: Add back removed
ddtrace.contrib.starlette.get_resource
andddtrace.contrib.starlette.span_modifier
.
fastapi: Add back removed
ddtrace.contrib.fastapi.span_modifier
.
internal: fix exception raised for invalid values of
DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH
.
flask_caching: fix redis tagging after the v2.0 release.
redis: create default Pin on asyncio client. Not having a Pin was resulting in no traces being produced for the async redis client.
Other Changes#
perf: don’t encode default parent_id value.
profiling: add support for protobuf >=4.0.
v1.2.0rc1#
Upgrade Notes#
The profiler
asyncio_loop_policy
attribute has been renamed toasyncio_loop_policy_class
to accept a user-defined class. This guarantees the same asyncio loop policy class can be used process children.
New Features#
Add tracing support for
aiomysql>=0.1.0
.
Add support for
grpc.aio
.
botocore: allow defining error status codes for specific API operations.
See our botocore document for more information on how to enable this feature.
ciapp: detect code owners of PyTest tests
The memory profile collector can now entirely disabled with the
DD_PROFILING_MEMORY_ENABLED
environment variable.
psycopg2: add option to enable tracing
psycopg2.connect
method.See our psycopg documentation for more information.
Add asyncio support of redis ≥ 4.2.0
Bug Fixes#
Fixes deprecation warning for
asyncio.coroutine
decorator.
internal: normalize header names in ASM
profiling: implement
__aenter__
and__aexit__
methods onasyncio.Lock
wrapper.
tracing: fix issue with
ddtrace-run
having the wrong priority order of tracer host/port/url env variable configuration.
django,redis: fix unicode decode error when using unicode cache key on Python 2.7
fastapi/starlette: when using sub-apps, formerly a call to
/sub-app/hello/{name}
would give a resource name of/sub-app
. Now the full path/sub-app/hello/{name}
is used for the resource name.
sanic: Don’t send non-500s error traces.
pin protobuf to version
>=3,<4
due to incompatibility with version4.21
.
Fixes a performance issue with the profiler when used in an asyncio application.
The profiler now copy all user-provided attributes on fork.
pytest: Add note for disabling ddtrace plugin as workaround for side-effects
Set required header to indicate top level span computation is done in the client to the Datadog agent. This fixes an issue where spans were erroneously being marked as top level when partial flushing or in certain asynchronous applications.
The impact of this bug is the unintended computation of stats for non-top level spans.
Other Changes#
The default number of events kept by the profiler has been reduced to decreased CPU and memory overhead.
v1.1.0rc1#
Prelude#
The Datadog APM Python team is happy to announce the release of v1.0.0 of ddtrace. This release introduces a formal versioning policy that simplifies the public interface and defines a release version policy for backwards compatible and incompatible changes to the public interface.
The v1.0.0 release is an important milestone for the library as it has grown substantially in scope. The first commit to the library was made on June 20, 2016. Nearly sixty minor releases later, the library now includes over sixty integrations for libraries. And the library has expanded from Tracing to support the Continuous Profiler and CI Visibility.
Important
Before upgrading to v1.0.0, we recommend users install
ddtrace>=0.60.0,<1.0.0
and enable deprecation warnings. All removals to
the library interface and environment variables were deprecated on 0.x
branch. Consult Upgrade 0.x for recommendations
on migrating from the 0.x release branch.
Note
The changes to environment variables apply only to the configuration of the ddtrace library and not the Datadog Agent.
Upgrading summary#
Functionality changes#
The default logging configuration functionality of ddtrace-run
has changed to address conflicts with application logging configuration. See note on the new default behavior and note on deprecation for future removal.
Removed legacy environment variables#
These environment variables have been removed. In all cases the same functionality is provided by other environment variables and replacements are provided as recommended actions for upgrading.
Removed legacy tracing interfaces#
These methods and module attributes have been removed. Where the same functionality is provided by a different public method or module attribute, a recommended action is provided for upgrading. In a few limited cases, because the interface was no longer used or had been moved to the internal interface, it was removed and so no action is provided for upgrading.
Module |
Method/Attribute |
Note |
---|---|---|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
||
|
||
|
||
|
||
|
||
|
||
|
|
|
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
|
|
|
|
|
|
|
Removed legacy integration tracing#
These tracing functions in integrations were no longer used for automatic instrumentation so have been removed. Any manual instrumentation code in an application will need to be replaced with ddtrace.patch_all()
or ddtrace.patch()
when upgrading.
Module |
Function/Class |
|
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Removed deprecated modules#
These modules have been removed. Many were moved to the internal interface as they were not intended to be used as part of the public interface. In these cases, no action is provided for upgrading. In a few cases, other modules are provided as alternatives to maintain functionality. See the notes for more information.
Upgrade Notes#
The deprecated attribute
ddtrace.Sampler.default_sampler
is removed.
Spans started after
Tracer.shutdown()
has been called will no longer be sent to the Datadog Agent.
Default value of
DD_CALL_BASIC_CONFIG
was updated fromTrue
toFalse
. Calllogging.basicConfig()
to configure logging in your application.
aiohttp_jinja2: use
patch(aiohttp_jinja2=True)
instead ofpatch(aiohttp=True)
for enabling/disabling the integration.
ddtrace.settings.Config.HTTPServerConfig
is removed.
cassandra:
get_traced_cassandra
is removed. Useddtrace.patch(cassandra=True)
orddtrace.patch_all()
instead.
celery:
ddtrace.contrib.celery.patch_task
is removed. Useddtrace.patch(celery=True)
orddtrace.patch_all()
instead.
celery:
ddtrace.contrib.celery.unpatch_task
is removed. Useddtrace.contrib.celery.unpatch()
instead.
ddrace.context.Context.clone
is removed. This is no longer needed since the tracer now supports asynchronous frameworks out of the box.
ddtrace.constants.FILTERS_KEY
is removed.
ddtrace.constants.NUMERIC_TAGS
is removed.
ddtrace.constants.LOG_SPAN_KEY
is removed.
The deprecated module
ddtrace.contrib.util
is removed.
The deprecated module
ddtrace.compat
is removed.
The deprecated module
ddtrace.encoding
is removed.
The deprecated modules
ddtrace.http
andddtrace.http.headers
are removed. Useddtrace.contrib.trace_utils.set_http_meta
to store request and response headers on a span.
Remove deprecated
ddtrace.install_excepthook
.
Remove deprecated
ddtrace.uninstall_excepthook
.
The deprecated module
ddtrace.monkey
is removed. Useddtrace.patch
orddtrace.patch_all
instead.
The deprecated module
ddtrace.propagation.utils
is removed.
- The deprecated module
ddtrace.utils
and its submodules are removed: ddtrace.utils.attr
ddtrace.utils.attrdict
ddtrace.utils.cache
ddtrace.utils.config
ddtrace.utils.deprecation
ddtrace.utils.formats
ddtrace.utils.http
ddtrace.utils.importlib
ddtrace.utils.time
ddtrace.utils.version
ddtrace.utils.wrappers
- The deprecated module
ddtrace.Tracer.sampler
is removed.
ddtrace.Tracer.priority_sampler
is removed.
ddtrace.Tracer.log
was removed.
The deprecated module
ddtrace.ext.errors
is removed. Use theddtrace.constants
module instead:from ddtrace.constants import ERROR_MSG from ddtrace.constants import ERROR_STACK from ddtrace.constants import ERROR_TYPE
The deprecated module
ddtrace.ext.priority
is removed. Use theddtrace.constants
module instead for setting sampling priority tags:from ddtrace.constants import USER_KEEP from ddtrace.constants import USER_REJECT
The deprecated module
ddtrace.ext.system
is removed. Useddtrace.constants.PID
instead.
The deprecated method
ddtrace.helpers.get_correlation_ids
is removed. Useddtrace.Tracer.get_log_correlation_context()
instead.
The legacy environment variables
DD_SERVICE_NAME
andDATADOG_SERVICE_NAME
are removed. UseDD_SERVICE
instead.
mongoengine: The deprecated method
ddtrace.contrib.mongoengine.trace_mongoengine
is removed. Useddtrace.patch(mongoengine=True)
orddtrace.patch()
instead.
mysql: The deprecated method
ddtrace.contrib.mysql.get_traced_mysql_connection
is removed. Useddtrace.patch(mysql=True)
orddtrace.patch_all()
instead.
Pin.app
is removed.
Pin.app_type
is removed.
psycopg:
ddtrace.contrib.psycopg.connection_factory
is removed. Useddtrace.patch(psycopg=True)
orddtrace.patch_all()
instead.
requests: The legacy distributed tracing configuration is removed. Use ddtrace.config.requests[‘distributed_tracing’] instead.
ddtrace.Span.meta
is removed. Useddtrace.Span.get_tag()
andddtrace.Span.set_tag()
instead.
ddtrace.Span.metrics
is removed. Useddtrace.Span.get_metric()
andddtrace.Span.set_metric()
instead.
ddtrace.Span.pprint
is removed.
ddtrace.Span.set_meta
is removed. Useddtrace.Span.set_tag()
instead.
ddtrace.Span.set_metas
is removed. Useddtrace.Span.set_tags()
instead.
Span.to_dict
is removed.
Span.tracer
is removed.
The deprecated tracer argument is removed from
ddtrace.Span.__init__()
.
sqlite3:
ddtrace.contrib.sqlite3.connection_factory
is removed. Useddtrace.patch(sqlite3=True)
orddtrace.patch_all()
instead.
Remove deprecated attribute
ddtrace.Tracer.debug_logging
. Set the logging level for theddtrace.tracer
logger instead:import logging log = logging.getLogger("ddtrace.tracer") log.setLevel(logging.DEBUG)
ddtrace.Tracer.__call__
is removed.
ddtrace.Tracer.global_excepthook
is removed.
ddtrace.Tracer.get_call_context
is removed. Useddtrace.Tracer.current_trace_context()
instead.
ddtrace.Tracer.set_service_info
is removed.
ddtrace.Tracer.writer
is removed. To force flushing of buffered traces to the agent, useddtrace.Tracer.flush()
instead.
ddtrace.warnings.DDTraceDeprecationWarning
is removed.
DD_TRACE_RAISE_DEPRECATIONWARNING
environment variable is removed.
The environment variables prefixed with
DATADOG_
are removed. Use environment variables prefixed withDD_
instead.
The environment variable
DD_LOGGING_RATE_LIMIT
is removed. UseDD_TRACE_LOGGING_RATE
instead.
The environment variable
DD_TRACER_PARTIAL_FLUSH_ENABLED
is removed. UseDD_TRACE_PARTIAL_FLUSH_ENABLED
instead.
The environment variable
DD_TRACER_PARTIAL_FLUSH_MIN_SPANS
is removed. UseDD_TRACE_PARTIAL_FLUSH_MIN_SPANS
instead.
ddtrace.util
is removed.
ddtrace.ext.SpanTypes
is no longer anEnum
. UseSpanTypes.<TYPE>
instead ofSpanTypes.<TYPE>.value
.
Tracer.write
has been removed.
Removed deprecated middleware
ddtrace.contrib.flask.middleware.py:TraceMiddleware
. Useddtrace.patch_all()
orddtrace.patch()
instead.
Removed deprecated function
ddtrace.contrib.pymongo.patch.py:trace_mongo_client
. Useddtrace.patch_all()
orddtrace.patch()
instead.
Removed deprecated function
ddtrace.contrib.pymysql.tracers.py:get_traced_pymysql_connection
. Useddtrace.patch_all()
orddtrace.patch()
instead.
Removed deprecated function
ddtrace.contrib.redis.tracers.py:get_traced_redis
. Useddtrace.patch_all()
orddtrace.patch()
instead.
Removed deprecated function
ddtrace.contrib.redis.tracers.py:get_traced_redis_from
. Useddtrace.patch_all()
orddtrace.patch()
instead.
Deprecation Notes#
DD_CALL_BASIC_CONFIG
is deprecated.
New Features#
Add
Span.get_tags
andSpan.get_metrics
.
aiohttp: add client integration. This integration traces requests made using the aiohttp client and includes support for distributed tracing. See the documentation for more information.
aiohttp_jinja2: move into new integration. Formerly the aiohttp_jinja2 instrumentation was enabled using the aiohttp integration. Use
patch(aiohttp_jinja2=True)
instead ofpatch(aiohttp=True)
. To support legacy behaviorpatch(aiohttp=True)
will still enable aiohttp_jinja2.
asyncpg: add integration supporting v0.18.0 and above. See the docs for more information.
fastapi: add support for tracing
fastapi.routing.serialize_response
.This will give an insight into how much time is spent calling
jsonable_encoder
within a given request. This does not provide visibility into how long it takes forResponse.render
/json.dumps
.
Add support to reuse HTTP connections when sending trace payloads to the agent. This feature is disabled by default. Set
DD_TRACE_WRITER_REUSE_CONNECTIONS=true
to enable this feature.
MySQLdb: Added optional tracing for MySQLdb.connect, using the configuration option here.
The profiler now supports profiling
asyncio.Lock
objects.
Add support for injecting and extracting B3 propagation headers.
See DD_TRACE_PROPAGATION_STYLE_EXTRACT and DD_TRACE_PROPAGATION_STYLE_INJECT configuration documentation to enable.
Bug Fixes#
botocore: fix incorrect context propagation message attribute types for SNS. This addresses Datadog/serverless-plugin-datadog#232
aiohttp: fix issue causing
ddtrace.contrib.aiohttp_jinja2.patch
module to be imported instead of thepatch()
function.
botocore: omit
SecretBinary
andSecretString
from span metadata for calls to Secrets Manager.
tracing/internal: fix encoding of propagated internal tags.
Fix issue building
ddtrace
from source on macOS 12.
Fix issue building
ddtrace
for the Pyston Python implementation by not building the_memalloc
extension anymore when using Pyston.
tracer.get_log_correlation_context()
: use active context in addition toactive span. Formerly just the span was used and this would break cross execution log correlation as a context object is used for the propagation.
- opentracer: update
set_tag
andset_operation_name
to return a reference to the span to match the OpenTracing spec.
- opentracer: update
The CPU profiler now reports the main thread CPU usage even when asyncio tasks are running.
Fixes wrong numbers of memory allocation being reported in the memory profiler.
pymongo: fix
write_command
being patched with the wrong method signature.
Other Changes#
tracing/internal: disable Datadog internal tag propagation
v0.59.0rc1#
Deprecation Notes#
ddtrace.constants.FILTERS_KEY
is deprecated. Usesettings={"FILTERS": ...}
instead when callingtracer.configure
.
ddtrace.constants.NUMERIC_TAGS
is deprecated.
ddtrace.constants.LOG_SPAN_KEY
is deprecated.
Pin.app
is deprecated.
ddtrace.Span.set_meta()
is deprecated. Useddtrace.Span.set_tag()
instead.
ddtrace.Span.set_metas()
is deprecated. Useddtrace.Span.set_tags()
instead.
ddtrace.Span.metrics
is deprecated. Useddtrace.Span.get_metric()
andddtrace.Span.set_metric()
instead.
ddtrace.Span.tracer
is deprecated.
ddtrace.Tracer.log
is deprecated. Useddtrace.tracer.log
instead.
ddtrace.Tracer.sampler
is deprecated.
ddtrace.Tracer.priority_sampler
is deprecated.
ddtrace.Tracer.tags
is deprecated. Use the environment variable DD_TAGS to set the global tags instead.
New Features#
The profiler now reports asyncio tasks as part as the task field in profiles. This is enabled by default by replacing the default asyncio loop policy. CPU time, wall time and threading lock times are supported.
Add tracing support for
sanic>=21.9.0
.
Bug Fixes#
Fix internal import of deprecated
ddtrace.utils
module.
Set correct service in logs correlation attributes when a span override the service.
Fixes import path to not reference
__init__
. This could otherwise be a problem formypy
.
flask: fix resource naming of request span when errors occur in middleware.
Fix issue when
httpx
service name isbytes
.
Fixes build issues on older MacOS versions by updating
libddwaf
to 1.0.18
pytest: fix unsafe access to xfail reason.
v0.58.0rc1#
Deprecation Notes#
HttpServerConfig is no longer part of the public API.
ddtrace.Span.meta
has been deprecated. Useddtrace.Span.get_tag()
andddtrace.Span.set_tag()
instead.
ddtrace.Span.pprint()
is deprecated and will be removed in v1.0.
ddtrace.Tracer.writer
is deprecated. To force flushing of buffered traces to the agent, useddtrace.Tracer.flush()
instead.
New Features#
botocore: add distributed tracing support for AWS EventBridge, AWS SNS & AWS Kinesis.
Only for CI Visibility (
pytest
integration): remove traces whose root span is not a test.
Bug Fixes#
Fix application crash on startup when using
channels >= 3.0
.
Fix parenting of Redis command spans when using aioredis 1.3. Redis spans should now be correctly attributed as child of any active parent spans.
Fixes incompatibility of wrapped aioredis pipelines in
async with
statements.
Fixes issue with aioredis when empty pool is not available and execute returns a coroutine instead of a future. When patch tries to add callback for the span using add_done_callback function it crashes because this function is only for futures.
Escape non-Unicode bytes when decoding aioredis args. This fixes a
UnicodeDecodeError
that can be thrown from the aioredis integration when interacting with binary-encoded data, as is done in channels-redis.
Ensure
gevent
is automatically patched.
grpc: ensure grpc.intercept_channel is unpatched properly
Fix JSON encoding error when a
bytes
string is used for span metadata.
Profiler raises a typing error when
Span.resource
is unicode on Python 2.7.
Fix a bug in the heap profiler that could be triggered if more than 2^16 memory items were freed during heap data collection.
Fix a possible bug in the heap memory profiler that could trigger an overflow when too many allocations were being tracked.
Fix an issue in the heap profiler where it would iterate on the wrong heap allocation tracker.
Pymongo instrumentation raises an AttributeError when
tracer.enabled == False
v0.57.0#
Deprecation Notes#
ddtrace.sampler.DatadogSampler.default_sampler
property is deprecated and will be removed in 1.0.
ddtrace.propagation.utils
has been deprecated and will be removed in version 1.0.
New Features#
Add new environment variables to enable/disable django database and cache instrumentation.
DD_DJANGO_INSTRUMENT_DATABASES
,DD_DJANGO_INSTRUMENT_CACHES
Add tracing support for the
aioredis
library. Version 1.3+ is fully supported.
Add django 4.0 support.
The profiler now automatically injects running greenlets as tasks into the main thread. They can be seen within the wall time profiles.
Bug Fixes#
The thread safety of the custom buffered encoder was fixed in order to eliminate a potential cause of decoding errors of trace payloads (missing trace data) in the agent.
Fix handling of Python exceptions during trace encoding. The tracer will no longer silently fail to encode invalid span data and instead log an exception.
Fix error when calling
concurrent.futures.ThreadPoolExecutor.submit
withfn
keyword argument.
Configure a writer thread in a child process after forking based on writer configuration from its parent process.
Only for CI Visibility (
pytest
integration): Fix calculation of pipeline URL for GitHub Actions.
v0.56.0rc3#
Deprecation Notes#
The
ddtrace.utils
module and all of its submodules have been copied over intoddtrace.internal
in an effort to internalize these modules. Their public counterparts will be removed entirely in version 1.0.0.
New Features#
Profiling now supports tracing greenlets with gevent version prior to 1.3.
The heap profiler is now enabled by default.
Add yaaredis ≥ 2.0.0 support.
Bug Fixes#
Allow the elasticsearch service name to be overridden using the integration config or the DD_SERVICE_MAPPING environment variable.
Fixes parsing of
botocore
env variables to ensure they are parsed as booleans.
Ensure tornado spans are marked as an error if the response status code is 500 <= x < 600.
v0.56.0rc2#
Bug Fixes#
Fix memory leak caused when the tracer is disabled.
v0.56.0rc1#
Upgrade Notes#
The aredis integration is now enabled by default.
Deprecation Notes#
The contents of
monkey.py
have been moved into_monkey.py
in an effort to internalize the module. Public methods have been imported back intomonkey.py
in order to retain compatibility, but monkey.py will be removed entirely in version 1.0.0.
Bug Fixes#
Fix Pyramid caller_package level issue which resulted in crashes when starting Pyramid applications. Level now left at default (2).
Set the correct package name in the Pyramid instrumentation. This should fix an issue where the incorrect package name was being used which would crash the application when trying to do relative imports within Pyramid (e.g. when including routes from a relative path).
v0.55.0rc1#
Upgrade Notes#
Instead of using error constants from
ddtrace.ext.errors
. Use constants fromddtrace.constants
module. For example:ddtrace.ext.errors.ERROR_MSG
->ddtrace.constants.ERROR_MSG
Instead of using priority constants from
ddtrace.ext.priority
. Use constants fromddtrace.constants
module. For Example::ddtrace.ext.priority.AUTO_KEEP
->ddtrace.constants.AUTO_KEEP
Instead of using system constants from
ddtrace.ext.system
. Use constants fromddtrace.constants
module. For Example::ddtrace.ext.system.PID
->ddtrace.constants.PID
Deprecation Notes#
Deprecate DATADOG_TRACE_AGENT_HOSTNAME, DATADOG_TRACE_AGENT_PORT, DATADOG_PRIORITY_SAMPLING, DATADOG_PATCH_MODULES in favor of their DD equivalents.
[Deprecated environment variable] | [Recommended environment variable]
For
DATADOG_TRACE_AGENT_HOSTNAME
, useDD_AGENT_HOST
For
DATADOG_TRACE_AGENT_PORT
useDD_AGENT_PORT
For
DATADOG_PRIORITY_SAMPLING
, follow ingestion controlsFor
DATADOG_PATCH_MODULES
, useDD_PATCH_MODULES
Moved
ddtrace.ext.errors
constants into theddtrace.constants
module.ddtrace.ext.errors
will be removed in v1.0. Shorthand error constant (MSG,TYPE,STACK) inddtrace.ext.errors
will be removed in v1.0. Functionget_traceback()
in ddtrace.ext.errors is now deprecated and will be removed v1.0.
Moved
ddtrace.ext.priority
constants intoddtrace.constants
module.
Moved
ddtrace.ext.system
constants intoddtrace.constants
module.
New Features#
Add aredis support >= 1.1.0
Add automatic unix domain socket detection for Dogstatsd. The expected path for the socket is
/var/run/datadog/dsd.socket
which if exists, will be used instead of the previous UDP default,udp://localhost:8125/
. To be used in conjunction withdogstatsd_socket
in yourdatadog.yaml
file, or theDD_DOGSTATSD_SOCKET
environment variable set on the Datadog agent.
Add new
DD_TRACE_SAMPLING_RULES
environment variable to override default sampling rules. For Example::DD_TRACE_SAMPLING_RULES='[{"sample_rate":0.5,"service":"my-service"}]'
Add support for snowflake-connector-python >= 2.0.0. Note that this integration is in beta and is not enabled by default. See the snowflake integration documentation for how to enable.
Only for CI Visibility (
pytest
integration): includepytest
version as a tag in the test span.
Added official support for Python 3.10
Only for CI Visibility (
pytest
integration): Extract stage and job name from environment data in Azure Pipelines.
Bug Fixes#
Fixes an issue where all Django function middleware will share the same resource name.
Fixed an issue with gevent worker processes that caused them to crash and stop.
Fixes exceptions raised when logging during tracer initialization when
DD_LOGS_INJECTION
is enabled.
The
ddtrace.utils.wrappers.unwrap
function now raises an error if trying to unwrap a non-wrapped object.
Only for CI Visibility (
pytest
integration): Fix extraction of branch in GitLab CI.
v0.54.0rc1#
Deprecation Notes#
Deprecate the DATADOG_ENV environment variable in favor of DD_ENV. The use of DD_ENV should follow Unified Service Tagging recommendations.
New Features#
Add automatic unix domain socket detection for traces. The expected path for the socket is
/var/run/datadog/apm.socket
which if exists, will be used instead of the previous http default,http://localhost:8126/
. To be used in conjunction withapm_config.receiver_socket
in yourdatadog.yaml
file, or theDD_APM_RECEIVER_SOCKET
environment variable set on the Datadog agent.
Update the –info command to be easier to read and provide more helpful information.
Add support for
DD_PROFILING_ENDPOINT_COLLECTION_ENABLED
env variable to disable endpoint name collection in profiler.
Add rq integration.
Tag traces with HTTP headers specified on the
DD_TRACE_HEADER_TAGS
environment variable. Value must be either comma or space separated. e.g.Host:http.host,User-Agent,http.user_agent
orreferer:http.referer Content-Type:http.content_type
.
Bug Fixes#
pytest: report exception details directly instead of through a RuntimeWarning exception.
Fixed the support for Celery workers that fork sub-processes with Python 3.6 and earlier versions.
Fix the reporting of the allocated memory and the number of allocations in the profiler.
Fixes cases in which the
test.status
tag of a test span frompytest
would be missing becausepytest_runtest_makereport
hook is not run, like whenpytest
has an internal error.
Pin
protobuf
version to<3.18
for Python <=3.5 due to support being dropped.
Make sure that correct endpoint name collected for profiling.
Other Changes#
Added runtime metrics status and sampling rules to start-up logs.
v0.53.0rc1#
Upgrade Notes#
Replace DD_TRACER_PARTIAL_FLUSH_ENABLED with DD_TRACE_PARTIAL_FLUSH_ENABLED Replace DD_TRACER_PARTIAL_FLUSH_MIN_SPANS with DD_TRACE_PARTIAL_FLUSH_MIN_SPANS
Deprecation Notes#
The DD_TRACER_PARTIAL_FLUSH_ENABLED and DD_TRACER_PARTIAL_FLUSH_MIN_SPANS environment variables have been deprecated and will be removed in version 1.0 of the library.
New Features#
The
ddtrace.Tracer.get_log_correlation_context
method has been added to replaceddtrace.helpers.get_correlation_ids
. It now returns a dictionary which includes the current span’s trace and span ids, as well as the configured service, version, and environment names.
The gevent tasks are now tracked by the threading lock events
Bug Fixes#
Fixes an issue where a manually set
django.request
span resource would get overwritten by the integration.
Pin
setup_requires
dependencysetuptools_scm[toml]>=4,<6.1
to avoid breaking changes.
The profiler now updates the trace resource when capturing span information with the stack and lock collectors. That means that if the trace resource changes after the profiling events are created, the profiler samples will also be updated. This avoids having trace resource being empty when profiling, e.g., WSGI middleware.
v0.52.0rc1#
Deprecation Notes#
Removed the
collect_metrics
argument fromTracer.configure
. See the release notes for v0.49.0 for the migration instructions.
New Features#
Add tracing support for the
httpx
library. Supported versions>=0.14.0
.
- ASGI: store the ASGI span in the scope. The span can be retrieved with the
ddtrace.contrib.asgi.span_from_scope
function.
Submit runtime metrics as distribution metrics instead of gauge metrics.
Support flask-caching (>= 1.10.0) with the Flask-Cache tracer.
Only for CI Visibility (pytest integration): It is now possible to specify any of the following git metadata through environment variables: * DD_GIT_REPOSITORY_URL: The url of the repository where the code is stored * DD_GIT_TAG: The tag of the commit, if it has one * DD_GIT_BRANCH: The branch where this commit belongs to * DD_GIT_COMMIT_SHA: The commit hash of the current code * DD_GIT_COMMIT_MESSAGE: Commit message * DD_GIT_COMMIT_AUTHOR_NAME: Commit author name * DD_GIT_COMMIT_AUTHOR_EMAIL: Commit author email * DD_GIT_COMMIT_AUTHOR_DATE: The commit author date (ISO 8601) * DD_GIT_COMMIT_COMMITTER_NAME: Commit committer name * DD_GIT_COMMIT_COMMITTER_EMAIL: Commit committer email * DD_GIT_COMMIT_COMMITTER_DATE: The commit committer date (ISO 8601)
Bug Fixes#
ASGI: handle decoding errors when extracting headers for trace propagation.
Corrected some typing annotations for PEP 484 compliance
Django: add support for version 3.1+ ASGI applications. A different codepath is taken for requests starting in Django 3.1 which led to the top level span not being generated for requests. The fix introduces automatic installation of the ASGI middleware to trace Django requests.
- dogpile.cache: handle both kwargs and args in the wrapper functions (using
only kwargs would result in an IndexError.
Fixes an issue with the Django integration where if the
urlconf
changes at any point during the handling of the request then the resource name will only be<METHOD> 404
. This fix moves resource name resolution to the end of the request.
Fixes error with tagging non-string Flask view args.
werkzeug.exceptions.NotFound
404 errors are no longer raised and logged as a server error in the Flask integration.
Fixes type hinting for
**patch_modules
parameter forpatch
/patch_all
functions.
Fixes an issue when using the pytest plugin with doctest which raises an
AttributeError
onDoctestItem
.
Fixes a bug in the pytest plugin where xfail test cases in a test file with a module-wide skip raises attribute errors and are marked as xfail rather than skipped.
Fixed the handling of sanic endpoint paths with non-string arguments.
- opentracer: don’t override default tracing config for the ENABLED,
AGENT_HOSTNAME, AGENT_HTTPS or AGENT_PORT settings.
v0.51.0rc2#
Upgrade Notes#
botocore: Update trace propagation format for directly invoked Lambda functions. This breaks compatibility with Lambda functions instrumented with datadog-lambda-python < v41 or datadog-lambda-js < v3.57.0. Please upgrade datadog-lambda-* in invoked lambda functions, or engage legacy compatibility mode in one of two ways: - ddtrace.config.botocore.invoke_with_legacy_context = True - DD_BOTOCORE_INVOKE_WITH_LEGACY_CONTEXT=true
New Features#
Added support for
aiopg~=0.16.0
.
Add MariaDB integration.
The profiler now exports active tasks for CPU and wall time profiles.
Bug Fixes#
Fixed JSON encoding errors in the pytest plugin for parameterized tests with dictionary parameters with tuple keys. The pytest plugin now always JSON encodes the string representations of test parameters.
Fixed JSON encoding errors in the pytest plugin for parameterized tests with complex Python object parameters. The pytest plugin now defaults to encoding the string representations of non-JSON serializable test parameters.
Fix a possible NoneType error in the WSGI middleware start_response method.
v0.51.0rc1#
Upgrade Notes#
The legacy Django configuration method (deprecated in 0.34) has been removed.
Deprecation Notes#
monkey.patch_module
is deprecated.
monkey.get_patch_module
is deprecated.
New Features#
Added support for
jinja2~=3.0.0
.
The pytest integration now uses the name of the repository being tested as the default test service name.
Bug Fixes#
Fixes an issue with enabling the runtime worker introduced in v0.49.0 where no runtime metrics were sent to the agent.
Fix pymongo 3.12.0+ spans not being generated.
v0.50.0rc5#
New Features#
The Python heap profiler can now be enabled by setting the
DD_PROFILING_HEAP_ENABLED
environment variable to1
.
Other Changes#
The botocore integration excludes AWS endpoint call parameters that have a name ending with
Body
from the set of span tags.
v0.50.0rc4#
New Features#
The pytest plugin now includes support for automatically tagging spans with parameters in parameterized tests.
v0.50.0rc3#
Deprecation Notes#
The deprecated dbapi2 configuration has been removed. The integration-specific configuration should be used instead. Look at the v0.48.0 release notes for migration instructions.
New Features#
Add new environment variables to configure the internal trace writer.
DD_TRACE_WRITER_MAX_BUFFER_SIZE
,DD_TRACE_WRITER_INTERVAL_SECONDS
,DD_TRACE_WRITER_MAX_PAYLOAD_SIZE_BYTES
The exception profiler now gathers and exports the traces and spans information.
Bug Fixes#
Fixed the handling of the Django template name tag causing type errors.
Fixes an issue when trying to manually start the runtime metrics worker:
AttributeError: module 'ddtrace.internal.runtime' has no attribute 'runtime_metrics'
sanic: update instrumentation to support version 21.
Performance of the Celery integration has been improved.
Fix runtime-id and system.pid tags not being set on distributed traces.
v0.50.0rc2#
Deprecation Notes#
The reuse of a tracer that has been shut down has been deprecated. A new tracer should be created for generating new traces.
New Features#
Add new
DD_TRACE_AGENT_TIMEOUT_SECONDS
to override the default connection timeout used when sending data to the trace agent. The default is2.0
seconds.
The CI tagging for the pytest plugin now includes OS and Python Runtime metadata including system architecture, platform, version, and Python runtime name and version.
Bug Fixes#
The shutdown task is re-registered when a tracer is reused after it has been shut down.
Fixed the optional argument of
Span.finish
toOptional[float]
instead ofOptional[int]
.
Other Changes#
The pytest plugin now includes git metadata tags including author name and email as well as commit message from CI provider environments.
The profiler won’t be ignoring its own resource usage anymore and will report it in the profiles.
v0.50.0rc1#
Prelude#
Major changes to context management. See the upgrade section for the specifics. Note that only advanced users of the library should be affected by these changes. For the details please refer to the Context section of the docs: https://ddtrace.readthedocs.io/en/v0.50.0/advanced_usage.html
Upgrade Notes#
ddtrace.contrib.asyncio
-AsyncioContextProvider
can now return and activateNone
,Span
orContext
objects.
ddtrace.contrib.gevent
-GeventContextProvider
can now return and activateNone
,Span
orContext
objects.
ddtrace.contrib.tornado
-TracerStackContext
can now return and activateNone
,Span
orContext
objects.
ddtrace.context.Context
no longer maintains the active/current span state.get_current_root_span()
has been removed. Usetracer.current_root_span()
instead.get_current_span()
has been removed. Usetracer.current_span()
instead.add_span()
has been removed. To activate a span in an execution usetracer.context_provider.activate()
instead.close_span()
has been removed. To deactivate a span in an execution usetracer.context_provider.activate()
instead.
ddtrace.provider.BaseContextProvider
active()
now returnsNone
,Span
orContext
objects.activate()
now acceptsNone
,Span
orContext
objects.
ddtrace.span.Span
-Span.context
will now return aContext
ddtrace.tracer.Tracer
tracer.get_call_context()
will now return a one-offContext
reference. This is to maintain backwards compatibility with the API but the functionality differs slightly.tracer.start_span()
passing aspan.context
forchild_of
no longer adds the strong_parent
reference to the new span.
Support for MySQL-python has been removed.
Support for psycopg < 2.7 has been removed.
New Features#
Add
DD_CALL_BASIC_CONFIG={true,false}
environment variable to control whetherddtrace
callslogging.basicConfig
. By default when usingddtrace-run
or running in debug modelogging.basicConfig
is called to ensure there is always a root handler. This has compatibility issues for some logging configurations.DD_CALL_BASIC_CONFIG=false
can be used to skip callinglogging.basicConfig
. The default value istrue
to maintain existing behavior.
agent: support URL with a base path
Automated context management should now work in all asynchronous frameworks that use
contextvars
.
tracer.start_span()
now accepts anactivate
argument (defaultFalse
) to allow manual context management.
tracer.current_trace_context()
has been added to be used to access the trace context of the active trace.
A warning has been added to alert when gevent monkey patching is done after ddtrace has been imported.
Add support for Flask 2
Added retry logic to the tracer to mitigate potential networking issues, like timeouts or dropped connections.
Bug Fixes#
The OpenTracing
tracer.start_span
method no longer activates spans.
Datadog active spans will no longer take precedence over OpenTracing active spans.
django: fix a bug where multiple database backends would not be instrumented.
django: fix a bug when postgres query is composable sql object.
A possible memory leak that occurs when tracing across a fork has been fixed. See https://github.com/DataDog/dd-trace-py/pull/2497 for more information.
Fix double patching of
pymongo
client topology.
v0.49.0rc1#
Prelude#
Several deprecations have been made to Context
as we prepare to move active span management out of this class.
Upgrade Notes#
Support for aiohttp previous to 2.0 has been removed.
Support for deprecated DD_PROFILING_API_URL environment variable has been removed. Use DD_SITE instead.
Support for deprecated DD_PROFILING_API_KEY environment variable has been removed. Use DD_API_KEY instead.
Profiling support for agentless mode must now be explicitly enabled.
- The ddtrace pytest plugin can now label spans from test cases marked xfail with the tag “pytest.result”
and the reason for being marked xfail under the tag “pytest.xfail.reason”.
Removed ddtrace.ext.AppTypes and its usages in the tracer library.
requests: spans will no longer inherit the service name from the parent.
The return value of
Span.pprint()
has been changed to a single line in the tracer debug logs rather than the previous custom multiline format.
Spans are now processed per tracer instance. Formerly spans were stored per-Context which could be shared between tracer instances. Note that context management is not affected. Tracers will still share active spans.
Spans from asynchronous executions (asyncio, gevent, tornado) will now be processed and flushed together. Formerly the spans were handled per-task.
tracer.write()
will no longer have filters applied to the spans passed to it.
The function
ddtrace.utils.merge_dicts
has been removed.
Deprecation Notes#
Context.clone
is deprecated. It will not be required in 0.50.
Context.add_span
is deprecated and will be removed in 0.50.
Context.add_span
is deprecated and will be removed in 0.50.
Context.close_span
is deprecated and will be removed in 0.50.
Context.get_current_span
is deprecated and will be removed in 0.50 please use Tracer.current_span instead.
Context.get_current_root_span
is deprecated and will be removed in 0.50 please useTracer.current_root_span
instead.
Deprecate the configuration of the analytics through the generic dbapi2 configuration. This should now be configured via integration configurations, for example:
# Before export DD_TRACE_DBAPI2_ANALYTICS_ENABLED=1 # After export DD_TRACE_SQLITE3_ANALYTICS_ENABLED=1
ddtrace.compat has been deprecated and will be removed from the public API in ddtrace version 1.0.0.
Deprecate ddtrace.config.dbapi2 as default for TracedCursor and TracedConnection as well as DD_DBAPI2_TRACE_FETCH_METHODS. Use IntegrationConfig and DD_<INTEGRATION>_TRACE_FETCH_METHODS specific to each dbapi-compliant library. For example:
# Before config.dbapi2.trace_fetch_methods = True # After config.psycopg2.trace_fetch_methods = True
The use of
ddtrace.encoding
has been deprecated and will be removed in version 1.0.0.
The ddtrace.http module has been deprecated and will be removed in version 1.0.0, with the ddtrace.http.headers module now merged into ddtrace.trace_utils.
The
collect_metrics
argument of thetracer.configure
method has been deprecated. Runtime metrics should be enabled only via theDD_RUNTIME_METRICS_ENABLED
environment variable.
New Features#
The futures integration is now enabled by default.
requests: add global config support. This enables the requests service name to be configured with
ddtrace.config.requests['service']
or theDD_REQUESTS_SERVICE
environment variable.
Bug Fixes#
Fix broken builds for Python 2.7 on windows where
<stdint.h>
was not available. This change also ensures we build and publishcp27-win
wheels.
CGroup file parsing was fixed to correctly parse container ID with preceding characters.
grpc: handle None values for span tags.
grpc: handle no package in fully qualified method
grpc: Add done callback in streaming response to avoid unfinished spans if a StopIteration is never raised, as is found in the Google Cloud libraries.
grpc: handle IPv6 addresses and no port in target.
Fix DD_LOGS_INJECTION incompatibility when using a
logging.StrFormatStyle
(logging.Formatter(fmt, style="{")
) log formatter.
Fixed a bug that prevented the right integration name to be used when trying to patch a module on import that is already loaded.
Fix
urllib3
patching not properly activating the integration.
gRPC client spans are now marked as measured by default.
Fixes issue of unfinished spans when response is not a grpc.Future but has the same interface, as is the case with the base future class in google-api-core.
In certain circumstances, the profiles generated in a uWSGI application could have been empty. This is now fixed and the profiler records correctly the generated events.
The default agent timeout for profiling has been restored from 2 to 10 seconds to avoid too many profiles from being dropped.
Fix issue with missing traces when using
pymemcache.client.hash.HashClient
.
Added missing pymongo integration configuration, which allows overriding the service name for all the emitted spans.
Other Changes#
Added environment variable DD_BOTTLE_DISTRIBUTED_TRACING to enable distributed tracing for bottle.
The attrs library has been unvendored and is now required as a normal Python dependency with a minimum version requirement of 19.2.0.
The six library has been removed from vendor and the system-wide version is being used. It requires version 1.12.0 or later.
Documentation on how to use Gunicorn with the
gevent
worker class has been added.
Added environment variable DD_FALCON_DISTRIBUTED_TRACING to enable distributed tracing for falcon.
When extracting context information from HTTP headers, a new context is created when the trace ID is either 0 or not available within the headers.
Added environment variable DD_PYLONS_DISTRIBUTED_TRACING to enable distributed tracing for pylons.
Update
pymemcache
test suite to test latest versions.
Added config.pyramid.distributed_tracing setting to integration config for pyramid.
The
ddtrace.payload
submodule has been removed.
Added environment variable DD_TORNADO_DISTRIBUTED_TRACING to enable distributed tracing for tornado.
v0.48.0rc2#
New Features#
Adds support for the Datadog Lambda Extension. The tracer will send traces to the extension by default if it is present.
Add support for space-separated DD_TAGS.
urllib3: add urllib3 integration
v0.48.0rc1#
Upgrade Notes#
The deprecated dogstatsd_host and dogstatsd_port arguments to tracer.configure() have been removed.
Support for gevent 1.0 has been removed and gevent >= 1.1 is required.
flask: deprecated configuration option extra_error_codes has been removed.
The deprecated
pyddprofile
wrapper has been removed. Useddtrace-run
withDD_PROFILING_ENABLED=1
set instead.
A ValueError will now be raised on tracer initialization if the Agent URL specified to the initializer or with the environment variable DD_TRACE_AGENT_URL is malformed.
uWSGI is no longer supported with
ddtrace-run
due to a limitation of how tracer initialization occurs. See the updated instructions for enabling tracing in the library uWSGI documentation.
New Features#
dogpile.cache: is now automatically instrumented by default.
pylons: now supports all the standard http tagging including query string, custom error codes, and request/response headers.
The ddtrace pytest plugin can now call
ddtrace.patch_all
via the--ddtrace-patch-all
option.
Span
now accepts aon_finish
argument used for specifying functions to call when a span finishes.
Bug Fixes#
The
Records
parameter toFirehose
endpoint calls is being excluded from the tags to avoid generating traces with a large payload.
Tracer: fix configuring tracer with dogstatsd url.
elasticsearch: patch versioned elasticsearch modules (elasticsearch1, …, elasticsearch7).
botocore: Do not assume that ResponseMeta exists in the results.
django: handle erroneous middleware gracefully.
The tracer now captures the task ID from the cgroups file for Fargate >= 1.4.0 and reports it to the agent as the Datadog-Container-ID tag.
Fix a bug when tracing a mako
DefTemplate
or anyTemplate
that does not have afilename
property.
flask: fix a bug when the query string would contain non-Unicode characters
Fix a formatting issue on error exporting profiles.
A workaround is provided for the problem with uWSGI worker processes failing to respawn. This can occur when using
ddtrace-run
for automatic instrumentation and configuration or manual instrumentation and configuration without the necessary uWSGI options. The problem is caused by how the tracer can end up starting threads in the master process before uWSGI forks to initialize the workers processes. To avoid this, we have provided updated instructions for enabling tracing in the library uWSGI documentation.
Other Changes#
The logic behind the header extraction for distributed tracing has been improved.
The default connection timeout for the profiling agent has now been reduced from 10 to 2 seconds to match the tracer behavior.
The tracemalloc memory profiler, which was disabled by default, has been removed.
Query strings are stripped out from URLs by default when setting URL metadata on a span. This change affects all integrations that store HTTP metadata, like aiohttp, falcon, requests, urllib3.
v0.47.0rc1#
Upgrade Notes#
elasticsearch: removed get_traced_transport method and ddtrace.contrib.elasticsearch.transport module.
The profiler now automatically sets up uWSGI compatibility in auto mode or with profile_children=True. Make sure that you don’t have custom code instrumenting the profiler in those cases.
The
Tracer
class properties DEFAULT_HOSTNAME, DEFAULT_PORT, DEFAULT_DOGSTATSD_PORT, DEFAULT_DOGSTATSD_URL, DEFAULT_AGENT_URL have been removed.
New Features#
cherrypy: introduce TraceMiddleware for the CherryPy web framework.
django: tag root spans as measured.
elasticsearch: add support for version 7.
fastapi: add integration.
Introduce support for the DD_SERVICE_MAPPING environment variable to allow remapping service names on emitted spans.
httplib: distributed tracing is now enabled by default.
The profiler now supports most operation mode from uWSGI without much configuration. It will automatically plug itself in post fork hooks when multiprocess mode is used.
wsgi: add tracing middleware.
Bug Fixes#
Resolves an issue in Django tracing where, if query_string is not present in request.META, a KeyError is raised, causing the request to 500
Deprecate the DD_LOGGING_RATE_LIMIT variable in favor of the standard DD_TRACE_LOGGING_RATE for configuring the logging rate limit.
sampler: removed bug causing sample_rate of 0 to be reset to 1, and raise ValueError instead of logging.
starlette: unpatch calls correctly.
flask: fix memory leak of sampled out traces.
Fix CPU time and wall time profiling not ignoring the profiler tasks with gevent.
Other Changes#
The default maximum CPU time used for the stack profiler (CPU time, wall time and exceptions profiling) has been decreased from 2% to 1%.
v0.46.0rc1#
Upgrade Notes#
The profiler will only load tags from the DD_TAGS environment variable once at start.
Deprecation Notes#
flask: Use HTTP Custom Error Codes instead of
ddtrace.config.flask['extra_error_codes']
.
New Features#
aiohttp: store request and response headers.
bottle: store request and response headers.
flask: store response headers.
molten: store request headers.
pyramid: store request and response headers.
flask: store request and response headers when auto-instrumented.
The ddtrace.profiling.auto module will warn users if gevent monkey patching is done after the profiler is auto-instrumented.
The Profiler object can now be passed tags with the tags keyword argument.
Bug Fixes#
dbapi: avoid type error with potential non-compliance in db libraries when setting tag for row count.
django: add legacy resource format of {handler}.
grpc: fix wrapper for streaming response to support libraries that call an internal method directly.
sanic: use path parameter names instead of parameter values for the resource.
The profiler won’t deadlock on fork when gevent monkey patch is enabled.
requests: fix TracedSession when patches are not applied.
v0.45.0rc1#
Prelude#
Build and deploy Python 3.9 wheels for releases
Upgrade Notes#
Context.get() has been removed and the functionality has been rolled into Context.close_span().
Tracer.record() has similarly been removed as it is no longer useful with Context.get() removed.
The deprecated compatibility module ddtrace.profile has been removed.
The profiler now uses the tracer configuration is no configuration is provided.
Deprecation Notes#
The pyddprofile wrapper is deprecated. Use DD_PROFILING_ENABLED=true ddtrace-run instead.
The profiler does not catch uncaught exception anymore.
New Features#
botocore: added distributed_tracing configuration setting which is enabled by default.
- The ddtrace-run command now supports the following arguments:
-h, –help -d, –debug enable debug mode (disabled by default) -i, –info print library info useful for debugging -p, –profiling enable profiling (disabled by default) -v, –version show program’s version number and exit
It now also has friendlier error messages when used incorrectly.
Add functionality to call gevent.monkey.patch_all() with ddtrace-run by setting the environment variable DD_GEVENT_PATCH_ALL=true. This ensures that gevent patching is done as early as possible in the application.
botocore: inject distributed tracing data to ClientContext to trace lambda invocations.
botocore: inject tracing data to MessageAttributes.
The profiler now tracks the running gevent Greenlet and store it as part of the CPU and wall time profiling information.
The profiler is now able to upload profiles to the Datadog Agent by using a Unix Domain Socket.
It is now possible to pass a url parameter to the Profiler to specify the Datadog agent location.
The new memory profiler for Python is now enabled by default. This improves the profiler memory consumption and performance impact. It can still be disabled by setting DD_PROFILING_MEMALLOC=0 as an environment variable.
The profiler now uses the tracer configuration is no configuration is provided.
pytest integration. This enables the pytest runner to trace test executions.
Bug Fixes#
core: always reset the current_span in the context.
django: Http404 exceptions will no longer be flagged as errors
django: add safe guards for building http.url span tag.
aiobotocore: set span error for 5xx status codes.
elasticsearch: set span error for 5xx status codes.
django, DRF, ASGI: fix span type for web request spans.
Fixes span id tagging in lock profiling.
Fix UDS upload for profiling not using the correct path.
Fixed an issue in profiling exporting profiles twice when forking.
core: fix race condition in TracerTagCollector.
Other Changes#
Start-up logs are now disabled by default. To enable start-up logs use DD_TRACE_STARTUP_LOGS=true or DD_TRACE_DEBUG=true.
v0.44.0#
Prelude#
Add support for Python 3.9
New Features#
Store request headers in Flask integration.
pyodbc integration. This enables the pyodbc library to trace queries.
starlette integration resource aggregation This aggregates endpoints to the starlette application resource that was accessed. It occurs by default but it is configurable through config.starlette[“aggregate_resources”].
The profiler now captures the traces information with the lock profiling.
The Profiler instances now restart automatically in child process when the main program is forked. This only works for Python ≥ 3.7.
Bug Fixes#
dbapi: add support for connection context manager usage
django: check view before instrumenting MRO.
core: use loose types when encoding.
Patch pynamodb on import to prevent patching conflicts with gevent.
tornado: handle when the current span is None in log_exception().