API#

Tracing#

Importing ddtrace.auto installs Datadog instrumentation in the runtime. It should be used when ddtrace-run is not an option. Using it with ddtrace-run is unsupported and may lead to undefined behavior:

# myapp.py

import ddtrace.auto  # install instrumentation as early as possible
import mystuff

def main():
    print("It's my app!")

main()

If you’d like more granular control over instrumentation setup, you can call the patch* functions directly.

ddtrace.patch(raise_errors: bool = True, **patch_modules: Union[list[str], bool]) → None#

Patch only a set of given modules.

Parameters

raise_errors (bool) – Raise error if one patch fail.
patch_modules (dict) –
List of modules to patch.
```
>>> patch(psycopg=True, elasticsearch=True)
```

class ddtrace.trace.Tracer#

Tracer is used to create, sample and submit spans that measure the execution time of sections of code.

If you’re running an application that will serve a single trace per thread, you can use the global tracer instance:

from ddtrace.trace import tracer
trace = tracer.trace('app.request', 'web-server').finish()

current_trace_context(*args, **kwargs) → Optional[ddtrace._trace.context.Context]#

Return the context for the current trace.

If there is no active trace then None is returned.

get_log_correlation_context(active: Optional[Union[ddtrace._trace.span.Span, ddtrace._trace.context.Context]] = None) → dict[str, str]#: Retrieves the data used to correlate a log with the current active trace. Generates a dictionary for custom logging instrumentation including the trace id and span id of the current active span, as well as the configured service, version, and environment names. If there is no active span, a dictionary with an empty string for each value will be returned.

configure(context_provider: Optional[ddtrace._trace.provider.BaseContextProvider] = None, compute_stats_enabled: Optional[bool] = None, appsec_enabled: Optional[bool] = None, appsec_enabled_origin: Optional[str] = '', iast_enabled: Optional[bool] = None, apm_tracing_disabled: Optional[bool] = None, trace_processors: Optional[list[ddtrace._trace.processor.TraceProcessor]] = None) → None#

Configure a Tracer.

Parameters

context_provider (object) – The ContextProvider that will be used to retrieve automatically the current call context. This is an advanced option that usually doesn’t need to be changed from the default value.
appsec_enabled (bool) – Enables Application Security Monitoring (ASM) for the tracer.
iast_enabled (bool) – Enables IAST support for the tracer
apm_tracing_disabled (bool) – When APM tracing is disabled ensures ASM support is still enabled.
trace_processors (list[TraceProcessor]) – This parameter sets TraceProcessor (ex: TraceFilters). Trace processors are used to modify and filter traces based on certain criteria.

start_span(name: str, child_of: Optional[Union[ddtrace._trace.span.Span, ddtrace._trace.context.Context]] = None, service: Optional[str] = None, resource: Optional[str] = None, span_type: Optional[str] = None, activate: bool = False, span_api: str = 'datadog') → ddtrace._trace.span.Span#

Return a span that represents an operation called name.

Note that the trace() method will almost always be preferred over this method as it provides automatic span parenting. This method should only be used if manual parenting is desired.

Parameters

name (str) – the name of the operation being traced.
child_of (object) – a Span or a Context instance representing the parent for this span.
service (str) – the name of the service being traced.
resource (str) – an optional name of the resource being tracked.
span_type (str) – an optional operation type.
activate – activate the span once it is created.

To start a new root span:

span = tracer.start_span("web.request")

To create a child for a root span:

root_span = tracer.start_span("web.request")
span = tracer.start_span("web.decoder", child_of=root_span)

Spans from start_span are not activated by default:

with tracer.start_span("parent") as parent:
    assert tracer.current_span() is None
    with tracer.start_span("child", child_of=parent):
        assert tracer.current_span() is None

new_parent = tracer.start_span("new_parent", activate=True)
assert tracer.current_span() is new_parent

Note: be sure to finish all spans to avoid memory leaks and incorrect parenting of spans.

trace(name: str, service: Optional[str] = None, resource: Optional[str] = None, span_type: Optional[str] = None, span_api: str = 'datadog') → ddtrace._trace.span.Span#

Activate and return a new span that inherits from the current active span.

Parameters

name (str) – the name of the operation being traced
service (str) – the name of the service being traced. If not set, it will inherit the service from its parent.
resource (str) – an optional name of the resource being tracked.
span_type (str) – an optional operation type.

The returned span must be finish’d or it will remain in memory indefinitely:

>>> span = tracer.trace("web.request")
    try:
        # do something
    finally:
        span.finish()

>>> with tracer.trace("web.request") as span:
        # do something

Example of the automatic parenting:

parent = tracer.trace("parent")     # has no parent span
assert tracer.current_span() is parent

child  = tracer.trace("child")
assert child.parent_id == parent.span_id
assert tracer.current_span() is child
child.finish()

# parent is now the active span again
assert tracer.current_span() is parent
parent.finish()

assert tracer.current_span() is None

parent2 = tracer.trace("parent2")
assert parent2.parent_id is None
parent2.finish()

current_root_span() → Optional[ddtrace._trace.span.Span]#

Returns the local root span of the current execution/process.

Note: This cannot be used to access the true root span of the trace in a distributed tracing setup if the actual root span occurred in another execution/process.

This is useful for attaching information to the local root span of the current execution/process, which is often also service entry span.

For example:

# get the local root span
local_root_span = tracer.current_root_span()
# set the host just once on the root span
if local_root_span:
    local_root_span.set_tag('host', '127.0.0.1')

current_span() → Optional[ddtrace._trace.span.Span]#

Return the active span in the current execution context.

Note that there may be an active span represented by a context object (like from a distributed trace) which will not be returned by this method.

property agent_trace_url: Optional[str]#: Trace agent url

flush()#: Flush the buffer of the trace writer. This does nothing if an unbuffered trace writer is used.

wrap(name: Optional[str] = None, service: Optional[str] = None, resource: Optional[str] = None, span_type: Optional[str] = None) → Callable[[ddtrace._trace.tracer.AnyCallable], ddtrace._trace.tracer.AnyCallable]#

A decorator used to trace an entire function. If the traced function is a coroutine, it traces the coroutine execution when is awaited.

Parameters

name (str) – the name of the operation being traced. If not set, defaults to the fully qualified function name.
service (str) – the name of the service being traced. If not set, it will inherit the service from its parent.
resource (str) – an optional name of the resource being tracked.
span_type (str) – an optional operation type.

>>> @tracer.wrap('my.wrapped.function', service='my.service')
    def run():
        return 'run'

>>> # name will default to 'execute' if unset
    @tracer.wrap()
    def execute():
        return 'executed'

>>> # or use it in asyncio coroutines
    @tracer.wrap()
    async def coroutine():
        return 'executed'

>>> @tracer.wrap()
    @asyncio.coroutine
    def coroutine():
        return 'executed'

>>> # or use it on generators
    @tracer.wrap()
    def gen():
        yield 'executed'

>>> @tracer.wrap()
    async def gen():
        yield 'executed'

You can access the current span using tracer.current_span() to set tags:

>>> @tracer.wrap()
    def execute():
        span = tracer.current_span()
        span.set_tag('a', 'b')

set_tags(tags: dict[str, str]) → None#

Set some tags at the tracer level. This will append those tags to each span created by the tracer.

Parameters: tags (dict) – dict of tags to set at tracer level

shutdown(timeout: Optional[float] = None) → None#

Shutdown the tracer and flush finished traces. Avoid calling shutdown multiple times.

Parameters: timeout (int | float | None) – How long in seconds to wait for the background worker to flush traces before exiting or None to block until flushing has successfully completed (default: None)

class ddtrace.trace.Span(name: str, service: Optional[str] = None, resource: Optional[str] = None, span_type: Optional[str] = None, trace_id: Optional[int] = None, span_id: Optional[int] = None, parent_id: Optional[int] = None, start: Optional[int] = None, context: Optional[ddtrace._trace.context.Context] = None, on_finish: Optional[list[Callable[[ddtrace._trace.span.Span], NoneType]]] = None, span_api: str = 'datadog', links: Optional[list[ddtrace._trace._span_link.SpanLink]] = None)#

finish(finish_time: Optional[float] = None) → None#

Mark the end time of the span and submit it to the tracer. If the span has already been finished don’t do anything.

Parameters: finish_time – The end time of the span, in seconds. Defaults to now.

set_tag(key: str, value: Optional[str] = None) → None#

Set a tag key/value pair on the span.

Keys must be strings, values must be str-able.

Parameters

key (str) – Key to use for the tag
value (str-able value) – Value to assign for the tag

get_tag(key: str) → Optional[str]#: Return the given tag or None if it doesn’t exist.

get_tags() → dict[str, str]#: Return all tags.

set_tags(tags: dict[str, str]) → None#: Set a dictionary of tags on the given span. Keys and values must be strings (or stringable)

set_metric(key: str, value: Union[int, float]) → None#: This method sets a numeric tag value for the given key.

set_metrics(metrics: dict[str, Union[int, float]]) → None#: Set a dictionary of metrics on the given span. Keys must be must be strings (or stringable). Values must be numeric.

get_metric(key: str) → Optional[Union[int, float]]#: Return the given metric or None if it doesn’t exist.

get_metrics() → dict[str, Union[int, float]]#: Return all metrics.

set_traceback(limit: Optional[int] = None)#: If the current stack has an exception, tag the span with the relevant error info. If not, tag it with the current python stack.

set_exc_info(exc_type: type[BaseException], exc_val: BaseException, exc_tb: Optional[types.TracebackType], limit: Optional[int] = None) → None#: Tag the span with an error tuple as from sys.exc_info().

record_exception(exception: BaseException, attributes: Optional[dict[str, Union[str, bool, int, float, Sequence[str], Sequence[bool], Sequence[int], Sequence[float]]]] = None) → None#

Records an exception as a span event. Multiple exceptions can be recorded on a span.

Parameters

exception – The exception to record.
attributes – Optional dictionary of additional attributes to add to the exception event. These attributes will override the default exception attributes if they contain the same keys. Valid attribute values include (homogeneous array of) strings, booleans, integers, floats.

link_span(context: ddtrace._trace.context.Context, attributes: Optional[dict[str, Any]] = None) → None#: Defines a causal relationship between two spans

class ddtrace.trace.Context(trace_id: Optional[int] = None, span_id: Optional[int] = None, dd_origin: Optional[str] = None, sampling_priority: Optional[float] = None, meta: Optional[dict[str, str]] = None, metrics: Optional[dict[str, Union[int, float]]] = None, lock: Optional[_thread.RLock] = None, span_links: Optional[list[ddtrace._trace._span_link.SpanLink]] = None, baggage: Optional[dict[str, Any]] = None, is_remote: bool = True)#

Represents the state required to propagate a trace across execution boundaries.

trace_id: Optional[int]#

span_id: Optional[int]#

property sampling_priority: Optional[Union[int, float]]#: Return the context sampling priority for the trace.

property dd_origin: Optional[str]#: Get the origin of the trace.

property dd_user_id: Optional[str]#: Get the user ID of the trace.

set_baggage_item(key: str, value: Any) → None#: Sets a baggage item in this span context. Note that this operation mutates the baggage of this span context

copy(trace_id: int, span_id: int) → ddtrace._trace.context.Context#: Return a shallow copy of the context with the given correlation IDs.

get_baggage_item(key: str) → Optional[Any]#: Gets a baggage item in this span context.

get_all_baggage_items() → dict[str, Any]#: Returns all baggage items in this span context.

remove_baggage_item(key: str) → None#: Remove a baggage item from this span context.

remove_all_baggage_items() → None#: Removes all baggage items from this span context.

class ddtrace.propagation.http.HTTPPropagator#

A HTTP Propagator using HTTP headers as carrier. Injects and Extracts headers according to the propagation style set by ddtrace configurations.

static inject(context: Union[ddtrace._trace.context.Context, ddtrace._trace.span.Span], headers: dict[str, str]) → None#

Inject Context attributes that have to be propagated as HTTP headers.

Here is an example using requests:

import requests

from ddtrace.propagation.http import HTTPPropagator

def parent_call():
    with tracer.start_span('parent_span') as span:
        headers = {}
        # Preferred: Pass span object to context argument
        HTTPPropagator.inject(span, headers)
        url = '<some RPC endpoint>'
        r = requests.get(url, headers=headers)

    with tracer.start_span('child_span2') as span:
        headers = {}
        # Alternative: Pass context, but ensure sampling_priority is set
        tracer.sample(span)
        HTTPPropagator.inject(span.context, headers)
        url = '<some RPC endpoint>'
        r = requests.get(url, headers=headers)

Parameters

context (Union[Span, Context]) – Pass a Span object (preferred) or Context object. Span objects automatically trigger sampling decisions. Context objects should have sampling_priority set to avoid inconsistent downstream sampling.
headers (dict) – HTTP headers to extend with tracing attributes.

static extract(headers)#

Extract a Context from HTTP headers into a new Context. For tracecontext propagation we extract tracestate headers for propagation even if another propagation style is specified before tracecontext, so as to always propagate other vendor’s tracestate values by default. This is skipped if the tracer is configured to take the first style it matches.

Here is an example from a web endpoint:

from ddtrace.propagation.http import HTTPPropagator

def my_controller(url, headers):
    context = HTTPPropagator.extract(headers)
    if context:
        tracer.context_provider.activate(context)

    with tracer.trace('my_controller') as span:
        span.set_tag('http.url', url)

Parameters: headers (dict) – HTTP headers to extract tracing attributes.
Returns: New Context with propagated attributes.

OpenTelemetry API#

The dd-trace-py library provides an implementation of the OpenTelemetry API for distributed tracing, metrics, and logs.

Note: Datadog-specific configurations take precedence over OpenTelemetry settings. By default, telemetry data is routed to a Datadog Agent. The minimum supported Datadog Agent version is v7.33.0 (but recommended to use >=7.66 for best performance).

Note: OpenTelemetry support is lazily loaded when the opentelemetry package is first imported. Ensure import opentelemetry... or from opentelemetry.... is present in your code.

Tracing#

dd-trace-py supports OpenTelemetry tracing by mapping OpenTelemetry spans to Datadog spans. The TracerProvider sends Datadog-formatted payloads and is compatible only with the Datadog Agent (it does not emit OTLP).

Enable OpenTelemetry tracing support by setting DD_TRACE_OTEL_ENABLED=true and using ddtrace.auto or ddtrace-run. Manual configuration is also supported through ddtrace.opentelemetry.TracerProvider.

For supported configurations, see OpenTelemetry Tracing Configuration.

Usage example:

import os
os.environ["DD_TRACE_OTEL_ENABLED"] = "true"

import ddtrace.auto

from opentelemetry import trace

# Use tracing
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("operation") as span:
    span.set_attribute("key", "value")
    # Your code here

Metrics#

dd-trace-py supports OpenTelemetry metrics. Metrics are emitted in OTLP format and can be routed to any OTLP-compatible receiver (Datadog Agent, OpenTelemetry Collector, or directly to the Datadog intake).

Enable metrics support by setting DD_METRICS_OTEL_ENABLED=true and using ddtrace.auto or ddtrace-run. ddtrace automatically configures the appropriate providers and exporters.

You must install an OTLP exporter (minimum supported version: v1.15.0):

pip install opentelemetry-exporter-otlp>=1.15.0

For supported configurations, see OpenTelemetry Metrics Configuration.

Usage example:

import os
os.environ["DD_METRICS_OTEL_ENABLED"] = "true"
import ddtrace.auto

from opentelemetry import metrics

# ddtrace automatically configures the MeterProvider
meter = metrics.get_meter(__name__)
counter = meter.create_counter("requests_total")
counter.add(1, {"method": "GET"})

Logs#

dd-trace-py supports OpenTelemetry logs. Logs are emitted in OTLP format and can be routed to any OTLP-compatible receiver (Datadog Agent, OpenTelemetry Collector, or directly to the Datadog intake).

Enable logs support by setting DD_LOGS_OTEL_ENABLED=true and using ddtrace.auto or ddtrace-run. ddtrace automatically configures the appropriate providers and exporters.

You must install an OTLP exporter (minimum supported version: v1.15.0):

pip install opentelemetry-exporter-otlp>=1.15.0

For supported configurations, see OpenTelemetry Logs Configuration.

Usage example:

import os
import logging
os.environ["DD_LOGS_OTEL_ENABLED"] = "true"
os.environ["DD_TRACE_OTEL_ENABLED"] = "true"
import ddtrace.auto

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

# Logs within a trace context are automatically correlated
with tracer.start_as_current_span("user_operation") as span:
    span.set_attribute("user.id", "12345")
    logging.info("User operation started", extra={"operation": "login"})
    # Your business logic here
    logging.info("User operation completed", extra={"status": "success"})

Supported Configuration#

The Datadog SDK supports many of the configurations available in the OpenTelemetry SDK. The following environment variables are supported:

DD_{LOGS,TRACE,METRICS}_OTEL_ENABLED Enable OpenTelemetry logs, metrics, or traces (default: false). These features are off by default, allowing Datadog to track adoption by customer or service when the configuration is set to true.
OTEL_{METRICS,LOGS}_EXPORTER This value defaults to otlp. We can gauge the prevalence of other exporters by observing other values, such as prometheus, console, or none.
OTEL_METRIC_EXPORT_INTERVAL Interval (in milliseconds) between metric exports.
OTEL_METRIC_EXPORT_TIMEOUT Timeout (in milliseconds) for metric export requests.
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE For the best Datadog experience, we encourage customers to use delta temporality for monotonic sums, histograms, and exponential histograms. The default value in the Datadog SDKs is delta, which differs from the OpenTelemetry specified default of cumulative. Tracking this configuration helps understand how many customers prefer Cumulative Temporality and why.
OTEL_EXPORTER_OTLP_{METRICS,LOGS}_ENDPOINT When metrics export is enabled with DD_METRICS_OTEL_ENABLED=true, this defines the target endpoint for metrics export (default: http://<agent_hostname>:4317).
OTEL_EXPORTER_OTLP_{METRICS,LOGS}_HEADERS Optional headers for metrics or logs export in JSON format (default: {}).
OTEL_EXPORTER_OTLP_{METRICS,LOGS}_TIMEOUT Request timeout in milliseconds for metrics or logs export (default: 10000).
OTEL_EXPORTER_OTLP_{METRICS,LOGS}_PROTOCOL OTLP protocol used for metrics or logs. Supported protocols are grpc (default) and http/protobuf.

Important: Metrics and logs are exported using the OTLP protocol and can be routed to any OTLP-compatible receiver. See Send OpenTelemetry Data to Datadog for more details. Traces, however, use Datadog’s custom MsgPack format and require a Datadog Agent.

Trace Mapping#

OpenTelemetry spans are mapped to Datadog spans. This mapping is described by the following table, using the protocol buffer field names used in OpenTelemetry and Datadog.

OpenTelemetry	Datadog	Description
`trace_id`	`traceID`
`span_id`	`spanID`
`trace_state`	`meta["tracestate"]`	Datadog-specific data is stored in the trace state using the `dd=` prefix
`parent_span_id`	`parentID`
`name`	`resource`
`kind`	`meta["span.kind"]`
`start_time_unix_nano`	`start`
`end_time_unix_nano`	`duration`	Derived from the start and end times
`attributes[<key>]`	`meta[<key>]`	Each OpenTelemetry attribute is stored as a Datadog tag (`meta`)
`links[]`	`meta["_dd.span_links"]`
`status`	`error`	Derived from span status
`events[]`	`meta["events"]`	If the v0.4 API is used, span events are stored as `Span.span_events`

class ddtrace.opentelemetry.TracerProvider#

Entry point of the OpenTelemetry API and provides access to OpenTelemetry compatible Tracers. One TracerProvider should be initialized and set per application.

get_tracer(instrumenting_module_name: str, instrumenting_library_version: Optional[str] = None, schema_url: Optional[str] = None, attributes: Optional[dict] = None) → opentelemetry.trace.Tracer#: Returns an opentelemetry compatible Tracer.

Runtime Metrics#

class ddtrace.runtime.RuntimeMetrics#

Runtime metrics service API.

This is normally started automatically by ddtrace-run when the DD_RUNTIME_METRICS_ENABLED variable is set.

To start the service manually, invoke the enable static method:

from ddtrace.runtime import RuntimeMetrics
RuntimeMetrics.enable()

static enable(tracer: Optional[ddtrace._trace.tracer.Tracer] = None, dogstatsd_url: Optional[str] = None) → None#

If the service has already been activated before, this method does nothing. Use disable to turn off the runtime metric collection service.

Parameters: tracer – The tracer instance to correlate with.

static disable() → None#

Disable the runtime metrics collection service.

Once disabled, runtime metrics can be re-enabled by calling enable again.

Dynamic Instrumentation#

Enablement#

Dynamic Instrumentation can be enabled by setting the DD_DYNAMIC_INSTRUMENTATION_ENABLED variable to true in the environment, when using the ddtrace-run command. Alternatively, when ddtrace-run cannot be used, it can be enabled programmatically with:

import ddtrace.auto

Note that the DD_DYNAMIC_INSTRUMENTATION_ENABLED still needs to be set to true in the environment.

Alternatively, Dynamic Instrumentation can be enabled programmatically with:

from ddtrace.debugging import DynamicInstrumentation

# Enable dynamic instrumentation
DynamicInstrumentation.enable()

...

# Disable dynamic instrumentation
DynamicInstrumentation.disable()

However, this method requires that Dynamic Instrumentation is started manually in every forked process.

Configuration#

See the Configuration page for more details on how to configure Dynamic Instrumentation.

Source Code Integration#

Datadog Source Code Integration is supported for Git by the addition of the repository URL and commit hash in the Python package metadata field Project-URL with name source_code_link.

Format of source_code_link: <repository url>#<commit hash>

setuptools#

The ddtrace provides automatic instrumentation of setuptools to embed the source code link into the project metadata. ddtrace has to be installed as a build dependency.

Packages with pyproject.toml can update the build system requirements:

[build-system]
requires = ["setuptools", "ddtrace"]
build-backend = "setuptools.build_meta"

The instrumentation of setuptools can be automatically enabled to embed the source code link with a one-line import in setup.py (before setuptools import):

import ddtrace.sourcecode.setuptools_auto
from setuptools import setup

setup(
    name="mypackage",
    version="0.0.1",
    #...
)