Build System#

Native Pre-Dependencies#

Some native extensions are written in Rust, Cython, or C++, requiring specific dependencies to build them.

For Rust, you need the rustc and cargo tools installed. Install them by following these steps:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain stable -y

For C++, requirements vary by system, but you’ll need a C++ compiler and cmake. Install them as follows:

  • On Ubuntu:

    When using GCC, you need versions that have fixes for this bug

sudo apt-get install build-essential cmake

  • On macOS:

brew install cmake
xcode-select --install

Build Dependencies#

To see the current build dependencies, check the [build-system] section in the pyproject.toml file. For example, at the time of writing, the dependencies are:

[build-system]
requires = ["cython", "cmake>=3.24.2,<3.28; python_version>='3.8'", "setuptools-rust<2"]

To install all dependencies in one step, use:

pip install 'cython' 'cmake>=3.24.2,<3.28' 'setuptools-rust<2'

Note that pip install -e (described below) also installs these build dependencies automatically.

Local Installation from a Local Repository#

The simplest way to build and install the library in your Python environment is with pip:

pip install .

Local Installation in Development Mode#

To install the library in development mode, use the -e flag:

pip install -e .

This installs the library in “editable” mode, meaning changes to the source code are immediately reflected in the installed package. This works only if you run the command from the repository’s root directory. To use ddtrace from a different directory (e.g., another project running under ddtrace), add the repository path to your PYTHONPATH environment variable:

export PYTHONPATH=/path/to/ddtrace:$PYTHONPATH

If you skip pip install -e . and use PYTHONPATH, you must manually install the build dependencies (see above) and compile the native extensions with:

python setup.py build_ext --inplace

Then, if you want to run ddtrace from the repo with another project in another directory, instead of using ddtrace-run (which with an editable or PYTHONPATH install would not find the one in the repository), do:

python -m ddtrace.commands.ddtrace_run python my_traced_app.py

Using sccache to Speed Up Builds#

If you frequently rebuild native extensions or deploy to multiple containers, consider using sccache to accelerate compilation. sccache caches compilation results, reducing build times. Install it using cargo, which you should have if you followed the Rust installation steps earlier:

cargo install sccache

For the build system to locate sccache, either ensure its path is in your PATH environment variable or set the SCCACHE_PATH (or SCCACHE) environment variable to its location:

export SCCACHE_PATH=/home/doe/.cargo/bin/sccache

Additionally, enable sccache by setting the DD_USE_SCCACHE environment variable:

export DD_USE_SCCACHE=1

Then, build the tracer as usual.

To verify that sccache is working after a build, check its statistics:

sccache --show-stats

If cache-hits is greater than 0 (or increases after a cached build), sccache is functioning correctly.

To change the sccache cache directory—e.g., to share it with containers via copying or mounting a volume—set the SCCACHE_DIR environment variable:

export SCCACHE_DIR=/path/to/cache

The sccache –show-stats command also displays the current cache directory.

Configuration Environment Variables#

These environment variables modify aspects of the build process.

DD_COMPILE_DEBUG(DEPRECATED)#

This deprecated variable will be removed in a future release. If set to 1, it compiles the tracer with debug symbols—useful for debugging the tracer itself but resulting in a slower and larger binary. This is not recommended for production. If set to 0, extensions are compiled in Release mode.

Type: Boolean

Default: (no value)

New in version v0.44.0.

DD_USE_SCCACHE#

If set to 1, it enables sccache to accelerate native extension compilation (see above). This is beneficial for frequent rebuilds or multi-container deployments.

Type: Boolean

Default: (no value)

New in version v2.12.0.

DD_COMPILE_MODE#

Specifies the compilation mode for native extensions. Depending on your CMake version, options typically include Release, Debug, RelWithDebInfo, and MinSizeRel. Note that Debug produces slower, larger binaries; RelWithDebInfo increases size but retains the performance of Release; and MinSizeRel reduces binary size at the cost of performance.

Type: String

Default: Release

New in version v3.3.0.

DD_COMPILE_ABSEIL#

If set to 1, the tracer is compiled with the Abseil library, enhancing performance for Runtime Code Analysis features when active (DD_IAST_ENABLED=1). If set to 0, the Runtime Code Analysis extension is built without Abseil, speeding up the build process at the cost of some performance.

Type: Boolean

Default: True

New in version v3.3.0.

DD_CYTHONIZE#

If enabled, then Cython extensions are included in the build. Disabling will exclude them. This is mostly useful for source distribution builds so we can skip calling cythonize on source files.

Type: Boolean

Default: True

DD_FAST_BUILD#

If set to 1, the tracer is compiled with minimal optimizations (-g0) and DD_COMPILE_ABSEIL is forced to 0. This is not recommended for production due to reduced performance.

Type: Boolean

Default: (no value)

New in version v3.3.0.

DD_PROFILING_MEMALLOC_ASSERT_ON_REENTRY#

If set to 1, it enables a memalloc-specific native build guard that aborts on reentrant allocator hook calls (malloc -> malloc or malloc -> free). This is intended for memalloc testing and debugging builds, not for production use.

Type: Boolean

Default: (no value)

DD_CMAKE_INCREMENTAL_BUILD#

Enables support for incremental builds of CMake extensions when doing an in-place install (e.g. editable mode).

Type: Boolean

Default: True

New in version v3.10.0.

DD_SETUP_CACHE_DOWNLOADS#

Caches the download of artifacts needed by the build process.

Type: Boolean

Default: True

New in version v3.10.0.

DD_DOWNLOAD_MAX_RETRIES#

Maximum number of retry attempts for transient download failures from GitHub. Retries are triggered by HTTP 429 (rate limit), 502/503/504 (server errors), and network timeouts. Uses exponential backoff with jitter between retries.

Type: Integer

Default: 10

New in version v4.1.0.

DD_DOWNLOAD_INITIAL_DELAY#

Initial delay in seconds before the first retry attempt. Delay increases exponentially with backoff_factor=1.618 (Fibonacci-like). Useful for tuning retry behavior in different environments.

Type: Float

Default: 1.0

New in version v4.1.0.

DD_DOWNLOAD_MAX_DELAY#

Maximum delay in seconds between retry attempts. Prevents excessive wait times during exponential backoff.

Type: Integer

Default: 120

New in version v4.1.0.

_DD_DEBUG_EXT#

If set to any non-empty value, enables DebugMetadata timing output in setup.py. Per-phase and per-extension build times are written to the file specified by _DD_DEBUG_EXT_FILE (default: debug_ext_metadata.txt). Useful for diagnosing slow warm builds.

Type: Boolean

Default: (no value)

_DD_DEBUG_EXT_FILE#

Override the output filename for DebugMetadata timing data when _DD_DEBUG_EXT is set.

Type: String

Default: debug_ext_metadata.txt

Using a system-provided libddwaf#

By default the build downloads the prebuilt libddwaf binaries from GitHub releases and bundles the one matching the target architecture into the package. Distribution packagers cannot do that: they build from source with no network access, and package libddwaf separately rather than vendoring it.

The build_py command therefore takes a --no-bundle-libddwaf option. With it, nothing is downloaded and no library is bundled. Being a command option rather than an environment variable, it is set through setup.cfg, which is how it reaches build_py through a PEP 517 frontend such as pip:

[build_py]
no_bundle_libddwaf = 1

It can also be passed on the command line for a direct python setup.py build_py --no-bundle-libddwaf invocation. Default builds read no such section and bundle the library as before.

How the library is found at runtime#

The loader uses the bundled library when the package contains one. When it does not — which is what the option produces, but also what a partial or damaged install looks like — it asks the dynamic linker instead, trying libddwaf.so.2 and then libddwaf.so. Both names are tried because libddwaf’s own CMake sets no SOVERSION: an install built from upstream sources is plain libddwaf.so, while a distribution that adds a SOVERSION ships libddwaf.so.2 in its runtime package and keeps libddwaf.so in -devel. The versioned name is tried first so that the runtime package is preferred over a development symlink.

Because an unversioned SONAME guarantees no ABI, the loader checks ddwaf_get_version() once the library is in: anything that is not 2.x is refused.

What the packaging must guarantee#

  • The library is installed where the dynamic linker looks for it — a normal /usr/lib64 install registered in ld.so.cache is enough, and -devel is not required since the versioned name is tried first. LD_LIBRARY_PATH also works.

  • The package declares a runtime dependency on libddwaf. The library is loaded at import time, not linked at build time, so the build succeeds whether or not libddwaf is installed.

  • libddwaf 2.x, at least 2.0.0: every ddwaf_* symbol ddtrace resolves is present in 2.0.0, whose public header is identical to 2.0.1’s. LIBDDWAF_VERSION in setup.py is the version ddtrace pins and tests against, so prefer that one; a 3.x will need a new ddtrace release.

If the library cannot be loaded, or is not 2.x, AppSec logs a warning and disables itself; the rest of the tracer is unaffected. The version actually loaded is reported in telemetry, so a mismatch is visible.

Linux only: elsewhere the runtime has no system library to fall back to, so the build fails rather than produce a package whose AppSec cannot load.

Debugging Build Performance#

The build system compiles many native extensions (CMake C++, Cython, Rust). In CI, ext_cache.py caches compiled .so files between runs. On a warm run (extensions already in cache), the entire pip install -e . should complete in under 30s.

How the Build Works#

scripts/run-tests
  └─ pip install -e .
       ├─ build_py  → LibraryDownloader.run()
       │    ├─ CleanLibraries.remove_artifacts()  ← SKIPPED when INCREMENTAL=1
       │    └─ LibDDWafDownload.run()
       └─ build_ext → CustomBuildExt.run()
            ├─ build_rust()          → Rust _native extension
            ├─ build_libdd_wrapper() → libdd_wrapper.so (C++)
            ├─ build_shared_deps()   → absl (once, cached by sentinel file)
            └─ super().run()         → build_extension() for every ext
                 ├─ CMakeExtension  → build_extension_cmake()
                 │    └─ skip if .so newer than sources (INCREMENTAL check)
                 └─ Cython/C ext   → skip if .so newer than .pyx sources

ext_cache.py flow (CI and local testing):

ext_cache.py restore  →  copies .so files into source tree
pip install -e .      →  skip checks fire, nothing recompiles
ext_cache.py save     →  copies .so files into cache

Key Files#

File

Purpose

setup.py

CustomBuildExt, CMakeExtension.get_sources(), DebugMetadata, LibraryDownloader

scripts/ext_cache.py

Cache/restore .so files and shared C++ dependency install trees

cmake/abseil/CMakeLists.txt

Standalone abseil build (shared between extensions, built once)

cmake/AbseilDep.cmake

CMake module included by consuming extensions to resolve abseil

Shared Dependency (abseil) Cache#

Abseil is built once via CustomBuildExt.build_shared_deps() and installed to .download_cache/_cmake_deps/absl_install_<arch>. The SharedDep.is_built() sentinel (.dep_build_info file containing a config hash) prevents rebuilding on every run.

scripts/ext_cache.py caches the entire install tree under .ext_cache/shared_deps/absl/<config_hash>/. The config hash is keyed on: version + compile mode + platform + machine arch + ARCHFLAGS.

Known Root Causes of Warm Rebuilds#

  1. ``CleanLibraries.remove_artifacts()`` deletes restored ``.so`` files

    setup.py → LibraryDownloader.run() → CleanLibraries.remove_artifacts() used to wipe all .so files unconditionally on every run, defeating any cache restore. Fixed by guarding with if not CustomBuildExt.INCREMENTAL:.

  2. CMakeExtension skip check gated on ``IS_EDITABLE``

    The skip check if IS_EDITABLE and self.INCREMENTAL never fired during the test environment’s pip install -e . because IS_EDITABLE was never set in that context. Fixed by removing the IS_EDITABLE guard.

  3. Cython-generated ``.c`` files poison CMakeExtension hashes

    cythonize() writes .c files with the current timestamp during the same pip install run. Those .c files appear newer than the just-restored .so, causing a spurious rebuild. Fixed by excluding .c, .so, .dylib, .dll, and .pyd from CMakeExtension.get_sources().

  4. No skip check for Cython/C extensions

    Cython extensions had no mtime-based skip check, so they always rebuilt even when the .so was restored from cache. Fixed by adding a newer_group check using .pyx file mtimes (not .c files, which are touched by cythonize()), plus all .pxd files as invalidation inputs.

  5. Double extension processing

    CustomBuildExt.run() called super().run() then looped over extensions explicitly again. The second pass overwrote timing data in DebugMetadata, masking true rebuild costs. Fixed by removing the explicit loop.