How To Write an Integration#

An integration should provide concise, insightful data about the library or framework that will aid developers in monitoring their application’s health and performance.

The best way to get started writing a new integration is to refer to existing integrations. Looking at the integration for a similar library or framework is a great starting point. To write a new integration for memcached, for example, we might refer to the existing redis integration as a starting point since both are datastores that focus on in-memory storage.

Your development process might look like this:

  • Research the library or framework that you’re instrumenting. Reading through its docs and code examples will reveal what APIs are meaningful to instrument.

  • Copy the skeleton module provided in templates/integration and replace foo with the integration name. The integration name typically matches the library or framework being instrumented:

    cp -r templates/integration ddtrace/contrib/<integration>
    
  • Create a test file for the integration under tests/contrib/<integration>/test_<integration>.py.

  • Write the integration (see more on this below).

  • Open a pull request containing your changes.

All integrations live in ddtrace/contrib/ and contain at least two files, __init__.py and patch.py. A skeleton integration is available under templates/integration which can be used as a starting point:

cp -r templates/integration ddtrace/contrib/<integration>

The Pin API in ddtrace.pin is used to configure the instrumentation at runtime, including enabling and disabling the instrumentation and overriding the service name.

Library support#

ddtrace supports versions of integrated libraries according to these guidelines:

  • Test the oldest and latest minor versions of the most latest major version going back 2 years.

  • Test the latest minor version of any previous major version going back 2 years.

  • If there are no new releases in the past 2 years, test the latest released version.

  • For legacy Python versions (2.7,3.5,3.6), test the latest minor version known to support that legacy Python version.

For libraries with many versions it is recommended to pull out the version of the library to use when instrumenting volatile features. A great example of this is the Flask integration:

Exceptions and Errors#

Exceptions provide a lot of useful information and are usually easy to deal with. Exceptions are a great place to start instrumenting. There are a couple of considerations when dealing with exceptions in ddtrace.

Re-raise exceptions when your integration code catches them, because it is crucial that ddtrace does not change the contract between the application and the integrated library. See the bottle exception handling instrumentation for an example.

Gather relevant information from exceptions. ddtrace provides a helper for pulling out common exception data and adding it to a span. See the cassandra exception handling instrumentation for an example.

Tracing across execution boundaries#

Some integrations need to propagate traces across execution boundaries, to other threads, processes, tasks, or other units of execution. For example, here’s how the requests integration handles this, and here’s the django integration’s implementation.

Web frameworks#

A web framework integration should:

  • Install the WSGI or ASGI trace middlewares already provided by ddtrace

  • Emit a trace covering the entire duration of a request

  • Assign one resource name per application “route” (a “route” is a URL pattern on which the application listens for requests)

  • Use trace_utils.set_http_meta to set the standard http tags

  • Set an internal service name

  • Support configurable distributed tracing

  • Provide insight to middlewares and views, if applicable

  • Use the SpanTypes.WEB span type

Some example web framework integrations::

Database libraries#

ddtrace already provides base instrumentation for the Python database API (PEP 249) which most database client libraries implement in the ddtrace.contrib.dbapi module.

Check out some of our existing database integrations for how to use the dbapi:

Testing#

The tests for your integration should be defined in their own module at tests/contrib/<integration>/.

Testing is the most important part of the integration. We have to be certain that the integration submits meaningful information to Datadog and does not impact the library or application by disturbing state, performance or causing errors. The integration should be invisible to users.

Snapshot Tests#

Many of the tests are based on “snapshots”: saved copies of actual traces sent to the APM test agent.

To update the snapshots expected by a test, first update the library and test code to generate new traces. Then, delete the snapshot file corresponding to your test. Use docker-compose up -d testagent to start the APM test agent, and re-run the test. Use –pass-env as described here to ensure that your test run can talk to the test agent. Once the run finishes, the snapshot file will have been regenerated.

Writing Integration Tests for Your Integration#

These instructions describe the general approach of writing new integration tests for a library integration. They use the Flask integration tests as a teaching example. Referencing these instructions against tests/contrib/flask/test_flask_snapshot.py and tests/contrib/flask/app.py may be helpful.

  1. Make sure a directory for your integration exists under tests/contrib

  2. Create a new file tests/contrib/<integration>/test_<integration>_snapshot.py

  3. Make sure a Venv instance exists in riotfile.py that references your contrib subdirectory. Create one if it doesn’t exist. Note the name of this Venv - this is the “test suite name”.

  4. In this directory, write a simple “Hello World” application that uses the library you’re integrating with similarly to how customers will use it. Depending on the library, this might be as simple as a function in the snapshot test file that imports the library. It might also be a new file in the test directory app.py as in the cases of Flask or Gunicorn.

  5. Instrument your “hello world” app with ddtrace. In the case of Flask, this is accomplished by running the app server in a subprocess started with a ddtrace-run command. The app server is started by a Pytest fixture function that’s defined in the snapshot test file.

  6. If the library you’re integrating with requires communication with a datastore, make sure there’s an image for that datastore referenced in docker-compose.yml. If there is not, add one. You can find a suitable image by searching on Dockerhub.

  7. Write a simple test. In your new snapshot test file, define a function testing your app’s happy path. Here’s an example from the Flask test suite:

@pytest.mark.snapshot
def test_flask_200(flask_client):
    assert flask_client.get("/", headers=DEFAULT_HEADERS).status_code == 200

This function accepts a client object, defined elsewhere in the file, as a fixture. The client has been initialized to communicate with the server running the “hello world” app we created in step 3. The function makes a simple request to the app server and checks the status code.

  1. Add the pytest.mark.snapshot decorator to your test function.

@pytest.mark.snapshot
def test_flask_200(flask_client):
    ...

This decorator causes Pytest to collect the spans generated by your instrumented test app and compare them against a stored set of expected spans. Since the integration test we’re writing is new, there are not yet any expected spans stored for it, so we need to create some.

  1. Start the “test agent”, as well as any necessary datastore containers, and run your new test:

$ docker-compose up -d testagent <container>
$ scripts/ddtest
> DD_AGENT_PORT=9126 riot -v run --pass-env <test_suite_name>
  1. Check git status and observe that some new files have been created under tests/snapshots/. These files contain JSON representations of the spans created by the instrumentation that ran during your test function. Look over these spans to make sure that they’re what you’d expect from the integration.

  2. Commit the new snapshot files. The next time the snapshot test runs, it will compare the real spans generated by the test to these committed span JSON objects, and will fail on any differences found.

  3. Test that this works: delete any attribute from one of the snapshot JSON objects, and then run the test again. You should observe that the test fails with a message indicating that the received and expected spans do not match.

  4. Repeat steps 7 through 9 until you’ve achieved test coverage for the entire “happy path” of normal usage for the library you’re integrating with, as well as coverage of any known likely edge cases.

Trace Examples#

If in the process of writing tests for your integration you create a sample application, consider adding it to the trace examples repository along with screenshots of some example traces in the PR description.