> ## Documentation Index
> Fetch the complete documentation index at: https://launchdarkly-preview.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Datadog Agent ingestion

> This topic explains how to send traces, metrics, and logs from the Datadog Agent to LaunchDarkly's observability features.

> For the complete documentation index, see [llms.txt](/llms.txt). For the full documentation snapshot, see [llms-full.txt](/llms-full.txt).

<Note title="This feature is in Early Access">
  LaunchDarkly's observability features are publicly available in early access. Enable observability in the billing page.

  They currently require the LaunchDarkly [observability SDKs](/sdk/observability) and the JavaScript, React Web, or Vue SDK.

  If you are interested in participating in the Early Access Program for our upcoming observability plugins for server-side SDKs, [sign up here](https://launchdarkly.com/early-access/).
</Note>

## Overview

This topic explains how to send traces, metrics, and logs from the [Datadog Agent](https://docs.datadoghq.com/agent/) to LaunchDarkly's observability features.

LaunchDarkly's OpenTelemetry Collector includes a Datadog-compatible receiver. If you already run the Datadog Agent in your infrastructure, you can configure it to send APM traces, metrics, and logs to LaunchDarkly without changing your application instrumentation. Once received, this telemetry is available in the [Traces](/home/observability/traces), [Logs](/home/observability/logs), and observability dashboards in the LaunchDarkly UI.

## Prerequisites

Before you configure Datadog Agent ingestion, you must:

* Have LaunchDarkly observability enabled for your project
* Have the [Datadog Agent](https://docs.datadoghq.com/agent/) v6.0 or later installed and running in your infrastructure
* Know your LaunchDarkly client-side IDTo find your client-side ID:

1. In the LaunchDarkly UI, click the project dropdown to open the project menu.
2. Select **Project settings**.
3. Click **Environments**.
4. Find the environment you want to use and copy the **Client-side ID** value.

## Configure the Datadog Agent

To send telemetry to LaunchDarkly, configure the Datadog Agent to use LaunchDarkly's Datadog-compatible endpoint. You can replace the standard Datadog intake, or send data to both Datadog and LaunchDarkly simultaneously using dual shipping.**Endpoint:** `datadog.observability.app.launchdarkly.com:8126`

### Configure using the agent configuration file

To configure the Datadog Agent using the `datadog.yaml` configuration file, include these lines:

<CodeGroup>
  ```yaml title="datadog.yaml configuration" theme={null}
      apm_config:
        enabled: true
        apm_dd_url: https://datadog.observability.app.launchdarkly.com:8126

      logs_config:
        logs_dd_url: datadog.observability.app.launchdarkly.com:8126
        use_http: true
        expected_tags_duration: "87600h"

      dd_url: https://datadog.observability.app.launchdarkly.com:8126
  ```
</CodeGroup>

### Configure using environment variables

If you run the Datadog Agent in a container, you can use environment variables to provide the configuration:

<CodeGroup>
  ```bash title="Environment variable configuration" theme={null}
      DD_APM_ENABLED=true
      DD_APM_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
      DD_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
      DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION=87600h
  ```
</CodeGroup>

### Configure using Docker Compose

If you run the Datadog Agent as a Docker container, add the configuration to your `docker-compose.yml` file:

<CodeGroup>
  ```yaml title="docker-compose.yml configuration" theme={null}
      services:
        datadog-agent:
          image: gcr.io/datadoghq/agent:latest
          environment:
            - DD_APM_ENABLED=true
            - DD_APM_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
            - DD_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
            - DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION=87600h
            - DD_API_KEY=placeholder
  ```
</CodeGroup>

<Info title="DD_API_KEY is required">
  The Datadog Agent requires a `DD_API_KEY` value to start, even when routing to a non-Datadog endpoint. You can use any non-empty placeholder string. LaunchDarkly does not validate or use this value.
</Info>

### Sending data to both Datadog and Launch

DarklyThe examples above replace the default Datadog intake with LaunchDarkly's endpoint. If you want to continue sending data to Datadog while also sending it to LaunchDarkly, you can configure the Datadog Agent to dual ship telemetry to both destinations.

Dual shipping lets you keep your existing Datadog dashboards, alerts, and workflows while also using LaunchDarkly's observability features and guarded rollouts.

To learn how to configure dual shipping, read [Dual Shipping](https://docs.datadoghq.com/agent/configuration/dual-shipping/) in the Datadog documentation.

When configuring dual shipping, use `https://datadog.observability.app.launchdarkly.com:8126` as the additional endpoint for APM traces, metrics, and logs.

## Associate telemetry with your Launch

Darkly projectLaunchDarkly uses the `launchdarkly.project_id` resource attribute to route telemetry to the correct project. Set this to your LaunchDarkly client-side ID.You can set this attribute in your application's OpenTelemetry SDK configuration, or use the `OTEL_RESOURCE_ATTRIBUTES` environment variable for services that send data through the agent:

<CodeGroup>
  ```bash title="Environment variable" theme={null}
      export OTEL_RESOURCE_ATTRIBUTES="launchdarkly.project_id=YOUR_CLIENT_SIDE_ID"
  ```
</CodeGroup>

Alternatively, if you use the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) in front of the Datadog Agent, you can inject this attribute using a `resource` processor. To learn more, read [OpenTelemetry in server-side SDKs](/sdk/features/opentelemetry-server-side).

## Attaching feature flag context to traces

To use Datadog trace data with LaunchDarkly features like [guarded rollouts](/home/releases/guarded-rollouts) and [autogenerated metrics](/home/metrics/autogen/opentelemetry), your traces must include feature flag evaluation data. LaunchDarkly uses this data to correlate traces with specific flag evaluations and contexts.

You can attach feature flag context to your Datadog traces by creating a custom [SDK hook](/sdk/features/hooks). The hook instruments each flag evaluation as a child span on your existing Datadog traces, adding attributes that LaunchDarkly uses to connect traces to flag evaluations.

The hook must set the following span attributes on each flag evaluation:

\| Attribute

| Description |
| - |

\| --- |
\| <code>feature\_flag.key</code>

\| The key of the evaluated flag. |
\| <code>feature\_flag.context.id</code>

\| The fully-qualified key of the LaunchDarkly context. |
\| <code>feature\_flag.contextKeys</code>

\| A JSON object mapping each context kind to its key, for example <code>{'{"user":"user-123"}'}</code> . |
\| <code>feature\_flag.provider.name</code>

\| Set to <code>LaunchDarkly</code> . |
\| <code>feature\_flag.result.value</code>

\| The string representation of the evaluation result. |

### Go example

This example uses a `BeforeEvaluation` hook to start a Datadog span with flag metadata, and an `AfterEvaluation` hook to record the result and finish the span.

<Accordion title="Expand Go hook implementation">
  First, define the hook:

  <CodeGroup>
    ```go title="Go Datadog tracing hook" theme={null}
          import (
          	"context"
          	"encoding/json"
          	"fmt"

          	"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
          	"github.com/launchdarkly/go-server-sdk/v7/ldhooks"
          	"github.com/launchdarkly/go-sdk-common/v3/ldreason"
          )

          type DatadogHook struct {
          	ldhooks.Unimplemented
          	metadata ldhooks.Metadata
          }

          func NewDatadogHook() DatadogHook {
          	return DatadogHook{
          		metadata: ldhooks.NewMetadata("datadog-tracing"),
          	}
          }

          func (h DatadogHook) Metadata() ldhooks.Metadata {
          	return h.metadata
          }

          func (h DatadogHook) BeforeEvaluation(
          	ctx context.Context,
          	seriesContext ldhooks.EvaluationSeriesContext,
          	data ldhooks.EvaluationSeriesData,
          ) (ldhooks.EvaluationSeriesData, error) {
          	span, _ := tracer.StartSpanFromContext(ctx, "feature_flag.evaluation",
          		tracer.ResourceName(seriesContext.FlagKey()),
          		tracer.SpanType("launchdarkly"),
          	)

          	ldCtx := seriesContext.Context()
          	contextKeys := make(map[string]string)
          	for i := 0; i < ldCtx.IndividualContextCount(); i++ {
          		if individualCtx := ldCtx.IndividualContextByIndex(i); individualCtx.IsDefined() {
          			contextKeys[string(individualCtx.Kind())] = individualCtx.Key()
          		}
          	}
          	if ldCtx.IndividualContextCount() == 0 && ldCtx.IsDefined() {
          		contextKeys[string(ldCtx.Kind())] = ldCtx.Key()
          	}
          	contextKeysJSON, _ := json.Marshal(contextKeys)

          	span.SetTag("feature_flag.context.id", ldCtx.FullyQualifiedKey())
          	span.SetTag("feature_flag.contextKeys", string(contextKeysJSON))
          	span.SetTag("feature_flag.key", seriesContext.FlagKey())
          	span.SetTag("feature_flag.provider.name", "LaunchDarkly")

          	return ldhooks.NewEvaluationSeriesBuilder(data).Set("span", span).Build(), nil
          }

          func (h DatadogHook) AfterEvaluation(
          	ctx context.Context,
          	seriesContext ldhooks.EvaluationSeriesContext,
          	data ldhooks.EvaluationSeriesData,
          	detail ldreason.EvaluationDetail,
          ) (ldhooks.EvaluationSeriesData, error) {
          	if spanData, ok := data.Get("span"); ok {
          		if span, ok := spanData.(*tracer.Span); ok {
          			span.SetTag("feature_flag.result.value",
          				fmt.Sprintf("%v", detail.Value.AsArbitraryValue()))
          			span.Finish()
          		}
          	}
          	return data, nil
          }
    ```
  </CodeGroup>

  Then register the hook when you initialize the LaunchDarkly client:

  <CodeGroup>
    ```go title="Registering the hook" theme={null}
          import (
          	ld "github.com/launchdarkly/go-server-sdk/v7"
          	"github.com/launchdarkly/go-server-sdk/v7/ldhooks"
          )

          ldConfig := ld.Config{
          	Hooks: []ldhooks.Hook{NewDatadogHook()},
          }

          ldClient, err := ld.MakeCustomClient("YOUR_SDK_KEY", ldConfig, 5*time.Second)
    ```
  </CodeGroup>

  When you evaluate flags, use the context-aware variation methods to pass the request context through to the hook:

  <CodeGroup>
    ```go title="Evaluating a flag with context" theme={null}
          value, err := ldClient.BoolVariationCtx(ctx, "my-flag-key", ldContext, false)
    ```
  </CodeGroup>
</Accordion>

### Configuring the Datadog tracer

Configure the Datadog tracer in your application to send traces to LaunchDarkly's Datadog-compatible endpoint. Set the `X-LaunchDarkly-Project` global tag to your LaunchDarkly project ID so that traces are routed to the correct project:

<CodeGroup>
  ```go title="Go tracer configuration" theme={null}
      import "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"

      tracer.Start(
      	tracer.WithAgentURL("https://datadog.observability.app.launchdarkly.com:8126"),
      	tracer.WithService("your-service-name"),
      	tracer.WithEnv("production"),
      	tracer.WithGlobalTag("X-LaunchDarkly-Project", "YOUR_PROJECT_ID"),
      )
      defer tracer.Stop()
  ```
</CodeGroup>

### Using flag data with guarded rollouts

After you configure the hook and tracer, LaunchDarkly automatically processes your Datadog trace data and associates it with the feature flags and contexts in your evaluations. This enables you to:

* Use [autogenerated OpenTelemetry metrics](/home/metrics/autogen/opentelemetry), such as HTTP error rates and latencies, with [guarded rollouts](/home/releases/guarded-rollouts)
* Monitor how flag changes impact your application performance in [Traces](/home/observability/traces)
* Correlate errors and latency regressions with specific flag evaluations and context attributesTo learn more about setting up guarded rollouts, read [Creating guarded rollouts](/home/releases/creating-guarded-rollouts).

## What data is collected

The Datadog Agent sends the following telemetry types to LaunchDarkly:

* **Traces**: APM traces from your instrumented services, visible in [Traces](/home/observability/traces)
* **Metrics**: Infrastructure and application metrics
* **Logs**: Application logs collected by the agent log collection feature, visible in [Logs](/home/observability/logs)

## Verify that data is being received

After you configure the Datadog Agent, telemetry begins flowing to LaunchDarkly.

To verify that traces are being received:

1. In the LaunchDarkly UI, expand **Observe** in the left navigation.
2. Click **Traces**.
3. Look for traces from your Datadog-instrumented services.

To verify that logs are being received:

1. In the LaunchDarkly UI, expand **Observe** in the left navigation.
2. Click **Logs**.
3. Look for logs from your services.

It may take a few minutes for data to appear after you first configure the agent.

## Filtering ingested data

You can configure ingestion filters to control which logs are stored in LaunchDarkly. This is useful for reducing noise or staying within your observability quotas.

To learn more, read [Filters](/home/observability/settings#filters).