Datadog Agent ingestion

This topic explains how to send traces, metrics, and logs from the Datadog Agent to LaunchDarkly’s observability features.

LaunchDarkly’s observability features are publicly available in early access. Enable observability in the billing page.

They currently require the LaunchDarkly observability SDKs and the JavaScript, React Web, or Vue SDK.

If you are interested in participating in the Early Access Program for our upcoming observability plugins for server-side SDKs, sign up here.

Overview

This topic explains how to send traces, metrics, and logs from the Datadog Agent to LaunchDarkly’s observability features.

LaunchDarkly’s OpenTelemetry Collector includes a Datadog-compatible receiver. If you already run the Datadog Agent in your infrastructure, you can configure it to send APM traces, metrics, and logs to LaunchDarkly without changing your application instrumentation. Once received, this telemetry is available in the Traces, Logs, and observability dashboards in the LaunchDarkly UI.

Prerequisites

Before you configure Datadog Agent ingestion, you must:

• Have LaunchDarkly observability enabled for your project
• Have the Datadog Agent v6.0 or later installed and running in your infrastructure
• Know your LaunchDarkly client-side IDTo find your client-side ID:

1. In the LaunchDarkly UI, click the project dropdown to open the project menu.
2. Select Project settings.
3. Click Environments.
4. Find the environment you want to use and copy the Client-side ID value.

Configure the Datadog Agent

To send telemetry to LaunchDarkly, configure the Datadog Agent to use LaunchDarkly’s Datadog-compatible endpoint. You can replace the standard Datadog intake, or send data to both Datadog and LaunchDarkly simultaneously using dual shipping.Endpoint: datadog.observability.app.launchdarkly.com:8126

Configure using the agent configuration file

To configure the Datadog Agent using the datadog.yaml configuration file, include these lines:

datadog.yaml configuration

    apm_config:
      enabled: true
      apm_dd_url: https://datadog.observability.app.launchdarkly.com:8126

    logs_config:
      logs_dd_url: datadog.observability.app.launchdarkly.com:8126
      use_http: true
      expected_tags_duration: "87600h"

    dd_url: https://datadog.observability.app.launchdarkly.com:8126

Configure using environment variables

If you run the Datadog Agent in a container, you can use environment variables to provide the configuration:

Environment variable configuration

    DD_APM_ENABLED=true
    DD_APM_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
    DD_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
    DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION=87600h

Configure using Docker Compose

If you run the Datadog Agent as a Docker container, add the configuration to your docker-compose.yml file:

docker-compose.yml configuration

    services:
      datadog-agent:
        image: gcr.io/datadoghq/agent:latest
        environment:
          - DD_APM_ENABLED=true
          - DD_APM_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
          - DD_DD_URL=https://datadog.observability.app.launchdarkly.com:8126
          - DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION=87600h
          - DD_API_KEY=placeholder

The Datadog Agent requires a DD_API_KEY value to start, even when routing to a non-Datadog endpoint. You can use any non-empty placeholder string. LaunchDarkly does not validate or use this value.

Sending data to both Datadog and Launch

DarklyThe examples above replace the default Datadog intake with LaunchDarkly’s endpoint. If you want to continue sending data to Datadog while also sending it to LaunchDarkly, you can configure the Datadog Agent to dual ship telemetry to both destinations.

Dual shipping lets you keep your existing Datadog dashboards, alerts, and workflows while also using LaunchDarkly’s observability features and guarded rollouts.

To learn how to configure dual shipping, read Dual Shipping in the Datadog documentation.

When configuring dual shipping, use https://datadog.observability.app.launchdarkly.com:8126 as the additional endpoint for APM traces, metrics, and logs.

Associate telemetry with your Launch

Darkly projectLaunchDarkly uses the launchdarkly.project_id resource attribute to route telemetry to the correct project. Set this to your LaunchDarkly client-side ID.You can set this attribute in your application’s OpenTelemetry SDK configuration, or use the OTEL_RESOURCE_ATTRIBUTES environment variable for services that send data through the agent:

Environment variable

    export OTEL_RESOURCE_ATTRIBUTES="launchdarkly.project_id=YOUR_CLIENT_SIDE_ID"

Alternatively, if you use the OpenTelemetry Collector in front of the Datadog Agent, you can inject this attribute using a resource processor. To learn more, read OpenTelemetry in server-side SDKs.

Attaching feature flag context to traces

To use Datadog trace data with LaunchDarkly features like guarded rollouts and autogenerated metrics, your traces must include feature flag evaluation data. LaunchDarkly uses this data to correlate traces with specific flag evaluations and contexts.

You can attach feature flag context to your Datadog traces by creating a custom SDK hook. The hook instruments each flag evaluation as a child span on your existing Datadog traces, adding attributes that LaunchDarkly uses to connect traces to flag evaluations.

The hook must set the following span attributes on each flag evaluation:

| Attribute

Description

| --- | | feature_flag.key

| The key of the evaluated flag. | | feature_flag.context.id

| The fully-qualified key of the LaunchDarkly context. | | feature_flag.contextKeys

| A JSON object mapping each context kind to its key, for example . | | feature_flag.provider.name

| Set to LaunchDarkly . | | feature_flag.result.value

| The string representation of the evaluation result. |

Go example

This example uses a BeforeEvaluation hook to start a Datadog span with flag metadata, and an AfterEvaluation hook to record the result and finish the span.

Expand Go hook implementation

First, define the hook:

Go Datadog tracing hook

      import (
      	"context"
      	"encoding/json"
      	"fmt"

      	"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
      	"github.com/launchdarkly/go-server-sdk/v7/ldhooks"
      	"github.com/launchdarkly/go-sdk-common/v3/ldreason"
      )

      type DatadogHook struct {
      	ldhooks.Unimplemented
      	metadata ldhooks.Metadata
      }

      func NewDatadogHook() DatadogHook {
      	return DatadogHook{
      		metadata: ldhooks.NewMetadata("datadog-tracing"),
      	}
      }

      func (h DatadogHook) Metadata() ldhooks.Metadata {
      	return h.metadata
      }

      func (h DatadogHook) BeforeEvaluation(
      	ctx context.Context,
      	seriesContext ldhooks.EvaluationSeriesContext,
      	data ldhooks.EvaluationSeriesData,
      ) (ldhooks.EvaluationSeriesData, error) {
      	span, _ := tracer.StartSpanFromContext(ctx, "feature_flag.evaluation",
      		tracer.ResourceName(seriesContext.FlagKey()),
      		tracer.SpanType("launchdarkly"),
      	)

      	ldCtx := seriesContext.Context()
      	contextKeys := make(map[string]string)
      	for i := 0; i < ldCtx.IndividualContextCount(); i++ {
      		if individualCtx := ldCtx.IndividualContextByIndex(i); individualCtx.IsDefined() {
      			contextKeys[string(individualCtx.Kind())] = individualCtx.Key()
      		}
      	}
      	if ldCtx.IndividualContextCount() == 0 && ldCtx.IsDefined() {
      		contextKeys[string(ldCtx.Kind())] = ldCtx.Key()
      	}
      	contextKeysJSON, _ := json.Marshal(contextKeys)

      	span.SetTag("feature_flag.context.id", ldCtx.FullyQualifiedKey())
      	span.SetTag("feature_flag.contextKeys", string(contextKeysJSON))
      	span.SetTag("feature_flag.key", seriesContext.FlagKey())
      	span.SetTag("feature_flag.provider.name", "LaunchDarkly")

      	return ldhooks.NewEvaluationSeriesBuilder(data).Set("span", span).Build(), nil
      }

      func (h DatadogHook) AfterEvaluation(
      	ctx context.Context,
      	seriesContext ldhooks.EvaluationSeriesContext,
      	data ldhooks.EvaluationSeriesData,
      	detail ldreason.EvaluationDetail,
      ) (ldhooks.EvaluationSeriesData, error) {
      	if spanData, ok := data.Get("span"); ok {
      		if span, ok := spanData.(*tracer.Span); ok {
      			span.SetTag("feature_flag.result.value",
      				fmt.Sprintf("%v", detail.Value.AsArbitraryValue()))
      			span.Finish()
      		}
      	}
      	return data, nil
      }

Then register the hook when you initialize the LaunchDarkly client:

Registering the hook

      import (
      	ld "github.com/launchdarkly/go-server-sdk/v7"
      	"github.com/launchdarkly/go-server-sdk/v7/ldhooks"
      )

      ldConfig := ld.Config{
      	Hooks: []ldhooks.Hook{NewDatadogHook()},
      }

      ldClient, err := ld.MakeCustomClient("YOUR_SDK_KEY", ldConfig, 5*time.Second)

When you evaluate flags, use the context-aware variation methods to pass the request context through to the hook:

Evaluating a flag with context

      value, err := ldClient.BoolVariationCtx(ctx, "my-flag-key", ldContext, false)

Configuring the Datadog tracer

Configure the Datadog tracer in your application to send traces to LaunchDarkly’s Datadog-compatible endpoint. Set the X-LaunchDarkly-Project global tag to your LaunchDarkly project ID so that traces are routed to the correct project:

Go tracer configuration

    import "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"

    tracer.Start(
    	tracer.WithAgentURL("https://datadog.observability.app.launchdarkly.com:8126"),
    	tracer.WithService("your-service-name"),
    	tracer.WithEnv("production"),
    	tracer.WithGlobalTag("X-LaunchDarkly-Project", "YOUR_PROJECT_ID"),
    )
    defer tracer.Stop()

Using flag data with guarded rollouts

After you configure the hook and tracer, LaunchDarkly automatically processes your Datadog trace data and associates it with the feature flags and contexts in your evaluations. This enables you to:

• Use autogenerated OpenTelemetry metrics, such as HTTP error rates and latencies, with guarded rollouts
• Monitor how flag changes impact your application performance in Traces
• Correlate errors and latency regressions with specific flag evaluations and context attributesTo learn more about setting up guarded rollouts, read Creating guarded rollouts.

What data is collected

The Datadog Agent sends the following telemetry types to LaunchDarkly:

• Traces: APM traces from your instrumented services, visible in Traces
• Metrics: Infrastructure and application metrics
• Logs: Application logs collected by the agent log collection feature, visible in Logs

Verify that data is being received

After you configure the Datadog Agent, telemetry begins flowing to LaunchDarkly.

To verify that traces are being received:

1. In the LaunchDarkly UI, expand Observe in the left navigation.
2. Click Traces.
3. Look for traces from your Datadog-instrumented services.

To verify that logs are being received:

1. In the LaunchDarkly UI, expand Observe in the left navigation.
2. Click Logs.
3. Look for logs from your services.

It may take a few minutes for data to appear after you first configure the agent.

Filtering ingested data

You can configure ingestion filters to control which logs are stored in LaunchDarkly. This is useful for reducing noise or staying within your observability quotas.

To learn more, read Filters.