Quickstart for AI Configs

This topic explains how to get started with the LaunchDarkly AI Configs product.

Overview

This topic explains how to get started with the LaunchDarkly AI Configs product.

You can use AI Configs to customize, test, and roll out new large language models (LLMs) within your generative AI applications. Use AI Configs to:

• Manage your model configuration and messages outside of your application code.
• Upgrade your app to the newest model version, then gradually release to your customers.
• Add a new model provider and progressively move your production traffic.
• Compare variations to find the best-performing prompt and model.

AI Configs can be created in two modes:

• Completion mode: Configure prompts with messages and roles for single-step model responses. Completion mode supports multi-message prompts, including chat-style prompts. In LaunchDarkly, completion mode is a product mode and does not map to a specific provider API.
• Agent mode: Configure structured multi-step workflows using instructions. Agent mode is designed for coordinating structured, multi-step workflows within your application. For agent-based variations, invoke a judge programmatically using the AI SDK. To learn more, read Agents in AI Configs.

Tool integration depends on your SDK and model provider integration, not on the AI Config mode. Agent mode is optimized for structured multi-step workflows, but tool usage is not limited to agent mode.

This quickstart focuses on AI Configs in completion mode.

AI Configs can also be created in agent mode, which uses instructions to define multi-step workflows. Agent mode is documented separately in Agents in AI Configs.

You can attach judges to completion-mode AI Config variations in the LaunchDarkly UI. For other variations, invoke a judge programmatically using the AI SDK. To learn more, read Online evaluations in AI Configs.

With AI Configs, messages and model configuration are customized for each end user at runtime. You can update messages and model configuration without redeploying your application.

Working with AI Configs requires a role that allows AI Config actions. The LaunchDarkly Project Admin, Maintainer, and Developer project roles, as well as the Admin and Owner base roles, include this permission.

Follow the steps below to incorporate AI Configs into your app, or use the in-app onboarding experience to set up your first AI Config directly in the LaunchDarkly UI.

To learn from an example that’s built on a specific generative AI application, read one of our guides:

• Managing AI model configuration outside of code with the Node.js AI SDK
• Using targeting to manage AI model usage by tier with the Python AI SDK

You can use AI Configs with any model provider, including Gemini, OpenAI, and Anthropic. To learn how to use OpenAI or Anthropic models with AI Configs, read the examples in Customize the AI Config. To learn how to use Gemini models, including how to cache prompts, read Use Gemini prompt caching with AI Configs.

You can also use the LaunchDarkly demo sandbox.

AI Configs is an add-on feature. Access depends on your organization’s LaunchDarkly plan. If AI Configs does not appear in your project, your organization may not have access to it.

To enable AI Configs for your organization, contact your LaunchDarkly account team. They can confirm eligibility and assist with activation.

For information about pricing, visit the LaunchDarkly pricing page or contact your LaunchDarkly account team.

Step 1, in your app: Install an AI SDK

First, install one of the LaunchDarkly AI SDKs in your app:

.NET AI SDK

    Install-Package LaunchDarkly.ServerSdk
    Install-Package LaunchDarkly.ServerSdk.Ai

Next, import the LaunchDarkly AI client in your app and initialize a single, shared instance of it:

.NET AI SDK

    using LaunchDarkly.Sdk.Server.Ai;
    using LaunchDarkly.Sdk.Server.Ai.Adapters;
    using LaunchDarkly.Sdk.Server.Ai.Config;

    var baseClient = new LdClient(Configuration.Builder("sdk-key-123").StartWaitTime(TimeSpan.FromSeconds(5)).Build());
    var aiClient = new LdAiClient(new LdClientAdapter(baseClient));

The AI SDKs each require that you specify your SDK key to authorize your application to connect to a particular environment within LaunchDarkly. SDK keys are specific to each project and environment. They are available from Project settings, on the Environments list in the LaunchDarkly UI. To learn more, read Keys.

Then, set up the context. Contexts are the people or resources who will encounter generated AI content in your application. The context attributes determine which variation of the AI Config LaunchDarkly serves to the end user, based on the targeting rules in your AI Config. If you are using template variables in the messages in your AI Config’s variations, the context attributes also fill in values for the template variables.

Here’s how:

.NET AI SDK

    using LaunchDarkly.Sdk;

    var context = Context.Builder("example-user-key")
      .Kind("user")
      .Name("Sandy")
      .Build();

Step 2, in Launch

Darkly: Create an AI Config

Next, create an AI Config in LaunchDarkly:

1. In the left navigation, select AI Configs, then click Create AI Config.

2. In the “Create AI Config” dialog, select Completion. This quickstart covers completion mode only. For agent mode, read Agents in AI Configs.

3. Enter a name for your AI Config, and optionally assign a maintainer.

4. Click Create.

After you create the AI Config, the Variations tab appears.

Then, create a variation. Every completion-based AI Config has one or more variations. Each variation includes a model configuration and, optionally, one or more messages.

Here’s how:

1. In the Variations tab, replace “Untitled variation” with a variation Name. You’ll use this to refer to the variations when you set up targeting rules, below.
2. Click Select a model and choose the model to use.
   • LaunchDarkly provides a list of common models, and updates it regularly.
   • You can also choose + Add a model and create your own. To learn more, read Create AI model configurations.
3. (Optional) Select a message role and enter the message for the variation. If you’d like to customize the message at runtime, use {{ example_variable }} or {{ ldctx.example_context_attribute }} within the message. The LaunchDarkly AI SDK will substitute the correct values when you customize the AI Config from within your app.
   • To learn more about how variables and context attributes are inserted into messages at runtime, read Customizing AI Configs.
4. Click Review and save.

You can also select Import from playground in your variation to bring your model, model parameters, and messages from an external playground.

Here’s an example of a completed variation:

Expand to copy variation message

Here’s the variation message for this example. You can copy this if you’re working through this quickstart in your own project:

jokes' variation, system message"

      You are a system designed to tell knock-knock jokes. Each joke should be addressed to {{ ldctx.name }} and include at least one {{ example_variable }}.

Step 3, in Launch

Darkly: Set up targeting rules

Next, set up targeting rules for your AI Config. These rules determine which of your customers receive a particular variation of your AI Config.

To specify the AI Config variation to use by default when the AI Config is toggled on:

1. Select the Targeting tab for your AI Config.
2. In the “Default rule” section, click Edit.
3. Configure the default rule to serve a specific variation.
4. Click Review and save.

Here is an example of a default rule:

The AI Config is enabled by default. After you’ve added code to your application to pull the customized messages and model configuration from LaunchDarkly, your AI Config will be active.

When an end user opens your application, they’ll get the AI Config variation you’ve defined, either in the default rule or in a custom targeting rule. Your app’s AI content generation will use the model and messages from the AI Config variation, and the messages will be customized for each end user.

When you’re ready to add more AI Config variations, come back to this step and set up additional targeting rules. If you are familiar with LaunchDarkly’s flag targeting, the process is very similar: with AI Configs, you can target individuals or segments, or target contexts with custom rules. To learn how, read Target with AI Configs.

You can release a new variation gradually using a guarded rollout. Guarded rollouts reduce risk and let you monitor key metrics as the change rolls out.

Step 4, in your app: Customize the AI Config and track metrics

Now that your AI Config is set up, you can use it in your app:

1. Customize the AI Config: First, use your SDK’s completion-mode config function to customize the AI Config. This function returns the customized messages and model configuration along with a tracker instance for recording metrics.
2. Call your generative AI model, track metrics: Then, call your generative AI model, passing in the result of the config function. In the LaunchDarkly AI SDKs, you use a track[Model]Metrics function to record metrics from your AI model generation. This function takes a completion from your AI model generation.

Customize the AI Config

In your application code, call your SDK’s completion-mode config function each time you generate content. This lets LaunchDarkly evaluate targeting and return the latest messages and model settings.

The completion-mode config function returns the customized messages and model configuration along with a tracker instance you can use to record metrics. Customization means that any variables you include in the AI Config variation’s messages have their values set to the context attributes and variables you pass to your SDK’s function. Then, you can pass the customized messages directly to your AI. Set up a fallback value to use in case of an error, and handle the fallback case appropriately in your application.

Here’s how:

.NET AI SDK

    var fallbackConfig = LdAiConfig.New()
      .SetEnabled(false)
      .Build();

    var tracker = aiClient.Config(
      "example-ai-config-key",
      context,
      fallbackConfig,
      new Dictionary<string, object> {
        { "example_variable", "puppy" }
      }
    );

    // Based on the example AI Config variation shown in step 2,
    // tracker.Config.Messages[0].Content will be:
    // "You are a model designed to tell knock-knock jokes. Each joke should be addressed to Sandy and include at least one puppy."

Call your generative AI model, track metrics

Next, make a call to your generative AI model and pass in the result of your SDK’s completion-mode config function.

Use one of the track[Model]Metrics or TrackRequest functions to record metrics from your AI model generation. These functions take the response from your model provider. Call your SDK’s completion-mode config function each time you generate content so LaunchDarkly can evaluate targeting and return the latest messages and model settings.

LaunchDarkly provides specific functions for completions for several common AI model families, and an option to record this information yourself.

Here’s how to use a provider-specific track[Model]Metrics function to call a supported provider or framework, such as OpenAI, Amazon Bedrock, or LangChain, and record metrics from your AI model generation:

Node.js AI SDK (TypeScript), OpenAI model

    // Call completionConfig() each time you want to call the OpenAI operation

    const aiConfig = await aiClient.completionConfig(
      'example-ai-config-key',
      context,
      fallbackConfig,
      { example_variable: 'elephant' },
    );

    const { tracker } = aiConfig;

    if (aiConfig.enabled) {
      try {
        const completion = await tracker.trackOpenAIMetrics(async () =>
          client.chat.completions.create({
            messages: aiConfig.messages || [],
            model: aiConfig.model?.name || 'gpt-4',
            temperature: (aiConfig.model?.parameters?.temperature as number) ?? 0.5,
            max_tokens: (aiConfig.model?.parameters?.maxTokens as number) ?? 4096,
          }),
        );

        // use completion here
      } catch (err) {
        // handle exception from OpenAI or LaunchDarkly tracker
      }
    } else {
      // Application path to take when the aiConfig is disabled
    }

Here’s how to use the general TrackRequest function to call any AI model provider and record metrics from your AI model generation:

.NET AI SDK, any model

    if (tracker.Config.Enabled == true) {

      var response = tracker.TrackRequest(() =>
        {
          // Make request to a provider, which automatically tracks metrics in LaunchDarkly.
          // When sending the request to a provider, use details from tracker.Config.
          // For instance, you can pass tracker.Config.Model and tracker.Config.Messages.
          // Optionally, return response metadata, for example to do your own additional logging.
          //
          // CAUTION: If the call inside of Task.Run() throws an exception,
          // the SDK will re-throw that exception.

          return new Response
          {
            Usage = new Usage { Total = 1, Input = 1, Output = 1 }, /* Token usage data */
            Metrics = new Metrics { LatencyMs = 100 } /* Metrics data */
          };
        }
      ));

    } else {

      // Application path to take when the tracker.Config is disabled

    }

    // Call Config() again each time you want to call the generative AI operation.
    var tracker = aiClient.Config(
      "example-ai-config-key",
      context,
      fallbackConfig,
      new Dictionary<string, object> {
        { "example_variable", "elephant" }
      }
    );

    var response = tracker.TrackRequest(...)

Whether you use track[Model]Metrics or TrackRequest, the SDK automatically flushes these pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, such as a script or unit test, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using flush() or close().

Here’s how:

.NET AI SDK

    baseClient.Flush();

LaunchDarkly provides specific functions for completions for several common AI model families, including OpenAI, Bedrock, and LangChain, and an option to record this information yourself.

If you are using another provider, or want to record additional metrics, each AI SDK also includes individual track* methods to record duration, token usage, generation success, generation error, time to first token, output satisfaction, and more. To learn more, read AI metrics.

For a complete example application, you can review some of our sample applications:

• Node.js AI SDK, using OpenAI
• Node.js AI SDK, using Bedrock
• Python AI SDK, using OpenAI
• Python AI SDK, using Bedrock
• Python AI SDK, using LangChain
• Python AI SDK, using Gemini
• Ruby AI SDK, using Bedrock
• Ruby AI SDK, using OpenAI

Use Gemini prompt caching with AI Configs

To use Gemini’s explicit prompt caching with AI Configs, we recommend structuring your variation with two system messages:

1. Use the first system message (messages[0]) for static content that should be cached.
2. Use the second system message (messages[1]) for dynamic content that changes per request, such as user input or context variables.

In your application, call aiClient.completionConfig() to evaluate the AI Config. Extract messages[0] and pass it to Gemini’s system_instruction field or use it when creating a cached prompt using the Gemini SDK. Use messages[1] as your dynamic prompt content.

To detect when the cached prompt should be refreshed, access tracker.config.version. This value changes when LaunchDarkly serves a different variation or when the variation’s content is updated. Alternatively, compare the role and content of messages[0] to a previously cached version.

Step 5, in Launch

Darkly: Monitor the AI Config

Select the Monitoring tab for your AI Config. When end users use your application, LaunchDarkly monitors AI Config performance. Metrics update approximately every minute.

To score response quality in production, you can attach judges to completion-mode AI Config variations in the LaunchDarkly UI. For other variations, invoke a judge programmatically using the AI SDK.

Each evaluated response sends an additional request to your model provider, which increases token usage and provider costs. Start with a lower sampling percentage and increase it only if you need more evaluation coverage.

Attach judges for online evaluations

You can evaluate the quality of AI Config responses by attaching judges.

Judges are AI Configs that evaluate responses generated by other AI Configs. When a response is generated, an attached judge runs automatically and uses its evaluator prompt to generate a numeric score, for example to indicate how accurate or relevant the response is. LaunchDarkly records these judge scores as evaluation metrics.

These evaluation metrics appear on the Monitoring tab alongside other AI Config metrics. You can use them to monitor quality over time, compare variations, and assess the impact of changes during guarded rollouts and experiments.

Use judges when you want to:

• Monitor response quality in production without manual review
• Detect quality regressions after prompt or model changes
• Compare variations using automated quality scores
• Apply consistent quality standards across teams and environments

You can attach judges to completion-mode AI Config variations in the LaunchDarkly UI. For other variations, invoke a judge programmatically using the AI SDK.

See how online evaluations run in your application

After you attach judges to a completion-mode AI Config variation in the LaunchDarkly UI, your application continues to use the same invocation flow. You do not need to call judges directly.

For sampled requests, attached judges run automatically and record evaluation results.

Completion-mode AI Config with automatic judge evaluation

    chat = await ai_client.create_chat(
        "example-ai-config",
        context,
        fallback,
        {"example_variable": "value"},
    )

    response = await chat.invoke("How can LaunchDarkly help me?")

    print("Model response:", response.message.content)

    if response.evaluations is not None:
        eval_results = await asyncio.gather(*response.evaluations)
        print("Evaluation results:", [r.to_dict() for r in eval_results])

Evaluation results appear on the Monitoring tab as metrics such as accuracy, relevance, and toxicity.

To learn more about evaluation patterns and programmatic usage, read Online evaluations in AI Configs.

Set up and attach judges

Online evaluations use your existing AI model provider credentials. Before you enable judges, make sure your organization has connected a supported provider, such as OpenAI or Anthropic.

To attach a judge to a variation:

1. In LaunchDarkly, click AI Configs.

2. Click the name of the Completion AI Config you want to edit.

3. Select the Variations tab.

4. Open a variation or create a new variation.

5. In the “Judges” section, click + Attach judges.

6. Select one or more of the judges. If necessary, enter text in the search bar to search for a judge. By default, LaunchDarkly provides three judges:

   • Accuracy
   • Relevance
   • Toxicity

7. (Optional) Set the sampling percentage to control how many model responses are evaluated.

8. Click Review and save.

Each evaluated response triggers an additional call to your model provider. Higher sampling percentages can increase token usage and cost. For guidance on choosing a sampling rate, read Online evaluations in AI Configs.

Attached judges remain connected to the variation until you remove them.

For more information about judges, read Online evaluations in AI Configs.

Learn more about AI Configs

The following sections provide answers to common questions about working with AI Configs.

Integration with AI providers

In the AI Configs product, LaunchDarkly does not handle the integration to the AI provider. The LaunchDarkly AI SDKs provide your application with model configuration details for providers and frameworks such as OpenAI, Amazon Bedrock, and LangChain, including customized messages and model parameters such as temperature and tokens. It is your application’s responsibility to pass this information to the AI provider or framework.

The LaunchDarkly AI SDKs provide methods to help you track how your AI model generation is performing, and in some cases, these methods take the completion from common AI providers as a parameter. However, it is still your application’s responsibility to call the AI provider. To learn more, read Tracking AI metrics.

Privacy and personally identifiable information (PII)

LaunchDarkly does not send any of the information you provide to any models, and does not use any of the information to fine tune any models.

You should follow your own organization’s policies regarding if or when it may be acceptable to send end-user data either to LaunchDarkly or to an AI provider. To learn more, read AI Configs and information privacy.

Availability of new models

When you create a new AI Config variation, you can select a model from the provided list. LaunchDarkly updates this list regularly. To request a new model, click the Give feedback option and let us know what models you’d like to have included.

You can also add your own model at any time. To learn how, read Create AI model configurations.