enable-ssi

Configure SSI and Unified Service Tags on Linux

Before doing anything else: Fully resolve all variables in ## Context to resolve before acting. Do not begin Step 0 until every variable has a concrete value.

Triggers

Invoke this skill when:

• The Datadog Agent is already installed with SSI (DD_APM_INSTRUMENTATION_ENABLED=host was used) and you need to configure Unified Service Tags on the application service
• The user wants to set DD_SERVICE, DD_ENV, DD_VERSION on a running service
• SSI is installed but /proc/<pid>/maps doesn't show the language tracer (launcher-only injection)

Do NOT invoke this skill if:

• The Datadog Agent is not yet installed — run agent-install first
• SSI packages are missing from /opt/datadog-packages/ — re-run agent-install
• The target is a Kubernetes cluster — use enable-ssi at dd-apm/k8s-ssi/enable-ssi instead

Background

When the install script runs with DD_APM_INSTRUMENTATION_ENABLED=host, it:

1. Installs datadog-apm-inject and language library packages under /opt/datadog-packages/
2. Writes the launcher path into /etc/ld.so.preload
3. SSI is now armed — every new process on the host gets the launcher injected at startup

What SSI does NOT configure automatically:

• DD_SERVICE, DD_ENV, DD_VERSION — these must be set on the application process for traces to be tagged correctly
• Without DD_SERVICE, the tracer auto-detects a service name (often the process name or framework name), which may not match what the user expects

Prerequisites

Verify SSI is armed:

Claude runs

ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "cat /etc/ld.so.preload && ls /opt/datadog-packages/ | grep apm"

If /etc/ld.so.preload contains a path to the launcher, and /opt/datadog-packages/datadog-apm-inject exists — SSI is armed.

ERROR: Either missing — run agent-install first.

Check for existing manual instrumentation:

Claude runs

ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> "
grep -r 'import ddtrace\|from ddtrace\|require .dd-trace.\|opentelemetry' <SOURCE_DIR> 2>/dev/null | head -5 || echo 'No manual instrumentation found'
"

ERROR: Manual instrumentation found — SSI silently disables itself when it detects an existing tracer. Remove the manual import/package before proceeding.

Check base libc:

Claude runs

ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "ldd --version 2>&1 | head -1"

ERROR: musl — SSI requires glibc. No workaround; must use a glibc-based OS.

Context to resolve before acting

Before substituting any value into a command, check it. The unit name may contain only letters, digits, dots, dashes, underscores and the at sign. The process ID must be digits only. Service name, env and version may contain only letters, digits, dots, dashes, underscores, colons and slashes. The source directory must be an absolute path made only of letters, digits, dots, dashes, underscores and slashes. If any value fails these checks, stop and ask the user.

Before the first connection, ask the user to verify the host fingerprint out of band and add it to their known hosts. If ssh reports an unknown or changed host key, stop and report it to the user.

Step 0 (Only if existing instrumentation detected): Remove Manual Instrumentation

Report the detected manual instrumentation to the user and give them the removal steps below to carry out themselves. The agent leaves packages and application code as they are.

• Python: pip uninstall ddtrace, remove import ddtrace / ddtrace-run from CMD
• Node.js: npm uninstall dd-trace, remove require('dd-trace')
• Java: remove -javaagent:/path/to/dd-java-agent.jar JVM flag
• Ruby: gem uninstall ddtrace, remove require 'ddtrace'
• .NET: remove Datadog.Trace NuGet and profiler env vars

After removing, restart the service. Confirm with the user before restarting. Tell the user: "I need to restart <SYSTEMD_SERVICE_NAME> to remove the old instrumentation. This will cause a brief outage. Ready to proceed?" Wait for confirmation.

Step 1: Set Unified Service Tags on the Application Process

Without UST, traces arrive with an auto-detected service name that may not match user expectations, and won't be tagged with env or version.

For systemd-managed services (most common):

Claude runs

ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo systemctl cat <SYSTEMD_SERVICE_NAME>"

Add a drop-in override (preserves the original unit file):

What you need to do in a terminal

ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST>
sudo systemctl edit <SYSTEMD_SERVICE_NAME>

Add to the editor:

[Service]
Environment="DD_SERVICE=<SERVICE_NAME>"
Environment="DD_ENV=<ENV>"
Environment="DD_VERSION=<VERSION>"

Wait for the user to confirm they saved the drop-in before continuing.

Apply:

Claude runs

ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  sudo systemctl daemon-reload
ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo systemctl show <SYSTEMD_SERVICE_NAME> | grep -E 'DD_SERVICE|DD_ENV|DD_VERSION|DD_DATA_STREAMS_ENABLED'"

If the UST vars appear in the output — configuration applied.

For supervisord:

# In [program:<name>] section of supervisord.conf
environment=DD_SERVICE="<SERVICE_NAME>",DD_ENV="<ENV>",DD_VERSION="<VERSION>"

Confirm with the user before reloading: tell the user that this reload restarts every program supervisord manages, offer to restart only the affected program instead, and wait for confirmation. Reload: sudo supervisorctl reload

For pm2:

// ecosystem.config.js
env: { DD_SERVICE: "<SERVICE_NAME>", DD_ENV: "<ENV>", DD_VERSION: "<VERSION>" }

Confirm with the user before reloading: tell the user which app will restart and wait for confirmation. Reload: pm2 reload <app>

Step 1b: Check for Event-Driven Services

Skip this step when running non-interactively, and continue to the next step.

Otherwise, check whether the service looks event-driven from what the user describes or what is evident in the application.

• Fit found (messaging client, broker, queue-triggered Lambda, or the user describes services handing work to each other asynchronously) → offer the separate enable-dsm skill and invoke it only if the user agrees. It asks the user once, states the plan rule, and makes the config change without restarting.
• No fit → skip. Do not mention DSM.

If the user agrees, enable-dsm adds its variable to the drop-in from Step 1, so the restart in Step 2 picks up both.

Step 2: Restart the Service

Confirm with the user before restarting. Tell the user: "I need to restart <SYSTEMD_SERVICE_NAME> for SSI to inject into it. This will cause a brief outage. Ready to proceed?" Wait for confirmation.

Claude runs

ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo systemctl restart <SYSTEMD_SERVICE_NAME>"

Wait about three seconds, then check the service state:

ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo systemctl is-active <SYSTEMD_SERVICE_NAME>"

If active is returned — service is running.

ERROR: Returns failed — check logs:

ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo journalctl -u <SYSTEMD_SERVICE_NAME> --since '1 minute ago' | tail -30"

Step 3: Confirm Injection and UST in the Running Process

Use the main process ID of <SYSTEMD_SERVICE_NAME> resolved in the context table and confirmed with the user. Keep output limited to the filtered lines shown below, and keep full command lines and the unfiltered process environment out of output.

Claude runs

# Authoritative injection check
ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo cat /proc/<PID>/maps | grep -E 'launcher|apm-library|datadog'"

# UST vars in process environment
ssh -i <SSH_KEY> <SSH_USER>@<SSH_HOST> \
  "sudo cat /proc/<PID>/environ | tr '\0' '\n' | grep -E 'DD_SERVICE|DD_ENV|DD_VERSION|DD_DATA_STREAMS_ENABLED'"

If both the launcher and language library appear in maps, and UST vars are in environ (plus DD_DATA_STREAMS_ENABLED=true if DSM was enabled in Step 1b), SSI and tagging are fully configured.

ERROR: Launcher in maps but no language library — injection attempted but failed. Ask the user whether pup is already installed on their local machine from a trusted Datadog source. If the user confirms, run it on the local machine, not over ssh; it uses Datadog credentials the user has already configured, and those credentials stay out of files and chat. Otherwise go directly to troubleshoot-ssi. Run:

pup apm troubleshooting list --hostname <DD_HOSTNAME> --timeframe 15m

Go to troubleshoot-ssi if errors are present.

Done

Exit when ALL of the following are true:

• Launcher and language library visible in /proc/<PID>/maps
• DD_SERVICE, DD_ENV, DD_VERSION present in /proc/<PID>/environ
• Service is running and healthy

Tell the user configuration is complete and offer to run verify-ssi. Proceed only after the user confirms.

Security constraints

• Never write a raw API key into any file or chat message
• Always confirm with the user before restarting production services
• Do not modify application source code — configure only via environment variables in the service unit
• Use only the SSH key path the user provides, keep the key contents out of output, and prefer the user's ssh agent where available