configuring-experiment-rollout

Configuring experiment rollout

This skill answers: Who sees what variant?

Recommended approach: equal split + adjust rollout percentage

In most cases, experiments work best with an equal split. If you want to limit exposure to the test variant, adjust the rollout percentage instead.

Why equal splits are better:

• Equal splits maximize statistical power — each variant has the same sample size
• Equal splits balance traffic and thus reach significance faster
• Increasing user exposure throughout the experiment through increasing rollout is clean (changing split mid-experiment can cause users to switch variants, which is bad for user experience and data quality)

Always default to an equal split unless the user explicitly requests otherwise.

When an uneven split is required

Uneven splits combined with the default "Exclude from analysis" handling can introduce bias. If the experiment observes multi-variant users (users exposed to more than one variant) then those are dropped asymmetrically — the smaller variant loses a larger fraction of its assignments. If those users behave differently from the rest, the smaller variant's metrics will be skewed.

The right mitigation depends on experiment state:

1. Pre-launch, or live but with few exposures so far — use an equal split and reduce the overall rollout. Achieves the same test-variant exposure without the bias and preserves statistical power. See the disambiguation question below.
2. Live experiment with significant exposures — switch multivariate handling to "First seen variant". Changing the split mid-run reassigns users across variants (anti-pattern; see "Changing rollout on a running experiment" below). Switching handling instead keeps everyone in their original variant and avoids the asymmetric exclusion. See configuring-experiment-analytics for how to set this. Note that "first seen" handling can introduce other biases, but it's preferable to mid-run reassignment.

The two rollout controls

There are two separate controls that determine who sees what. Both live on the linked feature flag, sent through the feature_flag object in the flag's own shape (not the deprecated parameters keys).

1. Variant split (feature_flag.filters.multivariate.variants)

How users inside the experiment are distributed across variants.

• Array of {key, name, rollout_percentage}, where the rollout_percentage values must sum to 100
• Minimum 2 variants, maximum 20
• No specific variant key is required — the analysis baseline defaults to the variant keyed "control" when present, else the first variant
• Default: control 50% / test 50%

If the user says "A/B/C test" without naming keys, key the baseline "control" (the convention) and create additional variants for the others; if they ask for specific keys, use them as-is with the baseline first.

2. Overall rollout (feature_flag.filters.groups[0].rollout_percentage)

What percentage of all users enter the experiment at all, sent as a single rollout group: groups: [{ "properties": [], "rollout_percentage": N }]. Default: 100%.

Users not included are excluded entirely: they don't see any variant and are not part of the analysis.

Where these are sent

Both controls live inside feature_flag.filters:

{
  "feature_flag": {
    "filters": {
      "multivariate": {
        "variants": [
          { "key": "control", "name": "Control", "rollout_percentage": 50 },
          { "key": "test", "name": "Test", "rollout_percentage": 50 }
        ]
      },
      "groups": [{ "properties": [], "rollout_percentage": 100 }]
    }
  }
}

filters may also carry aggregation_group_type_index (to run the experiment on a group type rather than individual users) and payloads (JSON-encoded strings keyed by variant key). On a running experiment, any flag-config change must also send update_feature_flag_params: true, otherwise the API rejects the update before it reaches the flag (see "Changing rollout on a running experiment").

How they interact

These two controls multiply:

The disambiguation question

CRITICAL: If the user requests an uneven variant split (e.g. "60/40", "70/20/10") or mentions a specific percentage that could refer to either the split or the rollout (e.g. "roll out to 25%"), you MUST clarify before proceeding. This covers two cases:

Case 1: Single percentage ("25%", "roll out to 40%")

The percentage is ambiguous — it could mean a variant split or a rollout change. Ask:

There are two ways to get 25% of users seeing the test variant:

Reduced rollout with equal split (recommended): reduce the overall rollout and split variants equally. Only a subset of users enter the experiment, and of those, each variant gets the same share. Equal splits maximize statistical power and avoid bias. Asymmetric split: keep 100% rollout but give the test variant only 25%. All users enter the experiment, but the uneven split reduces power on the smaller variant and risks bias.

Which approach do you prefer?

Adjust the numbers to match whatever percentage the user requested.

Case 2: Uneven ratio ("60/40", "70/30", "80/20", etc.)

The ratio looks like an explicit variant split, but a reduced rollout with an equal split is almost always better. Explain the trade-off and recommend the alternative:

An uneven variant split works, but an equal split with reduced rollout is recommended:

Equal split + reduced rollout (recommended): reduce the overall rollout so that the same fraction of users sees the test variant, but split variants equally within the experiment. Equal splits maximize statistical power and avoid bias from asymmetric multivariate exclusion. Uneven split. Achieves the same user-facing outcome, but reduces power on the smaller variant and risks bias.

Would you like the equal split approach, or do you have a specific reason for the uneven split?

Adjust the numbers to match the ratio. For experiments with more than two variants, "equal" means each variant gets the same share (e.g. 34/33/33 for three variants). If the user confirms they want the uneven split after seeing the trade-off, proceed — but DO NOT skip the next section.

After the user picks the uneven split

If the user proceeds with an uneven split (option 2 in either case above), you MUST surface the multivariate-handling implication BEFORE creating or updating the experiment. The user has chosen the riskier rollout path and needs to make an informed choice about how to mitigate.

Ask:

One more thing — with an uneven split, the default "Exclude from analysis" handling drops users exposed to multiple variants asymmetrically. The smaller variant loses a larger fraction of its assignments, which can skew its metrics if those users behave differently from the rest.

Two options:

Switch multivariate handling to "Use first seen variant" (recommended for uneven splits) — keeps all users in the analysis and avoids asymmetric exclusion. Has its own caveats (other biases can creep in) but is preferable to the default for uneven splits. Keep the default "Exclude" handling and accept the bias risk.

Which would you like?

See configuring-experiment-analytics for how to set the multivariate handling. Apply the choice as part of the same operation (creation or update) — do not leave the user with an uneven split under default handling without an explicit, informed decision.

Bucketing and persistence across login

By default a person's variant follows their distinct ID. When the same person sees the flag before and after they are identified (usually at login), their distinct ID changes, and so can their variant. Choose before you create the flag or the experiment; never change bucketing or persistence on a live flag.

The share of anonymous visitors on a page does not settle this. It gives the mix on the page, not whether the same individuals cross identification there, and those are different questions: an even mix can be two populations that never meet, and an almost fully identified page can still route every one of those people through an anonymous first pageview. So present the three options with what each one needs, and let the user choose:

• User-id bucketing (the default): leave ensure_experience_continuity out, so the team's persistence default applies, except in the two cases below.
• Device-id bucketing (recipe below): needs a device ID on every flag call, because a call without one gets no variant. A server SDK must forward the browser's device ID, and local evaluation works when it does. The web SDK sends the device ID on flag requests from posthog-js 1.307.1; check the version, because an older one puts a device ID on events but not on flag requests. The flag service takes a device ID from any SDK that sends one, so check the device-ID shares rather than the platform. Those shares count events and not the flag request, so a near-0 share leaves this option unchecked rather than ruled out.
• Persistence (ensure_experience_continuity: true): cannot work where an SDK evaluates the flag locally, because a local evaluation never consults the override store. It also needs person profiles for anonymous users, no bootstrapping, and $anon_distinct_id on server flag calls. Learn more: https://posthog.com/docs/feature-flags/creating-feature-flags#persisting-feature-flags-across-authentication-steps

When ensure_experience_continuity is omitted, experiment-create applies the team's persistence default. The device-id recipe needs no such step: create-feature-flag leaves persistence off unless you set it, and experiment-create rejects a feature_flag object for a flag that already exists. Device-id bucketing and persistence can't be combined, and the API refuses the pair.

Two cases need the field set rather than omitted:

• Moving a flag experiment-create already made onto device-id bucketing. This applies only to a draft experiment whose flag is still inactive; for a launched experiment, keep the current bucketing and recommend ending and restarting the experiment through managing-experiment-lifecycle. That flag carries the team's persistence default, so pass ensure_experience_continuity: false in the same call, or the refusal above blocks the change.
• The team's persistence default is on and an SDK evaluates the flag locally. The flag then inherits a persistence setting that cannot work there. Tell the user, and set the field to false only if they ask, because the team chose that default.

Device-id bucketing recipe

experiment-create cannot set bucketing, so create the flag first, then link it:

1. Call create-feature-flag with the experiment's key, bucketing_identifier: "device_id", and active: false so no one gets a variant before launch (a new flag is active by default):

   {
     "key": "kebab-case-key",
     "name": "Experiment name",
     "active": false,
     "bucketing_identifier": "device_id",
     "filters": {
       "groups": [{ "properties": [], "rollout_percentage": 100 }],
       "multivariate": {
         "variants": [
           { "key": "control", "rollout_percentage": 50 },
           { "key": "test", "rollout_percentage": 50 }
         ]
       }
     }
   }

   filters.groups is required: a flag without a group is rejected.

2. Call experiment-create with that feature_flag_key and no feature_flag object. The experiment links the flag as it is, and launching the experiment turns the flag on.

experiment-setup-context reports the facts these rules need (the page's share of unidentified visitors, and per SDK the device-ID and local-evaluation shares). creating-experiments (references/setup-decisions.md) applies them.

Resolving experiments

Rollout changes require an experiment ID. If the user refers to an experiment by name or description (e.g. "change rollout on my signup test"), load the finding-experiments skill to resolve it to a concrete ID before proceeding. After resolving, show the matched experiment's name, ID, linked flag key, and status, and get user confirmation before making any change. If more than one experiment matches, ask the user to pick one.

Changing rollout on a running experiment

Any change to rollout or variant split on a running experiment affects both user experience and statistical validity. You MUST warn the user and get explicit confirmation before making the change.

Do NOT silently apply the change — even if the user asked for it directly. Present the warning covering both perspectives:

1. Who sees what variant? — will users switch variants or lose a feature?
2. Who is in my analysis? — how does this affect data quality?

Exception: Increasing rollout (without changing the split) is generally safe — no users switch variants, more users are added cleanly. A lighter warning is enough here, but still state the experiment ID and the current and new rollout percentages, and get explicit user confirmation before sending update_feature_flag_params: true.

If the goal is "stop new users from entering" rather than a percentage change: reducing the rollout is the wrong tool — it drops already-enrolled users out of the experiment too. Freezing exposure (experiment-freeze-exposure) closes enrollment while enrolled users keep their variant and metrics keep flowing; see managing-experiment-lifecycle for its preconditions and limitations.

Mid-experiment fix for uneven-split bias: switching multivariate handling from "Exclude" to "First seen variant" is the recommended mitigation for already-launched experiments — no users switch variants and all collected data stays in the analysis. Changing the split to be even is an anti-pattern mid-run (typically requires resetting or ending the experiment) and is only preferred if the experiment hasn't been exposed to many users yet. See configuring-experiment-analytics for how to change the handling.

See references/changing-distribution-after-launch.md for detailed warnings, what to tell the user, and when to recommend alternatives.

Related skills

• configuring-experiment-analytics — the analysis side: exposure criteria, metrics, and multivariate handling
• diagnosing-experiment-health — when a mid-run split change has already skewed the results
• managing-experiment-lifecycle — reset or end-and-restart mechanics when a split change requires them