A/B testing onboarding with PostHog

Overview

An experiment splits visitors of one entry point across versions. A version is either a patch (copy, layout, theme, landing blocks) or a different form of the same workspace, so questions, order and logic can differ. Every response records the experiment, the version and the form version it saw. Goals go beyond completion: answers, time, and conversions your own backend or PostHog reports later.

Setup

1. Connect PostHog

In Integrations, add an analytics destination of kind posthog. It is workspace-level and stored encrypted like other credentials.

Use Send test event on the destination to deliver one formflow_test_event. Every delivery, with its attempts and error, is in the delivery log. Failed batches retry with backoff through the job queue (up to 6 attempts); permanent errors are not retried.

2. Turn on forwarding per form

Forwarding is off until a form enables it in its analytics settings, or the experiment sets config.forward to true. Choose which destinations receive it, which answers are shareable, the answer-to-person-property mapping and an optional group key. An experiment can also switch forwarding off for itself with forward: false.

uid, distinct_id and the URL parameters

uid is your app's stable user id. It is the key for stable assignment and, by default, the PostHog distinct_id. If your PostHog SDK uses a different distinct id, pass it as ph_distinct and events use that instead. Without a uid, an anonymous id (cookie ff_aid) is used.

Signed identity

A uid in a URL can be forged. Create the workspace identity secret in the workspace identity settings (the app calls PUT /api/internal/identity with rotate: true); it is shown once. Your backend computes uid_sig and hands it to the app. With require signed uids on, an unsigned or invalid uid is treated as anonymous: no PostHog person reads, no prefill from person data, no stable assignment by uid. Verification is constant time. With it off, an unsigned uid is accepted but not trusted for person data.

import { createHmac } from "node:crypto";
const uidSig = createHmac("sha256", process.env.FORMFLOW_IDENTITY_SECRET!).update(uid).digest("hex");

import CryptoKit
func uidSig(_ uid: String, secret: String) -> String {
  let mac = HMAC<SHA256>.authenticationCode(for: Data(uid.utf8), using: SymmetricKey(data: Data(secret.utf8)))
  return mac.map { String(format: "%02x", $0) }.joined()   // compute on your server, not in the app
}

import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec
fun uidSig(uid: String, secret: String): String {
  val mac = Mac.getInstance("HmacSHA256").apply { init(SecretKeySpec(secret.toByteArray(), "HmacSHA256")) }
  return mac.doFinal(uid.toByteArray()).joinToString("") { "%02x".format(it) }   // compute on your server
}

Never ship the secret in the app. The Swift and Kotlin snippets show the algorithm; run it where the secret lives.

Experiment kinds

Version types

A version with a formId shows a different form. It must be in the same workspace and published, and cannot be the original. The original (control) is always the experiment's own form and is never patched. Visitors see the variant form's published version. If you publish the variant form while the experiment runs, the new version is simply recorded: results list views and completions per variant per form version, so you can see which version each response saw.

An experiment may own an entry slug. Visitors open /f/<entrySlug>, are assigned there, and see the variant form; responses are stored under the variant form, tagged with experimentId and variantId. Without an entry slug the test runs on the original form's address.

Assignment

Order of decisions: forced variant, then the stored assignment, then a finished test (winner or per-segment rules), then eligibility, holdout, the PostHog flag if configured, and finally FormFlow's bucket.

• Deterministic bucket. SHA-256 of experiment id and uid, first 48 bits, mapped to [0, 1) and through the version weights. Same uid, same version, on any server and device. Without a uid the anonymous id is used, remembered in a first-party cookie.
• Stored assignments. At first exposure the (experiment, uid or anonymous id) to version mapping and the first exposure time are stored and never change, even when weights move. A user whose version is later paused by a guardrail is moved once.
• Exposure. The hosted page records exposure when the form is viewed (bots, headless browsers and visitors who declined consent are not counted). Over the API, decide records it.

PostHog as the assignment source

Set the experiment's assignment to { "source": "posthog", "flagKey": "onboarding-test", "variantMap": { "test": "b" } }. Let PostHog Experiments own the split while FormFlow serves the flows. FormFlow calls POST {host}/flags?v=2 with the project key and the uid as distinct_id, waits at most 1.5 seconds, and caches the result per (flag, uid) for 3 minutes. The flag variant is mapped through variantMap, or used directly when a FormFlow version has the same id. If the flag is off, unmapped, slow or failing, FormFlow assigns with its own bucket. A uid is required; anonymous users always use FormFlow assignment.

For FormFlow-assigned tests, the first exposure sends $feature_flag_called with $feature_flag set to config.exposureKey (default: the experiment id) and $feature_flag_response set to the version id, so PostHog can analyse the experiment too. Create a flag with that key in PostHog.

Lucida

Add an analytics destination of kind lucida in Integrations with a project key (lc_sk_ or lc_pk_), the capture host (blank is hosted) and the project id, which is not numeric. A read key (a secret project key with *:read scope, or lc_pat) is optional and enables flag reads, cohorts, person properties and conversions. Events use the same formflow_* names and properties as PostHog, with $geoip_disable set. See the Lucida capture docs.

• Assignment. Set { "source": "lucida", "flagKey": "onboarding-test", "variantMap": { "test": "b" } }. The flag is evaluated at the capture host with POST /v1/flags, with a 1.5 second timeout and a 3 minute cache, and falls back to FormFlow's bucket.
• Targeting. lucida.cohort (value: cohort id, a UUID) and lucida.person.<property> work like the PostHog rules and need a trusted uid and a read key.
• Conversions. Mark a conversion metric with source: "lucida". An hourly job counts the goal event with an LQL query on POST /v1/projects/{id}/sql.
• Exposures. The first exposure sends $feature_flag_called and $exposure, which Lucida experiments read.
• Ids. The ph_session and ph_distinct parameters carry Lucida's ids too. In the embed SDK, pass them as analyticsSessionId and analyticsDistinctId.

Holdout, targeting, traffic and layers

• Holdout. holdout percent of enrolled users get variantId: "holdout" and no form. The SDK ready or decide reports it and the app skips onboarding. Measure it with downstream conversions. On the hosted page set holdoutExit in the config: redirectUrl (https only) sends the visitor on with uid and the page's params kept (the signature and anonymous id are not forwarded), or message is shown on a minimal "You're all set" screen with a Continue button. Embeds post holdout then close to the host, and embed.js calls onHoldout or removes the embed. No form loads and nothing is submitted, but the view counts as the holdout arm's exposure. Visitors outside the traffic share or targeting are excluded instead: they see the original form, untracked, on every surface including decide.
• Targeting. Rules on attributes (platform, app_version with vgte/vlt, country, locale, source, plan, anything passed as attrs.*) with operators eq neq in nin gt gte lt lte contains exists vgte vlt, matched with all or any. Rules on posthog.cohort (value: cohort id) and posthog.person.<property> use the personal API key, need a trusted uid, and are cached 5 minutes. A failed lookup counts as no match.
• Traffic allocation. traffic percent of eligible users enter the test. The rest see the original and are not tracked.
• Layers. Tests in the same layer own disjoint slices [from, from + traffic) of one hash space, so a user is in at most one of them. Starting a test whose slice overlaps a running one in the same layer is refused.

Scheduling and ramp

schedule.startAt and schedule.stopAt (epoch ms) start and stop the test; ramp steps (for example 10, 50, 100 percent) advance on a date (at) or once the test has whenViews views. A job runs every five minutes, so timing is accurate to about that. Split-ratio checks are skipped while ramping or running a bandit.

Events sent to destinations

One event schema, several destinations. All names start with formflow_.

Properties on every event

form_id, form_version, experiment_id, variant_id, locale, device, utm_*, and $feature/<flagKey> (the version id) when assignment comes from PostHog. The distinct_id is ph_distinct, else the uid, else the anonymous id (anonymous events carry $process_person_profile: false). $session_id is set from ph_session. $groups is set when the form maps a group type and an attribute holding the company id.

What is never sent

No answer value leaves unless the field is listed as shareable, and never for sensitive, secret, signature, file or payment fields. Person-property mapping ("answer of field X becomes person property Y", set or set_once) skips the same fields. Under Protected data mode nothing is sent at all.

Other destinations

Each destination has an event filter (only these event names) and a property rename map, applied before the batch is queued.

Goals and metrics

A test has one primary metric and up to five secondary metrics.

Per-question drop-off is always shown: for each version, the number and share of visits that reached each question of that version's form. Time per question is the gap between the question being shown and answered, measured from beacon arrival, so it is accurate to a few seconds.

Reporting a downstream conversion

From your backend, with an API key that has the form:write scope (the key never goes into the app). The user must have been exposed; unknown users and events before exposure are acknowledged but not counted. A repeat of the same event counts once unless you send a different eventId.

curl -X POST https://forms.example.com/api/v1/experiments/exp_123/conversions \
  -H "Authorization: Bearer $FORMFLOW_API_KEY" -H "Content-Type: application/json" \
  -d '{ "uid": "user_42", "event": "subscribed", "value": 9.99 }'

await formflow.request("POST", "/experiments/exp_123/conversions", { body: { uid: "user_42", event: "subscribed", value: 9.99 } });

Or let PostHog report it: mark a conversion metric with source: "posthog". An hourly job counts the goal event per distinct_id after exposure with a HogQL query (needs the personal API key and project id) and stores the first occurrence. Only users identified by uid are pulled.

Statistics

• Frequentist. Two-proportion z-test against the original with a Wilson 95% interval and a Bonferroni correction across versions.
• Bayesian. Beta-Binomial with a uniform prior, 20,000 seeded Monte Carlo draws: probability to beat control, probability of being best, expected loss (rate you give up if you choose the version and the original was better) and expected lift. The same counts always give the same numbers.
• Sample ratio mismatch. Chi-square on assigned users against the configured split. p < 0.001 is flagged: the split is broken, so do not trust the results.
• Guardrails. For example "completion must not drop more than 10% relative". A breach needs the sample minimum, a drop beyond the limit, and 95% probability the version is truly worse. The version is paused (its traffic goes to the original) and experiment.guardrail_breached is sent.
• Sample-size guard. Results warn while the smallest version has under 100 users; a verdict needs at least 100 views per version and 20 completions.
• Over time. Daily cumulative rate, lift and a 95% interval on the difference. A novelty hint appears when the second half of the test is well below the first.
• Segments. By any attribute captured at exposure. Each cut is a separate comparison, so the screen shows a stricter threshold (0.05 divided by the number of comparisons). Treat segment leads as hypotheses.

Auto-optimise

• Bandit. With bandit.enabled, a daily job runs Thompson sampling on the primary metric (not available for time) and sets version weights from the share of draws each version wins, keeping at least floor percent on every live version. It waits until every live version has minViews users. Set contextualAttr (for example platform) to learn weights per attribute value. SRM is not checked for bandit tests.
• Auto-stop. With autoStop.enabled, when one version's probability of being best reaches probability and every live version has minViewsPerArm, the winner is marked and experiment.winner_declared fires. With autoPromote the test completes and everyone gets the winner (experiment.promoted); otherwise it stops and waits for you.
• Per-segment winners. Personalize keeps the clearly winning version per segment (probability best and of beating control above 90%, at least 30 users per version) as permanent targeting rules and completes the test. Everyone else gets the overall winner, or the original.

Webhooks

Subscribe a webhook endpoint to experiment.started, experiment.paused (a stop, manual or scheduled), experiment.guardrail_breached, experiment.winner_declared and experiment.promoted. Payloads carry experimentId, formId and name, plus the version, metric or evidence involved.

AI help

Suggest variants proposes two or three ideas with a hypothesis each: a ready patch, a single-question rewrite, or a draft copy of the form with reordered or reworded questions (created as a draft, which you publish and add as a version). Explain results writes a plain-language summary with the funnel drop-offs and a recommendation. The recommendation is computed from the statistics and the model only writes the wording, so it cannot call a winner the numbers do not support. Only labels, counts and statistics are sent, never answers. Both are off under Protected data mode, and fall back to rule-based output when no AI provider is configured.

End to end: a mobile onboarding test

Goal: compare the current onboarding (control) with a shorter flow (another form) and a holdout, and optimise for paid conversion.

1. Publish onboarding and onboarding-short. Connect PostHog and enable forwarding on both.
2. Create the experiment on onboarding with entry slug first-launch, versions control (45%) and short (45%, form onboarding-short), holdout 10%, primary metric conversion on subscribed within 168 hours, secondary completion, guardrail on completion (max drop 15%). Start it.
3. Your backend signs the uid and returns uid_sig to the app. The app passes uid, uid_sig, attributes and the PostHog ids.

let client = FormFlowClient(baseURL: URL(string: "https://forms.example.com")!)
let req = FormFlowDecideRequest(entry: "first-launch", uid: user.id, uidSignature: user.formflowSig,
                                attributes: ["plan": user.plan, "platform": "ios"],
                                posthogSessionId: PostHogSDK.shared.getSessionId())
Task {
  guard let d = try? await client.decide(req) else { return finishOnboarding() }   // fail open
  if !d.show { return finishOnboarding() }                                           // holdout: variantId == "holdout"
  let base = FormFlowConfig(baseURL: client.baseURL, slug: "", uid: user.id, uidSignature: user.formflowSig,
                            attributes: ["plan": user.plan, "platform": "ios"],
                            posthogSessionId: PostHogSDK.shared.getSessionId(), posthogDistinctId: PostHogSDK.shared.getDistinctId())
  guard let config = d.config(from: base) else { return finishOnboarding() }
  // present FormFlowView(config:); the ready event carries experimentId and variantId
}

val client = FormFlowClient(context, "https://forms.example.com")
val req = DecideRequest("first-launch", uid = user.id, uidSignature = user.formflowSig,
    attributes = mapOf("plan" to user.plan, "platform" to "android"),
    posthogSessionId = PostHog.getSessionId()?.toString(), posthogDistinctId = PostHog.distinctId())
val base = FormFlowConfig("https://forms.example.com", "", uid = user.id, uidSignature = user.formflowSig,
    attributes = req.attributes, posthogSessionId = req.posthogSessionId, posthogDistinctId = req.posthogDistinctId)
val config = runCatching { client.decide(req) }.getOrNull()?.config(base)   // null = holdout or offline
if (config == null) finishOnboarding() else FormFlowForm(config, onComplete = { finishOnboarding() }, onTimeout = ::finishOnboarding)

Without the SDK's decide call, open /embed/<slug>?native=1&uid=...&uid_sig=...&attrs.plan=pro&ph_session=... in a WebView. The ready event includes experimentId and variantId; log them in your own analytics if you like. The embed page is served by the mobile SDK work; check that your version of the embed page applies experiments before relying on it.

4. When the user pays, your backend reports it (see the conversion call above). Every holdout, control and short-flow user can convert, so the lift of onboarding itself is measurable.
5. In PostHog, build a funnel on formflow_form_viewed, formflow_form_started, formflow_form_completed, then your subscribed event, broken down by variant_id. Use formflow_form_abandoned with last_field_id to find where people leave.
6. Open the results in FormFlow: Bayesian and frequentist numbers, per-question funnel, the split check, then use Explain results. Promote the winner or personalize per segment.

Limitations

• The HogQL queries for cohort membership, person properties and conversion counts are written from PostHog's documented query API and have not been verified against a live PostHog project. Test with your project before depending on cohort targeting or the conversion pull.
• Time per question comes from when the browser beacon reached the server (it flushes every 5 seconds), so it is accurate to a few seconds.
• Under Protected data mode answers are sealed: answer goals, question distributions, person properties, forwarding and AI are unavailable.
• Segment and results queries read up to 200,000 rows per test.
• A view that arrives before the visitor has decided on a consent banner is recorded anonymously but not forwarded.