Experimentation
One name, two tools: picking the right Amplitude experiment for the job
Teams say 'Amplitude Experiment' and mean two different products. Amplitude's own docs split them: Web Experiment for marketers with a visual editor, Feature Experiment for engineering-led, flag-based product tests. Choosing wrong is how experiments stall.
What's the actual difference?
| Web Experiment | Feature Experiment | |
|---|---|---|
| Built for (per Amplitude's docs) | Digital marketers and growth marketing - 'test the website without an engineer' | Product, data engineering/operations, and analyst teams |
| How variants are made | Point-and-click visual editor: edit text, swap images, restyle elements on the page | Feature flags - switches that modify the experience without code changes - delivered via Experiment SDKs and APIs |
| Where it runs | Your website; variants applied by page/URL targeting | Web, mobile, and backend code paths |
| Targeting | URL, behavior, and user property | Properties, cohorts, and segments on the flag |
| Typical first test | Headline / hero / CTA on a landing page | New onboarding flow behind a flag |
Split and audience descriptions: Amplitude's feature-vs-web comparison doc (Sources).
Which one should your team start with?
Our field rule (judgment, informed by the docs' own audience split): start where your bottleneck is. If experiments die waiting for engineering sprints, Web Experiment's no-code editor removes that queue for marketing-surface tests. If your questions live inside the product - pricing gates, onboarding flows, algorithm changes - Feature Experiment's flags are the only honest way to test them, and the flags double as safe-rollout infrastructure.
Plan reality check, against amplitude.com/pricing at the time of writing: the Free tier includes 'limited' experiments - Amplitude does not publish the numeric cap - while Growth and Enterprise carry unlimited active experiments plus the advanced options (multi-armed bandits, stratified sampling, sticky bucketing). Budget accordingly before promising a testing program.
What's the statistics engine underneath?
Better than most teams assume. Amplitude Experiment defaults to sequential testing - specifically the mixture sequential probability ratio test (mSPRT) - which the docs describe plainly: results stay valid whenever you view them, and you can end an experiment early. That kills the classic peeking problem that invalidates naive t-tests.
You can also switch test types per your statistical preferences: sequential, T-test, Bayesian, or Thompson sampling. And CUPED - variance reduction using pre-experiment data - exists as an optional technique, toggled off by default; flip it on and Amplitude accounts for varying treatment effects across user segments. If your experiments are chronically underpowered, that one toggle is worth more than a month of traffic.
The mistake that wastes the whole tool
Experiments are only as trustworthy as the events they read. An experiment whose success metric fires unreliably, double-fires on retries, or means different things on web and mobile will produce a confident readout of noise - the stats engine cannot save an instrumentation problem. Before scaling a testing program, put the tracking plan through the same rigor as the test design; that ordering is the entire premise of our practice.
In practice: Experiment setup - flags, metrics, and guardrails wired to a taxonomy you trust - is part of our enablement work.
Sources
- Amplitude docs: Feature vs Web Experimentation
- Amplitude docs: Web Experiment
- Amplitude docs: Experiment overview
- Amplitude docs: Sequential testing (mSPRT)
- Amplitude docs: Statistical preferences (test types, CUPED)
- Amplitude pricing (experiment entitlements)
Product capabilities change - every claim above links to the primary source it came from. Judgment calls and field observations are ours and labeled in-line.
Frequently asked questions
Can I run Amplitude experiments without engineers?+
Website experiments, yes: Web Experiment's visual editor edits text, images, and styling on the page itself, targeted by URL, behavior, and user property - Amplitude markets it as testing without an engineer. In-product experiments, no: those run on feature flags through the Experiment SDKs, which is engineering work by design.
Is it safe to peek at Amplitude experiment results early?+
Under the default settings, yes - that's the point of sequential testing. Amplitude's mSPRT-based engine keeps results valid whenever you view them and supports stopping early. If you switch the test type to a classic T-test, the usual peeking rules apply again.
How many experiments does the Amplitude free plan include?+
Amplitude's pricing page says 'limited experiments' on the Free tier without publishing a number. Growth and Enterprise list unlimited active experiments. If the cap matters to your plan, confirm the current figure with Amplitude - and treat any specific number you read elsewhere as unverified.
Get a free, scored audit of your Amplitude instance
Send us read-only access and get a scored findings report within 48 hours: taxonomy health, duplicate events, governance gaps, and the three fixes with the highest data-trust payoff. No commitment.
Request the free audit