Basic Concepts

Control Group

The control group refers to the group of users who receive the existing or current version of your app or website. This group serves as a baseline for comparison against the experimental group.

Test Group

The test group, or experimental group, consists of users who are exposed to the modified or alternative version of your app or website. This group allows you to measure the impact of the changes being tested.

Traffic Diversion Entity

Traffic refers to the number of users who are directed to different variations of your app, while traffic diversion entity defines "users" with user ID, device ID, account ID, visitor ID or custom ID.

Mutually Exclusive Experiments

Imagine that you are running two or more experiments, but you don't want your users to be exposed to both experiments, as interaction effects may invalidate your test results. In this case, make the two experiments mutually exclusive.

Interdependent Experiments

Interdependent experiments do not impact each other, so your experiment results will not get tainted even though a single user could be included in multiple experiments at the same time.

Statistical Significance

Statistical significance determines whether the observed differences between the control and experimental groups are statistically meaningful or simply due to chance. It helps validate the results and ensure they are reliable.

Test Layer / Exclusion Group

On one hand, as traffic is not shared within the same test layer, you can make two or more experiments mutually exclusive to each other by assigning them into one test layer.

On the other hand, if the traffic pool is not large enough, which can affect the statistical significance of the test result, you can set up multiple test layers so that the same users can be exposed to multiple interdependent experiments.

Confidence Level

The confidence level refers to how confident we are that the observed effect or difference between variants is not due to random chance. It is typically expressed as a percentage, such as 95% or 99%.

Confidence Interval

A confidence interval is a range of values within which we estimate the true parameter to likely lie. With a certain confidence level (e.g., 95%), we can calculate the confidence interval:

Key Metrics

Key metrics are used to assess the performance or impact of the changes. These could include conversion rates, click-through rates, engagement metrics, revenue, or any other relevant metric for evaluating success.

T-Test

A t-test is a statistical test used to determine if there is a significant difference between the means of two groups, especially when the sample size is small. By calculating the t-value (see below) and comparing it to the p-value from the t-distribution, you can determine whether to reject or fail to reject the null hypothesis.

P-value

It represents the probability of observing the obtained results (or more extreme results) under the assumption that the null hypothesis is true. It quantifies the strength of evidence against a null hypothesis.