Create Programmatic Test

Entry: A/B Testing - Test List - Create Test - Programmatic Test

Basic Information

• Test Name
• Required & Unique;
• No more than 20 characters;
• Contains Chinese, uppercase and lowercase English letters, and numbers;
• Do not start with a number.
• Description
• Optional;
• No more than 100 characters.
• Operation Cycle (Test Duration)
• For experiments that have not been launched, you can set when the test goes online and offline by checking the "Operation Cycle" option.
• Once an experiment is launched, you can only adjust the end time, and it cannot be set earlier than the current time.

Select Test Layer

• Select a test layer for your A/B test. If no test layer exists, click "Add Test Layer" from the dropdown menu to build a new one.
• The total traffic allocation for a test layer is 100%. The remaining available traffic in the test layer = 100% - Traffic allocated for the current test.

Select Target Audience

• Diversion Entity

Device/User/Account/Visitor/Custom ID

• Condition Configuration
• All: All active users will participate in the test.
• Conditional Filter: Filter out the target audience with event conditions or property conditions. Device properties are available only if device ID is set as the entity.
• User Tag: Available only when the user tag and the A/B test use the same entity.
• File Import: Upload an ID list through a CSV file. The file size should not exceed 200MB. Uploading a new file will replace the old one.

Allocate Traffic

• Flow adjustment only applies to new users in the group.

This option is enabled by default. It means that existing users who already been assigned to a particular group will maintain this result, ensuring a consistent user experience. Any adjustments made to the traffic allocation will affect only the new users in the group.

• Allocation Method
• Manual Allocation

You can manually adjust the traffic for each group by entering integers between 1 and 100. The remaining traffic will stay "To Be Allocated" and will not participate in the experiment.

• Auto Allocation

100% traffic will be evenly distributed to each group. If the total traffic cannot be evenly divided, the remainder will be allocated to the last group. (e.g. Control group: 33%, Test group 1: 33%, Test group 2: 34%)

• Test Group Name

SolarEngine allows at most 10 groups running simultaneously, including 1 control group and 9 test groups.

Edit group names as needed.

• No more than 10 characters.
• Contains Chinese, uppercase and lowercase English letters, and numbers.
• Group Description

Enter a description or upload an image.

• Debugging Device
• Add debugging devices using "_distinct_id". Each debugging device can only participate in one of the groups within the same test layer.
• The debugging device will directly enter its assigned group, ignoring the target audience filtering conditions.
• Data generated by the debugging device will not be recorded or calculated in the report.
• By default, a newly created test is in the "debugging" status. During this time, only debugging devices can participate in the experiment.
• Parameter Settings

Assign parameter values for each group. Add up to 10 parameters. Note that parameters cannot be used repeatedly across different test layers.

Configure Metrics

• Add up to 10 metrics for each test. Metric names should be unique.
• A core metric needs to be configured to measure the effect of variations. You can also add more metrics to analyze the test results from multiple aspects. The core metric can only be:
• Times per User
• Average
• Per User
• Custom Metrics
• Set a Return Visit Event

If you would like to know the retention rate of users in each group, you also need to set a return visit event. The initial event is set as the enter group event by default.

The return visit event allows conditional filtering.

Test List

       The test list displays all the existing A/B tests and includes the following information:

• Test ID
• Test Type
• Test Name
• Test Layer
• Update Time
• Creator
• Status (Default "Debugging")

Launch the Test

Clicking the "Online" tab to bring the selected "Debugging" test online. The status will change to "Running" automatically.

End the Test

Clicking the "Offline" tab to bring the selected "Running" test offline. After the test goes offline, the requested test parameters will still be retained on the devices.

To update the parameters, select the parameters from one of the groups within the current experiment. Once you confirm the selection, the current experiment will be taken offline, and a corresponding rule will be added to the online parameters. This rule will be used to update the parameters on the devices of users who have already joined the groups.

Publish the Test

Clicking the "Publish" tab to publish the best strategy to the defined users. You can select the test audience users, filter out users by conditions, or import IDs through a file. After publishing, the test goes offline. The system creates online parameter rules based on the publishing configuration and executes the publishing plan.