User Guide

Using A/B Tests for Widgets

A/B testing (or split testing) is a methodology for researching user experience. When running a test, you can split your audience to test different variations of your widget or launcher and determine which of them is performing better.

In Reteno, you can run an A/B test for the following types of widgets:

  • Subscription forms
  • Informers
  • Launchers
  • Age gate

A test can include different types of widgets.

šŸ“˜

Note

You cannot delete the widgets participating in a running test.

You cannot run A/B tests for the widgets:

  • Containing errors.
  • Having non-saved changes.
  • Used in another A/B test.

To open the A/B tests screen, go to Site ā†’ Widgets and select A/B tests in the left-hand side panel. If you do not have any tests, the A/B tests landing page displays.

A/B tests landing page with no tests yet, showing the New test button and an illustration of two sample subscription widgets connected by an arc

Managing A/B Tests

To create a new A/B test, click the New test button.Ā 

You need at least 2 widgets to create and run an A/B test. If you donā€™t have them, pressing the New widget button opens the dropdown menu suggesting the creation of a widget.

Select the type of widget you want to create and follow the instructions in Setting Up Widgets for Your Site.

New widget dropdown menu open on the A/B tests page, listing widget types: All types, Subscription forms, Informers, Request forms, Launchers, and Age gate

When you have the widgets to compare, you can proceed with the creation of tests.

In Reteno, you can create and run A/B tests to compare:

  • 2 widgets.
  • 2 widgets when one has a connection to a launcher.
  • 2 pairs of widgets and launchers.

See the following sections to learn how to create and run the different types of tests.

When you have the created and completed tests, the A/B tests page shows the tests in progress marked with the play icon in the Status column. The completed tests are marked with the tick icon.

List of A/B tests showing test names, IDs, start and end dates, confidence percentages, and status column with play icons for in-progress tests and a checkmark for a completed test

The Confidence column shows the reliability of the test results in percent. The test results are reliable when the confidence is greater than 90%. In this case, the confidence value is displayed in green.

Stopping a Test in Progress

To stop the test in progress, expand the In progress dropdown menu in the Status column and select one of the options:

  • Stop: To stop the test and keep the widgets published.
  • Stop and unpublish the worse: To stop the test and unpublish the widget with the worst performance when the confidence is less than 90%.
  • Stop and unpublish loser: To stop the test and unpublish the widget with the worst performance when the confidence is greater than 90%.
  • Stop and unpublish both: To stop the test and unpublish both widgets when the confidence is 0%.
A/B tests list with the In progress status dropdown expanded, showing Stop and Stop and unpublish the worse options

The stopped tests change their status to Completed.

šŸ“˜

Note

When you stop a test in progress, you will not be able to relaunch it.

Deleting a Test

To delete a test from the list of A/B tests:Ā 

  1. Click the ellipses (three dots) icon on the right-hand side of the test row, and then click Delete in the dropdown menu.
List of A/B tests with an ellipsis (three dots) icon at the end of each test row
  1. Click Delete in the dialog box.
Delete test confirmation dialog asking "Are you sure you want to delete test?" with Cancel and Delete buttons

The test is permanently deleted from the list of A/B tests. The widgets that participated in the deleted test are not deleted.

Viewing the Test Statistics

To view the test statistics, click the test name in the Name column. The statistics page opens. Switch between the Desktop, Mobile, and All tabs to view the test details for the desktop, mobile, or both versions of the widgets.

A/B test statistics page showing status, confidence percentage, and start date, a table comparing Widget A and Widget B traffic allocation and conversion, and a line chart of unique views over time

To preview the widget, point to the widget name in the Widgets column. The widget shows in the bottom right corner of the screen.

To go to the widget parameters, click the widget name.

A/B test statistics page showing status, confidence percentage, and start date, a table comparing Widget A and Widget B traffic allocation and conversion, and a line chart of unique views over time

To preview the widget and its calling rules in a new window, click the Widget preview (eye) icon on the right-hand side of the row.Ā 

A/B test statistics page showing status, confidence percentage, and start date, a table comparing Widget A and Widget B traffic allocation and conversion, and a line chart of unique views over time

In the preview window, you can switch between the desktop and mobile preview by selecting the Desktop preview or the Mobile preview icons in the top left-hand side corner of the screen.Ā 

To go to the widget parameters, click the Open button.

Widget preview window with desktop and mobile toggle icons top left, an Open button top right, a launcher icon on the page, and a Widget triggering rules list on the right

Creating the A/B Test for 2 Variations of a Widget

You can run an A/B test to compare the performance of 2 widgets.

To create the A/B test for 2 widgets:

  1. In the A/B tests window, click the New test button.
A/B tests landing page with no tests yet, showing the New test button and an illustration of two sample subscription widgets connected by an arc
  1. In the New test window, enter the test name (required) and test description (optional) into the corresponding fields.
New test window with a validation error "This field cannot be blank" under the empty Name field, and empty widget preview placeholders for widget A and B
  1. Click on the first widget preview and select the widget from the list in the Select widget window.
Select widget window with a widget ID search and a Select button highlighted beside a matching subscription form result
šŸ“˜

Note

You can search for the widget using the Search box at the top of the Select widget window:

  • To search by the widget name, enter the widget name or a part of the name into the Search box and click the Search (magnifying glass) icon or press Enter.
  • To search by the widget ID, enter id: followed by space and the widget ID into the Search box and click the Search (magnifying glass) icon or press Enter. For example, id: 8230
Select widget window with the search box and the sorting and filter options highlighted in red
  1. Click Select widget B and select the widget from the list in the Select widget window.
šŸ“˜

Note

To replace the widget, point at the widget, click the Replace widget icon, and select another widget from the list in the Select widget window.

Widget A and B preview comparison with the Replace widget icon highlighted in red beside widget A
  1. Enter the value in the Traffic allocation field for one of the widgets.
Traffic allocation fields for widget A and B, with the widget A percentage field highlighted in red
šŸ“˜

Note

The traffic allocation value specifies the distribution of displays for widgets A and B. The default value is 50% for each widget. This means that the number of displays for both widgets is evenly distributed between all site or page visitors. The minimum value is 1, the maximum value is 99.

  1. Click Start test.

The test shows in the A/B test window with the In progress status.

Running the A/B Test for 2 Widgets with One Attached to the Launcher

If one of your widgets has the By click on launcher option enabled in the Widget calling settings, you can run an A/B test for 2 widgets attached to that launcher.

When the page visitor clicks the launcher during the test, it will show either widget A or widget B, depending on the traffic allocation values applied.

šŸ“˜

Note

For this type of test, only one widget must have the By click on launcher option enabled.

To run an A/B test for 2 widgets attached to the same launcher:

  1. In the left-hand side panel, select A/B Tests and click the New test button.
AB tests page with an "Install the script on the site" banner and the New test button highlighted in red
  1. In the New test window, enter the test name (required) and test description (optional) into the corresponding fields.
  2. Click Select widget A and select the widget from the list in the Select widget window. This widget must have the By click on launcher option enabled.
šŸ“˜

Note

You can search for the widget using the Search box at the top of the Select widget window:

  • To search by the widget name, enter the widget name or a part of the name into the Search box and click the Search (magnifying glass) icon or press Enter.
  • To search by the widget ID, enter id: followed by space and the widget ID into the Search box and click the Search (magnifying glass) icon or press Enter. For example, id: 8230
  1. Click Select widget B and select the widget from the list in the Select widget window. This widget must not have the By click on launcher option enabled.
New test form named "2 widgets 1 launcher" showing a popup informer for widget A and the Select widget B option highlighted in red, connected to a launcher icon
  1. Enter the value in the Traffic allocation field for one of the widgets.
  2. Click Start test.

Two tests ā€“ one for the widgets and one for the launchers ā€“ appear in the list of tests in the A/B tests screen.

List of A/B tests with two new rows, "2 widgets and 2 launchers Launchers" and "2 widgets and 2 launchers," both in progress and highlighted in red

Updated 4 days ago

Did this page help you?