Sixpack A/B Testing
Last updated
Last updated
The Phoenix platform has the capability of running a variety of A/B Test Experiments to gather insight on what small changes on the web experience between a set of variants perform better. The overall aim is to find what small changes lead to more conversions from our existing website visitor traffic.
Depending on the context for the kind of test being performed, there are 2 test approaches that Phoenix supports, A/B Tests executed in code, or A/B Tests executed using our Contentful CMS.
If a Sixpack A/B Test intends to experiment with layout changes, color variatons, or other design or feature variations, these experiments are best executed entirely in code and set up by one of the developers on the team.
The section provides more information and instructions regarding these types of experiments.
If a Sixpack A/B Test intends to experiment with content copy changes, content asset changes, or visual order of blocks on a page, these experiments are best executed within the Contentful CMS by a content editor, using the SixpackExperiment content type to help set the experiment up.
The section provides more information and instructions regarding these types of experiments.
To perform all these A/B Test Experiments, we utilize an open-source, language-agnostic A/B testing framework called . Sixpack runs as an instance on a server that Phoenix communicates with to exchange data and also pass along information to Sixpack regarding successful conversions on different experiments.
Regardless of whether a Sixpack A/B Test Experiment is a code test or a Contentful test, in both cases the core component that initiates and participates the current user in the experiment within Phoenix is the SixpackExperiment React component.
When a SixpackExperiment component is rendered on a page, upon mounting the following series of steps occur:
The component configures some settings to set the name for the experiment, and define what the alternative test options are to choose from.
It then calls the participate() method on the Sixpack class via the sixpack() helper function, which makes an API request to the Sixpack server, and participates the current user in the specified A/B test experiment.
The response from the Sixpack server returns with the name of the selected alternative test option that the current user has been participated in.
Within the Sixpack service class, this experiment along with the associated alternative test option that was selected for the user is added to the experiments object property on the class.
The participate() method resolves and returns the name of the selected alternative option for the user to the SixpackExperiment component, and assigns it to the selectedAlternative component state property.
When the SixpackExperiment component state changes with the name of the selected alternative test, it re-renders with the component specified by the selected alternative test name.
On the rendered page, the current user sees one of the various test alternatives; specifically the test alternative that they have been assigned by the Sixpack server.
Again, regardless of whether a Sixpack A/B Test Experiment is a code test or a Contentful test, in both cases a user is converted on a test alternative for an experiment when they take a particular action on a page that triggers the conversion. This action typically involves interacting with a user interface element, like a button, that can programmatically trigger Phoenix to count the conversion on a test.
Within Phoenix, conversions can be triggered by a variety of actions, like a campaign signup, a reportback post or clicking a button on a call to action block.
Thus, the following describes how a user is converted on a test in Phoenix within the context of a campaign signup action, but can be applied to other actions and their respective user interface element interaction.
While on a Campaign landing page, with a Sixpack A/B test experiment running, upon clicking the button to signup for the campaign, the following series of steps occur:
The storeCampaignSignup() function dispatches an action to store the signup, and within the payload for the action there is a sixpackExperiments property that specifies the conversion as signup to trigger converting Sixpack experiments on signup.
Next, the sixpackExperimentMiddleware catches the dispatched action and it checks for a payload.meta.sixpackExperiment.conversion property within the action payload. If it finds a conversion specified on the property, it proceeds to run the convertOnAction() method from the Sixpack service class.
The convertOnAction() method converts all available experiments within the experiments list of registered experiments for a page that match a specified convertable action. For example, this will convert all experiments on the page that specify conversion on a "signup" action.
After finding all matching experiments that should convert on the specified action, it calls the convert() method on the Sixpack service class, which makes an API request to the Sixpack server for each matching experiment, and coverts the current user on the specified test for a respective A/B experiment.
For other types of interface elements that trigger a conversion (other buttons, links, call to action, etc), they will need to ensure they also ultimately dispatch an action, similar to the convertOnSignupAction() method mentioned above, that can be caught by the sixpackExperimentMiddleware and finalize counting the conversion on the Sixpack server for a specific test alternative.
To communicate with the Sixpack server, Phoenix has a Sixpack service class located in . This class is only instantiated once during the page request cycle and only if an experiment is initiated; it contains a key-value store of all the experiments running on the current page being viewed.
There is a convenient sixpack() helper function you should use as an alias, which creates a new instance, or retrieves an existing instance of the Sixpack service class, ensuring that only one instance of the Sixpack is ever called. This function can be found in .
The SixpackExperiment component can be found in directory.
When clicked, the CampaignSignupForm component calls the storeCampaignSignup() function from .