Splash About - A/B Testing

After speaking to Splash About we agreed to conduct an analytics review of their site to gain insight into their customers behaviour and provide small CRO changes to help improve the overall efficiency of the site.

We were then tasked to go through a full CRO workshop and run a series of A/B Tests to help find those marginal gains that add up to big results.

Research and data collection

Before we sat down with Splash About, we went through a research and data collection workshop. Looking into qualitative and quantitative data, making use of Google Analytics, we were able to gather insight into how customers interact with the site . We were able to understand the customers journey, pain-points and potential conversion rate bottlenecks they may face.

Piecing together all of the data and research we were able to clearly define the primary metrics we would look at improving across the site.

Hypothesis generation

The next step after the research is hypothesis generation. We sat down with Splash About and started to brainstorm potential hypotheses to prove, leveraging the insights we found from the research stage.

Each hypothesis is written to predict a specific result based on a proposed change. For example, “Changing the Buy Now CTA button colour to a more vibrant colour will increase click-through rate because it stands out to the user more”.

Prioritisation matrix

Once we have created a number of hypotheses, it is important to run through a prioritisation matrix to help showcase which hypotheses should be prioritised over others.

The matrix uses business value (how significant is the potential impact on key metrics), implementation complexity (how difficult and resource intensive it is) and certainty of improvement (how confident are you that it will improve usability and CRO). Hypotheses that have high business value, low implementation complexity and high certainty of improvement will take priority.

Design and A/B Testing

Taking the high priority hypotheses, we then went through a design process where we created alternate designs that make use of the proposed change. These designs would go through a feedback stage and get final sign off from Splash About. Once we had a signed off design, the design was uploaded to the site using Personizely, an A/B Test third party application that runs the tests.

We would agree on a timeframe for each hypothesis, giving enough time for the test to collect the data and determine if the test was successful or not. The successful tests would then replace the original variant.

As these tests have both shown positive results, they have been promoted to the main variant. These increases have come from a few very simple tests and we will be continuing to conduct more tests to make sure we are maximising Splash Abouts CRO.