Menu

I have built and launched hundreds of A/B tests in my career across multiple ecommerce businesses.

Hypothesis

The test was created due to an increased number of customers who visit the site but don't purchase. During the Festive period, customers tend to commit to purchasing later on in the season and often research where the best places are to buy. This confuses conversion metrics therefore, we needed an experience to allow customers to not use the basket to store potential cards and gifts. We needed that experience to be an alternative to the standard conversion funnel and add more context to a customer's intentions.

Card Factory - Wish list

The solution was to create a session based wish list function using our AB testing platform. The purpose would be to add 'heart' icons to each item in a product list. When clicked, these items would be saved to a wish list that the user could utilise to save items for later. From there, the customer can click through to the item, see the item value and also see the overall total of their wish list.

Implementation

the old UI of the card size AB test

Due to the custom implementation of the website, this test had to be built from scratch. It also had to be available to both logged in and guest users. I decided to store the items saved in the users session storage and build in the option to remove the items which would also delete the data for that item from the local storage.

The build was involved but the test was launched in the first week of October, just in time for Festive research. The test was live to 25% of users which was increased from 10% for the first few days. This was to mitigate against any potential issues or conflicts with the existing experience. The test ran for just under a 6 weeks before being ramped up to 75% to capitalise on extra Christmas traffic.

The results

password revealed test screenshot

Results: The test showed us some promising uplifts across a range of metrics comparing to the original. Primary Goal: Purchase/transaction rate growth +1.23%

  • Secondary goals:
  • Overall exit rate: V1 reduced overall exit rate by -0.49%, which is a small but positive improvement.
  • Payment step: Conversion rate growth: +2.97%.
  • We saw 12k items added to wish list in the test period.
  • After an understandable slow start, users who saved a wish list showed a higher propensity to purchase.
Overall the test variant is expected to increase revenue by a median of +£4k per month (projection). (If 100% of traffic was allocated to this variation in the test period)

Next steps..

Given the effort to build this into the site. The next steps are to look at other research. I decided to launch a survey which showed only to customers who fell into the variant. I used GTM to capture the event that triggers into GA4 when a customer saw the test. I then used that event to trigger a Hotjar survey. This gave us specific feedback to users in the test. The questions included:

  • Hey there! Have you used the wish list feature on our website?
  • How useful did you find the wish list feature for planning your future purchases? (Duplicate question based on if the user did/didn't use the wish list feature).
  • What features would you want from a wish list? (Pick from a list)

The results showed a small portion of users who used Wishlist however, the majority of users who did/didn't use wish lists, found them useful in general. In terms of the features required, customers wanted it to be easy to add items to wish list and easy to view saved items. Customers weren't interested in sharing their wish lists or having multiple wish lists.

Now we had the quant and qual data to support the development of this feature. The next steps are to share this data with the wider teams and develop into the product backlog.

Overall this test showed that we can create alternative 'shopping' journeys to ensure we don't cause detriment to the conversion funnel. It also allows users to save items as guest and not have to rely on the basket.

Feel free to get in touch if you want to discuss any of the above in more detail.