One of the biggest challenges in web design is figuring out what works for your customers. There is no universal solution to making a website that resonates with everyone. Whether your site focuses on ecommerce, lead generation, sharing information, or something entirely different, each site will have a different demographic of users who will interact with the site in different ways.
A website’s look and feel should be allowed to evolve and improve, but minor changes to your storefront might lead to major changes in conversions. Maybe your customers feel safer giving payment details if you include a lock icon on your checkout buttons, or they prefer adding products to their bag instead of the traditional cart. Maybe they want to be able to bypass the cart summary and go straight to checkout.
Getting to know your customers is vital to tailor your site to their preferences but how can you do this effectively? A/B testing is one tool you can use to measure how customers react to different variations in site functionality, especially when it’s included in a reliable ecommerce platform like Salesforce Commerce Cloud.
What is A/B Testing and Why Should You Use it?
One of the best ways to explain A/B testing is to portray it as the scientific method, which we all learned about in school. In this case, it’s a process for testing content and layout variations and using analytics data to determine which version performs better.
If you follow the scientific method example, then you will start with a hypothesis on how to improve a web page. From there, you’ll set up a test with a control version of the page and a variant of the page with your proposed change, and then run the experiment by splitting visitors evenly into groups and showing the different versions of the webpages to them at the same time. You can then use analytics and other forms of tracking data to measure the results and determine which version performed better.
When used effectively, A/B testing should lead to improvements in site performance. You can improve conversion rates through UI changes, get more returning customers with user flow improvements, reduce bounce rate with PLP and PDP alterations, and lower the number of abandoned carts with a streamlined checkout process or cart page updates. Most importantly, you’ll have the data to back up your design decisions.
A/B Testing in Commerce Cloud
While there are many A/B testing tools available, like Optimizely and Google Optimize, I’d like to focus on the tools available in Salesforce Commerce Cloud. SFCC has a built-in set of A/B testing tools available in Business Manager. These tools can be used to run A/B tests against sorting rules, promotions, and content slots right out of the box. You can also run tests for template changes but this will require some code changes.
To set up a test you will need to navigate to Merchant Tools > Online Marketing > A/B Tests. Once here, you can create a new test.
While going through the form, you’ll find the email notification field which can be used to automatically send an email when the test results become statistically significant, or when the test is paused, resumes, or ends.
The Key Metric dropdown allows users to select the KPI that will be used for comparing performance between the different versions.
The participation trigger defines a certain action the customer must take to be sorted into a group for the A/B test. This can be used in combination with the Customer Groups qualifier to further refine your audience. For example, you could set the participation trigger to a category page view and set the customer group to MobileCustomer to limit the test to mobile category pages.
Participant expiration is used to define how the group assignment is handled for the test.
- Per session will last until the session expires and re-assign a group when a new session is started
- Per customer only works with logged in customers but will keep their group the same for the duration of the test, even across devices
- Per Browser will keep the group consistent across all sessions coming from the same browser
The start and end dates are used to define when the test will run. It’s worth noting that there is a 90-day maximum test length.
Finally, the test segments area is where you will define the variations to test. Note that the control version (the current experience live on the site) is already present and accounted for in the Control Group allocation. You will not need to set up a test segment for the control.
To add a new test segment, fill out the ID, description (optional) and allocation percent, then click the Add Experience dropdown choose the option that lines up with your test. Sorting rules, slot configurations, and promotions will all override the respective configs from the control group, so you can test how different configurations perform.
The ‘Enable Custom Scripting’ option is used to set up a test where the changes are made in the server side code (isml templates, pipelines/routes, and scripts). When using a custom scripting test segment, developers will need to update code using the ABTestMgr.isParticipant() method to check whether the current user is part of a specific test segment and, if so, alter the experience accordingly.
After you have set up the test, you can preview the different versions by using the storefront toolkit to preview the site as a member of a specific test segment. After the test has run, you can return to the A/B test list in Business Manager to view the test results. This view will show you how the test segments performed compared to the control group. You can use this data to determine whether the alternate version was more successful and move forward with making those variations the default experience.
A/B Testing Best Practices
So, now that you know how to set up your A/B tests, here are some tips to get the most out of A/B testing with Salesforce Commerce Cloud:
- Make sure your test segments are balanced. If you are testing a control group against a single variation, aim for a 50/50 split of customers. Having the split skewed one way or another can interfere with achieving statistical significance and may lead to inconclusive results.
- Keep your tests focused. Avoid adding too many variables into the mix when running a test. You want to be able to clearly attribute performance metrics to specific changes that were made.
- Don’t make changes mid-test. Early responses might not match the final results and altering the webpages in the middle of a test can throw off the results and invalidate your data. Don’t let early results push you towards making changes to counteract or reinforce the results you’re seeing. Let the test run to its conclusion and use those final results to drive your decision on what changes to make.
- Don’t be discouraged by failed tests. Sometimes your hypothesis will be wrong and may have a neutral or negative impact on the site. Even in cases like these, you finish the test with more insight into how your site is performing and what your customers are looking for.
Continue to run tests. A/B testing is an iterative process. Take what you have learned from previous tests and apply it to new elements of the site. Form new hypotheses around how to improve your sites and put them to the test.
Contact the A/B Testing Experts at CQL to Learn More
Reach out to CQL’s SFCC development team today If you’d like to learn more about A/B testing with Salesforce Commerce Cloud. Our talented developers can help you find bigger, better, and higher-converting solutions that your customers will love.
