What Is Amazon’s “Manage Your Experiments” and How to Use It

Updated February 2021: Amazon now lets you run A/B tests on your main product images.

After years of closely guarding their domain, causing a testing “black hole” for sellers with ambitions of optimization, Amazon is finally starting to roll out tools to enable sellers to squeeze every dollar of effectiveness out of their Amazon marketing efforts.

The latest tool in that growing seller arsenal is Amazon’s “Manage Your Experiments” feature.

In this post I’m going to walk through what Amazon’s “Manage Your Experiments” is, who has access to it, and how to get the most out of it.

 

What is Manage Your Experiments?

Manage Your Experiments is a new tool that allows eligible sellers to run A/B tests on their A+ content, product titles, and, as of 2021, their product images.

In the words of Amazon:

“You no longer have to spend months testing different product marketing strategies when creating A+ content on your brand’s listings. The Manage Your Experiments tool helps you test two versions of A+ content simultaneously, and shows you the results in an easy to understand dashboard. Through the use of A/B tests you can compare the two versions of content and see which version resonates with Amazon customers.”

This means you’ll be able to test many different aspects of your A+ Content and get data back that will show you definitively what is and isn’t converting the best for your listing (i.e. producing more sales).

By testing different elements of your A+ Content one-by-one, over time you’ll start to get a picture of the kind of text and images that appeal most to your target audience.

The results will also give you insights about what should work best across all your A+ Content.

Some of the A+ Content elements you’ll be able to test are:

• Headlines
• Images
• Text
• The order in which your A+ modules are stacked
• Adding extra modules to your existing A+ content
• Adding charts, graphs and graphic in place of photography for your images

The same principles apply when experimenting with your product titles and now main images.

2021 Update: Running A/B Tests on Your Images

As of February 2021, Amazon now let's you run A/B tests on your product images without any third-party tool.

You can start an A/B test on your product images on the Manage Your Experiments feature (You need to be logged in to Seller Central to access the link). If you're looking at the Home dashboard of Seller Central, go to Brands > Manage Your Experiments.

You should now see the Manage Your Experiments dashboard. As of February 2021, Amazon lets you run A/B tests on your A+ Content, Product Title, and Main Image.

Amazon has mentioned plans of rolling out similar A/B testing features for Description Bullets and Secondary Images in the future. This article will be updated once those features roll out.

Being able to A/B test your Main Image is big because it's the first thing anyone sees when they land on your listing. Knowing which image performs best will likely boost your traffic and conversion rate.

One head-scratcher for product image A/B tests, though, is that Amazon’s product image requirements are already quite strict and specific.

Creating a Version B that complies with the same requirements AND is different enough (experiments with versions that are very different from each other yield more meaningful results) from your Version A can be quite tricky.

Who Can Use Manage Your Experiments?

You must meet the 2 levels of eligibility (More on eligibility further down this article) to run experiments on your product images:

• Brand eligibility
• ASIN eligibility

Right now, Manage your Experiments is only available for vendors and brand registered sellers on the U.S. marketplace (If you’re not part of Amazon’s Brand Registry, you can learn all about how to do that here).

However, in order to access it you must have listings with existing A+ content that get a high enough volume of visitors so that Amazon can “provide statistically sound results for your A/B tests.”

The quick way to check for ASIN eligibility is to go ahead and select any of the options from the dropdown mentioned earlier. The following window will actually show you a list of eligible ASINs. Here, you can select a Reference ASIN from the list or look up a particular ASIN you want to experiment on.

As with a lot of things Amazon, there are no specifics as to what qualifies as a high enough volume of visitors.

For A+ Content, another way to check if your ASINs are eligible is to go to the A+ Content Manager in your Seller Central account:

Once there, A+ Manager will show which of your current ASINs that have A+ Content are eligible.

If you have no eligible ASINs you’ll get a message like this one:

If you do have eligible ASINs, you’ll be taken to the Manage Your Experiments dashboard where you’ll be able to create and manage all of your A/B tests for your A+ Content.

How Do I Create A/B Tests?

On the dashboard, simply hit the button that says Create a New Experiment and Amazon will let you choose which among the A+ Content, Product Title, or Main Image you'd like to run an experiment on.

This is also where you can check the status of you Active Experiments.

Once you select an item you want to run a test on, a workflow will then appear that will take you through the steps you need to take to set up your experiment. Once done, you'll have two versions of your element ready to test against each other.

Whichever item you're setting up an experiment for, there will be an info box requiring you to key in an Experiment Name, a Hypothesis, and a Schedule.

The Experiment Name is for your own reference. Make it clear and easy to distinguish in case you are running multiple experiments at once.

In our example, we also put in a hypothesis which is basically what your experiment is all about—Changing [X] into [Y] will [Z]. Amazon provides a couple of examples, so try to make it as simple as you can.

Lastly, set the time frame when you want your experiment to run. Somewhere around 8 to 10 weeks is ideal.

For the Main Image, you'll simply be pitting a new image against the one you are currently using, but remember to make it varied enough so that your results actually mean something. Here's what the setup page looks like:

If you want to run an experiment with your Product Title, there's an option to copy the Title you are currently using into your Version B. You can then just make changes to come up with your alternate Title. For meaningful results, make very specific and isolated changes, instead of changing, say, a string of several words at a time.

And here's the setup page for the A+ Content:

Then schedule the experiment, leave it to run its course and come back when it's done to check on the results.

One tip is to try out alternate images that are easier to understand and more info-rich than the one you currently have.

Once done, your experiment will be up for manual review by Amazon. Follow all the steps and make sure your Version B complies with Amazon's guidelines to avoid any do-overs.

Best Practices For A/B Testing In Manage Your Experiments

When A/B testing you should only ever test one element at a time.

The reason for this is, if you have version A with one headline, one set of images, and one set of text, and you have Version B with a completely different headline, different images, and different text, when the results come back and you learn Version B was the most effective, you’ll have no idea why.

Was it the different headline that made it more effective? Or was it the different images or the different text?

The only way to know is to test one element at a time.

So if you want to test the headline, run two identical pieces of A+ Content and only change the headline.

If you want to test your images, same thing. Use the same headline and text, and only change one of your images.

The same goes for titles and main images. Try to test one specific element at a time to isolate which ones actually have impact.

You also want to give it some time before interpreting results. Trying to interpret your results too early could be misleading.

It can be a slow and somewhat tedious process, but that’s the point of optimization. The goal is to squeeze every last drop of ‘conversion juice’ out of every element until you end up with the perfect combination of elements.

And then you can test that against something else.

How Long Should I Run Amazon A+ Content Experiments?

Manage Your Experiments allows you to choose any duration between 4 and 10 weeks.  Within that, you can cancel your experiment at any time.

However, as with the best practices section above, the whole point of A/B testing is to get as much data as possible before you make a decision. So for the most reliable results you’re going to let your experiments run for as long as possible.

One Of The Big Flaws Of Amazon’s Manage Your Experiments

Being able to now run experiments on Product Titles and Main Images without using third-party software is a big update to Amazon's Manage Your Experiments. But, I’d be remiss if I didn’t mention a big gotcha before concluding this article.

In typical Amazon fashion, the metrics to be eligible to run experiments are a bit vague.

Some sellers on Amazon’s seller forums mention qualifying with ASINs that have around 1000 to 2000 sessions/month—but others have ASINs with over 6000 sessions/month and still don’t qualify!

While I'm at it, the fact that you can only use Manage Your Experiments if you have a high-traffic ASIN seems a bit counterintuitive, since its entire purpose is to drive more traffic and sales.

That makes sense, of course, since higher traffic yields more meaningful data for your experiments.

But that puts sellers who might actually need a boost in those areas in some sort of catch-22: You can get high traffic using Manage Your Experiments; Manage Experiments requires you to have high traffic.

It's not the end of the world, though. Amazon says if you're not running a high-traffic ASIN, you could still drive more traffic to your listings using other advertising methods or some other means.

But here's to hoping the Manage Your Experiment tool evolves further in the coming years to address that drawback.

Conclusion

Manage Your Experiments is a nice feature that takes some of the guesswork out of optimizing your A+ Content, Product Title, and Main Image, and testing variables to create your highest converting listings possible.

Whether it’s worth the time and effort necessary will depend on where you are with your brand right now – if you’re just starting out there may be lower hanging fruit that will give you a better ROI for your time.

However, if you’re a more established brand, testing each certain elements under this tool could give you the data that helps you squeeze every last dollar out of your listings.