To Test or Not to Test?


To test or not to test: that is the question. Conventional knowledge holds that ecommerce website owners should never perform marketing tests during spikes in website performance, but some marketing experts actually have a very different opinion. According to some, testing during a spike can be effective because it gives you very different information than testing during normal traffic.

When does testing during a spike in activity help you? If you can catch a spike at the very beginning, testing can help you to get the most sales throughout this period of increased traffic. You can quickly identify how to take advantage of the trend and make the most out of every single visitor.

Testing during a spike is more time-efficient because you can get a statistically significant sample faster. If you are testing for factors that should not change even when traffic is low—a different check out button, for example—then the results will likely be the same that you would have gotten over a much longer period of time during a slower time.

Sometimes you want to find out how customers during a particular time react to different variables on your website. For instance, perhaps you are interested in the behavior of holiday shoppers. Obviously you will need to test during the holidays, even if these are a busy and atypical time for your website.

There are also a few good reasons not to test during a traffic spike. The most compelling of these is that you are probably not just having abnormally high traffic, but abnormal traffic in general. Your results will not necessarily be representative of the way your everyday customers would behave. Thus, argue many experts, test results gained during traffic spikes are almost useless. 

There are a few things you can do to make testing during a spike more effective. First, remember that it may not be representative of behavior during less busy times. Let common sense be your guide as always. In addition, make sure your test is relevant; that is, it needs to test the behavior that you expect during this time and also to have a limited number of variables so you can test quickly. Timing is crucial, because your test needs to start after the spike has begun and conclude before it is over in order to get a clean sample. Make sure your test can be held in the amount of time that you have available during the spike.

Tests taken during traffic spikes can be effective and indeed even priceless. However, you need to test at other times as well. After all, if you are like the average ecommerce website owner, your site spends a good amount of time with slow and steady traffic. Obviously you need to perform tests at these times as well.

A/B and other types of testing can be very effective ways of evaluating customer behavior and developing new ways of reaching your consumer base, but you need to make wise decisions in order to get the best data possible.