This is the second blog in a multi-part series that will outline how to create a successful optimization practice within your organization. To check out the first blog, please go here.

We left off the last article by discussing the importance of a data-driven mindset. Once you have this established, you are ready to get more specific around what to test and where.

First you want to figure out ‘What’ you are trying to influence and ‘Where’ you should focus your time. The ‘What’ question could be easy or difficult depending on your organization and the number of stakeholders at the table. For most e-commerce organizations it is best to optimize towards revenue, this will ensure you are always moving the bottom line. Optimizing towards orders could result in a reduction of revenue by increasing the volume of low value orders at the expense of your high value customers. For media companies, this is where the ‘What’ question can become a little more ambiguous. Some companies may have a membership, but should you put your content behind a paywall? Other companies make their money from ad revenue, but where is the diminishing return?

Optimization can help answer these questions, but not alone, you will need to analyze historic data and monitor performance over time to ensure you are on the right path. Simply conducting an A/B test that reaches statistical significance isn’t enough (WARNING: don’t place too much trust in your tool’s statistical significance calculation… but, more on this in a later post). Tests of paywalls on media sites may show promise at the beginning, but where is your point of saturation for paying members, how will this impact SEO, how much repeat traffic will you lose who are not willing to become a paying member?  These questions can all be answered, but will require more forethought, planning, and analysis to accomplish.

Once you know the ‘What’ you’ll have to identify ‘Where’ to test. The simplest method is to analyze your high traffic pages for their conversion rates. The pages with the highest traffic and lowest conversion rates should be the initial focus of your optimization efforts. Note, you will want to look at both the conversion rate and the missed potential to fully understand the opportunity available. Pages with lots of traffic and a high conversion rate could still have a high total opportunity, but squeezing additional conversions from these pages might be more difficult. Conversely, a page with a low conversion rate and mid-level traffic numbers may require a massive increase to hit the same absolute total as a page with more traffic and a better conversion rate. As you gain more experience testing on your site, these types of situations will be easier to identify and estimate outcomes.

So, now you have a list of pages that you can rank against your key metrics. From here, you can further narrow down the list by removing the pages that don’t have the same expected conversion as what we are currently focusing on. What does that mean? Simply put, any pages that were not designed to elicit the metric we are optimizing toward (ex: sitemaps, about us, company history, etc…) can be removed from the analysis to help you focus on the pages that will move the needle.

Now that we have the ‘What’ and the ‘Where’ identified, we want to work on hypothesizing ‘How’ the conversion rates for these pages can be improved. This involves working closely with the client team to look at the intent of the page and possible scenarios for how the page is failing to achieve its goal. Analytics data can help with hypotheses here, too, you can investigate where people predominately go when they aren’t going to your desired next action. Do they not have all the information they need to proceed? Are they going back to another page to clarify refund policies or warranty information? Other sources of data for hypothesis generation are voice of customer/survey data and usability testing. These data sources will give you more qualitative information on how the user is feeling or what they are thinking while trying to navigate your site. These sources are valuable insights for optimization, as they help you better understand the point of view of the end user, where analytics data simply shows you the outcome. From here, ideally, you are generating alternative test ideas – how to improve the page to improve the metric you are focused on.

Finally, let’s talk about the importance of tracking secondary KPIs. Above, we discussed the importance of identifying the ‘What’ you are trying to optimize, but most sites don’t have a single goal, they have multiple goals that could have an impact on the overall performance of the site. A good analogy is a race car team trying to improve the average sustained speed of their car. They conduct a series of tests and conclude that using new tires, increasing engine horse power, and altering the body will result in an increase of 5 mph of average sustained speed. Unfortunately, they did not measure tire wear and fuel consumption during their tests, resulting in more frequent pit stops and effectively negating any improvement in speed.

Optimization is a deceptively complicated process that, on the surface, seems so very simple: serve content A and content B and see which version gets the most conversions. Unfortunately, it is not that simple. Optimization programs are usually a complex undertaking, requiring the inclusion of multiple organizational stakeholders and business units. This complexity opens optimization up to numerous potential pitfalls.

We’ll talk more about the pitfalls in our next blog post. Additionally, we will start discussing the “who” aspect of testing – the personalization and segmentation dimensions of Optimization that can really drive results.  For today, remember that prioritizing the pages where you can have the most impact is a critical foundation to your testing process.  Using data to uncover the places to focus your efforts is your best bet to having a test plan that moves the needle and shows true impact of Optimization to the organization.

Further Reading

If you’re interested in more on Optimization
Part 1 of Optimization: The First Step

Six Best Practices for Maximizing Conversion Rates

Three Key Analytics Questions for 2015: Governance, Big Data and Customer Journeys

Advanced Analytics and Marketing Optimization:  Two Sides of the Same Coin