ab-testA Review of 2019 A/B Testing Tools

There are many A/B testing tools on the market right now. If you are planning to run an A/B test on your website or landing page, it’s often hard to tell which tool will work best for your business.

As experts in conversion rate optimization (CRO), we have used some of the A/B testing tools listed in this review. There are some newer tools on the market that we have not tested yet. Although this is not a full list of tool providers, it is also not a full review of each tool. It is a list of the tools that we recommend and a list of the newer tools available for you to investigate.

  1. Optimizely is an enterprise-level experimentation platform. It is the most well-known platform in this space due to a lot of VC funding and advertising. At $1,440 per month (paid annually) it is one of the most expensive of A/B testing tools. They also offer a pay-as-you-go pricing model that costs $49 per 1000 unique monthly visitors. The platform supports testing on the web, mobile web and mobile apps.
  2. VWO (Visual Website Optimizer) [Updated 1-31-19] I just finished a meeting with VWO where they provided a sneak peak of VWO’s new interface. It has some new features that make it very easy to see with visual graphs if a test is stable. They also removed some of the confusion when comparing variations and how they are doing against control. The data is presented so a novice can understand it and an expert can dig into the statistics. The data and graphs will also be exportable to an PDF to make reporting easier to clients. The new version should be out in second quarter this year I will update this post when they have a firm date. Regarding my next comment… they are reinvesting in the tool to justify cost and this will be going beyond just the presentation layer of the system.  [Updated 1-17-19] VWO has completely changed its pricing model to follow Optimzely. There are now only annual subscriptions and the price has doubled. However the A/B tool is still buggy so it is hard to justify the price. (Note every test system has it idiosyncrasies and bugs.)  The biggest issues is you can’t increase and decrease the number of test per month by switching plans as in the past. You are now purchasing tests per year which for consultants and agencies is hard to predict. 120k users under test is now $1188 a year or $99 a month. It was $59 a month. 30k users under test is now $2388 a year or $199 a month which comes to $66 for 10k.  You need to double check your tests to make sure the variation you created is what is actually being shown. I always do this outside of their testing system, and I do this with Optimizely as well. VWO offers other CRO tools like heatmaps and videos. VWO works across mobile, tablet and desktop websites.
  3. Freshworks has acquired Zarget. [Updated 1-17-19] The pricing is very good and the support is excellent. 50k users under test for $159 a month and no annual commitment. They also have a number of other conversion tools similar to VWO. The testing system presents nice reports and does conversion rate over time which I feel is a key visual indicator of test stability when comparing variations. Setting up some test is a bit different but it is clear they are building better functionality. The acquisition included all the developers so they did not loose any talent.
  4. Convert has received high praise from its users, who tend to be SMB customers. It was the first solid A/B testing tool on the market before Optimizely. Their focus is on being the best in the market for website A/B testing.  [Updated 1-17-19] Price starts at $699/month or $599 a month (paid annually).  It has an intuitive interface and excellent customer support.
  5. Crazyegg known for it’s excellent heatmap tools has added A/B testing. Pricing starts at $29 a month for 30k visitors. Note this is not users under test so it is harder to compare to other testing pricing plans and will have to throttle the traffic on the pages for larger sites. That said I would love some feedback on their A/B tool.
  6. Google Optimize is an A/B testing tool that connects to Google Analytics. Since you already have all your data in your Google Analytics account. You can’t beat the cost and convenience of this tool since it is free. It allows you to split test up to 10 full versions of a single page, each delivered from a distinct URL. However the goals setting is very cumbersome relying on Google Analytic goals and segments and in some cases would take Tag Manager to setup goals. Hopefully in the future they will build features that take seconds in other tools instead of 10 to 30 minutes. [Update 5-7-17] I asked Google about the goal issue and they said  that within a year you will be able to set goals on elements and that next month they will be adding Adwords to Optimize.] [Update 1-17-19] – Google launched new features today. I have not played with them yet but if they can set goals quickly without google analytics then they are moving in the right direction. [1-31-19] The new goals called Custom Objectives gets around the biggest issue of only using the Google Analytics goals which were limited. They still don’t have one click goal setting but it is getting there. So the time it takes to set all of these up compared to clicking a button and calling it a goal can be significant. What is nice is it is easy to set purchase and revenue goals if you have enhanced eCommerce already set up. This can be difficult in other systems.
  7. Omniconvert provides an A/B testing tool for tablet, mobile and desktop. Some of the unique features are ready-to-use overlays, multiple overlay formats, OnPage variables, frequency capping and IP address targeting. Pricing starts at $38/month (paid annually).
  8. AB Tasty is a tool that tests user re-engagement and it is designed for marketing teams. You can modify pages using a visual editor, so you don’t need a technical background to use it. This is a good tool to start A/B testing if you’ve never done it before and you don’t want to invest a lot of money. Monthly pricing is in British pounds.
  9. Intellimize  helps marketers drive conversions and revenue growth by using artificial intelligence to personalize websites to deliver the right experience to each visitor on any device.   Since this is a newer approach I asked some questions to clarify how they do what they do.
    Pricing? – We are a SaaS based pricing model with an annual term and monthly pricing.  Cost varies considerably based on the needs of the client and we don’t publish pricing as our work for clients.  We work mostly with middle market companies. As you noted, it’s common for companies to run their entire site through our system as we can optimize flows for them.
    Does this effect SEO? – Great question. Google has established guidelines for companies like us to follow.

    1. Google asks personalization companies like us to serve a random experience to them. We do not “listen” for Google.
    2. For keywords that are important to you, we ensure that variations of your page contain those keywords and metadata to ensure that your page will keep ranking.
    3. It’s important to remember we are changing elements of the page (headlines, CTAs, images, etc.) rather than setting up new pages so your existing work will remain intact.

    How do you measure/define the audiences for each variation? Is it from data points in the site or data found outside of the site or both?

    Our system does not require any data integration to get started and can accept data from 1st and 3rd party sources.  The most commonly used data by our system would be contextual data because it’s readily available for each visitor.  This includes features like: IP address, device type, browser, time of day, day of week, and geographic location (city level). We also commonly use firmographic and demographic data as well as behavioral data about how the user interacts on the site.

    Finally, our approach to A/B testing is rooted in the principle that A is probably best for some people and B is probably best for others.  Instead of finding a global winner (the best average) our AI-based system learns which version of your site is best for each visitor and automatically allocates more traffic to your best performing variations.

  10. Adobe Target provides a intuitive user interface and works best with sites that already use Adobe Analytics. It is easy to setup and test. Pricing is not available on their website but is around the cost of Optimizely or more.
  11. SiteSpect is a server-side testing solution and is the #1 choice for more technically savvy users and clients who are sensitive to security. Pricing was not available on the website.
  12. Sentient Ascend is an AI-powered conversion rate optimization platform. It is a new type of testing tool that uses machine learning algorithms which means the A/B test shows what combinations of your pages gives the best lift. Pricing was not available on the website.
  13. Oracle Maxymiser provides simple A/B tests all the way up through complex multivariate tests. You can test on any public or secure page with an easy-to-use visual editor. The tool automatically builds a unique customer profile for each online visitor. Pricing was not available on the website.
  14. Qubit is a testing platform focused on personalization and advanced segmentation. Like most A/B testing tools, it uses multivariate to show which combination of changes are more valuable than the individual changes. Pricing was not available on the website.
  15. Splitforce has an A/B testing tool for mobile apps, which is more challenging than testing a website or mobile website because apps run locally on mobile devices and don’t access a live internet connection through a browser. Pricing was not available on the website.
  16. Leanplum offers an integrated A/B testing tool for native mobile apps. Pricing was not available on the website.
  17. Taplytics is a visual A/B testing tool for native mobile apps and for web pages. Pricing was not available on the website.
  18. AppTunnel is a mobile A/B testing and feature flags platform for iOS and Android mobile apps. Pricing starts at $79/month.
  19. Monetate does A/B testing and personalization. Their product is targeted to retailers and e-commerce. Most of their customers are enterprises.
  20. Kameleoon is a website optimization solution with A/B testing for desktop and mobile. Their platform is devoted to web personalization. They offer a freemium pricing model.
  21. ChangeAgain.me is a A/B testing tool for websites. It integrates with Google Analytics and has a visual editor to update the visual elements on a website. Pricing starts at $14/month.
  22. Nelio A/B Testing is a website optimization solution for publishers and marketers using a WordPress site. Users can A/B test any piece of their WordPress site. Pricing starts at $29/month.
  23. ClickThroo is a landing page marketing solution. They offer an A/B testing tool for both desktop and mobile that has an auto-optimization engine. Pricing starts at $95/month.
  24. Invesp offers an A/B testing tool called Pii testing optimization engine. The website shows it as a free tool.
  25. Analytics-toolkit This site has a number to A/B statistics and analytics calculators. I always run my own statistics with tools that I created in Excel to double check results when using A/B tools.
  26. SiteGainer has a drag-n-drop website editor that loads directly on your website and allows you to A/B test any part of your website.
  27. TrustRadius collects in-depth user reviews on a variety of A/B testing tools and includes a user ranking for each product. TrustRadius also has a TrustMap comparison tool that allows you to compare products and services based on user satisfaction ratings.

If you have an A/B testing tool that you like and recommend, contact us and we will add it to the list.  

Follow Me

Greg Ahern

Greg Ahern Founder and President of Ometrics® is a fanatic about conversion rate optimization, AI chatbots and lead generation. Greg has been a successful Internet entrepreneur since 1994. He speaks at conferences and webinars and has built a number of internet businesses, including web marketing, web development and internet lead generation, which have been successfully acquired. Greg is the Denver Chapter Leader for the Digital Analytics Association. You can follow Greg on Twitter @gregahern and join his CRO Hacks Groups on Facebook and Slack.
Greg Ahern
Follow Me

Latest posts by Greg Ahern (see all)