Optimizely Web Experimentation Pros and Cons

What do you like best about Optimizely Web Experimentation?

The tool allows me to point and click in order to define/create my tests. I do not have to jump through a lot of hoops to test my hoptheses Review collected by and hosted on G2.com.

What do you dislike about Optimizely Web Experimentation?

The point and click can be a bit finicky if your site is not using the best coding standards. To compare two versions of pages to see what changes worked best, the interface was not all that intuitive and I had to count on the tutorials Review collected by and hosted on G2.com.

What do you like best about Optimizely Web Experimentation?

Optimizely starts as a free tool which is great. This makes it accessible to companies of all sizes. There are surprisingly a ton of capabilities with the free tool in terms of running quick split tests. I haven't had a need for the paid version of the tool yet. Review collected by and hosted on G2.com.

What do you dislike about Optimizely Web Experimentation?

There have been some instances where the preview of the test page doesn't match what is actually seen live. Given that it is a free tool I would expect there to be some bugs like this. Review collected by and hosted on G2.com.

What do you like best about Optimizely Web Experimentation?

Their user interface is ridiculously easy. Even if you don't code, you can use their interface. Their support team is awesome. They are so helpful and quick to answer questions. Review collected by and hosted on G2.com.

What do you dislike about Optimizely Web Experimentation?

The reporting is good, but could be better and more robust. Review collected by and hosted on G2.com.

What do you like best about Optimizely Web Experimentation?

Intuitive, data accessibility/visualization, can run multi-variate tests Review collected by and hosted on G2.com.

What do you dislike about Optimizely Web Experimentation?

Can't segment results by OS, would like to compare results between iOS and Android mobile web users, within a single experiment. Review collected by and hosted on G2.com.

What do you like best about Optimizely Web Experimentation?

I love the technical prowess of everyone I have met at Optimizely. The engineers and support folks have without exception, all been very smart and their customer service is really special. And I say this after dealing with many tech vendors .

Just as one example, when we first started out doing mobile integration, we experienced a few crashes, and they actually walked through code and debugger in live sessions with us, and the Opt-gineers (hey did I just invent a new term?) would spot something amiss, fix it in 10 to 20 minutes and send us a new .zip of the iOS SDK, and this would repeat until the problem was solved. They appeared in-exhaustable. They also seem never at a loss for an answer, a fix, a suggestion, all of which make complete sense and just work.

Also, I am truly impressed by the strength of the product. When you think about product architecture, and the end to end complexity you'd encounter if you ever attempted to implement a tool like this, from the backend infrastructure (which never appears to fail or show latency) to the intuitive interface for setting up, executing tests, viewing results, to the underlying algorithms and stats-engine at the heart of all of this (in itself the by-product of a Stanford PhD Thesis) the whole picture is one of some amazing engineering. Review collected by and hosted on G2.com.

What do you dislike about Optimizely Web Experimentation?

What I dislike, or rather what I wish for the product to have, is more of the unrealized potential.: for a consolidated end to end orchestration of tests across platforms.

For example, I'd like to see a test where a test starts on a mobile app and ends with conversion on a desktop site or mweb or vice versa, or also one where a multivariate test can be run across web with variations of app download ads and promos and resulting downloads and conversions. In other words a cross between referral marketing and Optimization or orchestrated events and goals across multiple platforms (web/mweb/mobile app) with consolidated progress, result and drill-down views on the Optimizely dashboard. At present these are tedious to set up and analyze (as one has to use a combination of referral tracking tools, analytics tools and Optimizely) Review collected by and hosted on G2.com.

What do you like best about Optimizely Web Experimentation?

A/B testing - this is the core functionality that Optimizely has always been great with and only got better as they add more functionality.

Test analytics - the reporting pages are great, the impact of the tests on revenue is extremely difficult to calculate correctly. Optimizely has made a lot of updates to their statistics engine, latest round of changes upset a lot of people, but it's still doing a great job presenting very complicated data in a simple way.

Personalization - still new and barely out of beta, but already looks promising and judging by the quality of the core product, A/B testing, this should be a hit with marketers as well. Review collected by and hosted on G2.com.

What do you dislike about Optimizely Web Experimentation?

Test results - the under the hood calculations could be more transparent, while the statistics engine does a ton of very complex calculations that we don't really need to see, in the end it provides one number for "statistical significance". As that number changes over time, sometimes its apparent fluctuations make no intuitive sense. This can make it difficult to justify test results to management and the rest of the team. Review collected by and hosted on G2.com.

What do you like best about Optimizely Web Experimentation?

Our experience with Optimizely has been nothing short of exceptional. They are very responsive to our needs and questions. The on-boarding experience with this tool was very easy and they walked us through each step of the process. Depending on the level of service you purchase it may come with at a strategic adviser who can help you with strategic decision about your business.

The tool itself is easy to use and has some very practical uses based on your business. Review collected by and hosted on G2.com.

What do you dislike about Optimizely Web Experimentation?

Data is king at Princess Cruises. This tool allows you to determine the need for so many additional tests that sometimes prioritization becomes an issue with all the test that our results can yield. One test can lead to another test and so on. Connecting all of the dots in your all your data points can be a challenge. Review collected by and hosted on G2.com.

What do you like best about Optimizely Web Experimentation?

Ease of use, simplicity with a robust feature set and consistent results and delivery. Review collected by and hosted on G2.com.

What do you dislike about Optimizely Web Experimentation?

Limited dislikes if any - enterprise pricing is high, but still worthwhile. Review collected by and hosted on G2.com.

What do you like best about Optimizely Web Experimentation?

Creating personalised experiences for targeted segments is easy with an easy to use UI Review collected by and hosted on G2.com.

What do you dislike about Optimizely Web Experimentation?

Navigation is at times a little confusing. Review collected by and hosted on G2.com.

What do you like best about Optimizely Web Experimentation?

When we started implementing Optimizely on mobile (iOS and Android), the team at Optimizely was particularly helpful in working with the engineers and product team. This really accelerated the process for us to dive in and get started using the product.

The Optimizely team seems very passionate about helping their customers get the most out of their platform. They are always available to answer an e-mail, support request, forum post, jump on a phone call and even come by for a visit. On the flip side, they are always asking for feedback on the product, which provides a sense that we are working with a living, breathing company.

Optimizely has spent a lot of time understanding the intricacies of multivariate testing and statistical analysis and made it dead simple to see the results in their product. Even if your focus isn't around data, you can plainly see how your experiments are performing and focus on what you are best at. If you want to learn more, they've provided a lot of resources to help you understand concepts like statistical significance. Review collected by and hosted on G2.com.

What do you dislike about Optimizely Web Experimentation?

I'd like to see more APIs to be able to export data and integrate it a data warehouse so that we could perform our own analytics.

Sometimes it's not obvious how long an experiment will need to run based on a product's existing traffic. Having an estimate integrated directly into the experiment would be very helpful.

Sometimes I am concerned that they've made things too simple to get in and start running experiments, without having proper constructs on what experiments you should be running and what you are actually testing. Just because you've seen results on testing a button in an early stage of a funnel does not mean your end conversion is going to change. Review collected by and hosted on G2.com.