“The SplitSignal tool is really powerful. It made the whole process of testing and changing metadata more efficient. We didn’t need to get the devs involved and had the ability to test each and every idea that came to our mind without having to run for approvals. This opens up so many possibilities we didn’t see before.”

Stephan Cude, Senior Tech SEO Manager at Leafly

About Leafly:

From strain and local dispensaries directory to how-to’s on weed growth and finding the best local deal, Leafly is what you’d call an educational and informational hub on all things cannabis. Based in Seattle, Leafly attracts over 125 million visitors each year and features more than 4,600 marijuana retailers and a 5,000+ strain database. With over 4 million orders processed through the site annually, Leafly is a one-stop shop for both legal weed retailers and consumers that are looking for a “near me” weed store or want to find a bargain and purchase directly from Leafly’s site.

Challenge: Run SEO split tests at scale

Leafly’s website has nearly 2 million pages, and its directory pages are all built on templates. This means that a change to a single page affects the rest.

Operating in a highly competitive landscape, the Leafly team had to ensure the ultimate visibility to each and every page so that they have both the rankings and the traffic (aka CTRs). The only way to do that is to run various SEO experiments, changing variables one at a time.

But SEO split-testing comes with a lot of challenges:

Unlike e-commerce where split-tests are built into the concept, traditional split-testing isn’t possible with SEO. Leafly knew that Google can only digest and evaluate one experience at a time and serving Google different user experiences with the same URL would negatively impact their SEO.
Implementing changes for a split test requires a lot of support from dev teams. And then there’s the wait time - you have to let the test live for at least a few weeks to come to any conclusions.
With a lot of internal and external factors affecting page and site-wide performance (think Google algo updates and various on-page changes that are hard to control in large companies where a lot of people have access to on-page content changes), test results aren’t always easy to interpret.

In Leafly’s case, they faced even more obstacles:

Leafly has a dynamic website, which means that small changes can affect the performance of the entire directory and each page within it. So even the slightest decrease in performance would imply drastic traffic consequences.
To reverse the changes, they’d once again need to turn to dev teams’ resources, which discouraged them from running split tests at scale.

Leafly knew they had a lot of missed opportunities. First of all, they were wasting their resources on tests that could potentially have a negative effect. And this, in turn, meant that they weren’t spending time and resources on tests that would bring in a positive impact.

Solution: SplitSignal tool

One way around all the challenges was to run control group testing. Yet with thousands of pages on the radar, this seemed like a mission impossible.

So Leafly turned to the split-testing power of SplitSignal - a tool designed by the Semrush team to allow SEOs go smart and efficient about SEO A/B testing.

“The general notion of controlling metadata in the cloud is so promising - it used to be a project impossible to pursue for a site with thousands of pages. And now it’s a mission possible”, says Stephan Cude, the Senior Tech SEO Manager at Leafly.

Leafly ran a test on 500 product pages (253 pages in the control group and 247 pages in the variant group) changing "marijuana" to "weed" in titles. In 21 days the test result was positive, increasing the number of clicks to the variant group by 2.7%.

One-by-one, each challenge they faced before was disappearing thanks to the tool:

No more need to involve dev teams in the test

SplitSignal allowed Leafly to easily run split tests without getting their dev teams involved. The only time their resources were needed was during the setup and for the site-wide implementation of the positive tests.

No more need to get any approvals or deal with concerns about the full-site impact

With SplitSignal, Leafly had full control over the scope of pages that were participating in the test. Moreover, they could control metadata on a URL level. So one change didn’t have to affect the whole directory, which, in the negative test result circumstances, implied a lot less impact to the rest of the site.

Ability to quickly spot negative results and stop the test before it leads to a dramatic impact

The tool gave great visibility in regard to each test performance.

While initially, Leafly’s SEO team would check each test’s results on a daily basis, with time, they’ve come up with weekly checks. As they were testing both local directories and strain databases - their most high-traffic pages - whenever they spotted a drop in either rankings or CTR, they simply stopped the test before it led to significant changes.

Losses become wins when you spot them quickly

As they could save time and resources while running more tests, even negative results were seen as good as they indicated things that should not get implemented, and hence, delivered a more data-driven way to manage site-wide SEO.

Leafly ran a test in October on 248 pages. The test has run for 30 days. Here is an example of how this would look in a mock-up of SERP:

Embracing more opportunities than ever before

The notion of missed opportunities was lingering in the air before the Leafly team became capable of running a lot of split tests with SplitSignal. Now, the problem was not with lost opportunities but with prioritization.

As split test ideas come from everything from intuition to competitive analysis, Leafly has to find a way to create a hierarchy of variables to test - something they could only dream of before they were able to run split testing at scale.

Results: 418k in Google clicks

By the end of November, Leafly had run over 20 full-scale tests that led to measurable - be it positive or negative - results.

Overall, they saw that the tool delivered on its promise:

10% of the tests were positive.
While the 90% were negative, this was a good opportunity to test ideas and see that they are not viable. And thanks to SplitSignal, only a small number of pages were temporarily affected by these results.
One change (moving from marijuana/cannabis to weed) went site-wide. Since implementing sitewide in late September, Leafly saw a +0.5% lift in click-through rate to strain pages accounting for a lift of ~418k in Google clicks compared to the previous time period before the change was made.

Key Takeaways

SEO split tests are not only possible, they offer no-code access to turning a creative hunch into a measurable business decision.

Small website title changes can make a big impact on clicks. In order to isolate the impact of seemingly minuscule copy changes and measure the true impact of SEO split testing, use a tool like SplitSignal.
Sitewide learnings can be made with continuous, micro split testing. For Leafly, this meant gaining enough insights over the course of 20 split test to confidently change references to "marijuana" or "cannabis" to "weed"

Want the same success? Free trial here