Increase Your CRO by 65%+ Through These A/B Testing Methods | Ayat Shukairy, AWasia 2018

Increase Your CRO by 65%+ Through These A/B Testing Methods | Ayat Shukairy, AWasia 2018


Hello Bangkok! I’m so excited to be here.
This is actually the furthest east I’ve ever travelled. So I’m really excited to
actually be here with you today, and I hope you’re gonna really enjoy my talk.
I’m gonna talk about increasing your conversion rates. We’ve developed a
methodology that I’ll take you through. Hopefully you’ll take some actionable
insights, so that you can apply it to your own programs. A little bit about
me. I am the queen of CRO. I’ve been doing conversion rate optimisation now for
12 years. I’ve done so much A/B testing, ran thousands of A/B tests. So
I’ve certainly earned the title of queen of CRO, and I’m also the mother of four
children, the youngest of which is kind of like a mini Elmyra, and if you forgot
who Elmyra is, you could see kind of the similarities between my child hugging
the cat and what Elmyra does to animals. So let’s get right into it. I have a lot
of slides to get through, conversion rate optimisation is not easy and you know, a
lot of times when people start working on it, they think, “Hey this is super easy.
I can totally do it.” So you get yourself ready or prepped up,
you’re going to go into the fighting ring and you’re so excited.
You could totally do this. So you put together your first test. You’re ready to
launch it. This is going to be super easy, and then wham! You’re just shocked.
Where are my increases in conversion rates? Where are the amazing results that I always hear about? And the reality is right now,
we’re always looking for those best practices. What’s going to help me increase
my conversion rates immediately? We want it now,
but the reality is we can’t have everything now. It takes a lot of time and effort,
in order to develop a solid conversion rate optimisation program.
And it’s not your fault. it’s not anybody’s fault. The reality is
you go online, you search for conversion rate optimisation, and you read articles
like this. Where from one, single A/B test, they were able to increase their
conversion rates 529%. When you hear that kind of thing,
you’re like, “Wow, I can totally do this.” Or for example, “An orange button changed
conversion rates 86%.” This is so easy. Anybody can do that. Anybody can see those amazing,
amazing conversion rates but is that really the truth? I mean, when we really
look at it, Google reports that only 10% of A/B tests that are launched, actually
result in an increase in conversions. Optimizely, that’s the kind of a large
testing engine, they also report that only 25% of the tests that
are launched on their testing engine, result in an increase in conversion rates. VWO, another testing engine says that
only 12% of tests that are run on that specific platform, result in an increase in conversion rates.
So you’re thinking, “Wait. Why is there this data confusion?
Why do we see these amazing case studies about 529% increase and 86% increase,
but then the reality is very different than that?” So there’s a couple of reasons. Number
one is, it could be that these are just outliers. Once in a while, you do launch a
test and you see amazing increases, but most of the time when you’re launching
an A/B test, you need to run it for a specific length of time. You need to also
make sure you reach a statistical significance, otherwise, you’re not going to be able to see those results. In most cases, those are the
reasons why A/B tests don’t perform. So we developed a methodology called
the “Ship Method”. it looks very, very simple,
but the devil is in the details. Each one of these is a process
that you have to really work through in order to see those
25%, 35% increases in conversion rates and I’m going to talk to you guys
a little bit about some of the lessons that you can learn from this process. In this day and age, a long time ago, a lot of companies were
always worried about the big fish. The big fish. They were gonna gobble up
all the small companies, and this isn’t entirely not true today.
You have the conglomerates, like Amazon that
are definitely overtaking a lot of the competition, but if you’re agile enough,
and that’s what this kind of testing program, CRO program teaches you, then
you’re going to be able to outrun the competition in general. So the reason why I emphasise experimentation. I don’t condone just
saying that conversion rate optimisation equals testing.
Definitely not. There’s a lot that goes
into conversion rate optimisation and you shouldn’t test everything. There’s
that whole saying, “A/B test it all!” And if you have somebody in
your office that’s telling you, “We should test this, and we should test that, and we
should just throw things at the wall, and hopefully something will stick.”
Then that person is definitely somebody that shouldn’t be in your organisation
because that doesn’t work, but at the same time, obviously, when you have an
Amazon.com. They have a large enough set of traffic that comes into their website.
They have so many different pages so they can run a lot of experiments, and
they have a team dedicated to experimentation. They’re not just
throwing things at the wall. There’s always some sort of research, and some
reason why they’re launching different tests, but if you do run many
A/B tests, you’re able to see a really big difference in the performance
on the site. The one thing that everything always comes down to is that
customer. Providing the customer experience. Everybody knows the Salt Bae. He became very, very famous.
An internet sensation. Everybody was watching his videos, but what was it
about what he offered? Actually, I spent almost five years in Istanbul,
Turkey and a lot of the restaurants there provide a similar thing, where they
have a show that they put on. It’s all about providing that experience
because that’s what people are going to be able to walk away from
and remember, “I love that restaurant” because nowadays, we’re in the era of
commoditisation. You go into any store, you look at any type of product,
everything looks the same. You remove the label, it all looks the same.
So if you’re not providing your visitors with a really unique experience
that they’re going to remember, then you’re certainly going to just be
like anything else, all like the rest of the competition. You want to
make sure that’s what you’re giving your visitors. You’re giving them that
experience, and the way that you’re going to be able to innovate and provide that
experience is through conversion rate optimisation and experimentation. So it
starts with testing. It starts with fixing. Of course, all the UI/UX issues
that you have on your website, and then it takes a lot of collaboration in order
to really reach what that big win is. They talk about this big win. Whether
it’s Amazon, whether it’s Facebook. They’re always testing and they’re
trying to get to that one test that’s going to be like their home run,
where they’re gonna really be able to see those amazing results. So lesson one that you should really think about when you’re trying to optimise the
experience of your visitor, is it’s not about template tweaking. What does this
mean? Again, we talked about this example in the beginning. When it comes to
changing in the colour of the CTA and that’s gonna give you an 86% increase.
I can just assure you that that’s not the case. That will not happen.
You will not see 86% and 96% increase because of a change of
colour of a CTA. It’s not significant enough for the visitor, for that increase
to happen on your website. We do a lot of training, when it comes to
conversion rate optimisation. So we go into these large organisations that
already have the team, but they don’t know what to do and how to start. So I
was giving a training and somebody came up to me after the training and
they’re like, “Well I thought you were gonna give me, I have to do XYZ
immediately after I leave. This is a checklist of
everything that I could do to really improve the experience of the visitor.”
and so I told her, “That’s basically, exactly the opposite of what
this training is all about. The training and conversion rate optimisation
is about your specific visitor. I can’t tell you you need to do
XYZ. You need to discover exactly what XYZ is for your particular visitor.” And what we’ve seen is that best practices, when they’re implemented, they often fail.
You’re always looking for repeatable, sustainable growth.
That’s what you want to achieve with any type of conversion rate
optimisation program that you’re running. So any typical conversion rate
optimisation program has the qualitative research, the quantitative
research. You prioritise the different areas that you found. You put
some sort of hypothesis together, and then you test, and so for any
agency, then there’s a lot of prayer. Like, “Oh god, I hope my variation
wins.” but if you’re looking – remember what I said was, it’s
all about the devil is in the details. You want to make sure that each
and every single one of these particular areas is something that you’re focusing
on, you’re making sure that you’re paying attention to, and each one of them is a
process by themselves. And when you implement these processes on your
website, you’re going to be able to generate, again, it’s that incremental.
Once in a while it’ll have like an amazing test that performs, but
you’re looking at generating repeatable, sustainable increases in your
conversion rate. Lesson two is you want to know the customer
as much as possible. So at the beginning of the brink of the Industrial Revolution in
the US. In New York city, what did they do? They built a lot of skyscrapers.
When they built these skyscrapers, of course, you don’t want people to
run up to whatever floor it is just by going on the stairs, you want to
put in some elevators so that they can get up to where they need to get to, but
people were getting fed up with the elevators. It was very exhausting.
You had to sit there, crunched in with like a zillion people because there
was one elevator for a single skyscraper, and it just took a lot of time to get to
where you need it to go. So they asked people, “Hey, what should we do to
help this experience of the elevators or the skyscrapers?” So people responded
with, of course, “You need to go ahead and build more elevators in the skyscrapers.”
So they wen’t back to the drawing board, but they said this is
gonna cost us hundreds of millions of dollars if we’re going to install more
elevators in these skyscrapers. So what they did was they provided a solution,
they satisfied people to a certain extent, and at the same time
they weren’t spending millions of dollars, and they
just put a mirror in. And that’s why you go to a lot of elevators today, and
there’s a mirror. The reason is because it gives the sense that when
you’re going into that elevator, it’s a lot bigger than it actually is, and at
the same time, it gives a chance for people to be able to stare somewhere and
not necessarily have to look at the ground because there’s an elevator and a
mirror in place. So the point of what I’m trying to say is that in general, is a
lot of people, when you ask them questions about what they want, they’re
gonna give you the top of mind answer. They’re not gonna really dig
deep and tell you exactly what they really want, they’re going to give you
whatever comes to mind. So one of the biggest marketing fails of all time
was – I don’t know if you guys know the company, I’m sure you guys do,
Heinz ketchup. They launched a campaign and I think it was the early or late
90s, early 2000s where they had coloured ketchup. Green ketchup, purple ketchup,
orange ketchup, and they had a focus group of moms and they asked them, “What
do you guys think of this?” and the moms were raving, “This is amazing. We’re gonna
buy it. We love it! Our kids are gonna love it!” So then they actually pushed
this product to market and nobody bought the product. Who wants snot-looking
ketchup on their hot dogs? Not many people. So it was a major fail for them,
but the point of it is that sometimes when you ask those surface questions,
people are going to give you the answer that they think you want to hear, not
what actually is bothering them, and what the actual experience is like. So you
have to kind of pry and really figure out that subconscious. What’s
really happening in their mind? What is really exhausting them? So we always have
this perception of how we think visitors behave. It’s like
everybody talks about the funnel, visitors are gonna go through
the awareness phase, and then they’re going to be a little bit more informed
about the product, and they’re gonna actually end up buying. Great. This is
what we all see, all the time in any type of marketing talk, people
discuss the funnel. But what’s the reality? I think we all know this. That
visitors don’t behave that way. Visitors actually are sporadic and especially in
this day and age. They go from one platform to the next. They’re coming in
from different sources of traffic. It’s a very confusing type of experience,
and it really doesn’t give us a lot of clarity because in the end, we need to
understand who is that visitor that’s coming to my website? Who are they? And I
need to understand exactly who that target market is so I can
understand their needs, and be able to address all of their concerns through my
website and provide them with the experience that they’re looking for.
So when it comes to packaging quality, when it comes to
customer service, when it comes to the content on your website, when it comes to
anything, it always comes down to that customer. So if I don’t understand who
they are, then any type of initiative that I’m running, anything that I’m doing,
I’m not going to be able to succeed. Because again, ultimately, what we’re
trying to provide is not the regular experience. We’re trying to give them
something that they’re always going to remember, a memorable experience so
they’ll come back more and more. Remember, we’re not going for just a single
conversion. We’re going for that lifetime value, where they’re gonna be able to
come back to our organisation and purchase more and more. If I want to conduct qualitative research, this is gonna help me get into the head
of my visitor as much as possible, really understand what they’re thinking, what’s
bothering them, how can I really provide them with the experience that they’re looking for. Number one is we always say in any type
of activity that you’re doing, when it comes to conversion rate optimisation,
you need to plan the research. Don’t go into it blindly. Don’t just throw things
at the wall. Make sure you’re planning and understanding exactly what your goal is,
then you, of course, conduct the research. This takes time and then it’s,
“What am I learning from all of this research that I’ve conducted? Am I just
running different surveys and polls, calling up my customers, and then I
have nothing to take away from it?” Then I’m really just wasting a lot of time.
And there’s different methodologies that you can use to get into the
head of your visitor. I can conduct different interviews. I
can run different focus groups, usability tests, and surveys or polls. So of
course there’s different steps that you can take to really understand
how to drill deep into that visitor and understand more about them. The number
one is you have to identify if you even need the research. Why am I
conducting this in the first place? And then the second thing is, and this is
something that we find a lot of companies are missing, is identify
what’s the goal? What am I supposed to achieve at the end of this? Once I’ve
identified the goal, then I can identify exactly who I need to get this
information from. It might not be all the visitors. So when you run a survey
or poll, and you’re just running it across everybody, all the
different visitors that are coming to your website, you’re missing out because
if you target a specific group, and you know exactly why you want these particular
participants. You’re gonna be able to learn a lot more. Then
you want to identify your personal bias. We had a client. They’re a SaaS
company and they sell male subscription products. So, of course, I come in thinking,
“Oh, I know everything that you need to know. I actually come from a family of
six. I have four brothers. I know everything about guys.
I have my husband. So there’s I know everything you have to know.” So as we
were talking with the group, a lot of the team members that
were males in my group were like, “No, I really don’t think that men would think
this or would would do this.” So clearly, I had a bias that I wasn’t aware of. So,
it’s important that you identify what is the bias that you have when you’re
talking to these particular participants. And then don’t limit your
research methods. Make sure you have research methods that again,
like I’m doing customer interviews, and I’m running polls, and I’m running
surveys. Each method provides me with insights. Each method is going to solve a
specific problem for me, and then it comes to just
formatting. What am I going to ask the visitors exactly? I need to make sure
that I’m formatting the questions correctly, and then what’s very
important is after I run whatever it is that I’m running, I want to make sure I
analyse and propagate that data because that’s where I’m going to be really able
to benefit the most. So this is actually a poll that we ran for an ecommerce site that sold these different accessories for cellphones,
and so we asked people on the cart page, what is preventing you
from moving forward with your purchase. So the visitors gave us a number of
different answers. They said we can’t find that information that we need, we
don’t know if this product fits our phone, although every single product had
within the title, what phone it actually was for. They wanted a different colour.
They couldn’t find the shipping cost, and they were just comparing prices. So we
took that data, we plotted it, and again for us it’s all about taking away the
insights, and what are we gonna do as a result of learning this information
about our client? Some things require us to run an A/B test. Some things require
us just to fix it. It just doesn’t make sense that this information isn’t
readily available to the visitor. Some things require us to investigate further,
okay, this has given me more reason to look into analytics and seeing what’s
happening with the visitor, and some require some sort of action. So
we changed the design of the page, and we provided the benefits up front. We
created an area where they could see exactly what phones this was compatible
with, and again, we gave them more options, so they can see
exactly what they were looking for. Then we conducted for a different client, this
was more of a SaaS subscription for running your own radio show
online, almost like a podcast, and what we did was we interviewed some of their
potential customers, and we wanted to understand from them what was preventing
them from signing up. So again, they gave us a number of different pieces of
information. What stood out the most was that visitors felt that they were
feeling rushed. They were feeling rushed. Everything was on one page
and they felt very rushed because although it wasn’t a very
expensive subscription, it was going to take them a lot of time and effort in
order to create a radio show. So again, we plotted all the data. We tried to
understand what are we going to test? What are we gonna fix? What are
we going to investigate? And we created a two-page process.
So rather than having a one-page checkout, we created two pages because we
were able to provide them more information, get them more
situated with exactly what type of packaging they’d want.
We also made the “free subscription” stand out a lot more for the visitor, and
we got a lot of amazing responses as a result of this. We also created a
clarified process. Now, I just want to note that when we got to this particular
design, it went through multiple iterations. What I’m showing you is the
final design that we came up with after a lot of different testing that we did. So when we talk about “who are you
targeting?” when we talk to our clients when we do a discovery phase,
we’ll ask them who’s the target market? We want to understand from their
perspective who it is. We’ll do our own investigation, but we want to understand
from them what they think. And you get these broad answers. Like,
all males and all females, anybody that wants to like purchase from us
and we always tell them, people that are coming to
your site might not fit this general criteria. You want to be really,
really specific with who they are because the copy is going to change as a
result of that. The offers are going to change as a result of that.
Everything’s going to change as a result of exactly who the target market is. Anybody familiar with Mad Men? Maybe? So this is Don Draper from Mad Men,
and it was a show that talked about these ad agencies in the 1940s in the US. And so he said in the show, “People tell you who
they are but we ignore it because we want them to be who we want them to be.”
We want them to be a certain way. We’ve designed the site for a certain
type of visitor, but hey, guess what? These might not be our visitors and this
actually stood out to us when we were working with a billion-dollar company. Most of
their their clients went to their stores. so they had a lot of stores and they had
of course this ecommerce presence so they assumed that all their visitors
that were coming to their website, were the people that actually would
visit their stores. So we told them, “You know what, that’s okay.
Let us just do our thing and let us run different experiments,
even if you don’t necessarily agree with them. It’s an experiment. We’re
gonna see whether or not we’re able to achieve increases in conversion rate.”
So the first experiment we launched with them, they were completely against it.
They’re were like, “No, our visitors do not behave this way. They’re not gonna like
this. This is not gonna achieve any goals.” And then, of course, when we increased
their conversion rates by 30%, they were just shocked. They knew that, “Oh okay, so the
visitors that are coming to our website might be a little bit different
than people that are coming to our stores.” And that’s very
important to kind of keep in mind, but what’s also important is okay, I’ve
identified the target market, but I also need to know, what is the experience that
brings people to my website? Remember I’m not working within a silo. I’m not just
looking at a website. I’m looking at the entire journey. I need to understand
where they’re coming from, what are their experiences, and what brought them here
because that’s going to really help me understand what I need to do, to make the
experience even better. Whenever we look at optimising, we try to think about “What person is coming to my website?” They’re coming to answer a specific question
that they have in mind. Maybe they’re looking for a particular
product or service. Maybe they’re just coming to
compare options. So again I’m mapping out what is bringing
people to my site exactly. We have 7 questions that we typically ask, and that
is, what’s the visitor intent coming to the website? What’s the context of their visit?
Where did they come from exactly? Where did their experience start?
Where does the experience finish? Again, I’m trying to map out that
customer journey from start to finish. Understanding exactly what keywords that
brought them to my website, what experience they had as a result of that,
and really try to understand their journey overall. What are the visitors
feeling when entering? Do they respond positively or negatively?
Again, you’re using analytics to give you an idea. I’m using
my qualitative research to give me an idea in order for me to understand the answers to some of these questions. What’s also
important is based on where they’re coming from, how brand aware are they?
Because depending on how brand aware, the visit and the experience might be very
different, and then where is the visitor in the buying funnel? Are they ready to
buy? Are they people that are just browsing? Where are they exactly? And that
gives me a really good vision of where my visitor is, what their
experience really looks like. It really gives me that overall customer
journey, and I’m able then – depending on what I understand from the visitors, to kind of target specific experiences better or provide them with a better overall experience. So this is what it looks like.
There’s this Excel file that will plot all this information to
really understand where the visitor is coming from depending on experience, the
context, all of that. Lesson 3 is you want to think outside of the box.
When we talk about the history of optimisation, think about it,
and this really applies to the US, Europe, a lot of these countries,
we created these websites, we optimised for online transactions, and
then mobile started becoming very popular and we started creating these
responsive websites on mobile. We took whatever we had on desktop, we just made
sure that it was “mobile-friendly”. So this is the eBay app over the years.
So this is how it first started when they first launched their mobile app.
As you can see they have everything listed, they have
the search area clearly listed there, and then it changed.
They went “What if we add these different elements? How is it going
to look?” And then changed it really drastically at one point, where they
completely removed the old design and then went with a design that’s
completely different. Again, changes. Small tweaks here and there, just trying to
improve the experience overall, and it will go on and on. How they
completely changed the app for many years. So this is what it looks
like and when you look at all the designs, and you guys will kind of see these – I don’t know if you’re gonna actually
see that particular part, but you can see the transition. How they moved
from one point to the next, and they’re small changes. There’s nothing
really major, but it’s the idea that we took whatever we had on desktop,
and we just made it responsive on mobile, and then when you look at
retail prices, ecommerce prices or sales, you see that it’s a growing
industry in the US. Nowadays, everywhere we go,
we’re always hearing about mobile first, mobile first, because what’s
happening is this realisation that actually people are using their mobile
devices a lot more than they are desktop. Most of our clients, at this point in
time 70% of their traffic is coming from mobile, and not coming from
desktop. So when we went from desktop to mobile, we really
lost that overall experience but when you look at China,
and actually just recently, they had a sale where within, I think it was a
really small amount of time, like 87 seconds or something like that. They were
able to reach $1 billion in sales because when they
created the mobile experience for their users, they thought of it strictly from
the mobile experience. They didn’t really think about desktop because they wanted
to provide that mobile experience overall, and that’s why they’re really
killing it when it comes to ecommerce sales. Where do I start? A lot of times
people hear, number one is you’re working within an organisation,
and your boss comes in saying, “I really hate this particular design. I want you
guys to change it.” So it’s about putting out fires and that’s why I move
forward with creating a new design or creating a different
experience for my visitors, but it should definitely never be like that. It’s never
about putting out fires. Every single marketing activity that I talked to you
about today should be grounded in some sort of a process. Now, you look
at this guy, if you don’t have a process going, he would lose his hands definitely.
So everything that you do should have some sort of a process. It should be
grounded in that, and a lot of companies will say, “Okay, we know that we need to have
a process but I really don’t understand how.” And so, what I’ve provided
you today, hopefully gave you a little bit of an idea of what type of a process
you have when it comes conversion rate optimisation. I’ve done
my qualitative research. I’ve done my quantitative research. I’ve run usability
tests. I really understand who my visitor is. So what we do, and I’ll provide you
guys with an Excel sheet that you can plot this data in, is we prioritise where
it is we’re going to be working first. What are the first problems that we’ve
identified on your website that we’re gonna actually be able to tackle first.
Because when we’ve been collecting that data, we classify it. There’s items like I
said, that are fixed right away. There are items that I need to add some tagging,
I need to be able to some event tracking on. There are items
that are a research opportunity, which are essentially experiments and there
are items that I need to investigate further, overall. So all of
those items together, that’s where my classification begins. I’m plotting
all of this information to understand what I’m going to be tackling to begin
with, and when we’re talking about prioritisation, there’s a lot of factors
that we consider. We actually consider 18 different factors when it comes
to actually the problem itself. So it’s really important to plot
this information and see, “Where am I going to start when it comes to
conversion rate optimisation?” You want to move away from things that are
your personal opinion and very subjective, to things that are more data-driven.
Everything that you do when it comes to conversion rate
optimisation should be extremely data-driven, and so there’s a number of different
points I’m not going to be able to get to them, but you guys will be able to see
these in the slide, and then once you’ve completed all of this, you’ve
plotted all this information, I have now a prioritisation sheet. I know I’m gonna
start with this particular problem, this is what I’m gonna tackle first, and then
you can come up with the solutions as a result of that. The final
lesson that I have to you is threat awareness. When we talk about CRO, it
really takes us back to almost the scientific model. If you guys
remember in the 8th grade. And that’s really what the process is all
about. It is extremely scientific and it should be nothing less than that. So one
thing that you see a lot, is people run into what we call false positive and
false negative. These are type I and type II errors and this is because
they don’t understand how long a test should run. They don’t understand
also that it needs to reach a specific statistical significance. So this is what
a sample hypothesis looks like. Again, because we have this
particular feedback and this research that we’ve done, we expect this
particular change to happen, and we’re gonna use this metric to measure it. So
this is the hypothesis that we had actually, for one of our clients. Where we
observed that visitors were leaving the cart before they actually made a
purchase, and so we said, by exposing free delivery earlier and repeatedly
on the site and on the cart itself, this will cause a reduction
in overall cart abandonment rate. So this is what it looked like
before, where we didn’t have that information plotted for the visitor, and
then we added “free delivery, cancel at any time” and also that they could adjust
their delivery as they wanted. So you want to measure and set
aggressive goals, in order to see those 25/35/45% increase in conversion rates.
Jeff Bezos says, “Our success at Amazon is a function of how many
experiments we do per year, per month, per week, and per day.” What’s reported to us
is that this is like old data, that they conduct almost 7,000
experiments a year, and I’m sure it’s more than that now. And this is a really
great sheet that gives you a good overview of how to plot your test
data. This is actually from Intuit, and then there’s actually another sheet, if
you guys get these slides afterwards, that has an example of how you would
plot this information, and then finally a special gift from me to you is a
complete conversion optimisation start-up kit. This includes the prioritisation
sheet that I already showed you. It also has an A/B testing guide that you can
download and get a lot of really excellent information from. Thank you.
Stay connected and thank you for being such an amazing crowd. you

3 comments

  1. Thanks for those uploads, appreciated! Can you please remove the ip block to Saudi Arabia for (Website: https://affiliateworldconferences.com ) Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *