All of your clickstream logs paint a (personalization) story

Lucky Gunasekara

February 28th, 2023

All of your clickstream logs paint a (personalization) story

For any platform-scale business, whether you’re a retailer, direct to consumer (DTC), marketplace, or a content media service, personalization is essential for a seamless user experience (UX) that drives conversions. But what actually drives conversions?

In many cases, crude personalization can hurt conversions because although it surfaces attractive products or content that drives clicks, it’s not enough for users to purchase or consume. A tell-tale sign of unfocused personalization is elevated page views met with oddly low key conversion metrics, such as Add to Cart and Checkout events.

This is because the machine learning (ML) driving the Learning to Rank on Search or Product Recommendations only looked at one part of the clickstream in its modeling—in this example, pageviews. And it did its job perfectly for a single objective optimization.

What’s much harder is multi-objective optimization—one that can look at many different events and metrics in tandem and drive lift in a coordinated and sophisticated manner. This is precisely the approach we’re taking here at Miso.ai.

Rather than sharing just one part of your clickstream, we ask partners to share as much of it as possible. Why? Because every search, browse and click from a user tells a story about their journey on your site. And if you really want to have personalization that drives seamless engagement, conversion lift, and high-quality revenue optimization, you’ll need a platform that can analyze this entire journey for every user at once, in real-time.

While this approach is effective, Miso.ai is not a one-size-fits-all solution for personalization or first-party clickstream data. All products and businesses are different, and a cookie cutter approach to personalization is boring. We're a small data model of ML for personalization, so even if you don't have the full clickstream picture that you wish you did, it's okay. As long as you have the bare minimum key clickstream events on hand, we can get to work.

What's Minimum Viable Clickstream for Personalization with Miso.ai?

Even if you're just starting out with your first-party clickstream data efforts, you can still get rolling with Miso.ai, so long with two key clickstream events:

Pageviews
Key Conversion of Interest (ex. Add to Cart, Checkouts, Bookings, 25%+ Watches, Follows, etc.)

More is always more, but if you can tell Miso.ai what your users have been looking at and when they took the key action you want to optimize and grow, we can help!.

From there you can expand naturally, and our API clickstream docs are a good guide for future events you can share with your Miso.ai engines. We realize that tracking every click can feel like boiling the ocean. The helpful heuristic we've used when building our own demos and personalized services is to make sure the clickstream we're using represents as full of a picture as possible of the click path funnel users take towards their ultimate destination and conversion goal.

Say you're a travel rental marketplace like Airbnb. Ultimately, the conversion you want to optimize for is bookings. But the path to this conversion isn't just a page view followed by a booking. It’s more likely users will run a location-based search with a date range (Clickstream: Search event logs).

Then they may filter in those results (Clickstream: Faceted Search Filtering). Then they may click on a given listing (Clickstream: Listing clickstream and page view). They may expand and browse reviews (Clickstream: Clicking to browse Reviews). They may expand the list of amenities and features of the listing (Clickstream: Amenities list expansion click). Or they may message the host (Clickstream: Host message). And then finally, after all of that, they will book the rental property (Clickstream: Booking).

It’s a lot to keep track of, but even if you’re logging just some of it, you're in good shape. Miso.ai will look at the key conversion you want to grow (ex. Bookings) alongside all of the clickstream data you provide and will construct a data-informed "funnel" of the typical events and behaviors it sees that drive the key conversion (ex. again, Bookings). As part of growing the key conversion, it will tackle the nuanced multi-objective optimization of all the events that typically drive the key conversion.

You Get a Gold Star if You Have Implicit Clickstream Events

A lot of what we’ve just unpacked falls into the category of explicit clicks, in which a user or visitor is directly clicking on something. While implicit behavior is fantastic, it’s not always something you may have on hand.So, what are examples of implicit signals?

Dwell time or page duration is a great implicit signal. The longer a user spends time actively browsing or reading a page, the better. It's a great sign that they're taking an interest, educating themselves, and are likely actively considering the product. This is especially relevant when considering content media. Taking the time to read an article, watch a video, or listen to a podcast all indicate strongly that they are finding the content interesting.

But, of course, dwell time on its own can be misleading. We've all opened a page tab and then just left the page there, telling ourselves we'll get back to it later. In this regard, active scrolling time is an even better implicit signal. Or how far a user has scrolled a page, or how much of a piece of linear content —articles, music, or videos—they've consumed.

All of these give richer context for a user journey in session and are great to provide Miso.ai if you have them. Our API docs cover the schema for these events in more detail. But remember, even if you don't have these implicit signals on-hand, we can still get to work and drive lift with what you have.

Conversion Lift is Great, but What about My Business Metrics?

One paradox of personalization and objective optimization is that even if we grow your checkout conversions, your revenue could still go down. It's weird, we know, but let's look at a simple example.

Say you're a grocery and convenience delivery app and you have a given user who is hangry. We could show them a bunch of cheap snacks like 99 cent Twinkies, Cheetos, and gummy bears to quickly buy and stuff their face with. But what's the cost of this, really? The user's core intent and need is filled but in a pretty low-quality way.

They might not have been price sensitive when it came to a good frozen pizza or pre-made, throw-in-the-oven food tray that you could have sold for $12. But because you created a shopping experience with cheaper junk food upfront, that path is closed and you cannibalized your AOV. Even worse, you could have sold them the pre-made meal tray AND a modest amount of junk food and hit higher AOV and a good cart size at the same time.

This is why we designed Miso.ai and our Dojo admin application—to allow you to dial in on the key conversion you want to optimize, and the metrics you want to dial up. This enables you to not only grow your Checkout events, but also your margins, which is especially helpful if you're in a tight margin business, like groceries. Or you can increase episode watches on your streaming platform, but with the duration of the viewing session optimized, too. Conversion growth alone is rarely enough, and we realized early on that you also want to see quality metrics growth.

What Clickstream Sources Can I Use?

A clickstream-driven approach to real-time personalization isn't new to the likes of Youtube, Amazon, Netflix, or Spotify, but for the 99% of the web that historically couldn’t afford ML technology and engineering teams to achieve this, it's been out of reach for too long. When we designed Miso.ai, we realized we should leverage the clickstream brands are already aggregating, which pretty much fall into a few key buckets:

Product Analytics: Like any growth product team, chances are you’re already processing first-party clickstream data as part of your product analytics. As such, we’ve built direct and easy support for getting this first-party clickstream data into Miso.ai, especially if you’re using Google Analytics, Mixpanel or Amplitude.
CDPs: Customer Data Platforms are another great source of clickstream events, given their very nature as a centralized datastore, including user-centric product event logs. Our webhook integration with Segment is by far one of our best integrations for real-time clickstream events, which you can read in more detail about here.
Homegrown Data Warehouses: With the rise of Snowplow, it’s becoming more common for teams to build their own first-party clickstream data warehouses themselves. They can do this by running Snowplow and sending their event streams into Snowflake, BigQuery, Redshift, Postgres, S3, or really any database or data warehouse they prefer. Our first integration model for this has been to build on Meltano as a great open source connector tool, which you can dig into more here.

But even with Meltano, we realized we could only get historical first-party clickstream data into your Miso.ai environments regularly, not in real-time, at ultra-low latency. That’s enough to get your first engines trained to a great baseline, but it’s not the magic of real-time, click-by-click personalization that’s fully cold-start ready.

To account for this, we created a full client side JavaScript SDK that you can install easily (more on that here) to stream key first-party clickstream data from your site to Miso.ai in real-time. Every platform and product team is different, and we recognize that there’s always fewer engineering resources than priorities for any product team. Our goal in building client and server side SDKs is to make life as easy as possible, so you can get the historical data you do have into your Miso.ai environments easily. Then, you can turbocharge your engines and performance from there with our SDKs.

Figuring Out What Works for You

Every product is different. Every product team is different. This is why we take a hands-on approach to onboarding every product team we work with and debugging their data integration into Miso.ai. We also collaborate on the first few successful A/B test rollouts together. Self-serve software sounds great until you face a scenario not in the manual. When things are complicated, you want a human and an experienced one at that; every Miso.ai partner gets 24/7 access to our team of Google, Alibaba, and Yahoo! veterans via our shared Slack channels.