Integrating Your Data

Miso requires two datasets to train your engines and keep them up-to-date:

• Your catalog of products (or content)
• Your site's log of historical and real-time interactions

It is also possible to provide data about your site's users and visitors, though this is not required. Since Miso is a privacy-first platform, many customers choose to provide hashed user data (and we ask that you please do not send any PII to Miso).

You have two options for providing data:

1. To quickly get started exploring what Miso can do on top of your data, you can use Dojo to upload your datasets as flat files.
2. As you get ready to build, test, and launch a Miso recipe, you should integrate your data fully using our Data APIs.

Uploading Data in Dojo

If you can access your datasets as flat files, you can upload them in the Dojo UI in just a few steps. See the guide to Data Sets in Dojo.

Integrating With Miso's Data APIs

Miso's Data APIs let you automatically upload and manage your data in Miso. These APIs all support high-throughput data ingestion through bulk insert and satisfy GDPR and CCPA compliance by letting users delete their data from Miso.

Uploading and managing your product catalog data

Miso's Product APIs let you upload, read, and delete Product records that represent your site's catalog of products (and/or content).

Much of Miso's personalization capability relies on understanding your catalog in-depth and drawing correlations between your catalog and your users' consumption or purchasing behaviors. In other words, Miso discovers that, with high correlation, users who are interested in certain product attributes will also be interested in products with similar or related attributes. You can think of the product_id as a join key that connects your product data with your interaction data.

API throughput

We recommend batching up your Product Upload API calls and sending around 100 records at a time to avoid timeout and memory risks.

Product data model design

To fully optimize your personalization, it is important to provide Miso with Product records that are complete and accurate. We define a set of common attributes that capture the basics of most eCommerce and content media products, such as title, description, categories, tags, material, brand, authors, etc. You can also use custom_attributes to specify any additional information from your catalog. Miso can handle hundreds of custom attributes, so don't feel reluctant to provide as much catalog metadata as you have.

How to use custom attributes

If your products' characteristics cannot be fully captured by the fields that Miso defines, you can also specify custom_attributes. The more complete the product information is, the smarter Miso becomes. For example, if your product summaries support multiple languages, you can have something like: custom_attributes.alternative_langs: ["en", "zh"].

You might also want to consider including attributes that are required for presenting summaries in the front end (e.g. cover image, raw rating scores etc.) so that you don't need additional requests to fetch those fields.

Sometimes the design of custom attributes can require some thought. Your Miso solutions architects are available to help you think through the engineering implications. For example, if you have products that are only available in certain regions, how should you represent that? There are a few options:

1. Have a list like: {"regional_availability": ["region_1", "region_3", ...]} that contains all the regions that currently have the product
2. Have multiple attributes like: {"region_1_availability": true, "region_2_availability": false, ....} , where each attribute represents the availability of a particular region.

For a "partial" update, option 2 might be easier, because you will only need to gather the availability for the specific regions you want to update.

Uploading and managing your interaction data

Miso's Interaction APIs let you upload and delete Interaction records. Interaction records tell Miso about user interactions with products and content on your site or application. Based on historical and real-time interactions, Miso understands how users move through your conversion funnels: which products or content assets attract the attention of each individual user, and which products or content ultimately will be purchased or consumed by each of them. Interaction data requires product_ids and user_ids to be present so that Miso can make these connections between users and products on your site.

Integrating your Interaction records with Miso can be thought of as two distinct phases of work:

• Historical Interaction Records: To start your Miso integration, you'll bulk upload at least three weeks to three months of interactions data. Your personalization engines will initially train on this data to inform their understanding of the behavior of users and visitors on your site.
• Real-Time Interaction Records:  After the historical backfill, you'll start streaming interactions to Miso in real-time. This ensures that your models can adapt instantly to what users are currently searching for, clicking on, adding to their cart, etc. With real-time data, Miso can personalize to new users and visitors at cold-start. Streaming this data makes sure your models stay constantly up-to-date, avoiding drift and driving steady conversion growth.

Uploading historical interaction data

Historical interactions can be sent via Miso's Interaction APIs, which are not rate-limited and maintain a complete interaction history (as opposed to sampling as in Google Analytics). We recommend batching up your calls to around 10,000 records at a time to avoid memory issues or timeout risks. Your Miso solutions engineer can help you if you have any questions about the bulk upload.

Streaming real-time interaction data from your frontend JS

To get the most out of your Miso performance, you need to send Interaction records to Miso as soon as the interactions take place. To do so, teams often call the Interaction Upload API from their frontend JavaScript code. Make sure that when doing this, you use your publishable API key (and never your secret API key).

Example JavaScript code

const INTERACTION_ENDPOINT = 'https://api.askmiso.com/v1/interactions'
const PUBLISHABLE_KEY = '<my publishable API key>'
const API_URL = `${INTERACTION_ENDPOINT}?api_key=${PUBLISHABLE_KEY}`

// when visitor add one item to shopping cart
const requestBody = {
  data: [{
    type: 'add_to_cart',
    quantities: [1],
    product_ids: ['<the product ID>'],
    user_id: '<ID of user that add this item>'
  }]
}

window.fetch(API_URL, {
    body: JSON.stringify(requestBody),
    cache: 'no-cache',
    headers: {
      'content-type': 'application/json'
    },
    method: 'POST',
    mode: 'cors'
  })
  .then((response) => {
    if (response.ok || response.status === 422) {
      return response.json()
    }
    throw new Error('Upload failed')
  })
  .then((response) => {
    if (response.errors) {
      console.log('Upload failed because:', response)
    } else {
      console.log('Upload succeeded!')
    }
  })

Important interactions to send


If interaction data is sparse or hard to get, teams often wonder what the most important interactions are for Miso. Here is the advice we give:

Send the interactions you want to optimize for

When you set up your Miso engines, you choose interactions that Miso should optimize for. These usually represent the key conversions on your site, like checkout interactions. It's essential to send this interaction data, otherwise Miso can't optimize for it and it won't show up as an option in your engine setup. See our guide to Understanding Your Engines for more information.

Send the interactions that happen as users step through your conversion funnels

We recommend sending Miso the interactions that represent steps through your user conversion funnels, as users follow your personalized UX from entry-point to endpoint. This user journey may depend on the precise recipe that you are implementing, but usually these interactions include:

1. search
2. category_page_view
3. product_detail_page_view
4. add_to_cart
5. checkout

Providing feedback to Miso's models

When users interact with results generated by Miso's engines, it's important to provide the miso_id (which you will receive from any Engine API response) in the corresponding Interaction data. This feeds back into Miso to help it know how predictions performed, so your engines can improve over time. Of course, when you first start sending historical and real-time interactions to Miso, you won't have a miso_id yet, and you can leave it blank.

Uploading and managing your user data (not required)

Miso’s User APIs let you upload, read, and delete User records. While optional, Miso's engines can use this data to inform its understanding, especially when the user has had zero or few interactions on your site.

Since Miso considers user data as any other feature for its models, and does not inspect or analyze this data in any way, we recommend that you provide hashed user data as a privacy best practice. For example, you might consider sending hashed user cohorts or any interests that your users have declared. It can also be useful to know the user’s hashed zip code location so Miso can extrapolate what products or product categories are trending in a specific area.

We recommend batching up your User Upload API calls and sending around 100 records at a time to avoid timeout and memory risks.

Viewing Data

In Dojo, you can see the data that you've uploaded to Miso in a visual dashboard. See the Dojo Data Sets guide for more details.

You can also read your Product and User data via API.


Deleting Data

Dojo provides a way to delete data records and even wipe all the data in your environment. See the Dojo Data Sets guide for more details.

Programmatically, data can be deleted via API as well. Here is an example of how to delete data in bulk:

Get all product ids

GET https://api.askmiso.com/v1/products/_ids
Body: NA
Response(200): {"data": {"ids": \["PRODUCT_1", "PRODUCT_2"]}}

Bulk delete products by ids

POST https://api.askmiso.com/v1/products/_delete
Body: {"data": {"product_ids": \["PRODUCT_1", "PRODUCT_2"]}}
Response(200): {"message": "success"}

Troubleshooting

422 Errors (Schema Validation)

When a data upload fails, it is usually due to a schema (formatting) error.

Any schema error will cause the whole request to fail: the API will return status_code=422, and none of the records will be inserted. You should check the data field in the response to see where the errors are located. For example, the response below means there are schema errors in the interaction record at index 0:

{
    "errors": true, // there are errors. please check!
    "message": "None of the records were inserted because at least one of them contained schema errors. Please see the `data` field for details.",
    "data": \[
        "data.0.product_ids is invalid. The attribute was expected to be of type ''array', 'null'' but type 'string' was given.",
        "data.0.timestamp is invalid. The attribute should match the 'date-time' format."
    ]
}

We log schema errors on our end as well. If you ever need help troubleshooting, contact your Miso solutions engineer — we're confident we can solve the problem together.

Incorrect data

It's a good idea to review your data in Dojo to make sure that it looks correct. If data has been uploaded in error, you can delete it, or overwrite the incorrect fields by uploading a new version. Miso automatically deduplicates records during upload and will keep the newest version.