Part 2- Optimizing the Algolia record size

In Part 1 of this example, we showed you how to create a Dynamic Content Algolia webhook integration that creates a record in Algolia when a content item is created in Dynamic Content.

Algolia recommends that, for performance reasons, the maximum size of a record should not exceed 5KB. In this part we build on the webhook created in Part 1 and explain how you can refine the custom payload to optimize the size of the record created in Algolia by including only those properties that you want to be searchable.

In Part 3 we explain how you can index large content items by splitting the output into multiple records.

Prerequisites

This example uses the configuration and content type schemas included in Part 1, but the concepts will be the same whatever schemas you are using in your own projects.

Step 1: The Algolia record from Part 1

If you worked through the steps included in Part 1, you will have a webhook integration that defines a custom payload to create a record in an Algolia index containing the entire content item that was created or updated in Dynamic Content.

For a blog content item the Algolia record will look something like that shown below.

{
  "date": "2020-02-26",
  "urlSlug": "power-your-integrations-with-dc-webhooks",
  "image": {
    "image": {
      "defaultHost": "kuifkfmrgfsv1qjsmei8dbbnq.staging.bigcontent.io",
      "endpoint": "ampproduct",
      "name": "five-bulb-lights-1036936",
      "id": "bd686cd6-0934-4340-b311-3085a72508c2",
      "_meta": {
        "schema": "http://bigcontent.io/cms/schema/v1/core#/definitions/image-link"
      }
    },
    "altText": "Blog hero image",
    "_meta": {
      "schema": "http://example.com/blog-image.json",
      "deliveryId": "0be6d50b-65bf-4e42-8a0e-a1eab48955dd",
      "name": "doc-example-blog-hero-image"
    }
  },
  "_meta": {
    "schema": "http://example.com/blogpost.json",
    "deliveryId": "a2b1d192-04d5-4219-81cf-7cf2e6e1b3cf",
    "deliveryKey": "dynamic-content/webhooks",
    "name": "doc-example-blog-post"
  },
  "description": "We take you through our great new webhook features",
  "readTime": 10,
  "title": "Powering your integrations with Dynamic Content webhooks",
  "content": [
    {
      "_meta": {
        "schema": "http://example.com/blog-text.json",
        "deliveryId": "d7939a9c-1609-4505-8f3a-af3fada878d2",
        "deliveryKey": "blogpost-text-1",
        "name": "doc-example-blog-post-text-1"
      },
      "text": "This blog post will walk you through the great new webhook customization features."
    },
    {
      "_meta": {
        "schema": "http://example.com/blog-text.json",
        "deliveryId": "63876325-6842-4046-9de0-cebd19762dec",
        "name": "doc-example-blog-post-text-2"
      },
      "text": "This is part 2 of the text explaining the new features."
    }
  ],
  "authors": [
    {
      "name": "Amplience",
      "_meta": {
        "schema": "http://example.com/blog-author.json",
        "deliveryId": "261291db-6e1e-4f84-b4b8-cd19affa2e78",
        "name": "doc-example-blog-author"
      },
      "avatar": {
        "image": {
          "defaultHost": "kuifkfmrgfsv1qjsmei8dbbnq.staging.bigcontent.io",
          "endpoint": "ampproduct",
          "name": "woman-in-hat",
          "id": "d1381a88-24ea-4c1b-87b6-950e01339a27",
          "_meta": {
            "schema": "http://bigcontent.io/cms/schema/v1/core#/definitions/image-link"
          }
        },
        "altText": "Author image",
        "_meta": {
          "schema": "http://example.com/blog-image.json",
          "deliveryId": "0d9840db-cdf8-4f9d-a72b-f3f10e23eada",
          "name": "doc-example-blog-post-author-image"
        }
      }
    }
  ],
  "objectID": "a2b1d192-04d5-4219-81cf-7cf2e6e1b3cf"
}

By looking at the record structure in Algolia you can begin to see which fields you want to index. You can then include these fields in your custom payload.

Step 2: Add the updated custom payload

The following custom payload can be used to create a record for each blog article, including the information we want to be indexable: title, description, author names, tags, date, readTime and the content itself. Your payload should include all the fields that are used in your search configuration: searchAttributes, attributesForFaceting and customRanking, together with any fields that you need to display your search frontend. For the blog post content, you might choose to include just an excerpt to prevent your record from becoming too large.

{
  {{#withDeliveryContentItem contentItemId=payload.id account="myAccountId" stagingEnvironment="myVseUrl"}}
  "title": "{{{title}}}",
  "description": "{{{description}}}",
  "deliveryKey": "{{{_meta.deliveryKey}}}",
  "schema": "{{{_meta.schema}}}",
  "authors": [
  {{#each authors}}{{#if @index}},{{/if}}
    {
      "name":"{{{name}}}"
    }
  {{/each}}
  ],
  {{#if tags}}
  "tags": {{{JSONstringify tags}}},
  {{/if}}
  "date": "{{{this.date}}}",
  "readTime": {{{readTime}}},
  "dateAsTimeStamp": {{{moment date format="X"}}},
  "content": [
  {{#each content}}
    {{#if text}}
        {{#if @index}},{{/if}}
        {{{JSONstringify text}}}
    {{/if}}
  {{/each}}
  ],
  "imagePath":"{{{image.image.endpoint}}}/{{{image.image.name}}}"
  {{/withDeliveryContentItem}}
}

If you're using the blog content type from Part 1, paste the handlebars shown above into your webhook's custom payload and save the webhook.

Some points to note about the custom payload:

• deliveryKey is a friendlier way of referring to the delivery key than _meta.deliveryKey
• in the custom payload we iterate through authors and create authors.name to use with attributesForFaceting
• dateAsTimeStamp contains the date converted from a string to unix timestamp format so that it can be used for sorting. To sort using dateAsTimeStamp you can update your custom ranking as follows:

{
   "customRanking": [
    "desc(dateAsTimeStamp)"
  ]
}

See the configuring your search indexes for more information about updating your search configuration when using Dynamic Content search.

• The image endpoint and name are concatenated to make path. This doesn't include mediaHost as this should be an environment variable in your app.

After updating your custom payload, if you have changed the names of any attributes, or added new ones, you may need to update your search settings. If you are using Dynamic Content search see search configuration, otherwise see the Algolia dashboard for these settings.

Step 3: Trigger the webhook and view the activity log

The next step is to trigger the webhook in order to view the webhook body created from the custom payload.

• Create or update a blog content item to trigger the webhook.
• Open the webhook activity log and view the most recent webhook delivery.

The webhook delivery details should look something the following image and include the request body that was generated from the custom payload.

If you view the Algolia record it will now contain just the fields that you specified in the custom payload.

{
  "title":"Powering your integrations with Dynamic Content webhooks",
  "description":"We take you through our great new webhook features.",
  "deliveryKey":"dynamic-content/webhooks",
  "schema":"http://example.com/blogpost.json",
  "authors":[
    {
      "name":"Amplience"
    }
  ],
  "date":"2020-02-26",
  "readTime":10,
  "dateAsTimeStamp":1582675200,
  "content":[
    "This blog post will walk you through the great new webhook customization features.",
    "This is part 2 of the text explaining the new features."
  ],
  "imagePath":"ampproduct/five-bulb-lights-1036936"
}

When to use the Algolia batch API

The custom payload shown in this example can be used as long each blog article is not too long, but there may be cases where it exceeds the recommended maximum size of 5KB. In this case we recommend breaking the blog, or other large content items, into multiple records and using the Algolia batch API. This is covered in Part 3.

Related pages

Example: Search integration (Algolia), Part 1

Example: Search integration (Algolia), Part 3

Webhooks overview

Customizable webhooks

Webhook payloads

Algolia documentation

What to put in your record

Reducing object size