Part 3- Splitting large content items into multiple Algolia records

In Parts 1 and 2 of this example we showed you how to update an Algolia search index when a content item is created or updated in Dynamic Content. We also explained how to refine the custom payload to keep the size of each record below the recommended maximum size of 5KB.

Content items created using the blog content type used in this example can often exceed 5KB. You could limit the size of the record created in Algolia by indexing an excerpt of the blog, but then only the excerpt of the blog would be searchable. In the cases where the resulting Algolia records are expected to be above 5KB, the best practice is to split the output into multiple records. To do this you need to use the Algolia batch end point.

In Part 3 we walk you through the steps required to update the configuration and custom payload of the webhook you built in parts 1 and 2 to create multiple records from a blog content item and link them together in Algolia.

Prerequisites

This example uses the content type schemas included in Part 1 and Part 2, but the concepts will be the same whatever schemas you are using in your own projects.

Step 1: Configuring the webhook URL

To generate multiple records from the blog content item you will send a POST request to the batch endpoint of the Algolia API.

• Update the webhook URL to use the batch endpoint. Your webhook URL should now look something like:

https://yourAlgoliaAppid.algolia.net/1/indexes/yourIndexName/batch

• Change the webhook method to POST

Your webhook settings should now look those in the image below.

Step 2: Defining the custom payload

The next step is to update the custom payload. The handlebars shown below will index each piece of linked content from the blog content item as a separate record in Algolia. So if the blog contains two text content items, two records will be created.

Some key points to note about the custom payload:

• "action": "updateObject" is a request to the Algolia batch API to add or replace an object with the specified objectid.
• The Algolia objectid for each record must be unique and is set using the deliveryid of the item for which the record is created.
• The "id" field will be used to deduplicate the records. This is set to the deliveryid of the root content item- in this case the blog content item.

{
  "requests": [
    {{#withDeliveryContentItem contentItemid=payload.id account="myAccountid" stagingEnvironment="vseURL"}}
      {{#each content}}
        {{#if text}}
        {{#if @index}},{{/if}}
            {
              "action": "updateObject",
              "body": {
                  "objectid": "{{../_meta/deliveryid}}_{{_meta/deliveryid}}",
                  "id": "{{../_meta/deliveryid}}",
                  "title": "{{{../title}}}",
                  "description": "{{{../description}}}",
                  "authors": [
                    {{#each authors}}
                        {{#if @index}},{{/if}}
                        "{{name}}"
                    {{/each}}
                  ],
                  "date": "{{../date}}",
                  "readTime": {{../readTime}},
                  "content": {{{JSONstringify text}}}
              }
            }
        {{/if}}
      {{/each}}
    {{/withDeliveryContentItem}}
  ]
}

The custom payload will now look like the image shown below.

Step 3: Trigger the webhook and view the activity log

To trigger the webhook you can create or update a blog content item or use the webhook testing feature.

To use the testing feature you need to have triggered the webhook at least once and choose a previous webhook delivery to test with the new settings. Open the testing pane, choose a recent webhook delivery (1 in the image below) and click the "Test" button (2). The webhook will be invoked in testing mode and the webhook request will be displayed, but no request will be sent to the webhook URL. The latest version of your settings will be used for testing, you don't need to save the changes before you test.

A successful webhook test is shown below. In this example, the webhook body includes the request that will be sent to the Algolia API to update records for each of the content items linked to the blog item.

The webhook body generated from the custom payload is as follows. The payload is generated from two content items, each linked to one blog content item. Note the unique objectids and the same id field, allowing for deduplication in Algolia.

{
  "requests": [

            {
              "action": "updateObject",
              "body": {
                  "objectid": "a2b1d192-04d5-4219-81cf-7cf2e6e1b3cf_d7939a9c-1609-4505-8f3a-af3fada878d2",
                  "id": "a2b1d192-04d5-4219-81cf-7cf2e6e1b3cf",
                  "title": "Powering your integrations with Dynamic Content webhooks",
                  "description": "We take you through our great new webhook features",
                  "authors": [
                  ],
                  "date": "2020-02-26",
                  "readTime": 10,
                  "content": "This blog post will walk you through the great new webhook customization features."
              }
            }
        ,
            {
              "action": "updateObject",
              "body": {
                  "objectid": "a2b1d192-04d5-4219-81cf-7cf2e6e1b3cf_63876325-6842-4046-9de0-cebd19762dec",
                  "id": "a2b1d192-04d5-4219-81cf-7cf2e6e1b3cf",
                  "title": "Powering your integrations with Dynamic Content webhooks",
                  "description": "We take you through our great new webhook features",
                  "authors": [
                  ],
                  "date": "2020-02-26",
                  "readTime": 10,
                  "content": "This is part 2 of the text explaining the new features."
              }
            }
  ]
}

Step 4: Deduplicating your Algolia index

You Algolia index will contain two new records (depending on the structure of the content item you updated). Records generated from the same blog item will have the same value for the id field.

In your Algolia index, you need to deduplicate your search index setting "attributeForDistinct" to the "id" field and "distinct" to true. More information can be found in the Algolia guide to indexing large documents.

Related pages

Example: Search integration (Algolia), Part 1

Example: Search integration (Algolia), Part 2

Webhooks overview

Customizable webhooks

Webhook payloads

Algolia documentation

What to put in your record

Reducing object size

Indexing long documents