Using webhooks with Dynamic Content
A step-by-step guide for creating your own integration with Dynamic Content using webhooks, Typescript and Express.js
Recipe for integrating with Dynamic Content using webhooks
Dynamic Content is great at managing content, but sometimes you might want to include other components in your architecture such as a search index, database or a file server. In these cases, it can be useful to synchronise content produced in Dynamic Content with these other systems. Webhooks provide a way to do just that.
Algolia is a hosted search engine that allows you to create a fast, as-you-type search experience with instant results.
Using Algolia to index your data in Amplience Dynamic Content provides a method for returning instant search results, which are nicely formatted and contain both your images and descriptive text.
In this article we talk about how we created a simple integration between Algolia and Dynamic Content (DC) which uses webhooks to synchronise changes between an Algolia index and content items when they are published in DC.
This article is a good starting point if you want to create your own integration between Dynamic Content and another third party service, such as Algolia. We provide a complete example of a third-party integration on GitHub.
What exactly are Webhooks then?
Webhooks are a means by which a web application can notify another of an event and pass information about the event. For this to work, your application must first 'subscribe' to certain events that it wants to be notified of.
In this context the web application is an 'observer' and will be automatically invoked whenever an event is broadcast by Dynamic Content, which is the 'subject' of the webhook.
You may have come across this pattern, known as the Observer Pattern, before. A webhook is simply an implementation of this pattern between two services over HTTP.
What do we need for a webhook integration?
Simply put, a web application. This will subscribe to certain events and then act upon the contents of these when it receives a notification: perhaps making a call to an API, pushing a message onto a queue; potentially almost anything.
We also need a way of building, deploying and serving the application. To speed up development, it would be nice if we could do this locally on our development machine, so we need some way of making the web application externally visible to the system we are observing (in our case, Dynamic Content).
Creating a web application
Setting up the project
We'll start by creating a bare-bones express application, get this running on our local machine and then iteratively add new features:
add a webhook handler
add calls to Dynamic Content to fetch content
add a call to the Algolia index to add our fetched content
add validation We'll call our app
dc-integrations-algoliaand create a folder of the same name in the current working directory. Under this we'll create some folders for our source and test files:
express-application.tsfile sets up Express and configures the router, error handling and any additional middleware that we want. We'll define a bare-bones implementation for now:
index.tsfile is the main entry point for our application and is where we start our Express app to listen for connections:
Again, this is just bare-bones at the moment; we'll add more functionality in here for validation and configuration later. The other two files,
tsconfig.json, define the npm package dependencies and configure the Typescript root files and compiler options, respectively. We won't go into the specifics of these here; instead we suggest that you download copies of these files from the full example integration on GitHub.
Running the application locally
To run the application locally, run the following (default port is 3000)
Exposing your local port using ngrok
ngrok to make our locally running application visible to the outside world. This will allow it to be invoked by the Dynamic Content webhook handler.
Ngrok is reverse proxy tunnelling software which establishes a secure tunnel from a public endpoint (i.e. what will be our webhook 'callback' url) to a locally running network service (our integration application).
Ngrok also provides a web UI where you can introspect all HTTP traffic running over your tunnels and this is immeasurably useful when it comes to debugging webhooks.
You can signup for an Ngrok account for free and then download and install the version of Ngrok suitable for your platform. The setup and installation step will show you your generated authtoken for your Ngrok account.
You can run the following command to add your account's authtoken to your ngrok.yml file. This will give you more features and all open tunnels will be listed in the Ngrok dashboard:
Adding a webhook handler
We'll create a folder to contain the Typescript class files for our webhook processor and route handling:
The Application Coordinator pattern
The default architecture for most web applications, Model View Controller (MVC), can begin to break down as Controllers become bloated and logic begins to creep into the view templates.
The Coordinator pattern addresses this problem by adding another layer of abstraction: a class representing the state of the view (this is usually called the 'Presenter'). The presenter forms a contract between components with different responsibilities.
We'll use this pattern to ensure that we explicitly handle all outcomes in our logic (i.e. responding differently depending on whether it is a success or failure scenario). This will help separate out the route handling logic in one component from the business logic which processes the webhook in another component.
We'll go into more details about the presenter interface when we talk about the webhook processor.
The webhook processing
In the webhooks directory, create a file named
snapshot-published-webhook.ts. This contains the class responsible for handling the processing of our webhook:
In this case we've named our class
SnapshotPublishedWebhook since it responds to Dynamic Content snapshot publishing events. This will allow us to index any new content when it is published.
On our class we define a method,
processWebhook, that will be called by our route handler function (which will in turn be invoked when a webhook is received from Dynamic Content).
The method is passed two parameters: a copy of the request that was received by the webhook endpoint and the presenter used to notify the route handler whenever an exit point (failure or success) is reached during the webhook processing logic.
The webhook processor must carry out the following operations:
validate the fields of the webhook request, also ensuring that the webhook event is the one we're expecting
create a connection to Dynamic Content and retrieve the snapshot
use the snapshot to retrieve the version of the content item related to the snapshot
check that the content item type is in the allowed 'type whitelist'
extract just the properties that we want to index from the content item using the 'property whitelist'
add the extracted properties to the Algolia index
Validating the webhook event type
We use the popular Class Validator library to validate the fields of the webhook request. This works using decorators that we define on our model:
Again, we invoke the presenter to return an error if the event is not a snapshot published event.
Creating a connection to Dynamic Content
We need to use the Dynamic Content SDK to create a client for the Dynamic Content API. We use the OAuth2 credentials (id and secret) passed into our processor to initialise the connection:
We also pass configuration details containing the URLs for the Dynamic Content API and the Amplience Authentication Service.
Retrieving the snapshot
Next, we'll fetch the snapshot (a representation of the content graph for a particular version of a content item) from the Dynamic Content API and then use a templated HAL link on the snapshot to fetch the correct version of the content item corresponding to our snapshot (the 'root' content item on the snapshot):
Again, we invoke the presenter to report any error.
Now that we have the correct version of the content item we need to check that the content type is one of the ones that we are expecting and exclude it if it is not: this narrows down the amount of content that will be indexed which will help us avoid exceeding the maximum capacity of the Algolia index.
We will keep a whitelist of content types which we handle in our web application: we need to check that the type of our fetched content item is in the whitelist. We do this using the schema identifier:
If we end up with no properties after our whitelist filtering then this is probably due to a misconfiguration so we report this an error via the presenter.
Updating the Algolia index
The final step is to build up the object that we want to pass to the Algolia indexing engine. This will have a copy of the keys/values of the properties that we've whitelisted from our content item.
It will also have the the unique reference (
objectID) for the Algolia index. In this case, we'll use the id of the content item so that we can retrieve it in full when we do a search using the index):
As usual, any error is reported via the presenter.
If this indexing operation completes without error, then we need to signal to the presenter that we have been successful. This is the last step in our processing function:
Adding a route handler for the webhook callback
Now that we've created our webhook handler we need to wire this up to the controller in our application so that it gets invoked when the webhook callback endpoint is called by Dynamic Content.
The route handler function
In the webhooks directory, create a Typescript file named
snapshot-published-webhook-route-handler.ts. This contains the Express route handling function:
The route handler function is invoked when a webhook is received by the application. It then invokes the webhook handler, passing it an instance of the presenter. The webhook handler uses the presenter to call back to the route handler (e.g. whenever a failure occurs) which is responsible for determining the response code.
Here we handle the case when we receive a webhook event type that we do not support (i.e. not a snapshot published event). The code throws an exception which is handled by the default error handling in Express.
Registering the route handler with Express
We must register the route handling function that we've defined with the Express Router instance. We'll insert a call to the router in
express-application.ts for our webhook callback endpoint:
This ensures that our handler is invoked when we receive an incoming webhook callback to our '/webhook' endpoint.
Create a webhook subscription in Dynamic Content
Follow these steps to setup the webhook subscription in Dynamic Content:
1. Go to the Webhook management section within Dynamic Content ("Development" -> "Webhooks" from the menu)
Copy the public URL displayed by the output from
The URL must end in "/webhook"
You can provide your own secret, or follow our recommended approach of using the generate button to create a random signature.
- This must be passed in as your WEBHOOK_SECRET environment variable which will be used later when validating the webhook signature
You must select the "Snapshot - published" Webhook trigger
Publish a content item
If you've followed the steps above and you now publish a content item in Dynamic Content it will trigger your Algolia integration. If you have the debug log enabled for Express (define the environment variable
DEBUG=express:* to enable this), you should see output similar to the following in your console:
Validating the webhook signature
It is important to secure your webhook handlers to avoid creating a back door into your system. Dynamic Content signs each webhook using a shared secret so you can cryptographically prove it came from us.
With Express, you can provide multiple route handler functions that behave like middleware to handle a request.
We will use this facility to define an additional step which performs validation of the webhook request. Validation is necessary to ensure that we have received a valid HTTP request with a content type of
application/json, and that the webhook signature (passed encrypted in the X-Amplience-Webhook-Signature header) is valid, by checking it against our known secret. Edit
express-application.ts and replace the router call for the /webhook endpoint with the following:
This makes a call to a factory function
ValidateWebhookRequest.middleware which will return a route handler function (or series of functions) that checks the content type and performs validation of the webhook body in the passed request against the known secret which we have configured as an environment variable.
The implementation of this function involves defining one or more validation handler functions:
We return a 400 HTTP error (Bad Request) if the content type if the content type does not match any in the content type filter list, otherwise we just continue with
The function to validate the webhook signature is a little bit more involved: we must calculate the expected signature by encrypting the body of the webhook using the webhook secret and compare this against the supplied signature: if they don't match then we return a HTTP error:
In this handler we need to use the raw request body of the webhook (i.e. before any parsing) so that the webhook signature can be calculated. For this we use the Express
json middleware and pass it a
verify function as a parameter.
Finally, we just need to insert these functions into the sequence of functions called in the middleware processing chain:
Where to go from here
The code examples above do not cover the full details of creating an integration. There are plenty of other refinements to the basic application that you can make, including configuring the application using environment variables defined in a file.
Take a look at the full sample application on GitHub for details of some other ways of enhancing the integration and ideas you can use to build you own.
The Algolia integration discussed in this post is just one example of how you can use webhooks to integrate Dynamic Content with other services. Webhooks allow developers to be notified of events in real time and can be used to trigger actions in apps such as JIRA or Slack - easy to do with our Zapier integration. We also use webhooks to trigger a rebuild of our blog site when content is updated and to integrate with e-commerce systems.
If you want to build your own integration with Dynamic Content, the sample code and the information in this blog post should be a good starting point.