June 22, 2021 | 10 Min
Part 3: How Amplience Helps to Simplify Your Picture Tags
Author
Neil Cooper
- Engineering
In part two we saw how you can use picture tags to enable developers to request images in the right format, the right dimensions, and even with optimized content (art direction).
This came at the cost of complexity and a very large number of lines of code. At best this means extra payload for the request, at worst it means extra maintenance overhead and an increase in the possibility of errors.
DAM / CDN providers like Amplience have tools to help simplify these 3 key areas.
Art Direction
So - you have a landscape or portrait image, but you need the opposite, or a square-shaped image. How can you keep the point of interest in the image without having to manually create two or three alternative images?
Amplience has two solutions to this problem, one is slightly manual, the other is entirely automated, and uses AI to identify the point of interest.
Point of Interest
Amplience allows you to store metadata against images that are uploaded. By default, the EXIF metadata and file metadata are captured, but you can also assign custom metadata. One such custom meta data set is the Point of Interest meta data set. Amplience has a user interface that allows us to set a point of interest without having to mess around with API calls:
1https://cdn.media.amplience.net/i/bccdemo/husky?w=300&poi=0.167,0.125,0.245,0.236&sm=aspect&aspect=2:3&scaleFit=poi
1<picture> 2 <source srcset="cats-landscape.jpg?w=1024 1024w, 3 cats-landscape.jpg?w=1440 1440w, 4 cats-landscape.jpg?w=1920 1920w 5 cats-landscape.jpg?w=2048 2048w" 6 media="(min-width: 1024px)" 7 sizes="(min-width:2048px) 2048px, 100vw"> 8 9 <source srcset="cats-landscape.webp?w=1024 1024w, 10 cats-landscape.webp?w=1440 1440w, 11 cats-landscape.webp?w=1920 1920w 12 cats-landscape.webp?w=2048 2048w" 13 media="(min-width: 1024px)" 14 sizes="(min-width:2048px) 2048px, 100vw"> 15 16 <source srcset="cats-portrait.jpg?w=768 768w, 17 cats-portrait.jpg?w=800 800ww, 18 cats-portrait.jpg?w=1536 1536w 19 cats-portrait.jpg?w=1600 1600w" 20 media="(min-width: 760px)" 21 sizes="100vw"> 22 23 <source srcset="cats-portrait.webp?w=768 768w, 24 cats-portrait.webp?w=800 800w, 25 cats-portrait.webp?w=1536 1536w 26 cats-portrait.webp?w=1600 1600w" 27 media="(min-width: 760px)" 28 sizes="100vw"> 29 30 <source srcset="cats-landscape-small.jpg?w=320 320w, 31 cats-landscape-small.jpg?w=360 360w, 32 cats-landscape-small.jpg?w=412 412w, 33 cats-landscape-small.jpg?w=640 640w, 34 cats-landscape-small.jpg?w=720 720w, 35 cats-landscape-small.jpg?w=824 824w" 36 media="(min-width: 320px)" 37 sizes="100vw"> 38 39 <source srcset="cats-landscape-small.webp?w=320 320w, 40 cats-landscape-small.webp?w=360 360w, 41 cats-landscape-small.webp?w=412 412w, 42 cats-landscape-small.webp?w=640 640w, 43 cats-landscape-small.webp?w=720 720w, 44 cats-landscape-small.webp?w=824 824w" 45 media="(min-width: 320px)" 46 sizes="100vw"> 47 <img src="cats-small-landscape.jpg" width="100%" border="0" alt="" title=""> 48</picture>
1<picture> 2 <source srcset="http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=320&sm=aspect&aspect=16:9 320w, 3 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=360&sm=aspect&aspect=16:9 360w, 4 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=412&sm=aspect&aspect=16:9 412w, 5 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=640&sm=aspect&aspect=16:9 640w, 6 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=720&sm=aspect&aspect=16:9 720w, 7 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=824&sm=aspect&aspect=16:9 824w, 8 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=768&sm=aspect&aspect=3:4 768w, 9 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=800&sm=aspect&aspect=3:4 800w, 10 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=1536&sm=aspect&aspect=3:4 1526w, 11 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=1600&sm=aspect&aspect=3:4 1600w, 12 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=1024&sm=aspect&aspect=16:9 1024w, 13 http://cdn.media.amplience.net/i/bccdemo/cats.webp?poi$&w=1440&sm=aspect&aspect=16:9 1440w, 14 http://cdn.media.amplience.net/i/bccdemo/cats.webp?$poi$&w=1920&sm=aspect&aspect=16:9 1920w 15 http://cdn.media.amplience.net/i/bccdemo/cats.webp?poi$&w=2048&sm=aspect&aspect=16:9 2048w" 16 media="(min-width: 320px)" 17 sizes="(min-width:320px) 320px, 100vw,(min-width:760px) 760px, 100vw,(min-width:1024px) 1024px, 100vw"> 18 <source srcset="http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=320&sm=aspect&aspect=16:9 320w, 19 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=360&sm=aspect&aspect=16:9 360w, 20 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=412&sm=aspect&aspect=16:9 412w, 21 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=640&sm=aspect&aspect=16:9 640w, 22 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=720&sm=aspect&aspect=16:9 720w, 23 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=824&sm=aspect&aspect=16:9 824w, 24 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=768&sm=aspect&aspect=3:4 768w, 25 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=800&sm=aspect&aspect=3:4 800w, 26 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=1536&sm=aspect&aspect=3:4 1526w, 27 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=1600&sm=aspect&aspect=3:4 1600w, 28 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=1024&sm=aspect&aspect=16:9 1024w, 29 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?poi$&w=1440&sm=aspect&aspect=16:9 1440w, 30 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=1920&sm=aspect&aspect=16:9 1920w 31 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=2048&sm=aspect&aspect=16:9 2048w" 32 media="(min-width: 320px)" 33 sizes="(min-width:320px) 320px, 100vw,(min-width:760px) 760px, 100vw,(min-width:1024px) 1024px, 100vw"> 34 <img src="http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=1024&sm=aspect&aspect=16:9)" width="100%" border="0" alt="" title=""> 35</picture>
not much shorter, but we've been able to remove the extra versions of the assets (cats-portrait, cats-landscape-small) by specifying the Point of Interest.
You can also see another Amplience feature on show here:
1$poi$ represents a template that contains common url parameters: &scaleFit=poi&poi={$this.metadata.pointOfInterest.x},{$this.metadata.pointOfInterest.y},{$this.metadata.pointOfInterest.w},{$this.metadata.pointOfInterest.h}
which dynamically pulls the values of the Point of Interest from the image metadata into the image request.
If you had a set of common aspect ratios you could also set the &sm=aspect&aspect= values into templates too so that only the width would be specified in the URL.
e.g.
1http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$$landscape$&w=2048where $landscape$ now represents &sm=aspect&aspect=16:9
Image Recognition
You may have seen another blog post about image recognition, where I talked about how Amplience uses AWS Rekognition to automatically identify objects, faces, and text within images as they're uploaded.
When Amplience's image recognition service detects a face, it puts the values for the bounding box into metadata which is available to the Dynamic imaging service.
We can use some of those values to populate the poi attribute that we saw above so instead of:
1poi={$this.metadata.pointOfInterest.x},{$this.metadata.pointOfInterest.y},{$this.metadata.pointOfInterest.w},{$this.metadata.pointOfInterest.h}
we can put:
1poi={$this.metadata.detectedFaces.data.boundingBox.left},{$this.metadata.detectedFaces.data.boundingBox.top},{$this.metadata.detectedFaces.data.boundingBox.width},{$this.metadata.detectedFaces.data.boundingBox.height}
and of course, this can be stored in a template as we saw earlier, so a URL like:
1http://cdn.media.amplience.net/i/bccdemo/pexels-burst-374044?$facePoi$&w=288&sm=aspect&aspect=1:1
Contains all the information Amplience Dynamic Media needs to render the original image while ensuring the face remains in the crop, regardless of the size or aspect ratio. The main difference here is that we were able to remove the manual step of assigning the Point of Interest.
Amplience Smart Images
One of Amplience's newest features is Amplience Smart Images - this enables developers to do away with picture tags for format switching. This new feature lets Ampliences Dynamic Imaging service use information from the browser to determine which format it should return. WEBP for Chrome / Chromium based browsers , JP2 for Safari and then JPEG for anything older than that. In effect this means that our last set of picture tags can now be reduced to:1<img srcset="http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=320&sm=aspect&aspect=16:9&qlt=default&fmt=auto 320w, 2 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=360&sm=aspect&aspect=16:9&qlt=default&fmt=auto 360w, 3 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=412&sm=aspect&aspect=16:9&qlt=default&fmt=auto 412w, 4 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=640&sm=aspect&aspect=16:9&qlt=default&fmt=auto 640w, 5 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=720&sm=aspect&aspect=16:9&qlt=default&fmt=auto 720w, 6 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=824&sm=aspect&aspect=16:9&qlt=default&fmt=auto 824w, 7 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=768&sm=aspect&aspect=3:4&qlt=default&fmt=auto 768w, 8 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=800&sm=aspect&aspect=3:4&qlt=default&fmt=auto 800w, 9 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=1536&sm=aspect&aspect=3:4&qlt=default&fmt=auto 1526w, 10 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=1600&sm=aspect&aspect=3:4&qlt=default&fmt=auto 1600w, 11 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=1024&sm=aspect&aspect=16:9&qlt=default&fmt=auto 1024w, 12 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?poi$&w=1440&sm=aspect&aspect=16:9&qlt=default&fmt=auto 1440w, 13 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=1920&sm=aspect&aspect=16:9&qlt=default&fmt=auto 1920w 14 http://cdn.media.amplience.net/i/bccdemo/cats.jpg?$poi$&w=2048&sm=aspect&aspect=16:9&qlt=default&fmt=auto 2048w" 15 size="(min-width:320px) 320px, 100vw"> 16 width="100%" border="0" alt="" title="">
Recall in an earlier blog post that srcset and size attributes enable browsers to determine the dimensions for the image and choose from the set of available sources. You don't necessarily need this many sources, but it does reduce how many extra bytes are requested. It's worth reviewing your users viewport sizes if you can in order to optimize your breakpoints.
You can find further information about Amplience Smart Images here.
Hopefully this has shown you a few of the tools that are available within Amplience to reduce the impact of art direction on your picture tags. In Part 4 we'll look at how an upcoming browser feature called client hints can be used to supercharge image requests when combined with an image API like Amplience's.
