Timed metadata describes a subset of a stream through data (bytes, text, image, ...), a start time and/or a stop time. For example, if you're watching a 60 seconds video, and if I would declare that an advertisement is playing in this video between 00:30 and 00:45, then I am declaring timed metadata because I'm referring to a subset of this video.
Timed metadata shouldn't be confused with general metadata. General metadata describes data related to your entire video content, like the title of the video, its total duration and its PG-rating. In this article we use timed metadata and metadata interchangeably, and consider general metadata out-of-scope.
This article provides information on the different types of timed metadata and how you can leverage it in an application. The story is told in the reverse order: first we'll consider some use-cases (e.g. highlights), next we'll talk about the role of a video player versus the application, and finally we'll list the types of metadata (e.g. ID3) you can insert.
When using timed metadata, you are trying to achieve a use-case through a technical implementation. In an ideal world, your video player provides an out-of-the-box solution for your use-case, and you don't need to write and maintain a lot of code.
Timed metadata enables all kinds of use-cases in the field across various domains.
A popular use-case is to associate some data (text, image, ...) with a specific moment in time to mark a highlight.
For example, in the screenshot above, we're leveraging metadata to highlight where a goal has been scored. If you click on the goal icon, you'll be taken to this moment in time in the video. You could of course expand this feature, and hovering the goal icon would toggle a tooltip which tells you who scored and what the new score is.
Timed metadata can help you define the start and end of an advertisement. This information can be used to enhance the UI and UX. For example, at https://cdn.theoplayer.com/demos/verizon-media/index.html, the application leverages metadata to render markers in the scrub bar and to count down to the end of the ad break, and on top of that, if you try to seek over an ad break, the cursor snaps back to the ad break.
You can also leverage timed metadata "behind the scenes". For example, the timed metadata might contain info on the creative ID of the advertisement, and the URLs of its event trackers. An application could then use this information to communicate with an ad server or analytics server.
Timed metadata could also be used to initiate events and actions. For example, perhaps you want to overlay some text or image when you are at position 00:40. Or perhaps you want to disable seeking through the scrub bar for a defined period of time.
The screenshot showcases a demo where a relevant hyperlink is being overlayed on top of the video within a specific time period. This hyperlink (and its start and stop time) was indicated through timed metadata.
You can get very creative with timed metadata.
If you have a use-case where a certain behavior can be associated with a certain moment in time, you can probably tackle it through timed metadata.
To programmatically extract your metadata and implement your use-case, you will always need to write custom application logic to some degree.
On top of this, you might need to leverage a video player API to extract your metadata, if you pass along your metadata in such a way it adheres to a certain streaming specification. More about this in the following sections.
To implement the use-cases listed above, applications developers have to write and maintain application code.
Sometimes, pre-integrations offered by the video player can take away some responsibilities of the application developer. For example, THEOplayer has a pre-integration with Verizon Media which allows developers to easily configure whether viewers can see ad break markers, whether viewers can skip over ad breaks, and much more.
At other times, there might be a recommended workflow and pre-integrated feature to achieve your use-case. For example, you can use Chapter Markers to highlight different chapters in a video.
THEOplayer also acts as the main contributor to Enriched Media Streaming Solutions (EMSS). EMSS describes a standardized way to insert metadata and how a video player should handle and visualize this metadata. It might be interesting for you to reach out to THEOplayer and ask if your use-case is supported by EMSS out-of-the-box.
Depending on how you insert the metadata, you have to use your video player API to extract the metadata, and/or write custom application logic.
When you use THEOplayer as your video player API, you can use the
TextTracks API to extract certain types of timed metadata like ID3, emsg, EXT-X-DATERANGE and EventStream,
and APIs like
player.currentProgramDateTime to determine the absolute date and time should you want to compare that with an external CMS.
Let's talk about the different types of timed metadata.
It depends on your use-case and back-end which type you should pick, and which considerations you should make. Perhaps your back-end doesn't allow you to insert any stream metadata (i.e. segment and manifest metadata), or perhaps you cannot use an external CMS.
Segment metadata is contained within a segment. The industry also refers to segment metadata as in-band media timed events. There are two popular types of segment metadata:
.ts) is commonly used in HLS. ID3 can also occur in MPEG-DASH through a carriage over emsg.
.mp4). This type of segment is commonly used in MPEG-DASH, and in HLS when your HLS stream is using fragmented MP4 segments instead of .ts segments.
Manifest metadata is contained in the manifest of your streaming protocol. The industry also refers to manifest metadata as out-band media timed events. There are two popular types of manifest metadata:
You can also store metadata externally, for example in a remote database or in a file.
You can store metadata in an external database, for example by using a content management system (CMS). In this CMS, you will probably associate an IDs with asset, and associate certain types of data with these IDs.
Furthermore, your database should offer an API to your application, in order for your application developer to successfully query your database through an asset ID.
You can store metadata in files.
For example, you could leverage Chapter Markers and thus use
That being said, you would still need to know which asset should load which
.vtt file, thus creating the need for an underlying CMS.
Your video player might be able to determine timeline metadata through your segments or manifests. Knowing your playhead position, video duration and/or absolute date may be necessary for you to implement your use-case.
For example, your HLS manifest might contain the
#PROGRAM-DATE-TIME tag which defines the absolute date and time.
This property allows your video player to calculate and expose the absolute date and time, which you can then leverage
in your use-case.
Check with your back-end (vendor) whether if they expose the manifest metadata required for video players to calculate the absolute date and time.
For HLS, this goes through the
#PROGRAM-DATE-TIME tag. For MPEG-DASH, this goes through the
availabilityTimeOffset attribute, and additionally through the
We've listed some remarks below which could influence your decision in selecting a specific type of metadata.
THEOplayer is able to extract all applicable types of timed metadata across all supported platforms:
THEOplayer offers a number of pre-integrations to facilitate the implementation of relevant SSAI use-cases. Reach out to THEOplayer if your vendor is missing from the list below, because the list is incomplete.
THEOplayer offers features which might implement your desired use-case out-of-the-box: