THEOplayer ❤️ Sideloaded Subtitles

THEOplayer-Connector-SideloadedSubtitle brings sideloaded subtitle support to THEOplayer 5.x.

By using an experimental API from THEOplayer this library aims to achieve sideloaded TextTrack support across all Apple devices and all media playback scenarios.

The presented technique is superior to any other similar options as it brings sideloading support to Airplay, Picture-in-Picture and DRM playback (and their combinations) and is out of the box compatible with the TextTrackStyling API from THEOplayer.

Installation

Cocoapods

1. Create a Podfile if you don't already have one. From the root of your project directory, run the following command: pod init
2. To your Podfile, add links to the registry of Cocoapods: source 'https://cdn.cocoapods.org/'. If this fails, bypass their CDN by using source 'https://github.com/CocoaPods/Specs.git'
3. To your Podfile, add the connector pods that you want to use in your app: pod 'THEOplayer-Connector-SideloadedSubtitle'
4. Install the pods using pod install , then open your .xcworkspace file to see the project in Xcode.

Usage

Import the THEOplayerConnectorSideloadedSubtitle module

import THEOplayerConnectorSideloadedSubtitle

After importing the module, a new API will be available on the THEOplayer instance called setSourceWithSubtitles(source: SourceDescription?)

If you use this method to set a source, THEOplayer will be able to present sideloaded subtitles (via the TextTrackDescription already known from THEOplayer 4.x) configured with your SourceDescription, e.g.:

public static var sourceWithSideloadedTextTrack : SourceDescription {
    let typedSource = TypedSource(src: "https://sourceURL.com/manifest.m3u8, type: "application/x-mpegurl")
    let textTrack = TextTrackDescription(src: "https://sideloadedurl.com/subtitle.vtt", srclang: "language_code", isDefault: false, kind: .subtitles, label:"Label", format: .WebVTT)
    return SourceDescription(source : typedSource, textTracks :[textTrack])
}

theoplayer.setSourceWithSubtitles(source: sourceWithSideloadedTextTrack)

NOTE: Replace every theopalyer.source = newSource call with theoplayer.setSourceWithSubtitles(source: newSource) in your application if you intend to use (at least once) subtitle sideloading. Even if you don't have sideloaded subtitle on some of your sources. Combining both approaches could lead to unexpected behavior.

Limitations

1. THEOplayer-Connector-SideloadedSubtitle relies on an experimental API from THEOplayer which is subject to change. Always use matching THEOplayer and Connector versions.
2. Supported formats are: iOS compatible WebVTT subtitles and SRT via this connector today.
3. The presented label within THEOplayer is auto-generated by the operating system based on the language code and the configured label will be disregarded. (e.g. "eng" --> "English" (if the device language is English; "eng" --> "Engels", if the device language is Dutch). If an invalid language code is specified, the system will use that one as label. (e.g. "MyCustomLanguageCode" --> "MyCustomLanguageCode")
4. The isDefault: true setting will be not taken into account on sideloaded TextTracks as it could lead to invalid source (if the stream already contains default subtitles). Enabling the desired sideloaded subtitle can be achieved via ADD_TRACK listener.

theoplayer.textTracks.addEventListener(type: TextTrackListEventTypes.ADD_TRACK) { ev in
    var texttrack = ev.track as! TextTrack
    if texttrack.language == "sideloaded_language_code" {
        tt.mode = .showing
    }
}

5. Loading thumbnail metadata through sideloaded WebVTT subtitles is not supported.

Setting a time offset

It is possible to shift the presentation of the cues by using the extended SSTextTrackDescription class instead of TextTrackDescription. It will provide an additional property vttTimestamp which allows to specify a X-TIMESTAMP-MAP.

The X-TIMESTAMP-MAP is optional and can be omitted from the WebVTT file and in general, it is not required. It is only necessary if you use a WEBVTT file inside an HLS subtitle rendition and you try to achieve subtitle synchronization. And this is exactly the way how this connector operates.

The APIs below provide a way of adding or modifying the timestamp in case it is required by mapping the PTS (e.g. from a video rendition) to a local time within the WEBVTT file.

When converting different formats to WebVTT, this module does not add the timestamp by default.

The API usage remains mostly the same, with the exception of adding the timestamp:

public static var sourceWithSideloadedTextTrack : SourceDescription {
    let typedSource = TypedSource(src: "https://sourceURL.com/manifest.m3u8, type: "application/x-mpegurl")
    let textTrack = SSTextTrackDescription(src: "https://sideloadedurl.com/subtitle.vtt", srclang: "language_code", isDefault: false, kind: .subtitles, label:"Label", format: .WebVTT)
    textTrack.vttTimestamp = .init(pts: "900000", localTime: "00:00:00.000")
    return SourceDescription(source : typedSource, textTracks :[textTrack])
}

theoplayer.setSourceWithSubtitles(source: sourceWithSideloadedTextTrack)

If the source WebVTT file specified in the SSTextTrackDescription already contains the X-TIMESTAMP-MAP timestamp, then it can be accessed by using the extractSourceTimestamp method:

let textTrack = SSTextTrackDescription(src: "https://sideloadedurl.com/subtitle.vtt", srclang: "language_code", isDefault: false, kind: .subtitles, label:"Label", format: .WebVTT)
textTrack.extractSourceTimestamp { timestamp, error in
    print(timestamp.pts, timestamp.localTime)   // outputs the values from "https://sideloadedurl.com/subtitle.vtt"
}

Note: The extractSourceTimestamp method does not set the vttTimestamp property, but only reads the timestamp value from the source. You would need this when you want to modify the timestamp according to the initial values.

More examples:

A sample WEBVTT file:

WEBVTT

1
00:00:11.000 --> 00:00:12.000
- Never drink liquid nitrogen.

2
00:00:13.000 --> 00:15:00.000
- It will perforate your stomach.
- You could die.

By using the sample file mentioned above, if your PTS is 900000 in the video rendition, then mapping 900000 to 00:00:00.000 will show the first subtitle cue exactly when you would expect it, at 11 seconds in the movie.

let textTrack = SSTextTrackDescription(src: "https://sideloadedurl.com/subtitle.vtt", srclang: "language_code", isDefault: false, kind: .subtitles, label:"Label", format: .WebVTT)
textTrack.vttTimestamp = .init(pts: "900000", localTime: "00:00:00.000")

(Experimental) Shifting the presentation of the cues forward (later) by 5 seconds.

You can achieve this by adding values to the first digit of the PTS timestamp with localTime set to 0:

let textTrack = SSTextTrackDescription(src: "https://sideloadedurl.com/subtitle.vtt", srclang: "language_code", isDefault: false, kind: .subtitles, label:"Label", format: .WebVTT)
// we assume here that the PTS in the video rendition is 900000, so 900000 + 500000 = 1400000
textTrack.vttTimestamp = .init(pts: "1400000", localTime: "00:00:00.000")

The timestamp is relative to the localTime, so the following settings will cancel each other out:

let textTrack = SSTextTrackDescription(src: "https://sideloadedurl.com/subtitle.vtt", srclang: "language_code", isDefault: false, kind: .subtitles, label:"Label", format: .WebVTT)
textTrack.vttTimestamp = .init(pts: "1400000", localTime: "00:00:05.000")

(Experimental) Shifting the presentation of the cues backwards (earlier) by 5 seconds.

You can achieve this by subtracting values to the first digit of the PTS timestamp with localTime set to 0:

let textTrack = SSTextTrackDescription(src: "https://sideloadedurl.com/subtitle.vtt", srclang: "language_code", isDefault: false, kind: .subtitles, label:"Label", format: .WebVTT)
// we assume here that the PTS in the video rendition is 900000, so 900000 - 500000 = 400000
textTrack.vttTimestamp = .init(pts: "400000", localTime: "00:00:00.000")

or set the localTime to 5 seconds while keeping the timestamp value the same:

let textTrack = SSTextTrackDescription(src: "https://sideloadedurl.com/subtitle.vtt", srclang: "language_code", isDefault: false, kind: .subtitles, label:"Label", format: .WebVTT)
// we assume here that the PTS in the video rendition is 900000
textTrack.vttTimestamp = .init(pts: "900000", localTime: "00:00:05.000")

Note: Every 100000 of PTS timestamp results in roughly 1 second in time mapping. Use this carefully as it is not officially documented.

Warnings

1. You should always use HLS subtitle rendition compliant WEBVTT files which are in sync with your streams and you should not be using the time shifting functionality.
2. Shifting cues backward is only possible until the timestamp of the first cue (e.g. if your first cue has to be rendered at 10 sec, you should not shift the subtitle with 15 sec).
3. Shifting forward is possible, but considered as hack, and it is not specification compliant manifest entry. Use it at your own risk!

Caching

The caching feature of this connector allows to download a source alongside a sideloaded subtitle to store it locally on the device and play it offline. To make this happen, the connector provides an API extension.

All that is needed is a source with a text track description:

public static var sourceWithSideloadedTextTrack: SourceDescription {
    let typedSource = TypedSource(src: "https://sourceURL.com/manifest.m3u8, type: "application/x-mpegurl")
    let textTrack = TextTrackDescription(src: "https://sideloadedurl.com/subtitle.vtt", srclang: "language_code", isDefault: false, kind: .subtitles, label: "Label", format: .WebVTT)
    return SourceDescription(source: typedSource, textTracks: [textTrack])
}

and to call the aforementioned API which creates a new caching task, and finally calling the start method:

let parameters = CachingParameters(expirationDate: .distantFuture)
let task = THEOplayer.cache.createTaskWithSubtitles(source: sourceWithSideloadedTextTrack, parameters: parameters)
task?.start()

For more information on how to implement offline playback with caching, please refer to our guide.

Limitations

1. Caching sources with sideloaded subtitles can only be done one task at a time. This is due to some technical complexities in the underlying implementation. This limitation may be addressed in future releases.
2. Caching is only available on iOS.