Resolve images source while generating ISD

[bbert]

Question: is there any reason for resolving images source while rendering HTML and not while generating ISD?

Resolving images at ISD generation step would ease the comparison of image subtitles before rendering the cues, by comparing the ISD contents.
And comparing subtitles may be useful to handle contigous subtitles with same content (same image).

[palemieux]

The ISD includes the smpte:backgroundImage attribute values and the application controls the image resolver, so it should be possible to compare images before HTML generation.

[bbert]

That means that you have to go through the returned ISD tree and resolve the sources.
Thus, these sources would be resolved twice: once while parsing the returned ISD, and once while rendering from within imscJS library.
I was just wondering if it could be more convenient to resolve while generating ISD, except if there was any valid reason I am not aware of.

[palemieux]

The assumption is that one can compare the value smpte:backgroundImage to determine whether images are identical, without resolving the image to a specific location. It might help if you shared a specific use case you have in mind.

[bbert]

Yes but the smpte:backgroundImage refers to an image id which can be different accross the chunks/segments but containing the same image data.
One specific use case is chunked live low latency streams, with consecutive chunks containing same image subtitle but with different id.

[palemieux]

Assuming the same image has two different smpte:backgroundImage URIs across two different chunks (which is not awesome), would resolving the URIs results in the same URL, or is the comparison done by hashing, or...? In other words, how does one determine if two images are identical?

[bbert]

No, agree this is not awesome, but for live low latency and chunked transfer encoding, you may have no other solution.
Thus you could compare the resolved URL or the base64 PNG representation

[palemieux]

How complex is the smpte:backgroundImage resolution process? Won't the UA cache the image when it is presented the first time around?

In the meantime, I will look to see if there were specific reasons not to resolve images at ISD creation time. Breaking the API is not great in any case.

[bbert]

This is not an issue of images caching, but rather an issue of comparing contigous subtitles in order to create a single cue instead of creating one cue per chunk.

[palemieux]

Ah. What is the drawback of creating one cue per ISD?

[bbert]

Sorry, I believed I already described this.
On browsers, contiguous cues with same content will produce some blinking effect, due to same content being hidden then immediatly rendered.
In dash.js we already implemented a patch to avoid this (Dash-Industry-Forum/dash.js#4103), but that works only for text subtitles.

[palemieux]

On browsers, contiguous cues with same content will produce some blinking effect, due to same content being hidden then immediatly rendered.

😭

[palemieux]

Does the blinking effect persist when the src attribute of the img element is modified, instead of a new img element created?

[bbert]

Since we have different cues, the renderHTML will create new HTML elements and the player will display these new elements in the dedicated subtitles rendering div, right after having hidden the previous one.

[palemieux]

Thanks for the detailed explanation. I suggest we explore the pain involved in moving the image resolver to ISD construction.

[bbert]

OK thanks.
For the time being, as an alternative we can eventually manage by going through the ISD and resolve the image sources before rendering HTML, but it maybe more relevant to resolve only once and avoid going through the ISD.

[nigelmegitt]

I'd suggest a mechanism for doing this would be to construct a data url from the dereferenced image resource, since the text of the data url can be compared directly, and that data url can be passed into the HTML also, avoiding a second fetch.

[bbert]

That sounds good. Let me know if you update the code so that I can try on my side.

[nigelmegitt]

Sorry to raise hopes - I will not be implementing it for the foreseeable future.