Exception thrown when adding an existing Decklink consumer in PAL format (identified for Quad 2 cards)

[leftearsons]

Expected behaviour

When adding a Decklink consumer to a channel, we would expect it to be added successfully, regardless of whether the consumer is already added or not, regardless of the channel output format and regardless of the Decklink card type used (Quad 1 or Quad 2).

Current behaviour

When adding a Decklink consumer to a PAL configured channel, in case the Decklink consumer is already added and the Decklink card type is Quad 2, an exception is thrown preventing it from loading successfully.
For non-PAL output format (720p5000 & 1080i5000 have been tested), as well as for Decklink Quad 1 cards, the issue is not present, allowing the Decklink consumer to be added successfully.

Below is the error log output after running the steps below:

DeckLink Quad 2 [1-7|PAL] Scheduled playback has stopped.
[decklink_consumer] Uninitialized.
DeckLink Quad 2 [1-7|PAL] Uninitialized.
Exception: D:\a\server\server\src\modules\decklink\consumer\decklink_consumer.cpp(441): Throw in function void __cdecl caspar::decklink::decklink_consumer::enable_video(enum _BMDDisplayMode)
Dynamic exception type: class boost::exception_detail::clone_impl<struct caspar::caspar_exception>
[struct caspar::tag_msg_info * __ptr64] = DeckLink Quad 2 [1-7|PAL] Could not enable fill video output.
[struct caspar::tag_stacktrace_info * __ptr64] =  0# 0x00007FF76F965217 in casparcg
 1# 0x00007FF76F98B756 in casparcg
 2# 0x00007FF76FAD2075 in casparcg
 3# 0x00007FF76FACDF67 in casparcg
 4# 0x00007FF76FACEDD7 in casparcg
 5# 0x00007FF76F999AFF in casparcg
 6# 0x00007FF76F9F974E in casparcg
 7# 0x00007FF76F9F3AC6 in casparcg
 8# configthreadlocale in ucrtbase
 9# BaseThreadInitThunk in KERNEL32
10# RtlUserThreadStart in ntdll



 0# 0x00007FF76F965217 in casparcg
 1# 0x00007FF76F964CA5 in casparcg
 2# 0x00007FF76FC1C8D7 in casparcg
 3# 0x00007FF9F7D61080 in VCRUNTIME140_1
 4# _NLG_Return2 in VCRUNTIME140_1
 5# RtlCaptureContext2 in ntdll
 6# 0x00007FF76FA7BCBF in casparcg
 7# 0x00007FF76F999AFF in casparcg
 8# 0x00007FF76F9F974E in casparcg
 9# 0x00007FF76F9F3AC6 in casparcg
10# configthreadlocale in ucrtbase
11# BaseThreadInitThunk in KERNEL32
12# RtlUserThreadStart in ntdll

---

Steps to reproduce

1. Make sure you are using a Decklink Quad 2 SDI card
2. Make sure you have a PAL channel configured
3. Add a Decklink consumer to the PAL channel by sending the AMCP command: ADD 1 DECKLINK 1 EMBEDDED_AUDIO LOW_LATENCY
4. Re-add the same Decklink consumer to the same channel by sending the same AMCP command: ADD 1 DECKLINK 1 EMBEDDED_AUDIO LOW_LATENCY

Result: An exception should be thrown and the Decklink consumer should be forcefully removed from the channel

Environment

• Commit: 3e35453
• Server version: v2.3.3
• Operating system: Windows 10

The issue is present on all versions from release 2.3.3 stable forward.
Issue was not present in version 2.07, but i have not tested versions in between.

---

[leftearsons]

I just identified that the issue is also present when using 720p5000 output format, but with a much lower frequency (it occured 4 times during two days, when regularly re-adding the Decklink consumer every one hour).

[Julusian]

This behaviour has been around for as long as I can remember, at least since 2.1. It happens because destroying the old consumer gets done asynchronously, and so initializing the new producer happens first resulting in the failure as the decklink is still in use.
I don't have any ideas on an easy fix for this, as it needs to be done asynchronously.

[didikunz]

I don't have any ideas on an easy fix for this, as it needs to be done asynchronously.

Just issue a REMOVE command first and the ADD command a few seconds later (from the client).

[leftearsons]

Indeed @didikunz, that is an easy and obvious workaround, but does not actually solve the problem because the output would become black for those seconds, which of course is not acceptable.

[Julusian]

the output would become black for those seconds, which of course is not acceptable.

I expect that having some black on the output is unavoidable. Even if CasparCG were to handle this case correctly, it would still be closing and re-initialising the decklink. At best it may freeze for some frames, but it could also show some black.

[leftearsons]

Thanks for your comment @Julusian.
The fact is that on version 2.07 that i was last using before switching to 2.3.3, there was no freeze or black frames in between when CasparCG was auto reloading the SDI output.

[Julusian]

Interesting.. I would have thought that because the decklink is being prerolled with black, that it would actually output that..

But the key change is that in 2.0 producers and consumers were destroyed synchronously, in 2.1 and later this is being done asynchronously. The benefit being that if removing a consumer/producer (where there arent the same access limitations) takes some time, the new consumer can start being created before that destruction completes.
I am struggling to see much benefit for consumers, as they are so rarely changed so am open to rethinking this, but that requires time.

I do think you have a very unusual use case here though, what is the reason for needing to remove and readd the same decklink to a channel?

[leftearsons]

@Julusian thank you for your time. The real reason for using this "technique" (reload an SDI output), is to deal with audio disappearing issue CasparCG/help#24

In practice, for PAL and 720p5000 formats i was reloading the SDI output every 1 hour, while for 1080i5000 format i was reloading the SDI output every 30 minutes.
This way, the audio buffers (as shown in the diagnostics window) were resetting while no black or frozen frames would exist in the channel output (at least up until version 2.07).

Based on the design you described above, what about using a synchronous approach only when the exact same producer/consumer is already in use?