Consumer disconnect() not called on container shutdown in Kubernetes

[royroyee]

We are running an ASGI app using Django Channels in a Kubernetes container environment.

The main issue we’re facing is that when a container is terminated or restarted, WebSocket connections are dropped, but the closed_at field of the related channel is not updated. It seems that the consumer’s disconnect() method is not being called properly in these cases.

Since our application logic relies on closed_at to determine whether a session is still active, this leads to inconsistencies.

Of course, we could define a shutdown task to update all closed_at fields when a container stops. However, we don’t know how to distinguish which sessions or channels belong to the specific container being shut down. If there’s a recommended way to identify or tag sessions per container, we’d really appreciate any guidance.

I’m wondering whether this behavior is expected due to limitations in Django Channels or ASGI, or if it can be mitigated by design.

asgi app command

  command: ["daphne", "-b", "0.0.0.0", "-p", "8000", "--application-close-timeout", "60", "--proxy-headers", "alpacon.asgi:application"]

disconnect() example

    async def disconnect(self, close_code):
        if hasattr(self, 'group_name'):
            await self.channel_layer.group_discard(
                self.group_name,
                self.channel.channel_name
            )
            await self.channel_layer.group_send(
                self.group_name,
                {
                    'type': 'leave_message',
                    'message': 'exit'
                }
            )
        if hasattr(self, 'channel'):
            if self.channel.closed_at is None:
                self.channel.closed_at = timezone.now()
                await database_sync_to_async(
                    self.channel.save
                )(update_fields=['closed_at'])

        await super().disconnect(close_code)

Thanks in advance! 🙏

[bigfootjon]

Do you have logs of what happens when Kubernetes attempts to shutdown the container?

AIUI Kubernetes issues a SIGTERM signal to PID1 of the Pod's container(s). Daphne runs on Twisted, so AIUI this code should control: https://github.com/twisted/twisted/blob/e09d0427fbd980d99c0a6ec6b9a8d329768cc623/src/twisted/internet/base.py#L829

Which means you should have Received SIGTERM, shutting down. in your logs. If so, something isn't hooked up between Twisted <-> Daphne <-> django-channels and that sounds like a bug that can be fixed (either in your code or channels itself). If you're not seeing that in the logs during pod termination, then I'm not sure where to start looking but it'll probably involve debugging unix signal handlers :(

[royroyee]

First of all, thank you so much for your helpful reply!

Here’s what I see in the logs when shutting down a pod in Kubernetes (I didn’t configure the Twisted logger, so this might be the only line being printed):

{"timestamp": "2025-04-25 14:43:20", "level": "INFO", "name": "daphne.server", "message": "Killed 0 pending application instances"}

From this line, it seems like the SIGTERM signal is being received correctly — would you agree?

I’ve also configured terminationGracePeriodSeconds generously to ensure graceful shutdown is possible.

That said, I’ve noticed that the disconnect() method doesn’t always get called — it seems a bit random when it does or doesn’t, which makes it even harder to debug

[bigfootjon]

From this line, it seems like the SIGTERM signal is being received correctly — would you agree?

Yes I agree!

That said, I’ve noticed that the disconnect() method doesn’t always get called — it seems a bit random when it does or doesn’t, which makes it even harder to debug

That's rather concerning. You say random, can you identify any pattern at all?

[royroyee]

Here’s why I suspect that disconnect() is not being called properly:

    async def disconnect(self, close_code):
       
       ...
    
        if self.channel.closed_at is None:
            self.channel.closed_at = timezone.now()
            await database_sync_to_async(
                self.channel.save
            )(update_fields=['closed_at'])

If disconnect() were properly triggered when the container shuts down, the closed_at field in the Channel model should be updated — but in most cases, it isn’t.

I described it as “random” because very occasionally the closed_at field is updated, but it’s so rare that in practice it almost never happens.

As a side question: in the pattern of using Django Channels, is updating a model field in disconnect() considered an anti-pattern? or could this issue be related to redis instead?