MyBinder.org Events Archive not recording launches

[rgaiacs]

https://archive.analytics.mybinder.org/ shows

Date	Filename	Number of Events
2025-01-16	events-2025-01-16.jsonl	0
2025-01-15	events-2025-01-15.jsonl	28
2025-01-14	events-2025-01-14.jsonl	0
2025-01-13	events-2025-01-13.jsonl	1310
2025-01-12	events-2025-01-12.jsonl	799
2025-01-11	events-2025-01-11.jsonl	865
2025-01-10	events-2025-01-10.jsonl	3489
2025-01-09	events-2025-01-09.jsonl	5216

On 2025-01-10, the OVH server went down as reported in #3160.

On 2025-01-13, #3165 change the traffic from mybinder.org.

On 2025-01-14, archive.analytics.mybinder.org recorded zero events. But Grafana shows otherwise.

On 2025-01-15, something happen that a few events were recorded.

cc @arnim

[minrk]

I don't see an event published from GESIS since 2025-01-15T22:04:14.729182Z Can you check the logs from the GESIS binderhub pod? This should be a direct communication between the BinderHub instance and Google's logging platform, initiated here.

[rgaiacs]

The log of the binder pod includes

[E 250116 11:09:13 background_thread:118] Failed to submit 1 logs.
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
        return callable_(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 1181, in __call__
        return _end_unary_response_blocking(state, call, False, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
        raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNAVAILABLE
    	details = "failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2001:4860:4802:34::174%5D:443: connect: Network is unreachable (101)"
    	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2001:4860:4802:34::174%5D:443: connect: Network is unreachable (101)", grpc_status:14, created_time:"2025-01-16T11:09:13.217412297+00:00"}"
    >
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
        result = target()
                 ^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
        raise exceptions.from_grpc_error(exc) from exc
    google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2001:4860:4802:34::174%5D:443: connect: Network is unreachable (101)

I will follow up with @arnim.

[minrk]

ok, so seems like a networking problem