Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose wsrelay metrics #15042

Open
wants to merge 1 commit into
base: devel
Choose a base branch
from

Conversation

chrismeyersfsu
Copy link
Member

@chrismeyersfsu chrismeyersfsu commented Mar 27, 2024

SUMMARY
  • We were already maintaining prometheus metrics for the websocket relay service. This change exposes those metrics over the local per-service prometheus server.
ISSUE TYPE
  • New or Enhanced Feature
COMPONENT NAME
  • API
AWX VERSION
devel
ADDITIONAL INFORMATION
docker exec -it tools_awx_1 /bin/bash
curl localhost:8016

# HELP wsrelay_messages_received_total Number of messages received, to be forwarded, by the broadcast websocket system
# TYPE wsrelay_messages_received_total counter
wsrelay_messages_received_total{remote_host="awx_3"} 0.0
wsrelay_messages_received_total{remote_host="awx_2"} 0.0
# HELP wsrelay_messages_received_created Number of messages received, to be forwarded, by the broadcast websocket system
# TYPE wsrelay_messages_received_created gauge
wsrelay_messages_received_created{remote_host="awx_3"} 1.712085502612824e+09
wsrelay_messages_received_created{remote_host="awx_2"} 1.712085502613021e+09
# HELP wsrelay_connection Websocket broadcast connection established status
# TYPE wsrelay_connection gauge
wsrelay_connection{remote_host="awx_3",wsrelay_connection="disconnected"} 0.0
wsrelay_connection{remote_host="awx_3",wsrelay_connection="connected"} 1.0
wsrelay_connection{remote_host="awx_2",wsrelay_connection="disconnected"} 0.0
wsrelay_connection{remote_host="awx_2",wsrelay_connection="connected"} 1.0
# HELP wsrelay_connection_start_time_seconds Time the connection was established since unix epoch in seconds
# TYPE wsrelay_connection_start_time_seconds gauge
wsrelay_connection_start_time_seconds{remote_host="awx_3"} 1.712085502621528e+09
wsrelay_connection_start_time_seconds{remote_host="awx_2"} 1.7120855026317036e+09
docker exec -it tools_awx_2 /bin/bash
bash-5.1$ curl localhost:8016
# HELP wsrelay_messages_received_total Number of messages received, to be forwarded, by the broadcast websocket system
# TYPE wsrelay_messages_received_total counter
wsrelay_messages_received_total{remote_host="awx_3"} 0.0
wsrelay_messages_received_total{remote_host="awx_1"} 0.0
# HELP wsrelay_messages_received_created Number of messages received, to be forwarded, by the broadcast websocket system
# TYPE wsrelay_messages_received_created gauge
wsrelay_messages_received_created{remote_host="awx_3"} 1.7120854948292418e+09
wsrelay_messages_received_created{remote_host="awx_1"} 1.71208549482941e+09
# HELP wsrelay_connection Websocket broadcast connection established status
# TYPE wsrelay_connection gauge
wsrelay_connection{remote_host="awx_3",wsrelay_connection="disconnected"} 0.0
wsrelay_connection{remote_host="awx_3",wsrelay_connection="connected"} 1.0
wsrelay_connection{remote_host="awx_1",wsrelay_connection="disconnected"} 0.0
wsrelay_connection{remote_host="awx_1",wsrelay_connection="connected"} 1.0
# HELP wsrelay_connection_start_time_seconds Time the connection was established since unix epoch in seconds
# TYPE wsrelay_connection_start_time_seconds gauge
wsrelay_connection_start_time_seconds{remote_host="awx_3"} 1.7120854948457365e+09
wsrelay_connection_start_time_seconds{remote_host="awx_1"} 1.712085494845462e+09
docker exec -it tools_awx_3 /bin/bash
bash-5.1$ curl localhost:8016
# HELP wsrelay_messages_received_total Number of messages received, to be forwarded, by the broadcast websocket system
# TYPE wsrelay_messages_received_total counter
wsrelay_messages_received_total{remote_host="awx_1"} 0.0
wsrelay_messages_received_total{remote_host="awx_2"} 0.0
# HELP wsrelay_messages_received_created Number of messages received, to be forwarded, by the broadcast websocket system
# TYPE wsrelay_messages_received_created gauge
wsrelay_messages_received_created{remote_host="awx_1"} 1.712085498118138e+09
wsrelay_messages_received_created{remote_host="awx_2"} 1.7120855081194005e+09
# HELP wsrelay_connection Websocket broadcast connection established status
# TYPE wsrelay_connection gauge
wsrelay_connection{remote_host="awx_1",wsrelay_connection="disconnected"} 0.0
wsrelay_connection{remote_host="awx_1",wsrelay_connection="connected"} 1.0
wsrelay_connection{remote_host="awx_2",wsrelay_connection="disconnected"} 0.0
wsrelay_connection{remote_host="awx_2",wsrelay_connection="connected"} 1.0
# HELP wsrelay_connection_start_time_seconds Time the connection was established since unix epoch in seconds
# TYPE wsrelay_connection_start_time_seconds gauge
wsrelay_connection_start_time_seconds{remote_host="awx_1"} 1.7120854981252277e+09
wsrelay_connection_start_time_seconds{remote_host="awx_2"} 1.712085508127356e+09

@chrismeyersfsu chrismeyersfsu marked this pull request as draft March 27, 2024 17:41
@chrismeyersfsu chrismeyersfsu force-pushed the td-expose-websocket-relay-metrics branch 7 times, most recently from e01067c to 02247c8 Compare April 3, 2024 11:35
@chrismeyersfsu chrismeyersfsu force-pushed the td-expose-websocket-relay-metrics branch 4 times, most recently from d6e8294 to 3bcce4c Compare April 10, 2024 23:13
@chrismeyersfsu chrismeyersfsu force-pushed the td-expose-websocket-relay-metrics branch from 3bcce4c to 9f1d853 Compare April 11, 2024 19:05
@chrismeyersfsu chrismeyersfsu marked this pull request as ready for review April 11, 2024 19:53
@chrismeyersfsu chrismeyersfsu force-pushed the td-expose-websocket-relay-metrics branch 3 times, most recently from 0bb5eb5 to 859ea7f Compare April 12, 2024 12:34
print('\n'.join(lines))

print("Websocket Relay (wsrelay) status is available as prometheus metrics. Run the below curl command to view them.\n")
print(f"curl http://localhost:{settings.METRICS_SUBSYSTEM_CONFIG['server'][settings.METRICS_SERVICE_WEBSOCKET_RELAY]['port']}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might not have curl in more locked down base images

* We were already maintaining prometheus metrics for the websocket relay
  service. This change exposes those metrics over the local per-service
  prometheus server.
  for this instead.
* Remove run_wsrelay --status in favor of directing users at prometheus
  metrics
@chrismeyersfsu chrismeyersfsu force-pushed the td-expose-websocket-relay-metrics branch from 859ea7f to 80812a2 Compare April 12, 2024 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants