[WARNING] data websocket error, restarting connection: no close frame received or sent

[kimboox44]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

websocket connection on sip is unusable for me, a connection lost every 10 seconds, one of tow errors

• [WARNING] data websocket error, restarting connection: no close frame received or sent
• [WARNING] data websocket error, restarting connection: sent 1011 (unexpected error) keepalive ping timeout; no close frame received
  simple subscribtion to bars for all symbols
  attempts:
• tested versions 1.5.0 1.5.1
• changed host (local machine, friend's machine, personal laptop, 2 vps servers) exact same behaviour
• changed websocket versions i can't even remember how many
  log websocket shows ping timeout sometimes

some suggested that api already reconnect automatically, that's not a solution because you're missing 5sec of timeout before the connection actually drop plus reconnection+subscription time, that leaves gaps in 1minute bars by the time you start receiving again.
for 24h run i got a total of 290 candles on AAPL (because of this error) out of possible 960 candles that 70% lost data.

my guess:

• Alpaca servers websocket timeout ping interval is too short for such high traffic.
• Alpaca servers deliberately drop random connections at saturation to prevent service failure (problem happens at peak hours)

Expected Behavior

Expected to work

Steps To Reproduce

simple 1minute barse subscription for all symbols

Anything else?

No response

[umitanuki]

I am also experiencing this while comparing with CoinbasePro websocket. I've just run some 10 minutes and 2 of these where data stalled for some seconds or maybe 10+, while Coinbase Pro connection keeps runnnig without problem

[ovresko]

I can't understand Alpaca focusing in adding more features/markets while the core of thier business (data) is getting worse everyday. i'm thinking of switching, for the same problem, i'm sure they're dropping random connections to reduce load. they don't care anymore

[ccnlui]

This is a challenge inherent to streaming data with websocket, if the client is not consuming data at the rate that the server is producing, client is bound to get disconnected. This is a pretty good article of why this happens.

Some suggestions:

• dont subscribe to more data than necessary
• make sure handler is not doing heavy computation and it is reading as fast as possible from the websocket
• avoid running client in VMs or containers
• try the more performant go-sdk

[ovresko]

This is a challenge inherent to streaming data with websocket, if the client is not consuming data at the rate that the server is producing, client is bound to get disconnected. This is a pretty good article of why this happens.

Some suggestions:

• dont subscribe to more data than necessary
• make sure handler is not doing heavy computation and it is reading as fast as possible from the websocket
• avoid running client in VMs or containers
• try the more performant go-sdk

for me i'm subscribing to 10 to 20 symbols trades max on a 64Gb RAM 8Cores VPS nothing else, and yet connection drop every 5 to 20 seconds (1006,1011), if it's a websocket client issue we would experience the same for other websockets, for example our binance subscribtion run with all available coins +500 live, for 7 months with no disconnections (python also).
i've analyzed the traffic from alpaca, and it's always ALWAYS dropping around 10th 15th or 20th second on a new minute and not when receiving bars at second 0 to 5 where traffic is higher and bellow 20sec ping timout. our system designed around alpaca's websocket so switching is a nasty task.
i've tried subscribing to only one symbol with no significant volume, and yet it drops at 15th second.
i'm giving up

[umitanuki]

100% agree that this needs to be addressed. As I reported, I'm also comparing with coinbase pro websocket connection and it doesn't disconnect.

With my internal hat on, Alpaca's backend team is working on it and we are hoping to get this sorted.

@ovresko you mentioned it got worse. Do you know around when it got worse, and was it not like this before?

[raja]

@umitanuki @ovresko @kimboox44 To help better address the issue, can you see if you can reproduce the undesirable interimittent disconnects by using https://github.com/hashrocket/ws with the same subscription message you are using with alpaca python sdk. This will help us better understand if this is specific to the websocket client implementation in our python sdk (which I believe it may be) vs our streaming endpoint.

[ovresko]

@raja i did try ws, for bars it just stops same behaviour, after a minute without any messages
but it did hold the connection for 1 trades subscription (no bars) for 20 minutes now

tried in two seperate networks (DSL, 4G) same, and i don't see anything useful except response data meaning low level logs

[ovresko]

I tried using bare minimum subscription using websockets, and it's workign fine so far with 1000 live trades and bars subscription.
i think it's a problem with python api too

[ovresko]

there are some data providers that offers dynamic price update interval on ws, live/100ms/500ms/1s to be set at subscription to address the situation where clients can't keep up, that would solve all performance related issues (just opt for lower rate)

[drew887]

Hey just to give an update here since its been over a week.

While we are still investigating this issue, we have recently merged a PR that we hope will help lessen the frequency and severity of this issue in the mean time for most users. We also have a second PR merged allowing users to configure the options we pass to the websocket library we use under the hood themselves if these defaults wont work for them.

I will try and get a new release out soon so that these changes are available (EDIT: v2.0.0 has been released and contains these changes) but all I can ask is that you please continue to be patient with us in the meantime while we investigate 😓.

[sshcli]

Still facing this problem in the new python-py SDK