Question: Ensuring no websockets messages are lost

[muhammadsaeedparacha]

Noob question.

Situation:
I ran a web socket server and client.
My client subscribes to public and private (user stream) channels.
I passed a callback function/script in daemon mode, so it's run every time there's a message on the websocket
Based on an algorithm, my script triggers orders, then listens to private channel updates. On every update it does something else as well.

Problem:
Now I'm missing messages on the private channel on both locally and on a server.

Question:

1. Does the websocket worker queue the events to ensure that the callback would run on every message? Or does it run the callback with whatever is the latest data once the previous callback is finished which would result in missing messages?

[zhouaini528]

Hello ,I suggest you split the two parts to test websocket, first try to see if the public channel can be successful. Then test whether the private channel can succeed.

There is no queue for public channel messages, and there is a queue for private channels and a maximum of 100 messages can be stored.

Are you "spot" or "coin future" or "usdt future"? Example: "spot" setting $binance->keysecret(); will subscribe to private channels by default.

The server must be a daemon

$binance=new BinanceWebSocket();
$binance->config([]);
$binance->start();

The client has no requirements for the daemon

//Subscribe to all private channels by default
$binance->keysecret([
    'key'=>'xxxxxxxxx',
    'secret'=>'xxxxxxxxx',
]);
$binance->subscribe([
    'btcusdt@depth',
    'bchusdt@depth',
]);

//You can get it like this
$binance->keysecret([
    'key'=>'xxxxxxxxx',
    'secret'=>'xxxxxxxxx',
]);
$binance->getSubscribes(function ($data){
    print_r(json_encode($data));
});

Return format

{
	"btcusdt@depth": {
		"stream": "btcusdt@depth",
		"data": {
			"e": "depthUpdate",
			"E": 1608262880475,
			"s": "BTCUSDT",
			"U": 7203491669,
			"u": 7203491855,
			"b": [
				["22863.42000000", "0.00000000"],
				["22862.59000000", "0.00000000"],
			],
			"a": [
				["22862.54000000", "3.93987700"],
				["22862.60000000", "0.00000000"],
			]
		}
	},
	"bchusdt@depth": {
		"stream": "bchusdt@depth",
		"data": {
			"e": "depthUpdate",
			"E": 1608262880476,
			"s": "BCHUSDT",
			"U": 2070318220,
			"u": 2070318370,
			"b": [
				["313.36000000", "0.00000000"],
				["313.35000000", "0.00000000"],
			],
			"a": [
				["313.37000000", "0.00000000"],
				["313.38000000", "0.00000000"],
			]
		}
	},
	"yz5uhiFVLpIeCJxCzOVfQW3gbkxrTLUks8n0DoYmedx0qKcBmKlK4tWyuSwFvHgW===executionReport": {
		"e": "executionReport",
		"E": 1608262750894,
		"s": "ADAUSDT",
		"c": "NO160826275071922070",
		"S": "SELL",
		"o": "LIMIT",
		"f": "GTC",
		"q": "121.40000000",
		"p": "0.16472000",
		"P": "0.00000000",
		"F": "0.00000000",
		"g": -1,
		"C": "",
		"x": "NEW",
		"X": "NEW",
		"r": "NONE",
		"i": 652111413,
		"l": "0.00000000",
		"z": "0.00000000",
		"L": "0.00000000",
		"n": "0",
		"N": null,
		"T": 1608262750893,
		"t": -1,
		"I": 1348047474,
		"w": true,
		"m": false,
		"M": false,
		"O": 1608262750893,
		"Z": "0.00000000",
		"Y": "0.00000000",
		"Q": "0.00000000"
	},
	"yz5uhiFVLpIeCJxCzOVfQW3gbkxrTLUks8n0DoYmedx0qKcBmKlK4tWyuSwFvHgW===outboundAccountInfo": {
		"e": "outboundAccountInfo",
		"E": 1608262750894,
		"m": 10,
		"t": 10,
		"b": 0,
		"s": 0,
		"T": true,
		"W": true,
		"D": true,
		"u": 1608262750893,
		"B": [{
			"a": "BTC",
			"f": "0.02337993",
			"l": "0.00000000"
		}, {
			"a": "NEO",
			"f": "0.00459000",
			"l": "0.00000000"
		}, {
			"a": "BNB",
			"f": "0.00001719",
			"l": "0.00000000"
		}, {
			"a": "GAS",
			"f": "0.04920539",
			"l": "0.00000000"
		}, {
			"a": "USDT",
			"f": "0.00210115",
			"l": "0.00000000"
		}, {
			"a": "TRX",
			"f": "0.04910000",
			"l": "0.00000000"
		}, {
			"a": "ETC",
			"f": "1.67832000",
			"l": "0.00000000"
		}, {
			"a": "ADA",
			"f": "6298.78374000",
			"l": "121.40000000"
		}, {
			"a": "BCH",
			"f": "6.12146206",
			"l": "0.00000000"
		}],
		"P": ["SPOT"]
	},
	"yz5uhiFVLpIeCJxCzOVfQW3gbkxrTLUks8n0DoYmedx0qKcBmKlK4tWyuSwFvHgW===outboundAccountPosition": {
		"e": "outboundAccountPosition",
		"E": 1608262750894,
		"u": 1608262750893,
		"B": [{
			"a": "BNB",
			"f": "0.00001719",
			"l": "0.00000000"
		}, {
			"a": "USDT",
			"f": "0.00210115",
			"l": "0.00000000"
		}, {
			"a": "ADA",
			"f": "6298.78374000",
			"l": "121.40000000"
		}]
	}
}

[zhouaini528]

This method has bugs

$binance->getSubscribe([
    'btcusdt@depth',
    'bchusdt@depth',
]);

You use these two methods first

$data=$binance->getSubscribes();
//or
$binance->getSubscribes(function ($data){
	print_r($data);
});

[muhammadsaeedparacha]

@zhouaini528

$binance->subscribe(['xrpusdt@bookTicker']);
$binance->getSubscribes(function ($data){
	print_r($data);
}, true);

For some reason with bookTicker, getSubscribes() doesn't update the $data.
But when I use getSubscribe() then it constantly updates.

This works

$binance->subscribe(['xrpusdt@bookTicker']);
$binance->getSubscribe(['xrpusdt@bookTicker'], function ($data){
	print_r($data);
}, true);

The first one works with xrpusdt@depth, but not with xrpusdt@bookTicker :/

[muhammadsaeedparacha]

I'm working with Futures. Both public and private are successful. The problem is that some of the private channel messages are being lost. Even in the beginning. Like some trade updates go missing.
I'm using the getSubscribe() method as I mentioned because the getSubscribes() method has a bug that public channel updates do not work with <symbol>@bookTicker, however they do work with <symbol>@depth

[muhammadsaeedparacha]

I can change config to reduce data read time like:
From 0.1 to 0.01

$binance = new BinanceWebSocket();
  $binance->config([
      'data_time'=>0.01,
      'baseurl'=>'ws://fstream.binance.com',//usdt future
  ]);

But if private channel has a queue of 100, then I shouldn't have the problem of missed messages in the first place

[zhouaini528]

I reserved two methods as debug

//View buffer pool variables
$binance->client()->test();
//View all data in the buffer pool
$binance->client()->test2();

The data returned by this method is shown in the figure, it will help you by analyzing the data.

I tested @bookTicke successful. the SDK public channel will lose data, and the public channel must choose a full subscription.

I suggest to decompose the design and place all public channel data in some redis. Do not combine business logic with subscription logic, which will cause maintenance difficulties.
The PHP daemon may crash for a long time, I don’t know why, I’m looking for the problem

[zhouaini528]

Changing the configuration can indeed avoid losing the private channel data "data_time => 0.01". In the next version, I will put the maximum limit of 100 messages in the configuration.

[muhammadsaeedparacha]

Oh my bad. I was using an outdated version. I noticed in composer that I had dev version and upgraded it and it fixed the bookTicker problem. It was the from 15 days ago and I see you added queue 12 days ago as per your commit message so i wasn't supposed to have it. I'll recheck this update you.

Thank you so much for giving me this much time. Forever grateful. I'll try to help whatever I can with the library if I find any things I can improve

[muhammadsaeedparacha]

Yup Queues are working perfectly as well. I'm not missing any messages now. Thank you for your time. I'll close this conversation.

[zhouaini528]

Thank you for your evaluation, please feel free to ask me questions.