Unable to send http requests after long time usage

[ferchor2003]

We have an application that places a lot of calls. We noticed that after many hours and in some servers (not always the same ones) the aricpp::HttpClient stops talking to Asterisk:

• New http requests are created with HttpClient::SendRequest but we do not see an error happening in the associated handler or any traffic going to the Asterisk server on Wireshark
• We have checked that Asterisk is alive and receiving requests using the browser on http://:8088/ari/asterisk/info on the same server
• netstat -a -o on the same machine does not show any hung sockets on the application server

The goal is to find out why new requests cannot be initiated.
Maybe the HttpClient::pending queue never goes empty and sockets are not closed? what else can be happening?

This is the first call the application makes when placing a call:

client->RawCmd(
	aricpp::Method::post,
	"/ari/bridges/" + id +
	"?type=" + type,
	[this, h(std::forward<CreationHandler>(h))](const boost::system::error_code & e, auto state, auto reason, auto body)

The CreationHandler function is not invoked.. not an error_code or a regular response. just silence.
The application has a timer that can detect this situation and abort placing the call, but at this point placing calls on this server is not possible. All our requests behave like this from this point in time on.

[daniele77]

Hi @ferchor2003 .
we have stress-tested aricpp on a Linux machine using the tool SIPp and never got a behavior like you're describing.
How many calls do you place every seconds?
Just be sure to use the latest version of boost (maybe they've fixed some bugs in boost::beast library).
You can also check the dimension of the pending queue does not grow too much: you can add a log like this:

void SendRequest(Method _method, std::string _url, ResponseHandler res, std::string body = {})
{
    // enqueue the new request
    pending.emplace(_method, std::move(_url), std::move(res), std::move(body));
    if (pending.size() > 100)
        std::cout << "WARNING. Pending queue size is " << pending.size();

please, let me know.

[ferchor2003]

Hi @daniele77
Each server places around 600-2000 calls per hour. That translates to 10-35 calls per minute. This is not our high volume season (we need to figure out this problem before that happens!).
The problem happens around once or twice a week on each server so it is difficult to point out what triggers it or how to replicate it in a development environment.
The solution is running on Windows server 2012 R2 on AWS extra-large. Using Visual Studio 2015 update 3 and boost 1.75.0 to compile.

I'm adding logging as you suggested as well as in the following places:

boost::beast::http::async_write(
            socket,
            request, 
            [this](boost::system::error_code e, std::size_t bytes_transferred)
            {
                boost::ignore_unused(bytes_transferred);
				if (e)
				{
					std::ostringstream os;
					os << "async_write failed: " << e.message() << '\n';
					std::cout << os.str();

and

boost::beast::http::async_read(
            socket,
            buffer,
            resp,
            [this](boost::system::error_code e, std::size_t bytes_transferred)
            {
                boost::ignore_unused(bytes_transferred);
#ifdef ARICPP_HTTP_TIMEOUT
                timer.cancel();
#endif
				if (e)
				{
					std::ostringstream os;
					os << "async_read failed: " << e.message() << '\n';
					std::cout << os.str();
					CallBack(e);

I'll keep updating this post with more info. Thanks for your help.

[daniele77]

Well, to tell the truth, 36 calls per minute should not represent an unbearable load for the library.
However, we never load tested aricpp on windows servers. While under Linux we did all sort of tests (load, stress, correctness, and so on), on windows we only checked for correctness.

It could be useful understand if the ARI request is blocked inside aricpp library or inside beast / asio or at OS level.
Maybe you could add a trace just before the statement:

boost::beast::http::async_write

to check whether aricpp asks beast to send a message or not.

[ferchor2003]

I agree. I have also tested my application with a sustained rate of 150 simultaneous calls for 20 minutes. That helped me find a few bugs in my code and added a lot of confidence to run on production environment.
The problem we see however shows up sporadically. Still trying to figure out what triggers it.
We have added a mechanism to recreate aricpp::Client when connection to the server is lost (checking websocket) which also happens on occasion, but in this specific case we can ping the websocket and we see the asterisk server is receiving http requests;
We're considering ideas like checking how many sockets are open or trying to customize socket timeouts behavior on Windows.