https://markets.bisq.network/ is often down those last days

[ghost]

https://markets.bisq.network/ is often down those last days

We got complaints from users on the slack, and myself monitoring https://markets.bisq.network
get a lot of alert messages
2019-07-17 06:45:08 ; https://markets.bisq.network ; _ssl.c:704: The handshake operation timed out
2019-07-17 07:00:07 ; https://markets.bisq.network ; timed out

Maybe there is some memory leak ? need to restart ?

Ping @dan-da

[dan-da]

I have no access to server or github repo, so nothing I can really do about it.

Ping @Emzy.

just a hunch... It may be worth looking at the size of the json files (esp trades.json) generated by the bisq stats node and associated processing times. trades.json stores all-time history and the file is re-written (I believe) after every trade, or possibly on a timer. It then must be read, parsed, and cached by the PHP app. With enough cumulative trades, this could become a real bottleneck. I don't know if bisq is at that point yet, or it is something else.

An improvement would be to write daily or even hourly trade.json files, so that only the current file is written to and (re-) parsed by php app when a new trade happens. Or possibly even one file per trade.

edit: @Emzy has reported repeated occurrences of out-of-memory issues. If caused by apache/php, then the problem could possibly be related to the caching of trade related data, which uses PHP 7's built-in code cache. A suggestion would be to watch memory usage (in a dev/staging environment) with trade cache turned on vs turned off.

[Emzy]

What I do is just a "service apache2 restart"
This always fixes it. And there is no time pattern. It can again happen after weeks or hours.

[huey735]

So I'm dumb. Is it possible to set it up so that I can have a Raspberry Pi dedicated for the dumpStatistics but have Bisq running on a separate computer?

[dan-da]

@Emzy suggestion: Create a file opcache-status.php under docroot:

<?php
echo "\n" . date('c') . "\n"; print_r( opcache_get_status() );

then make a crontab entry every 10 minutes with this command:

wget --output-document=/tmp/opcache-fetch.log https://bisq.network/markets/opcache-status.php && cat /tmp/opcache-fetch.log >> /tmp/opcache-status.log

This will create a timestamped log file with detailed info about opcache memory usage. It should stabilize after initial cache warmup period. If mem use keeps increasing forever or cache_full becomes true, that's a problem.

Also, here's some info re opcache settings: https://www.saotn.org/optimize-php-opcache-configuration/

This is what a log entry should look like:

2019-07-17T08:14:03-07:00
Array
(
    [opcache_enabled] => 1
    [cache_full] => 
    [restart_pending] => 
    [restart_in_progress] => 
    [memory_usage] => Array
        (
            [used_memory] => 9133184
            [free_memory] => 57974864
            [wasted_memory] => 816
            [current_wasted_percentage] => 0.001215934753418
        )

    [interned_strings_usage] => Array
        (
            [buffer_size] => 4194304
            [used_memory] => 200904
            [free_memory] => 3993400
            [number_of_strings] => 4790
        )

    [opcache_statistics] => Array
        (
            [num_cached_scripts] => 1
            [num_cached_keys] => 1
            [max_cached_keys] => 3907
            [hits] => 5
            [start_time] => 1559147469
            [last_restart_time] => 0
            [oom_restarts] => 0
            [hash_restarts] => 0
            [manual_restarts] => 0
            [misses] => 2
            [blacklist_misses] => 0
            [blacklist_miss_ratio] => 0
            [opcache_hit_rate] => 71.428571428571
        )

    [scripts] => Array
        (
            [/home/websites/random/opcache-status.php] => Array
                (
                    [full_path] => /home/websites/random/opcache-status.php
                    [hits] => 0
                    [memory_consumption] => 1056
                    [last_used] => Wed Jul 17 08:14:03 2019
                    [last_used_timestamp] => 1563376443
                    [timestamp] => 1563376433
                )

        )

)

Also, as a bandaid, you might want to automate a graceful apache restart every 12 hours or so. But it would be good to generate this logfile first to see if anything bad happens with cache usage around the time that mem fills up.

[dan-da]

@huey735 it's possible, yes. provided rpi can handle the load. You could also run a copy of the markets website app on the rpi if you wanted, for a web view and/or read-only api.