Redis gone away #22250
Comments
|
Some stats from redis |
|
Today night updated to 19.0.1.1 |
|
@nextcloud/server-triage can you, please, say something? |
|
For last two weeks there were no problem with 504 error. but once a day nextcloud misses css files and in logs there is The only place, where I fount server.css is ./theme/*/core/css folder. So, in my opinion, this problem closely connect with theming application. |
|
After some experiments, I fount solution, how to prevent the problem with but still have problems with lost css, described here #22250 (comment) The only way to solve it - flushdb in redis Guys, @kesselb, @rullzer, @MorrisJobke can you take a look, please? |
If that fixes it... not a nextcloud issue. Try help.nextcloud.com for server configuration questions. |
|
@dragon988 This IS nextcloud's problem. Read carefully. Again. The object exists in redis, I can take it with any custom command. But in the same time Nextcloud cannot. |
This information is not in any of your tickets. Please provide this information for troubleshooting. Describe in detail how nextcloud can't access the information but your custom command can. |
|
@krakazyabra If you ever find what creates the CSS problems, please let me know. |
|
After disabling Theming app, everything working fine. So, problem is 100% in Theming app and caching mechanism. |
|
Please provide the information previously requested regarding redis so we can narrow down the bug.
|
|
|
@skjnldsv @karakayasemi In my case (see the above link), CSS files were regenerated on every run because of a circular link, rendering the whole Nextcloud sluggish, including redis. |
|
Is this still the case with the latest maintenance releases? We had a bunch of SCSSCache related fixes lately with #23488 #23478 #23490 What seems odd to me is that disabling the theming app would fix it. The following log line actually looks interesting, do you maybe have a full one of the whole json output, as it seems to be cut of at the end: It seems like when copying the file from tmp to s3 that it somehow tries to fetch a part file: writeBack("/tmp/oc_tmp_lulwPS-.css","appdata_ocfl9qvwxaml/css/core/bcd6-aadb-server.css") |
|
Maybe you can just grep your logs for this: "Argument 1 passed to OC\Files\Cache\CacheQueryBuilder::whereFileId()" |
|
Is this Issue still valid in NC21.0.2? If not, please close this issue. Thanks! :) |
|
let's close this as there was no feedback for a long time. Feel free to reopen with the additionally requested information if it still occurs. |
How to use GitHub
Hi! Let me start from description of the problem. Several times a week Nextcloud is going down (504 gateway timeout, or loading the page, but without CSS). After lots of researches. we figured out, that the problem is somewhere on Nextcloud's side. It is closely connected with Redis.
After I saw 504 error or page without CSS, I performed a test: wrote simple script, which is connecting to current Redis, creating, putting, getting the key-value. The script is on the same machine, as NC. Nextcloud cannot connect to Redis, but script can. So, this is not Redis or network problem.
I tried different Redis configurations (single-master, classic master-slaves, sentinel with master-election), different versions, the result is the same - sometimes I see 504 or page without CSS.
In NC log there is only one message - redis server went away. Sorry now I don't have actual log, everything (except data) is not persistent in our environment. I will add logs, when new outage will happen. And I'm sure it will be soon.
In redis log - totally nothing.
There is no logic in when the problem occurs: it can happen in the night, when there are no online users, or during the day, when there is 40-50 online users.
For now, the only solution I fount - connect to redis and execute
flushallcommand. After this operation Nextcloud starts to work.In the same time we're using the same configuration in staging environment, but there are no active users. In staging there is no such problem. So it somehow connected with load. I could not reproduce this problem somewhere else. So, unfortunately, I cannot provide steps, how to reproduce it.
Steps to reproduce
Expected behaviour
Nextcloud is working :-)
Actual behaviour
Described upper
Server configuration
Operating system:
Debian 10 Buster
Web server:
nginx/1.19.1
Database:
Percona xtradb-cluster 5.7.19
PHP version:
PHP 7.3.20
Nextcloud version: (see Nextcloud admin page)
nextcloud:18.0.6-fpm
Updated from an older Nextcloud/ownCloud or fresh install:
Fresh
Where did you install Nextcloud from:
Docker image
Signing status:
I dunno why NC shows that this file has invalid hash. I compared my file with original - they are absolutely the same,
diff -stold me this.Signing status
List of activated apps:
App list
Nextcloud configuration:
Config report
Are you using external storage, if yes which one: local/smb/sftp/...
S3 as primary storage
Are you using encryption: yes/no
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
oAuth
Logs
Web server error log
Logs are from kibana, sorry for the format
Web server error log
Nextcloud log (data/nextcloud.log)
Nextcloud log
The text was updated successfully, but these errors were encountered: