Custom filter lists sometimes won't update + an exception throwed in a browser while pressing update button in uBO

[kulfoon]

Prerequisites

• I verified that this is not a filter issue
  • Filter issues MUST be reported at filter issue tracker
• This is not a support issue or a question
  • Support issues and questions are handled at /r/uBlockOrigin
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • Your issue may already be reported.
• I tried to reproduce the issue when...
  • uBlock Origin is the only extension
  • uBlock Origin with default lists/settings
  • using a new, unmodified browser profile
• I am running the latest version of uBlock Origin
• I checked the documentation to understand that the issue I report is not a normal behavior

Description

Custom filter lists sometimes won't update, the issue has been happening to me since several months already, but I wasn't sure whether it was uBO's issue, and I was hoping somehow that it was just a temporary issue, which might be fixed somehow in uBO or in a browser as the time passes, but it hasn't happened, and the issue is getting annoying more and more, so I decided to create an issue. I'm still not sure whether it's uBO's, a browser's or an user's issue, but knowing that it occurs in two different browsers (Firefox, Opera) (and in Firefox in two different profiles, which one of them is reasonablly fresh) it seems to be not a browser's issue. Also this guy krystian3w has the same issue, we both have talked in the past about it, I don;t remember exactly, but he was saying something about that to force a successfulludpate of the list in this case is to add some string at the end of the list path name and visit it in a browser, something like a question mark followed by a some number like this ?_=9 which is visible in the uBO logger while updating a filter list, it sometimes works but not always, it seems to not work anymore for me. Unfortunatelly this guy will not reply here in this thread to help us because I had to put him on my blocklist due to his ugly behaviour so he can't reply to my threads anymore.

A specific URL where the issue occurs

An example local list file path:
http://localhost/local_filters_lists_uBO/my_example_list_my_example_list.txt​

Steps to Reproduce

1. Add an example local list with some example filters in it (to do it in Firefox, you need to setup a virtual server first to use local files in browser).
2. Edit the list and add or remove some filters / domains or even a comment.
3. Click the list update icon - a clock icon at the end of the list name.
4. Click the main Update button in uBO.

Expected behavior:

Custom filter lists should be always updated.

Actual behavior:

Custom filter lists won't (SOMETIMES! - hard to reproduce it always! sometimes a filter list will be updated with no problems) update in uBO, despite the fact that the physically txt file has been modified and saved, uBO still showing the old content of the txt file. Repeating the update process: reediting / resaving the txt file and / or clicking update buttons in uBO multiple times or clearing the uBO cache, or hard reloading the page with filter list content bypassing the cache, or restarting a browser or a virtual server in Firefox, gives no result. After some random time, sometimes minutes, sometimes hours, the local filter list will finally update itself with no human input, just doing some random stuff on PC, not always internet related, and then randomly coming back to uBO to see that the list has finally updated itself as the time has passed.

In Firefox, in the browser console I noticed an strange exception which is being throwed exactly at the time I hit main Update button in uBO, not sure whether it's related and whether it's uBO's, a browser's or an user side's issue, (couldn't find a similiar browser's console in Opera, to see what it will say, and a normal Opera's (or Firefox's) console has throwed nothing) (also I don't remember whether the exception was occuring in older versions of uBO and the browser, than the current ones I'm using):

Handler function threw an exception: [Exception... "Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsICacheInfoChannel.isRacing]"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: resource://devtools/server/actors/network-monitor/network-response-listener.js :: NetworkResponseListener.prototype._getSecurityInfo< :: line 301"  data: no]
Stack: NetworkResponseListener.prototype._getSecurityInfo<@resource://devtools/server/actors/network-monitor/network-response-listener.js:301:26
exports.makeInfallible/<@resource://devtools/shared/ThreadSafeDevToolsUtils.js:109:22
onStartRequest@resource://devtools/server/actors/network-monitor/network-response-listener.js:209:10
Line: 301, column: 0 ThreadSafeDevToolsUtils.js:88:13

Your environment

• uBlock Origin version: uBlock Origin 1.21.5b1 / 1.21.7b0 and earlier versions from last several months
• Browser Name and version: Firefox 68.0.1 64bit / Opera 62.0.3331.72 64bit and earlier versions from last several months
• Operating System and version: Win 7 Pro SP1 64bit

[gwarser]

Does not look like issue. uBO downloads list from browser cache. If you wait one hour it should change this ?_=9 number at the end of url, and update fresh from server.

[gwarser]

About exception: ThreadSafeDevToolsUtils.js this looks like Developer Tools internal code, not related to uBO or normal browsing without DevTools opened.

[gwarser]

Feature :)

https://github.com/gorhill/uBlock/blob/869c63f82115f27ba9f8af939522d12fa3bd82fb/src/js/assets.js#L69

[gwarser]

I tried http://0.0.0.0:8080/my-filters.txt with Python server python3 -m http.server 8080 and cannot reproduce. Probably because lack of any caching headers:

[uBlock-user]

Browser is throwing an exception which isn't caught. Similar Firefox bugs -- https://bugzilla.mozilla.org/buglist.cgi?quicksearch=0x80040111&list_id=14820451

[gwarser]

@kulfoon What you use as a server? Can you enable XHR in browser console and check what is happening with this network request?

[uBlock-user]

This is how it looks like on Firefox on my end --

On updating the list --

[kulfoon]

gwarser : What you use as a server?

I think that it doesn't matter as the issue occurs on chrome-based browsers too and on the chrome based browsers there is no need to use a server, thus I don't use a server there.

Currently I can't reproduce as it is annoying kind of a random issue, but I'm sure it's a matter of time when I'll reproduce as the issue occurs during the whole last several months, thus when I reproduce I'll tell you then what Network tab shows, maybe it's like in #682 (comment), not an issue but a feature, will see.

[gorhill]

occurs on chrome-based browsers too

What errors do you get in uBO's dev console with Chromium-based browsers?

[kulfoon]

As for "Custom filter lists sometimes won't update" issue I don't get any errors in dev console neither in Firefox nor in Opera, nor I get any errors in uBO logger, only in Firefox's browser console I get the exception, but as it was pointed out it's most probably not related.

I was able to reproduce "Custom filter lists sometimes won't update" issue now and it seems it has to be what gwarser said in #682 (comment), a feature and not an issue.

When the issue occured now first time:

1. the xhr was labeled in "Sent" column as "in cache memory",
2. after I updated again, filter list content got updated and there was no xhr entry,

I changed the content again and updated the filter list and the issue occured the 2nd time:

3. the xhr was labeled in "Sent" column as "in cache memory",
4. after I updated again, filter list content got updated with an xhr entry in columnd "Sent" labeled as "plain"

So it seems a feature and not an issue, I just wonder:

a) after updating the modified filter list second time, in the both reproduced cases above I didn't need to wait an hour to get the list freshly updated (not from cache), I just updated the 2nd time instantly after the 1st time, so what about this "an hour" timer ? On the other hand, also I remember in the past sometimes clicking update list multiple times and it refused to update the list, now it updates just after 2nd try. So what's going on with all these random update time fluctuations ?

b) where is one of the XHR request Nr 2, as seen on the screenshot there are only 3 XHRs, but should be 4, any idea? But never mind.

[gwarser]

a) after updating the modified filter list second time, in the both reproduced cases above I didn't need to wait an hour to get the list freshly updated (not from cache), I just updated the 2nd time instantly after the 1st time, so what about this "an hour" timer ?

This "feature" (gorhill/uBlock#2592) fixes cashing issue causing Fanboy lists not updating for years (potentially), that's why it was set to one hour.

On your side browser may cache your list only for few minutes and you will see it as not updated / not working properly.

Side note: "Refresh" button on browser toolbar does not reload pages from network if you click it only once (without modifier keys). But if you click it few times it does hard reload. It should work in similar way with these XHR's.

[uBlock-user]

The hour timer is not a fixed value as per my experience. If your list doesn't get updated the first time, waiting few minutes and updating again will do.

[gwarser]

Search parameter is incremented every hour - ?_=9. This ensures URL is new/not cashed yet, so it should be downloaded from server, not from any cache. If it works after few minutes without parameter change, this means browser cache gave up.

[kulfoon]

Ok, I get it now, the issue was a bit annoying for me becasue it disrupts developing a local filter list, because it makes me unable to test my new freshly added filters, so I'll have to change my method of developing a local list and testing new filters, I will have to copy particular filters from a local list to My Filters pane and then do all the modifications and tests, and then copy-back to a local filter list, but this is not too safe, as during the copy-back some mistake might happen, and the final result of copy-paste might be buggy / different, it's always best to test on a final place/file to make sure that the entered data was not affected during entering, the way of developing I will have to stick to currently, might lead to a buggy filter list....

So I want to close this thread.

[gorhill]

I will try to come up with an advanced setting to force an unconditional bypass of the cache when fetching a filter list.

[uBlock-user]

so will it always work at the first attempt with the advanced setting ?

[kulfoon]

Oh, it will be a good idea gorhill.
uBlock-user : yes - I think that's exactly what gorhill means.

[kulfoon]

I confirm - working, very useful, finally no problems with instant testing freshly updated filters lists.

[kulfoon]

After X days working with no problem (updating instantly at the time I force and hit the update button), since several days it sometimes won't update again despite the fact "marking the clock icon and then pressing the update now button", even repeating the same X times, and nothing, then after like 5 minutes or repeating the update process it finally will update, also another case - at the moment just right now I've tried to update uBlock filters like 10 times and nothing, I had to disable the list and then press the accept button, and then enable the list back and press the accept button and it finally have updated, so I'm confused, is such behaviour a feature or a bug?

[gorhill]

What list was this? Did you investigate on your side first to ensure that uBO is really fetching with a different URL every time? Did you look at the network requests in dev tools and look at the server response?

[kulfoon]

What list was this?

"uBlock filters" (the main uBlock list) if u asking me for the today case. As for the previous cases from previous several days, I don't remember the list names.

Did you investigate on your side first to ensure that uBO is really fetching with a different URL every time? Did you look at the network requests in dev tools and look at the server response?

I did not yet, because the issue is always random, especially today, I was dealing with this uBlockOrigin/uAssets#6175, I had the anti-adblock message, so I updated all the lists as mapx suggested, still the anti-adblock message was appearing, I was kinda annoyed what's going on, so I was keep updating all the list several times, and still nothing, then I was disabling all the list one by one, still was having the anti-adblock message with some single filter list, then I was enabling back all the list ony by one, and at the time I enabled back all the lists, the anti-adblock was gone, and I saw uBlock filters list got updated (number of filters decerased from like over 19100 to 19 064​, (the SECOND number with count of the total filters, not the first number which might change sometimes despite the list being enabled / disabled.) I hate random issues, but I'll try to investigate. I have uBO 1.21.9b3, but the issue was happening last days with some earlier versions of uBo deta / dev, so I'm not sure when it exactly started. Also when it was happening last days, I thought it was maybe just some occasional glitch caused by hell knows what so I didn't bother, but now I see the issue keeps repeating from sometimes to sometimes.

[gorhill]

I had the anti-adblock message, so I updated all the lists as mapx suggested, still the anti-adblock message was appearing

There might be a good delay before GitHub serve an updated version after a commit. If uBO sends a different URL every time and GitHub still serve the old list, there is nothing uBO can do.

[gorhill]

I need to fix this again, going through some code review I realized I have introduced a regression in this commit which remove all ability to bypass browser cache.