feat: `$removeparam` #4528

seia-soto · 2024-12-13T07:11:28Z

This adds experimental $removeparam support for adblocker: https://github.com/gorhill/ublock/wiki/static-filter-syntax#removeparam

The suggested matching process changed by this PR is illustrated as follows:

$important (not subject to exceptions)
redirection ($removeparam and $redirect=resource)
normal filters
exceptions

The removeparam filter priorities are as the followings, and also described in the exception test as a code:

Details

s1) global removal + exception => global removal wins

@@||example.com$removeparam=utm
||example.com$removeparam

result.filter=global removal
result.exception=(none)

s2) removal + global exception => exception wins

||example.com$removeparam=utm
@@||example.com$removeparam

result.filter=removal
result.exception=global exception

s3) removal + exception => exception wins

||example.com$removeparam=utm
@@||example.com$removeparam=utm

result.filter=removal
result.exception=exception

global exception
global removal
exception
removal

TODO

Parsing regex as value (in separate PR with refactor and clean up of regex util funcs)

chrmod

https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/declarativeNetRequest/URLTransform

seia-soto · 2024-12-23T09:08:18Z

Note: we're going to apply filters one by one

seia-soto · 2024-12-25T10:29:05Z

Leaving a note: regexp wrapping: [\\?&](${value!.slice(1, flagSeparatorIndex)}[^=]*)

test: add `isRemoveParam` and `isRedirectable` chore: reject on `removeparam` negation test: parsing removeparam test: removeparam test: removeparam

packages/adblocker/src/filters/network.ts

packages/adblocker/test/parsing.test.ts

packages/adblocker/src/engine/engine.ts

refs ghostery#4528 (comment)

chrmod

Does this support exceptions like $removeparam=igshid,domain=instagram.com|threads.net?

packages/adblocker/src/filters/network.ts

packages/adblocker/test/engine/engine.test.ts

packages/adblocker/src/engine/engine.ts

* s1) global removal + exception => global removal wins ``` @@||example.com$removeparam=utm ||example.com$removeparam ``` result.filter=global removal result.exception=(none) * s2) removal + global exception => exception wins ``` ||example.com$removeparam=utm @@||example.com$removeparam ``` result.filter=removal result.exception=global exception * s3) removal + exception => exception wins ``` ||example.com$removeparam=utm @@||example.com$removeparam=utm ``` result.filter=removal result.exception=exception --- 0. global exception 1. global removal 2. exception 3. removal

We don't know what operations will be done in the meantime but we know that all operations are safe at the end.

packages/adblocker/test/engine/engine.test.ts

seia-soto · 2025-01-23T07:50:50Z

packages/adblocker/src/filters/network.ts

@@ -1151,6 +1162,8 @@ export default class NetworkFilter implements IFilter {
      }
    }

+    mask >>>= 0;


@chrmod @remusao Should we move this operator to constructor or another place? We need to make sure that the mask to be uint32 before it get serialized for safety. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Unsigned_right_shift_assignment

Problematic part: https://github.com/ghostery/adblocker/pull/4528/files#diff-f45592ce8a6bd8a6e8e39391572f4a38406614b3649a6628891ca6fc38a516a9R981-R987

@seia-soto can this be removed now?

chrmod

Lets move out the bit operations safety out of this PR. Besides, it looks good imo.

seia-soto · 2025-01-31T10:24:03Z

Waiting for #4634

remusao

The AdGuard documentation mentions:

Please note that such rules are only applied to GET, HEAD, OPTIONS, and sometimes POST requests.

Where do we intend to perform those checks?

remusao · 2025-02-08T20:15:20Z

packages/adblocker/src/engine/engine.ts

+          if (filter.isRemoveParam()) {
+            redirects.push(filter);


Why do we store removeparam as redirect rules? Aren't they of different nature?

remusao · 2025-02-08T20:18:57Z

packages/adblocker/src/engine/engine.ts

+        for (const filter of redirectableFilters) {
+          if (filter.isRemoveParam()) {
+            if (filter.isException()) {
+              removeparamExceptionFilters.set(filter.removeparam!, filter);


I think it might be possible to inform the type checker from isRemoveParam that filter.removeparam is not undefined/null.

remusao · 2025-02-08T20:19:35Z

packages/adblocker/src/engine/engine.ts

+            if (filter.isException()) {
+              removeparamExceptionFilters.set(filter.removeparam!, filter);
+            } else {
+              removeparamFilters.set(filter.removeparam!, filter);


Is it safe to use removeparam as a key in a Set? What about other possible constraints like 1p/3p, or request type such as script/xhr/etc.

remusao · 2025-02-08T20:20:57Z

packages/adblocker/src/engine/engine.ts

+        // * If redirect=none is found, then cancel all redirects.
+        // * Else if redirect-rule is found, only redirect if request would be blocked.
+        // * Else if redirect is found, redirect.
+        if (result.filter === undefined && redirectFilters.length !== 0) {


What is the rational behind giving priority to removeparam over redirects? Wouldn't it make sense to do it the other way around? A redirect rule is a block rule + a redirect rule (as per uBO semantics). Hence if a request is blocked it should have priority over keeping the request and remove some of the parameters.

remusao · 2025-02-08T20:25:34Z

packages/adblocker/src/filters/network.ts

@@ -888,6 +889,16 @@ export default class NetworkFilter implements IFilter {
            mask = setBit(mask, NETWORK_FILTER_MASK.isReplace);
            optionValue = value;

+            break;
+          case 'removeparam':
+            // TODO: Support regex


How about ~ negation supported by AdGuard? I see it might be handled below but maybe worth documenting the limitation (and having a unit test for each)

remusao · 2025-02-08T20:33:56Z

packages/adblocker/src/engine/engine.ts

+              }
+
+              if (redirectUrl.searchParams.has(key)) {
+                result.filter = filter;


That seems invalid in case there is more than one removeparam filter applying to the URL. I also think that in terms of semantics, the result.filter means a blocking filter, but removeparam filters aren't blocking requests. Maybe would it make more sense to add an additional key on result with a list of removeparams (or if we want to generalize, a list of URL/request modifiers)

remusao · 2025-02-08T20:36:09Z

packages/adblocker/test/engine/engine.test.ts

+        for (let i = 0; i < params.length; i++) {
+          const { redirect } = engine.match(request);
+          expect(redirect).not.to.be.undefined;
+          request = urlToDocumentRequest(redirect!.dataUrl);
+        }


It is surprising to me that we need to call the .match(...) multiple times to remove all params. Shouldn't a single call to .match(...) remove all matching params in a single pass?

We didn't know how to return the filters matched since the return type allows only one filter to be returned. That means not the all of filters matched will be sent to the return value of match unless we change the shape.

remusao · 2025-02-08T20:38:45Z

packages/adblocker/src/engine/engine.ts

+            result.redirect = {
+              body: '',
+              contentType: 'text/plain',
+              dataUrl: redirectUrl!.toString(),
+            };


It seems a bit counter-intuitive to me to reuse the dataUrl in order to express the removeparam result. Why not return a list of URL modifiers to be interpreted by the client code and adapted based on available capability of the platform? (iOS, Manifest v2, Manifest v3, might not have the best way to modify request parameters) Ideally the adblocker library should not need to think about these considerations and return a more "abstract" kind of response.

remusao · 2025-02-08T20:41:02Z

packages/adblocker/test/engine/engine.test.ts

Can we add more tests for corner cases? For example: if we have a global $removeparam and also @@||example.org$removeparam=utm, do we remove all parameters or skip utm?

remusao · 2025-02-08T20:41:54Z

packages/adblocker/test/parsing.test.ts

Let's add a test case to cover the regexp removeparam case (as being unsupported for now)

feat: $removeparam

94f5868

seia-soto added the PR: New Feature 🚀 Increment minor version when merged label Dec 13, 2024

seia-soto self-assigned this Dec 13, 2024

seia-soto added 2 commits December 13, 2024 16:18

fix: check parametersRemoved to prevent infinite loop

29e9c39

fix: parsing and restoration

3ec1181

seia-soto added the WIP label Dec 13, 2024

feat: removeparam without option value

fd43ad1

chrmod reviewed Dec 23, 2024

View reviewed changes

seia-soto added 2 commits December 25, 2024 16:33

refactor: drop use of URLSearchParams

9497047

feat: introduce NetworkFilter.prototype.isRedirectable

56f90cc

seia-soto removed the WIP label Dec 25, 2024

seia-soto added 4 commits December 25, 2024 19:37

refactor: improve matching search param key

47b1302

test: removeparam

7ea4488

test: add `isRemoveParam` and `isRedirectable` chore: reject on `removeparam` negation test: parsing removeparam test: removeparam test: removeparam

test: removeparam removes parameter sequentially

29e9022

fix(test): unintentional isRedirectable definition

3cb0861

seia-soto marked this pull request as ready for review December 27, 2024 11:03

seia-soto requested a review from remusao as a code owner December 27, 2024 11:03

seia-soto requested a review from chrmod December 27, 2024 11:03

remusao reviewed Jan 4, 2025

View reviewed changes

seia-soto added 2 commits January 6, 2025 14:58

refactor: use URL constructor

9c1e12b

refs ghostery#4528 (comment)

refactor: clarify removeparam getter

c8da3bb

seia-soto requested a review from remusao January 6, 2025 06:02

chrmod reviewed Jan 8, 2025

View reviewed changes

packages/adblocker/src/filters/network.ts Show resolved Hide resolved

chrmod reviewed Jan 13, 2025

View reviewed changes

packages/adblocker/test/engine/engine.test.ts Outdated Show resolved Hide resolved

packages/adblocker/test/engine/engine.test.ts Show resolved Hide resolved

chrmod reviewed Jan 13, 2025

View reviewed changes

packages/adblocker/src/engine/engine.ts Outdated Show resolved Hide resolved

seia-soto added 3 commits January 14, 2025 19:30

refactor: move heavy calls to before

533110e

fix(test): invalid scopes

1c07025

refactor: remove url transforms outside of before hook

90d051b

seia-soto added 2 commits January 16, 2025 16:28

fix(test): make global removeparam work with malformed urls

c1db908

seia-soto requested a review from chrmod January 16, 2025 08:10

seia-soto added 5 commits January 16, 2025 17:11

fix(test): correct bucket location

d48fe63

fix: convert to unsigned int on set

88f35a1

fix: convert to unsigned int on bit clear

160febd

fix: modify mask safely when applying cpt mask

4040a89

fix: make mask unsigned at the end of parse

90ce5c2

We don't know what operations will be done in the meantime but we know that all operations are safe at the end.

seia-soto commented Jan 23, 2025

View reviewed changes

chrmod requested changes Jan 30, 2025

View reviewed changes

seia-soto mentioned this pull request Jan 31, 2025

fix: keep integrity in network filter mask #4634

Merged

chore: drop unnecessary operation

da507f4

seia-soto marked this pull request as draft January 31, 2025 10:24

Merge branch 'master' into removeparam

3c6236b

chrmod marked this pull request as ready for review February 6, 2025 16:53

remusao requested changes Feb 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `$removeparam` #4528

feat: `$removeparam` #4528

seia-soto commented Dec 13, 2024 •

edited

Loading

chrmod left a comment

seia-soto commented Dec 23, 2024

seia-soto commented Dec 25, 2024

chrmod left a comment

seia-soto Jan 23, 2025

seia-soto Jan 27, 2025

chrmod Feb 6, 2025

chrmod left a comment

seia-soto commented Jan 31, 2025

remusao left a comment

remusao Feb 8, 2025

remusao Feb 8, 2025

remusao Feb 8, 2025

remusao Feb 8, 2025

remusao Feb 8, 2025

remusao Feb 8, 2025

remusao Feb 8, 2025

seia-soto Feb 8, 2025

remusao Feb 8, 2025

remusao Feb 8, 2025

remusao Feb 8, 2025

feat: $removeparam #4528

Are you sure you want to change the base?

feat: $removeparam #4528

Conversation

seia-soto commented Dec 13, 2024 • edited Loading

chrmod left a comment

Choose a reason for hiding this comment

seia-soto commented Dec 23, 2024

seia-soto commented Dec 25, 2024

chrmod left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrmod left a comment

Choose a reason for hiding this comment

seia-soto commented Jan 31, 2025

remusao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feat: `$removeparam` #4528

feat: `$removeparam` #4528

seia-soto commented Dec 13, 2024 •

edited

Loading