Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(dev/core#2077) Make 'civicrm/ajax/rest' interoperable with 'extern/rest.php' parameters #19727

Merged
merged 9 commits into from
Feb 5, 2022

Conversation

totten
Copy link
Member

@totten totten commented Mar 4, 2021

Overview (updated)

This addresses a gap with APIv3 REST access on Drupal 8/9. Specifically, it updates an existing AJAX interface to satisfy the traditional REST contract.

The general approach involves two main changes:

  1. Define an authentication-flow called legacyrest. This looks for ?key=...&api_key=.... Accept this on civicrm/ajax/rest and civicrm/ajax/api4/*.
  2. Update the APIv3/APIv4 end-points to relax the requirement for setting X-Requested-With:. If a request has a strong/secret/custom input like ?api_key=, ?_authx=, or X-Civi-Auth:, then we do not need X-Requested-With:.

See also:

ping @seamuslee001 @MikeyMJCO

Before

If you have a remote app and wish to call APIv3 via REST, the call looks like this:

curl -X 'POST' -d 'entity=Contact&action=get&json=1&key=MY_SITE_KEY&api_key=MY_API_KEY' \
  'http://dmaster.127.0.0.1.nip.io:8001/sites/all/modules/civicrm/extern/rest.php' 

(* Note: URL points to the physical path of extern/rest.php *)

However, this is not generally available on D8/D9 because the script extern/rest.php is not available/accessible/bootable.

Alternatively, following #19590, one can call APIv3 like this:

curl -X 'POST' -d 'entity=Contact&action=get&json=1&_authx=Bearer+MY_API_KEY' \
  'http://dmaster.127.0.0.1.nip.io:8001/civicrm/ajax/rest 

(* Note: This uses the routing system *)

These two entry-points are based on the same controller (CRM_Utils_REST), so they mostly accept the same options (eg entity=Contact&action=get&json=1). However, they are not quite interoperable, so any external script/app/integration needs to tuned to match the different authentication parameters.

  • extern/rest.php only accepts credentials as the civicrm_contact.api_key (?api_key=...).
  • civicrm/ajax/rest accepts credentials via query (?_authx=...) or header (Authorization:). This may be API key, username/password, or JWT. However, it does not accept ?api_key.

After

The civicrm/ajax/rest can be used as a replacement for extern/rest.php. It accepts the authentication via ?api_key=....

curl -X 'POST' -d 'entity=Contact&action=get&json=1&key=MY_SITE_KEY&api_key=MY_API_KEY' \
  'http://dmaster.127.0.0.1.nip.io:8001/civicrm/ajax/rest 

The E2E REST tests have been expanded -- instead one of one test for extern/rest.php, we have:

  • E2E_Extern_BaseRestTest - Common set of assertions/test-cases describing the REST contract
  • E2E_Extern_LegacyRestTest - Runs the common assertions using extern/rest.php (for environments that support it)
  • E2E_ExternAuthxRestTest - Runs the common assertions using civicrm/ajax/rest+authx (for all environments)

Technical Details (updated)

  • extern/rest.php requires ?site_key as a way to ensure that this user is approved for REST access. civicrm/ajax/rest does not currently require this. However, there will be another PR momentarily to allow this requirement.
  • extern/rest.php accepts the APIv3 entity+action in two ways:
    • The well-known way (which appears in explorer/docs/etc) is to pass ?entity=...&action=.... This is supported by both.
    • The more obscure way (which doesn't seem to be publicly documented anymore) is to pass ?q=civicrm/{$entity}/{$action}. This notation is not supported on civicrm/ajax/rest and would be problematic. Fortunately, I don't think it's used much, so...
  • legacyrest uses two top-level HTTP params (?key=...&api_key=...). Of course, it's entirely possible that current and future routes have (or will have) their own uses for ?key and ?api_key. (This is difficult to check across-the-board.) To avoid conflicts, legacyrest will only activate civicrm/ajax/rest and civicrm/ajax/api4 -- because this doesn't produce a new conflict.

Here are a few examples that I used for my own testing:

cv en authx
echo 'update civicrm_contact set api_key = "MY_API_KEY" where id = MY_CID;' | cv sql

## Send request to `extern/rest.php` using legacy auth. No `X-Requested-With`. OK.
curl -X 'POST' -d 'entity=Contact&action=get&json=1&key=MY_SITE_KEY&api_key=MY_API_KEY' \
  'http://localhost/sites/all/modules/civicrm/extern/rest.php'

## Send request to `civicrm/ajax/rest` using legacy auth. No `X-Requested-With`. OK.
curl -X 'POST' -d 'entity=Contact&action=get&json=1&key=MY_SITE_KEY&api_key=MY_API_KEY' \
  'http://localhost/civicrm/ajax/rest'

## Send request to `civicrm/ajax/api4` using legacy auth. No `X-Requested-With`. OK.
curl -X POST -d 'key=MY_SITE_KEY&api_key=MY_API_KEY&params=%7B%22limit%22%3A25%2C%22debug%22%3Atrue%7D&index=display_name' \
  'http://localhost/civicrm/ajax/api4/Contact/get'

## Send request to `civicrm/ajax/api4` using `?_authx`. No `X-Requested-With`. OK.
curl -X POST -d '_authx=Bearer+MY_API_KEY&params=%7B%22limit%22%3A25%2C%22debug%22%3Atrue%7D&index=display_name' \
  'http://localhost/civicrm/ajax/api4/Contact/get'

## Send request to `civicrm/ajax/api4` using HTTP Basic auth. Requires `X-Requested-With:`.
## Requires enabling "pass(word)" support "header" auth.

cv ev 'Civi::settings()->set("authx_header_cred", ["jwt", "api_key", "pass"]);'

curl -X POST -d 'params=%7B%22limit%22%3A25%2C%22debug%22%3Atrue%7D&index=display_name' \
  'http://MY_USER:MY_PASS@localhost/civicrm/ajax/api4/Contact/get'

curl -X POST -d 'params=%7B%22limit%22%3A25%2C%22debug%22%3Atrue%7D&index=display_name' \
  -H 'X-Requested-With: XMLHttpRequest' \
  'http://MY_USER:MY_PASS@localhost/civicrm/ajax/api4/Contact/get'

@civibot
Copy link

civibot bot commented Mar 4, 2021

(Standard links)

@civibot civibot bot added the master label Mar 4, 2021
@homotechsual
Copy link
Contributor

The concept here looks great :-)

I like the idea of Header based authentication since it's much, much easier to work with :-)

I'll dive into some proper in-depth testing of this next week!

@homotechsual
Copy link
Contributor

This will need documentation updates to cover how the authx param works.

@seamuslee001
Copy link
Contributor

@totten test fails look related here

@eileenmcnaughton
Copy link
Contributor

test this please

@totten totten force-pushed the master-authx-rest branch 2 times, most recently from 0a616c3 to d458b6b Compare April 9, 2021 20:26
@eileenmcnaughton
Copy link
Contributor

Test Result (2 failures / ±0)
E2E_Extern_AuthxRestTest.testAPICalls with data set #1
E2E_Extern_AuthxRestTest.testGetCorrectUserBack

@totten
Copy link
Member Author

totten commented Apr 10, 2021

Yup, the test failures are related. Basically, the tests pass if the site has debug_enabled but fail on a normal site. The difference is CSRF enforcement. In normal mode, CSRF protections are active, and you need extra work send a request.

The trick is that we don't want to disable CSRF protection completely - but we do want a carve-out which allows REST-style authentication to be an adequate substitute. This requires some related work - e.g. next step/pre-req is #20026

@totten totten force-pushed the master-authx-rest branch from d458b6b to 1f13ee7 Compare April 13, 2021 02:38
@kainuk
Copy link
Contributor

kainuk commented Jun 16, 2021

Hi @totten, I just used this PR to make the rest endpoint on Drupal9 working. I had to make to changes in the code.

In the LegacyRestAuthenticator class the login function has the following signature:

protected function login(AuthenticatorTarget $tgt)

I added a hack in on line 41 of auth.php

$_SERVER['HTTP_X_REQUESTED_WITH'] = "XMLHttpRequest";

The ajax rest function checks if cal comes from Ajax. This line simulates this.

(and I had to replace the drupal 8 user_load function but that is already covered in another PR.

@jensschuppe
Copy link
Contributor

I added a hack in on line 41 of auth.php

$_SERVER['HTTP_X_REQUESTED_WITH'] = "XMLHttpRequest";

The ajax rest function checks if cal comes from Ajax. This line simulates this.

This seems a blocker to me for using the REST API on Drupal 8/9 environments with the AuthX extension, since many webhook-issuing applications don't provide configurable HTTP headers. I thought this would have been resolved with #20026, although @totten stated there:

(This patch does not address those example use-cases -- it merely retains data so that it's possible.)

@totten totten force-pushed the master-authx-rest branch 2 times, most recently from 0a38d14 to b8f084a Compare December 11, 2021 07:14
@totten
Copy link
Member Author

totten commented Dec 11, 2021

Rebased to address some merge-conflicts and to fix compatibility with phpunit8.

(@kainuk) In the LegacyRestAuthenticator class the login function has the following signature:

Oh, good point. Added that fix.

... $_SERVER['HTTP_X_REQUESTED_WITH'] = "XMLHttpRequest"; ...

(@jensschuppe) I thought this would have been resolved with #20026, although... [it's not...]

Quite right! I've added 69af731 + b8f084a to carve-out another way to get past the CSRF protection -- by supplying some valid secret as a bespoke param (ie &api_key=<SECRET>). And it does build on #200026.

The test now passes for me locally.

For r-running, I thought it was interesting to compare these requests:

$ curl -X 'POST' -d 'entity=Contact&action=get&json=1&key=ZZZZZZZZZZZ&api_key=ZZZZZZZZZZZ' 'http://dmaster.127.0.0.1.nip.io:8001/civicrm/ajax/rest'

{"is_error":0,"version":3,"count":25,"values":{"117":{"contact_id":"117",...

## This request is allowed because it gives a strong/valid secret `&key=...`.
## This wouldn't appear in a CSRF attacker; if the attacker could formulate a valid URL with `&key=...`, then
## wouldn't gain anything by going through CSRF. They're authorized to hit you directly.
$ curl -X 'POST' -d 'entity=Contact&action=get&json=1' 'http://USER:PASS@dmaster.127.0.0.1.nip.io:8001/civicrm/ajax/rest'

{"IP":"127.0.0.1","level":"security","referer":"","reason":"CSRF suspected","is_error":1,"error_message":"SECURITY ALERT: Ajax requests can only be issued by javascript clients, eg. CRM.api3()."}[bknix-min:~/bknix/build/dmaster/web/sites/all/modules/civicrm]

## This request is not sufficient to get past CSRF protection. While there is a secret PASS, the secret is conveyed
## by a standard `Authorization:` header. The standard `Authorization:` is conveyed implicitly with <A HREF>, <IMG>,
## and other CSRF vectors. 

🤞 Fingers crossed for the test to pass in CI as well.

@totten
Copy link
Member Author

totten commented Jan 25, 2022

Rebased to fix recent MCs. Tests pass. Updated labels (add has-test; remove needs-documentation).

Documentation links:

@colemanw
Copy link
Member

colemanw commented Feb 4, 2022

@totten I'm just waiting for you to patch the APIv4 endpoint and we'll get this merged.

_Overview_:  `civicrm/ajax/api4` and `returnJsonResponse()` inspect the
web-request to see if it comes via AJAX/REST. If so, the call is
allowed and formatted as JSON. The patch refines the test.

_Before_: `X-Requested-With:` signals that a call is AJAX/REST.

_After_: `X-Requested-With:` still signals that a call is AJAX/REST.
Additionally, if `authx` is enabled, then some requests will be treated as
AJAX/REST based on how they are authenticated (ie `xheader`/`X-Civi-Auth:`
and `param`/`?_authx=` are AJAX/REST).
@totten totten force-pushed the master-authx-rest branch from 24de1f1 to 8d5feab Compare February 5, 2022 00:29
@totten
Copy link
Member Author

totten commented Feb 5, 2022

@colemanw I've updated the PR and description in a couple ways:

  • Extend legacyrest (?key=...&api_key=...) support to APIv4. (I hadn't done this initially because I wasn't sure if it would cause conflict. But I looked closer, and I think they're compatible. I think it'll be easier to document - and more forgiving to use - if v3+v4 accept all the same auth mechanisms. However, legacyrest is still annoying to use b/c it requires site-key... so I'd still push people toward ?_authx= and X-Civi-Auth:)
  • Extend the X-Requested-With: changes to APIv4. Both v3+v4 will accept legacyrest (?key=...&api_key=...), xheader (X-Civi-Auth:), or param (?_authx=...) in lieu of X-Requested-With:.

@colemanw
Copy link
Member

colemanw commented Feb 5, 2022

Looks good @totten - let's merge this before the RC branch

@colemanw colemanw merged commit e859e64 into civicrm:master Feb 5, 2022
@totten totten deleted the master-authx-rest branch February 5, 2022 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants