Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(NFC) MenuXmlTest - Improve debug output. Use richer helpers. #28073

Merged
merged 2 commits into from
Nov 9, 2023

Conversation

totten
Copy link
Member

@totten totten commented Nov 9, 2023

Overview

In automated test-runs, the e2e test for mixin/menu-xml@1 (MenuXmlTest) has been failing sporadically. The failure always involves one of the HTTP sub-requests, but it's never clear what actually happened in the HTTP sub-request. This patch improves the debug info (so we might now how the sub-request failed).

Before

Send HTTP sub-request with simple primitives (cv url + file_get_contents). No detailed information about the failed request.

After

Send HTTP sub-request with HttpTestTrait (guzzle). Use rich assertions that log more detailed information.

For example, if I hack the test to provoke a failure, it looks like this:

1) E2E_Shimmy_LifecycleTest::testLifecycleWithSubprocesses
Expected HTTP response to indicate a failure (e.g. 404). Received HTTP response 200.
{
    "request": {
        "uri": "http://dmaster.127.0.0.1.nip.io:8001/civicrm/shimmy/foobar",
        "method": "GET",
        "headers": {
            "Host": "dmaster.127.0.0.1.nip.io:8001",
            "User-Agent": "GuzzleHttp/6.5.5 curl/7.83.1 PHP/7.4.29"
        },
        "body": ""
    },
    "response": {
        "status": "200 OK",
        "headers": {
            "Date": "Thu, 09 Nov 2023 00:22:53 GMT",
            "Server": "Apache/2.4.53 (Unix) OpenSSL/1.1.1o",
            "X-Powered-By": "PHP/8.1.19",
            "Expires": "Sun, 19 Nov 1978 05:00:00 GMT",
            "Cache-Control": "no-cache, must-revalidate",
            "X-Content-Type-Options": "nosniff",
            "Transfer-Encoding": "chunked",
            "Content-Type": "text/plain;charset=UTF-8"
        },
        "body": "hello world 1699489374.1819"
    }
}

/Users/totten/bknix/build/dmaster/web/sites/all/modules/civicrm/Civi/Test/HttpTestTrait.php:157
/Users/totten/bknix/build/dmaster/web/sites/default/files/civicrm/ext/example-mixin/tests/mixin/MenuXmlTest.php:30
/Users/totten/bknix/build/dmaster/web/sites/default/files/civicrm/ext/example-mixin/tests/phpunit/E2E/Shimmy/LifecycleTest.php:96
/Users/totten/bknix/build/dmaster/web/sites/default/files/civicrm/ext/example-mixin/tests/phpunit/E2E/Shimmy/LifecycleTest.php:73
/Users/totten/bknix/build/dmaster/web/sites/default/files/civicrm/ext/example-mixin/tests/phpunit/E2E/Shimmy/LifecycleTest.php:43
/Users/totten/bknix/extern/phpunit9/phpunit9.phar:2307

In automated test-runs, the e2e test for `mixin/menu-xml@1` (`MenuXmlTest`) has been failing sporadically.
The failure always involves one of the HTTP sub-requests, but it's never clear what actually happened
in the HTTP sub-request. This patch improves debug info.

Before
-------

Send HTTP sub-request with simple primitives (`cv` + `file_get_contents`).  No detailed information about the failed request.

After
-----

Send HTTP sub-request with `HttpTestTrait` (`guzzle`). Use rich assertions that log more detailed information.
Copy link

civibot bot commented Nov 9, 2023

🤖 Thank you for contributing to CiviCRM! ❤️ We will need to test and review this PR. 👷

Introduction for new contributors...
  • If this is your first PR, an admin will greenlight automated testing with the command ok to test or add to whitelist.
  • A series of tests will automatically run. You can see the results at the bottom of this page (if there are any problems, it will include a link to see what went wrong).
  • A demo site will be built where anyone can try out a version of CiviCRM that includes your changes.
  • If this process needs to be repeated, an admin will issue the command test this please to rerun tests and build a new demo site.
  • Before this PR can be merged, it needs to be reviewed. Please keep in mind that reviewers are volunteers, and their response time can vary from a few hours to a few weeks depending on their availability and their knowledge of this particular part of CiviCRM.
  • A great way to speed up this process is to "trade reviews" with someone - find an open PR that you feel able to review, and leave a comment like "I'm reviewing this now, could you please review mine?" (include a link to yours). You don't have to wait for a response to get started (and you don't have to stop at one!) the more you review, the faster this process goes for everyone 😄
  • To ensure that you are credited properly in the final release notes, please add yourself to contributor-key.yml
  • For more information about contributing, see CONTRIBUTING.md.
Quick links for reviewers...

➡️ Online demo of this PR 🔗

@civibot civibot bot added the master label Nov 9, 2023
@totten totten changed the base branch from master to 5.68 November 9, 2023 00:27
@civibot civibot bot added 5.68 and removed master labels Nov 9, 2023
@totten
Copy link
Member Author

totten commented Nov 9, 2023

The branch was written in a way that can be targeted at master or 5.68. I felt a little torn: we don't really know if there's any bug in the code (let alone a regression), but this sporadic test-failure is recent. The failure is also randomish (showing up on X% of matrix test-runs, spread across all versions), so it takes several runs in order to provoke the failure.

I switched it to 5.68 because this maximizes our ability to understand the failures, wherever they may happen to be (and because it doesn't change the actual behavior; it's runtime-NFC).

@seamuslee001 seamuslee001 merged commit 2068e7c into civicrm:5.68 Nov 9, 2023
@totten totten deleted the master568-http-log branch November 9, 2023 02:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants