Avoid the test-data plugin occasionally missing in the E2E test environment

[eason9487]

Changes proposed in this Pull Request:

Closes #2175

💡 The issue probably only happens on MacOS. Usually I can reproduce it after running the E2E tests 1-3 rounds.

Although I'm not fully confident this PR can fix it, after some time and validation, it seems to be working so far. The solution was mentioned by @mikkamp in #2080 (comment).

This PR:

• Fix the indentation for the .wp-env.json file first as it was mixed with spaces and tabs.
• Make the test-data.php plugin have a dedicated directory and change to use the plugins directive to set it up.

Detailed test instructions:

💡 Considering the issue seems to only happen on MacOS, it would be more appropriate to have someone who uses the same OS or can reproduce the issue to review and test it.

1. Running npm run wp-env destroy should not be necessary but it would be ideal to have a clean start to test it.
2. Run npm run wp-env:up to see if the E2E test env can be started without errors.
3. Log in http://localhost:8889/wp-admin/ and go to the Installed Plugins page.
4. Check if the GLA Test Data plugin is activated.
5. Run npm run test:e2e to see if all E2E tests can pass.
6. Refresh the Installed Plugins page to see if the GLA Test Data plugin is activated.
7. Repeat steps 5-6 several times to verify if the plugin missing issue no longer occurs.

Additional details:

📌 Investigation

Although I didn't find the exact cause, the process that led to the issue is roughly as follows.

1. From the docker-compose.yml generated by wp-env, it's shown there are nested mounts under the /var/www/html directory.

   version: '3.7'
   services:
     # ... omitted
     tests-wordpress:
       # ... omitted
       volumes:
         - '/Users/eason/.wp-env/fa3f9473ab3c866d9e18213e3637074d/tests-WordPress:/var/www/html'
         - '/Users/eason/.wp-env/fa3f9473ab3c866d9e18213e3637074d/tests-WordPress-PHPUnit/tests/phpunit:/wordpress-phpunit'
         - 'tests-user-home:/home/eason'
         - '/Users/eason/wp-local-docker-sites/wp-test/wordpress/wp-content/plugins/google-listings-and-ads/tests/e2e/config/wp-cli.yml:/var/www/html/wp-cli.yml'
         - '/Users/eason/wp-local-docker-sites/wp-test/wordpress/wp-content/plugins/google-listings-and-ads/tests/e2e/bin/test-data.php:/var/www/html/wp-content/plugins/test-data.php'
         - '/Users/eason/.wp-env/fa3f9473ab3c866d9e18213e3637074d/master:/var/www/html/wp-content/plugins/master'
         - '/Users/eason/wp-local-docker-sites/wp-test/wordpress/wp-content/plugins/google-listings-and-ads:/var/www/html/wp-content/plugins/google-listings-and-ads'
   

2. In the container, the test-data.php is mounted in the beginning.

   root@969c943abe08:/var/www/html# mount | grep www
   grpcfuse on /var/www/html type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
   grpcfuse on /var/www/html/wp-cli.yml type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
   grpcfuse on /var/www/html/wp-content/plugins/test-data.php type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
   grpcfuse on /var/www/html/wp-content/plugins/google-listings-and-ads type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
   grpcfuse on /var/www/html/wp-content/plugins/master type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)

   In the container, its file size is 3.0K.

   root@969c943abe08:/var/www/html# ls -lh /var/www/html/wp-content/plugins/test-data.php
   -rw-r--r-- 1 eason dialout 3.0K Feb 29 07:02 /var/www/html/wp-content/plugins/test-data.php

3. Due to the nested mounts, a test-data.php file is created back to the host side with 0 file size.
   
4. After a few runs of E2E test, somehow the test-data.php is unmounted.

   root@969c943abe08:/var/www/html# mount | grep www
   grpcfuse on /var/www/html type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
   grpcfuse on /var/www/html/wp-content/plugins/google-listings-and-ads type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
   grpcfuse on /var/www/html/wp-content/plugins/master type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)

   In the container, its file size is 0 now. I think the empty content file is synced from the host.
   (See the screenshot in step 3)

   root@969c943abe08:/var/www/html# ls -lh /var/www/html/wp-content/plugins/test-data.php
   -rwxr-xr-x 1 eason dialout 0 Feb 29 07:06 /var/www/html/wp-content/plugins/test-data.php

5. Now the issue occurs.
   • The Installed Plugins page
     
   • E2E test always failed
     

I couldn't find the same issue report, but I suspect the reason for this is similar to these past issues:

• Volume mounted inside another sometimes disappears after file change docker/compose#6975
• Nested mounts disappear docker/for-mac#3479
• If you touch a submount point on the host the volume is unmounted in the container docker/for-mac#3976
• Volume get detached after error with osxfs notifications docker/for-mac#2777

📌 About the wp-cli.yml file

Please note that the issue also happens on the wp-cli.yml file. If I understood it right, it's a one-time setup during initialization and won't break when the wp-cli.yml file becomes unavailable. Therefore, this PR doesn't adjust it.

google-listings-and-ads/tests/e2e/bin/test-env-setup.sh

Lines 9 to 10 in e6c03c1

	echo -e 'Update URL structure \n'
	wp-env run tests-cli -- wp rewrite structure '/%postname%/' --hard

Changelog entry

Dev - Avoid the test-data plugin occasionally missing in the E2E test environment

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.9%. Comparing base (a6e8a4c) to head (e6c03c1).

Additional details and impacted files

@@             Coverage Diff             @@
##             develop   #2286     +/-   ##
===========================================
+ Coverage       62.2%   63.9%   +1.7%     
  Complexity      4190    4190             
===========================================
  Files            748     455    -293     
  Lines          21547   17832   -3715     
  Branches         532       0    -532     
===========================================
- Hits           13406   11400   -2006     
+ Misses          7690    6432   -1258     
+ Partials         451       0    -451     

Flag	Coverage Δ
js-unit-tests	?
php-unit-tests	63.9% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 293 files with indirect coverage changes

[martynmjones]

Hey @eason9487, thanks for working on this fix.

I can't confirm if it definitely resolves the issue but I've not been able to trigger the problem using this branch after running multiple tests so looks good to merge and will be interesting to see if we encounter it again.