tests are not being discovered

[gbajson-area22]

Type: Bug

Tests for python project are not appearing in VS Code.

In the terminal output I foud the command that is executed.
This command returns proper tests

$ ~/.cache/pypoetry/virtualenvs/vox-7_rKBGgW-py3.11/bin/python ~/.vscode/extensions/ms-python.python-2023.16.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear .
[{"rootid": ".", "root": "/storage/amoje/Sync/area22/vox2", "parents": [{"id": "./src", "kind": "folder", "name": "src", "parentid": ".", "relpath": "./src"}, {"id": "./src/vox", "kind": "folder", "name": "vox", "parentid": "./src", "relpath": "./src/vox"}, {"id": "./src/vox/test_main.py", "kind": "file", "name": "test_main.py", "parentid": "./src/vox", "relpath": "./src/vox/test_main.py"}], "tests": [{"id": "./src/vox/test_main.py::test_read_main", "name": "test_read_main", "source": "./src/vox/test_main.py:13", "markers": [], "parentid": "./src/vox/test_main.py"}, {"id": "./src/vox/test_main.py::test_comedy_tom_hanks", "name": "test_comedy_tom_hanks", "source": "./src/vox/test_main.py:20", "markers": [], "parentid": "./src/vox/test_main.py"}]}]

VS Code version: Code 1.82.2 (abd2f3db4bdb28f9e95536dfa84d8479f1eb312d, 2023-09-14T05:51:20.981Z)
OS version: Linux x64 6.2.0-32-generic snap
Modes:

System Info

Item	Value
CPUs	AMD Ryzen 9 5900X 12-Core Processor (24 x 2200)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	0, 1, 1
Memory (System)	62.71GB (51.43GB free)
Process Argv	--no-sandbox --force-user-env --unity-launch --crash-reporter-id cccdcc23-b3ed-4860-8b66-fadc67e8d915
Screen Reader	no
VM	0%
DESKTOP_SESSION	ubuntu
XDG_CURRENT_DESKTOP	Unity
XDG_SESSION_DESKTOP	ubuntu
XDG_SESSION_TYPE	x11

Extensions (43)

Extension	Author (truncated)	Version
terraform	4op	0.2.5
vscode-markdownlint	Dav	0.51.0
gitlens	eam	14.3.0
remotehub	Git	0.60.0
go	gol	0.39.1
terraform	has	2.27.2
vscode-python-timeit	Has	1.3.8
file-downloader	min	1.0.12
prettify-json	moh	0.0.3
black-formatter	ms-	2023.4.1
isort	ms-	2023.10.1
pylint	ms-	2023.6.0
python	ms-	2023.16.0
vscode-pylance	ms-	2023.9.10
jupyter	ms-	2023.8.1002501831
jupyter-keymap	ms-	1.1.2
jupyter-renderers	ms-	1.0.17
vscode-jupyter-cell-tags	ms-	0.1.8
vscode-jupyter-slideshow	ms-	0.1.5
remote-containers	ms-	0.309.0
remote-ssh	ms-	0.106.4
remote-ssh-edit	ms-	0.86.0
remote-wsl	ms-	0.81.3
vscode-remote-extensionpack	ms-	0.24.0
azure-repos	ms-	0.36.0
remote-explorer	ms-	0.4.1
remote-repositories	ms-	0.38.1
remote-server	ms-	1.4.3
remote-kubernetes	Okt	0.4.10
vscode-circular-dependencies-finder	ole	1.1.0
r-debugger	RDe	0.5.4
java	red	1.22.1
vscode-yaml	red	1.14.0
markdown-preview-enhanced	shd	0.7.9
intellicode-api-usage-examples	Vis	0.2.8
vscodeintellicode	Vis	1.2.30
keyoti-changeallendoflinesequence	vs-	0.0.3
vscode-java-debug	vsc	0.54.0
vscode-java-dependency	vsc	0.23.1
vscode-java-pack	vsc	0.25.14
vscode-java-test	vsc	0.39.1
vscode-maven	vsc	0.42.0
markdown-all-in-one	yzh	3.5.1

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
vscccc:30803845
282f8724:30602487
89544117:30613380
showlangstatbar:30737416
962ge761:30835153
03d35959:30757346
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30805730
pythonnoceb:30805159
asynctok:30821568
dsvsc013:30795093
dsvsc014:30804076
diffeditorv1:30821571
dsvsc015:30829745

[gbajson-area22]

This started to work when I opened my project from the realpath.
I have a symlink to project dirs in my home dir and in that case it doesn't work.

VS Code file opened as:
/home/gbajson/Sync/area22/vox2/vox.code-workspace -- tests are not discovered

VS Code file opened as:
/storage/amoje/Sync/area22/vox2/vox.code-workspace -- tests are discovered

$ ls -ld ~/Sync
lrwxrwxrwx 1 gbajson gbajson 20 cze  3  2022 /home/gbajson/Sync -> /storage/amoje/Sync/

[eleanorjboyd]

Hello! Could you try this on the rewrite of the testing infrastructure to see if it occurs here? You can do so by adding this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"].

You can confirm you have the rewrite enabled by setting "python.analysis.logLevel": "Trace", in your user settings then check Experiment 'pythonTestAdapter' is active in your python logs.

Let me know if the rewrite fixes your issue. Thanks!

[kbokh]

The same problem with a symlink. Linux (EndeavourOS), vscode 1.82.2, a python 3.11 env managed by the poetry. The unittest can find tests but the pytest cannot. Pytest works well with a realpath

[eleanorjboyd]

Can you clarify what your setup is with the two root paths? Are those on the same computer? I have been trying to repro this symlink issue but have been unable to. If you are able to review the steps I took here and let me know how your setup differs that would be very helpful! #21373

Hello! I got a setup working on a parallels VM. I have a windows VM with parallels then have WSL on that machine to a Ubuntu machine. These are the steps I took and am still not able to repro. Any steps I am missing / did incorrectly?

1. connect via WSL
2. open vscode while connected via WSL
3. create a folder called "purple_tiger"
4. add a folder called "pink_tiger"
5. create a test file "test_pink_tiger.py" with a single pytest in it
6. get pytest running and the test explorer working in a vscode window with this folder
7. create a symlink between the "purple_tiger" folder on my WSL and "tiger_alias" on my windows parallels VM, with a powershell command like this New-Item -ItemType SymbolicLink -Path "C:\Users\eleanorboyd\Documents\python-tester-files\tiger_alias" -Target "\\wsl$\Ubuntu\home\linux_user\linux_root\purple_tiger"
8. run cd "C:\Users\eleanorboyd\Documents\python-tester-files\tiger_alias"
9. now in that folder run code .
10. now with vscode launched, open the python file so the extension is activated
11. click the test icon on the left hand bar
12. click to configure tests, selected "pink_tiger" as my test folder
13. tests show up and run as expected

[rohanaras]

Can confirm that I have a similar issue as well.

Can you clarify what your setup is with the two root paths? Are those on the same computer?

My file paths look like this:

~/repos/[git-repo-name]

where I have the symlink

repos -> /mnt/data_science/rohana/repos,

~ is /home/rohana1,

and /mnt/data_science is a nfs file share.

Opening any repository from /home/rohana/repos results in no tests being discovered, whereas all tests are discovered when I open any repository from /mnt/data_science/rohana/repos

Running pytest from the terminal works in both places.

Could you try this on the rewrite of the testing infrastructure to see if it occurs here? You can do so by adding this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"].

Tried doing this, couldn't figure out how to confirm that the rewrite was working (should Experiment 'pythonTestAdapter' have shown up under OUTPUT > Python?), but it didn't seem to fix the issue.

System information:

• Linux Rhel-8.7
• VS Code 1.82.2
• python 3.11.5 and 3.10 conda environments
• pytest 7.4.2 and 7.4.0

[rohanaras]

Oh and one more thing: the output of

conda run -n default --no-capture-output python ~/.vscode-server/extensions/ms-python.python-2023.16.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2023.16.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear .

appears to list all tests when I run the command myself from the symlink version of the repositories

[eleanorjboyd]

Hi @rohanaras could you try this on the rewrite? It looks from the output you are still on the older code.

To try it yourself, add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"].

You can confirm you have the rewrite enabled by setting "python.analysis.logLevel": "Trace", in your user settings then check for Experiment 'pythonTestAdapter' is active in your python logs.

Let me know if the rewrite fixes your issue. Thanks!

[AdrianB-sovo]

For those using WSL on Windows, the "python.experiments.optOutFrom" will not be supported on your local user settings, you should use the remote user settings.
ctrl + shift + p → "Preferences: Open Remote Settings (JSON)"

[alanwilter]

Hi @eleanorjboyd

It's me again.

So I have the same issue, on macOS Sonoma, with Vscode 1.84.0. I use symlink like this:

~/Projects/face-ai -> "/Users/alan/Library/Mobile Documents/com~apple~CloudDocs/Projects/face-ai"

• Using Symlink:
  If I start code ~/Projects/face-ai, no pytest Explorer at all, neither with old or new "rewrite". Nada. However, if using "rewrite", Python Test Log does show something,

But no matter what I do, Configure Python Tests never work. I tried unitest and pytest. Nada.

• Using REALPATH:

If using "/Users/alan/Library/Mobile Documents/com~apple~CloudDocs/Projects/face-ai" (note the whitespace, poor thoughtless Apple developer work IMO), so you understand why I prefer symlink (but I use iCloud to backup my work).

Old Test discovery works for me, but the "rewrite" has the same issues.

When using rewrite I do see in my Python Log:

2023-11-06 13:59:13.695 [info] Experiment 'pythonTestAdapter' is active

Is there a public repo with pytest that you can clearly see this rewrite working for you? Can you share a video showing how do you setup and get it to work?

I'm plagued with this issue now matter system I work, being it Mac, Linux or WSl.

[eleanorjboyd]

do you have any logs after the test structure is returned in the logs? It seems it is discovering it on the pytest side (as shown by the logs) so the disconnect must be either sending the data to the vscode extension or parsing the data from the vscode extension and populating the UI. This is the common example I use to send my team members so they can test this feature:

1. Setup:
   1. Clone the test repo from [here](https://github.com/eleanorjboyd/inc_dec_example_repo) or use your own repo of pytests.
   2. Launch VSCode and open the inc_dec_example_repo/inc_dec_example_test_suite testing repo folder.
   3. Add the following to your settings in VSCode: "python.experiments.optInto": ["pythonTestAdapter"],
   4. Use the "create environment" command from the command palette, select venv and a Python version >=3.7.
   5. (for pytest)
      1. In the terminal, run python -m pip install pytest.
      2. From the command palette, run "python: configure tests", select pytest and then "." (root directory)
   6. click the test explorer icon (beaker)

interesting about the space, I will think on that / investigate. I am going on vacation for a few weeks so sadly I cannot investigate now

[alanwilter]

So I used you repo, it's somewhat worked but I got this message (just part of it):

==================================== ERRORS ====================================
___________ ERROR collecting testing3/testfolder2/tests/test_math.py ___________
import file mismatch:
imported module 'test_math' has this __file__ attribute:
  /Users/alan/Downloads/inc_dec_example_repo/testing3/testfolder1/tests/test_math.py
which is not the same as the test file we want to collect:
  /Users/alan/Downloads/inc_dec_example_repo/testing3/testfolder2/tests/test_math.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
___________ ERROR collecting testing3/testfolder2/tests/test_str.py ____________
import file mismatch:
imported module 'test_str' has this __file__ attribute:
  /Users/alan/Downloads/inc_dec_example_repo/testing3/testfolder1/tests/test_str.py
which is not the same as the test file we want to collect:
  /Users/alan/Downloads/inc_dec_example_repo/testing3/testfolder2/tests/test_str.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
=========================== short test summary info ============================
ERROR testing3/testfolder2/tests/test_math.py
ERROR testing3/testfolder2/tests/test_str.py
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
==================== 55 tests collected, 2 errors in 0.07s =====================

Then I went to my repo. My project settings.json has:

  "python.testing.pytestArgs": ["tests", "-v", "-c", "/dev/null"],
  "python.testing.unittestEnabled": false,
  "python.testing.pytestEnabled": true,

But once I copied your local settings.json something happened, logs showed something, Explorer as well:

your settings.json:

{
  "python.testing.pytestArgs": ["."],
  "python.testing.unittestEnabled": false,
  "python.testing.pytestEnabled": true
}

However, it's still useless as nothing happens when I click on play test button.

I also use pyproject.toml with

[tool.pytest.ini_options]
addopts = "tests --cov=faceapp --cov-report=term-missing:skip-covered --cov-report=xml -n 8"

But even commenting that line it still have the same problem.

After some try-and-error I found out that your new Test rewrite don't like "-c", "/dev/null" args in my settings.json

However, this is there to avoid using the set given in pyproject.toml.

Then, I remembered the "symlink" issue and when I used the realpath (even if having a whitespace)... it worked!

Bottomline:

• pyproject.toml is not the issue, I put back my settings for pytest
• remove "-c", "/dev/null" from "python.testing.pytestArgs"
• don't use symlink
• Please, fix the symlink bug 😄
• Have a nice holidays first

[alanwilter]

Ok, just one glitch.

Now I see the issues reported in other tickets. I have a project for which some tests are not working:

CLIENT: Server listening on port 63891...
Received JSON data in run script
PYTEST ERROR: TEST_UUID and/or TEST_PORT are not set at the time of pytest starting. Please confirm these environment variables are not being changed or removed as they are required for successful test discovery and execution. 
TEST_UUID = faf77953-ba88-42e3-8991-dac1d3fcc08a
TEST_PORT = 63611

============================= test session starts ==============================
platform darwin -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /Users/alan/.virtualenvs/acpype/bin/python
cachedir: .pytest_cache
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
>>>	Version: 2023.11.2

rootdir: /Users/alan/acpype
plugins: random-order-1.1.0, xdist-3.4.0, typeguard-4.1.5, cov-3.0.0
collecting ... collected 1 item

tests/test_acpype.py::test_args_wrong_inputs[None-2- error: ]

[eleanorjboyd]

thank you for your investigation @alanwilter! All these notes are extremely helpful. So just to make sure I have it right, the repo I sent you worked when you removed it from using symlinks? (when you said "I used the realpath" not sure where you are saying you used the realpath). Therefore to replicate this issue I only need a symlink on my computer similar to ~/Projects/face-ai -> "/Users/alan/Library/Mobile Documents/com~apple~CloudDocs/Projects/face-ai" in that its a symlink but it can have the same home directory (which is the current user), and the maybe try with the symlink not being an absolute path but relative one.

[alanwilter]

Realpath is absolutepath. I only tried symlink to absolutepath. Try relative as well. And try paths with whitespace: absolute, symlink to absolute and to relative, all you may wonder, etc. Add these scenarios to your pytest units.
And thanks! I saw you sorted #22447 so it may have an impact here. When it will be in the public release so I can test myself?

[eleanorjboyd]

Thanks for the extra info! The fix to issue #22447 is in our next release so that will go out around thursday next week!

[alanwilter]

I've tested this new release and after a lot of headache, I got it to work again but it's not ok. Still, sorry, useless in my case.

This new Test definitely uses pyproject.toml:

[tool.pytest.ini_options]
addopts = "tests --cov=faceapp --cov-report=term-missing:skip-covered --cov-report=xml -n 8"

And I can't deactivate it with

"python.testing.pytestArgs": ["tests", "-vv"], //, "-c", "/dev/null"],

because if "-c", "/dev/null" is back, then new Test does not work.

The problem is: if I keep my [tool.pytest.ini_options], I can't use Vscode debug. In fact, whenever or whatever test I do in vscode, it will will follow pyproject.toml directive.

I'm going back to the Old "good" Test.

The new Test really need to be independent of pyproject.toml.

PS: I've only tested the absolutepath case.

[eleanorjboyd]

@alanwilter, I am seeing the following problems from what you have discussed that I need to investigate. Please correct me if I have one of these wrong

1. symlinks not working
2. -c arg causing testing to not work at all
3. vscode debugging not working when [tool.pytest.ini_options] is being used

In terms of the pyproject.toml and its independence pytest is actually the one that looks for that file. Here is that precedence as I understand it. If you do not think this is correct and you think pytest natively behaves differently please let me know the reasoning.

1. Command Line Options
2. pytest.ini
3. pyproject.toml
4. tox.ini
5. setup.cfg
6. conftest.py Files (scoped to their directories)
7. Environment Variables

@paulacamargo25, can you look at number 3 I listed above about "vscode debugging not working when [tool.pytest.ini_options] is being used"?

Thanks

[alanwilter]

Perfect summary. So, the fact that the "Old" Test works is because I can pass -c /dev/null, like emulating the "CL option" (top priority).
So, addressing 2) and we can ignore 3).

[eleanorjboyd]

Hi @alanwilter, sorry for all the back and forth. I have created the following sample project and am having the '-c' option working for me. I also created a pyproject.toml file that I then ignore using the '-c' to a different, blank pytest.ini file. Does this work for you? How does this project structure differ from yours?
cwd_specified.zip

[alanwilter]

Thanks @eleanorjboyd, using dummy pytest.ini did the trick indeed. So it seems that the New Test does not accept /dev/null.
It's a small price to pay.

So, the only issue pending now it the symlink.

If I do:

ln -s cwd_specified llll
code llll

your example test won't work.

[eleanorjboyd]

@alanwilter thank you so much for the example, that was finally the symlink repro that worked for me and I was able to investigate the problem!!

I have created this issue to consolidate all symlink related bugs: #22658. I worked out a solution on Friday but it ended up being more complicated than expected so it will require some more work. I will post updates there, thanks

[eleanorjboyd]

The issue mentioned above now include a proposed solution. Please check it out here and contribute any thoughts you may have: #22658

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

[eleanorjboyd]

Hello! This bug should be fixed and on vscode insiders! The release to vscode stable will be out by the end of the week. Please let me know if you try it on insiders or stable when it comes out and still doesn't work and I can investigate your specific issue further.

[eleanorjboyd]

Hi @alanwilter, just got back from vacation and unlocked this. Could you describe the issue you are still seeing? Is it still the same setup as this? I have been testing it in a few scenarios and it has been working for me recently but I must still be missing a scenario. Thanks

Thanks @eleanorjboyd, using dummy pytest.ini did the trick indeed. So it seems that the New Test does not accept /dev/null. It's a small price to pay.

So, the only issue pending now it the symlink.

If I do:

ln -s cwd_specified llll
code llll

your example test won't work.

[eleanorjboyd]

Also can you send over your pytest args when you describe your setup? I think this could be contributing if you already specific a cwd or rootdir but I would have to check further.

[eleanorjboyd]

Hello again! Recently I have merged this PR which was very similar to your symlink issue. I think that merge should resolve this but please let me know if not and I can keep investigating. Thanks!