[perf] Automatically prefer Windows Dev Drive for default tmp_path

[zooba]

What's the problem this feature will solve?

Windows Dev Drive is a new OS feature that allows users to create a high performance disk that's designed for development activities. The performance improvement is based on new optimisations to the file system driver, and separating it out from the default OS drive reduces other overheads.

It's perfect for storing temporary files, however, it was decided that it's not safe for Windows to redirect all TEMP accesses to a Dev Drive by default. But our testing showed the improvements get significantly better when you do it, and we believe apps that can switch ought to switch.

I was part of the design team for this feature, so happy to add as much more context as you'd like and as I'm able to. Obviously these are public statements, so I have to be careful about things that might be interpreted as promises and not merely dreams, hopes and ambitions 😉

Describe the solution you'd like

Currently PyTest has ways for callers to redirect where temp files are stored, but I'd like to propose it could detect that the current project (or environment, or CWD, or something, I'm not so fussy about it) is on a Dev Drive, and if so, create a temp directory on that same drive.

I'm in the process of adding an os.path.isdevdrive function to CPython for 3.12, so would love to hear whether this is something you'd consider using, and whether a simple test like this suits your needs.¹

The kind of logic I'd expect to see (around here):

from_devdrive = None
if not from_env and hasattr(os.path, 'isdevdrive'):
    # passing through "tbd_project_root" is TBD
    root = ''.join(os.path.splitroot(self.tbd_project_root)[:2]) # get 'D:\\' or equivalent path
    if os.path.isdevdrive(root):
        from_devdrive = os.path.join(root, '.pytest-temp')
temproot = Path(from_devdrive or from_env or tempfile.gettempdir()).resolve()

isdevdrive will work on the full path, but I strip back to the drive name first because it could change which drive is actually used (e.g. the project might be in a mounted directory). There's not really an efficient way to handle this case, and on balance it makes the most sense to just ignore the optimisation anyway (access through a mounted directory is typically slow).

Obviously this can only light up in 3.12 and not earlier (unless you want to port the code into ctypes, which I'll understand if you don't bother). Dev Drive doesn't become widely available until the end of the year anyway, so 3.12 will also be available. Just another reason for people to upgrade! 😄

Alternative Solutions

An alternative would be to detect and suggest to users that since their code is on a Dev Drive, they should also manually provide the setting to store temporary files on it as well. I'm quite okay with this as an approach, and it might make more sense anyway (especially as a path towards changing the default in some unspecified future).

Another alternative would be to merely document it. I'd be disappointed by this, because we really want the fact that a Dev Drive is being used to be the signal that dev-related files should be stored there, and we don't want to encourage users to set global environment variables for something like this. It also seems unlikely to be discovered by existing users, who probably aren't going back to read this documentation regularly.

Python 3.12 already has os.listdrives(), so there is the possibility of searching for a Dev Drive. I don't think you'd want to go down this path though, unless there's genuinely no usable info about where the user is currently working. If they've got a Dev Drive but aren't using it, they probably have a good reason and we should leave them alone.

Additional context

If you prefer video demonstrations: https://build.microsoft.com/en-US/sessions/7ed9bb72-b4f4-4490-9b26-911d1ac263d1?source=/home

The docs again: https://learn.microsoft.com/en-us/windows/dev-drive/

Right now it's available on the Dev Channel of Windows Insider, which means it is public and people are trying it out. Python and Pytest was one of the key scenarios we were testing during development, so it'd be awesome to have it be the best it can.

Footnotes

1. Obviously we're past the first beta, but I got an allowance to add it before beta 2 (next Tuesday!) because of NDAs surrounding the feature. ↩

[RonnyPfannschmidt]

If there's a platform specific default way to get the “local" tmpdir, which is either tmp,or a default tmp of the dev drive, I'm all for enable by default,else we need opt in plus hints

[zooba]

the “local" tmpdir, which is either tmp,or a default tmp of the dev drive

About the only thing missing here is that we didn't decide on a convention for a "default tmp of the dev drive". I'm personally inclined towards dotted directories in the root of the drive, though there's a good argument for everyone agreeing to create and use a D:\TEMP like the old days. We're kind of waiting to see which convention apps think is best for them.

(I like the dotted names because it allows apps to feel like they can keep long caches in there too.)

(Aside: I actually learned through this process that the Windows GetTempPath API is literally just reading the environment variables 😆 )

[RonnyPfannschmidt]

So with the state as is,the first iteration will be opt in, once there's a standard we can switch to opt out

[nicoddemus]

Personally I'm fine with 1) detecting the project is in a dev drive and 2) using a custom temporary directory, like .pytest-temp as you suggest.

My rationale is that dev drive itself is a new feature, so old users will not be affected, and because of that we are free with setting new defaults for it.

We can even leave a note about D:\.pytest-temp being experimental, in the sense that if in the future Dev Env promotes an official temporary directory, we switch to using it (say to D:\TEMP\.pytest-temp).

[alexlambson]

Is there a system environment variable we can check for the existence of? I'm assuming it won't always be D:\ because on my machine that's the data drive.

If there's a way to test this on windows 10 then I can work on this.

[RonnyPfannschmidt]

as per original comment a new api will be available in python 3.12+
as per the documentation link the feature is available in windows 11 insider previews

[zooba]

Is there a system environment variable we can check for the existence of?

I proposed this, but it doesn't work because it's totally possible to have multiple Dev Drives. So all we can really do from this side of things is check whether a particular path is on one (or use the new-in-3.12 os.listdrives() to search, but I don't think that makes sense for this scenario).

And yeah, it's not coming to Windows 10 I'm afraid. Gotta offer some reason for people to upgrade 🤷‍♂️