Windows Python 3.8+ os.add_dll_directory search order is defined as unspecified

[tbttfox]

Describe the bug
Random users across our studio would get this dll error when running from pxr import Tf. Unfortunately, this was machine specific as far as we could tell. Identical setups on different machines would give different results.

# #     from pxr import Tf
# #   File "c:\Program Files\Autodesk\MayaUSD\Maya2023\0.16.0\mayausd\USD\lib\python\pxr\Tf\__init__.py", line 163, in <module>
# #     PreparePythonModule()
# #   File "c:\Program Files\Autodesk\MayaUSD\Maya2023\0.16.0\mayausd\USD\lib\python\pxr\Tf\__init__.py", line 88, in PreparePythonModule
# #     module = importlib.import_module(
# #   File "C:\Program Files\Autodesk\Maya2023\Python\lib\importlib\__init__.py", line 127, in import_module
# #     return _bootstrap._gcd_import(name[level:], package, level)
# # ImportError: DLL load failed while importing _tf: The specified procedure could not be found.

Steps to reproduce
I don't think it's possible to reproduce. See Additional Context

Expected behavior
No DLL Error

Specs (if applicable):

• Windows 10
• Maya 2023.3
• Mayausd 0.20.0 and 0.16.0

Additional context

We found that USD looks at the system's PATH variable to load its dll's on python 3.8+ here:
https://github.com/PixarAnimationStudios/USD/blob/release/pxr/base/tf/__init__.py#L47

So we started chasing the DLL's, and the breaking machines were loading Bifrost's usd dll's before maya-usd's. This seemed very strange because maya-usd came before Bifrost in the PATH, and it wasn't happening on other machines.

On a hunch, we checked the cpython source for os.add_dll_directory, which ends up calling this function:
https://learn.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-adddlldirectory

And, oh boy, is there a doozy in those docs:

If AddDllDirectory is used to add more than one directory to the process DLL search path, the order in which those directories are searched is unspecified.

That says to me, that the paths added to the dll search are not necessarily searched in order, and without knowing the internal Windows DLL search mechanism, we apparently can't know that order.

Proposed solution
We don't need to search the entire PATH for libraries when we know exactly where they are. This is easily accomplished by appending to PXR_USD_WINDOWS_DLL_PATH instead of PATH in modules/mayaUSD.mod.template
Even if you figure out some other reason for this err, I would still recommend doing this.

I don't know about modules/alUSD_Win.mod.template or modules/pxrUSD_Win.mod.template though. Somebody with more experience should chime in.

[santosd]

@seando-adsk is this something that would require USD changes or could it be taken care of from the Maya USD Plugin Side?

[seando-adsk]

Thanks for the information about PXR_USD_WINDOWS_DLL_PATH. I wasn't aware of that variable. We can look into adding it to the .mod file to help when loading the python modules. However we still require adding the USD folders to the PATH (for Windows). This is required so that the regular DLLs, such as MayaUsd, which links with USD can find USD. I would suggest adding a PXR_USD_WINDOWS_DLL_PATH line for each PATH line. So for example in the USD section of mayaUsd.mod adding:

PXR_USD_WINDOWS_DLL_PATH+:=bin
PXR_USD_WINDOWS_DLL_PATH+:=lib
PXR_USD_WINDOWS_DLL_PATH+:=plugin/usd

Would you be able to try this on your machines with the problem and see if it fixes it.

Sean

[tbttfox]

@seando-adsk That was the fix we came up with as well, and it seems to work. Or (more precisely) It doesn't immediately break :)

@santosd Even if this falls mostly on the Maya side to "fix"... This error is so esoteric that I think it would be a good idea to update WindowsImportWrapper to check (first time only) for multiple USD dll's if they're falling back on PATH, and at least show a warning.

[seando-adsk]

@tbttfox Would you be able to submit a pull-request with the fix to the mod file? I didn't see a CLA on file for your company, so please have a look at CONTRIBUTING.md.

As for making a change to WindowsImportWrapper - that function is in the Pixar USD code. So you would need to submit either an issue or pull-request to the USD repo: https://github.com/PixarAnimationStudios/USD

[maya-usd-git-sync]

Issue synced internally to EMSUSD-751

[tbttfox]

Oh, I'm sorry. I forgot to close this issue when my PR got merged.