Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[service] windows long paths #14216

Closed
ericLemanissier opened this issue Nov 16, 2022 · 8 comments · Fixed by #14738
Closed

[service] windows long paths #14216

ericLemanissier opened this issue Nov 16, 2022 · 8 comments · Fixed by #14738
Labels
infrastructure Waiting on tools or services belonging to the infra type: question

Comments

@ericLemanissier
Copy link
Contributor

ericLemanissier commented Nov 16, 2022

Is the long path feature enabled on C3I's windows workers ? cf https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#enable-long-paths-in-windows-10-version-1607-and-later

I have the following failure in #13867 (comment):

ERROR: qt/6.4.1: Error in source() method, line 451
	strip_root=True, destination="qt6")
	FileNotFoundError: [Errno 2] No such file or directory: 'qt6\\qtwebengine\\src\\3rdparty\\chromium\\third_party\\devtools-frontend\\src\\node_modules\\eslint-plugin-lit-a11y\\node_modules\\eslint-plugin-lit\\node_modules\\parse5\\lib\\extensions\\location-info\\open-element-stack-mixin.js'

the error happens in conan.tools.files.get, probably because the path C:\J\w\prod\BuildSingleReference@2/s\c889c3\1\qt6\qtwebengine\src\3rdparty\chromium\third_party\devtools-frontend\src\node_modules\eslint-plugin-lit-a11y\node_modules\eslint-plugin-lit\node_modules\parse5\lib\extensions\location-info\open-element-stack-mixin.js is too long (261 characters)

I can reproduce the error by removing the short_paths attribute to the recipe, and removing the LongPathsEnabled registry value

The internal stacktrace:

Traceback (most recent call last):
  File "C:\Users\eric\.conan\data\qt\6.4.1\_\_\export\conanfile.py", line 481, in untargz
    tarredgzippedFile.extractall(destination, members=members)
  File "C:\Users\Eric Lemanissier\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2036, in extractall
    self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "C:\Users\Eric Lemanissier\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2077, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "C:\Users\Eric Lemanissier\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2150, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "C:\Users\Eric Lemanissier\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2191, in makefile
    with bltn_open(tan\data\qt\6.4.1\_\_\export\conanfile.py", line 481, in untargz
    tarredgzippedFile.extractall(destination, members=members)
  File "C:\Users\Eric Lemanissier\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2036, in extractall
    self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "C:\Users\Eric Lemanissier\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2077, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "C:\Users\Eric Lemanissier\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2150, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "C:\Users\Eric Lemanissier\AppData\Local\Programs\Python\Python39\lib\tarfile.py", line 2191, in makefile
    with bltn_open(targetpath, "wb") as target:
FileNotFoundError: [Errno 2] No such file or directory: 'qt6\\qtwebengine\\src\\3rdparty\\chromium\\third_party\\devtools-frontend\\src\\node_modules\\eslint-plugin-lit-a11y\\node_modules\\eslint-plugin-lit\\node_modules\\parse5\\lib\\extensions\\location-info\\open-element-stack-mixin.js'
```argetpath, "wb") as target:
FileNotFoundError: [Errno 2] No such file or directory: 'qt6\\qtwebengine\\src\\3rdparty\\chromium\\third_party\\devtools-frontend\\src\\node_modules\\eslint-plugin-lit-a11y\\node_modules\\eslint-plugin-lit\\node_modules\\parse5\\lib\\extensions\\location-info\\open-element-stack-mixin.js'

The setup of python proposes to do the LongPathsEnabled registry setting at the end:
image

@ericLemanissier
Copy link
Contributor Author

@danimtb @uilianries @jcar87 I'm sorry to bother you again. This is another blocking issue for qt (>= 6.4.0). Is there a plan to enable long paths on windows workers ?

@EricAtORS
Copy link
Contributor

In my experience, the long path registry is only handled by windows applications that support it. If they don't support it, like older versions of VS (as recent as VS2015 didn't support it), that registry key will have no affect.

To support it, you need to tell windows that your path is a long path by prependeing the path with \\?\ according to microsoft's documentation

For example, the long path you mention at the top:
C:\J\w\prod\BuildSingleReference@2/s\c889c3\1\qt6\qtwebengine\src\3rdparty\chromium\third_party\devtools-frontend\src\node_modules\eslint-plugin-lit-a11y\node_modules\eslint-plugin-lit\node_modules\parse5\lib\extensions\location-info\open-element-stack-mixin.js
would become:
\\?\C:\J\w\prod\BuildSingleReference@2/s\c889c3\1\qt6\qtwebengine\src\3rdparty\chromium\third_party\devtools-frontend\src\node_modules\eslint-plugin-lit-a11y\node_modules\eslint-plugin-lit\node_modules\parse5\lib\extensions\location-info\open-element-stack-mixin.js

But this could introduce breaking changes because there's a / in the path and that would have potentially been replaced by \ according to the documentation, but I've never had a forward slash in the path and windows explorer won't let me create a filename or folder with a / in it.

I don't know if this helps you find a solution, but I hope it does as I want to see / use qt 6.4.1

@ericLemanissier
Copy link
Contributor Author

Thanks for your feedback, I'm not sure where you expect a crash. As said above, I reproduced the crash on my computer by removing the short_paths attribute from the recipe, and removing the LongPathsEnabled registry value: There was an exception in get(self, ...) during unzipping. Simply setting LongPathsEnabled registry value allowed get(self, ...) to pass, and the rest of the build passed too (without any other modification). Could you please give the full context of the crash you have ? Thanks!

@uilianries uilianries added infrastructure Waiting on tools or services belonging to the infra type: question labels Nov 23, 2022
@EricAtORS
Copy link
Contributor

I'm not sure what gives you the impression that I have a crash.

Ultimately, what I'm saying is that you can bypass the long paths in certain cases by explicitly prepending the paths with \\?\
We routinely hit paths that are greater than 250 on windows in our application and we've never run in to an issue with it when our paths are prepended with \\?\

I'm not sure if this can be incorporated in to the solution, or if the solution really is to have the build machines have the LongPathsEnabled registry key enabled.

@ericLemanissier
Copy link
Contributor Author

sorry for my misunderstanding. I was confused and thought you were the author of this message. Thanks for the \\?\ workaround. I'll definitely keep this in my mind when path length issue comes back.

@ericLemanissier
Copy link
Contributor Author

ericLemanissier commented Nov 25, 2022

@EricAtORS many thanks for the workaround. 7cafdde just happened to pass on windows ! https://c3i.jfrog.io/c3i/misc/summary.html?json=https://c3i.jfrog.io/c3i/misc/logs/pr/13867/21-configs/windows-visual_studio/qt/6.4.1//summary.json

@Nekto89
Copy link
Contributor

Nekto89 commented Dec 9, 2022

is there any point in disabling short_paths? In VS2019 msbuild still can't handle long paths so libraries like ITK/hdf5 can't be built in cases where conan root is long because of user/channel. Is it fixed in VS2022?

@ericLemanissier
Copy link
Contributor Author

yes, it allows to decompress the source archives, which allows to actually build the package in all the cases where the files with long paths are not used during compilation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Waiting on tools or services belonging to the infra type: question
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants