Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Failed to create hardlink" due to "No such file or directory" in 2.17.0.dev #18661

Closed
stuhood opened this issue Apr 3, 2023 · 6 comments · Fixed by #18768
Closed

"Failed to create hardlink" due to "No such file or directory" in 2.17.0.dev #18661

stuhood opened this issue Apr 3, 2023 · 6 comments · Fixed by #18768
Assignees
@thejcannon
Milestone
2.17.x

Comments

@stuhood
Copy link
Member

stuhood commented Apr 3, 2023 •
edited
Loading

Even after #18600, there appear to be periodic "No such file or directory" errors (likely caused by #18153), similar to this one: https://github.com/pantsbuild/pants/actions/runs/4585391355/jobs/8097532089?pr=18652

AFAICT, we have only seen this on 2.17.0.dev/main, and not in 2.16.x.
@stuhood stuhood added this to the 2.17.x milestone Apr 3, 2023
@stuhood stuhood mentioned this issue Apr 3, 2023
Closed
@stuhood
Copy link
Member Author

stuhood commented Apr 6, 2023

#18696 should increase debuggability here.

@stuhood
Copy link
Member Author

stuhood commented Apr 6, 2023

The first repro from https://github.com/pantsbuild/pants/actions/runs/4633199550/jobs/8198452894?pr=18696:

native_engine.IntrinsicError: Failed to create hardlink to /home/runner/.cache/pants/lmdb_store/immutable/files/97/971d930a44d67d79f358b6c35905744c7509767aa61313e662352cf407c10174 at /tmp/pants-sandbox-O6nGOE/src/python/pants/engine/internals/native_engine.so: No such file or directory (os error 2)

@stuhood stuhood changed the title "No such file or directory" in 2.17.0.dev "Failed to create hardlink" due to "No such file or directory" in 2.17.0.dev Apr 13, 2023
@stuhood stuhood mentioned this issue Apr 13, 2023
Closed
@thejcannon
Copy link
Member

Just saw it myself: https://github.com/pantsbuild/pants/actions/runs/4691537632/jobs/8316196245?pr=18737#step:11:449

@thejcannon thejcannon mentioned this issue Apr 14, 2023
Merged
@thejcannon
Copy link
Member

And a failure on Mac runner using the copy: https://github.com/pantsbuild/pants/actions/runs/4702225938/jobs/8339851741

thejcannon added a commit that referenced this issue Apr 14, 2023
@thejcannon
Clear FSDB materialization cache when removeing a file (#18747)
9616025 
Scenario:
- daemon starts
- materializes FSDB file, entry added to `dest_initializer`
- GC eventually kicks in and calls `remove` on an FSDB file
- `all_fingerprints` doesn't return a fingerprint for the file because
it was removed
- Attempt to `store` the file, but we re-use the `OnceCell` entry in
`dest_initializer`, so it's a no-op
- Hardlinking fails 😢 

New test fails before, passes after

I don't think this bug is
#18661 though :/
@cognifloyd

This comment was marked as off-topic.

Sign in to view
@cognifloyd cognifloyd mentioned this issue Apr 17, 2023
Closed
@thejcannon thejcannon mentioned this issue Apr 19, 2023
Merged
@thejcannon thejcannon self-assigned this Apr 19, 2023
thejcannon added a commit that referenced this issue Apr 19, 2023
@thejcannon
Flush metadata to disk before renaming immutable store files (#18768)
e87c3c6 
Otherwise if we're really unlucky, the metadata isn't persisted to disk
before we try to `hard_link` it and the OS returns an error.

Fixes #18661

(Tested locally using work repo. Always reproduced before PR. Doesn't
reproduce after)
@pietrobolcato
Copy link

Setting the temporary directory in the root folder, gives this error:

IntrinsicError: Failed to create hardlink to /workspaces/kittl-python/.cache/immutable/files/fa/faad51a6a108fba9d40b2a10e82a2646fccbaf8c3d9be47818f4bffae02d94b8 at /workspaces/kittl-python/.temp/pants-sandbox-9z0B1O/pex: No such file or directory (os error 2)

It seems that it is not creating: .temp/pants-sandbox-9z0B1O/pex

My [GLOBAL] pants.toml is:

[GLOBAL]
pants_version = "2.17.0"
pants_ignore = [
  "!.env", # for tests to work
]
local_store_dir = ".cache"
local_cache = true
local_execution_root_dir = ".temp"

Here the log of the execution with -ldebug and --keep-sandboxes=always: https://gist.github.com/pietrobolcato/889887fe2aaf8570ab6f5e9c6bc09e0f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thejcannon thejcannon

Labels
None yet
Projects
None yet
Milestone
2.17.x
Development

Successfully merging a pull request may close this issue.

Flush metadata to disk before renaming immutable store files thejcannon/pants
4 participants
@stuhood @cognifloyd @pietrobolcato @thejcannon