Fix path resolution for Windows Symbolic Links

[Vivraan]

Describe the project you are working on

A 2D game as a tutorial in 3.2.2.stable, although the given proposal is applicable for any game using plugins which are passed around as git repos. I'm using Windows 10, but I expect this to also work as expected in *nix and macOS.

Describe the problem or limitation you are having in your project

I want to create symlinks of git cloned plugins' addons/ directories so that my own repo stays lean, but Godot doesn't recognise symlinks that well. Godot behaves with symlinked files in two ways as I've observed:

1. Treating them as read-only (symlink clones) - if saving after modfying, the editor shows a pop-up saying:

   
   Note, here DebugOverlay.gd is derived from my fork of GammaGames' godot-debug-overlay, a small plugin for displaying debug information on screen. The script is actually a symlinked to a location outside my project and is being read correctly by Godot.

   and erroneously produces this in the console:

   Safe save failed. This may be a permissions problem, but also may happen because you are running a paranoid antivirus. If this is the case, please switch to Windows Defender or disable the 'safe save' option in editor settings. This makes it work, but increases the risk of file corruption in a crash.

   The plus side is that a .tmp file is created which can then be diffed and merged on the original file.

   This read-only hack can be achieved by using symlink clones - keeping concrete directory structures but maintaining symlinks to individual files.

   While it's pretty useful since I'd expect to affect changes in my plugin by opening it as a separate Godot project (using a project.godot file likely supplied by the plugin maker(s) or by creating one myself), the drawback is that Git will treat it like any other file, and if I pull any of these files in case they're available on the remote repo, Windows can't tell that they're symlinks anymore (opening the file will just contain a system path to the original resource) and I effectively lose the symlink.

2. Crashing unexpectedly (junction hardlinks) - When using junction hardlinks, Godot may flip the table trying to import a resource which isn't actually there. I learnt this the hard way - the editor may crash and the adjoining log window returns a waterfall of blood-red error logs. However, this doesn't always happen - Godot may play nice and reflect changes between simultaneously running Godot engine processes. The downside is that Git now assumes that the files actually exist in the repo containing the symlink, and probably creates artefacts assuming this.

There is no workaround that allows me to include minimal addon content which is also versionable (I could use a git submodule but I'll also need to understand git filter and git sparsecheckout, so if this is something that may work I'm all ears.. er... eyes), so I ignore my project's addons/ folder except for a file at ../addons/REQUIRES.md that links to the plugin repo should I (or anyone on my team) wish to clone my project elsewhere, regardless of whether anyone here uses symlink clones (as per the Link Shell Extension extension which makes this as easy as two clicks.)

Describe the feature / enhancement and how it helps to overcome the problem or limitation

I do not know the specifics of how a file subsystem understands what's a symlink or not, and my knowledge of symlinks in *nix is limited to ln -s, but while I've read some proposals for introducing dependency and package management (too many to link, if you find them please don't hesitate to link them here) and come to understand the complexity in remaking the whole AssetLib system, I think a method which relies on submodules and symlinks can help bridge some gaps, and if implementing all of it is too hard, at least let Godot know about symlinks.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams

Godot's file system could be updated to understand symlinks in different OSs, and treat them as special directories with references to the global filesystem, maybe with a greyed out icon, or an @ symbol - anything that marks it from normal folders which are actual children of res://.

The vision I had (Note that that either junction or symlink are optional)

I cannot point to the exact proposal but there was one that wanted to allow Godot to understand the global filesystem, and the owner came up with the DSCN plugin, a method to pass around scene data outside the project folder, as well as a proposal for plugin management at the system level which was dropped.

Sure, this could be a plugin level content again: write a script that
- detects the OS
- recognises the file path in a symlink clone file
- correctly assigns the kind of symlink to make
- runs the appropriate symlink command in the appropriate terminal with the appropriate privileges

effectively rewiring the files pulled from the Git repo as symlinks in the appropriate platform/OS.
This satisfies my project's current requirements since I don't plan on including the godot-debug-overlay plugin content in a build, but Godot may need to correctly include the scripts being linked to if they're needed in a build, which AFAICT must be built into core.

One could use a placeholder file (akin to __init__.py, gdignore, or similar) to mark these directories as containing symlinks, which could also be a REQUIRES file - while I made mine a Markdown file for readability, it should probably be a plaintext file, ideally a kind of .cfg file that denotes:
- platform/OS agnostic symlink type, or
- a list of OS-specific symlink types applicable to different platforms the project intends to be edited on

EDIT: On second thought, it might be useful to provide such a file outside the addons/ tree, and it should thus also include
- relative path to symlinked directories or files (in users://? addons://? Allow that setting to be passed through project.godot in a [requires/[symlink] section? Put the whole thing outside project.godot in a requires.cfg/links.cfg file? I see a pattern similar to a packages.xxx file.)

EDIT 2: It's preferable to make such an annotator agnostic to the type of symlink used, so it's better not to assume that only symlink clones will be used. The links would then rightfully belong in project.godot until a dependency management solution is conceived (or not).

If this enhancement will not be used often, can it be worked around with a few lines of script?

It needs to be an editor plugin tool that listens to changes in the filesystem or currently checked out branch/commit, which may not be a trivial 10-line script but could be a single monolithic script, if it doesn't support adding symlinked files to builds. If needed, PoC can make use of node metadata.

The difficulty is that the files and their symlinks look exactly the same by name from the outside to Git (Windows correctly marks symlink clones as such):

The IDE knows that both are essentially the same:

Some Git GUIs detect what the symlinks actually contain:

When uploaded to Git, they will also look like these single line files containing some absolute or relative path.

This is a non-problem for tiny plugins - indeed even in my case I could paste the files and call it a day - but the occasionally large plugin may raise eyebrows if passed around by (figurative) value, and not as a (figurative) reference, and eyes may roll when these are copy-pasted around projects. In these cases, an optional REQUIRES.cfg/requires.cfg/symlinks.cfg file or optional [requires]/[symlinks] section to project.godot can be added.

Is there a reason why this should be core and not an add-on in the asset library?

It could be another plugin that needs to be pasted into my project, but as I stated earlier:

Godot may need to correctly include the scripts being linked to if they're needed in a build, which AFAICT must be built into core.

It could be something the engine provides so that the experience with symlinks becomes seemless, without possibly getting outdated, and devs can begin to grok the problem of dependency management in its possibly its simplest, most low-level form, possibly even forming a basic dependency system that can symlink git repos (not necessarily as submodules) to give absolute freedom to users to pass their plugins around by value or by reference.

Most of this content is largely incorrect.

[Calinou]

Related to #554.

[Calinou]

@SHIVAMMUKHERJEE As far as I know, this is only an issue on Windows since its symlinks implementation differs from the one that's common to macOS and Linux.

(Also, Git might be able to port over symlinks in a repository, but I'm not sure.)

[Vivraan]

@SHIVAMMUKHERJEE As far as I know, this is only an issue on Windows since its symlinks implementation differs from the one that's common to macOS and Linux.

(Also, Git might be able to port over symlinks in a repository, but I'm not sure.)

Read somewhere that Git used to allow symlinks but beyond 1.6.0 it didn't due to trouble.

I'd really love to see this tested on some Linux or macOS editor build. Could try to do this myself.

EDIT: symlinks seem to work best in Git if they are inside the parent project to a subproject, which is totally not what I'm looking for.

[Vivraan]

Adding a short note: Unity uses its Asset Browser for copy-paste content, which it also downloads to system local storage, and uses a dedicated Package manager for other stuff. My proposal doesn't necessarily touch the functionality of the latter.

[Vivraan]

I checked Godot in Manjaro (booted from a pen drive): I can't create directory hardlinks possibly since it's from a bootable medium (not allowed on Linux), so I tried to create a regular old symlink from some plugin located outside a test project.

But this symlink doesn't show up in Godot (I assume this is similar to Windows shortcuts).
Please advise on whether I should check this behaviour by fully installing this distro or another distro.

I was incorrect about this. If the linked directory doesn't have a project.godot file in it, it links correctly.

[Vivraan]

I'm confirming that Git is able to understand Linux soft symbolic links containing both absolute and relative paths.

[Vivraan]

Windows soft symbolic links are detected properly by Godot's FileSystem, but the paths of the added scripts cannot be resolved, which is not a problem on Linux.

Can this be fixed?

[Vivraan]

Maybe this relates as well: #1205

[bruvzg]

As far as I know, this is only an issue on Windows since its symlinks implementation differs from the one that's common to macOS and Linux.

macOS APFS volumes also have "firmlinks" used to overlay locations on the multiple volumes to the single folder.

For example non of the system apps are accessible from the Godot file selector:

Since "firmlinks" are used only for overlaying read-only system volume and are not user creatable, it's not as important as symlinks, but still can be a bit inconvenient.

[Vivraan]

A TL;DR clarification:
In Windows, symlinked files throw an error when trying to write on them, creating a .tmp copy, symlinked folders are not parsed properly, and Git doesn't recognise Windows symlinks.

[DanielKinsman]

Even on linux, symlinks do not work as expected. Godot 4.2.2 will simply overwrite any symlinks with a copy of the actual file and all its contents instead. This is the case for gd scripts saved in the editor and even .glb files imported by the editor. Though symlinking whole directories works as expected I think.

Not a high priority to change but I guess it would be nice.