Fix file recursion overflow problems

[rjmholt]

Aims to fix PowerShell/vscode-powershell#1613.

Rather than using method recursion to process recursive directories, we now use a while loop and a stack and stop recursing at a depth of 64.

It's a naive solution to the problem, but I didn't want to do any sophisticated expensive checks like building a hashmap of paths or testing for symlinks in case that would cause a performance hit (although it might not be very important).

I also added a new logging method to log exceptions in the log that aren't really fatal.

[TylerLeonhardt]

In the original issue, you said "We might have to build a HashSet every time and ignore files that are already in it."

I think we should have a HashSet so that we don't end up creating ScriptFiles for the same file over and over again.

[rjmholt]

My only real concern with the HashSet is that in a file with a lot of directories, it could get pretty big. So we'd allocate and allocate and allocate and then collect every time we search a path. I'll do some research though.

[SeeminglyScience]

I think we should have a HashSet so that we don't end up creating ScriptFiles for the same file over and over again.

For a HashSet to be useful, wouldn't we have to get the real path for every file? iirc the process for doing that is pretty expensive (at this scale at least) and platform specific.

[SeeminglyScience]

LGTM! Just one suggestion (that may or may not be misguided 🙂)

[TylerLeonhardt]

@rjmholt + @SeeminglyScience you both made good points on the HashMap approach. It could be too expensive.

this LGTM

[rjmholt]

Oh oh oh, so I now see the problem with the HashSet method -- the same directory deeper down will be given back as a deeper path from Directory.GetDirectories(); say that odir is a directory and ldir is a link to that directory, then we'll see the following paths:

• 0 loops: /path/odir/
• 1 loop: /path/odir/sub/ldir/
• 2 loops: /path/odir/sub/ldir/sub/ldir/

We couldn't just compare the paths. We'd have to do symlink inspection and that would be costly.

[rjmholt]

We also need to do something about the BuildInfo file like @rkeithhill suggested

[rjmholt]

This seems to be the best suggestion I've seen.

[rkeithhill]

LGTM. Thanks for the logging changes!