Glob patterns with subdirectories whose parent directory does not exist fail

[Xiphoseer]

While writing my second Builder, I encountered the following:

$ dart run build_runner build
Building package executable...
Built build_runner:build_runner.
[INFO] Generating build script completed, took 235ms
[WARNING] Invalidated precompiled build script due to missing asset graph.
[INFO] Precompiling build script... completed, took 4.6s
[SEVERE] Nothing can be built, yet a build was requested.
[INFO] Initializing inputs

You have hit a bug in build_runner
Please file an issue with reproduction steps at https://github.com/dart-lang/build/issues


FileSystemException: Directory listing failed, path = 'c:\<path to project>\not\*' (OS Error: The system cannot find the path specified.
, errno = 3)
Instance of '_StringStackTrace'

Steps to reproduce

1. Create a new dart package (e.g. using dart create -t package-simple bug_report)
2. Add build_runner: to dev_dependencies in pubspec.yaml
3. Add the build.yaml below
4. Invoke build_runner (e.g. using dart run build_runner build) on Windows

# build.yaml
targets:
  $default:
    sources:
      include:
      - $package$
      - lib/$lib$
      - not/exists/**

Expected Results

Nothing. <path to project>/not doesn't exist and shouldn't cause anything. I.e. exactly what happens when includeing just not/**. Under WSL/Ubuntu, the same example package doesn't crash.

What builder(s) you are using (or writing yourself).

I'm writing a builder that is supposed to take data from a JSON file and turn it into dart classes at build time.

Context: We're looking for a way to turn ISO alpha-2 codes into localized country names using CLDR data without using a plugin (which limits platform choices) or one of the many packages with vendored lists (usually limited to english names). Essentially, something akin to how date_symbol_data_local is generated.

In practice I'm npm installing https://www.npmjs.com/package/cldr-localenames-modern as a lightweight alternative to vendoring https://github.com/unicode-org/cldr-json/tree/main/cldr-json/cldr-localenames-modern as files or a git submodule.

I'm currently debugging why the JSON files (located in <path to project>/sub_package/node_modules/...) are not passed to the builder instance created (it prints a log statement) for the $default target in sub_package when running build_runner build on the root project. I still don't know why not, but that's a different issue (also hit dart-lang/build#2835 in the process).

However, includeing sub_package/node_modules/** as test on whether the working directory was somehow wrong is how I encountered this bug.

Dart SDK Version

$ flutter pub --version
Flutter 2.10.4 • channel stable • https://github.com/flutter/flutter.git
Framework • revision c860cba910 (4 weeks ago) • 2022-03-25 00:23:12 -0500
Engine • revision 57d3bac3dd
Tools • Dart 2.16.2 • DevTools 2.9.2

What package(s) from this repo you are using, and the version

build: 2.3.0
build_config: 1.0.0
build_runner: 2.1.10

Whether you are using Windows, MacOSX, or Linux

Windows 11

[jakemac53]

It looks like we have a test which is generally supposed to be checking this and runs on windows.

Can you run the same build with --verbose so we can get a larger stack trace? I want to try and figure out where in the system that error is coming from.

[Xiphoseer]

@jakemac53 Can you run the same build with --verbose so we can get a larger stack trace?

$ dart run build_runner build --verbose
[INFO] Entrypoint:Generating build script...
[INFO] Entrypoint:Generating build script completed, took 232ms

[WARNING] Bootstrap:
Invalidated precompiled build script due to missing asset graph.
[INFO] Bootstrap:Precompiling build script......
[INFO] Bootstrap:Precompiling build script... completed, took 673ms

[SEVERE] Build:
Nothing can be built, yet a build was requested.
[INFO] BuildDefinition:Initializing inputs


You have hit a bug in build_runner
Please file an issue with reproduction steps at https://github.com/dart-lang/build/issues


FileSystemException: Directory listing failed, path = 'c:\users\<path>\not\*' (OS Error: Das System kann den angegebenen Pfad nicht finden.
, errno = 3)
Instance of '_StringStackTrace'

[Xiphoseer]

For comparison, this is the WSL/Ubuntu output for the package:

$  dart run build_runner build --verbose
[INFO] Entrypoint:Generating build script...
[INFO] Entrypoint:Generating build script completed, took 223ms

[INFO] Bootstrap:Precompiling build script......
[INFO] Bootstrap:Precompiling build script... completed, took 4.2s

[FINE] Bootstrap:Core package locations file does not exist
[SEVERE] Build:
Nothing can be built, yet a build was requested.
[INFO] BuildDefinition:Initializing inputs
[INFO] BuildDefinition:Building new asset graph...
[INFO] BuildDefinition:Building new asset graph completed, took 1.6s

[INFO] BuildDefinition:Checking for unexpected pre-existing outputs....
[INFO] BuildDefinition:Checking for unexpected pre-existing outputs. completed, took 1ms

[INFO] Build:Running build...
[INFO] Build:Running build completed, took 2ms

[INFO] Build:Caching finalized dependency graph...
[INFO] Build:Caching finalized dependency graph completed, took 32ms

[INFO] Build:Succeeded after 46ms with 0 outputs (0 actions)

[jakemac53]

Hmm, it still didn't seem to give the actual stack trace unfortunately :/. I am not sure what is different for you compared to what is happening in our test. I wonder if its a windows 11 specific thing? Not sure which version of windows we are on.

[jakemac53]

As far as I can tell this must be coming from here, so likely an issue with the Glob package itself.

[jakemac53]

Ahah! I have a repro for this issue with the glob package. It happens if you have an explicit subdir listed whos parent dir does not exist (so foo/bar/** fails but foo/** succeeds). Moving to that repo, but I will work on a fix there.

[jakemac53]

Actually, there appear to be some tests today which are validating the existing behavior.

cc @natebosch wdyt?

[natebosch]

The behavior in glob seems strange to me, it would make more sense to me to drop that exception, but if it's too breaking it's not worth the effort to push it out. I'd be fine with either changing glob it it's fairly easy, or catching the exception in build_runner if we're worried other code depends on the current behavior.

[jakemac53]

cc @nex3 would you be opposed to a change here?

Specifically today, you can call list on a glob like foo/** even if foo does not exist. But if there are more directories (such as foo/bar/**) it throws. That behavior is tested today, which implies it is that way for a reason, but it seems surprising to me.

I doubt changing it to not throw would break anything, but maybe you are aware of some specific patterns or tools (sass?) that rely on that behavior.

[nex3]

I don't have any problem changing that behavior. Since it's changing an error to a non-error, it's not even technically a breaking change.

[xvrh]

Can we have a pub release of the Glob package 2.1.0 with this new behaviour (non throwing on missing directory)?

Thanks.

[natebosch]

#61

[natebosch]

https://pub.dev/packages/glob/versions/2.1.0