Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Files inside directories aren't included when running pack #2498

Closed
squiddy opened this issue Jan 19, 2017 · 5 comments · Fixed by #3175 or GulajavaMinistudio/yarn#24
Closed

Files inside directories aren't included when running pack #2498

squiddy opened this issue Jan 19, 2017 · 5 comments · Fixed by #3175 or GulajavaMinistudio/yarn#24

Comments

@squiddy
Copy link

squiddy commented Jan 19, 2017

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Package built with yarn pack doesn't include files inside directories.

If the current behavior is a bug, please provide the steps to reproduce.

$ yarn init
$ mkdir a
$ touch a/b

Edit package.json and add files setting:

{
  "name": "tmp_package",
  "version": "1.0.0",
  "main": "index.js",
  "repository": {},
  "license": "MIT",
  "files": [
	  "a/"
  ]
}
$ yarn pack
$ tar --list -zf tmp_package-v1.0.0.tgz
package
package/a
package/package.json

What is the expected behavior?

The package includes file a/b.

$ tar --list -zf tmp_package-v1.0.0.tgz
package
package/a/b
package/package.json

When running npm pack the file is included.

Perhaps this is merely an documentation issue:

If you name a folder in the array, then it will also include the files inside that folder. (Unless they would be ignored by another rule.)

https://docs.npmjs.com/files/package.json#files

You can specify single files, whole directories or [...]
https://yarnpkg.com/en/docs/package-json#toc-files

Please mention your node.js, yarn and operating system version.

node: 6.6.0
yarn: 0.19.1
os: Linux 4.8.13-1-ARCH #1 SMP PREEMPT Fri Dec 9 07:24:34 CET 2016 x86_64 GNU/Linux
@squiddy
Copy link
Author

squiddy commented Jan 19, 2017

On a related note, packages built with yarn pack that include directories in the files setting aren't installable.

Trying to install the package I build in the steps above results in this error:

$ yarn add file:/tmp/tmp_package/tmp_package-v1.0.0.tgz                                                                                                                                                                                [11:57]
yarn add v0.19.1
[1/4] Resolving packages...
error An unexpected error occurred: "invalid tar file".
info If you think this is a bug, please open a bug report with the information provided in "/tmp/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

Same error as in #1253

@ghost
Copy link

ghost commented Feb 6, 2017

We are seeing this issue too. Replacing "foldername/" with "foldername/*", "foldername/**/*" appears to be a workaround.

@jamiewinder
Copy link

I've just encountered this too, but the workaround @majgis posted doesn't seem to work either:

  "files": [
    "dist/",
    "dist/*",
    "dist/**/*",
    (...)
  ]

Doesn't include anything from 'dist'. Tried every combination of these I can think of, and only the 'dist' directory is included, but never the contents.

@faassen
Copy link

faassen commented Mar 13, 2017

I ran into this too. To @jamiewinder -- it worked for me but I didn't include "dist/" itself, just "dist/" and "dist/**/". And then a lot of struggling due to the old situation being in the yarn cache after installation.

@fvgs
Copy link

fvgs commented Mar 17, 2017

I encountered this same issue and found "files": ["dirName/**"] to be the cleanest workaround. Based on my tests, yarn will include all contents of the dirName/ directory.

josephfrazier added a commit to josephfrazier/prettier-diff that referenced this issue Apr 16, 2017
Work around `yarn publish` being broken: yarnpkg/yarn#2498
josephfrazier added a commit to josephfrazier/prettier_d that referenced this issue Apr 16, 2017
josephfrazier added a commit to josephfrazier/yarn that referenced this issue Apr 18, 2017
josephfrazier added a commit to josephfrazier/yarn that referenced this issue Apr 18, 2017
This makes it so that you don't have to put '/**' after a directory in
the `files` field of package.json to ensure that the contents of the
directory will be published.

Fixes yarnpkg#2498
Fixes yarnpkg#2942
Fixes yarnpkg#2851

Includes and closes yarnpkg#3170
arcanis pushed a commit that referenced this issue Apr 28, 2017
* Use tar-fs instead of tar-stream in `yarn pack` (and fix packed emojis)

This lets tar-fs do the [header construction] for us.

[header construction]: https://github.com/mafintosh/tar-fs/blob/b79d82a79c5e21f6187462d7daaba1fc03cdd1de/index.js#L101-L127

I tested this by comparing the output of this command before and after
the change:

    ./bin/yarn.js pack >/dev/null && tar tvf yarn-v0.24.0-0.tgz | sort && wc -c < yarn-v0.24.0-0.tgz && rm *tgz

Here's the diff between the outputs:

```diff
diff --git a/before.txt b/after.txt
index 5e7f370e..5565a808 100644
--- a/before.txt
+++ b/after.txt
@@ -7,13 +7,13 @@
 -rw-r--r--  0 0      0         657 Mar  4 07:19 package/Dockerfile.dev
 -rw-r--r--  0 0      0        1346 Mar  4 07:19 package/LICENSE
 -rw-r--r--  0 0      0        1789 Apr 17 15:10 package/jenkins_jobs.groovy
--rw-r--r--  0 0      0        3061 Mar  4 07:19 package/README.md
--rw-r--r--  0 0      0        3438 Apr 17 16:18 package/package.json
+-rw-r--r--  0 0      0        3057 Mar  4 07:19 package/README.md
+-rw-r--r--  0 0      0        3430 Apr 17 16:18 package/package.json
 -rwxr-xr-x  0 0      0          42 Mar  4 07:19 package/bin/yarnpkg
 -rwxr-xr-x  0 0      0         172 Mar  4 07:19 package/bin/node-gyp-bin/node-gyp
 -rwxr-xr-x  0 0      0         906 Mar  4 07:19 package/bin/yarn
 -rwxr-xr-x  0 0      0         929 Apr 10 15:59 package/bin/yarn.js
 drwxr-xr-x  0 0      0           0 Apr 10 15:59 package/bin
 drwxr-xr-x  0 0      0           0 Apr 17 17:04 package
 drwxr-xr-x  0 0      0           0 Mar  4 07:19 package/bin/node-gyp-bin
-    6206
+    6177
```

I extracted the tarballs into `./package-master` and `./package-feature`,
then diffed them to find that this change has the side effect of
fixing emojis in the tarball. You can see examples of the broken emoji
here:

* https://unpkg.com/yarn@0.22.0/package.json
* https://unpkg.com/yarn@0.22.0/README.md

```diff
diff --git a/package-master/README.md b/package-feature/README.md
index aabfc24f..6aff13d8 100644
--- a/package-master/README.md
+++ b/package-feature/README.md
@@ -30,7 +30,7 @@
 * **Network Performance.** Yarn efficiently queues up requests and avoids request waterfalls in order to maximize network utilization.
 * **Network Resilience.** A single request failing won't cause an install to fail. Requests are retried upon failure.
 * **Flat Mode.** Yarn resolves mismatched versions of dependencies to a single version to avoid creating duplicates.
-* **More emojis.** �
+* **More emojis.** 🐈

 ## Installing Yarn

diff --git a/package-master/package.json b/package-feature/package.json
index c89ad7a6..8e7e3bc7 100644
--- a/package-master/package.json
+++ b/package-feature/package.json
@@ -4,7 +4,7 @@
   "version": "0.24.0-0",
   "license": "BSD-2-Clause",
   "preferGlobal": true,
-  "description": "�� Fast, reliable, and secure dependency management.",
+  "description": "📦🐈 Fast, reliable, and secure dependency management.",
   "dependencies": {
     "babel-runtime": "^6.0.0",
     "bytes": "^2.4.0",
```

* When testing `yarn pack`, use fs.walk instead of fs.readdir

This ensures that files inside directories are listed too.

* Add failing test for packing directories recursively

#2498

* `pack`: include contents of directories in `files` field

This makes it so that you don't have to put '/**' after a directory in
the `files` field of package.json to ensure that the contents of the
directory will be published.

Fixes #2498
Fixes #2942
Fixes #2851

Includes and closes #3170

* `pack` test: Use path.join() to create nested path

* `path` test: Make output easier to understand

Now, we can see just what the expected/actual difference is, rather than
just getting a -1 vs 0 from an indexOf test.

* `pack`: transform each [ "file-name" ] into [ "file-name", "file-name/**" ], whether it's a file or a folder

See #3175 (comment)

* Account for backslashes in paths when filtering files

See #3175 (comment)

* Use `path.sep` instead of slashes

See #3175 (comment)

* Revert "Use `path.sep` instead of slashes"

This reverts commit c2df043.

It caused an additional test to fail:
https://ci.appveyor.com/project/kittens/yarn/build/2195/job/q5u26f85qlroy533#L3011

* Revert "Account for backslashes in paths when filtering files"

This reverts commit 20646f5.

I don't think it actually helps, see
#3175 (comment)

* Keep pattern in IgnoreFilter, use with minimatch() in matchesFilter

This should help with Windows support. See
#3175 (comment)

* Update ignoreLinesToRegex tests
lovelypuppy0607 added a commit to lovelypuppy0607/yarn that referenced this issue May 11, 2023
* Use tar-fs instead of tar-stream in `yarn pack` (and fix packed emojis)

This lets tar-fs do the [header construction] for us.

[header construction]: https://github.com/mafintosh/tar-fs/blob/b79d82a79c5e21f6187462d7daaba1fc03cdd1de/index.js#L101-L127

I tested this by comparing the output of this command before and after
the change:

    ./bin/yarn.js pack >/dev/null && tar tvf yarn-v0.24.0-0.tgz | sort && wc -c < yarn-v0.24.0-0.tgz && rm *tgz

Here's the diff between the outputs:

```diff
diff --git a/before.txt b/after.txt
index 5e7f370e..5565a808 100644
--- a/before.txt
+++ b/after.txt
@@ -7,13 +7,13 @@
 -rw-r--r--  0 0      0         657 Mar  4 07:19 package/Dockerfile.dev
 -rw-r--r--  0 0      0        1346 Mar  4 07:19 package/LICENSE
 -rw-r--r--  0 0      0        1789 Apr 17 15:10 package/jenkins_jobs.groovy
--rw-r--r--  0 0      0        3061 Mar  4 07:19 package/README.md
--rw-r--r--  0 0      0        3438 Apr 17 16:18 package/package.json
+-rw-r--r--  0 0      0        3057 Mar  4 07:19 package/README.md
+-rw-r--r--  0 0      0        3430 Apr 17 16:18 package/package.json
 -rwxr-xr-x  0 0      0          42 Mar  4 07:19 package/bin/yarnpkg
 -rwxr-xr-x  0 0      0         172 Mar  4 07:19 package/bin/node-gyp-bin/node-gyp
 -rwxr-xr-x  0 0      0         906 Mar  4 07:19 package/bin/yarn
 -rwxr-xr-x  0 0      0         929 Apr 10 15:59 package/bin/yarn.js
 drwxr-xr-x  0 0      0           0 Apr 10 15:59 package/bin
 drwxr-xr-x  0 0      0           0 Apr 17 17:04 package
 drwxr-xr-x  0 0      0           0 Mar  4 07:19 package/bin/node-gyp-bin
-    6206
+    6177
```

I extracted the tarballs into `./package-master` and `./package-feature`,
then diffed them to find that this change has the side effect of
fixing emojis in the tarball. You can see examples of the broken emoji
here:

* https://unpkg.com/yarn@0.22.0/package.json
* https://unpkg.com/yarn@0.22.0/README.md

```diff
diff --git a/package-master/README.md b/package-feature/README.md
index aabfc24f..6aff13d8 100644
--- a/package-master/README.md
+++ b/package-feature/README.md
@@ -30,7 +30,7 @@
 * **Network Performance.** Yarn efficiently queues up requests and avoids request waterfalls in order to maximize network utilization.
 * **Network Resilience.** A single request failing won't cause an install to fail. Requests are retried upon failure.
 * **Flat Mode.** Yarn resolves mismatched versions of dependencies to a single version to avoid creating duplicates.
-* **More emojis.** �
+* **More emojis.** 🐈

 ## Installing Yarn

diff --git a/package-master/package.json b/package-feature/package.json
index c89ad7a6..8e7e3bc7 100644
--- a/package-master/package.json
+++ b/package-feature/package.json
@@ -4,7 +4,7 @@
   "version": "0.24.0-0",
   "license": "BSD-2-Clause",
   "preferGlobal": true,
-  "description": "�� Fast, reliable, and secure dependency management.",
+  "description": "📦🐈 Fast, reliable, and secure dependency management.",
   "dependencies": {
     "babel-runtime": "^6.0.0",
     "bytes": "^2.4.0",
```

* When testing `yarn pack`, use fs.walk instead of fs.readdir

This ensures that files inside directories are listed too.

* Add failing test for packing directories recursively

yarnpkg/yarn#2498

* `pack`: include contents of directories in `files` field

This makes it so that you don't have to put '/**' after a directory in
the `files` field of package.json to ensure that the contents of the
directory will be published.

Fixes yarnpkg/yarn#2498
Fixes yarnpkg/yarn#2942
Fixes yarnpkg/yarn#2851

Includes and closes yarnpkg/yarn#3170

* `pack` test: Use path.join() to create nested path

* `path` test: Make output easier to understand

Now, we can see just what the expected/actual difference is, rather than
just getting a -1 vs 0 from an indexOf test.

* `pack`: transform each [ "file-name" ] into [ "file-name", "file-name/**" ], whether it's a file or a folder

See yarnpkg/yarn#3175 (comment)

* Account for backslashes in paths when filtering files

See yarnpkg/yarn#3175 (comment)

* Use `path.sep` instead of slashes

See yarnpkg/yarn#3175 (comment)

* Revert "Use `path.sep` instead of slashes"

This reverts commit c2df043343092dcc408f8792ad16eb86cad6ba3a.

It caused an additional test to fail:
https://ci.appveyor.com/project/kittens/yarn/build/2195/job/q5u26f85qlroy533#L3011

* Revert "Account for backslashes in paths when filtering files"

This reverts commit 20646f5d4ac5207dbf929af4e96acebeca29d07f.

I don't think it actually helps, see
yarnpkg/yarn#3175 (comment)

* Keep pattern in IgnoreFilter, use with minimatch() in matchesFilter

This should help with Windows support. See
yarnpkg/yarn#3175 (comment)

* Update ignoreLinesToRegex tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants