Update pkg

[Page-]

The isSocketValue patch was merged in the latest 4.x of pkg which means we can drop it from the cli 🎉

[ghost]

Your landr site preview has been successfully deployed to https://landr-balena-io-repo-balena-cli-preview-2428.netlify.app

Deployed with Landr 6.36.2

[pdcastro]

Let's see what this one does, and test it. I seem to remember issues preventing the update of pkg, and I was simply hoping to replace pkg with caxa (I've done some work towards that, but it is not ready). I will try to find the thread where the pkg problems were discussed.

[pdcastro]

Some notes I found when I last attempted it back in May 2021:

--- Everything below this line is a copy-and-paste from May 2021 ---
Original thread: https://www.flowdock.com/app/rulemotion/i-cli/threads/lqsgv7orNqQAinI5hwKYDs66nLw

• Updating to pkg v4.5.0 or v4.5.1 (latest v4) causes balena os configure to fail with:

TypeError: handle.close is not a function
    at Object.withOpenFile (/snapshot/balena-cli/node_modules/file-disk/build/index.js:109:22)

I haven't dug there yet, but it sounds like the kind of issue that we previously patched pkg for.

• Updating to pkg v5.0.0 or v5.1.0 (latest v5) causes several CLI commands to fail with:

dyld: lazy symbol binding failed: Symbol not found: _napi_module_register
  Referenced from: /var/folders/1s/52g85bgj3wlchqh6szld3fp40000gn/T/f2e40016d01c60e8913da09524e6322533f0a40147a7871226a7f82bd016a877/node.napi.node
  Expected in: flat namespace

dyld: Symbol not found: _napi_module_register
  Referenced from: /var/folders/1s/52g85bgj3wlchqh6szld3fp40000gn/T/f2e40016d01c60e8913da09524e6322533f0a40147a7871226a7f82bd016a877/node.napi.node
  Expected in: flat namespace

I see that someone is working on a new pkg-fetch release (v3.1.0) that might fix the Symbol not found issue above -- fingers crossed.

• As for the pkg v4.5.0 feature of bundling native node modules (.node files) in the single-ish large executable, it relies on scanning source code files to find clauses like require('fsevents.node') which, guess what, failed for some CLI dependencies in the same way that such scanning fails for non-native-module assets: dynamically computed require clauses. And there's no easy way to identify which .node files were missed by the scanning. I think (not sure because I couldn't fully test because of the other bugs above) that it would be possible to tell pkg to bundle all .node files by adding **/*.node to the pkg section of the CLI's package.json file.

While googling for the pkg problems above, I came across another project, caxa -

• https://www.npmjs.com/package/caxa
• https://github.com/leafac/caxa#readme

It's a new project, v1.0.0 published 2 months ago, but it builds on the tried and tested approach of self extracting executables (if you're neither too young nor too old, you may remember self extracting zip files for Windows!). caxa's Readme is surprisingly thorough and the author even created YouTube videos. It struck the right chord with me, especially as I found it at the right "pkg v5.1.0 frustration" time. :-) Some of caxa's highlights, compared to pkg:

• Doesn’t patch the Node.js source code, doesn’t build Node.js from source. Ergo, no pkg-like bugs resulting from pkg's patching of the fs module for pkg's "snapshot filesystem".
• Takes whatever existing version of Node.js is available locally, including Node.js from any balenalib image that we already build for arm v6 etc, and literally just copies the Node.js executable.
• No traversal of require() trying to find which files to include: No need to hardcode assets in the CLI's package.json file.
• No limitations/bugs around native modules.
• All the positives of pkg that we care about in a CLI installer for Linux (as far as I can tell), without the worst negatives.
• caxa extracts to a temporary directory by default, but if we were otherwise happy to adopt it, I'd submit a PR to allow the extraction directory to be chosen ([Feature] Allow specifying alternative extraction / temporary directory leafac/caxa#20). I've already checked their relatively concise source code, it should be easy enough to do it, and it would be a more rewarding "do it once and forget" simple feature PR than the never ending patch PRs to fix pkg incompatibilities in reaction to bug reports by CLI end users.
• Could also be helpful in combination with the CLI's existing GUI installers for Windows and macOS, to improve installation and update time.
• https://github.com/leafac/caxa#features

In addition to being a self extracting executable, caxa adds some convenience features for Node.js applications, like running npm prune --production prior to compressing, and allowing the application's entry point to be customized.

I have added a row for caxa in the comparison spreadsheet: https://docs.google.com/spreadsheets/d/1VboFnh0U74Nl0xwdesG6z3pnMfPPFcn19_NrD6SYxKQ/edit#gid=0

[Page-]

@balena-ci retest

[pdcastro]

The build is still failing, but now with a brand new "Package Invalid" notarization error:

Notarizing package...
Error: "build:installer": Apple failed to notarize your application, check the logs for more info

Status Code: 2
Message: Package Invalid
Logs: https://osxapps-ssl.itunes.apple.com/itunes-assets/Enigma126/v4/d2/ca/7d/d2ca7dbb-cf09-ad73-9c00-e4d49cec3fa2/developer_log.json?accessKey=1641090823_3469808323622775986_KEUuaUZk6OeDrWCqvu6n5CovwwXU3jpsjMDCuFrmf3kC%2FEZAoRLMpZV2vTSmsDKuacg5d3Ra4J4JEceE3ByfA5KV7VHfrx3hmPDlAwjDCUxqr8iitKy7XuATICk1khY02dRdTTYU5M3RBExscqBdz6VDCg81Hsbyw4BPj3%2BrcvI%3D

The logs from the URL above read:

{
  "logFormatVersion": 1,
  "jobId": "b44228b6-a8bc-4321-900c-45ac5fb61d8e",
  "status": "Invalid",
  "statusSummary": "Archive contains critical validation errors",
  "statusCode": 4000,
  "archiveFilename": "balena-cli-v13.1.2-macOS-x64-installer.pkg.zip",
  "uploadDate": "2021-12-30T20:24:38Z",
  "sha256": "cd768bf16970f4aefcde7b04530ed34708ee53ccd63fff2722a78d79d372285b",
  "ticketContents": null,
  "issues": [
    {
      "severity": "warning",
      "code": null,
      "path": "balena-cli-v13.1.2-macOS-x64-installer.pkg.zip/balena-cli-v13.1.2-macOS-x64-installer.pkg/balena-cli-v13.1.2-macOS-x64-installer.pkg Contents/Payload/usr/local/lib/balena-cli/node_modules/unzip-stream/aa.zip",
      "message": "b\"ditto: stdin: No such file or directory\\nditto: Couldn't read pkzip signature.\\n\"",
      "docUrl": null,
      "architecture": null
    },
    {
      "severity": "error",
      "code": null,
      "path": "balena-cli-v13.1.2-macOS-x64-installer.pkg.zip/balena-cli-v13.1.2-macOS-x64-installer.pkg/balena-cli-v13.1.2-macOS-x64-installer.pkg Contents/Payload/usr/local/lib/balena-cli/node_modules/@ronomon/direct-io/binding.node.bak",
      "message": "The binary is not signed.",
      "docUrl": null,
      "architecture": "x86_64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "balena-cli-v13.1.2-macOS-x64-installer.pkg.zip/balena-cli-v13.1.2-macOS-x64-installer.pkg/balena-cli-v13.1.2-macOS-x64-installer.pkg Contents/Payload/usr/local/lib/balena-cli/node_modules/@ronomon/direct-io/binding.node.bak",
      "message": "The signature does not include a secure timestamp.",
      "docUrl": null,
      "architecture": "x86_64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "balena-cli-v13.1.2-macOS-x64-installer.pkg.zip/balena-cli-v13.1.2-macOS-x64-installer.pkg/balena-cli-v13.1.2-macOS-x64-installer.pkg Contents/Payload/usr/local/lib/balena-cli/node_modules/xxhash/build/Release/hash.node.bak",
      "message": "The binary is not signed.",
      "docUrl": null,
      "architecture": "x86_64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "balena-cli-v13.1.2-macOS-x64-installer.pkg.zip/balena-cli-v13.1.2-macOS-x64-installer.pkg/balena-cli-v13.1.2-macOS-x64-installer.pkg Contents/Payload/usr/local/lib/balena-cli/node_modules/xxhash/build/Release/hash.node.bak",
      "message": "The signature does not include a secure timestamp.",
      "docUrl": null,
      "architecture": "x86_64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "balena-cli-v13.1.2-macOS-x64-installer.pkg.zip/balena-cli-v13.1.2-macOS-x64-installer.pkg/balena-cli-v13.1.2-macOS-x64-installer.pkg Contents/Payload/usr/local/lib/balena-cli/node_modules/fsevents/fsevents.node.bak",
      "message": "The binary is not signed.",
      "docUrl": null,
      "architecture": "x86_64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "balena-cli-v13.1.2-macOS-x64-installer.pkg.zip/balena-cli-v13.1.2-macOS-x64-installer.pkg/balena-cli-v13.1.2-macOS-x64-installer.pkg Contents/Payload/usr/local/lib/balena-cli/node_modules/fsevents/fsevents.node.bak",
      "message": "The signature does not include a secure timestamp.",
      "docUrl": null,
      "architecture": "x86_64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "balena-cli-v13.1.2-macOS-x64-installer.pkg.zip/balena-cli-v13.1.2-macOS-x64-installer.pkg/balena-cli-v13.1.2-macOS-x64-installer.pkg Contents/Payload/usr/local/lib/balena-cli/node_modules/fsevents/fsevents.node.bak",
      "message": "The binary is not signed with a valid Developer ID certificate.",
      "docUrl": null,
      "architecture": "arm64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "balena-cli-v13.1.2-macOS-x64-installer.pkg.zip/balena-cli-v13.1.2-macOS-x64-installer.pkg/balena-cli-v13.1.2-macOS-x64-installer.pkg Contents/Payload/usr/local/lib/balena-cli/node_modules/fsevents/fsevents.node.bak",
      "message": "The signature does not include a secure timestamp.",
      "docUrl": null,
      "architecture": "arm64"
    }
  ]
}

I guess the fix would be to amend the following code:

• Exclude further zip files (listed in the logs above) at
  https://github.com/balena-io/balena-cli/blob/master/automation/build-bin.ts#L318-L335
• Include further binary files (listed in the logs above) at
  https://github.com/balena-io/balena-cli/blob/master/automation/build-bin.ts#L364-L396

[Page-]

@balena-ci retest

[Page-]

@balena-ci retest

[Page-]

@balena-ci I self-certify!