Updating package registry snapshot distribution version #89776

ycombinator · 2021-01-29T19:13:23Z

This PR updates the version of the snapshot package registry that's used by Fleet tests.

ycombinator · 2021-02-01T14:05:38Z

cc: @mtojek @ruflin @ph

nchaulet · 2021-02-01T14:41:18Z

@ycombinator looks like some test need to be updated, I can do it if you want.

ycombinator · 2021-02-01T16:13:09Z

looks like some test need to be updated, I can do it if you want.

@nchaulet that would be great, if you don't mind. Thanks!

ycombinator · 2021-02-01T16:14:25Z

@nchaulet I just invited you to be a collaborator on my kibana fork, if you just want to push commits to this PR's branch there.

…-reg-dist-version

elasticmachine · 2021-02-01T18:12:17Z

Pinging @elastic/fleet (Team:Fleet)

nchaulet · 2021-02-01T18:31:45Z

Currently test are failing because we are trying to create an existing transform (related to #89802)

@skh What are you though on that should we ignore that error when we create an already existing transform?

skh · 2021-02-02T11:53:14Z

@skh What are you though on that should we ignore that error when we create an already existing transform?

I agree to @kevinlog 's comment on #89802 , let's ignore it.

ycombinator · 2021-02-02T13:38:47Z

I agree to @kevinlog 's comment on #89802 , let's ignore it.

Does this mean this PR can be merged as-is, or is there some more work needed to mark a test as ignored or something like that?

nchaulet · 2021-02-02T18:44:24Z

Does this mean this PR can be merged as-is, or is there some more work needed to mark a test as ignored or something like that?

There is some work to do to fix the issue, I will create a separate PR for that.

ycombinator · 2021-02-02T18:48:20Z

There is some work to do to fix the issue, I will create a separate PR for that.

Thanks. If it makes sense for you to push commits to this PR directly, please feel free to do so. I sent you an invite to collaborate on my kibana fork.

…-reg-dist-version

nchaulet

🚀

ycombinator · 2021-02-02T23:44:20Z

Test failure looks relevant, @nchaulet?

             └- ✖ fail: Fleet Endpoints fleet_agent_flow should work
             │      Error: Agent should run the 'system' package
             │       at Assertion.assert (/dev/shm/workspace/parallel/1/kibana/packages/kbn-expect/expect.js:100:11)
             │       at Assertion.contain (/dev/shm/workspace/parallel/1/kibana/packages/kbn-expect/expect.js:447:10)
             │       at Context.<anonymous> (test/fleet_api_integration/apis/agents/complete_flow.ts:118:52)
             │       at Object.apply (/dev/shm/workspace/parallel/1/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

nchaulet · 2021-02-02T23:53:41Z

Yes the error is relevant I missed a part during the install transform

[00:01:40]               │ proc [kibana]   log   [23:24:35.257] [error][fleet][plugins] [status_exception] Cannot start transform [endpoint.metadata_current-default-0.17.0] as it is already started. response from /_transform/endpoint.metadata_current-default-0.17.0/_start: {"error":{"root_cause":[{"type":"status_exception","reason":"Cannot start transform [endpoint.metadata_current-default-0.17.0] as it is already started."}],"type":"status_exception","reason":"Cannot start transform [endpoint.metadata_current-default-0.17.0] as it is already started."},"status":409}

skh · 2021-02-03T13:54:49Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/install.ts

+      body: transform.content,
+    });
+  } catch (err) {
+    // swallow the error if the transform already exists.


LGTM, but I'd like to double-check with @kevinlog that this is safe.

this looks okay with me - I will get @pzl to comment as well asap

putting the same version, yeah we should logically be able to assume (aside from local dev situations) that it contains the same content trying to be installed then. Looks good.

Generally, the transform is deleted before install, right? I wonder why that isn't happening here

The transform is deleted if the saved object contains the transform, if our saved object and the state of ES differ this could happen

skh · 2021-02-03T13:55:06Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/install.ts

  await callCluster('transport.request', {
    method: 'POST',
    path: `/_transform/${transform.installationName}/_start`,
+    // Ignore error if the transform is already started
+    ignore: [409],


Same as above, cc @kevinlog

same here, looks OK - will check with @pzl

as long as we don't think we're swallowing other problems that might be in a 409, sure

ycombinator · 2021-02-04T11:53:39Z

@elasticmachine merge upstream

skh · 2021-02-04T12:29:17Z

I think we can merge this once CI is happy -- @kevinlog any objections? I believe this should fix #89802 as well?

kevinlog · 2021-02-04T12:34:31Z

Looks OK to me - it seems we'd keep moving and not error when a transform of the same version is already installed. This seems OK to me as we'd always increment the version on the Endpoint package if there were new mappings, components, etc.

I'll get the final 👍 from @pzl

pzl

The only thing I would worry about is the ignore all 409's when starting

We might end up with a false confidence if other classes of errors come back with 409's. If we think a transform is running, and it is not.. then a lot of data will never populate.

That said, none of these changes are introducing anything worse, so it's 👍 , just want to point out that caution

kibanamachine · 2021-02-04T13:36:47Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: cb762ff

Metrics [docs]

✅ unchanged

History

💚 Build #103402 succeeded 854a251
💔 Build #103274 failed 289e005
💔 Build #103253 failed 5333b48
💔 Build #103194 failed f701a13
💔 Build #102691 failed 586b3cb

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

)

jen-huang · 2021-02-04T20:06:49Z

Thanks for this @ycombinator @nchaulet! We haven't updated the distribution in a while :)

ycombinator · 2021-02-04T21:52:47Z

@jen-huang Going forward, the distribution should be updated more frequently. I've added a manual step to the package registry release process (see last step in https://github.com/elastic/package-registry/blob/master/README.md#release).

ycombinator added 3 commits January 29, 2021 11:06

Updating package registry snapshot distribution version

e46addd

Updating comment to be clearer

798fe9c

Updating link to build

be6f22e

ycombinator marked this pull request as ready for review February 1, 2021 14:04

ycombinator requested review from jonathan-buttner, skh, jen-huang and nchaulet February 1, 2021 14:05

Merge branch 'master' of github.com:elastic/kibana into fleet-upd-pkg…

586b3cb

…-reg-dist-version

nchaulet mentioned this pull request Feb 1, 2021

[Bug] When loading server after upgrades and reverts, the Endpoint package failed to install momentarily in Fleet start #89802

Open

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Feb 1, 2021

nchaulet added release_note:skip Skip the PR/issue when compiling release notes v7.12.0 v8.0.0 labels Feb 1, 2021

nchaulet added 2 commits February 2, 2021 16:02

Merge branch 'master' of github.com:elastic/kibana into fleet-upd-pkg…

f144483

…-reg-dist-version

Ignore transform errors if already exists

f701a13

nchaulet approved these changes Feb 2, 2021

View reviewed changes

Ignore error when started an already started transform

5333b48

nchaulet added 2 commits February 2, 2021 20:20

Fix unit test

289e005

Fix last test

854a251

skh reviewed Feb 3, 2021

View reviewed changes

Merge branch 'master' into fleet-upd-pkg-reg-dist-version

cb762ff

skh approved these changes Feb 4, 2021

View reviewed changes

pzl approved these changes Feb 4, 2021

View reviewed changes

nchaulet merged commit f3a9c76 into elastic:master Feb 4, 2021

nchaulet mentioned this pull request Feb 4, 2021

[7.x] Updating package registry snapshot distribution version (#89776) #90278

Merged

nchaulet pushed a commit to nchaulet/kibana that referenced this pull request Feb 4, 2021

Updating package registry snapshot distribution version (elastic#89776)

e571de6

nchaulet added a commit that referenced this pull request Feb 4, 2021

Updating package registry snapshot distribution version (#89776) (#90278

378764b

)

ycombinator deleted the fleet-upd-pkg-reg-dist-version branch February 4, 2021 21:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updating package registry snapshot distribution version #89776

Updating package registry snapshot distribution version #89776

ycombinator commented Jan 29, 2021 •

edited

Loading

ycombinator commented Feb 1, 2021

nchaulet commented Feb 1, 2021

ycombinator commented Feb 1, 2021

ycombinator commented Feb 1, 2021

elasticmachine commented Feb 1, 2021

nchaulet commented Feb 1, 2021

skh commented Feb 2, 2021

ycombinator commented Feb 2, 2021

nchaulet commented Feb 2, 2021

ycombinator commented Feb 2, 2021

nchaulet left a comment

ycombinator commented Feb 2, 2021

nchaulet commented Feb 2, 2021

skh Feb 3, 2021

kevinlog Feb 4, 2021

pzl Feb 4, 2021

nchaulet Feb 4, 2021

skh Feb 3, 2021

kevinlog Feb 4, 2021

pzl Feb 4, 2021

ycombinator commented Feb 4, 2021

skh commented Feb 4, 2021

kevinlog commented Feb 4, 2021

pzl left a comment

kibanamachine commented Feb 4, 2021

jen-huang commented Feb 4, 2021

ycombinator commented Feb 4, 2021

Updating package registry snapshot distribution version #89776

Updating package registry snapshot distribution version #89776

Conversation

ycombinator commented Jan 29, 2021 • edited Loading

ycombinator commented Feb 1, 2021

nchaulet commented Feb 1, 2021

ycombinator commented Feb 1, 2021

ycombinator commented Feb 1, 2021

elasticmachine commented Feb 1, 2021

nchaulet commented Feb 1, 2021

skh commented Feb 2, 2021

ycombinator commented Feb 2, 2021

nchaulet commented Feb 2, 2021

ycombinator commented Feb 2, 2021

nchaulet left a comment

Choose a reason for hiding this comment

ycombinator commented Feb 2, 2021

nchaulet commented Feb 2, 2021

skh Feb 3, 2021

Choose a reason for hiding this comment

kevinlog Feb 4, 2021

Choose a reason for hiding this comment

pzl Feb 4, 2021

Choose a reason for hiding this comment

nchaulet Feb 4, 2021

Choose a reason for hiding this comment

skh Feb 3, 2021

Choose a reason for hiding this comment

kevinlog Feb 4, 2021

Choose a reason for hiding this comment

pzl Feb 4, 2021

Choose a reason for hiding this comment

ycombinator commented Feb 4, 2021

skh commented Feb 4, 2021

kevinlog commented Feb 4, 2021

pzl left a comment

Choose a reason for hiding this comment

kibanamachine commented Feb 4, 2021

💚 Build Succeeded

Metrics [docs]

History

jen-huang commented Feb 4, 2021

ycombinator commented Feb 4, 2021

ycombinator commented Jan 29, 2021 •

edited

Loading