[Ingest Manager] 7.9 Cannot access Ingest Manager due to 'package not found' 500 server response from api/ingest_manager/setup

[EricDavisX]

Kibana version:
7.9.0 latest SNAPSHOT Cloud-staging deploy as of 10:30 AM Sunday July 5

Browser version:
Chrome on macOS

Describe the bug:
Ingest not available due to package / registry problem (we know changes were done recently on Registry side... maybe the 7.9 latest SNAPSHOT build is not new enough to properly take advantage, but this seemed worthwhile to log and confirm instead of ignoring it.

Steps to reproduce:

1. deploy latest 7.9 snapshot to cloud, I used staging
2. click on 'Ingest Manager' in left hand side-bar
3. see error on main page

None of the pages respond with any different UI, that I see. And the API noted gives this response:

api/ingest_manager/setup
{"statusCode":500,"error":"Internal Server Error","message":"package not found"}

Expected behavior:
app is available and works!

Screenshots (if relevant):

[EricDavisX]

I don't know if anyone will know the goings on here better than maybe Sonja? @skh can I assign to you just to help confirm whats up, if you are able to? We can re-assign as needed to fix after initial triaging. Thank you in advance.

[EricDavisX]

I'll note that the 8.0 latest SNAPSHOT behavior is different... it works well enough to get to Ingest Manager, but it does not successfully deploy an Agent either, see separate bug here:
#70726

[EricDavisX]

tagging @mtojek as he knows much and may be able to help too? We're anxious and very urgently awaiting help on this on Monday to unblock the Endpoint team from finishing dev. Thanks all. @ruflin

[skh]

My first guess is that after all the changes that went in last week, we need very current snapshots of all components in order for everything to work together again.

Assuming this error is because of the base package missing, it was removed from requirements in this backport: #70645

Can you check on the /status page (just append /status to the main url, without /app or anything) what commit the snapshot was built from?

[EricDavisX]

it says: BUILD 34285 COMMIT a921bbf

• we know its old. There is a problem in ES that is failing the build. I'm uncertain, with the packages changes, do we explicitly need a newer version of Elasticsearch built as well for the system to work? If so, the 8.0 line won't be unblocked until ES is fixed (not just skipped in the snapshot build). I don't have links to that issue handy, but I had seen it somewhere in chat or email.

[skh]

do we explicitly need a newer version of Elasticsearch built as well for the system to work?

From what I know, I don't think so.

it says: BUILD 34285 COMMIT a921bbf

That's a master build, not 7.x, is that correct? If yes, there's something else going awry -- that commit came in after the change was merged which removes the base package ( 5b8fb95 )

[skh]

Cc @jonathan-buttner as he was involved with the latest changes to default packages too. Any ideas what could be wrong here?

[EricDavisX]

Oh I'm sorry, I got mixed up on which branch I should post for which issue. This is my failing 7.9 commit:
BUILD 32848 COMMIT accfd2f

[skh]

BUILD 32848 COMMIT accfd2f

Ok, that one is indeed missing 7d6e1d7 so a newer snapshot, if you can get one, will hopefully resolve your issue.

[jonathan-buttner]

@EricDavisX Hmm, Yeah see if it gets fixed with a new snapshot version of ES. The issue is probably related to the fact that we removed the base package since ES now ships with that.

[skh]

since ES now ships with that.

That means that you DO need a fairly new elasticsearch too, or you'll see other errors because of missing mappings. @jonathan-buttner do you know when that change was added to ES?

[jonathan-buttner]

since ES now ships with that.

That means that you DO need a fairly new elasticsearch too, or you'll see other errors because of missing mappings. @jonathan-buttner do you know when that change was added to ES?

The change for ES went in 6 days ago: elastic/elasticsearch@3b68df2

It should definitely been in the snapshot build of ES that's available today. It was in the snapshot that was available last Thursday too.

[jonathan-buttner]

BUILD 32848 COMMIT accfd2f

Ok, that one is indeed missing 7d6e1d7 so a newer snapshot, if you can get one, will hopefully resolve your issue.

Oh you're right, we need the removal of the base package commit.

[EricDavisX]

we think this is fixed if we were to have the latest 7.9 snapshots (and nothing else broke the integration) track the builds here: https://internal-ci.elastic.co/job/elastic+release-manager+7.x+unified-snapshot/485/

[EricDavisX]

we got new builds of 7.9 artifacts this morning (last night it wasn't working). it looks like this error isn't happening. the new build can install the new agent. so closing yay.