Create an Ingest API #5213

Bargs · 2015-10-27T23:07:52Z

This feature was removed from 5.0. It does not currently have a target release because it will most likely require some changes.

Closes #5199

Documentation

If you like Postman, import this collection with example requests to play around with.

Endpoints

POST /api/kibana/ingest

Create a new index pattern and index template. Template mappings are given sensible defaults based on their type. Fields of type number will be created as a double in elasticsearch. Strings will be mapped as multi fields with an analyzed version and a non-analyzed version (called {fieldName}.raw). Fields with a . in their name will be mapped as object type fields with mappings included for each of their subfields.

Note: You can only create a template if your pattern does not match existing indices.

Payload Schema

Each payload is a JSON object.

The object has the following properties:

id: String (Required) This will be the ID of the index pattern that gets created.
title: String (Required)
time_field_name: String (optional, only needed for time based index patterns)
interval_name: String (optional)
fields: Array (required)
field_format_map: Object (optional, map of non-default formatters to user for fields)
not_expandable: Boolean (optional)

An object in the fields array must have the following attributes:

name: String (required)
type: String (required)

Valid values for type include:

string
date
boolean
number
geo_point
geo_shape
ip

Errors

400 Bad Request for an invalid payload
409 Conflict when a pattern or template with the given ID already exists
409 Conflict when the pattern matches existing indices

Example

Request:

POST /api/kibana/ingest

{
          "id": "logstash-*",
          "title": "logstash-*",
          "time_field_name": "@timestamp",
          "fields": [{
            "name": "ip",
            "type": "ip"
          }, {
            "name": "@timestamp",
            "type": "date"
          }, {
            "name": "agent",
            "type": "string"
          }, {
            "name": "bytes",
            "type": "number"
          },
          {
            "name": "geo.coordinates",
            "type": "geo_point"
          },
          {
            "name": "geo.src",
            "type": "string"
          },
          {
            "name": "geo.dest",
            "type": "string"
          },
          {
            "name": "geo.srcdest",
            "type": "string"
          }]
}

Response:

Status 204

DELETE /api/kibana/ingest/{id}

Deletes the index pattern with the given id and its associated index template.

Errors

404 Not Found if the given pattern does not exist

Example

Request:

DELETE /api/kibana/ingest/logstash-*

Response:

Status 200

{"success": true}

Done

Moved API endpoints into a plugin.
Implemented basic initial version of all the endpoints.
Error handling
Need to handle errors that the elasticsearch client might throw at me
https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/errors.html
Create an elasticsearch index template when an index pattern is created
Ignore type specific mappings, use default
- Accept json payload that ES expects for mappings.
- If a pattern matches no existing indices, a template will be created.
- If a pattern matches existing indices, a template will not be created.
- If a user is trying to update an existing pattern with a matching template, we should only accept new field mappings and reject changes to existing ones (add a warning in the UI that uses this API to describe this behavior?) (going to punt on this until we feel like we actually need it because it's going to be rather complicated).
Payload validation
Create a Joi schema
Testing
Primarily going to rely on functional tests for API testing, do unit testing where appropriate.
Security
Should accept API tokens for auth once Kibana API Tokens #5267 is merged.
Until then, how should we secure this? It's a bit different than most ways we access elasticsearch right now because it accesses the .kibana index, but it will be initiated from the frontend. Should look at how the current pattern creation screen does it with a shielded cluster.

kimjoar · 2015-10-29T14:18:22Z

What are pros and cons of plugin vs core? If there are no big cons to putting this in a plugin, that seems like the best way to go (smaller core is nice).

Bargs · 2015-10-29T14:21:59Z

@kjbekkelund yup I agree. The last couple days I've been learning more about Hapi and the current state of the plugin system and that's the direction I'm going to move in. Should have some updated code today.

Bargs · 2015-11-02T18:36:27Z

@rashidkpc what level of validation do you think we should do on the request body JSON, if any? For instance should we parse the "fields" field and make sure it looks like it's formatted correctly?

From a security standpoint, it looks like the /elasticsearch endpoint allows the user to send pretty much anything to the .kibana index. This new API forces the .kibana index and index-pattern type for all requests since it only deals with index patterns, but can you think of any potentially malicious payloads we should try to detect and reject?

I also have some questions related to ES index template creation in the description of this PR that I'd appreciate if you could take a look at.

Also forgot, I was curious if there was a reason why the "fields" field is stored as a string instead of a regular json array?

rashidkpc · 2015-11-03T14:51:06Z

I can't think of anything particularly malicious you could add to a mapping.

Fields is stored as a string to keep elasticsearch from parsing it into real elasticsearch fields, causing the cluster state to be much larger than it would need to be. This is probably not a good move, we probably could've just turned off indexing of that field in the mapping. Not sure if that works on object level fields.

We could try to validate that each item in the fields object at least has a name and a type. Those are really the only things Elasticsearch needs for a successful mapping. We're going to hit an odd issue here though, and thats that we denormalized short/long/double/integer/float into number in our index patterns. We're now talking about creating a template and will need to go the other way. I suppose when they select "number" as the type, we'll need to prompt for something more specific.

Bargs · 2015-11-03T15:53:07Z

@rashidkpc we'd really only be creating templates for new index patterns, right? Could we deprecate and hide the number type and only allow new index patterns to use one of the elasticsearch types?

rashidkpc · 2015-11-03T15:55:06Z

We certainly could, though we'd need to update a lot of code that looks for 'number' specifically.

Bargs · 2015-11-14T01:44:39Z

@rashidkpc I think the API is pretty much functionally complete. I still have a few things I'd like to do (testing, investigate security/shield integration, refactor some common code into modules), but I'd love it if you could take a look when you get a chance next week and provide any feedback you might have. Thanks!

kimjoar · 2015-11-14T14:56:48Z

src/plugins/api/index_pattern_endpoints.js

+      const isWildcard = _.contains(indexPattern.title, '*') || (indexPattern.title.match(/\[.*]/) !== null);
+      const mappings = _.omit(_.mapValues(_.indexBy(req.payload.fields, 'name'), (value) => {
+        return value.mapping;
+      }), _.isUndefined);


For readability I usually prefer something like this when I use a lot of lodash methods:

_(req.payload.fields) .indexBy('name') .mapValues(value => value.mapping) .omit(_.isUndefined) .value();

Didn't know about that syntax. Mind blown!

rashidkpc · 2015-11-17T20:16:43Z

@Bargs can you add a thorough description of how the API is used to the pull description. I know you have an initial description in #5199, but I'd like to see the final implementation here.

Bargs · 2015-11-18T00:07:07Z

@rashidkpc Updated. I think I covered all of the functionality as it currently stands. Let me know if you have any questions or concerns. My goal was to have the API combine all the info from the .kibana index, index templates, and indices in a way that's mostly seamless to the client, without duplicating data in any of those three locations.

Bargs · 2015-11-20T18:51:11Z

@rashidkpc @spalger Now that the tests are complete and passing I've slapped the official review label on this guy. The only outstanding items from the last batch of feedback are:

Is using callWithRequest all I need to do to work properly with Shield?
Resolution to the discussion about template creation.

I updated the PR description with some thorough documentation as well so please give that a read as you review.

rashidkpc · 2015-11-24T15:44:36Z

@Bargs can you update the pull description to fit the new routes?

rashidkpc · 2015-11-24T16:27:39Z

src/plugins/kibana/server/routes/api/index_pattern_endpoints.js

+      return error;
+    }
+  };
+


Can you break these routes out into their own files, under an index_pattern directory? Then just include them in this file?

Bargs · 2016-01-11T17:58:53Z

@spalger Just pushed some updates, I think I covered all of your feedback, please take another look!

spalger · 2016-01-11T18:02:09Z

LGTM

Bargs · 2016-01-11T19:41:38Z

@spalger Whoooooo! @rashidkpc one final look for you as well?

rashidkpc · 2016-01-11T21:20:50Z

Implementation looks good, can you squash the commits?

Create an Ingest API

kmoe · 2016-01-26T14:21:09Z

Is the Postman collection up to date? When I run the GET request against my local Kibana 5, it complains about a missing kbn-version header, and still gives a 400 when I add one in.

Bargs · 2016-01-26T15:25:46Z

@kmoe the Postman examples should be up to date, it was working for another team member before. However it sounds like you're getting an old version, because there's no longer a GET endpoint. You might try deleting the postman collection and re-importing it, I've seen the sync act pretty flaky in the past.

megastef · 2016-01-27T11:13:04Z

How to use this API? Why is a kbn-version Header required? And why is there a browser version check for an API?
Here are the things I've tried:

 curl -XPOST localhost:5601/api/kibana/ingest -d @pattern.txt
{"statusCode":400,"error":"Bad Request","message":"Missing kbn-version header"}MacBook-Pro-2:kibana stefan$ 
MacBook-Pro-2:kibana stefan$ curl -XPOST -H "kbn-version: 4.4.0" localhost:5601/api/kibana/ingest -d @pattern.txt
{"statusCode":400,"error":"Bad Request","message":"Browser client is out of date, please refresh the page"} 
curl -XPOST -H "kbn-version: 4.4.0" -H "mime-type: application/json" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:43.0) Gecko/20100101 Firefox/43.0"  localhost:5601/api/kibana/ingest -d @pattern.txt
{"statusCode":400,"error":"Bad Request","message":"Browser client is out of date, please refresh the page"}

epixa · 2016-01-27T14:44:00Z

@megastef This change only exists in master, which is version 5.0.0-snapshot rather than 4.4.0.

megastef · 2016-01-27T14:59:25Z

I'm using "master". I wonder more about the error messages ""Browser client is out of date, please refresh the page" - there was no complain about the version I used in the header (only when not present). And the API does not allow to specify the Kibana-Index (which is configured in the kibana.yml - static/gloabal config), while an API should give the freedom to be used for more general things.

epixa · 2016-01-27T15:03:08Z

The "browser out of date" error you're getting actually is the error message from the version mismatch. When the error message was created, there were no APIs to access, so you'd only ever see it if you were using Kibana from your browser. We should update the error message.

epixa · 2016-01-27T15:08:01Z

I created issue #6021 for updating the error message.

Bargs · 2016-01-27T20:35:26Z

Also note that this API is primarily meant for Kibana's own internal use
right now. It's possible to hit it from outside of Kibana's front end code,
but you will run into some rough edges like this.
On Wed, Jan 27, 2016 at 10:08 AM Court Ewing notifications@github.com
wrote:

I created issue #6021 #6021 for
updating the error message.

—
Reply to this email directly or view it on GitHub
#5213 (comment).

tamsky · 2018-01-19T23:30:37Z

The "browser out of date" error you're getting actually is the error message from the version mismatch.

For us to solve this, we first needed to discover that the version of kibana installed on all nodes handling web requests for kibana:5601 were not running the same version.

sandstrom · 2018-04-18T12:52:59Z

If you would prefer a method for loading dashboards and indexpatterns from the filesystem (for use with Chef, Puppet, Salt, etc) please vouch for it here: #2310

Bargs force-pushed the indexPatternApi branch from cc28c1c to 266ff03 Compare October 29, 2015 20:51

Bargs force-pushed the indexPatternApi branch from b574171 to 398a916 Compare November 2, 2015 18:51

Bargs force-pushed the indexPatternApi branch 2 times, most recently from 04239de to 09b207b Compare November 11, 2015 01:07

Bargs force-pushed the indexPatternApi branch from 968fd50 to 7e73b3f Compare November 13, 2015 20:48

kimjoar reviewed Nov 14, 2015
View reviewed changes

spalger mentioned this pull request Nov 17, 2015

Decide casing in the api #5412

Closed

Bargs force-pushed the indexPatternApi branch 3 times, most recently from f070657 to bb9aee0 Compare November 19, 2015 01:11

Bargs added the review label Nov 20, 2015

Bargs assigned rashidkpc Nov 20, 2015

rashidkpc reviewed Nov 24, 2015
View reviewed changes

spalger assigned Bargs Jan 10, 2016

Bargs added 3 commits January 11, 2016 12:24

fix errors in tests

22be314

clean up formatting and leverage error ignoring feature of ES js client

df9ef8c

Use more descriptive names for the case conversion helper methods

e5a8b12

Bargs assigned spalger and unassigned Bargs Jan 11, 2016

Bargs assigned rashidkpc and unassigned spalger Jan 11, 2016

rashidkpc added a commit that referenced this pull request Jan 11, 2016

Merge pull request #5213 from Bargs/indexPatternApi

1043884

Create an Ingest API

rashidkpc merged commit 1043884 into elastic:master Jan 11, 2016

rashidkpc added the v5.0.0 label Jan 11, 2016

Bargs mentioned this pull request Jan 21, 2016

Tail a File Wizard #5974

Closed

15 tasks

epixa mentioned this pull request Jan 27, 2016

Clearer error message for mismatched kbn-version on api request #6021

Closed

tbragin added the Feature:http label Mar 18, 2016

tbragin mentioned this pull request Mar 18, 2016

Kibana 5.0.0-alpha1 docs / release notes #6572

Closed

7 tasks

epixa added the v5.0.0-alpha1 label Mar 31, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create an Ingest API #5213

Create an Ingest API #5213

Bargs commented Oct 27, 2015 •

edited

Loading

kimjoar commented Oct 29, 2015

Bargs commented Oct 29, 2015

Bargs commented Nov 2, 2015

rashidkpc commented Nov 3, 2015

Bargs commented Nov 3, 2015

rashidkpc commented Nov 3, 2015

Bargs commented Nov 14, 2015

kimjoar Nov 14, 2015

Bargs Nov 19, 2015

rashidkpc commented Nov 17, 2015

Bargs commented Nov 18, 2015

Bargs commented Nov 20, 2015

rashidkpc commented Nov 24, 2015

rashidkpc Nov 24, 2015

Bargs commented Jan 11, 2016

spalger commented Jan 11, 2016

Bargs commented Jan 11, 2016

rashidkpc commented Jan 11, 2016

kmoe commented Jan 26, 2016

Bargs commented Jan 26, 2016

megastef commented Jan 27, 2016

epixa commented Jan 27, 2016

megastef commented Jan 27, 2016

epixa commented Jan 27, 2016

epixa commented Jan 27, 2016

Bargs commented Jan 27, 2016

tamsky commented Jan 19, 2018

sandstrom commented Apr 18, 2018 •

edited

Loading

Create an Ingest API #5213

Create an Ingest API #5213

Conversation

Bargs commented Oct 27, 2015 • edited Loading

Documentation

Endpoints

POST /api/kibana/ingest

Payload Schema

Errors

Example

DELETE /api/kibana/ingest/{id}

Errors

Example

Done

kimjoar commented Oct 29, 2015

Bargs commented Oct 29, 2015

Bargs commented Nov 2, 2015

rashidkpc commented Nov 3, 2015

Bargs commented Nov 3, 2015

rashidkpc commented Nov 3, 2015

Bargs commented Nov 14, 2015

kimjoar Nov 14, 2015

Choose a reason for hiding this comment

Bargs Nov 19, 2015

Choose a reason for hiding this comment

rashidkpc commented Nov 17, 2015

Bargs commented Nov 18, 2015

Bargs commented Nov 20, 2015

rashidkpc commented Nov 24, 2015

rashidkpc Nov 24, 2015

Choose a reason for hiding this comment

Bargs commented Jan 11, 2016

spalger commented Jan 11, 2016

Bargs commented Jan 11, 2016

rashidkpc commented Jan 11, 2016

kmoe commented Jan 26, 2016

Bargs commented Jan 26, 2016

megastef commented Jan 27, 2016

epixa commented Jan 27, 2016

megastef commented Jan 27, 2016

epixa commented Jan 27, 2016

epixa commented Jan 27, 2016

Bargs commented Jan 27, 2016

tamsky commented Jan 19, 2018

sandstrom commented Apr 18, 2018 • edited Loading

Bargs commented Oct 27, 2015 •

edited

Loading

sandstrom commented Apr 18, 2018 •

edited

Loading