Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add strict_allow_templates option for the dynamic mapping parameter #7745

Merged
merged 27 commits into from
Jul 18, 2024
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
968048a
Add strict_allow_templates option for the dynamic mapping parameter
gaobinlong Jul 17, 2024
27e2b62
Fix typo
gaobinlong Jul 17, 2024
b08b46c
Fix header
gaobinlong Jul 17, 2024
032d1f9
Update dynamic.md
vagimeli Jul 18, 2024
4fa3811
Merge branch 'main' into strict_allow_templates
vagimeli Jul 18, 2024
637996f
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
8c3e4c2
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
4564725
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
16ede9f
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
e4eb343
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
c0f4c76
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
e043ecc
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
480ee1e
Update _field-types/supported-field-types/object.md
vagimeli Jul 18, 2024
92baea1
Update _field-types/supported-field-types/object.md
vagimeli Jul 18, 2024
a23dad8
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
44f2f36
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
86d670f
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
f27ff68
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
37750a7
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
445757c
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
cf8b350
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
d2a9abc
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
c62ad9e
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
dc723de
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
50ea608
Update _field-types/supported-field-types/nested.md
vagimeli Jul 18, 2024
ee1953f
Update dynamic.md
vagimeli Jul 18, 2024
b0568fa
Merge branch 'main' into strict_allow_templates
vagimeli Jul 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
342 changes: 342 additions & 0 deletions _field-types/dynamic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,342 @@
---
layout: default
title: Dynamic parameter
nav_order: 10
redirect_from:
- /opensearch/dynamic/
---

# Dynamic parameter

The `dynamic` parameter specifies whether newly detected fields can be added dynamically to a mapping. It accepts the parameters listed in the following table.

Parameter | Description
:--- | :---
`true` | Specfies that new fields can be added dynamically to the mapping. Default is `true`.
`false` | Specifies that new fields cannot be added dynamically to the mapping. If a new field is detected, then it is not indexed or searchable but can be retrieved from the `_source` field.
`strict` | Throws an exception. The indexing operation fails when new fields are detected.
`strict_allow_templates` | Adds new fields if they match predefined dynamic templates in the mapping.

---

## Example: Create an index with `dynamic` set to `true`

1. Create an index with `dynamic` set to `true` by sending the following request:

```json
PUT testindex1
{
"mappings": {
"dynamic": true
}
}
```
{% include copy-curl.html %}

2. Index a document with an object field `patient` containing two string fields by sending the following request:

```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
}
}
```
{% include copy-curl.html %}

3. Confirm the mapping works as expected by sending the following request:

```json
GET testindex1/_mapping
```
{% include copy-curl.html %}

The object field `patient` and two subfields `name` and `id` are added to the mapping, as shown in the following response:

```json
{
"testindex1": {
"mappings": {
"dynamic": "true",
"properties": {
"patient": {
"properties": {
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
}
```

---

## Example: Create an index with `dynamic` set to `false`

1. Create an index with explicit mappings and `dynamic` set to `false` by sending the following request:

```json
PUT testindex1
{
"mappings": {
"dynamic": false,
"properties": {
"patient": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
}
}
}
}
```
{% include copy-curl.html %}

2. Index a document with an object field `patient` containing two string fields and additional unmapped fields by sending the following request:

```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
},
"room": "room1",
"floor": "1"
}
```
{% include copy-curl.html %}

3. Confirm the mapping works as expected by sending the following request:

```json
GET testindex1/_mapping
```
{% include copy-curl.html %}

The following response shows that the new fields `room` and `floor` were not added to the mapping, which remained unchanged:

```json
{
"testindex1": {
"mappings": {
"dynamic": "false",
"properties": {
"patient": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
}
}
}
}
}
```

4. Get the unmapped fields `room` and `floor` from the document by sending the following request:

```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
},
"room": "room1",
"floor": "1"
}
```

The following request searches for the fields `room` and `floor`:

```json
POST testindex1/_search
{
"query": {
"term": {
"room": "room1"
}
}
}
```

The response returns no results:

```json
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
}
}
```

---

## Example: Create an index with `dynamic` set to `strict`

1. Create an index with explicit mappings and `dynamic` set to `strict` by sending the following request:

```json
PUT testindex1
{
"mappings": {
"dynamic": strict,
"properties": {
"patient": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
}
}
}
}
```
{% include copy-curl.html %}

2. Index a document with an object field `patient` containing two string fields and additional unmapped fields by sending the following request:

```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
},
"room": "room1",
"floor": "1"
}
```
{% include copy-curl.html %}

Note that an exception is thrown, as shown in the following response:

```json
{
"error": {
"root_cause": [
{
"type": "strict_dynamic_mapping_exception",
"reason": "mapping set to strict, dynamic introduction of [room] within [_doc] is not allowed"
}
],
"type": "strict_dynamic_mapping_exception",
"reason": "mapping set to strict, dynamic introduction of [room] within [_doc] is not allowed"
},
"status": 400
}
```

---

## Example: Create an index with `dynamic` set to `strict_allow_templates`

1. Create an index with predefined dynamic templates and `dynamic` set to `strict_allow_templates` by sending the following request:

```json
PUT testindex1
{
"mappings": {
"dynamic": "strict_allow_templates",
"dynamic_templates": [
{
"strings": {
"match": "room*",
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
}
],
"properties": {
"patient": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
}
}
}
}
```
{% include copy-curl.html %}

2. Index a document with an object field `patient` containing two string fields and a new field `room` that matches one of the dynamic templates by sending the following request:

```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
},
"room": "room1"
}
```
{% include copy-curl.html %}

Indexing succeeds because the new field `room` matches the dynamic templates. However, indexing fails for the new field `floor` because it does not match one of the dynamic templates and is not explicitly mapped, as shown in the following response:

```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
},
"room": "room1",
"floor": "1"
}
```
2 changes: 1 addition & 1 deletion _field-types/supported-field-types/nested.md
Original file line number Diff line number Diff line change
@@ -308,7 +308,7 @@ The following table lists the parameters accepted by object field types. All par

Parameter | Description
:--- | :---
[`dynamic`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/object#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to this object. Valid values are `true`, `false`, and `strict`. Default is `true`.
[`dynamic`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/object#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to the object. Valid values are `true`, `false`, `strict`, and `strict_allow_templates`. Default is `true`.
`include_in_parent` | A Boolean value that specifies whether all fields in the child nested object should also be added to the parent document in flattened form. Default is `false`.
`include_in_root` | A Boolean value that specifies whether all fields in the child nested object should also be added to the root document in flattened form. Default is `false`.
`properties` | Fields of this object, which can be of any supported type. New properties can be dynamically added to this object if `dynamic` is set to `true`.
3 changes: 2 additions & 1 deletion _field-types/supported-field-types/object.md
Original file line number Diff line number Diff line change
@@ -73,7 +73,7 @@ The following table lists the parameters accepted by object field types. All par

Parameter | Description
:--- | :---
[`dynamic`](#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to this object. Valid values are `true`, `false`, and `strict`. Default is `true`.
[`dynamic`](#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to the object. Valid values are `true`, `false`, `strict`, and `strict_allow_templates`. Default is `true`.
`enabled` | A Boolean value that specifies whether the JSON contents of the object should be parsed. If `enabled` is set to `false`, the object's contents are not indexed or searchable, but they are still retrievable from the _source field. Default is `true`.
`properties` | Fields of this object, which can be of any supported type. New properties can be dynamically added to this object if `dynamic` is set to `true`.

@@ -149,6 +149,7 @@ Value | Description
`true` | New fields can be added to the mapping dynamically. This is the default.
`false` | New fields cannot be added to the mapping dynamically. If a new field is detected, it is not indexed or searchable. However, it is still retrievable from the _source field.
`strict` | When new fields are added to the mapping dynamically, an exception is thrown. To add a new field to an object, you have to add it to the mapping first.
`strict_allow_templates` | If the newly detected fields match any of the predefined dynamic templates in the mapping, then they are added to the mapping; if they do not match any of them, then an exception is thrown.

Inner objects inherit the `dynamic` parameter value from their parent unless they declare their own `dynamic` parameter value.
{: .note }
Loading