Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add strict_allow_templates option for the dynamic mapping parameter #7745

Merged
merged 27 commits into from
Jul 18, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
968048a
Add strict_allow_templates option for the dynamic mapping parameter
gaobinlong Jul 17, 2024
27e2b62
Fix typo
gaobinlong Jul 17, 2024
b08b46c
Fix header
gaobinlong Jul 17, 2024
032d1f9
Update dynamic.md
vagimeli Jul 18, 2024
4fa3811
Merge branch 'main' into strict_allow_templates
vagimeli Jul 18, 2024
637996f
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
8c3e4c2
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
4564725
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
16ede9f
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
e4eb343
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
c0f4c76
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
e043ecc
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
480ee1e
Update _field-types/supported-field-types/object.md
vagimeli Jul 18, 2024
92baea1
Update _field-types/supported-field-types/object.md
vagimeli Jul 18, 2024
a23dad8
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
44f2f36
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
86d670f
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
f27ff68
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
37750a7
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
445757c
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
cf8b350
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
d2a9abc
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
c62ad9e
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
dc723de
Update _field-types/dynamic.md
vagimeli Jul 18, 2024
50ea608
Update _field-types/supported-field-types/nested.md
vagimeli Jul 18, 2024
ee1953f
Update dynamic.md
vagimeli Jul 18, 2024
b0568fa
Merge branch 'main' into strict_allow_templates
vagimeli Jul 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
342 changes: 342 additions & 0 deletions _field-types/dynamic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,342 @@
---
layout: default
title: Dynamic parameter
nav_order: 10
redirect_from:
- /opensearch/dynamic/
---

# Dynamic parameter

The `dynamic` parameter specifies whether new detected fields can be added dynamically to the mapping. It accepts the parameters listed in the following table.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

Parameter | Description
:--- | :---
`true` | Specfies that new fields can be added dynamically to the mapping. Default is `true`.
`false` | Specifies that new fields cannot be added dynamically to the mapping. If a new field is detected, then it is not indexed or searchable but can be retrieved from the `_source` field.
`strict` | Throws an exception. The indexing operation fails when new fields are detected.
`strict_allow_templates` | Adds new fields matching dynamic templates if the new fields match predefined dynamic templates in the mapping; otherwise, indexing fails.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the first phrase a bit redundant? Can it just be "Adds new fields if they match predefined dynamic templates in the mapping"?


---

## Example: Create index with dynamic parameter set to `true`
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

1. Create an index with `dynamic` set as `true` using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT testindex1
{
"mappings": {
"dynamic": true
}
}
```
{% include copy-curl.html %}

2. Index a document with an object field `patient` containing two string fields using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
}
}
```
{% include copy-curl.html %}

3. Check the mapping using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we mean something slightly more precise than "Check"?

Copy link
Contributor

@vagimeli vagimeli Jul 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revised to read "Confirm the mapping works as expected by sending the following request:"


```json
GET testindex1/_mapping
```
{% include copy-curl.html %}

Object field `patient` and two subfields `name` and `id` are added to the mapping, as shown in the following response:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
{
"testindex1": {
"mappings": {
"dynamic": "true",
"properties": {
"patient": {
"properties": {
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
}
```

---

## Example: Create index with dynamic parameter set to `false`
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

1. Create an index with `dynamic` set to `false` and explicit mappings using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reverse the order here?: "Create an index with explicit mappings and dynamic set to false..."


```json
PUT testindex1
{
"mappings": {
"dynamic": false,
"properties": {
"patient": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
}
}
}
}
```
{% include copy-curl.html %}

2. Index a document with an object field patient containing two string fields and additional unmapped fields using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
},
"room": "room1",
"floor": "1"
}
```
{% include copy-curl.html %}

3. Check the mapping using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we mean something slightly more precise than "Check"?

Copy link
Contributor

@vagimeli vagimeli Jul 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revised to read "Confirm the mapping works as expected by sending the following request:"


```json
GET testindex1/_mapping
```
{% include copy-curl.html %}

The following response shows that the new fields `room` and `floor` were not added to the mapping, which remained unchanged:

```json
{
"testindex1": {
"mappings": {
"dynamic": "false",
"properties": {
"patient": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
}
}
}
}
}
```

4. Get the unmapped fields `room` and `floor` from the document using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
},
"room": "room1",
"floor": "1"
}
```

The following request searches the fields `room` or `floor`, with a response confirming no results returned:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The following request searches for the fields room and floor:"?


```json
POST testindex1/_search
{
"query": {
"term": {
"room": "room1"
}
}
}
```

The following response confirms no returned results:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
}
}
```

---

## Example: Create an index with `dynamic` set to `strict`

1. Create an index with `dynamic` set to `strict` and explicit mappings using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reverse the order here?: "Create an index with explicit mappings and dynamic set to strict..."


```json
PUT testindex1
{
"mappings": {
"dynamic": strict,
"properties": {
"patient": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
}
}
}
}
```
{% include copy-curl.html %}

2. Index a document with an object field `patient` containing two string fields, and additional unmapped fields using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
},
"room": "room1",
"floor": "1"
}
```
{% include copy-curl.html %}

Note that an exception is thrown, as shown in the following response:

```json
{
"error": {
"root_cause": [
{
"type": "strict_dynamic_mapping_exception",
"reason": "mapping set to strict, dynamic introduction of [room] within [_doc] is not allowed"
}
],
"type": "strict_dynamic_mapping_exception",
"reason": "mapping set to strict, dynamic introduction of [room] within [_doc] is not allowed"
},
"status": 400
}
```

---

## Example: Create an index with `dynamic` set to `strict_allow_templates`

1. Create an index with `dynamic` set to `strict_allow_templates` and dynamic templates using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reverse the order here?: "Create an index with predefined dynamic templates and dynamic set to strict_allow_templates...". Added "predefined" to match language used elsewhere.


```json
PUT testindex1
{
"mappings": {
"dynamic": "strict_allow_templates",
"dynamic_templates": [
{
"strings": {
"match": "room*",
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
}
],
"properties": {
"patient": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
}
}
}
}
```
{% include copy-curl.html %}

2. Index a document with an object field `patient` containing two string fields, and a new field `room` matching the dynamic templates using the following request:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirm that we do not mean "one of the dynamic templates".


```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
},
"room": "room1"
}
```
{% include copy-curl.html %}

Indexing succeeds because the new field `room` matches the dynamic templates. However, indexing fails for the new field `floor` because it does not match the dynamic templates and is not explicitly mapped, as shown in the following response:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirm that we do not mean "one of the dynamic templates".


```json
PUT testindex1/_doc/1
{
"patient": {
"name" : "John Doe",
"id" : "123456"
},
"room": "room1",
"floor": "1"
}
```
2 changes: 1 addition & 1 deletion _field-types/supported-field-types/nested.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ The following table lists the parameters accepted by object field types. All par

Parameter | Description
:--- | :---
[`dynamic`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/object#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to this object. Valid values are `true`, `false`, and `strict`. Default is `true`.
[`dynamic`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/object#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to this object. Valid values are `true`, `false`, `strict` and `strict_allow_templates`. Default is `true`.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
`include_in_parent` | A Boolean value that specifies whether all fields in the child nested object should also be added to the parent document in flattened form. Default is `false`.
`include_in_root` | A Boolean value that specifies whether all fields in the child nested object should also be added to the root document in flattened form. Default is `false`.
`properties` | Fields of this object, which can be of any supported type. New properties can be dynamically added to this object if `dynamic` is set to `true`.
3 changes: 2 additions & 1 deletion _field-types/supported-field-types/object.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ The following table lists the parameters accepted by object field types. All par

Parameter | Description
:--- | :---
[`dynamic`](#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to this object. Valid values are `true`, `false`, and `strict`. Default is `true`.
[`dynamic`](#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to this object. Valid values are `true`, `false`, `strict` and `strict_allow_templates`. Default is `true`.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
`enabled` | A Boolean value that specifies whether the JSON contents of the object should be parsed. If `enabled` is set to `false`, the object's contents are not indexed or searchable, but they are still retrievable from the _source field. Default is `true`.
`properties` | Fields of this object, which can be of any supported type. New properties can be dynamically added to this object if `dynamic` is set to `true`.

Expand Down Expand Up @@ -149,6 +149,7 @@ Value | Description
`true` | New fields can be added to the mapping dynamically. This is the default.
`false` | New fields cannot be added to the mapping dynamically. If a new field is detected, it is not indexed or searchable. However, it is still retrievable from the _source field.
`strict` | When new fields are added to the mapping dynamically, an exception is thrown. To add a new field to an object, you have to add it to the mapping first.
`strict_allow_templates` | If the new detected fields can match any predefined dynamic template in the mapping, then they will be added to the mapping, if not match an exception will be thrown as same as `strict`.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please confirm the accuracy of my rewrite. The use of "any" here is what prompted my comments re: "one of the dynamic templates" in dynamic.md.


Inner objects inherit the `dynamic` parameter value from their parent unless they declare their own `dynamic` parameter value.
{: .note }
Loading