[ML] expand allowed NER labels to be any I-O-B tagged labels

[benwtrent]

Named entity recognition (NER) is a special form of token classification. The specific kind of labelling we support is Inside-Outside-Beginning (IOB) tagging. These labels indicate if they are the inside of a token (with a I- or I_), the beginning (B- or B_) or outside (O).

Each valid token classification label starts with the require prefix or O.

Before this commit, we restricted the labels to a specific set:

O(Entity.NONE),      // Outside a named entity
B_MISC(Entity.MISC), // Beginning of a miscellaneous entity right after another miscellaneous entity
I_MISC(Entity.MISC), // Miscellaneous entity
B_PER(Entity.PER),   // Beginning of a person's name right after another person's name
I_PER(Entity.PER),   // Person's name
B_ORG(Entity.ORG),   // Beginning of an organization right after another organization
I_ORG(Entity.ORG),   // Organisation
B_LOC(Entity.LOC),   // Beginning of a location right after another location
I_LOC(Entity.LOC);   // Location

But now, any entity is allowed, as long as the naming of the labels adhere to IOB tagging rules.

Here is an inference response containing other token labels:

{
    "predicted_value": "[Birth defects](ADR&Birth+defects) associated with [thalidomide](DRUG&thalidomide).",
    "entities": [
        {
            "entity": "birth defects",
            "class_name": "ADR",
            "class_probability": 0.9664951378636988,
            "start_pos": 0,
            "end_pos": 13
        },
        {
            "entity": "thalidomide",
            "class_name": "DRUG",
            "class_probability": 0.7323781805751934,
            "start_pos": 30,
            "end_pos": 41
        }
    ]
}

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @benwtrent, I've created a changelog YAML for you.

[dimitris-athanasiou]

Looks good! Just a question on whether we can remove the Entity enum entirely now.

Also, could we add a test for labels that are custom and do not match the default labels?

[benwtrent]

@elasticmachine update branch

[dimitris-athanasiou]

LGTM