DIETClassifier adds a confidence value to entity predictions

[tabergma]

Proposed changes:

• Update the CRF to return scores that can be used as confidence values.
• DIETClassifier now adds a confidence value to entity predictions.

closes #5481

(Will also make a PR to tensorflow addons directly. If they take the changes and adapt the CRF, we can remove our custom CRF again.)

Status (please check what you already did):

• added some tests for the functionality
• updated the documentation
• updated the changelog (please check changelog for instructions)
• reformat files using black (please check Readme for instructions)

[tabergma]

Tensorflow Addons PR: tensorflow/addons#1935

[Ghostvv]

@tabergma could you please apply my comments to tf addons PR as well

[vish0701]

This is a good feature to add. However, current rasa documentation at https://rasa.com/docs/rasa/nlu/entity-extraction/#entity-extraction-custom-features says

Does the documentation need to be updated to indicate that now DIET returns a confidence value?

[tabergma]

@vish0701 The feature is not yet released. It is merged into master and we also updated the documentation. Once we do the next release from master, you will have the confidence value for entities coming from the DIETClassifier and the documentation will be updated as well.

[aminekechaou]

Was this already included in a release? From my understanding it is not in 1.10.14.

[tabergma]

This will be included in the Rasa 2.0 release, which should happen any day now.