diff --git a/CHANGELOG b/CHANGELOG
index 44fe5ad74c7851..54b2672ec857b1 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,13 @@
+========
+4.2.8
+========
+----------------
+Bug Fixes & Enhancements
+----------------
+* Fix the issue with optional keys (labels) in metadata when using XXXForSequenceClassitication annotators. This fixes `Some(neg) -> 0.13602075` as `neg -> 0.13602075` to be in harmony with all the other classifiers. https://github.com/JohnSnowLabs/spark-nlp/pull/13396
+* Introducing a config to skip `LightPipeline` validation for `inputCols` on the Python side for projects depending on Spark NLP. This toggle should only be used for specific annotators that do not follow the convention of predefined `inputAnnotatorTypes` and `outputAnnotatorType`.
+
+
========
4.2.7
========
diff --git a/README.md b/README.md
index a8908a97c12dcd..ee8b7fa37bf8dc 100644
--- a/README.md
+++ b/README.md
@@ -152,7 +152,7 @@ To use Spark NLP you need the following requirements:
**GPU (optional):**
-Spark NLP 4.2.7 is built with TensorFlow 2.7.1 and the following NVIDIA® software are only required for GPU support:
+Spark NLP 4.2.8 is built with TensorFlow 2.7.1 and the following NVIDIA® software are only required for GPU support:
- NVIDIA® GPU drivers version 450.80.02 or higher
- CUDA® Toolkit 11.2
@@ -168,7 +168,7 @@ $ java -version
$ conda create -n sparknlp python=3.7 -y
$ conda activate sparknlp
# spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==4.2.7 pyspark==3.2.3
+$ pip install spark-nlp==4.2.8 pyspark==3.2.3
```
In Python console or Jupyter `Python3` kernel:
@@ -213,7 +213,7 @@ For more examples, you can visit our dedicated [repository](https://github.com/J
## Apache Spark Support
-Spark NLP *4.2.7* has been built on top of Apache Spark 3.2 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:
+Spark NLP *4.2.8* has been built on top of Apache Spark 3.2 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:
| Spark NLP | Apache Spark 2.3.x | Apache Spark 2.4.x | Apache Spark 3.0.x | Apache Spark 3.1.x | Apache Spark 3.2.x | Apache Spark 3.3.x |
|-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
@@ -247,7 +247,7 @@ Find out more about `Spark NLP` versions from our [release notes](https://github
## Databricks Support
-Spark NLP 4.2.7 has been tested and is compatible with the following runtimes:
+Spark NLP 4.2.8 has been tested and is compatible with the following runtimes:
**CPU:**
@@ -291,7 +291,7 @@ NOTE: Spark NLP 4.0.x is based on TensorFlow 2.7.x which is compatible with CUDA
## EMR Support
-Spark NLP 4.2.7 has been tested and is compatible with the following EMR releases:
+Spark NLP 4.2.8 has been tested and is compatible with the following EMR releases:
- emr-6.2.0
- emr-6.3.0
@@ -329,11 +329,11 @@ Spark NLP supports all major releases of Apache Spark 3.0.x, Apache Spark 3.1.x,
```sh
# CPU
-spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7
+spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8
-pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7
+pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8
-spark-submit --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7
+spark-submit --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8
```
The `spark-nlp` has been published to the [Maven Repository](https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp).
@@ -341,11 +341,11 @@ The `spark-nlp` has been published to the [Maven Repository](https://mvnreposito
```sh
# GPU
-spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.7
+spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.8
-pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.7
+pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.8
-spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.7
+spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.8
```
@@ -354,11 +354,11 @@ The `spark-nlp-gpu` has been published to the [Maven Repository](https://mvnrepo
```sh
# AArch64
-spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.7
+spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.8
-pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.7
+pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.8
-spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.7
+spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.8
```
@@ -367,11 +367,11 @@ The `spark-nlp-aarch64` has been published to the [Maven Repository](https://mvn
```sh
# M1
-spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.7
+spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.8
-pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.7
+pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.8
-spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.7
+spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.8
```
@@ -383,7 +383,7 @@ The `spark-nlp-m1` has been published to the [Maven Repository](https://mvnrepos
spark-shell \
--driver-memory 16g \
--conf spark.kryoserializer.buffer.max=2000M \
- --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7
+ --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8
```
## Scala
@@ -399,7 +399,7 @@ Spark NLP supports Scala 2.12.15 if you are using Apache Spark 3.0.x, 3.1.x, 3.2
com.johnsnowlabs.nlpspark-nlp_2.12
- 4.2.7
+ 4.2.8
```
@@ -410,7 +410,7 @@ Spark NLP supports Scala 2.12.15 if you are using Apache Spark 3.0.x, 3.1.x, 3.2
com.johnsnowlabs.nlpspark-nlp-gpu_2.12
- 4.2.7
+ 4.2.8
```
@@ -421,7 +421,7 @@ Spark NLP supports Scala 2.12.15 if you are using Apache Spark 3.0.x, 3.1.x, 3.2
com.johnsnowlabs.nlpspark-nlp-aarch64_2.12
- 4.2.7
+ 4.2.8
```
@@ -432,7 +432,7 @@ Spark NLP supports Scala 2.12.15 if you are using Apache Spark 3.0.x, 3.1.x, 3.2
com.johnsnowlabs.nlpspark-nlp-m1_2.12
- 4.2.7
+ 4.2.8
```
@@ -442,28 +442,28 @@ Spark NLP supports Scala 2.12.15 if you are using Apache Spark 3.0.x, 3.1.x, 3.2
```sbtshell
// https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp" % "4.2.7"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp" % "4.2.8"
```
**spark-nlp-gpu:**
```sbtshell
// https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-gpu
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-gpu" % "4.2.7"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-gpu" % "4.2.8"
```
**spark-nlp-aarch64:**
```sbtshell
// https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-aarch64" % "4.2.7"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-aarch64" % "4.2.8"
```
**spark-nlp-m1:**
```sbtshell
// https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-m1
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-m1" % "4.2.7"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-m1" % "4.2.8"
```
Maven Central: [https://mvnrepository.com/artifact/com.johnsnowlabs.nlp](https://mvnrepository.com/artifact/com.johnsnowlabs.nlp)
@@ -483,7 +483,7 @@ If you installed pyspark through pip/conda, you can install `spark-nlp` through
Pip:
```bash
-pip install spark-nlp==4.2.7
+pip install spark-nlp==4.2.8
```
Conda:
@@ -511,7 +511,7 @@ spark = SparkSession.builder \
.config("spark.driver.memory","16G")\
.config("spark.driver.maxResultSize", "0") \
.config("spark.kryoserializer.buffer.max", "2000M")\
- .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7")\
+ .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8")\
.getOrCreate()
```
@@ -579,7 +579,7 @@ Use either one of the following options
- Add the following Maven Coordinates to the interpreter's library list
```bash
-com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7
+com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8
```
- Add a path to pre-built jar from [here](#compiled-jars) in the interpreter's library list making sure the jar is available to driver path
@@ -589,7 +589,7 @@ com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7
Apart from the previous step, install the python module through pip
```bash
-pip install spark-nlp==4.2.7
+pip install spark-nlp==4.2.8
```
Or you can install `spark-nlp` from inside Zeppelin by using Conda:
@@ -614,7 +614,7 @@ The easiest way to get this done on Linux and macOS is to simply install `spark-
$ conda create -n sparknlp python=3.8 -y
$ conda activate sparknlp
# spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==4.2.7 pyspark==3.2.3 jupyter
+$ pip install spark-nlp==4.2.8 pyspark==3.2.3 jupyter
$ jupyter notebook
```
@@ -630,7 +630,7 @@ export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook
-pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7
+pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8
```
Alternatively, you can mix in using `--jars` option for pyspark + `pip install spark-nlp`
@@ -655,7 +655,7 @@ This script comes with the two options to define `pyspark` and `spark-nlp` versi
# -s is for spark-nlp
# -g will enable upgrading libcudnn8 to 8.1.0 on Google Colab for GPU usage
# by default they are set to the latest
-!wget https://setup.johnsnowlabs.com/colab.sh -O - | bash /dev/stdin -p 3.2.3 -s 4.2.7
+!wget https://setup.johnsnowlabs.com/colab.sh -O - | bash /dev/stdin -p 3.2.3 -s 4.2.8
```
[Spark NLP quick start on Google Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/quick_start_google_colab.ipynb) is a live demo on Google Colab that performs named entity recognitions and sentiment analysis by using Spark NLP pretrained pipelines.
@@ -676,7 +676,7 @@ This script comes with the two options to define `pyspark` and `spark-nlp` versi
# -s is for spark-nlp
# -g will enable upgrading libcudnn8 to 8.1.0 on Kaggle for GPU usage
# by default they are set to the latest
-!wget https://setup.johnsnowlabs.com/colab.sh -O - | bash /dev/stdin -p 3.2.3 -s 4.2.7
+!wget https://setup.johnsnowlabs.com/colab.sh -O - | bash /dev/stdin -p 3.2.3 -s 4.2.8
```
[Spark NLP quick start on Kaggle Kernel](https://www.kaggle.com/mozzie/spark-nlp-named-entity-recognition) is a live demo on Kaggle Kernel that performs named entity recognitions by using Spark NLP pretrained pipeline.
@@ -694,9 +694,9 @@ This script comes with the two options to define `pyspark` and `spark-nlp` versi
3. In `Libraries` tab inside your cluster you need to follow these steps:
- 3.1. Install New -> PyPI -> `spark-nlp==4.2.7` -> Install
+ 3.1. Install New -> PyPI -> `spark-nlp==4.2.8` -> Install
- 3.2. Install New -> Maven -> Coordinates -> `com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7` -> Install
+ 3.2. Install New -> Maven -> Coordinates -> `com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8` -> Install
4. Now you can attach your notebook to the cluster and use Spark NLP!
@@ -744,7 +744,7 @@ A sample of your software configuration in JSON on S3 (must be public access):
"spark.kryoserializer.buffer.max": "2000M",
"spark.serializer": "org.apache.spark.serializer.KryoSerializer",
"spark.driver.maxResultSize": "0",
- "spark.jars.packages": "com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7"
+ "spark.jars.packages": "com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8"
}
}]
```
@@ -753,7 +753,7 @@ A sample of AWS CLI to launch EMR cluster:
```.sh
aws emr create-cluster \
---name "Spark NLP 4.2.7" \
+--name "Spark NLP 4.2.8" \
--release-label emr-6.2.0 \
--applications Name=Hadoop Name=Spark Name=Hive \
--instance-type m4.4xlarge \
@@ -817,7 +817,7 @@ gcloud dataproc clusters create ${CLUSTER_NAME} \
--enable-component-gateway \
--metadata 'PIP_PACKAGES=spark-nlp spark-nlp-display google-cloud-bigquery google-cloud-storage' \
--initialization-actions gs://goog-dataproc-initialization-actions-${REGION}/python/pip-install.sh \
- --properties spark:spark.serializer=org.apache.spark.serializer.KryoSerializer,spark:spark.driver.maxResultSize=0,spark:spark.kryoserializer.buffer.max=2000M,spark:spark.jars.packages=com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7
+ --properties spark:spark.serializer=org.apache.spark.serializer.KryoSerializer,spark:spark.driver.maxResultSize=0,spark:spark.kryoserializer.buffer.max=2000M,spark:spark.jars.packages=com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8
```
2. On an existing one, you need to install spark-nlp and spark-nlp-display packages from PyPI.
@@ -856,7 +856,7 @@ spark = SparkSession.builder \
.config("spark.kryoserializer.buffer.max", "2000m") \
.config("spark.jsl.settings.pretrained.cache_folder", "sample_data/pretrained") \
.config("spark.jsl.settings.storage.cluster_tmp_dir", "sample_data/storage") \
- .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7") \
+ .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8") \
.getOrCreate()
```
@@ -870,7 +870,7 @@ spark-shell \
--conf spark.kryoserializer.buffer.max=2000M \
--conf spark.jsl.settings.pretrained.cache_folder="sample_data/pretrained" \
--conf spark.jsl.settings.storage.cluster_tmp_dir="sample_data/storage" \
- --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7
+ --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8
```
**pyspark:**
@@ -883,7 +883,7 @@ pyspark \
--conf spark.kryoserializer.buffer.max=2000M \
--conf spark.jsl.settings.pretrained.cache_folder="sample_data/pretrained" \
--conf spark.jsl.settings.storage.cluster_tmp_dir="sample_data/storage" \
- --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7
+ --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8
```
**Databricks:**
@@ -1147,12 +1147,12 @@ spark = SparkSession.builder \
.config("spark.driver.memory","16G")\
.config("spark.driver.maxResultSize", "0") \
.config("spark.kryoserializer.buffer.max", "2000M")\
- .config("spark.jars", "/tmp/spark-nlp-assembly-4.2.7.jar")\
+ .config("spark.jars", "/tmp/spark-nlp-assembly-4.2.8.jar")\
.getOrCreate()
```
- You can download provided Fat JARs from each [release notes](https://github.com/JohnSnowLabs/spark-nlp/releases), please pay attention to pick the one that suits your environment depending on the device (CPU/GPU) and Apache Spark version (3.0.x, 3.1.x, 3.2.x, and 3.3.x)
-- If you are local, you can load the Fat JAR from your local FileSystem, however, if you are in a cluster setup you need to put the Fat JAR on a distributed FileSystem such as HDFS, DBFS, S3, etc. (i.e., `hdfs:///tmp/spark-nlp-assembly-4.2.7.jar`)
+- If you are local, you can load the Fat JAR from your local FileSystem, however, if you are in a cluster setup you need to put the Fat JAR on a distributed FileSystem such as HDFS, DBFS, S3, etc. (i.e., `hdfs:///tmp/spark-nlp-assembly-4.2.8.jar`)
Example of using pretrained Models and Pipelines in offline:
diff --git a/build.sbt b/build.sbt
index 3f04c3e73bc281..c9d78fbd79b942 100644
--- a/build.sbt
+++ b/build.sbt
@@ -6,7 +6,7 @@ name := getPackageName(is_m1, is_gpu, is_aarch64)
organization := "com.johnsnowlabs.nlp"
-version := "4.2.7"
+version := "4.2.8"
(ThisBuild / scalaVersion) := scalaVer
diff --git a/conda/meta.yaml b/conda/meta.yaml
index 08b910b87acadd..5a0093c36adbd3 100644
--- a/conda/meta.yaml
+++ b/conda/meta.yaml
@@ -1,15 +1,15 @@
package:
name: "spark-nlp"
- version: 4.2.7
+ version: 4.2.8
app:
entry: spark-nlp
summary: Natural Language Understanding Library for Apache Spark.
source:
- fn: spark-nlp-4.2.7.tar.gz
- url: https://files.pythonhosted.org/packages/1d/e0/c123346f12e9d312c0b6bfecbd96db9e899882e01bc1adb338349d9e1088/spark-nlp-4.2.7.tar.gz
- sha256: 071f5b06ae10319cffe5a4fa22586a5b269800578e8a74de912abf123fd01bdf
+ fn: spark-nlp-4.2.8.tar.gz
+ url: https://files.pythonhosted.org/packages/5a/af/9c73a6a6a74f2848209001194bef19b74cfe04fdd070aec529d290ce239d/spark-nlp-4.2.8.tar.gz
+ sha256: 0573d006538808fd46a102f7efc79c6a7a37d68800e1b2cbf0607d0128a724f1
build:
noarch: generic
number: 0
diff --git a/docs/_includes/docs-healthcare-pagination.html b/docs/_includes/docs-healthcare-pagination.html
index 834b60f481577c..5079faa845f660 100644
--- a/docs/_includes/docs-healthcare-pagination.html
+++ b/docs/_includes/docs-healthcare-pagination.html
@@ -10,7 +10,7 @@
diff --git a/docs/_posts/Ahmetemintek/2022-10-19-ner_jsl_en.md b/docs/_posts/Ahmetemintek/2022-10-19-ner_jsl_en.md
index fd9dc457234dac..bde185afdffde8 100644
--- a/docs/_posts/Ahmetemintek/2022-10-19-ner_jsl_en.md
+++ b/docs/_posts/Ahmetemintek/2022-10-19-ner_jsl_en.md
@@ -10,6 +10,7 @@ language: en
edition: Spark NLP for Healthcare 4.2.0
spark_version: 3.0
supported: true
+recommended: true
annotator: MedicalNerModel
article_header:
type: cover
diff --git a/docs/_posts/C-K-Loan/2021-03-23-explain_document_dl_en.md b/docs/_posts/C-K-Loan/2021-03-23-explain_document_dl_en.md
index c080d8ef64fcab..a7a4e501a8785c 100644
--- a/docs/_posts/C-K-Loan/2021-03-23-explain_document_dl_en.md
+++ b/docs/_posts/C-K-Loan/2021-03-23-explain_document_dl_en.md
@@ -36,16 +36,16 @@ It performs most of the common text processing tasks on your dataframe
{% include programmingLanguageSelectScalaPythonNLU.html %}
```python
-from sparknlp.pretrained import PretrainedPipelinein
+from sparknlp.pretrained import PretrainedPipeline
pipeline = PretrainedPipeline('explain_document_dl', lang = 'en')
-annotations = pipeline.fullAnnotate(""Hello from John Snow Labs ! "")[0]
+annotations = pipeline.fullAnnotate("The Mona Lisa is an oil painting from the 16th century.")[0]
annotations.keys()
```
```scala
val pipeline = new PretrainedPipeline("explain_document_dl", lang = "en")
-val result = pipeline.fullAnnotate("Hello from John Snow Labs ! ")(0)
+val result = pipeline.fullAnnotate("The Mona Lisa is an oil painting from the 16th century.")(0)
```
@@ -54,7 +54,7 @@ val result = pipeline.fullAnnotate("Hello from John Snow Labs ! ")(0)
```python
import nlu
-text = [""Hello from John Snow Labs ! ""]
+text = ["The Mona Lisa is an oil painting from the 16th century."]
result_df = nlu.load('en.explain.dl').predict(text)
result_df
@@ -64,10 +64,11 @@ result_df
## Results
```bash
-| | document | sentence | token | checked | lemma | stem | pos | embeddings | ner | entities |
-|---:|:---------------------------------|:---------------------------------|:-------------------------------------------------|:------------------------------------------------|:------------------------------------------------|:-----------------------------------------------|:---------------------------------------|:-----------------------------|:-------------------------------------------------------|:---------------------------------|
-| 0 | ['Hello fronm John Snwow Labs!'] | ['Hello fronm John Snwow Labs!'] | ['Hello', 'fronm', 'John', 'Snwow', 'Labs', '!'] | ['Hello', 'front', 'John', 'Snow', 'Labs', '!'] | ['Hello', 'front', 'John', 'Snow', 'Labs', '!'] | ['hello', 'front', 'john', 'snow', 'lab', '!'] | ['UH', 'NN', 'NNP', 'NNP', 'NNP', '.'] | [[0.2668800055980682,.,...]] | ['B-ORG', 'I-ORG', 'I-ORG', 'I-ORG', 'I-ORG', 'I-ORG'] | ['Hello fronm John Snwow Labs!'] || | document | sentence | token | checked | lemma | stem | pos | embeddings | ner | entities |
-
++--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------+-----------+
+| text| document| sentence| token| checked| lemma| stem| pos| embeddings| ner| entities|
++--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------+-----------+
+|The Mona Lisa is an oil painting from the 16th ...|[The Mona Lisa is an oil painting from the 16th...|[The Mona Lisa is an oil painting from the 16th...|[The, Mona, Lisa, is, an, oil, painting, from, ...|[The, Mona, Lisa, is, an, oil, painting, from, ...|[The, Mona, Lisa, be, an, oil, painting, from, ...|[the, mona, lisa, i, an, oil, paint, from, the,...|[DT, NNP, NNP, VBZ, DT, NN, NN, IN, DT, JJ, NN, .]|[[-0.038194, -0.24487, 0.72812, -0.39961, 0.083...|[O, B-PER, I-PER, O, O, O, O, O, O, O, O, O]|[Mona Lisa]|
++--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------------+--------------------------------------------+-----------+
```
{:.model-param}
diff --git a/docs/_posts/Cabir40/2022-05-10-sbiobertresolve_cpt_procedures_measurements_augmented_en_3_0.md b/docs/_posts/Cabir40/2022-05-10-sbiobertresolve_cpt_procedures_measurements_augmented_en_3_0.md
index 8b6ff56af4ae07..ce958e88306ad8 100644
--- a/docs/_posts/Cabir40/2022-05-10-sbiobertresolve_cpt_procedures_measurements_augmented_en_3_0.md
+++ b/docs/_posts/Cabir40/2022-05-10-sbiobertresolve_cpt_procedures_measurements_augmented_en_3_0.md
@@ -10,6 +10,7 @@ language: en
edition: Healthcare NLP 3.5.1
spark_version: 3.0
supported: true
+recommended: true
article_header:
type: cover
use_language_switcher: "Python-Scala-Java"
@@ -42,55 +43,54 @@ This model maps medical entities to CPT codes using Sentence Bert Embeddings. Th
```python
document_assembler = DocumentAssembler()\
-.setInputCol("text")\
-.setOutputCol("document")
+ .setInputCol("text")\
+ .setOutputCol("document")
sentenceDetectorDL = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models") \
-.setInputCols(["document"])\
-.setOutputCol("sentence")
+ .setInputCols(["document"])\
+ .setOutputCol("sentence")
tokenizer = Tokenizer()\
-.setInputCols(["sentence"])\
-.setOutputCol("token")
+ .setInputCols(["sentence"])\
+ .setOutputCol("token")
word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")\
-.setInputCols(["sentence", "token"])\
-.setOutputCol("word_embeddings")
+ .setInputCols(["sentence", "token"])\
+ .setOutputCol("word_embeddings")
ner = MedicalNerModel.pretrained("ner_jsl", "en", "clinical/models") \
-.setInputCols(["sentence", "token", "word_embeddings"]) \
-.setOutputCol("ner")\
+ .setInputCols(["sentence", "token", "word_embeddings"]) \
+ .setOutputCol("ner")\
ner_converter = NerConverterInternal()\
-.setInputCols(["sentence", "token", "ner"])\
-.setOutputCol("ner_chunk")\
-.setWhiteList(["Procedure", "Test"])
+ .setInputCols(["sentence", "token", "ner"])\
+ .setOutputCol("ner_chunk")\
+ .setWhiteList(["Procedure", "Test"])
c2doc = Chunk2Doc()\
-.setInputCols("ner_chunk")\
-.setOutputCol("ner_chunk_doc")
+ .setInputCols("ner_chunk")\
+ .setOutputCol("ner_chunk_doc")
sbert_embedder = BertSentenceEmbeddings.pretrained("sbiobert_base_cased_mli", "en","clinical/models")\
-.setInputCols(["ner_chunk_doc"])\
-.setOutputCol("sentence_embeddings")\
-.setCaseSensitive(False)
+ .setInputCols(["ner_chunk_doc"])\
+ .setOutputCol("sentence_embeddings")\
+ .setCaseSensitive(False)
cpt_resolver = SentenceEntityResolverModel.pretrained("sbiobertresolve_cpt_procedures_measurements_augmented", "en", "clinical/models")\
-.setInputCols(["ner_chunk", "sentence_embeddings"]) \
-.setOutputCol("cpt_code")\
-.setDistanceFunction("EUCLIDEAN")
+ .setInputCols(["sentence_embeddings"]) \
+ .setOutputCol("cpt_code")\
+ .setDistanceFunction("EUCLIDEAN")
resolver_pipeline = Pipeline(stages = [
-document_assembler,
-sentenceDetectorDL,
-tokenizer,
-word_embeddings,
-ner,
-ner_converter,
-c2doc,
-sbert_embedder,
-cpt_resolver
-])
+ document_assembler,
+ sentenceDetectorDL,
+ tokenizer,
+ word_embeddings,
+ ner,
+ ner_converter,
+ c2doc,
+ sbert_embedder,
+ cpt_resolver])
model = resolver_pipeline.fit(spark.createDataFrame([[""]]).toDF("text"))
@@ -104,54 +104,54 @@ result = model.transform(data)
```
```scala
val document_assembler = new DocumentAssembler()
-.setInputCol("text")
-.setOutputCol("document")
+ .setInputCol("text")
+ .setOutputCol("document")
val sentenceDetectorDL = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare", "en", "clinical/models")
-.setInputCols(Array("document"))
-.setOutputCol("sentence")
+ .setInputCols(Array("document"))
+ .setOutputCol("sentence")
val tokenizer = new Tokenizer()
-.setInputCols(Array("sentence"))
-.setOutputCol("token")
+ .setInputCols(Array("sentence"))
+ .setOutputCol("token")
val word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
-.setInputCols(Array("sentence", "token"))
-.setOutputCol("word_embeddings")
+ .setInputCols(Array("sentence", "token"))
+ .setOutputCol("word_embeddings")
val ner = MedicalNerModel.pretrained("ner_jsl", "en", "clinical/models")
-.setInputCols(Array("sentence", "token", "word_embeddings"))
-.setOutputCol("ner")
+ .setInputCols(Array("sentence", "token", "word_embeddings"))
+ .setOutputCol("ner")
val ner_converter = new NerConverterInternal()
-.setInputCols(Array("sentence", "token", "ner"))
-.setOutputCol("ner_chunk")
-.setWhiteList(Array("Procedure", "Test"))
+ .setInputCols(Array("sentence", "token", "ner"))
+ .setOutputCol("ner_chunk")
+ .setWhiteList(Array("Procedure", "Test"))
val c2doc = new Chunk2Doc()
-.setInputCols(Array("ner_chunk"))
-.setOutputCol("ner_chunk_doc")
+ .setInputCols(Array("ner_chunk"))
+ .setOutputCol("ner_chunk_doc")
val sbert_embedder = BertSentenceEmbeddings.pretrained("sbiobert_base_cased_mli", "en", "clinical/models")
-.setInputCols(Array("ner_chunk_doc"))
-.setOutputCol("sentence_embeddings")
-.setCaseSensitive(False)
+ .setInputCols(Array("ner_chunk_doc"))
+ .setOutputCol("sentence_embeddings")
+ .setCaseSensitive(False)
val cpt_resolver = SentenceEntityResolverModel.pretrained("sbiobertresolve_cpt_procedures_measurements_augmented", "en", "clinical/models")
-.setInputCols(Array("ner_chunk", "sentence_embeddings"))
-.setOutputCol("cpt_code")
-.setDistanceFunction("EUCLIDEAN")
+ .setInputCols(Array("sentence_embeddings"))
+ .setOutputCol("cpt_code")
+ .setDistanceFunction("EUCLIDEAN")
val resolver_pipeline = new PipelineModel().setStages(Array(
- document_assembler,
-sentenceDetectorDL,
-tokenizer,
-word_embeddings,
- ner,
-ner_converter,
-c2doc,
-sbert_embedder,
-cpt_resolver))
+ document_assembler,
+ sentenceDetectorDL,
+ tokenizer,
+ word_embeddings,
+ ner,
+ ner_converter,
+ c2doc,
+ sbert_embedder,
+ cpt_resolver))
val data = Seq("She was admitted to the hospital with chest pain and found to have bilateral pleural effusion, the right greater than the left. CT scan of the chest also revealed a large mediastinal lymph node. We reviewed the pathology obtained from the pericardectomy in March 2006, which was diagnostic of mesothelioma. At this time, chest tube placement for drainage of the fluid occurred and thoracoscopy, which were performed, which revealed epithelioid malignant mesothelioma.").toDS.toDF("text")
@@ -210,6 +210,4 @@ At this time, chest tube placement for drainage of the fluid occurred and thorac
Trained on Current Procedural Terminology dataset with `sbiobert_base_cased_mli` sentence embeddings.
-
+
diff --git a/docs/_posts/Cabir40/2022-12-20-generic_re.md b/docs/_posts/Cabir40/2022-12-20-generic_re.md
index 0dafa5daaaa9d7..66deeb35140dbf 100644
--- a/docs/_posts/Cabir40/2022-12-20-generic_re.md
+++ b/docs/_posts/Cabir40/2022-12-20-generic_re.md
@@ -2,7 +2,7 @@
layout: model
title: Relation Extraction between generic entities
author: John Snow Labs
-name: posology_re
+name: generic_re
date: 2022-12-20
task: Relation Extraction
language: en
@@ -137,7 +137,7 @@ val re_Model = RelationExtractionModel()
.pretrained("generic_re")
.setInputCols(Array("embeddings", "pos_tags", "ner_chunks", "dependencies"))
.setOutputCol("relations")
- .setRelationPairs(["Biomarker-Biomarker_Result", "Biomarker_Result-Biomarker", "Oncogene-Biomarker_Result", "Biomarker_Result-Oncogene", "Pathology_Test-Pathology_Result", "Pathology_Result-Pathology_Test"]) \
+ .setRelationPairs(Array("Biomarker-Biomarker_Result", "Biomarker_Result-Biomarker", "Oncogene-Biomarker_Result", "Biomarker_Result-Oncogene", "Pathology_Test-Pathology_Result", "Pathology_Result-Pathology_Test")) \
.setMaxSyntacticDistance(4)
val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, tokenizer, words_embedder, pos_tagger, ner_tagger, ner_chunker, dependecy_parser, re_Model))
diff --git a/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_base_cuad_finetuned_en.md b/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_base_cuad_finetuned_en.md
deleted file mode 100644
index 6c0324e01a35a4..00000000000000
--- a/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_base_cuad_finetuned_en.md
+++ /dev/null
@@ -1,88 +0,0 @@
----
-layout: model
-title: English RobertaForQuestionAnswering Base Cased model (from gustavhartz)
-author: John Snow Labs
-name: roberta_qa_base_cuad_finetuned
-date: 2022-12-02
-tags: [en, open_source, roberta, question_answering, tensorflow]
-task: Question Answering
-language: en
-edition: Spark NLP 4.2.4
-spark_version: 3.0
-supported: true
-engine: tensorflow
-article_header:
- type: cover
-use_language_switcher: "Python-Scala-Java"
----
-
-## Description
-
-Pretrained RobertaForQuestionAnswering model, adapted from Hugging Face and curated to provide scalability and production-readiness using Spark NLP. `roberta-base-cuad-finetuned` is a English model originally trained by `gustavhartz`.
-
-{:.btn-box}
-
-
-[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/roberta_qa_base_cuad_finetuned_en_4.2.4_3.0_1669986303600.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
-[Copy S3 URI](s3://auxdata.johnsnowlabs.com/public/models/roberta_qa_base_cuad_finetuned_en_4.2.4_3.0_1669986303600.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
-
-## How to use
-
-
-
-
-{% include programmingLanguageSelectScalaPythonNLU.html %}
-```python
-Document_Assembler = MultiDocumentAssembler()\
- .setInputCols(["question", "context"])\
- .setOutputCols(["document_question", "document_context"])
-
-Question_Answering = RoBertaForQuestionAnswering.pretrained("roberta_qa_base_cuad_finetuned","en")\
- .setInputCols(["document_question", "document_context"])\
- .setOutputCol("answer")\
- .setCaseSensitive(True)
-
-pipeline = Pipeline(stages=[Document_Assembler, Question_Answering])
-
-data = spark.createDataFrame([["What's my name?","My name is Clara and I live in Berkeley."]]).toDF("question", "context")
-
-result = pipeline.fit(data).transform(data)
-```
-```scala
-val Document_Assembler = new MultiDocumentAssembler()
- .setInputCols(Array("question", "context"))
- .setOutputCols(Array("document_question", "document_context"))
-
-val Question_Answering = RoBertaForQuestionAnswering.pretrained("roberta_qa_base_cuad_finetuned","en")
- .setInputCols(Array("document_question", "document_context"))
- .setOutputCol("answer")
- .setCaseSensitive(True)
-
-val pipeline = new Pipeline().setStages(Array(Document_Assembler, Question_Answering))
-
-val data = Seq("What's my name?","My name is Clara and I live in Berkeley.").toDS.toDF("question", "context")
-
-val result = pipeline.fit(data).transform(data)
-```
-
-
-{:.model-param}
-## Model Information
-
-{:.table-model}
-|---|---|
-|Model Name:|roberta_qa_base_cuad_finetuned|
-|Compatibility:|Spark NLP 4.2.4+|
-|License:|Open Source|
-|Edition:|Official|
-|Input Labels:|[document, token]|
-|Output Labels:|[class]|
-|Language:|en|
-|Size:|454.1 MB|
-|Case sensitive:|true|
-|Max sentence length:|256|
-
-## References
-
-- https://huggingface.co/gustavhartz/roberta-base-cuad-finetuned
-- https://github.com/gustavhartz/transformers-legal-tasks
\ No newline at end of file
diff --git a/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_base_on_cuad_en.md b/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_base_on_cuad_en.md
deleted file mode 100644
index 1cf15af1cb2997..00000000000000
--- a/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_base_on_cuad_en.md
+++ /dev/null
@@ -1,93 +0,0 @@
----
-layout: model
-title: English RobertaForQuestionAnswering Base Cased model (from Rakib)
-author: John Snow Labs
-name: roberta_qa_base_on_cuad
-date: 2022-12-02
-tags: [en, open_source, roberta, question_answering, tensorflow]
-task: Question Answering
-language: en
-edition: Spark NLP 4.2.4
-spark_version: 3.0
-supported: true
-engine: tensorflow
-article_header:
- type: cover
-use_language_switcher: "Python-Scala-Java"
----
-
-## Description
-
-Pretrained RobertaForQuestionAnswering model, adapted from Hugging Face and curated to provide scalability and production-readiness using Spark NLP. `roberta-base-on-cuad` is a English model originally trained by `Rakib`.
-
-{:.btn-box}
-
-
-[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/roberta_qa_base_on_cuad_en_4.2.4_3.0_1669986363986.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
-[Copy S3 URI](s3://auxdata.johnsnowlabs.com/public/models/roberta_qa_base_on_cuad_en_4.2.4_3.0_1669986363986.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
-
-## How to use
-
-
-
-
-{% include programmingLanguageSelectScalaPythonNLU.html %}
-```python
-Document_Assembler = MultiDocumentAssembler()\
- .setInputCols(["question", "context"])\
- .setOutputCols(["document_question", "document_context"])
-
-Question_Answering = RoBertaForQuestionAnswering.pretrained("roberta_qa_base_on_cuad","en")\
- .setInputCols(["document_question", "document_context"])\
- .setOutputCol("answer")\
- .setCaseSensitive(True)
-
-pipeline = Pipeline(stages=[Document_Assembler, Question_Answering])
-
-data = spark.createDataFrame([["What's my name?","My name is Clara and I live in Berkeley."]]).toDF("question", "context")
-
-result = pipeline.fit(data).transform(data)
-```
-```scala
-val Document_Assembler = new MultiDocumentAssembler()
- .setInputCols(Array("question", "context"))
- .setOutputCols(Array("document_question", "document_context"))
-
-val Question_Answering = RoBertaForQuestionAnswering.pretrained("roberta_qa_base_on_cuad","en")
- .setInputCols(Array("document_question", "document_context"))
- .setOutputCol("answer")
- .setCaseSensitive(True)
-
-val pipeline = new Pipeline().setStages(Array(Document_Assembler, Question_Answering))
-
-val data = Seq("What's my name?","My name is Clara and I live in Berkeley.").toDS.toDF("question", "context")
-
-val result = pipeline.fit(data).transform(data)
-```
-
-
-{:.model-param}
-## Model Information
-
-{:.table-model}
-|---|---|
-|Model Name:|roberta_qa_base_on_cuad|
-|Compatibility:|Spark NLP 4.2.4+|
-|License:|Open Source|
-|Edition:|Official|
-|Input Labels:|[document, token]|
-|Output Labels:|[class]|
-|Language:|en|
-|Size:|453.7 MB|
-|Case sensitive:|true|
-|Max sentence length:|256|
-
-## References
-
-- https://huggingface.co/Rakib/roberta-base-on-cuad
-- https://github.com/facebookresearch/fairseq/tree/main/examples/roberta
-- https://arxiv.org/abs/1907.11692
-- https://aclanthology.org/2021.acl-long.330.pdf
-- https://dl.acm.org/doi/pdf/10.1145/3442188.3445922
-- https://mlco2.github.io/impact#compute
-- https://arxiv.org/abs/1910.09700
\ No newline at end of file
diff --git a/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_deepset_base_squad2_en.md b/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_deepset_base_squad2_en.md
index 4277237aa2eafe..5e9709db4c5d83 100644
--- a/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_deepset_base_squad2_en.md
+++ b/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_deepset_base_squad2_en.md
@@ -10,6 +10,7 @@ language: en
edition: Spark NLP 4.2.4
spark_version: 3.0
supported: true
+recommended: true
engine: tensorflow
article_header:
type: cover
diff --git a/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_marshmellow77_base_cuad_en.md b/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_marshmellow77_base_cuad_en.md
deleted file mode 100644
index 6dd00808ae502a..00000000000000
--- a/docs/_posts/Damla-Gurbaz/2022-12-02-roberta_qa_marshmellow77_base_cuad_en.md
+++ /dev/null
@@ -1,91 +0,0 @@
----
-layout: model
-title: English RobertaForQuestionAnswering Base Cased model (from marshmellow77)
-author: John Snow Labs
-name: roberta_qa_marshmellow77_base_cuad
-date: 2022-12-02
-tags: [en, open_source, roberta, question_answering, tensorflow]
-task: Question Answering
-language: en
-edition: Spark NLP 4.2.4
-spark_version: 3.0
-supported: true
-engine: tensorflow
-article_header:
- type: cover
-use_language_switcher: "Python-Scala-Java"
----
-
-## Description
-
-Pretrained RobertaForQuestionAnswering model, adapted from Hugging Face and curated to provide scalability and production-readiness using Spark NLP. `roberta-base-cuad` is a English model originally trained by `marshmellow77`.
-
-{:.btn-box}
-
-
-[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/roberta_qa_marshmellow77_base_cuad_en_4.2.4_3.0_1669986245674.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
-[Copy S3 URI](s3://auxdata.johnsnowlabs.com/public/models/roberta_qa_marshmellow77_base_cuad_en_4.2.4_3.0_1669986245674.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
-
-## How to use
-
-
-
-
-{% include programmingLanguageSelectScalaPythonNLU.html %}
-```python
-Document_Assembler = MultiDocumentAssembler()\
- .setInputCols(["question", "context"])\
- .setOutputCols(["document_question", "document_context"])
-
-Question_Answering = RoBertaForQuestionAnswering.pretrained("roberta_qa_marshmellow77_base_cuad","en")\
- .setInputCols(["document_question", "document_context"])\
- .setOutputCol("answer")\
- .setCaseSensitive(True)
-
-pipeline = Pipeline(stages=[Document_Assembler, Question_Answering])
-
-data = spark.createDataFrame([["What's my name?","My name is Clara and I live in Berkeley."]]).toDF("question", "context")
-
-result = pipeline.fit(data).transform(data)
-```
-```scala
-val Document_Assembler = new MultiDocumentAssembler()
- .setInputCols(Array("question", "context"))
- .setOutputCols(Array("document_question", "document_context"))
-
-val Question_Answering = RoBertaForQuestionAnswering.pretrained("roberta_qa_marshmellow77_base_cuad","en")
- .setInputCols(Array("document_question", "document_context"))
- .setOutputCol("answer")
- .setCaseSensitive(True)
-
-val pipeline = new Pipeline().setStages(Array(Document_Assembler, Question_Answering))
-
-val data = Seq("What's my name?","My name is Clara and I live in Berkeley.").toDS.toDF("question", "context")
-
-val result = pipeline.fit(data).transform(data)
-```
-
-
-{:.model-param}
-## Model Information
-
-{:.table-model}
-|---|---|
-|Model Name:|roberta_qa_marshmellow77_base_cuad|
-|Compatibility:|Spark NLP 4.2.4+|
-|License:|Open Source|
-|Edition:|Official|
-|Input Labels:|[document, token]|
-|Output Labels:|[class]|
-|Language:|en|
-|Size:|447.9 MB|
-|Case sensitive:|true|
-|Max sentence length:|256|
-
-## References
-
-- https://huggingface.co/marshmellow77/roberta-base-cuad
-- https://github.com/TheAtticusProject/cuad
-- https://github.com/marshmellow77/cuad-demo
-- https://towardsdatascience.com/how-to-set-up-a-machine-learning-model-for-legal-contract-review-fe3b48b05a0e
-- https://towardsdatascience.com/how-to-set-up-a-machine-learning-model-for-legal-contract-review-part-2-6ecbbe680ba
\ No newline at end of file
diff --git a/docs/_posts/Damla-Gurbaz/2023-01-03-distilbert_qa_base_cased_squadv2_xx.md b/docs/_posts/Damla-Gurbaz/2023-01-03-distilbert_qa_base_cased_squadv2_xx.md
index 13a0396f9bc60b..5b9d65a0e1449d 100644
--- a/docs/_posts/Damla-Gurbaz/2023-01-03-distilbert_qa_base_cased_squadv2_xx.md
+++ b/docs/_posts/Damla-Gurbaz/2023-01-03-distilbert_qa_base_cased_squadv2_xx.md
@@ -10,6 +10,7 @@ language: xx
edition: Spark NLP 4.3.0
spark_version: 3.0
supported: true
+recommended: true
engine: tensorflow
article_header:
type: cover
diff --git a/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_alcohol_usage_binary_sbiobert_cased_mli_en.md b/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_alcohol_usage_binary_sbiobert_cased_mli_en.md
index 18109047eff82d..5f33f05a360bee 100644
--- a/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_alcohol_usage_binary_sbiobert_cased_mli_en.md
+++ b/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_alcohol_usage_binary_sbiobert_cased_mli_en.md
@@ -1,6 +1,6 @@
---
layout: model
-title: SDOH Alcohol Usege For Binary Classification
+title: SDOH Alcohol Usage For Binary Classification
author: John Snow Labs
name: genericclassifier_sdoh_alcohol_usage_binary_sbiobert_cased_mli
date: 2023-01-14
@@ -10,6 +10,7 @@ language: en
edition: Healthcare NLP 4.2.4
spark_version: 3.0
supported: true
+recommended: true
article_header:
type: cover
use_language_switcher: "Python-Scala-Java"
@@ -134,4 +135,4 @@ val result = pipeline.fit(data).transform(data)
accuracy - - 0.85 1380
macro-avg 0.85 0.85 0.85 1380
weighted-avg 0.85 0.85 0.85 1380
-```
\ No newline at end of file
+```
diff --git a/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_alcohol_usage_sbiobert_cased_mli_en.md b/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_alcohol_usage_sbiobert_cased_mli_en.md
index 01d9f2a9e14424..8e6eaad24f571d 100644
--- a/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_alcohol_usage_sbiobert_cased_mli_en.md
+++ b/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_alcohol_usage_sbiobert_cased_mli_en.md
@@ -1,6 +1,6 @@
---
layout: model
-title: SDOH Alcohol Usege For Classification
+title: SDOH Alcohol Usage For Classification
author: John Snow Labs
name: genericclassifier_sdoh_alcohol_usage_sbiobert_cased_mli
date: 2023-01-14
@@ -10,6 +10,7 @@ language: en
edition: Healthcare NLP 4.2.4
spark_version: 3.0
supported: true
+recommended: true
article_header:
type: cover
use_language_switcher: "Python-Scala-Java"
@@ -141,4 +142,4 @@ val result = pipeline.fit(data).transform(data)
macro-avg 0.73 0.70 0.71 1380
weighted-avg 0.78 0.79 0.78 1380
-```
\ No newline at end of file
+```
diff --git a/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_economics_binary_sbiobert_cased_mli_en.md b/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_economics_binary_sbiobert_cased_mli_en.md
index cbac233cb3e747..0661c246bd1acb 100644
--- a/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_economics_binary_sbiobert_cased_mli_en.md
+++ b/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_economics_binary_sbiobert_cased_mli_en.md
@@ -10,6 +10,7 @@ language: en
edition: Healthcare NLP 4.2.4
spark_version: 3.0
supported: true
+recommended: true
article_header:
type: cover
use_language_switcher: "Python-Scala-Java"
diff --git a/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_substance_usage_binary_sbiobert_cased_mli_en.md b/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_substance_usage_binary_sbiobert_cased_mli_en.md
index 2054a87ec657af..94e53f8b092cbc 100644
--- a/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_substance_usage_binary_sbiobert_cased_mli_en.md
+++ b/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_substance_usage_binary_sbiobert_cased_mli_en.md
@@ -10,6 +10,7 @@ language: en
edition: Healthcare NLP 4.2.4
spark_version: 3.0
supported: true
+recommended: true
article_header:
type: cover
use_language_switcher: "Python-Scala-Java"
@@ -17,7 +18,7 @@ use_language_switcher: "Python-Scala-Java"
## Description
-This Generic Classifier model is intended for detecting substance use in clinical notes and trained by using GenericClassifierApproach annotator. `Present:` if the patient was a current consumer of alcohol or the patient was a consumer in the past and had quit or if the patient had never consumed alcohol. `None:` if there was no related text.
+This Generic Classifier model is intended for detecting substance use in clinical notes and trained by using GenericClassifierApproach annotator. `Present:` if the patient was a current consumer of substance or the patient was a consumer in the past and had quit or if the patient had never consumed substance. `None:` if there was no related text.
## Predicted Entities
diff --git a/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_tobacco_usage_sbiobert_cased_mli_en.md b/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_tobacco_usage_sbiobert_cased_mli_en.md
index 1e082276ba6110..5f4a18775d7761 100644
--- a/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_tobacco_usage_sbiobert_cased_mli_en.md
+++ b/docs/_posts/Damla-Gurbaz/2023-01-14-genericclassifier_sdoh_tobacco_usage_sbiobert_cased_mli_en.md
@@ -10,6 +10,7 @@ language: en
edition: Healthcare NLP 4.2.4
spark_version: 3.0
supported: true
+recommended: true
article_header:
type: cover
use_language_switcher: "Python-Scala-Java"
@@ -17,7 +18,7 @@ use_language_switcher: "Python-Scala-Java"
## Description
-This Generic Classifier model is intended for detecting tobacco use in clinical notes and trained by using GenericClassifierApproach annotator. `Present:` if the patient was a current consumer of alcohol. `Past:` the patient was a consumer in the past and had quit. `Never:` if the patient had never consumed alcohol. `None: ` if there was no related text.
+This Generic Classifier model is intended for detecting tobacco use in clinical notes and trained by using GenericClassifierApproach annotator. `Present:` if the patient was a current consumer of tobacco. `Past:` the patient was a consumer in the past and had quit. `Never:` if the patient had never consumed tobacco. `None: ` if there was no related text.
## Predicted Entities
diff --git a/docs/_posts/HashamUlHaq/2021-02-04-redl_drug_drug_interaction_biobert_en.md b/docs/_posts/HashamUlHaq/2021-02-04-redl_drug_drug_interaction_biobert_en.md
index a815ea212d9b41..9d7e5f12e6410c 100644
--- a/docs/_posts/HashamUlHaq/2021-02-04-redl_drug_drug_interaction_biobert_en.md
+++ b/docs/_posts/HashamUlHaq/2021-02-04-redl_drug_drug_interaction_biobert_en.md
@@ -172,10 +172,11 @@ they should be used with caution because the pharmacologically predictable sympa
## Results
```bash
-| | relation | entity1 | entity1_begin | entity1_end | chunk1 | entity2 | entity2_begin | entity2_end | chunk2 | confidence |
-|---:|:-----------|:----------|----------------:|--------------:|:--------------|:----------|----------------:|--------------:|:-------------|-------------:|
-| 0 | DDI-advise | DRUG | 5 | 17 | carbamazepine | DRUG | 62 | 73 | aripiprazole | 0.99238 |
-
++---------+-------+-------------+-----------+-------------+-------+-------------+-----------+------------+----------+
+| relation|entity1|entity1_begin|entity1_end| chunk1|entity2|entity2_begin|entity2_end| chunk2|confidence|
++---------+-------+-------------+-----------+-------------+-------+-------------+-----------+------------+----------+
+|DDI-false| DRUG| 5| 17|carbamazepine| DRUG| 62| 73|aripiprazole|0.91685396|
++---------+-------+-------------+-----------+-------------+-------+-------------+-----------+------------+----------+
```
{:.model-param}
diff --git a/docs/_posts/HashamUlHaq/2021-02-04-redl_temporal_events_biobert_en.md b/docs/_posts/HashamUlHaq/2021-02-04-redl_temporal_events_biobert_en.md
index ea23e6153f40f0..3d1b212314c99b 100644
--- a/docs/_posts/HashamUlHaq/2021-02-04-redl_temporal_events_biobert_en.md
+++ b/docs/_posts/HashamUlHaq/2021-02-04-redl_temporal_events_biobert_en.md
@@ -166,12 +166,20 @@ nlu.load("en.relation.temporal_events").predict("""She is diagnosed with cancer
## Results
```bash
-| | | relation | entity1 | entity1_begin | entity1_end | chunk1 | entity2 | entity2_begin | entity2_end | chunk2 | confidence |
-|---:|----:|:-----------|:-----------|----------------:|--------------:|:---------|:--------------|----------------:|--------------:|:------------|-------------:|
-| 0 | 0 | OVERLAP | PROBLEM | 20 | 25 | cancer | DATE | 30 | 33 | 1991 | 0.868028 |
-| 1 | 1 | AFTER | OCCURRENCE | 49 | 56 | admitted | CLINICAL_DEPT | 61 | 71 | Mayo Clinic | 0.798134 |
-| 2 | 2 | BEFORE | OCCURRENCE | 49 | 56 | admitted | OCCURRENCE | 89 | 98 | discharged | 0.63638 |
-
++--------+-------------+-------------+-----------+-----------+-------------+-------------+-----------+------------+----------+
+|relation| entity1|entity1_begin|entity1_end| chunk1| entity2|entity2_begin|entity2_end| chunk2|confidence|
++--------+-------------+-------------+-----------+-----------+-------------+-------------+-----------+------------+----------+
+| BEFORE| OCCURRENCE| 7| 15| diagnosed| PROBLEM| 22| 27| cancer|0.78168863|
+| OVERLAP| PROBLEM| 22| 27| cancer| DATE| 32| 35| 1991| 0.8492274|
+| AFTER| OCCURRENCE| 51| 58| admitted|CLINICAL_DEPT| 63| 73| Mayo Clinic|0.85629463|
+| BEFORE| OCCURRENCE| 51| 58| admitted| OCCURRENCE| 91| 100| discharged| 0.6843513|
+| OVERLAP|CLINICAL_DEPT| 63| 73|Mayo Clinic| DATE| 78| 85| May 2000| 0.7844673|
+| BEFORE|CLINICAL_DEPT| 63| 73|Mayo Clinic| OCCURRENCE| 91| 100| discharged|0.60411876|
+| OVERLAP|CLINICAL_DEPT| 63| 73|Mayo Clinic| DATE| 105| 116|October 2001| 0.540761|
+| BEFORE| DATE| 78| 85| May 2000| OCCURRENCE| 91| 100| discharged| 0.6042761|
+| OVERLAP| DATE| 78| 85| May 2000| DATE| 105| 116|October 2001|0.64867175|
+| BEFORE| OCCURRENCE| 91| 100| discharged| DATE| 105| 116|October 2001| 0.5302478|
++--------+-------------+-------------+-----------+-----------+-------------+-------------+-----------+------------+----------+
```
{:.model-param}
diff --git a/docs/_posts/HashamUlHaq/2021-07-12-redl_ade_biobert_en.md b/docs/_posts/HashamUlHaq/2021-07-12-redl_ade_biobert_en.md
index 23c875c7fb28a7..dd2980eac1896e 100644
--- a/docs/_posts/HashamUlHaq/2021-07-12-redl_ade_biobert_en.md
+++ b/docs/_posts/HashamUlHaq/2021-07-12-redl_ade_biobert_en.md
@@ -190,10 +190,10 @@ nlu.load("en.relation.adverse_drug_events.clinical.biobert").predict("""Been tak
```bash
-| relation | entity1 | entity1_begin | entity1_end | chunk1 | entity2 | entity2_begin | entity2_end | chunk2 | confidence |
-|---------:|:--------|--------------:|------------:|:----------|:--------|--------------:|------------:|:---------------|-----------:|
-| 1 | DRUG | 12 | 18 | Lipitor | ADE | 52 | 65 | severe fatigue | 0.999317 |
-| 0 | DRUG | 97 | 105 | voltarene | ADE | 144 | 156 | muscle cramps | 0.774904 |
+| relation | entity1 | entity1_begin | entity1_end | chunk1 | entity2 | entity2_begin | entity2_end | chunk2 | confidence |
+|-----------:|:----------|----------------:|--------------:|:----------|:----------|----------------:|--------------:|:---------------|-------------:|
+| 1 | DRUG | 12 | 18 | Lipitor | ADE | 52 | 65 | severe fatigue | 0.998156 |
+| 1 | DRUG | 97 | 105 | voltarene | ADE | 144 | 156 | muscle cramps | 0.985513 |
```
diff --git a/docs/_posts/HashamUlHaq/2021-11-16-redl_nihss_biobert_en.md b/docs/_posts/HashamUlHaq/2021-11-16-redl_nihss_biobert_en.md
index b9f320dfeb5188..e9f324080eb1dd 100644
--- a/docs/_posts/HashamUlHaq/2021-11-16-redl_nihss_biobert_en.md
+++ b/docs/_posts/HashamUlHaq/2021-11-16-redl_nihss_biobert_en.md
@@ -159,17 +159,17 @@ nlu.load("en.extract_relation.nihss").predict("""There , her initial NIHSS score
```bash
-| chunk1 | entity1 | entity1_begin | entity1_end | entity2 | chunk2 | entity2_begin | entity2_end | relation |
-|:--------------------------------------|:-------------|----------------:|--------------:|:------------|---------:|----------------:|--------------:|:-----------|
-| initial NIHSS score | NIHSS | 12 | 30 | Measurement | 4 | 36 | 36 | Has_Value |
-| left leg | 6a_LeftLeg | 111 | 118 | Measurement | 2 | 89 | 89 | Has_Value |
-| subtle ataxia in her left arm and leg | 7_LimbAtaxia | 149 | 185 | Measurement | 2 | 124 | 124 | Has_Value |
-| left leg | 6a_LeftLeg | 111 | 118 | Measurement | 4 | 36 | 36 | 0 |
-| initial NIHSS score | NIHSS | 12 | 30 | Measurement | 2 | 124 | 124 | 0 |
-| subtle ataxia in her left arm and leg | 7_LimbAtaxia | 149 | 185 | Measurement | 4 | 36 | 36 | 0 |
-| subtle ataxia in her left arm and leg | 7_LimbAtaxia | 149 | 185 | Measurement | 2 | 89 | 89 | 0 |
-
-
++---------+-----------+-------------+-----------+-----------+------------+-------------+-----------+--------------------+----------+
+| relation| entity1|entity1_begin|entity1_end| chunk1| entity2|entity2_begin|entity2_end| chunk2|confidence|
++---------+-----------+-------------+-----------+-----------+------------+-------------+-----------+--------------------+----------+
+|Has_Value| NIHSS| 20| 30|NIHSS score| Measurement| 36| 36| 4| 0.9998851|
+|Has_Value|Measurement| 89| 89| 2| 6a_LeftLeg| 111| 118| left leg| 0.9987311|
+| 0|Measurement| 89| 89| 2| Measurement| 124| 124| 2|0.97510725|
+| 0|Measurement| 89| 89| 2|7_LimbAtaxia| 156| 185|ataxia in her lef...| 0.999889|
+| 0| 6a_LeftLeg| 111| 118| left leg| Measurement| 124| 124| 2|0.99989617|
+| 0| 6a_LeftLeg| 111| 118| left leg|7_LimbAtaxia| 156| 185|ataxia in her lef...| 0.9999521|
+|Has_Value|Measurement| 124| 124| 2|7_LimbAtaxia| 156| 185|ataxia in her lef...| 0.9896319|
++---------+-----------+-------------+-----------+-----------+------------+-------------+-----------+--------------------+----------+
```
diff --git a/docs/_posts/Mary-Sci/2022-12-02-bert_embeddings_netbert_en.md b/docs/_posts/Mary-Sci/2022-12-02-bert_embeddings_netbert_en.md
index 54516abce0fc65..ff3cd218a4a441 100644
--- a/docs/_posts/Mary-Sci/2022-12-02-bert_embeddings_netbert_en.md
+++ b/docs/_posts/Mary-Sci/2022-12-02-bert_embeddings_netbert_en.md
@@ -10,6 +10,7 @@ language: en
edition: Spark NLP 4.2.4
spark_version: 3.0
supported: true
+recommended: true
article_header:
type: cover
use_language_switcher: "Python-Scala-Java"
diff --git a/docs/_posts/Meryem1425/2022-12-09-roberta_classifier_autotrain_neurips_chanllenge_1287149282_en.md b/docs/_posts/Meryem1425/2022-12-09-roberta_classifier_autotrain_neurips_chanllenge_1287149282_en.md
index bb94d5f690f882..670127716e0c9b 100644
--- a/docs/_posts/Meryem1425/2022-12-09-roberta_classifier_autotrain_neurips_chanllenge_1287149282_en.md
+++ b/docs/_posts/Meryem1425/2022-12-09-roberta_classifier_autotrain_neurips_chanllenge_1287149282_en.md
@@ -10,6 +10,7 @@ language: en
edition: Spark NLP 4.2.4
spark_version: 3.0
supported: true
+recommended: true
engine: tensorflow
article_header:
type: cover
diff --git a/docs/_posts/Meryem1425/2023-01-14-redl_ade_biobert_en.md b/docs/_posts/Meryem1425/2023-01-14-redl_ade_biobert_en.md
index 0d338e67e036a1..898efb1a0fee51 100644
--- a/docs/_posts/Meryem1425/2023-01-14-redl_ade_biobert_en.md
+++ b/docs/_posts/Meryem1425/2023-01-14-redl_ade_biobert_en.md
@@ -176,10 +176,10 @@ val result = pipeline.fit(data).transform(data)
## Results
```bash
-| relation | entity1 | entity1_begin | entity1_end | chunk1 | entity2 | entity2_begin | entity2_end | chunk2 | confidence |
-|---------:|:--------|--------------:|------------:|:----------|:--------|--------------:|------------:|:---------------|-----------:|
-| 1 | DRUG | 12 | 18 | Lipitor | ADE | 52 | 65 | severe fatigue | 0.999317 |
-| 0 | DRUG | 97 | 105 | voltarene | ADE | 144 | 156 | muscle cramps | 0.774904 |
+| relation | entity1 | entity1_begin | entity1_end | chunk1 | entity2 | entity2_begin | entity2_end | chunk2 | confidence |
+|-----------:|:----------|----------------:|--------------:|:----------|:----------|----------------:|--------------:|:---------------|-------------:|
+| 1 | DRUG | 12 | 18 | Lipitor | ADE | 52 | 65 | severe fatigue | 0.998156 |
+| 1 | DRUG | 97 | 105 | voltarene | ADE | 144 | 156 | muscle cramps | 0.985513 |
```
{:.model-param}
diff --git a/docs/_posts/ahmedlone127/2022-10-12-pipeline_image_classifier_vit_base_patch16_224_finetuned_eurosat_en.md b/docs/_posts/ahmedlone127/2022-10-12-pipeline_image_classifier_vit_base_patch16_224_finetuned_eurosat_en.md
index 281c12d71a9cc3..ad4939a1ddc0cf 100644
--- a/docs/_posts/ahmedlone127/2022-10-12-pipeline_image_classifier_vit_base_patch16_224_finetuned_eurosat_en.md
+++ b/docs/_posts/ahmedlone127/2022-10-12-pipeline_image_classifier_vit_base_patch16_224_finetuned_eurosat_en.md
@@ -10,6 +10,7 @@ language: en
edition: Spark NLP 4.2.1
spark_version: 3.0
supported: true
+recommended: true
annotator: PipelineModel
article_header:
type: cover
diff --git a/docs/_posts/aymanechilah/2023-01-03-image_processing_en_3_2.md b/docs/_posts/aymanechilah/2023-01-03-image_processing_en_3_2.md
new file mode 100644
index 00000000000000..e8d6055f6303ff
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-03-image_processing_en_3_2.md
@@ -0,0 +1,244 @@
+---
+layout: model
+title: Image Processing algorithms to improve Document Quality
+author: John Snow Labs
+name: image_processing
+date: 2023-01-03
+tags: [en, licensed, ocr, image_processing]
+task: Document Image Processing
+language: en
+edition: Visual NLP 4.0.0
+spark_version: 3.2.1
+supported: true
+annotator: ImageProcessing
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+The processing of documents for the purpose of discovering knowledge from them in an automated fashion is a challenging task and hence an open issue for the research community. Sometimes, the quality of the input images makes it much more difficult to perform these procedures correctly.
+
+To avoid this, it is proposed the use of different image processing algorithms on documents images to improve its quality and the performance of the next step of computer vision algorithms as text detection, text recognition, ocr, table detection... Some of these image processing algorithms included in this project are: scale image, adaptive thresholding, erosion, dilation, remove objects, median blur and gpu image transformation.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/tutorials/Certification_Trainings/1.2.Image_processing.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+
+
+## How to use
+
+
+
+## Example
+
+{%- capture input_image -%}
+
+{%- endcapture -%}
+
+{%- capture output_image -%}
+
+{%- endcapture -%}
+
+
+{% include templates/input_output_image.md
+input_image=input_image
+output_image=output_image
+%}
\ No newline at end of file
diff --git a/docs/_posts/aymanechilah/2023-01-03-ner_deid_large_en_3_2.md b/docs/_posts/aymanechilah/2023-01-03-ner_deid_large_en_3_2.md
new file mode 100644
index 00000000000000..e94db610c3362e
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-03-ner_deid_large_en_3_2.md
@@ -0,0 +1,271 @@
+---
+layout: model
+title: Image De-Identification
+author: John Snow Labs
+name: ner_deid_large
+date: 2023-01-03
+tags: [en, licensed, ocr, image_deidentification]
+task: Image DeIdentification
+language: en
+edition: Visual NLP 4.0.0
+spark_version: 3.2.1
+supported: true
+annotator: ImageDeIdentification
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Deidentification NER (Large) is a Named Entity Recognition model that annotates text to find protected health information that may need to be deidentified. The entities it annotates are `Age`, `Contact`, `Date`, `Id`, `Location`, `Name`, and `Profession`. This model is trained with the `embeddings_clinical` word embeddings model, so be sure to use the same embeddings in the pipeline.
+
+It protects specific health information that could identify living or deceased individuals. The rule preserves patient confidentiality without affecting the values and the information that could be needed for different research purposes.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/tutorials/Certification_Trainings/3.1.SparkOcrImageDeIdentification.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/ner_deid_large_en_3.0.0_3.0_1617209688468.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+## How to use
+
+
+
+## Example
+
+{%- capture input_image -%}
+
+{%- endcapture -%}
+
+{%- capture output_image -%}
+
+{%- endcapture -%}
+
+
+{% include templates/input_output_image.md
+input_image=input_image
+output_image=output_image
+%}
+
+## Output text
+
+```bash
++------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|ner_chunk |
++------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|[{chunk, 193, 202, 04/04/2018, {entity -> DATE, sentence -> 1, chunk -> 0, confidence -> 0.9999}, []}, {chunk, 3290, 3290, ., {entity -> NAME, sentence -> 17, chunk -> 1, confidence -> 0.6035}, []}, {chunk, 3388, 3397, 04/12/2018, {entity -> DATE, sentence -> 20, chunk -> 2, confidence -> 1.0}, []}]|
++------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+## Model Information
+
+{:.table-model}
+|---|---|
+|Model Name:|ner_clinical_large|
+|Compatibility:|Healthcare NLP 4.0.0+|
+|License:|Licensed|
+|Edition:|Official|
+|Input Labels:|[sentence, token, embeddings]|
+|Output Labels:|[ner]|
+|Language:|en|
+
+## Data Source
+
+Trained on augmented 2010 i2b2 challenge data with 'embeddings_clinical'.
+[https://portal.dbmi.hms.harvard.edu/projects/n2c2-nlp/](https://portal.dbmi.hms.harvard.edu/projects/n2c2-nlp/)
+
+## Benchmarking
+
+```bash
+ label tp fp fn prec rec f1
+ I-TREATMENT 6625 1187 1329 0.848054 0.832914 0.840416
+ I-PROBLEM 15142 1976 2542 0.884566 0.856254 0.87018
+ B-PROBLEM 11005 1065 1587 0.911765 0.873968 0.892466
+ I-TEST 6748 923 1264 0.879677 0.842237 0.86055
+ B-TEST 8196 942 1029 0.896914 0.888455 0.892665
+ B-TREATMENT 8271 1265 1073 0.867345 0.885167 0.876165
+Macro-average 55987 7358 8824 0.881387 0.863166 0.872181
+Micro-average 55987 7358 8824 0.883842 0.86385 0.873732
+```
+
+
+
diff --git a/docs/_posts/aymanechilah/2023-01-03-ocr_restapi_en_3_2.md b/docs/_posts/aymanechilah/2023-01-03-ocr_restapi_en_3_2.md
new file mode 100644
index 00000000000000..5cc10dbb9c7c78
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-03-ocr_restapi_en_3_2.md
@@ -0,0 +1,122 @@
+---
+layout: model
+title: Ocr pipeline with Rest-Api
+author: John Snow Labs
+name: ocr_restapi
+date: 2023-01-03
+tags: [en, licensed, ocr, RestApi]
+task: Ocr RestApi
+language: en
+edition: Visual NLP 4.0.0
+spark_version: 3.2.1
+supported: true
+annotator: OcrRestApi
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+RestAPI pipeline implementation for the OCR task, using tesseract models. Tesseract is an Optical Character Recognition (OCR) engine developed by Google. It is an open-source tool that can be used to recognize text in images and convert it into machine-readable text. The engine is based on a neural network architecture and uses machine learning algorithms to improve its accuracy over time.
+
+Tesseract has been trained on a variety of datasets to improve its recognition capabilities. These datasets include images of text in various languages and scripts, as well as images with different font styles, sizes, and orientations. The training process involves feeding the engine with a large number of images and their corresponding text, allowing the engine to learn the patterns and characteristics of different text styles. One of the most important datasets used in training Tesseract is the UNLV dataset, which contains over 400,000 images of text in different languages, scripts, and font styles. This dataset is widely used in the OCR community and has been instrumental in improving the accuracy of Tesseract. Other datasets that have been used in training Tesseract include the ICDAR dataset, the IIIT-HWS dataset, and the RRC-GV-WS dataset.
+
+In addition to these datasets, Tesseract also uses a technique called adaptive training, where the engine can continuously improve its recognition capabilities by learning from new images and text. This allows Tesseract to adapt to new text styles and languages, and improve its overall accuracy.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/tutorials/Certification_Trainings/6.2.SparkOcrRestApi.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+
+
+## How to use
+
+
+
+## Example
+
+### Input:
+
+
+## Output text
+
+```bash
+Response:
+
+STARBUCKS Store #19208
+11902 Euclid Avenue
+Cleveland, OH (216) 229-U749
+
+CHK 664250
+12/07/2014 06:43 PM
+112003. Drawers 2. Reg: 2
+
+¥t Pep Mocha 4.5
+Sbux Card 495
+AMXARKERARANG 228
+Subtotal $4.95
+Total $4.95
+Change Cue BO LOO
+- Check Closed ~
+
+"49/07/2014 06:43 py
+
+oBUX Card «3228 New Balance: 37.45
+Card is registertd
+```
diff --git a/docs/_posts/aymanechilah/2023-01-03-ocr_streaming_en_3_0.md b/docs/_posts/aymanechilah/2023-01-03-ocr_streaming_en_3_0.md
new file mode 100644
index 00000000000000..80b35ec16d5058
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-03-ocr_streaming_en_3_0.md
@@ -0,0 +1,139 @@
+---
+layout: model
+title: Ocr pipeline in streaming
+author: John Snow Labs
+name: ocr_streaming
+date: 2023-01-03
+tags: [en, licensed, ocr, streaming]
+task: Ocr Streaming
+language: en
+edition: Visual NLP 4.0.0
+spark_version: 3.0
+supported: true
+annotator: OcrStreaming
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Streaming pipeline implementation for the OCR task, using tesseract models. Tesseract is an Optical Character Recognition (OCR) engine developed by Google. It is an open-source tool that can be used to recognize text in images and convert it into machine-readable text. The engine is based on a neural network architecture and uses machine learning algorithms to improve its accuracy over time.
+
+Tesseract has been trained on a variety of datasets to improve its recognition capabilities. These datasets include images of text in various languages and scripts, as well as images with different font styles, sizes, and orientations. The training process involves feeding the engine with a large number of images and their corresponding text, allowing the engine to learn the patterns and characteristics of different text styles. One of the most important datasets used in training Tesseract is the UNLV dataset, which contains over 400,000 images of text in different languages, scripts, and font styles. This dataset is widely used in the OCR community and has been instrumental in improving the accuracy of Tesseract. Other datasets that have been used in training Tesseract include the ICDAR dataset, the IIIT-HWS dataset, and the RRC-GV-WS dataset.
+
+In addition to these datasets, Tesseract also uses a technique called adaptive training, where the engine can continuously improve its recognition capabilities by learning from new images and text. This allows Tesseract to adapt to new text styles and languages, and improve its overall accuracy.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/tutorials/Certification_Trainings/6.1.SparkOcrStreamingPDF.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+
+
+## How to use
+
+
+{% include programmingLanguageSelectScalaPythonNLU.html %}
+
+```python
+# Transform binary to image
+pdf_to_image = PdfToImage() \
+ .setInputCol("content") \
+ .setOutputCol("image")
+
+# Run OCR for each region
+ocr = ImageToText() \
+ .setInputCol("image") \
+ .setOutputCol("text") \
+ .setConfidenceThreshold(60)
+
+# OCR pipeline
+pipeline = PipelineModel(stages=[
+ pdf_to_image,
+ ocr])
+
+# fill path to folder with PDF's here
+dataset_path = "data/pdfs/*.pdf"
+# read one file for infer schema
+pdfs_df = spark.read.format("binaryFile").load(dataset_path).limit(1)
+
+# count of files in one microbatch
+maxFilesPerTrigger = 4
+
+# read files as stream
+pdf_stream_df = spark.readStream \
+ .format("binaryFile") \
+ .schema(pdfs_df.schema) \
+ .option("maxFilesPerTrigger", maxFilesPerTrigger) \
+ .load(dataset_path)
+
+# process files using OCR pipeline
+result = pipeline.transform(pdf_stream_df).withColumn("timestamp", current_timestamp())
+
+# store results to memory table
+query = result.writeStream \
+ .format('memory') \
+ .queryName('result') \
+ .start()
+
+# show results
+spark.table("result").select("timestamp","pagenum", "path", "text").show(10)
+```
+```scala
+# Transform binary to image
+val pdf_to_image = new PdfToImage()
+ .setInputCol("content")
+ .setOutputCol("image")
+
+# Run OCR for each region
+val ocr = new ImageToText()
+ .setInputCol("image")
+ .setOutputCol("text")
+ .setConfidenceThreshold(60)
+
+# OCR pipeline
+val pipeline = new PipelineModel().setStages(Array(
+ pdf_to_image,
+ ocr))
+
+# fill path to folder with PDF's here
+val dataset_path = "data/pdfs/*.pdf"
+# read one file for infer schema
+val pdfs_df = spark.read.format("binaryFile").load(dataset_path).limit(1)
+
+# count of files in one microbatch
+val maxFilesPerTrigger = 4
+
+# read files as stream
+val pdf_stream_df = spark.readStream
+ .format("binaryFile")
+ .schema(pdfs_df.schema)
+ .option("maxFilesPerTrigger", maxFilesPerTrigger)
+ .load(dataset_path)
+
+# process files using OCR pipeline
+val result = pipeline.transform(pdf_stream_df).withColumn("timestamp", current_timestamp())
+
+# store results to memory table
+val query = result.writeStream
+ .format("memory")
+ .queryName("result")
+ .start()
+
+# show results
+spark.table("result").select(Array("timestamp","pagenum", "path", "text")).show(10)
+```
+
+
+## Results
+
+```bash
++--------------------+-------+--------------------+--------------------+
+| timestamp|pagenum| path| text|
++--------------------+-------+--------------------+--------------------+
+|2022-07-20 21:45:...| 0|file:/content/dat...|FOREWORD\n\nElect...|
+|2022-07-20 21:45:...| 0|file:/content/dat...|C nca Document fo...|
+|2022-07-20 21:56:...| 0|file:/content/dat...|6/13/22, 11:47 AM...|
++--------------------+-------+--------------------+--------------------+
+```
diff --git a/docs/_posts/aymanechilah/2023-01-03-pdf_processing_en_3_2.md b/docs/_posts/aymanechilah/2023-01-03-pdf_processing_en_3_2.md
new file mode 100644
index 00000000000000..1ba7d6d8331279
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-03-pdf_processing_en_3_2.md
@@ -0,0 +1,124 @@
+---
+layout: model
+title: Pdf processing
+author: John Snow Labs
+name: pdf_processing
+date: 2023-01-03
+tags: [en, licensed, ocr, pdf_processing]
+task: Document Pdf Processing
+language: en
+edition: Visual NLP 4.0.0
+spark_version: 3.2.1
+supported: true
+annotator: PdfProcessing
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+This model obtain the text of an input PDF document with a Tesseract pretrained model. Tesseract is an Optical Character Recognition (OCR) engine developed by Google. It is an open-source tool that can be used to recognize text in images and convert it into machine-readable text. The engine is based on a neural network architecture and uses machine learning algorithms to improve its accuracy over time.
+
+Tesseract has been trained on a variety of datasets to improve its recognition capabilities. These datasets include images of text in various languages and scripts, as well as images with different font styles, sizes, and orientations. The training process involves feeding the engine with a large number of images and their corresponding text, allowing the engine to learn the patterns and characteristics of different text styles. One of the most important datasets used in training Tesseract is the UNLV dataset, which contains over 400,000 images of text in different languages, scripts, and font styles. This dataset is widely used in the OCR community and has been instrumental in improving the accuracy of Tesseract. Other datasets that have been used in training Tesseract include the ICDAR dataset, the IIIT-HWS dataset, and the RRC-GV-WS dataset.
+
+In addition to these datasets, Tesseract also uses a technique called adaptive training, where the engine can continuously improve its recognition capabilities by learning from new images and text. This allows Tesseract to adapt to new text styles and languages, and improve its overall accuracy.
+
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/tutorials/Certification_Trainings/2.1.Pdf_processing.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+
+
+## How to use
+
+
+
+## Example
+
+### Input:
+Representation of images 2 and 3 of the example:
+
+
+## Output text
+
+```bash
++--------------------+-------------------+------+--------------------+--------------------+-----------------+------------------+--------------------+--------------------+-----------+-------+-----------+----------------+---------+
+| path| modificationTime|length| text| positions| height_dimension| width_dimension| content| image|total_pages|pagenum|documentnum| confidence|exception|
++--------------------+-------------------+------+--------------------+--------------------+-----------------+------------------+--------------------+--------------------+-----------+-------+-----------+----------------+---------+
+|file:/Users/nmeln...|2022-07-14 15:38:51|693743|Patient Nam\nFina...|[{[{Patient Nam\n...|1587.780029296875|1205.8299560546875|[25 50 44 46 2D 3...|{file:/Users/nmel...| 1| 0| 0|81.2276874118381| null|
+|file:/Users/nmeln...|2022-07-14 15:38:51|693743|Patient Name\nFin...|[{[{Patient Name\...|1583.780029296875|1217.8299560546875|[25 50 44 46 2D 3...|{file:/Users/nmel...| 1| 0| 0|78.5234429732613| null|
+|file:/Users/nmeln...|2022-07-14 15:38:51| 70556|Alexandria is the...|[{[{A, 0, 72.024,...| 792.0| 612.0|[25 50 44 46 2D 3...| null| null| 0| 0| null| null|
+|file:/Users/nmeln...|2022-07-14 15:38:51| 70556|Alexandria was fo...|[{[{A, 1, 72.024,...| 792.0| 612.0|[25 50 44 46 2D 3...| null| null| 0| 0| null| null|
+|file:/Users/nmeln...|2022-07-14 15:38:51| 11601|8 i , . ! \n9 i ,...|[{[{8, 0, 72.0604...| 843.0| 596.0|[25 50 44 46 2D 3...| null| null| 0| 0| null| null|
++--------------------+-------------------+------+--------------------+--------------------+-----------------+------------------+--------------------+--------------------+-----------+-------+-----------+----------------+---------+
+```
+```bash
+text
+0 Patient Nam Financial Numbe Random Hospital Date of Birth Patient Location Chief Complaint Shortness of breath History of Present Illness Patient is an 84-year-old male wilh a past medical history of hypertension, HFpEF last known EF 55%, mild to moderate TA, pulmonary hypertension, permanent atrial fibrillation on Eliquis, history of GI blesd, CK-M8, and anemia who presents with full weeks oi ccneralized fatigue and fecling unwell. He also notes some shortness oi Breath and worsening dyspnea willy minimal exerlion. His major complaints are shoulder and joint pains. diffusely. He also complains of "bone pain’. He denics having any fevers or cnills. e demes having any chest pain, palpitalicns, He denies any worse extremity swelling than his baseline. He states he’s been compliant with his mcdications. Although he stales he ran out of his Eliquis & few weeks ago. He denies having any blood in his stools or mc!ena, although he does take iron pills and states his stools arc irequently black. His hemoglobin Is al baseline. Twelve-lead EKG showing atrial fibrillation, RBBB, LAFB, PVC. Chest x-ray showing new small right creater than left pleural effusions with mild pulmonary vascular congestion. BNP increased to 2800, up fram 1900. Tropoain 0.03. Renal function at baseline. Hemoaglopin at baseline. She normally takes 80 mq of oral Lasix daily. He was given 80 mg of IV Lasix in the ED. He is currently net negative close to 1 L. He is still on 2 L nasal cannula. ' Ss 5 A 10 system roview af systems was completed and negative except as documented in HPI. Physical Exam Vitals & Measurements T: 36.8 °C (Oral) TMIN: 36.8 "C (Oral) TMAX: 37.0 °C (Oral) HR: 54 RR: 7 BP: 140/63 WT: 100.3 KG Pulse Ox: 100 % Oxygen: 2 L'min via Nasal Cannula GENERAL: no acute distress HEAD: normecephalic EYES‘EARS‘NOSE/THAOAT: nupils are equal. normal oropharynx NECK: normal inspection RESPIRATORY: no respiratory distress, no rales on my exam CARDIOVASCULAR: irregular. brady. no murmurs, rubs or galleps ABDOMEN: soft, non-tendes EXTREMITIES: Bilateral chronic venous stasis changes NEUROLOGIC: alert and osieniec x 3. no gross motar or sensory deficils AssessmenvPlan Acute on chronic diastolic CHF (congestive heart failure) Acute on chronic diastolic heart failure exacerbation. Small pleural effusions dilaterally with mild pulmonary vascular congesiion on chest x-ray, slighi elevation in BNR. We'll continue 1 more day af IV diuresis with 80 mg IV Lasix. He may have had a viral infection which precipilated this. We'll add Tylenol jor his joint paias. Continue atenclol and chiorthalidone. AF - Atrial fibrillation Permanent atrial fibrillation. Rates bradycardic in the &0s. Continue atenolol with hola parameters. Coniinue Eliquis for stroke prevention. No evidence oj bleeding, hemog'abin at baseline. Printed: 7/17/2017 13:01 EDT Page 16 of 42 Arincitis CHF - Congestive heart failure Chronic kidney disease Chronic venous insufficiency Edema GI bleeding Glaucoma Goul Hypertension Peptic ulcer Peripheral ncuropathy Peripheral vascular disease Pulmonary hypertension Tricuspid regurgitation Historical No qualifying data Procedure/Surgical History duodenal resection, duodenojcjunostomy. small bowel enterolomy, removal of foreign object and repair oi enterotomy (05/2 1/20 14), colonoscopy (12/10/2013), egd (1209/2013), H/O endoscopy (07/2013), H’O colonoscopy (03/2013), pilonidal cyst removal at base of spine (1981), laser eye surgery ior glaucoma. lesions on small intestine closed up. Home Medications Home allopurinol 300 mg oral tablet, 300 MG= 1 TAB, PO. Daily atenolol 25 mg oral tablet, 25 MG= 1 TAB, PO, Daily chtorthalidone 25 mg oral tablet, 23 MG= 1 TAB, PO, MVE Combigan 0.2%-0.5% ophthalmic solution, 1 DROP, Both Eyes, Q12H Eliquis 5 mg oral lablet, 5 MG= 1 TAB, PO, BID lerrous sulfate 925 mg (65 nig elemental iron) oral tablet, 325 MG= 1 TAB, PO, Daily Lasix 80 mg oral tabic:. 80 MG= | TAB. PO, BID omeprazole 20 mg oral delayed scicasc capsule, 20 MG= 1 CAP, PO, BID Percocei 5/325 oral tablet. | TAB, PO. QAM potassium chloride 20 mEq oral tablet, extended release, 20 MEO= 1 TAB, PO, Daily sertraline 50 mg oral tablet, 75 MG= 1,5 TAB, PQ. Daiiy triamcinolone 0.71% lopical cream, 1 APP, Topical, Daily lriamcmnolone 0.1% lopical ominient, 1 APP. Topical, Daily PowerChart
+1 Patient Name Financial Number Date of Girth Patient Location Random Hospital H & P Anemia Vitamin D2 50,000 intl units (1.25 ma) oral ALBASeRne capsule, 1 TAS, PO, Veexly-Tue Arthritis Allergies Tylenol for pain. Patient also takes Percocet alt home, will add this cn. Chronic kidney disease AY baseline. Monitor while divresing. Hypertension Blood pressures within tolerable ranges. Pulmonary hypertension Tricuspid regurgitation Mild-to-moderaie on echocardiogram last year sholliisn (cout) sulfa drug (maculopapular rash) Social History Ever Smoked tobacco: Former Smoker Alcohol use - frequency; None Drug use: Never Lab Results O7/16/9 7 05:30 to O7/16/17 05:30 Attending physician note-the patient was interviewed and examined. The appropriatc information in power chart was reviewed. The patient was discussed wilh Dr, Persad. 143 1L 981H 26? Patient may have @ mild degree of heart failure. He and his wife were more concernes with ee Ins peripheral edema. He has underlying renal insufficiency as well. We'll try to diurese him to his “dry" weight. We will then try to adjust his medications to kcep him within & narrow range of [hat weight. We will stop his atenolol this point since he is relatively bradycardic anc observe his heart rate on the cardiac monitor. He will progress with his care and aclivily as tolerated. 102 07/16/17 05:30 to O7/ 16/17 05:30 05:30 GLU 102 mg/dL Printed: 7/1 7/2017 13:01 EDT Page 17 of 42 NA K CL TOTAL COZ BUN CRT ANION GAP CA CBC with diff WBC HGB HCT RBC MCV MICH MCHC RDW MPV 143 MMOL/L 3.6 MMOL/L 98 MMOL/L 40 MMOL/L 26 mg/dL. 1.23 mg/dL 5 7.9 mg/dL 07/16/17 05:30 3.4/ nl 10.1 G/DL 32.4 %o 3.41 /PL 95.0 FL 29.6 pg 31.2 % 15,9 %o 10.7 FL PowerChart
+2 Alexandria is the second-largest city in Egypt and a major economic centre, extending about 32 km (20 mi) along the coast of the Mediterranean Sea in the north central part of the country. Its low elevation on the Nile delta makes it highly vulnerable to rising sea levels. Alexandria is an important industrial center because of its natural gas and oil pipelines from Suez. Alexandria is also a popular tourist destination.
+3 Alexandria was founded around a small, ancient Egyptian town c. 332 BC by Alexander the Great,[4] king of Macedon and leader of the Greek League of Corinth, during his conquest of the Achaemenid Empire. Alexandria became an important center of Hellenistic civilization and remained the capital of Ptolemaic Egypt and Roman and Byzantine Egypt for almost 1,000 years, until the Muslim conquest of Egypt in AD 641, when a new capital was founded at Fustat (later absorbed into Cairo). Hellenistic Alexandria was best known for the Lighthouse of Alexandria (Pharos), one of the Seven Wonders of the Ancient World; its Great Library (the largest in the ancient world); and the Necropolis, one of the Seven Wonders of the Middle Ages. Alexandria was at one time the second most powerful city of the ancient Mediterranean region, after Rome. Ongoing maritime archaeology in the harbor of Alexandria, which began in 1994, is revealing details of Alexandria both before the arrival of Alexander, when a city named Rhacotis existed there, and during the Ptolemaic dynasty.
+4 8 i , . ! 9 i , . ! 10 i , . ! 11 i , . ! 12 i , . ! 13 i , . ! 14 i , . !```
diff --git a/docs/_posts/aymanechilah/2023-01-03-table_recognition_en_3_3.md b/docs/_posts/aymanechilah/2023-01-03-table_recognition_en_3_3.md
new file mode 100644
index 00000000000000..7473fc72a6a316
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-03-table_recognition_en_3_3.md
@@ -0,0 +1,199 @@
+---
+layout: model
+title: Table recognition
+author: John Snow Labs
+name: table_recognition
+date: 2023-01-03
+tags: [en, licensed, ocr, table_recognition]
+task: Table Recognition
+language: en
+edition: Visual NLP 4.1.0
+spark_version: 3.3.0
+supported: true
+annotator: TableRecognition
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+This model shows the capabilities for table recognition and free-text extraction using OCR techniques.
+For table recognition is proposed a CascadTabNet model.
+CascadeTabNet is a machine learning model for table detection in document images. It is based on a cascaded architecture, which is a two-stage process where the model first detects candidate regions that may contain tables, and then classifies these regions as tables or non-tables. The model is trained using a dataset of document images, where the tables have been manually annotated.
+
+The benchmark results show that the model is able to detect tables in document images with high accuracy.
+On the ICDAR2013 table competition dataset, CascadeTabNet achieved an F1-score of 0.85, which is considered a good score in this dataset. On the COCO-Text dataset, the model achieved a precision of 0.82 and a recall of 0.79, which are also considered good scores. In addition, the model has been evaluated on the UNLV dataset, where it achieved a precision of 0.87 and a recall of 0.83.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/tutorials/Certification_Trainings/2.2.Spark_OCR_training_Table_recognition.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+
+
+
+## How to use
+
+
+{% include programmingLanguageSelectScalaPythonNLU.html %}
+
+```python
+binary_to_image = BinaryToImage() \
+ .setInputCol("content") \
+ .setOutputCol("image") \
+ .setImageType(ImageType.TYPE_3BYTE_BGR)
+
+# Detect tables on the page using pretrained model
+# It can be finetuned for have more accurate results for more specific documents
+table_detector = ImageTableDetector.pretrained("general_model_table_detection_v2", "en", "clinical/ocr") \
+ .setInputCol("image") \
+ .setOutputCol("region")
+
+# Draw detected region's with table to the page
+draw_regions = ImageDrawRegions() \
+ .setInputCol("image") \
+ .setInputRegionsCol("region") \
+ .setOutputCol("image_with_regions") \
+ .setRectColor(Color.red)
+
+# Extract table regions to separate images
+splitter = ImageSplitRegions() \
+ .setInputCol("image") \
+ .setInputRegionsCol("region") \
+ .setOutputCol("table_image") \
+ .setDropCols("image")
+
+# Detect cells on the table image
+cell_detector = ImageTableCellDetector() \
+ .setInputCol("table_image") \
+ .setOutputCol("cells") \
+ .setAlgoType("morphops") \
+ .setDrawDetectedLines(True)
+
+# Extract text from the detected cells
+table_recognition = ImageCellsToTextTable() \
+ .setInputCol("table_image") \
+ .setCellsCol('cells') \
+ .setMargin(3) \
+ .setStrip(True) \
+ .setOutputCol('table')
+
+# Erase detected table regions
+fill_regions = ImageDrawRegions() \
+ .setInputCol("image") \
+ .setInputRegionsCol("region") \
+ .setOutputCol("image_1") \
+ .setRectColor(Color.white) \
+ .setFilledRect(True)
+
+# OCR
+ocr = ImageToText() \
+ .setInputCol("image_1") \
+ .setOutputCol("text") \
+ .setOcrParams(["preserve_interword_spaces=1", ]) \
+ .setKeepLayout(True) \
+ .setOutputSpaceCharacterWidth(8)
+
+pipeline_table = PipelineModel(stages=[
+ binary_to_image,
+ table_detector,
+ draw_regions,
+ fill_regions,
+ splitter,
+ cell_detector,
+ table_recognition,
+ ocr
+])
+
+imagePath = "/content/cTDaR_t10096.jpg"
+df = spark.read.format("binaryFile").load(imagePath)
+
+tables_results = pipeline_table.transform(df).cache()
+```
+```scala
+val binary_to_image = new BinaryToImage()
+ .setInputCol("content")
+ .setOutputCol("image")
+ .setImageType(ImageType.TYPE_3BYTE_BGR)
+
+# Detect tables on the page using pretrained model
+# It can be finetuned for have more accurate results for more specific documents
+val table_detector = ImageTableDetector
+ .pretrained("general_model_table_detection_v2", "en", "clinical/ocr")
+ .setInputCol("image")
+ .setOutputCol("region")
+
+# Draw detected region's with table to the page
+val draw_regions = new ImageDrawRegions()
+ .setInputCol("image")
+ .setInputRegionsCol("region")
+ .setOutputCol("image_with_regions")
+ .setRectColor(Color.red)
+
+# Extract table regions to separate images
+val splitter = new ImageSplitRegions()
+ .setInputCol("image")
+ .setInputRegionsCol("region")
+ .setOutputCol("table_image")
+ .setDropCols("image")
+
+# Detect cells on the table image
+val cell_detector = new ImageTableCellDetector()
+ .setInputCol("table_image")
+ .setOutputCol("cells")
+ .setAlgoType("morphops")
+ .setDrawDetectedLines(True)
+
+# Extract text from the detected cells
+val table_recognition = new ImageCellsToTextTable()
+ .setInputCol("table_image")
+ .setCellsCol("cells")
+ .setMargin(3)
+ .setStrip(True)
+ .setOutputCol("table")
+
+# Erase detected table regions
+val fill_regions = new ImageDrawRegions()
+ .setInputCol("image")
+ .setInputRegionsCol("region")
+ .setOutputCol("image_1")
+ .setRectColor(Color.white)
+ .setFilledRect(True)
+
+# OCR
+val ocr = new ImageToText()
+ .setInputCol("image_1")
+ .setOutputCol("text")
+ .setOcrParams(Array("preserve_interword_spaces=1", ))
+ .setKeepLayout(True)
+ .setOutputSpaceCharacterWidth(8)
+
+val pipeline_table = new PipelineModel().setStages(Array(
+ binary_to_image,
+ table_detector,
+ draw_regions,
+ fill_regions,
+ splitter,
+ cell_detector,
+ table_recognition,
+ ocr))
+
+val imagePath = "/content/cTDaR_t10096.jpg"
+val df = spark.read.format("binaryFile").load(imagePath)
+
+val tables_results = pipeline_table.transform(df).cache()
+```
+
+
+## Result
+
+
+
+```bash
++--------------------+--------------------+--------------------+--------------------+-------+--------------------+--------------------+--------------------+------+--------------------+--------------------+--------------------+-----------------+---------+--------------------+--------------------+
+| table_image| region| image_with_regions| image_1|pagenum| modificationTime| path| image|length| cells| output_image| table| confidence|exception| text| positions|
++--------------------+--------------------+--------------------+--------------------+-------+--------------------+--------------------+--------------------+------+--------------------+--------------------+--------------------+-----------------+---------+--------------------+--------------------+
+|{file:/content/cT...|{0, 0, 214.0, 437...|{file:/content/cT...|{file:/content/cT...| 0|2022-09-28 18:54:...|file:/content/cTD...|{file:/content/cT...|385071|[[[[0, 0, 408, 32...|{file:/content/cT...|{0, 0, 0.0, 0.0,...|95.43954744611467| null| ...|[{[{ , 0, 170.0, ...|
++--------------------+--------------------+--------------------+--------------------+-------+--------------------+--------------------+--------------------+------+--------------------+--------------------+--------------------+-----------------+---------+--------------------+--------------------+
+```
diff --git a/docs/_posts/aymanechilah/2023-01-10-dit_base_finetuned_rvlcdip_en_3_2.md b/docs/_posts/aymanechilah/2023-01-10-dit_base_finetuned_rvlcdip_en_3_2.md
new file mode 100644
index 00000000000000..41e3c552cd234c
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-10-dit_base_finetuned_rvlcdip_en_3_2.md
@@ -0,0 +1,115 @@
+---
+layout: model
+title: DiT model pretrained on IIT-CDIP and finetuned on RVL-CDIP for document classification
+author: John Snow Labs
+name: dit_base_finetuned_rvlcdip
+date: 2023-01-10
+tags: [en, licensed]
+task: OCR Document Classification
+language: en
+edition: Visual NLP 4.0.0
+spark_version: 3.2.1
+supported: true
+annotator: VisualDocumentClassifierv3
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+DiT was proposed in DiT: Self-supervised Pre-training for Document Image Transformer by Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei. DiT applies the self-supervised objective of BEiT (BERT pre-training of Image Transformers) to 42 million document images. This model was trained for document image classification in the RVL-CDIP dataset (a collection of 400,000 images belonging to one of 16 classes).
+
+The abstract from the paper is the following: Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose DiT, a self-supervised pre-trained Document Image Transformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, as well as table detection. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 → 92.69), document layout analysis (91.0 → 94.9) and table detection (94.23 → 96.55).
+
+
+## Predicted Entities
+
+
+
+{:.btn-box}
+
+[Open in Colab](https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/tutorials/Certification_Trainings/5.2.Visual_Document_Classifier_v3.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/dit_base_finetuned_rvlcdip_en_3.3.0_3.0_1654798502586.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+## How to use
+
+
+
+## Example
+
+### Input:
+
+
+## Output text
+```bash
++-------+
+|label |
++-------+
+|invoice|
++-------+
+```
+
+
+{:.model-param}
+## Model Information
+
+{:.table-model}
+|---|---|
+|Model Name:|dit_base_finetuned_rvlcdip|
+|Type:|ocr|
+|Compatibility:|Visual NLP 4.0.0+|
+|License:|Licensed|
+|Edition:|Official|
+|Language:|en|
+|Size:|319.6 MB|
+
+## References
+
+IIT-CDIP, RVL-CDIP
\ No newline at end of file
diff --git a/docs/_posts/aymanechilah/2023-01-10-general_model_table_detection_v2_en_3_2.md b/docs/_posts/aymanechilah/2023-01-10-general_model_table_detection_v2_en_3_2.md
new file mode 100644
index 00000000000000..509e40aa578d19
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-10-general_model_table_detection_v2_en_3_2.md
@@ -0,0 +1,133 @@
+---
+layout: model
+title: General model for table detection
+author: John Snow Labs
+name: general_model_table_detection_v2
+date: 2023-01-10
+tags: [en, licensed]
+task: OCR Table Detection
+language: en
+edition: Visual NLP 4.1.0
+spark_version: 3.2.1
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+For table detection it is proposed the use of CascadeTabNet. It is a Cascade mask Region-based CNN High-Resolution Network (Cascade mask R-CNN HRNet) based model that detects tables on input images. The model is evaluated on ICDAR 2013, ICDAR 2019 and TableBank public datasets. It achieved 3rd rank in ICDAR 2019 post-competition results for table detection while attaining the best accuracy results for the ICDAR 2013 and TableBank dataset.
+
+Here it is used the CascadeTabNet general model for table detection inspired by https://arxiv.org/abs/2004.12629
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Cards/SparkOcrImageTableDetection.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/general_model_table_detection_v2_en_3.3.0_3.0_1623301511401.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+## How to use
+
+
+
+
+
+## Example
+
+{%- capture input_image -%}
+
+{%- endcapture -%}
+
+{%- capture output_image -%}
+
+{%- endcapture -%}
+
+
+{% include templates/input_output_image.md
+input_image=input_image
+output_image=output_image
+%}
+
+
+{:.model-param}
+## Model Information
+
+{:.table-model}
+|---|---|
+|Model Name:|ocr_table_detection_general_model|
+|Type:|ocr|
+|Compatibility:|Visual NLP 3.0.0+|
+|License:|Licensed|
+|Edition:|Official|
+|Output Labels:|[table regions]|
+|Language:|en|
+
+## Benchmarking
+
+```bash
+3rd rank in ICDAR 2019 post-competition
+1st rank in ICDAR 2013 and TableBank dataset
+```
\ No newline at end of file
diff --git a/docs/_posts/aymanechilah/2023-01-10-image_text_detector_v2_en_3_2.md b/docs/_posts/aymanechilah/2023-01-10-image_text_detector_v2_en_3_2.md
new file mode 100644
index 00000000000000..29381f4e9d6e83
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-10-image_text_detector_v2_en_3_2.md
@@ -0,0 +1,132 @@
+---
+layout: model
+title: Text Detection
+author: John Snow Labs
+name: image_text_detector_v2
+date: 2023-01-10
+tags: [en, licensed]
+task: OCR Text Detection & Recognition
+language: en
+edition: Visual NLP 4.1.0
+spark_version: 3.2.1
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+CRAFT: Character-Region Awareness For Text detection, is designed with a convolutional neural network producing the character region score and affinity score. The region score is used to localize individual characters in the image, and the affinity score is used to group each character into a single instance. To compensate for the lack of character-level annotations, we propose a weaklysupervised learning framework that estimates characterlevel ground truths in existing real word-level datasets.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Cards/SparkOcrImageTextDetection.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/image_text_detector_v2_en_3.3.0_2.4_1643618928538.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+## How to use
+
+
+
+
+## Example
+
+{%- capture input_image -%}
+
+{%- endcapture -%}
+
+{%- capture output_image -%}
+
+{%- endcapture -%}
+
+
+{% include templates/input_output_image.md
+input_image=input_image
+output_image=output_image
+%}
+
+
+{:.model-param}
+## Model Information
+
+{:.table-model}
+|---|---|
+|Model Name:|text_detection_v2|
+|Type:|ocr|
+|Compatibility:|Visual NLP 4.1.0+|
+|License:|Licensed|
+|Edition:|Official|
+|Output Labels:|[text_regions]|
+|Language:|en|
+
diff --git a/docs/_posts/aymanechilah/2023-01-10-ocr_small_handwritten_en_2_4.md b/docs/_posts/aymanechilah/2023-01-10-ocr_small_handwritten_en_2_4.md
new file mode 100644
index 00000000000000..29851422037f2d
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-10-ocr_small_handwritten_en_2_4.md
@@ -0,0 +1,160 @@
+---
+layout: model
+title: Оcr small for handwritten text
+author: John Snow Labs
+name: ocr_small_handwritten
+date: 2023-01-10
+tags: [en, licensed]
+task: OCR Text Detection & Recognition
+language: en
+edition: Visual NLP 3.3.3
+spark_version: 2.4
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Ocr small model for recognise handwritten text based on TrOcr architecture.
+The TrOCR model was proposed in TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei. TrOCR consists of an image Transformer encoder and an autoregressive text Transformer decoder to perform optical character recognition (OCR).
+The abstract from the paper is the following: Text recognition is a long-standing research problem for document digitalization. Existing approaches for text recognition are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on both printed and handwritten text recognition tasks.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Cards/SparkOcrImageToTextHandwritten.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/ocr_small_handwritten_en_3.3.3_2.4_1645080334390.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+## How to use
+
+
+
+
+## Example
+
+{%- capture input_image -%}
+
+{%- endcapture -%}
+
+{%- capture output_image -%}
+
+{%- endcapture -%}
+
+
+{% include templates/input_output_image.md
+input_image=input_image
+output_image=output_image
+%}
+
+## Output text
+
+```bash
+This is an example of handwritten
+beerxt
+Let's # check the performance?
+I hope it will be awesome
+```
+
+
+{:.model-param}
+## Model Information
+
+{:.table-model}
+|---|---|
+|Model Name:|ocr_small_handwritten|
+|Type:|ocr|
+|Compatibility:|Visual NLP 3.3.3+|
+|License:|Licensed|
+|Edition:|Official|
+|Language:|en|
+|Size:|146.7 MB|
\ No newline at end of file
diff --git a/docs/_posts/aymanechilah/2023-01-10-ocr_small_printed_en_2_4.md b/docs/_posts/aymanechilah/2023-01-10-ocr_small_printed_en_2_4.md
new file mode 100644
index 00000000000000..9d96d694af6fd4
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-10-ocr_small_printed_en_2_4.md
@@ -0,0 +1,245 @@
+---
+layout: model
+title: Оcr small for printed text
+author: John Snow Labs
+name: ocr_small_printed
+date: 2023-01-10
+tags: [en, licensed]
+task: OCR Text Detection & Recognition
+language: en
+edition: Visual NLP 3.3.3
+spark_version: 2.4
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Ocr small model for recognise printed text based on TrOcr architecture. The TrOCR model was proposed in TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei. TrOCR consists of an image Transformer encoder and an autoregressive text Transformer decoder to perform optical character recognition (OCR). The abstract from the paper is the following: Text recognition is a long-standing research problem for document digitalization. Existing approaches for text recognition are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on both printed and handwritten text recognition tasks.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-ocr-workshop/blob/master/tutorials/Certification_Trainings/1.3.Trasformer_based_Text_Recognition.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/ocr_small_printed_en_3.3.3_2.4_1645007455031.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+
+## How to use
+
+
+
+
+```
+
+
+
diff --git a/docs/_posts/aymanechilah/2023-01-10-text_cleaner_v1_en_3_2.md b/docs/_posts/aymanechilah/2023-01-10-text_cleaner_v1_en_3_2.md
new file mode 100644
index 00000000000000..b71ad780c539fb
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-10-text_cleaner_v1_en_3_2.md
@@ -0,0 +1,184 @@
+---
+layout: model
+title: Text cleaner v1
+author: John Snow Labs
+name: text_cleaner_v1
+date: 2023-01-10
+tags: [en, licensed]
+task: OCR Text Cleaner
+language: en
+edition: Visual NLP 4.1.0
+spark_version: 3.2.1
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Model for cleaning image with text. It is based on text detection model with extra post-processing.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Cards/SparkOcrImageCleaner.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/text_cleaner_v1_en_3.0.0_2.4_1640088709401.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+
+## How to use
+
+
+## Example
+
+{%- capture input_image -%}
+
+{%- endcapture -%}
+
+{%- capture output_image -%}
+
+{%- endcapture -%}
+
+
+{% include templates/input_output_image.md
+input_image=input_image
+output_image=output_image
+%}
+
+## Output text
+
+```bash
++--------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
+|filename|exploded_entities |
++--------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
+|t01.jpg |{named_entity, 268, 269, OTHERS, {confidence -> 96, width -> 14, x -> 822, y -> 1101, word -> of, token -> of, height -> 34}, []} |
+|t01.jpg |{named_entity, 271, 273, OTHERS, {confidence -> 89, width -> 33, x -> 837, y -> 1112, word -> the, token -> the, height -> 13}, []} |
+|t01.jpg |{named_entity, 275, 277, OTHERS, {confidence -> 89, width -> 30, x -> 874, y -> 1113, word -> Act., token -> act, height -> 12}, []} |
+|t01.jpg |{named_entity, 280, 282, KEY-B, {confidence -> 94, width -> 26, x -> 910, y -> 1113, word -> Yes, token -> yes, height -> 12}, []} |
+|t01.jpg |{named_entity, 284, 285, VALUE-B, {confidence -> 45, width -> 13, x -> 944, y -> 1112, word -> LI, token -> li, height -> 13}, []} |
+|t01.jpg |{named_entity, 287, 288, KEY-B, {confidence -> 83, width -> 22, x -> 963, y -> 1113, word -> No, token -> no, height -> 12}, []} |
+|t01.jpg |{named_entity, 290, 295, HEADER-B, {confidence -> 96, width -> 89, x -> 1493, y -> 13, word -> UNITED, token -> united, height -> 16}, []} |
+|t01.jpg |{named_entity, 297, 302, HEADER-I, {confidence -> 95, width -> 83, x -> 1590, y -> 13, word -> STATES, token -> states, height -> 16}, []} |
+|t01.jpg |{named_entity, 304, 313, HEADER-B, {confidence -> 95, width -> 221, x -> 1186, y -> 45, word -> SECURITIES, token -> securities, height -> 25}, []} |
+|t01.jpg |{named_entity, 315, 317, HEADER-I, {confidence -> 95, width -> 80, x -> 1415, y -> 45, word -> AND, token -> and, height -> 25}, []} |
+|t01.jpg |{named_entity, 319, 326, HEADER-I, {confidence -> 96, width -> 212, x -> 1507, y -> 45, word -> EXCHANGE, token -> exchange, height -> 25}, []} |
+|t01.jpg |{named_entity, 328, 337, HEADER-I, {confidence -> 95, width -> 249, x -> 1732, y -> 45, word -> COMMISSION, token -> commission, height -> 25}, []} |
+|t01.jpg |{named_entity, 339, 348, HEADER-B, {confidence -> 96, width -> 125, x -> 1461, y -> 86, word -> Washington,, token -> washington, height -> 21}, []} |
+|t01.jpg |{named_entity, 351, 351, HEADER-I, {confidence -> 93, width -> 43, x -> 1595, y -> 86, word -> D.C., token -> d, height -> 16}, []} |
+|t01.jpg |{named_entity, 356, 360, HEADER-I, {confidence -> 93, width -> 59, x -> 1646, y -> 86, word -> 20549, token -> 20549, height -> 16}, []} |
+|t01.jpg |{named_entity, 362, 365, HEADER-B, {confidence -> 93, width -> 112, x -> 1484, y -> 159, word -> FORM, token -> form, height -> 25}, []} |
+|t01.jpg |{named_entity, 367, 368, HEADER-I, {confidence -> 91, width -> 77, x -> 1609, y -> 159, word -> 10-K, token -> 10, height -> 25}, []} |
++--------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+{:.model-param}
+## Model Information
+
+{:.table-model}
+|---|---|
+|Model Name:|visualner_keyvalue_10kfilings|
+|Type:|ocr|
+|Compatibility:|Visual NLP 4.0.0+|
+|License:|Licensed|
+|Edition:|Official|
+|Language:|en|
+|Size:|744.3 MB|
+
+## References
+
+Sec 10K filings
\ No newline at end of file
diff --git a/docs/_posts/aymanechilah/2023-01-17-docvqa_donut_base_en_3_2.md b/docs/_posts/aymanechilah/2023-01-17-docvqa_donut_base_en_3_2.md
new file mode 100644
index 00000000000000..024af9e7f1fa79
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-17-docvqa_donut_base_en_3_2.md
@@ -0,0 +1,120 @@
+---
+layout: model
+title: Document Visual Question Answering with DONUT
+author: John Snow Labs
+name: docvqa_donut_base
+date: 2023-01-17
+tags: [en, licensed]
+task: Document Visual Question Answering
+language: en
+edition: Visual NLP 4.3.0
+spark_version: 3.2.1
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+
+## Description
+
+Document understanding transformer (Donut) model pretrained for Document Visual Question Answering (DocVQA) task in the dataset is from Document Visual Question Answering [competition](https://rrc.cvc.uab.es/?ch=17) and consists of 50K questions defined on more than 12K documents.
+Donut is a new method of document understanding that utilizes an OCR-free end-to-end Transformer model. Donut does not require off-the-shelf OCR engines/APIs, yet it shows state-of-the-art performances on various visual document understanding tasks, such as visual document classification or information extraction (a.k.a. document parsing). Paper link [OCR-free Document Understanding Transformer](https://arxiv.org/abs/2111.15664) developed by Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han and Seunghyun Park.
+DocVQA seeks to inspire a “purpose-driven” point of view in Document Analysis and Recognition research, where the document content is extracted and used to respond to high-level tasks defined by the human consumers of this information.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Cards/SparkOcrVisualQuestionAnswering.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/docvqa_donut_base_en_4.3.0_3.0_1673269990044.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+
+## How to use
+
+
+{% include programmingLanguageSelectScalaPythonNLU.html %}
+
+```python
+binary_to_image = BinaryToImage()\
+ .setInputCol("content") \
+ .setOutputCol("image") \
+ .setImageType(ImageType.TYPE_3BYTE_BGR)
+
+visual_question_answering = VisualQuestionAnswering()\
+ .pretrained("docvqa_donut_base", "en", "clinical/ocr")\
+ .setInputCol(["image"])\
+ .setOutputCol("answers")\
+ .setQuestionsCol("questions")
+
+# OCR pipeline
+pipeline = PipelineModel(stages=[
+ binary_to_image,
+ visual_question_answering
+])
+
+test_image_path = pkg_resources.resource_filename('sparkocr', 'resources/ocr/vqa/agenda.png')
+bin_df = spark.read.format("binaryFile").load(test_image_path)
+
+questions = [["When it finish the Coffee Break?", "Who is giving the Introductory Remarks?", "Who is going to take part of the individual interviews?"]]
+questions_df = spark.createDataFrame([questions])
+questions_df = questions_df.withColumnRenamed("_1", "questions")
+image_and_questions = bin_df.join(questions_df)
+
+results = pipeline.transform(image_and_questions).cache()
+results.select(results.answers).show(truncate=False)
+```
+```scala
+val binary_to_image = new BinaryToImage()
+ .setInputCol("content")
+ .setOutputCol("image")
+ .setImageType(ImageType.TYPE_3BYTE_BGR)
+
+val visual_question_answering = VisualQuestionAnswering()
+ .pretrained("docvqa_donut_base", "en", "clinical/ocr")
+ .setInputCol(Array("image"))
+ .setOutputCol("answers")
+ .setQuestionsCol("questions")
+
+# OCR pipeline
+val pipeline = new PipelineModel().setStages(Array(
+ binary_to_image,
+ visual_question_answering))
+
+val test_image_path = pkg_resources.resource_filename("sparkocr", "resources/ocr/vqa/agenda.png")
+val bin_df = spark.read.format("binaryFile").load(test_image_path)
+
+val questions = Array("When it finish the Coffee Break?", "Who is giving the Introductory Remarks?", "Who is going to take part of the individual interviews?")
+val questions_df = spark.createDataFrame(Array(questions))
+val questions_df = questions_df.withColumnRenamed("_1", "questions")
+val image_and_questions = bin_df.join(questions_df)
+
+val results = pipeline.transform(image_and_questions).cache()
+results.select(results.answers).show(truncate=False)
+```
+
+
+## Example
+
+### Input:
+```bash
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|questions |
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|[ When it finish the Coffee Break?, Who is giving the Introductory Remarks?, Who is going to take part of the individual interviews?
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+
+
+### Output:
+```bash
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|answers |
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|[ When it finish the Coffee Break? -> 11:44 to 11:39 a.m., Who is giving the Introductory Remarks? -> lee a. waller, trrf vice presi- dent, Who is going to take part of the individual interviews? -> trrf treasurer]|
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+
+
diff --git a/docs/_posts/aymanechilah/2023-01-17-docvqa_donut_base_opt_en_3_2.md b/docs/_posts/aymanechilah/2023-01-17-docvqa_donut_base_opt_en_3_2.md
new file mode 100644
index 00000000000000..a1f307bebd0fd8
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-17-docvqa_donut_base_opt_en_3_2.md
@@ -0,0 +1,118 @@
+---
+layout: model
+title: Document Visual Question Answering optimized with DONUT
+author: John Snow Labs
+name: docvqa_donut_base_opt
+date: 2023-01-17
+tags: [en, licensed]
+task: Document Visual Question Answering
+language: en
+edition: Visual NLP 4.3.0
+spark_version: 3.2.1
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Document understanding transformer (Donut) model pretrained for Document Visual Question Answering (DocVQA) task in the dataset is from Document Visual Question Answering [competition](https://rrc.cvc.uab.es/?ch=17) and consists of 50K questions defined on more than 12K documents. This model was optimized with ONNX.
+Donut is a new method of document understanding that utilizes an OCR-free end-to-end Transformer model. Donut does not require off-the-shelf OCR engines/APIs, yet it shows state-of-the-art performances on various visual document understanding tasks, such as visual document classification or information extraction (a.k.a. document parsing). Paper link [OCR-free Document Understanding Transformer](https://arxiv.org/abs/2111.15664) developed by Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han and Seunghyun Park.
+DocVQA seeks to inspire a “purpose-driven” point of view in Document Analysis and Recognition research, where the document content is extracted and used to respond to high-level tasks defined by the human consumers of this information.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Cards/SparkOcrVisualQuestionAnswering_opt.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/docvqa_donut_base_opt_en_4.3.0_3.0_1673269990047.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+
+## How to use
+
+
+{% include programmingLanguageSelectScalaPythonNLU.html %}
+
+```python
+binary_to_image = BinaryToImage()\
+ .setInputCol("content") \
+ .setOutputCol("image") \
+ .setImageType(ImageType.TYPE_3BYTE_BGR)
+
+visual_question_answering = VisualQuestionAnswering()\
+ .pretrained("docvqa_donut_base_opt", "en", "clinical/ocr")\
+ .setInputCol(["image"])\
+ .setOutputCol("answers")\
+ .setQuestionsCol("questions")
+
+# OCR pipeline
+pipeline = PipelineModel(stages=[
+ binary_to_image,
+ visual_question_answering
+])
+
+test_image_path = pkg_resources.resource_filename('sparkocr', 'resources/ocr/vqa/agenda.png')
+bin_df = spark.read.format("binaryFile").load(test_image_path)
+
+questions = [["When it finish the Coffee Break?", "Who is giving the Introductory Remarks?", "Who is going to take part of the individual interviews?"]]
+questions_df = spark.createDataFrame([questions])
+questions_df = questions_df.withColumnRenamed("_1", "questions")
+image_and_questions = bin_df.join(questions_df)
+
+results = pipeline.transform(image_and_questions).cache()
+results.select(results.answers).show(truncate=False)
+```
+```scala
+val binary_to_image = new BinaryToImage()
+ .setInputCol("content")
+ .setOutputCol("image")
+ .setImageType(ImageType.TYPE_3BYTE_BGR)
+
+val visual_question_answering = VisualQuestionAnswering()
+ .pretrained("docvqa_donut_base_opt", "en", "clinical/ocr")
+ .setInputCol(Array("image"))
+ .setOutputCol("answers")
+ .setQuestionsCol("questions")
+
+# OCR pipeline
+val pipeline = new PipelineModel().setStages(Array(
+ binary_to_image,
+ visual_question_answering))
+
+val test_image_path = pkg_resources.resource_filename("sparkocr", "resources/ocr/vqa/agenda.png")
+val bin_df = spark.read.format("binaryFile").load(test_image_path)
+
+val questions = Array("When it finish the Coffee Break?", "Who is giving the Introductory Remarks?", "Who is going to take part of the individual interviews?")
+val questions_df = spark.createDataFrame(Array(questions))
+val questions_df = questions_df.withColumnRenamed("_1", "questions")
+val image_and_questions = bin_df.join(questions_df)
+
+val results = pipeline.transform(image_and_questions).cache()
+val results.select(results.answers).show(truncate=False)
+```
+
+
+## Example
+
+### Input:
+```bash
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|questions |
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|[ When it finish the Coffee Break?, Who is giving the Introductory Remarks?, Who is going to take part of the individual interviews?
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+
+### Output:
+```bash
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|answers |
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|[ When it finish the Coffee Break? -> 11:39 a.m., Who is giving the Introductory Remarks? -> lee a. waller, trrf vice presi- ident, Who is going to take part of the individual interviews? -> trrf]|
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+
+
diff --git a/docs/_posts/aymanechilah/2023-01-17-ocr_base_handwritten_v2_en_3_2.md b/docs/_posts/aymanechilah/2023-01-17-ocr_base_handwritten_v2_en_3_2.md
new file mode 100644
index 00000000000000..a956c3906823ec
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-17-ocr_base_handwritten_v2_en_3_2.md
@@ -0,0 +1,146 @@
+---
+layout: model
+title: Оcr base v2 for handwritten text
+author: John Snow Labs
+name: ocr_base_handwritten_v2
+date: 2023-01-17
+tags: [en, licensed]
+task: OCR Text Detection & Recognition
+language: en
+edition: Visual NLP 4.2.4
+spark_version: 3.2.1
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Ocr base handwritten model v2 for recognise handwritten text based on TrOCR model pretrained on handwritten datasets. It is an Ocr base model for recognise handwritten text based on TrOcr architecture. The TrOCR model was proposed in TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei. TrOCR consists of an image Transformer encoder and an autoregressive text Transformer decoder to perform optical character recognition (OCR). The abstract from the paper is the following: Text recognition is a long-standing research problem for document digitalization. Existing approaches for text recognition are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on both printed and handwritten text recognition tasks.
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Cards/SparkOcrImageToTextHandwritten_V2.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/ocr_base_handwritten_v2_en_4.2.2_3.0_1670602309000.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+
+## How to use
+
+
+
+## Example
+
+{%- capture input_image -%}
+
+{%- endcapture -%}
+
+{%- capture output_image -%}
+
+{%- endcapture -%}
+
+
+{% include templates/input_output_image.md
+input_image=input_image
+output_image=output_image
+%}
+
+## Output text
+```bash
+This is an example of handwritten
+beerxt
+Let's # check the performance !
+I hope it will be awesome
+```
+
+
+
diff --git a/docs/_posts/aymanechilah/2023-01-17-ocr_base_handwritten_v2_opt_en_3_2.md b/docs/_posts/aymanechilah/2023-01-17-ocr_base_handwritten_v2_opt_en_3_2.md
new file mode 100644
index 00000000000000..8f42471b9826d9
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-17-ocr_base_handwritten_v2_opt_en_3_2.md
@@ -0,0 +1,148 @@
+---
+layout: model
+title: Оcr base v2 optimized for handwritten text
+author: John Snow Labs
+name: ocr_base_handwritten_v2_opt
+date: 2023-01-17
+tags: [en, licensed]
+task: OCR Text Detection & Recognition
+language: en
+edition: Visual NLP 4.2.4
+spark_version: 3.2.1
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Ocr base handwritten model v2 optimized with ONNX for recognise handwritten text based on a TrOCR model pretrained with handwritten datasets. It is an Ocr base model for recognise handwritten text based on TrOcr architecture. The TrOCR model was proposed in TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei. TrOCR consists of an image Transformer encoder and an autoregressive text Transformer decoder to perform optical character recognition (OCR). The abstract from the paper is the following: Text recognition is a long-standing research problem for document digitalization. Existing approaches for text recognition are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on both printed and handwritten text recognition tasks.
+
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Cards/SparkOcrImageToTextHandwritten_V2_opt.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/ocr_base_handwritten_v2_opt_en_4.2.2_3.0_1670608549000.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+
+## How to use
+
+
+
+## Example
+
+{%- capture input_image -%}
+
+{%- endcapture -%}
+
+{%- capture output_image -%}
+
+{%- endcapture -%}
+
+
+{% include templates/input_output_image.md
+input_image=input_image
+output_image=output_image
+%}
+
+## Output text
+
+```bash
+This is an example of handwritten
+beerxt
+Let's # check the performance !
+I hope it will be awesome
+```
+
+
+
diff --git a/docs/_posts/aymanechilah/2023-01-17-ocr_base_printed_v2_en_3_2.md b/docs/_posts/aymanechilah/2023-01-17-ocr_base_printed_v2_en_3_2.md
new file mode 100644
index 00000000000000..8b67638bcc6d85
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-17-ocr_base_printed_v2_en_3_2.md
@@ -0,0 +1,155 @@
+---
+layout: model
+title: Оcr base v2 for printed text
+author: John Snow Labs
+name: ocr_base_printed_v2
+date: 2023-01-17
+tags: [en, licensed]
+task: OCR Text Detection & Recognition
+language: en
+edition: Visual NLP 4.2.4
+spark_version: 3.2.1
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Ocr base printed model v2 for recognise printed text based on a TrOCR model pretrained with printed datasets. It is an Ocr base model for recognise handwritten text based on TrOcr architecture. The TrOCR model was proposed in TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei. TrOCR consists of an image Transformer encoder and an autoregressive text Transformer decoder to perform optical character recognition (OCR). The abstract from the paper is the following: Text recognition is a long-standing research problem for document digitalization. Existing approaches for text recognition are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on both printed and handwritten text recognition tasks.
+
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Cards/SparkOcrImageToTextPrinted_V2.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/ocr_base_printed_v2_en_4.2.2_3.0_1670623909000.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+
+## How to use
+
+
+
+## Example
+
+{%- capture input_image -%}
+
+{%- endcapture -%}
+
+{%- capture output_image -%}
+
+{%- endcapture -%}
+
+
+{% include templates/input_output_image.md
+input_image=input_image
+output_image=output_image
+%}
+
+## Output text
+
+```bash
+STARBUCKS STORE #10208
+11302 EUCLID AVENUE
+CLEVELAND, OH (216) 229-0749
+CHK 664290
+12/07/2014 06:43 PM
+1912003- DRAWER: 2. REG: 2
+VT PEP MOCHA
+SBUX CARD : 4.95
+XXXXXXXXXXXX3228
+SUBTOTAL: $4.95
+TOTAL @ 6.95
+CHANGE DUE $0.00
+................ CCHECK CLOSED
+12/07/2014 06:43 PM
+SBUX CARD X3228 NEW BALANCE: 37.45
+CARD IS REGISTERED
+```
+
+
+
diff --git a/docs/_posts/aymanechilah/2023-01-17-ocr_base_printed_v2_opt_en_3_2.md b/docs/_posts/aymanechilah/2023-01-17-ocr_base_printed_v2_opt_en_3_2.md
new file mode 100644
index 00000000000000..7cac95870d559b
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-17-ocr_base_printed_v2_opt_en_3_2.md
@@ -0,0 +1,155 @@
+---
+layout: model
+title: Оcr base v2 optimized for printed text
+author: John Snow Labs
+name: ocr_base_printed_v2_opt
+date: 2023-01-17
+tags: [en, licensed]
+task: OCR Text Detection & Recognition
+language: en
+edition: Visual NLP 4.2.4
+spark_version: 3.2.1
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Ocr base printed model v2 optimized with ONNX for recognise handwritten text based on a TrOCR pretrained model with printed datasets. It is an Ocr base model for recognise handwritten text based on TrOcr architecture. The TrOCR model was proposed in TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei. TrOCR consists of an image Transformer encoder and an autoregressive text Transformer decoder to perform optical character recognition (OCR). The abstract from the paper is the following: Text recognition is a long-standing research problem for document digitalization. Existing approaches for text recognition are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on both printed and handwritten text recognition tasks.
+
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Cards/SparkOcrImageToTextPrinted_V2_opt.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/ocr_base_printed_v2_opt_en_4.2.2_3.0_1670605909000.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+
+
+## How to use
+
+
+## Example
+
+{%- capture input_image -%}
+
+{%- endcapture -%}
+
+{%- capture output_image -%}
+
+{%- endcapture -%}
+
+
+{% include templates/input_output_image.md
+input_image=input_image
+output_image=output_image
+%}
+
+## Output text
+
+```bash
+STARBUCKS STORE #10208
+11302 EUCLID AVENUE
+CLEVELAND, OH (216) 229-0749
+CHK 664290
+12/07/2014 06:43 PM
+1912003- DRAWER: 2. REG: 2
+VT PEP MOCHA
+SBUX CARD : 4.95
+XXXXXXXXXXXX3228
+SUBTOTAL: $4.95
+TOTAL @ 6 @ < $4.95
+CHANGE DUE $0.00
+................ CCHECK CLOSED
+12/07/2014 06:43 PM
+SBUX CARD X3228 NEW BALANCE: 37.45
+CARD IS REGISTERED
+```
+
+
+
diff --git a/docs/_posts/aymanechilah/2023-01-23-hocr_table_recognition_en_3_2.md b/docs/_posts/aymanechilah/2023-01-23-hocr_table_recognition_en_3_2.md
new file mode 100644
index 00000000000000..1c46a56263e562
--- /dev/null
+++ b/docs/_posts/aymanechilah/2023-01-23-hocr_table_recognition_en_3_2.md
@@ -0,0 +1,211 @@
+---
+layout: model
+title: Hocr for table recognition
+author: John Snow Labs
+name: hocr_table_recognition
+date: 2023-01-23
+tags: [en, licensed]
+task: HOCR Table Recognition
+language: en
+edition: Visual NLP 4.2.4
+spark_version: 3.2.1
+supported: true
+article_header:
+ type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Table structure recognition based on hocr with Tesseract architecture.
+
+Tesseract has been trained on a variety of datasets to improve its recognition capabilities. These datasets include images of text in various languages and scripts, as well as images with different font styles, sizes, and orientations. The training process involves feeding the engine with a large number of images and their corresponding text, allowing the engine to learn the patterns and characteristics of different text styles. One of the most important datasets used in training Tesseract is the UNLV dataset, which contains over 400,000 images of text in different languages, scripts, and font styles. This dataset is widely used in the OCR community and has been instrumental in improving the accuracy of Tesseract. Other datasets that have been used in training Tesseract include the ICDAR dataset, the IIIT-HWS dataset, and the RRC-GV-WS dataset.
+
+In addition to these datasets, Tesseract also uses a technique called adaptive training, where the engine can continuously improve its recognition capabilities by learning from new images and text. This allows Tesseract to adapt to new text styles and languages, and improve its overall accuracy.
+
+
+## Predicted Entities
+
+{:.btn-box}
+
+[Open in Colab](https://github.com/JohnSnowLabs/spark-ocr-workshop/tree/master/jupyter/Cards/SparkOcrImageTableRecognitionWHOCR.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
+
+
+
+## How to use
+
+
Live Demo
-
-[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/bert_qa_bert_large_question_answering_finetuned_legal_en_4.0.0_3.0_1654536412110.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
-[Copy S3 URI](s3://auxdata.johnsnowlabs.com/public/models/bert_qa_bert_large_question_answering_finetuned_legal_en_4.0.0_3.0_1654536412110.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
-
-## How to use
-
-
-
-
-{% include programmingLanguageSelectScalaPythonNLU.html %}
-```python
-document_assembler = MultiDocumentAssembler() \
-.setInputCols(["question", "context"]) \
-.setOutputCols(["document_question", "document_context"])
-
-spanClassifier = BertForQuestionAnswering.pretrained("bert_qa_bert_large_question_answering_finetuned_legal","en") \
-.setInputCols(["document_question", "document_context"]) \
-.setOutputCol("answer") \
-.setCaseSensitive(True)
-
-pipeline = Pipeline().setStages([
-document_assembler,
-spanClassifier
-])
-
-example = spark.createDataFrame([["What's my name?", "My name is Clara and I live in Berkeley."]]).toDF("question", "context")
-
-result = pipeline.fit(example).transform(example)
-```
-```scala
-val document = new MultiDocumentAssembler()
-.setInputCols("question", "context")
-.setOutputCols("document_question", "document_context")
-
-val spanClassifier = BertForQuestionAnswering
-.pretrained("bert_qa_bert_large_question_answering_finetuned_legal","en")
-.setInputCols(Array("document_question", "document_context"))
-.setOutputCol("answer")
-.setCaseSensitive(true)
-.setMaxSentenceLength(512)
-
-val pipeline = new Pipeline().setStages(Array(document, spanClassifier))
-
-val example = Seq(
-("Where was John Lenon born?", "John Lenon was born in London and lived in Paris. My name is Sarah and I live in London."),
-("What's my name?", "My name is Clara and I live in Berkeley."))
-.toDF("question", "context")
-
-val result = pipeline.fit(example).transform(example)
-```
-
-
-{:.nlu-block}
-```python
-import nlu
-nlu.load("en.answer_question.bert.large").predict("""What's my name?|||"My name is Clara and I live in Berkeley.""")
-```
-
-
-
-{:.model-param}
-## Model Information
-
-{:.table-model}
-|---|---|
-|Model Name:|bert_qa_bert_large_question_answering_finetuned_legal|
-|Compatibility:|Spark NLP 4.0.0+|
-|License:|Open Source|
-|Edition:|Official|
-|Input Labels:|[sentence, token]|
-|Output Labels:|[embeddings]|
-|Language:|en|
-|Size:|1.3 GB|
-|Case sensitive:|true|
-|Max sentence length:|512|
-
-## References
-
-- https://huggingface.co/atharvamundada99/bert-large-question-answering-finetuned-legal
\ No newline at end of file
diff --git a/docs/_posts/gadde5300/2022-06-20-roberta_qa_legal_qa_en_3_0.md b/docs/_posts/gadde5300/2022-06-20-roberta_qa_legal_qa_en_3_0.md
deleted file mode 100644
index 4e405052345464..00000000000000
--- a/docs/_posts/gadde5300/2022-06-20-roberta_qa_legal_qa_en_3_0.md
+++ /dev/null
@@ -1,103 +0,0 @@
----
-layout: model
-title: English RobertaForQuestionAnswering (from Beri)
-author: John Snow Labs
-name: roberta_qa_legal_qa
-date: 2022-06-20
-tags: [en, open_source, question_answering, roberta]
-task: Question Answering
-language: en
-edition: Spark NLP 4.0.0
-spark_version: 3.0
-supported: true
-annotator: RoBertaForQuestionAnswering
-article_header:
-type: cover
-use_language_switcher: "Python-Scala-Java"
----
-
-## Description
-
-Pretrained Question Answering model, adapted from Hugging Face and curated to provide scalability and production-readiness using Spark NLP. `legal-qa` is a English model originally trained by `Beri`.
-
-{:.btn-box}
-
-
-[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/roberta_qa_legal_qa_en_4.0.0_3.0_1655728938245.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
-[Copy S3 URI](s3://auxdata.johnsnowlabs.com/public/models/roberta_qa_legal_qa_en_4.0.0_3.0_1655728938245.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
-
-## How to use
-
-
-
-
-{% include programmingLanguageSelectScalaPythonNLU.html %}
-```python
-document_assembler = MultiDocumentAssembler() \
-.setInputCols(["question", "context"]) \
-.setOutputCols(["document_question", "document_context"])
-
-spanClassifier = RoBertaForQuestionAnswering.pretrained("roberta_qa_legal_qa","en") \
-.setInputCols(["document_question", "document_context"]) \
-.setOutputCol("answer") \
-.setCaseSensitive(True)
-
-pipeline = Pipeline().setStages([
-document_assembler,
-spanClassifier
-])
-
-example = spark.createDataFrame([["What's my name?", "My name is Clara and I live in Berkeley."]]).toDF("question", "context")
-
-result = pipeline.fit(example).transform(example)
-```
-```scala
-val document = new MultiDocumentAssembler()
-.setInputCols("question", "context")
-.setOutputCols("document_question", "document_context")
-
-val spanClassifier = RoBertaForQuestionAnswering
-.pretrained("roberta_qa_legal_qa","en")
-.setInputCols(Array("document_question", "document_context"))
-.setOutputCol("answer")
-.setCaseSensitive(true)
-.setMaxSentenceLength(512)
-
-val pipeline = new Pipeline().setStages(Array(document, spanClassifier))
-
-val example = Seq(
-("Where was John Lenon born?", "John Lenon was born in London and lived in Paris. My name is Sarah and I live in London."),
-("What's my name?", "My name is Clara and I live in Berkeley."))
-.toDF("question", "context")
-
-val result = pipeline.fit(example).transform(example)
-```
-
-
-{:.nlu-block}
-```python
-import nlu
-nlu.load("en.answer_question.roberta.by_Beri").predict("""What's my name?|||"My name is Clara and I live in Berkeley.""")
-```
-
-
-
-{:.model-param}
-## Model Information
-
-{:.table-model}
-|---|---|
-|Model Name:|roberta_qa_legal_qa|
-|Compatibility:|Spark NLP 4.0.0+|
-|License:|Open Source|
-|Edition:|Official|
-|Input Labels:|[question, context]|
-|Output Labels:|[answer]|
-|Language:|en|
-|Size:|447.9 MB|
-|Case sensitive:|true|
-|Max sentence length:|512|
-
-## References
-
-- https://huggingface.co/Beri/legal-qa
\ No newline at end of file
diff --git a/docs/_posts/gadde5300/2022-06-20-roberta_qa_marshmellow77_roberta_base_cuad_en_3_0.md b/docs/_posts/gadde5300/2022-06-20-roberta_qa_marshmellow77_roberta_base_cuad_en_3_0.md
deleted file mode 100644
index 4068e76ff822f1..00000000000000
--- a/docs/_posts/gadde5300/2022-06-20-roberta_qa_marshmellow77_roberta_base_cuad_en_3_0.md
+++ /dev/null
@@ -1,107 +0,0 @@
----
-layout: model
-title: English RobertaForQuestionAnswering (from marshmellow77)
-author: John Snow Labs
-name: roberta_qa_marshmellow77_roberta_base_cuad
-date: 2022-06-20
-tags: [en, open_source, question_answering, roberta]
-task: Question Answering
-language: en
-edition: Spark NLP 4.0.0
-spark_version: 3.0
-supported: true
-annotator: RoBertaForQuestionAnswering
-article_header:
-type: cover
-use_language_switcher: "Python-Scala-Java"
----
-
-## Description
-
-Pretrained Question Answering model, adapted from Hugging Face and curated to provide scalability and production-readiness using Spark NLP. `roberta-base-cuad` is a English model originally trained by `marshmellow77`.
-
-{:.btn-box}
-
-
-[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/roberta_qa_marshmellow77_roberta_base_cuad_en_4.0.0_3.0_1655730484239.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
-[Copy S3 URI](s3://auxdata.johnsnowlabs.com/public/models/roberta_qa_marshmellow77_roberta_base_cuad_en_4.0.0_3.0_1655730484239.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
-
-## How to use
-
-
-
-
-{% include programmingLanguageSelectScalaPythonNLU.html %}
-```python
-document_assembler = MultiDocumentAssembler() \
-.setInputCols(["question", "context"]) \
-.setOutputCols(["document_question", "document_context"])
-
-spanClassifier = RoBertaForQuestionAnswering.pretrained("roberta_qa_marshmellow77_roberta_base_cuad","en") \
-.setInputCols(["document_question", "document_context"]) \
-.setOutputCol("answer") \
-.setCaseSensitive(True)
-
-pipeline = Pipeline().setStages([
-document_assembler,
-spanClassifier
-])
-
-example = spark.createDataFrame([["What's my name?", "My name is Clara and I live in Berkeley."]]).toDF("question", "context")
-
-result = pipeline.fit(example).transform(example)
-```
-```scala
-val document = new MultiDocumentAssembler()
-.setInputCols("question", "context")
-.setOutputCols("document_question", "document_context")
-
-val spanClassifier = RoBertaForQuestionAnswering
-.pretrained("roberta_qa_marshmellow77_roberta_base_cuad","en")
-.setInputCols(Array("document_question", "document_context"))
-.setOutputCol("answer")
-.setCaseSensitive(true)
-.setMaxSentenceLength(512)
-
-val pipeline = new Pipeline().setStages(Array(document, spanClassifier))
-
-val example = Seq(
-("Where was John Lenon born?", "John Lenon was born in London and lived in Paris. My name is Sarah and I live in London."),
-("What's my name?", "My name is Clara and I live in Berkeley."))
-.toDF("question", "context")
-
-val result = pipeline.fit(example).transform(example)
-```
-
-
-{:.nlu-block}
-```python
-import nlu
-nlu.load("en.answer_question.cuad.roberta.base").predict("""What's my name?|||"My name is Clara and I live in Berkeley.""")
-```
-
-
-
-{:.model-param}
-## Model Information
-
-{:.table-model}
-|---|---|
-|Model Name:|roberta_qa_marshmellow77_roberta_base_cuad|
-|Compatibility:|Spark NLP 4.0.0+|
-|License:|Open Source|
-|Edition:|Official|
-|Input Labels:|[question, context]|
-|Output Labels:|[answer]|
-|Language:|en|
-|Size:|447.9 MB|
-|Case sensitive:|true|
-|Max sentence length:|512|
-
-## References
-
-- https://huggingface.co/marshmellow77/roberta-base-cuad
-- https://towardsdatascience.com/how-to-set-up-a-machine-learning-model-for-legal-contract-review-fe3b48b05a0e
-- https://github.com/TheAtticusProject/cuad
-- https://towardsdatascience.com/how-to-set-up-a-machine-learning-model-for-legal-contract-review-part-2-6ecbbe680ba
-- https://github.com/marshmellow77/cuad-demo
\ No newline at end of file
diff --git a/docs/_posts/gadde5300/2022-06-20-roberta_qa_roberta_base_on_cuad_en_3_0.md b/docs/_posts/gadde5300/2022-06-20-roberta_qa_roberta_base_on_cuad_en_3_0.md
deleted file mode 100644
index e0e40b4f428a48..00000000000000
--- a/docs/_posts/gadde5300/2022-06-20-roberta_qa_roberta_base_on_cuad_en_3_0.md
+++ /dev/null
@@ -1,103 +0,0 @@
----
-layout: model
-title: English RobertaForQuestionAnswering (from Rakib)
-author: John Snow Labs
-name: roberta_qa_roberta_base_on_cuad
-date: 2022-06-20
-tags: [en, open_source, question_answering, roberta]
-task: Question Answering
-language: en
-edition: Spark NLP 4.0.0
-spark_version: 3.0
-supported: true
-annotator: RoBertaForQuestionAnswering
-article_header:
-type: cover
-use_language_switcher: "Python-Scala-Java"
----
-
-## Description
-
-Pretrained Question Answering model, adapted from Hugging Face and curated to provide scalability and production-readiness using Spark NLP. `roberta-base-on-cuad` is a English model originally trained by `Rakib`.
-
-{:.btn-box}
-
-
-[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/roberta_qa_roberta_base_on_cuad_en_4.0.0_3.0_1655734616590.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
-[Copy S3 URI](s3://auxdata.johnsnowlabs.com/public/models/roberta_qa_roberta_base_on_cuad_en_4.0.0_3.0_1655734616590.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
-
-## How to use
-
-
-
-
-{% include programmingLanguageSelectScalaPythonNLU.html %}
-```python
-document_assembler = MultiDocumentAssembler() \
-.setInputCols(["question", "context"]) \
-.setOutputCols(["document_question", "document_context"])
-
-spanClassifier = RoBertaForQuestionAnswering.pretrained("roberta_qa_roberta_base_on_cuad","en") \
-.setInputCols(["document_question", "document_context"]) \
-.setOutputCol("answer") \
-.setCaseSensitive(True)
-
-pipeline = Pipeline().setStages([
-document_assembler,
-spanClassifier
-])
-
-example = spark.createDataFrame([["What's my name?", "My name is Clara and I live in Berkeley."]]).toDF("question", "context")
-
-result = pipeline.fit(example).transform(example)
-```
-```scala
-val document = new MultiDocumentAssembler()
-.setInputCols("question", "context")
-.setOutputCols("document_question", "document_context")
-
-val spanClassifier = RoBertaForQuestionAnswering
-.pretrained("roberta_qa_roberta_base_on_cuad","en")
-.setInputCols(Array("document_question", "document_context"))
-.setOutputCol("answer")
-.setCaseSensitive(true)
-.setMaxSentenceLength(512)
-
-val pipeline = new Pipeline().setStages(Array(document, spanClassifier))
-
-val example = Seq(
-("Where was John Lenon born?", "John Lenon was born in London and lived in Paris. My name is Sarah and I live in London."),
-("What's my name?", "My name is Clara and I live in Berkeley."))
-.toDF("question", "context")
-
-val result = pipeline.fit(example).transform(example)
-```
-
-
-{:.nlu-block}
-```python
-import nlu
-nlu.load("en.answer_question.cuad.roberta.base.by_Rakib").predict("""What's my name?|||"My name is Clara and I live in Berkeley.""")
-```
-
-
-
-{:.model-param}
-## Model Information
-
-{:.table-model}
-|---|---|
-|Model Name:|layoutlmv3_finetuned_funsd|
-|Type:|ocr|
-|Compatibility:|Visual NLP 3.14.0+|
-|License:|Licensed|
-|Edition:|Official|
-|Language:|en|
-|Size:|467.0 MB|
\ No newline at end of file
diff --git a/docs/_posts/muhammetsnts/2021-07-24-redl_drug_drug_interaction_biobert_en.md b/docs/_posts/muhammetsnts/2021-07-24-redl_drug_drug_interaction_biobert_en.md
index 5c6fda8a00800a..78f1192ed6cde4 100644
--- a/docs/_posts/muhammetsnts/2021-07-24-redl_drug_drug_interaction_biobert_en.md
+++ b/docs/_posts/muhammetsnts/2021-07-24-redl_drug_drug_interaction_biobert_en.md
@@ -185,9 +185,11 @@ they should be used with caution because the pharmacologically predictable sympa
```bash
-| | relation | entity1 | entity1_begin | entity1_end | chunk1 | entity2 | entity2_begin | entity2_end | chunk2 | confidence |
-|---:|:-----------|:----------|----------------:|--------------:|:--------------|:----------|----------------:|--------------:|:-------------|-------------:|
-| 0 | DDI-advise | DRUG | 5 | 17 | carbamazepine | DRUG | 62 | 73 | aripiprazole | 0.99238 |
++---------+-------+-------------+-----------+-------------+-------+-------------+-----------+------------+----------+
+| relation|entity1|entity1_begin|entity1_end| chunk1|entity2|entity2_begin|entity2_end| chunk2|confidence|
++---------+-------+-------------+-----------+-------------+-------+-------------+-----------+------------+----------+
+|DDI-false| DRUG| 5| 17|carbamazepine| DRUG| 62| 73|aripiprazole|0.91685396|
++---------+-------+-------------+-----------+-------------+-------+-------------+-----------+------------+----------+
```
diff --git a/docs/_posts/muhammetsnts/2021-07-24-redl_temporal_events_biobert_en.md b/docs/_posts/muhammetsnts/2021-07-24-redl_temporal_events_biobert_en.md
index 5ab8dd295d67c6..f658d56bc44b98 100644
--- a/docs/_posts/muhammetsnts/2021-07-24-redl_temporal_events_biobert_en.md
+++ b/docs/_posts/muhammetsnts/2021-07-24-redl_temporal_events_biobert_en.md
@@ -176,11 +176,20 @@ nlu.load("en.relation.temporal_events").predict("""She is diagnosed with cancer
```bash
-| | | relation | entity1 | entity1_begin | entity1_end | chunk1 | entity2 | entity2_begin | entity2_end | chunk2 | confidence |
-|---:|----:|:-----------|:-----------|----------------:|--------------:|:---------|:--------------|----------------:|--------------:|:------------|-------------:|
-| 0 | 0 | OVERLAP | PROBLEM | 20 | 25 | cancer | DATE | 30 | 33 | 1991 | 0.868028 |
-| 1 | 1 | AFTER | OCCURRENCE | 49 | 56 | admitted | CLINICAL_DEPT | 61 | 71 | Mayo Clinic | 0.798134 |
-| 2 | 2 | BEFORE | OCCURRENCE | 49 | 56 | admitted | OCCURRENCE | 89 | 98 | discharged | 0.63638 |
++--------+-------------+-------------+-----------+-----------+-------------+-------------+-----------+------------+----------+
+|relation| entity1|entity1_begin|entity1_end| chunk1| entity2|entity2_begin|entity2_end| chunk2|confidence|
++--------+-------------+-------------+-----------+-----------+-------------+-------------+-----------+------------+----------+
+| BEFORE| OCCURRENCE| 7| 15| diagnosed| PROBLEM| 22| 27| cancer|0.78168863|
+| OVERLAP| PROBLEM| 22| 27| cancer| DATE| 32| 35| 1991| 0.8492274|
+| AFTER| OCCURRENCE| 51| 58| admitted|CLINICAL_DEPT| 63| 73| Mayo Clinic|0.85629463|
+| BEFORE| OCCURRENCE| 51| 58| admitted| OCCURRENCE| 91| 100| discharged| 0.6843513|
+| OVERLAP|CLINICAL_DEPT| 63| 73|Mayo Clinic| DATE| 78| 85| May 2000| 0.7844673|
+| BEFORE|CLINICAL_DEPT| 63| 73|Mayo Clinic| OCCURRENCE| 91| 100| discharged|0.60411876|
+| OVERLAP|CLINICAL_DEPT| 63| 73|Mayo Clinic| DATE| 105| 116|October 2001| 0.540761|
+| BEFORE| DATE| 78| 85| May 2000| OCCURRENCE| 91| 100| discharged| 0.6042761|
+| OVERLAP| DATE| 78| 85| May 2000| DATE| 105| 116|October 2001|0.64867175|
+| BEFORE| OCCURRENCE| 91| 100| discharged| DATE| 105| 116|October 2001| 0.5302478|
++--------+-------------+-------------+-----------+-----------+-------------+-------------+-----------+------------+----------+
```
diff --git a/docs/_posts/murat-gunay/2022-07-07-bert_qa_legal_en_3_0.md b/docs/_posts/murat-gunay/2022-07-07-bert_qa_legal_en_3_0.md
deleted file mode 100644
index d326b05883548b..00000000000000
--- a/docs/_posts/murat-gunay/2022-07-07-bert_qa_legal_en_3_0.md
+++ /dev/null
@@ -1,87 +0,0 @@
----
-layout: model
-title: English BertForQuestionAnswering Cased model (from intogen)
-author: John Snow Labs
-name: bert_qa_legal
-date: 2022-07-07
-tags: [en, open_source, bert, question_answering]
-task: Question Answering
-language: en
-edition: Spark NLP 4.0.0
-spark_version: 3.0
-supported: true
-annotator: BertForQuestionAnswering
-article_header:
- type: cover
-use_language_switcher: "Python-Scala-Java"
----
-
-## Description
-
-Pretrained Question Answering model, adapted from Hugging Face and curated to provide scalability and production-readiness using Spark NLP. `legal-bert-qa` is a English model originally trained by `intogen`.
-
-{:.btn-box}
-
-
-[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/bert_qa_legal_en_4.0.0_3.0_1657189719594.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
-[Copy S3 URI](s3://auxdata.johnsnowlabs.com/public/models/bert_qa_legal_en_4.0.0_3.0_1657189719594.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
-
-## How to use
-
-
-
-
-{% include programmingLanguageSelectScalaPythonNLU.html %}
-```python
-documentAssembler = MultiDocumentAssembler() \
- .setInputCols(["question", "context"]) \
- .setOutputCols(["document_question", "document_context"])
-
-spanClassifier = BertForQuestionAnswering.pretrained("bert_qa_legal","en") \
- .setInputCols(["document_question", "document_context"]) \
- .setOutputCol("answer")\
- .setCaseSensitive(True)
-
-pipeline = Pipeline(stages=[documentAssembler, spanClassifier])
-
-data = spark.createDataFrame([["What is my name?", "My name is Clara and I live in Berkeley."]]).toDF("question", "context")
-
-result = pipeline.fit(data).transform(data)
-```
-```scala
-val documentAssembler = new MultiDocumentAssembler()
- .setInputCols(Array("question", "context"))
- .setOutputCols(Array("document_question", "document_context"))
-
-val spanClassifer = BertForQuestionAnswering.pretrained("bert_qa_legal","en")
- .setInputCols(Array("document", "token"))
- .setOutputCol("answer")
- .setCaseSensitive(true)
-
-val pipeline = new Pipeline().setStages(Array(documentAssembler, spanClassifier))
-
-val data = Seq("What is my name?", "My name is Clara and I live in Berkeley.").toDF("question", "context")
-
-val result = pipeline.fit(data).transform(data)
-```
-
-
-{:.model-param}
-## Model Information
-
-{:.table-model}
-|---|---|
-|Model Name:|bert_qa_legal|
-|Compatibility:|Spark NLP 4.0.0+|
-|License:|Open Source|
-|Edition:|Official|
-|Input Labels:|[document_question, document_context]|
-|Output Labels:|[answer]|
-|Language:|en|
-|Size:|407.9 MB|
-|Case sensitive:|true|
-|Max sentence length:|512|
-
-## References
-
-- https://huggingface.co/intogen/legal-bert-qa
\ No newline at end of file
diff --git a/docs/_posts/xyutech/2021-12-30-layoutlmv2_funsd_en.md b/docs/_posts/xyutech/2021-12-30-layoutlmv2_funsd_en.md
deleted file mode 100644
index fe90b7816cc72d..00000000000000
--- a/docs/_posts/xyutech/2021-12-30-layoutlmv2_funsd_en.md
+++ /dev/null
@@ -1,108 +0,0 @@
----
-layout: model
-title: LayoutLMv2 model for document understanding
-author: John Snow Labs
-name: layoutlmv2_funsd
-date: 2021-12-30
-tags: [en, licensed]
-task: OCR Text Detection & Recognition
-language: en
-edition: Healthcare NLP 3.3.0
-spark_version: 3.0
-supported: true
-article_header:
- type: cover
-use_language_switcher: "Python-Scala-Java"
----
-
-## Description
-
-Pretrained model for text and understanding. This model was trained on FUNSD dataset.
-
-## Predicted Entities
-
-
-
-{:.btn-box}
-
-
-[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/ocr/layoutlmv2_funsd_en_3.3.0_3.0_1640854497540.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden}
-[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/ocr/layoutlmv2_funsd_en_3.3.0_3.0_1640854497540.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
-
-## How to use
-
-
-
-
