New multitask 9in1 (Singlelabel version)

annotators/combined_classification/Dockerfile

@@ -1,14 +1,8 @@
-FROM deeppavlov/base-gpu:0.12.1
-RUN pip install git+https://github.com/deeppavlov/DeepPavlov.git@0.12.1
+FROM deeppavlov/base-gpu:0.17.5
 
-#RUN rm DeepPavlov
+RUN pip install git+https://github.com/deeppavlov/DeepPavlov.git@1f0bda76c7c3fd6ccd4a1c0c880c0fffb73522d1
 
-#Set up git lfs for your user account: git lfs install
-WORKDIR /base
-RUN rm -rf DeepPavlov
-RUN git clone https://github.com/dimakarp1996/DeepPavlov.git 
 WORKDIR /base/DeepPavlov
-RUN git checkout pal-bert+ner
 
 ARG CONFIG
 
@@ -24,8 +18,6 @@ RUN pip install -r requirements.txt
 
 COPY annotators/combined_classification/ ./
 COPY common/ common/
-RUN ls /tmp
 
-ARG DATA_URL=http://files.deeppavlov.ai/alexaprize_data/pal_bert_7in1/model.pth.tar
-ADD $DATA_URL /tmp
 CMD gunicorn --workers=1 --bind 0.0.0.0:8087 --timeout=1200 server:app
+

annotators/combined_classification/README.md

@@ -1 +1,25 @@
-BERT Base model for 6 tasks - cobot topics cobot dialogact topics cobot dialogact intent emotion sentiment toxic
+This model is based on the transformer-agnostic multitask neural architecture. It can solve several tasks similtaneously, almost as good as single-task models. 
+
+The models were trained on the following datasets:
+
+**Factoid classification** : For the Factoid task, we used the same Yahoo ConversVsInfo dataset that was used to train the Dream socialbot in Alexa Prize . Note that the valid set in this task was equal to the test set. 
+
+**Midas classification** : For the Midas task, we used the same Midas classification dataset that was used to train the Dream socialbot in Alexa Prize . Note that the valid set in this task was equal to the test set. 
+
+**Emotion classification** :For the Emotion classification task, we used the emo\_go\_emotions dataset, with all the 28 classes compressed into the seven basic emotions as in the original paper. Note that these 7 emotions are not exactly the same as the 7 emotions in the original Dream socialbot in Alexa Prize: 1 emotion differs (love VS disgust), so the scores are incomparable with the original model. Note that this task is multiclass. 
+
+**Topic classification**: For the Topic classification task, we used the dataset made by Dilyara Zharikova. The dataset was further filtered and improved for the final model version, to make the model suitable for DREAM. Note that the original topics model doesn’t account for that dataset changes(which were also about class number) and thus its scores are not compatible with the scores we have.
+
+**Sentiment classification** : For the Sentiment classification task, we used the Dynabench dataset (r1 + r2). 
+
+**Toxic classification** : For the toxic classification task, we used the dataset from kaggle <https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/datawith> the 7 toxic classes that pose an interest to us. Note that this task is multilabel.
+
+The model also contains 3 replacement models for Amazon services.
+
+The models (multitask and comparative single task) were trained with initial learning rate 2e-5(with validation patience 2 it could be dropped 2 times), batch size 32,optimizer adamW(betas (0.9,0.99) and early stop on 3 epochs. The criteria on early stopping was average accuracy for all tasks for multitask models, or the single-task accuracy for singletask models.
+
+This model(with a distilbert-base-uncased backbone) takes only 2439 Mb for 9 tasks, whereas single-task models with the same backbone for every of these tasks take up almost the same memory(~2437 Mb for every of these 9 tasks).
+
+CPU memory use of this model is 2909 Mb. 
+
+