Does SpERT work with GPT models?

[sameeravithana]

Hi, just wondering whether there are any efforts in the SpERT + GPT model integration?

[markus-eberts]

Hi,
I think nothing speaks against simply replacing BERT with GPT. Just have a look at the huggingface library (which we are using for the BERT implementation) for best practices. You especially need to replace the BERT related calls in spert/models.py and BertTokenizer/BertConfig in spert/spert_trainer.py. Other than that, the preprocessing, training and model architecture related code should also work with GPT.