Frank Ferraro, Department of Computer Science and Electrical Engineering
Mohammad Umair, Department of Computer Science and Electrical Engineering
Language encoders are deep neural nets with sequence-to-sequence architecture that enables transfer learning and parallelization, reducing training time greatly. In this project, we apply semantic midtuning on different language encoders, where we take a pre-trained encoder and perform the same pre-training steps on it, but with a different domain/corpus. Semantic midtuning is applied on a number of transformers including BERT and several of its modifications like SBERT, SemBERT etc and the performance is tested on several known datasets like FEVER, GLUE, SUPERGLUE, and SQUAD.