xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein

Developments The authors reveal an innovative manner of training protein language models using novel Masked Language Model training. They also investigate LORA and MLP adapter layers at the end for finetuning methods and show a significant gain when using LORA.

Results The resulting models are made with both standard [MASK] tokens masking tokens that indicate short-spans that are masked [sMASK] and spans marked at the end with [gMASK]. Training with both standard and block masking, at a ratio of 20% to 80%, respectively, they train models with notable improvement over models.