NLP Research in the Era of LLMs

Update Dec 30: Added mentions of BabyLM and the Languini Kitchen.NLP research has undergone a paradigm shift over the last year. A range of large language models (LLMs) has validated the unreasonable effectiveness of scale1. Currently, the state of the art on most benchmarks is held by LLMs that are expensive to fine-tune and prohibitive to pre-train outside of a few industry labs.In the past, a barrier to doing impactful research has often been a lack of awareness of fruitful research areas and...