Current Landscape of Machine Learning

In this brief post, I discuss some of the trends of ML and list some of the notable recent works. The way we train SotA models is slightly different from a few years ago for the purpose of optimizing the performance: We would first build a massive (often multimodal) dataset crawled from Web and model-parallelize your model with techniques from DeepSpeed, GSPMD, etc.We would then scale the following variables based on the compute budget according to the existing scaling laws or based on our explo...