The case for fine-grained tracking of compute for AI by Katherine Biewer
TL;DRCurrent approaches to tracking AI compute primarily rely on a handful of hardware proxies (like FLOP/s and bandwidth) that primarily track GPU progress. These metrics are becoming less useful for accurately tracking compute for AI because they (1) measure theoretical ceilings rather than actual performance, (2) as architectures diversify away from a GPU/TPU-dominant paradigm, the metrics are becoming less comparable across different architectur...
Read full article →