Attention Through Arithmetic Intensity
Reduce MHA to its arithmetic intensity and you get exactly 1. GQA, MQA and MLA come out just as clean -- head counts and nothing else. That last constant is what puts MLA at odds with MTP.
Read full article →Reduce MHA to its arithmetic intensity and you get exactly 1. GQA, MQA and MLA come out just as clean -- head counts and nothing else. That last constant is what puts MLA at odds with MTP.
Read full article →