The sample efficiency black hole

One definition of intelligence is sample efficiency - that is to say, how much data do you need to see in a given domain in order to operate fluently and competently. It’s not clear that we’ve actually made much progress on training sample efficiency over the last few years - it seems like more so we’ve dramatically widened and improved the data distribution.The main way that AIs have been getting better is from adding more and better data, and scaling the compute to develop that data in the fir...