New KV cache compaction technique cuts LLM memory 50x without accuracy loss

MIT researchers developed Attention Matching, a KV cache compaction technique that compresses LLM memory by 50x in seconds — ...
EDN

Last-level cache has become a critical SoC design element

LLC, positioned between external memory and internal subsystems, stores frequently accessed data close to compute resources.
Ars Technica

cache size vs. transistor count? OR a challenge to my professor!!

so i got in this pissing match with my cs instructor. he was telling the class that there are four transistors per bit of L2 cache on any given cpu with on-die, full-speed cache (not actually the ...