MeMo's memory model lets teams upgrade their LLM without retraining it — and performance jumps 26%

Researchers' MeMo keeps AI memory separate from reasoning, so teams can upgrade their LLM without retraining it and see a 26% ...
Hackaday

TurboQuant: Reducing LLM Memory Usage With Vector Quantization

Large language models (LLMs) aren’t actually giant computer brains. Instead, they are massive vector spaces in which the probabilities of tokens occurring in a specific order is encoded. Billions of ...
Hosted on MSN

Google says TurboQuant cuts LLM KV-cache memory use 6x, boosts speed

Google researchers have published a new quantization technique called TurboQuant that compresses the key-value (KV) cache in large language models to 3.5 bits per channel, cutting memory consumption ...
Semiconductor Engineering

Dynamic KV Cache Scheduling in Heterogeneous Memory Systems for LLM Inference (Rensselaer Polytechnic Institute, IBM)

A new technical paper titled “Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System” was published by researchers at Rensselaer Polytechnic Institute and IBM. “Large ...
Tom's Hardware on MSN

Enthusiast runs 1-trillion parameter LLM from 768GB of Intel Optane DIMM memory sticks

Redditor found 768GB of affordable Optane sticks second-hand.