DeepSeek open sources DSpark, a new framework to speed up LLM inference by up to 85%

DSpark can make decoding faster, but acceptance quality still determines how much speed the system actually realizes.
Semiconductor Engineering

Four Architectural Opportunities for LLM Inference Hardware (Google)

“Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundamentally different from training. Exacerbated by recent AI ...

OpenAI and Broadcom announce chip designed for LLM inference at scale

OpenAI, the company behind ChatGPT and Codex and the models those tools use, and Broadcom, an established silicon supplier, ...
Forbes

Cerebras Now The Fastest LLM Inference Processor; Its Not Even Close

The company tackled inferencing the Llama-3.1 405B foundation model and just crushed it. And for the crowds at SC24 this week in Atlanta, the company also announced it is 700 times faster than ...
InfoQ

NVIDIA Dynamo Addresses Multi-Node LLM Inference Challenges

Serving Large Language Models (LLMs) at scale is complex. Modern LLMs now exceed the memory and compute capacity of a single GPU or even a single multi-GPU node. As a result, inference workloads for ...

AI inference provider Baseten reportedly raising $1.5B in funding

Baseten Inc., a startup with a platform for running artificial intelligence inference workloads, is raising $1.5 billion in ...
Semiconductor Engineering

LLM Inference On CPUs (Intel)

“Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. However, deploying these models has been challenging due to the ...
Opinion
Database Trends and ApplicationsOpinion

OpenAI and Broadcom Debut LLM-Optimized Inference Chip

OpenAI and Broadcom are debuting 'Jalapeño,' OpenAI's first Intelligence Processor: an accelerator architected around OpenAI's vision for the future of LLM inference. According to the OpenAI and ...
Barchart on MSN

Broadcom just quietly became OpenAI’s preferred choice for AI inference. Here’s what that means for AVGO stock.

In the second half of last year, OpenAI and Broadcom (AVGO) announced a deal for 10 gigawatts worth of compute capacity. Just ...
Gizmodo

ChatGPT Can ‘Infer’ Personal Details From Anonymous Text

New research shows how popular LLMs are able to accurately guess a user’s race, occupation, or location, after being fed seemingly trivial chats. Reading time 4 minutes Quiz time: If you or your ...