Inference is now forcing a different kind of conversation.

LLM inference has two phases, prefill and decode. Prefill processes the prompt and builds the initial KV cache. Decode is what happens next: the model generates one token, feeds that token back in, then generates the next token, and so on. That loop (one token at a time) is inherently serial, which is why decode often becomes the limiting step in production inference. It shows up directly as user-visible latency, because it determines how long it takes to produce a complete response. The technical term for this is autoregressive decoding.

One of the most promising responses to the serial nature of the decode phase is speculative decoding. Instead of asking an expensive trillion-parameter model to do all the work, it splits the decode phase into two parts:

• A smaller, faster “draft model” that proposes tokens quickly, and

• A larger “target model” that verifies these proposals and decides which tokens to accept and commit.

Once you look at inference in this way, the hardware implications become clear. Drafting and verification have different memory, compute, and latency requirements. The target model (larger parent) benefits from the high-throughput characteristics of large GPU systems. The draft model (smaller child) often benefits from hardware that is optimized for low-latency, small batches, and fast token-by-token generation. As a result, the two models can benefit from very different silicon architectures, each tuned to its part of the pipeline.

That’s why I find the recent NVIDIA–Groq deal strategically interesting. It’s less about any single chip, and more about what it signals. The industry is treating draft-model-class inference seriously, and speculative decoding is one of the cleanest system-level mechanisms to turn that into lower latency and better economics.

This post systematically explores this topic and is organized as follows:

• A simple explanation of speculative decoding, a metric called acceptance rate, and why it matters

• Why disaggregating the draft and target models is a system-level win

• Why running draft models on Blackwell or Rubin GPUs can be an awkward fit, and why specialized architectures like d-Matrix and Groq are a strong alternative

• 3D DRAM based XPUs versus HBM based GPUs

• NVLink Fusion and how it can enable heterogeneous chip ecosystems

• What speculative decoding means for this era of Agentic AI

• Conclusion — SRAM vs 3D DRAM vs HBM

Disclosure: I work at d-Matrix. This post reflects my personal views and does not represent the views of the company.

Speculative decoding, simply explained

In traditional LLM serving, you deploy a single model, say Llama-70B, and it generates every token itself. During the decode phase, it repeats the same serial loop over and over. Generate a token, append it, run again, and continue until the answer is complete.

Speculative decoding, an idea introduced by Google and DeepMind in 2023, discovered that this large model does not need to generate every token by itself. Instead, the end-to-end latency can be reduced while keeping output quality high by splitting decode into two phases.

• Draft phase: A small model that closely approximates the large model (for example, Llama-7B) generates the next K tokens sequentially, but quickly.

• Target phase: The larger, more accurate model (for example, Llama-70B) verifies those K draft tokens in a single pass, treating them as a batch, and decides which tokens to accept and commit. The rest are rejected.

Then the process repeats. Draft a few more tokens, verify the next chunk, commit what’s accepted. This draft → verify loop continues until the full response is generated.

The figure below, from Google’s paper, shows how speculative decoding reduces wall-clock time compared to standard inference. This is quite promising because it can deliver meaningful gains in both latency and throughput without sacrificing output quality.

The promise of speculative decoding

A technical post by NVIDIA from Dec 2024 shows just how dramatic speculative decoding can be in practice.

• Llama-70B on one Hopper H200 yields about 51 tokens/sec without speculative decoding. But when they pair it with a tiny Llama-1B draft model, still on the same GPU, the throughput jumps to about 146 tokens/sec.

• For a much larger model, Llama-405B on 4× H200, they show about 33 tokens/sec without speculative decoding. With speculative decoding using a Llama-3B draft model, throughput jumps roughly 3.6× to about 120 tokens/sec.

A 3x improvement in inference throughput from one technique is quite incredible.

Acceptance Rate

The original intuition with speculative decoding was that there are simple, predictable continuations where a draft model can easily guess what comes next, like completing:

“The quick brown fox …”

But in more esoteric cases (rare subject domains, weird proper nouns, highly constrained formatting, tricky reasoning steps) the draft may diverge quickly, and most of the tokens get rejected by the target model.

A key metric to notice here is the acceptance rate. The draft model still has to do a decent job. If the target model accepts something like 80–90% of the draft tokens, then speculative decoding starts to feel useful. It will reduce end-to-end latency and raise throughput. But if most of the draft tokens are rejected then the whole trick falls apart. You’re doing extra work to draft tokens that you end up throwing away.

Medusa & EAGLE: Alternative mechanisms of speculative decoding

Since its introduction in 2023, speculative decoding has remained an active research area and has gotten better over time. The core idea is still the same: keep output quality high while reducing how often the expensive target model has to step through the serial decode loop.

A lot of follow-on work focuses on improvements such as higher acceptance rates, lower overhead, and simpler deployment.

An important approach is avoiding the requirement to load two completely separate models. Medusa and EAGLE take this approach. They add a draft stage inside the target model itself so you get speculative style candidates without standing up a full separate draft model. This is equivalent to grafting a small draft head onto the LLM’s pipeline.

A useful reference here is the recent survey of speculative decoding techniques.

Subscribe now

Why disaggregating draft and target models is a system-level win

In June 2025, the ByteDance team published a remarkable paper called SwiftSpec and argued that existing speculative decoding systems were leaving significant performance on the table. They traced it to two main issues.

First, the draft stage sits idle when the target stage is performing verification, i.e., the draft → verify loop still happens sequentially. The N^th draft stage cannot execute until the (N-1)^th verify step has completed. So the end-to-end latency is not optimized.

Second, large target models usually require multiple GPUs with model parallelism. Earlier deployments often co-located draft and target, and then tried to scale them both the same way across GPUs. SwiftSpec points out why this is inefficient in small-batch serving. The draft and target have very different compute needs, and when you force both through the same tensor-parallel setup, you run into compute imbalance, KV cache consistency issues, and communication overhead that is hard to amortize at low batch sizes.

They solved these two problems by first disaggregating the draft and the target models and running them on different groups of NVIDIA Hopper H200 GPUs, as shown in the figure below.

This allowed each side to scale differently depending upon the models deployed, whether it’s Llama, Qwen, or DeepSeek. The figure below shows how end-to-end performance changes as you vary how many H200 GPUs are allocated to the target versus the draft model.

They also developed a novel scheme to let the two models run asynchronously. The draft model generates a tree of tokens and can move ahead and generate candidate tokens for the next iteration while the target stage is still verifying the previous one. After each step, the two models synchronize state, and the draft “re-roots” by keeping only the portion of its token tree that the target has verified. This is represented in the figure below.

With this setup, they report an average 1.75X speedup over previous speculative decoding systems. The highlight of the paper is their claim that SwiftSpec serves Llama3-70B at 348 tokens/s on 8 NVIDIA Hopper GPUs, which they describe as the fastest known system for low-latency LLM serving at this scale.

Why Blackwell and Rubin are awkward draft engines and why heterogeneous XPUs can win

Decode is memory bound and draft decoding is the worst-case version of that

SwiftSpec has a section called “GPU Constraints for Low Batch Size”. The basic point is that GPUs are incredible when you can keep them busy with large batches and big, regular kernels.

Read more

To understand why Arista Networks exists today, and how it grew into a company valued at over $100 billion, you have to rewind to the mid-2000s and step inside John Chambers’ Cisco. You have to understand this unique, highly lucrative, and ultimately controversial business model known as the “Spin-In”.

This is the story of the team behind Pensando (acquired by AMD in 2022), the rise of Jayshree Ullal, and how internal conflicts inadvertently funded the creation of Cisco’s fiercest competitor, Arista.

I worked at Cisco from the mid-2000s to early 2010s and witnessed some of these dynamics firsthand.

The “Spin-In” Innovation Machine

In the 2000s, Cisco CEO John Chambers had a problem. Cisco was becoming a behemoth, and as with many other big companies, it started moving slowly and losing its competitive edge. Chambers worried that top talent would leave to start their own companies, or that Cisco would miss the next big technology wave because of bureaucratic inertia.

His solution was the “Spin-In.”

Instead of forcing his best engineers to navigate HR and finance approvals to build a new product, Cisco would fund them to leave. They would form a “startup” just down the street. Cisco would own the rights to buy them back later at a pre-determined, massive valuation if they hit their milestones.

It was genius, but it created a two-tier system: the chosen few got startup equity and autonomy; everyone else got a salary and corporate red tape.

The “MPLS” Team

The primary beneficiaries of this model were a legendary quartet of executives. Inside Cisco, they were known by the moniker MPLS. It was a cheeky pun on the famous networking protocol (Multiprotocol Label Switching), but it stood for their first names:

• Mario Mazzola

• Prem Jain

• Luca Cafiero

• Soni Jiandani

This team had a Midas touch. Every few years, the MPLS team would “leave,” build the next big thing, and get bought back by Cisco for hundreds of millions. Of course, to their credit, the products developed by each of their companies went on to become important pieces in Cisco’s roadmap.

1. Crescendo (1993): Acquired for ~$95M. This became the Catalyst 5000/6000 series, arguably the most successful networking product in history.

2. Andiamo (2002): A storage spin-in acquired for ~$750M. This became the MDS 9000 (SAN switching).

3. Nuova Systems (2008): A data center spin-in acquired for ~$678M. This became the Nexus switch and the UCS server line.

4. Insieme (2013): An SDN spin-in acquired for ~$863M. This became Cisco ACI (Application Centric Infrastructure).

The Collision Course

One of the key figures who had joined Cisco via the Crescendo acquisition in 1993 was Jayshree Ullal. She went onto become a seminal figure within Cisco, loved and admired by the majority of Cisco employees. By 2008, she was Senior VP of the Data Center, Switching, and Services Group. She was responsible for over $10 billion in revenue.

So, while the MPLS team was operating in their lucrative semi-private bubble, Jayshree Ullal was running the actual business within Cisco. She was the one executing, scaling, and managing the massive Catalyst portfolio that paid the bills.

The friction reached a breaking point with Nuova Systems.

Nuova (the MPLS team’s 2008 spin-in) was building the Nexus switch. This wasn’t just a complementary product; it was the future of the data center. It was designed to handle a unified fabric of Ethernet and storage.

The problem was that it directly cannibalized the Catalyst business Ullal was running.

Ullal found herself in an impossible position. She was managing the company’s biggest vertical, yet the future roadmap was being built by a “startup” team that operated outside her control. A team that was guaranteed a massive payout while her own team was bound by standard corporate compensation.

Realizing that the structure favored the “spin-in” inner circle over operational leadership, Ullal resigned in May 2008.

The Vindication: Arista Networks

Five months later, Ullal joined Arista Networks as CEO, teaming up with founders Andy Bechtolsheim and David Cheriton.

At Arista, Ullal didn’t just compete with Cisco; she exploited the weakness of the MPLS philosophy.

The MPLS team (and Cisco generally) believed in Custom ASICs. They believed that to get the best performance, you had to design your own chips. While custom chips offer differentiation and have legitimate advantages, they are proprietary and time consuming to produce.

Ullal and Arista took a different bet: Merchant Silicon. They believed that generic chips from Broadcom would eventually catch up to custom silicon in speed, but at a fraction of the cost. Arista focused their engineering power not on chips, but on software building a programmable, Linux-based stack that the hyperscalers (Google, Microsoft, Meta) were desperate for.

She was right.

While Cisco was busy integrating their expensive Nuova/Insieme acquisitions and fighting complexity, Arista’s “commodity chip + superior software” approach swept through the cloud market.

The piece recounts the bitter rivalry between Cisco and Jayshree Ullal. What started as respect between them, turned personal as Arista’s success eroded Cisco’s market share in switching — a business Jayashree Ullal once ran.

Cisco responded with internal “war room” efforts and legal battles, accusing Arista of copying its technology, while Arista characterized those tactics as defensiveness from a legacy company losing ground to a faster, more innovative competitor.

The Aftermath: End of the spin-in model

The spin-in era eventually ended. When Chuck Robbins took over as Cisco CEO in 2015, he dismantled the model, recognizing that it destroyed internal morale and siloed innovation.

The MPLS team left Cisco one final time to found Pensando Systems (focused on DPUs/SmartNICs). Cisco did not buy them back this time. Instead, AMD acquired Pensando for $1.9 billion in 2022.

Meanwhile, Jayshree Ullal went on to build Arista into one of the most successful networking companies of the cloud era. Today she is widely regarded as one of Silicon Valley’s most effective CEOs. And at a net worth north of $5 billion, if you assumed the richest Indian-American CEO in the Valley was Satya Nadella or Sundar Pichai — you’d be mistaken.

It’s Jayshree.

Share

Read this next …

This article traces the pivotal innovations that emerged from this partnership and how it put NVIDIA in the dominant position it is today. We’ll dissect how Jensen played his hand in five phases, each one pushing the GPU further out of its role as a graphics “sidekick”, to becoming the heart of the modern AI data center.

1. 2009: GPUDirect Shared Memory — Breaking the ice between the GPU and the NIC.

2. 2013: GPUDirect RDMA — NICs and GPUs learn to talk directly, accelerating the GPU-to-GPU path.

3. 2019: GPUDirect Storage — The storage bottleneck is removed, accelerating the path from SSD to GPU.

4. 2022–Present: Spectrum-X & BlueField. Building AI factories, and taking over the data center to challenge Broadcom.

5. Retrospective analysis and the Future — What’s next for the GPU-centric architecture?

By the end, you’ll see how the signal for companies like SNDK 0.00%↑ , WD 0.00%↑ were visible much earlier than most of us realized — it was in front of us all along.

The story goes like this…

2009-10: Tianhe-1A and China’s Emergence in Supercomputing

In 2009, during the Tesla and Fermi era of NVIDIA GPUs, there was a surge of interest from the scientific and high-performance computing (HPC) communities in using GPUs to accelerate computation.

This interest wasn’t speculative. Real work was already showing that GPUs could fundamentally reshape the computing landscape. For instance, NVIDIA demonstrated that LINPACK, the benchmark used to rank the world’s fastest supercomputers, could be accelerated with GPUs.

At the same time, China’s ambition to enter the supercomputing arena had become obvious. System by system, they climbed the TOP500 rankings. Then, in November 2010, they did it. Tianhe-1A, installed at the National Supercomputing Center in Tianjin, surpassed the American Cray XT5 supercomputer, installed at the Oak Ridge National Laboratory to claim the #1 spot. It marked the end of a decade-long period of U.S. dominance.

What’s remarkable isn’t just that China took the top position — it’s how quickly the underlying architecture shifted. If you compare the Top 5 supercomputers from November 2009 to November 2010, we see that in the span of a single year, three of the top five supercomputers are now GPU-accelerated, using NVIDIA GPUs.

The promises that GPUs were living up to in this era had nothing to do with AlexNet, or deep learning, or transformers. This was pure high-performance computing. Which makes me think that Jensen’s conviction in GPUs as a foundational computing primitive long pre-dated AI as we know it today.

It’s not hard to draw architectural parallels between Tianhe-1A’s compute trays and something like a modern GB200. The machine was composed of:

• 14,336 Xeon processors and 7,168 NVIDIA GPUs. Arranged in 3,548 compute trays, each containing two nodes.

• Each node had 2× Intel Xeon X5670 6-core CPUs and 1× NVIDIA Tesla M2050 GPU.

• The interconnect was proprietary, which delivered roughly 2× InfiniBand bandwidth at the time.

While Tianhe-1A and several other signs showed that GPUs were the future of high-performance computing, there was a fundamental problem.

In order for large compute clusters to work together cohesively as one unit, they have to communicate through high-speed links. But, GPUs were stuck behind CPUs. Every movement of data had to be orchestrated by the host CPU. GPUs couldn’t talk to the network directly. They couldn’t talk to storage directly. They were fast, but fenced in.

That bottleneck set the stage for what came next.

In 2009, NVIDIA and Mellanox formed a partnership to address exactly this problem — how to liberate GPUs from the CPU-centric data path and make them first-class citizens in the data center.

Subscribe now

2010: GPUDirect “Shared Memory” — Fermi + Connect-X2

At SC09 (Supercomputing 2009), NVIDIA and Mellanox formally announced their partnership.

At the time, a typical GPU-accelerated server looked like this: a CPU running Linux acted as the master, with a GPU and an InfiniBand NIC attached to it over PCIe. During PCIe enumeration, the GPU and the Mellanox card were assigned separate address regions in CPU’s memory (host memory). As a result, the CPU was responsible for initiating and orchestrating all data movement between the GPU and the InfiniBand network.

This led to an especially wasteful stage in the data path called buffer copy. The GPU puts data in System RAM (Buffer A). The CPU reads Buffer A and copies it to Buffer B (which the Network card can see). Their first goal with this partnership was quite simple — eliminate this buffer copy step.

Sounds deceptively simple, but in order to make this happen there were 3 changes necessary:

• Linux kernel updates to allow NVIDIA and Mellanox drivers to share host memory. This made it possible for the InfiniBand NIC to directly access buffers allocated by the CUDA library, eliminating the need for buffer copies.

• Driver updates on both sides. The NVIDIA and Mellanox card drivers had to shake hands and agree to use Buffer A for everything.The Mellanox driver registered callbacks that allowed the GPU to notify any changes performed during run time in the shared buffers.

This simple update accelerated GPU communications over Infiniband by 30%. More importantly, it broke the ice between the GPU and the NIC. This was the first step toward direct, coordinated data movement.

2013: GPUDirect RDMA — Kepler, Connect-X3, accelerating GPU-GPU path

Even with the 2010 GPUDirect improvements, data still had to detour through the CPU’s system memory. That extra hop added latency and capped bandwidth at the speed of the host memory bus.

Which naturally raised the question — why can’t the NIC just read directly from the GPU’s memory?

That question is exactly what GPUDirect RDMA (Remote Direct Memory Access) answered and it became possible with the Kepler-class GPUs (NVIDIA Tesla K40).

Why this wasn’t possible before Kepler

On older GPUs (Fermi and earlier), the interface between PCIe and GPU VRAM was fundamentally constrained. Although a GPU might have had 6 GB of VRAM, only a relatively small aperture (typically ~256 MB, exposed via BAR1) was addressable from the PCIe bus at any given time.

To access data outside that window, the NVIDIA driver (running on the CPU) had to reprogram the BAR1 mapping to point to a different region of VRAM. The GPU would slide the window, and only then could the CPU read the data. This remapping was entirely CPU and driver-controlled, and devices like a Network Card are not smart enough to issue these “Move Window” commands. It just sends a “Read” request to a physical address. Without a stable, fully mapped view of GPU memory, direct access was impossible.

What changed with Kepler

Kepler fundamentally changed this model.

With Kepler, NVIDIA exposed the entire GPU VRAM address space directly to the PCIe bus. This meant that a NIC could now access any location in GPU memory at any time without CPU involvement and without remapping windows.

For the first time, the network could treat GPU memory as a first-class RDMA target.

The software stack catches up

On the software side, NVIDIA introduced GPUDirect RDMA support in CUDA 4.0, while Mellanox updated their MLNX_OFED drivers (i.e., the infiniband drivers) to enable true peer-to-peer RDMA paths between GPU memory and Mellanox adapters such as the ConnectX-3. Together, this enabled a direct data path between the GPU and the NIC.

GPU VRAM ⇄ Mellanox NIC ⇄ Network

How programmers actually used it: CUDA-Aware MPI

Next, it’s important to understand how exactly programmers used this feature. The real unsung heroes are the developers/maintainers of MPI (Message Passing Interface), which is a standard library used in HPC. It’s how partial results of computations are exchanged between nodes of a cluster, when trying to solve one giant physics problem.

Before GPUDirect RDMA, the MPI code had to copy over data from the GPU to the CPU’s memory using cudaMemCpy(*gpu_mem_ptr, *host_mem_ptr), and then sent the data to the peer using MPI_Send(*host_mem_ptr).

With GPUDirect RDMA, the MPI libraries were updated so that you could invoke MPI_Send(*gpu_mem_ptr) directly. Underneath the hood, this would tell the NIC to go read the GPU memory directly. The cudaMemCpy(gpu2host) step was eliminated entirely.

Once again, the change to MPI looks deceptively simple but the implications and work involved to enable it were profound. In 2013, the MPI core developers presented this update publicly, showing concrete improvements in both latency and throughput enabled by GPUDirect RDMA. It’s well worth watching that presentation to understand just how significant the shift was.

NVIDIA also published a detailed developer blog walking through how to use CUDA-aware MPI in practice, and benchmarking CUDA-Aware MPI.

2019: GPUDirect Storage — Accelerating GPU to SSD data path

By the end of the 2010s, you could see NVIDIA’s attention shifting to storage.

Read more

From a hardware perspective, Supermicro sells a product, called a JBOF (Just a Bunch of Flash), much like this one, which can be purchased today. This device uses the previous generation NVIDIA BlueField-3 DPU.

From a software perspective, NVIDIA has also invested in and partnered with WEKA, which recently introduced a software offering called the Augmented Memory Grid. This system leverages NVIDIA Dynamo and NIXL to connect GPU clusters like GB200 to petabyte-scale NVMe systems and provide fast, persistent KV cache storage. Crucially, it enables KV cache to move directly in and out of GPU HBM with minimal overhead. They call it the WEKA Token Warehouse (more on that later).

In the rest of this article, I’ll take a step back and unpack what all of this means, focusing on three key dimensions.

First, we’ll put some numbers in context and examine how big this problem is with KV cache in modern inference workloads and why is it a limiting factor for scale and utilization.

Next, I’ll walk through the system architecture. This includes what a DPU actually is, how the platform integrates with NVIDIA’s GB200, GB300, and Vera Rubin systems, and why the combination of this hardware with NVIDIA’s Dynamo software stack is the real differentiator, and not just the hardware alone.

Finally, I’ll focus on what I see as the most important part of the discussion: the strategy. I’ll zoom out and speculate on NVIDIA’s broader play here. Specifically exploring how this mirrors AWS’s strategy with Annapurna Labs, and how the $900M Enfabrica deal from last September might be the key to what comes next in this product’s roadmap. I’ll also look at how this fits into NVIDIA’s continued expansion across the data-center stack through its growing portfolio of six core chips and investments in companies such as VAST Data and WEKA.

Subscribe now

The Problem

How bad is the KV-Cache memory footprint?

Let’s consider a classic LLM like GPT-3 with 175B parameters.

• Every token that the model generates has a memory footprint of roughly ~4.5 MB.

• For a user chat session with around 2,048 tokens (that’s roughly 1,500 words), this translates to about 10 GB of memory just for KV cache.

In a GB200 compute tray with 4× Blackwell GPUs and 744 GB total HBM3e, storing the model weights alone takes about 350 GB (175B parameters × 2 bytes for FP16). That leaves roughly 400 GB for everything else — including KV cache.

At ~10 GB per user session, that’s only ~40 concurrent users per tray. Scaling to millions of users with these numbers is unrealistic.

Of course, GPT-3 is old by today’s standards. Since then, there have been significant optimizations in attention mechanisms. Grouped-Query Attention (GQA), introduced by Google and popularized by Meta’s Llama 2, and more recently DeepSeek’s Multi-Head Latent Attention (MLA), have dramatically reduced KV cache size. These techniques bring the per-token KV cache footprint down from ~4.5 MB (4608 KB to be precise) to around ~71 KB.

But this doesn’t actually make the problem go away.

We are now firmly in the era of reasoning models. Even if the user-visible conversation is only a few hundred words, the model’s internal chain of thought can generate tens of thousands of tokens. All of that intermediate state becomes part of the KV cache, and it has to be stored somewhere.

If we assume a more realistic average of 10,000 tokens per session, a GB200 tray can still support fewer than 1,000 users per compute tray. This is why Jensen repeatedly emphasized in the keynote that KV cache management is one of the biggest pain points for AI labs, cloud service providers, and customers.

There’s also a product-level issue here. If I think about my own usage of ChatGPT or Gemini, I often pick up a chat session from days or weeks ago. That means KV cache can’t just live in GPU memory and disappear. It needs to be offloaded, stored, and retrieved later when the session resumes.

KV cache is no longer a temporary artifact. It’s becoming persistent state.

Note: I’ll be diving deep into the history of KV Cache optimization in my next post — including MHA, GQA, MLA, quantization, and compression. Subscribe to get notified when that drops.

Subscribe now

The Solution

Expanding KV-Cache outside the rack

Today, there is some breathing room. On a GB200 compute tray, alongside the 4 GPUs, you have 2 Vera CPUs with a combined ~1 TB of LPDDR5 memory. KV cache can be evicted from GPU HBM into CPU DRAM.

This is what Jensen was referring to in the keynote when he said, “Right now, the GPUs in each node have 1 TB of space.”

But this is still nowhere near enough. CPU memory is just another tier in the same rack, and it doesn’t solve the problem of large-scale, persistent context storage.

NVIDIA’s solution is to expand KV cache outside the rack entirely, using network-attached storage. To make this viable at scale, NVIDIA needs both a hardware solution and a software stack that understands KV cache as something that can move fluidly across the memory hierarchy.

Hardware: BlueField-4 and the role of a DPU

At the center of this solution is the BlueField-4 DPU. But, what exactly is a DPU?

At a high level, a DPU lets you attach large amounts of storage (typically NVMe SSDs) on one side, and connect that storage to a GPU or CPU cluster on the other side using high-speed networking. The goal is to make this remote storage appear to the compute node as if it were locally attached.

For this to work, the connection must be both low latency and high bandwidth. Without a DPU, you would need a separate NIC, a CPU, and a PCIe switch to achieve something similar, and the result would be slower, more complex, and harder to scale.

A DPU collapses all of this into a single ASIC. The BlueField-4 combines networking, general-purpose processing, PCIe switching, and hardware acceleration for features that are mandatory for storage expansion, such as VXLAN, encryption and decryption, and traffic management. Remember, these racks and pods are multi-tenant environments.

The BlueField line came to NVIDIA through the Mellanox acquisition in 2020, and BlueField-4 is the latest generation of that architecture.

In the Inference Context Memory Storage Platform, NVIDIA uses four BlueField-4 DPUs. Each DPU is connected to roughly 150 TB of storage, for a total of about 600 TB per appliance.

Other notable companies that made DPUs were Fungible (acquired by Microsoft in 2023) and Pensando (acquired by AMD in 2022). Fungible was a failure, while Pensando was a success. We’ll explore these DPUs in a separate article.

Software: NVIDIA Dynamo, NIXL, KV Block Management, and DOCA

Hardware alone isn’t sufficient. As I mentioned in the introduction, you can already buy a Supermicro JBOF today that uses BlueField-3 DPU. There is nothing fundamentally new about attaching NVMe over the network. The real differentiation is software.

At GTC 2025, NVIDIA announced Dynamo, a new inference framework built from the ground up. Dynamo does many things, but one of its core goals is KV block management. This includes native support for evicting KV cache from GPU memory, offloading it to CPU memory or external storage, and retrieving it later.

A key part of this is the new asynchronous transport library called NIXL, which allows KV cache to move anywhere in the memory hierarchy—HBM, Grace or Vera CPU memory, or fully off-rack storage—without interrupting ongoing GPU computation.

The Inference Context Memory Storage Platform is the hardware counterpart to Dynamo. It is a purpose-built appliance with the necessary networking and data-processing features required to make large-scale KV cache offload and retrieval practical in NVL72 and NVL144 racks.

While discussing NVIDIA Dynamo and Mellanox, it’s worth remembering that NVIDIA’s push toward greater control over AI storage goes back to its 2022 acquisition of Excelero. That acquisition brought key technologies (such as the NVMesh, a low-latency block storage layer) into DOCA (Data Center Infrastructure-on-a-Chip Architecture), the software stack that runs on BlueField DPUs. It can be thought of as CUDA for NVIDIA’s networking and data-center infrastructure stack.

The Strategy

Disaggregating Compute and Context

This entire announcement strongly reminds me of AWS’s approach to disaggregated compute and storage.

AWS acquired Annapurna Labs in 2015 for roughly $350–370 million to solve a fundamental efficiency problem in their data centers known as the "virtualization tax". At the time, AWS was reliant on commodity chips from Intel and AMD. They faced two major issues:

1. The "Virtualization Tax": A significant portion (estimated around 30%) of a server's processing power was being wasted just managing the overhead of the cloud (running the hypervisor, networking, and security protocols) rather than running customer applications.

2. Lack of Control: AWS was stuck following Intel's roadmap and release cycles, which were slowing down their ability to innovate or reduce costs.

Annapurna Labs provided the solution with what eventually became the AWS Nitro System. This technology allowed AWS to offload those "overhead" tasks (storage, networking, security) onto a dedicated, low-cost card.

This move unlocked massive value. AWS could now sell nearly 100% of a server's resources to customers, as the main CPU was no longer burdened by administrative tasks. It also dramatically improved network and storage performance, making remote storage feel like a local drive.

This was a massive win for the end users and customers as well. One of AWS’s most important innovations was Elastic Block Storage (EBS). Compute and storage were decoupled, and the software layer along with specialized hardware made it easy to independently choose instance type and storage size depending on application.

NVIDIA appears to be following a very similar playbook. Mellanox, Excelero, and BlueField DPU are the Nitro equivalent. The Context Memory Storage Platform is the mechanism for scaling storage attached to a GB200 or Vera Rubin superchip.

Except instead of block storage, the resource being disaggregated is model context.

Scaling up to PetaByte/ExaByte token warehouses, the Enfabrica deal, and what might come next

Read more

The Grace-Blackwell CPU + GPU couplet will be succeeded by the new Vera-Rubin platform. In addition to the LPDDR-based Vera CPU and the HBM4-based Rubin GPU, there’s now a third processor in the mix: the GDDR7-based Rubin CPX.

The idea behind this new processor is straightforward — make inference cheaper for providers while improving performance for users. This is done by separating the two stages of LLM inference, prefill and decode, and have the CPX run the prefill step and let the Rubin GPU run the decode step.

At GTC in March 2025 NVIDIA released Dynamo, their new inference framework, which makes disaggregate inference a first-class citizen and at the AI Summit in September 2025 they announced Rubin CPX. This marks NVIDIA’s firm commitment towards this disaggregated model of inference.

In this article, we’ll take a step back and analyze from first principles:

• The core issues with LLM inference and what are prefill and decode.

• Why separating prefill and decode helps, and the research that uncovered these optimizations.

• Then we’ll walk through the Rubin CPX architecture, examine its capabilities, limitations, and compare it with Blackwell and Rubin GPUs.

Subscribe now

What makes LLM inference difficult

Quality of Service

If you’re a paying user of ChatGPT, Gemini, or Claude Code, you have certain expectations about how responsive the service should be. The industry term for these expectations is called Service Level Agreement (SLA). For LLMs, two metrics determine whether that SLA is being met: Time to First Token (TTFT) and Time Per Output Token (TPOT).

These metrics are fairly intuitive. In a chatbot, TTFT is the time between submitting a prompt and seeing the first token appear. This phase of of processing is called Prefill. While TPOT is the speed at which each subsequent token is generated. This is the Decode phase.

Different applications have different requirements. A chatbot like ChatGPT needs a fast TTFT so the response feels immediate, but TPOT only needs to keep up with normal reading speed. In summarization, users tolerate a slower TTFT, but once the summary starts, they expect the rest of the output to appear quickly, so TPOT matters more. With tools like Claude Code, both TTFT and TPOT need to be fast.

What complicates things further for inference providers is that memory and compute demands vary dramatically across applications. The size of the input, (prefill phase) and the amount of output (decode phase) can differ by orders of magnitude. Text-to-image prompts are short but produce large outputs. Summarization and code generation often have long inputs but shorter outputs, and chat sits somewhere in between.

Economics

Managing the economics of LLM serving while meeting SLA is tricky. Different workload shapes from different customers make it hard to use the same hardware efficiently without breaking someone’s SLA.

For simplicity, imagine OpenAI is running ChatGPT on a single NVIDIA H100. To maximize profit, they want to serve as many user requests as possible on that one GPU while still meeting TTFT and TPOT targets. That effective “packing” of users per GPU is the batch size. If they try to fit too many users, latency spikes and people get frustrated. If they’re too conservative, they end up needing a prohibitively large GPU cluster to serve their millions of users.

To make things worse, you don’t always need many users to overload a system, you just need the wrong mix. Even with just two users, one user might ask, “I want to learn the piano. Give me step-by-step instructions and resources.” At the same time, another user uploads a 50,000-word document and asks for a summary. On a shared GPU, your TTFT might look fine, but once the long summarization request arrives, your TPOT can get hammered, and you’ll see a noticeable pause in the middle of your chat.

So in order to maximize revenue while meeting SLAs and keeping users happy, we need to do two things:

1. Understand how different applications stress the GPU, i.e., what their workloads actually look like in terms of compute and memory

2. Find an efficient way to batch and schedule these requests so we can get the most throughput out of the hardware while still hitting TTFT and TPOT targets.

How we got to today’s LLM inference

Next, we’ll look at how LLM inference has evolved over the years through the lens of four seminal papers. These papers offer valuable insights, and understanding what they uncovered is key to seeing the path that led to Rubin CPX.

ORCA: 2022

Paper: ORCA: A Distributed Serving System for Transformer-Based Generative Models

Key innovations:
  - Iteration-level scheduling
  - Selective and continuous batching

Inference serving systems usually have two parts: an inference server or scheduler, which receives and batches user requests, and an execution engine, which issues kernels to the GPU.

For models like ResNet, used in image recognition, this setup is simple. Every request is a single forward pass through the network, all inputs have the same shape, batching is trivial, and the end-to-end latency is predictable. When the batch is done processing, the users get their responses. This is called request-level scheduling.

The authors of this paper showed that when you apply this kind of request-level serving system to LLMs, where each request varies widely in length and compute time, you run into two major problems.

• For instance, in a setup with NVIDIA Triton (as the server) + FasterTransformer (as the execution engine), if three user requests were batched together, even if one finished early, that user wouldn’t get a response until all jobs in the batch were done.

• If new requests arrived while a batch was running, they had to wait for the entire batch to finish before being scheduled, even if there were empty slots available.

The diagram below shows how these issues lead to high latency and poor GPU utilization, but the bigger consequence is its implication on cost of inference.

ORCA replaced request level scheduling with iteration level scheduling, so batching happens on a per-token basis instead of per-request. So when a user’s job is done, it returns immediately. This also allowed new requests to be batched in continuously as older ones relinquished their slot.

Fixing these issues increased throughput. With ORCA, the cost to run a GPT3-175B model, running on 2 nodes each with 8 x A100 GPUs, went down from $476,000/month to $14,000/month.

SARATHI: 2023

Paper: SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

Key innovation: Chunked prefill

This paper built on ORCA and reduced inference costs by another 25%. The key insight was that the prefill and decode stages have very different compute utilization patterns. Prefill can saturate a GPU even with a single request, while decode only becomes compute-efficient at large batch sizes. In other words, prefills were efficient, but decode suffered from poor GPU utilization.

Another issue in ORCA was, even within an active batch, there was some serialization happening, the requests were not parallelized efficiently. This caused “bubbles”, especially when a large model was split across GPUs. ORCA itself acknowledges this limitation, and this paper focused on addressing it.

SARATHI split each prefill into equal-sized chunks and built batches containing one prefill chunk and multiple decode requests. With this setup, during inference, the prefill chunk fully saturated the GPU, allowing the decode requests to “piggyback” at a much lower cost, up to an order of magnitude cheaper than running a decode-only batch.

Although SARATHI improved decode throughput by up to 10×, it introduced one drawback. Chunking made prefills slower. As a result, the overall speedup over ORCA was only about 1.25×. Here’s how the paper describes it:

We note that although we improve decode efficiency by up to an order of magnitude, the end-to-end speedups and in turn monetary savings in inference cost are in the order of [only] 25%. This is because our technique only improves decodes and not prefills.

Splitwise & DistServe: 2024

Papers: Splitwise & DistServe

Key innovation: Disaggregated prefill and decode

This brings us to the two papers most relevant to Rubin CPX: Splitwise and DistServe. These parallel efforts showed the benefits of fully disaggregating prefill and decode and running them on separate GPU clusters. Splitwise demonstrated that disaggregation can deliver up to 1.4× higher throughput at 20% lower cost compared to SARATHI, or 2.35× more throughput under the same power and cost budget. They also showed that the decode phase can run on less compute-capable hardware with better perf/W and perf/$.

The authors of Splitwise clearly captured the impact of disaggregation. Their baseline was the standard setup where prefill and decode run together on A100 or H100 GPUs. They then compared this against configurations where prefill and decode were split across different machine types. The two charts below show throughput under iso-power (same power budget) and iso-cost (same server and operating cost). My annotations explain how to read them.

Machine combinations simulated:

+ A100.Prefill + A100.Decode
+ H100.Prefill + H100.Decode
+ H100.Prefill + A100.Decode
+ H100.Prefill + H100Cap.Decode (H100 cluster for decode were power capped).

DistServe contributed additional ideas. Given a model, workload, latency target, and machine types, their algorithm determines how many machines to allocate for prefill and decode and what parallelism strategies to use for each of them.

KV-Cache transfer

One inherent overhead in disaggregation is KV-cache transfer. After the prefill cluster processes the prompt, the KV-cache must be moved into the decode cluster’s memory. Both papers analyzed this cost and proposed strategies to reduce its impact.

NVIDIA Dynamo: GTC 2025

Finally, in March 2025 at GTC, NVIDIA announced Dynamo, their new inference framework. Like ORCA, Splitwise, and DistServe, it acts as the orchestrator, bringing together many of the ideas from these papers and adding several new capabilities of its own.

Notable:

In the GTC session, NVIDIA mentioned that, in 2024 and 2025 they met some Chinese customers who were pairing H800 GPUs for prefill and H20 GPUs for decode and that brought a lot of cost savings.

Some of Dynamo’s key features include:

• KV-cache aware request routing, which sends user requests to the right deployment cluster based on cache locality.

  

• A KV-Block Manager that can offload and retrieve KV-cache, significantly improving performance for workloads like code generation and multi-turn conversations.

  

• Production-grade serving tools, including fault tolerance and auto-scaling for separate prefill and decode clusters.

• The NIXL library, which provides an asynchronous peer-to-peer transfer API for moving data, such as KV-cache, between prefill and decode machines and across memory hierarchies. The async behavior is important because it lets communication overlap with computation. This differs from NCCL, which is mainly designed for collective operations and is not asynchronous.

  

• An AI Configurator, which recommends the best deployment setup (cluster size, parallelism strategies, and so on) based on the model and latency requirements.

Rubin CPX

This finally brings us to Rubin CPX. In my article Three examples of how ASICs and FPGAs are used as accelerators, I describe a recurring pattern in engineering: whenever possible, software optimizations transition into specialized hardware. With NVIDIA Dynamo and inference engines like vLLM pushing the industry toward disaggregated inference, the logical next step is to formalize the architecture and build hardware tailored to the use-case rather than relying on general-purpose GPUs, and Rubin CPX is exactly that.

To appreciate the capabilities of this new processor, it helps to compare it with NVIDIA’s broader GPU lineup. A quick look at the spec sheet shows that Rubin CPX isn’t a watered-down B200 or Rubin GPU. It’s much closer to an enhanced version of the NVIDIA RTX 6000 PRO Blackwell workstation-class GPU.

This is evident from the inclusion of NVENC/NVDEC video engines (to accelerate video generation workloads). Primary connectivity is PCIe and there’s no NVLink. The memory is GDDR7 and not HBM. Let’s dig into these further.

Compute

This CPX delivers 30 PFLOPS of sparse FP4 compute, which is substantial considering the dual-die Rubin GPU does 50 PFLOPS. So, on a per-die basis, the CPX provides more FP4 compute than a single Rubin GPU die.

Read more

• January 2025 : NVIDIA announced Project DIGITS at CES, revealing the form factor, compute capabilities, and a new partnership with MediaTek to co-develop the GB10 chip.

• March 2025: The project was officially renamed DGX Spark, replacing the DIGITS codename.

• September 2025: Chief Architect of the GB10 SoC, Andi Skende, shared its architecture and development process in great detail at the Hot Chips conference.

• October 2025: DGX Spark officially began shipping.

Architecture

2.5D

Like other Grace-Hopper (GH) and Grace-Blackwell (GB) configurations, the GB10 has a CPU and a GPU component. However, unlike its higher-end counterparts, the GH200 and GB200, where the CPU and GPU are housed in separate packages, the GB10 uses a multi-die package. In this design, the CPU and GPU dies are connected via C2C interconnect over an interposer, following a 2.5D integration approach.

Not really the Grace CPU

In my opinion, this was the biggest surprise. Although the CPU is labeled “Grace,” it’s not a cut-down version of the Grace CPU used in GH200, GB200, or the Grace Superchip systems. Instead, it’s a 20-Core ARM CPU IP developed by MediaTek.

While NVIDIA had announced its partnership with MediaTek back in January, I did not expect such a deep collaboration. This isn’t a small integration, it’s a full-fledged co-design effort.

Memory subsystem

The Blackwell iGPU die, as expected, is based on NVIDIA’s Blackwell GPU architecture. It doesn’t have dedicated memory of its own. Instead, it connects to the system’s LPDDR5X DRAM through memory controllers located in the MediaTek CPU die. The C2C interconnect provides around 600 GB/s of aggregate bandwidth, more than enough for the iGPU to access the full system memory bandwidth.

This kind of data flow, where the GPU accesses memory attached to the CPU in a unified fashion, isn’t new. It’s an approach we saw in Grace Hopper presentations, and the DGX Spark simply extends that same design philosophy.

Here are 2 slides from the Grace CPU and Grace Hopper architecture document, where this unified memory model is emphasized.

• The GPU can access LPDDR5X memory over NVLink C2C in a coherent and transparent manner.

• It’s more efficient to have 4× Grace + 4× Hopper nodes rather than 16× Hopper, to reach the 2.5 TB memory footprint required for training.

GB10 Memory Complex

Regarding the memory subsystem itself:

• The memory controllers are part of the MediaTek CPU IP.

• There are 8 LPDDR5X DRAM packages soldered to the board, each with a 32-bit interface, for a total interface width of 256 bits.

• The datasheet lists total memory bandwidth at 273 GB/s, which corresponds to a memory speed of 8.533 Gbps.

  (256 bits × 8.533 Gbps)/8 = 273 GB/s

• In the Hot Chips presentation, the memory clock was specified as 9.4 Gbps (300 GB/s of bandwidth). Suggesting some capability to overclock, perhaps?

Methodology

In his Hot Chips presentation, Andi Skende, GB10’s chief architect, spent considerable time detailing the verification strategies used in developing the chip.

Here are few excerpts from the talk, with my comments in italics:

• We had to do very extensive performance modeling of the GPU IP traffic into MediaTek’s memory controller subsystem to make sure we achieve the best efficiency while also giving the CPU core reasonable latency. It is a challenging problem to balance UMA (Unified Memory Access) architecture where latency and efficiency are competing with each other.

• We followed a very strict IP model when delivering our IP to MediaTek for integration into their SoC, IP such as the C2C and Display. (NVIDIA sent over their IPs to MediaTek. So, MediaTek closed timing, did synthesis, and all backend work for the CPU die?)

• On the verification side, we took a multi-prong approach, we used BFMs (Bus Functional Models) where possible and a hierarchical approach to verification.

• We did co-simulation for very complex features that required end-to-end simulation by bringing the 2 dies together into a co-simulation platform. CPU to GPU coherency is one such feature that requires a lot of verification and co-simulation to make sure you’ve covered all the corner cases, there are no race conditions, no deadlocks, etc.

• Last but not the least we relied on emulation very heavily, to bring together MediaTek’s hardware, NVIDIA hardware, firmware and software, working together pre-silicon. Where we booted the whole OS including running full app frames before we taped-out, and all that enabled us to tape-out and go to production with first silicon, A0 silicon, one tape-out no respins.

Andi Skende, Hot Chips 2025

---

Chiplog Note: Simulation and emulation are pre-silicon verification strategies. Simulation can only be done on smaller portions of the design and aperture of time for which the design can be checked is small, in the order of milliseconds.

In emulation, larger portions of the design are synthesized and run on dedicated hardware like Cadence’s Palladium or Siemens Mentor Graphics’ Veloce. The hardware and licenses for emulation itself cost millions of dollars.

Executing collaborations of this scale and shipping A0 silicon is an admirable achievement, even for a company like NVIDIA with virtually unlimited resources.

Conclusion

As expected, there’s been plenty of discussion in forums and comment sections about whether the DGX Spark justifies its $4,000 price tag, and how it stacks up against the RTX 5090 or an equivalently configured Apple Mac Studio. But regardless of raw performance metrics, this project is a major win for NVIDIA.

Designing this chip entirely in-house would have required a dedicated team and measurably more engineering resources, likely pulling focus away from NVIDIA’s main high-end SoC efforts. Instead, by co-designing with MediaTek, leveraging their CPU and memory controller IPs, and establishing a collaborative workflow that lead to a successful A0 tape-out is a significant feather in NVIDIA’s cap.

NVIDIA can now pursue the DGX Spark product line and build successors to the GB10 with lesser internal overhead.

Share

Support Chiplog

Chiplog is a reader-support publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Right now memberships are 50% off. That’s $37.5/year (~$3/month).

Subscribe

I’ve categorized the sections based on where the memory physically sits relative to the compute/processor die:

1. DRAM on the Board – Memory soldered on the PCB. Common with DDR, GDDR, and sometimes LPDDR.

2. DRAM on Package – Memory-on-Package (MoP) and Package-on-Package (PoP) configurations, used with LPDDR.

3. DRAM on Interposer – Found with HBM.

4. DRAM on Die – Early look at 3D-DRAM. (d-Matrix, the company I work for, had a big announcement on this just last week.)

5. DRAM in Die – Embedded DRAM.

🔒 For Members

• A brief comment on hybrid bonding.

• Deep dive into NVIDIA Grace CPU’s LPDDR5X memory complex

  • The Grace CPU is arguably more interesting than Hopper or Blackwell GPUs.

  • Grace uses special off-catalog DRAM components, and I’ll walk through why that is and what makes them unique.

  • NVIDIA is exercising its special relationship with Micron, and the topology is changing from the Grace-Hopper (GH) super chip to the latest Grace-Blackwell (GB) motherboards.

• A comparison of NVIDIA Hopper versus Blackwell HBM complex.

Chiplog is a reader supported newsletter. Right now memberships are 25% off. That’s $74/year (~$6/month).

Subscribe

Acknowledgement

This article wouldn’t have been possible without the excellent images from TechPowerUp and ServeTheHome. Many thanks to them for their fantastic work.

DRAM on Board (DDR & GDDR)

This type of memory integration is the most common. We see it in laptops, desktops, and servers. In this setup, the DRAM package is either soldered directly onto the PCB or first mounted onto a smaller daughter card (a DIMM module) that plugs into the main PCB through a socket. The processor and memory communicate through traces routed across the motherboard. Here are a few examples.

Example 1: GDDR in NVIDIA GeForce cards

Below are annotated images from three generations of NVIDIA graphics cards. In each case, GDDR memory is soldered directly onto the board. The DRAM devices are width-cascaded, forming a wide memory interface that the GPU can access in parallel.

1. NVIDIA GTX 780 Ti with GDDR5

2. NVIDIA RTX 4080/4090 with GDDR6X

3. NVIDIA RTX 5080/5090 with GDDR7

The GTX 780 Ti is fed by 12 GDDR5 chips, each with a 32-bit data bus. Together, these 12 devices provide the GPU with a 384-bit wide memory interface.

Next, let’s compare the RTX 4080 and RTX 4090.

• The RTX 4080 uses 8 GDDR6X devices, each with a 32-bit data bus, giving the GPU a 256-bit wide memory interface. With each device providing 2 GB of capacity, the card has a total of 16 GB of memory.

• In comparison, the RTX 4090 has 12 GDDR6X devices for a 384-bit interface, with a total capacity of 24 GB.

Finally, the latest generation RTX 5080 and 5090 move to GDDR7. The 5080 is equipped with 8 devices for a 256-bit bus, while the 5090 doubles that with 16 devices, delivering a massive 512-bit bus width.

While it may seem obvious, it’s worth emphasizing that every additional DRAM device comes at a cost. It consumes PCB area, requires dedicated memory controller logic on the processor die, and increases routing complexity on the board.

Eventually, physical limits take over, you either run out of PCB real estate or processor die area to accommodate more controllers. These constraints ultimately define the maximum bus width and the upper bound on memory capacity for a given graphics card.

Example 2: DDR5 in Intel 6700/6500 SuperMicro cards

From the graphics cards examples, we saw that even at the very high end, memory tops out at around 32 GB of DRAM capacity. But 32 GB isn’t much if your application demands more. So how do you scale beyond that?

This is where memory modules come in, a form of integration more common in CPUs. For example, this Micron 128 GB DDR5 RDIMM, priced at roughly $1,000, packs 40 DDR5 DRAM devices (20 soldered on the front and 20 on the back). This dual-sided design is what gives it the name Dual Inline Memory Module or DIMM.

These modules then plug into the system’s motherboard. For instance, a Supermicro X14 server board designed for Intel’s latest 6700-series CPUs supports 8 DIMM slots. Fully populated with 128 GB sticks, the system reaches a total capacity of 1 TB (1024 GB) of ECC protected DRAM.

The scaling method here is different from GPUs. While GPUs expand capacity by width-cascading DRAM devices (wider buses), DIMMs expand via depth-cascading, adding more memory devices behind the same interface width. This approach increases total memory capacity without widening the bus.

The Intel 6700 CPU, for instance, supports 8 DDR5 memory channels, and system capacity can be scaled by populating those channels with different DIMM sizes.

GPUs optimize for bandwidth, CPUs optimize for capacity. The type of integration reflects the workload priorities. Fast, parallel access for GPUs versus large, expandable memory pools for CPUs.

Types of DIMMs

There are several different types of DIMMs to choose from, depending on form factor and functionality.

• Form factor: A standard DIMM measures 133.35 mm × 31.25 mm, whereas a SODIMM (Small Outline DIMM) is more compact at 69.6 mm × 30 mm, making it suitable for laptops and other space-constrained devices.

• Functionality: Different applications impose different requirements. Laptops are cost-sensitive, so unbuffered DIMMs (UDIMMs) are usually sufficient. Servers in data centers, however, demand higher reliability and capacity, so they use ECC-protected RDIMMs (Registered DIMMs). In RDIMMs, a chip called the RCD (Register Clock Driver), located on the DIMM module, buffers and regenerates critical signals, like clocks, before reaching the DRAM components. This improves signal integrity and allows for larger, more stable memory configurations.

Beyond standard offerings, specialized vendors such as Virtium and Smart Modular provide custom DIMM modules. They can even source DRAM components from vendors like Micron or Samsung and build modules tailored to specific needs.

• For example, Virtium manufactures VLP (Very Low Profile) and ULP (Ultra Low Profile) DIMMs to reduce z-height in space-constrained systems. These are not available in Micron’s catalog.

• Smart Modular, on the other hand, offers Liquid Immersion DDR5 modules and ECC “C” SODIMMs, which regenerate and amplify the host clock signal for even stronger signal integrity.

DRAM on Package (LPDDR)

Like GDDR and DDR5, LPDDR memory can be directly soldered onto the PCB or mounted on a LPCAMM module (Low Power Compression Attached Memory Module) and then attached to the motherboard.

However, the more common and increasingly important approach is the style used in Apple’s M-series and A-series SoCs, bringing LPDDR DRAM closer to the processor through advanced packaging.

Memory-on-Package MoP (or On-Package Memory OPM)

In Intel’s Meteor/Lunar Lake and Apple’s M-series chips, LPDDR DRAM and the logic die share the same package substrate. The dies are connected through wires routed within the substrate itself.

Tightly integrating the memory and logic die sacrifices the flexibility of PCB-based DRAM (where modules can be swapped or scaled), but it saves valuable PCB area. That space can then be repurposed, for example, to fit a larger battery or to reduce the number of PCB layers, lowering assembly and cost.

Package-on-Package (PoP)

Chips that go into phones, such as Apple’s A-series processors, Qualcomm Snapdragon, and Samsung Exynos, take integration a step further. Here, the DRAM package is physically stacked on top of the SoC. This PoP design tightens the coupling between compute and memory, minimizing the footprint even further.

Below is a snippet from TechInsights’ A14 Bionic package breakdown, which illustrates this approach in practice:

[A14 Bionic uses] package-on-package (PoP) assembly using TSMC’s integrated fan out (InFO) technology.

The top package contains four DRAM dies configured in two stacks of two dies. The memory is wire bonded to a printed wiring board (PWB) ball grid array (BGA). The BGA is connected to the bottom of the package by through molding vias (TMVs).

DRAM on Interposer (HBM)

With HBM, the integration gets even tighter than LPDDR’s Memory-on-Package. Instead of sitting on the substrate (as in MoP), the compute die and DRAM stacks are placed side by side on an interposer. A setup commonly called 2.5D integration.

The biggest advantage of using an interposer is routing density. Interposers support much finer connections than substrates. The compute die and HBM stack connect through micro-bumps, while substrates rely on larger bumps called C4 (Controlled-Collapse Chip Connections). To put numbers on it, micro-bump pitch is around 40 μm, compared to 400 μm (0.4mm) for C4 bumps, an order of magnitude tighter. That tighter pitch means you can route far more wires across the interface.

This density shows up in bandwidth. HBM3e provides 1024 data lanes per stack, and HBM4 will double that to 2048. For example, NVIDIA’s Blackwell GPU integrates eight stacks of HBM3e, giving a combined bus width of 8,192 lanes. Each stack can deliver roughly 1 TB/s of bandwidth, which adds up to a staggering ~8 TB/s total.

DRAM on Die (3D DRAM)

The next step beyond HBM’s 2.5D architecture is 3D DRAM. At d-Matrix (where I work), we had an exciting announcement about this.

In this approach, the DRAM is attached directly to the logic die like the interposer. The image below (from our public release) illustrates this integration.

As some of the coverage around our announcement noted, 3D DRAM holds tremendous promise for pushing bandwidth and efficiency even further.

Silicon Angle

According to Bhoja [CTO], by combining 3D DRAM with its specialized interconnects, Raptor will be able to smash through the memory wall and unlock significant gains in terms of AI performance and cost-efficiency. He said the company is targeting an ambitious 10-times improvement in memory bandwidth and 10-times better energy efficiency with Raptor when running inference workloads, compared with existing HBM4 memory technology.

“These are not incremental gains — they are step-function improvements that redefine what’s possible for inference at scale,” Bhoja said.

Block And Files

“Our next-generation architecture, Raptor, will incorporate 3DIMC into its design – benefiting from what we and our customers learn from testing on Pavehawk. By stacking memory vertically and integrating tightly with compute chiplets, Raptor promises to break through the memory wall and unlock entirely new levels of performance and TCO.”

I’ll leave it at that for now. For the latest updates, keep an eye on d-Matrix or follow me on LinkedIn and X.

DRAM in Die (eDRAM)

The last chip I worked on with embedded DRAM (eDRAM) was back in the late 2000s and early 2010s, when IBM was building chips on 32nm and larger nodes.

In IBM’s eDRAM, the DRAM cells are part of the logic die itself (similar to how SRAM is integrated) and are built using deep trench capacitors. That means the DRAM cells are fabricated in the same process node as the rest of the chip.

IBM’s Power7 and Power8 processors used eDRAM for their L3 cache

You might also remember it from Microsoft’s Xbox 360, which used eDRAM for graphics. Intel’s Haswell generation also advertised eDRAM, though in that case it was implemented differently, the eDRAM was a separate chip integrated with the CPU through a Multi-Chip Module (MCM) approach.

Share

In a nutshell

This diagram summarizes the types of DRAM integrations we’ve discussed this far.

🔒 Hybrid bonding, Grace, Hopper, Blackwell

In the members-only section, we’ll dive into:

• A brief comment on hybrid bonding.

• A deep dive into NVIDIA Grace CPU’s LPDDR5X memory complex. How NVIDIA is exercising its special relationship with Micron, and how the motherboards are evolving from the Grace-Hopper (GH) to Grace-Blackwell (GB) systems.

• A comparison of NVIDIA Hopper vs. Blackwell HBM complexes.

Chiplog is a member supported newsletter. Right now memberships are 25% off. That’s $74/year(~$6/month).

Subscribe

Read more

1. Physical Structure - We tackled this last time, breaking down how DRAM is organized at the cell, bank, and die levels.

2. Timing Parameters - Reading and Writing to DRAM is like a carefully choreographed dance. Timing parameters dictate the rules that have to be followed while accessing DRAMs and have a big effect on the performance.

3. Initialization and Calibration - Over its lifetime, a DRAM device will be subject to variations in Process, Voltage and Temperatures (PVT). Memory controllers use special circuits and algorithms to periodically tune the interface so it runs reliably in all conditions.

This post focuses on Dimension #2: DRAM timing parameters and their effect on performance. We’ll approach the topic from first principles, because this foundation is essential when deciding,

• For ML architectures, how best to store and access weights, activations, embedding tables, KV caches, and other critical data structures.

• For cyber security and networking architectures, how to store hash tables.

• For compute offload, how to design algorithms which efficiently read from memory and maximize performance.

Specifically, we’ll cover:

• The difference between bandwidth and throughput.

• A systematic exploration of the most important DRAM timing parameters, and how they shape the effective bandwidth you can extract from modern memories like HBM, LPDDR, and GDDR.

• Practical insights into what makes a good memory access pattern versus a bad one.

• Help you develop an intuition, so you can design an efficient data storage architecture for your application.

---

🔒 For members:

• We’ll take a look at two more important timing parameters - tFAW and tWTR, which play a crucial role in performance.

• We’ll take a brief look at how memory controllers work, and the structures at your disposal to help maximize throughput.

• How companies like Meta evaluate and study the performance of their DRAM architecture.

• Tools and strategies that are used to study performance.

• Common mistakes made during performance estimation.

Chiplog is a member supported newsletter. Right now memberships are 25% off, that’s $74/year (~$6/month).

Subscribe now

---

Bandwidth vs Throughput

Bandwidth is the theoretical peak performance that a memory can achieve. For example, in LPDDR5 the maximum bandwidth of a memory channel is ~102Gbps.

Each memory channel has 16 data lanes1, with each lane capable of supporting 6400Mbps. So 6400*16 = ~102Gbps. This is the raw bandwidth.

In reality, this bandwidth is not attainable. The actual data rate that can be achieved is called throughput. To understand why the attainable throughput is less than the maximum bandwidth, we need to understand DRAM timing parameters.

What are timing parameters

Due to the physics governing DRAM devices, certain timing constraints or delays have to be observed while performing operations on the memory. The effects of timing parameters on memory bandwidth is best understood in the context of specific memory operations. We’ll next look at the Refresh command and a Read operation.

REFRESH command

At the lowest level, a DRAM cell is essentially a capacitor that holds charge with a transistor acting as a switch. Since the capacitor discharges over time, it has to be Refreshed periodically to ensure data is not lost.

There are 2 main timing parameters related to the refresh operation.

1. tREFI (Refresh Interval): This parameter specifies how often the memory has to be refreshed so that data is not lost.

2. tRFC (Refresh Cycle Time): This parameter specifies for how much time the DRAM is unavailable once it enters Refresh.

For example, in case of LPDDR5,

• The memory needs to be refreshed every 3.9us (tREFI).

• Once the memory enters refresh it is unavailable for ~200ns (tRFC).

So, if the memory is unavailable for ~200ns every 3.9us (i.e., 3900ns), then we lose nearly 5% of the bandwidth JUST to Refresh.

200/(3900+200) = 4.8%

Simple READ operation

Going up one level, DRAM cells are arranged in a grid of Rows and Columns. One such grid is called a memory bank.

From the outside world, it may look like reading from a memory bank is an atomic operation. But, in reality it is a multi-step process between the processor (memory controller, to be precise) and the DRAM. The 3 steps typically involved in a READ operation are:

1. Activate (ACT): A portion of the read address identifies which row within the bank the data is stored in. The processor first sends an ACT command to the DRAM to transfer the data from the memory cells into the bank’s Sense Amps.

2. Column-Address-Strobe (CAS):

   1. The activation process takes some time, so only after a delay of tRCD (Row-to-Column Delay) the column address can be sent to the DRAM.

   2. Then, after a latency of tCL (CAS Latency), the data is returned.

3. Precharge(PRE):

   1. Before accessing another row within the same bank, the currently open row has to be deactivated. This process is called Precharging.

   2. Once the current row is deactivated, it has to remain idle for tRP time before another row can be activated.

   3. Also, care has to be taken to ensure that the time between two ACT commands is at least tRC. This is the row cycle time.

These timing parameters are illustrated below. As you can see, a basic read operation involves a number of delays and latencies.

In the context of LPDDR5 running at 6400Mbps, tRCD is 18ns and tCL is ~22ns and after all that setup, it takes just 2.5ns to read the data back from that column. So the overhead is substantial.

In the upcoming section we’ll see how hiding these latencies is crucial to extract maximum performance out of the DRAM.

Hiding latencies to maximize throughput

Let’s stick with READ operations and examine 4 different back-to-back read scenarios, and how the performance varies widely for each of them.

Page hits vs page miss

• Scenario 1: Back-to-back reads to the same bank but different rows.

• Scenario 2: Back-to-back reads to the same bank and same row.

In scenario 1, since the relevant data is located in 2 different rows, after the first row is accessed it has to be deactivated, and the second row has to go through the activation process all over again. This scenario is also called a Page Miss.

But in scenario 2, both the reads have the same row address, only the column address is different. Here the row activation process only happens once. After which, two CAS (column-address-strobes) are sent from the processor and the data is read back. This scenario leads to what is called a Page Hit.

In scenario 2, the two CAS commands are separated by a timing parameter called tCCD (col-to-col delay)

As evident from the figure below, in case of a page miss it takes a lot longer to fetch the data from the DRAM.

Bank Groups vs Banks

While page hits are important, accessing the same bank in back-to-back reads is not good. We can do better to hide some of these latencies, especially the tRCD.

Fortunately for us, the DRAM die is made up of not just one bank but a collection of banks, which are arranged in Bank Groups. Next, we'll look at how bank groups help us improve performance.

Each bank has a set of Sense Amps. They are like different books, you are allowed keep a page (i.e., row) open in each Bank. So, when the memory controller sees two reads to different banks,

• It issues an ACTIVATE to the first bank, and while waiting for tRCD (Row-to-Col-Delay) it issues a second ACTIVATE to the second bank and prepares it for the second read. By doing back-to-back reads to different banks, you have allowed the memory controller the opportunity to hide the tRCD latency for the second bank.

• Similarly, once the controller issues the CAS (column address strobe) for the first read, it issues the second CAS to the different bank while waiting for tCL (Read Latency).

• This process of doing things in parallel reduces the total latency to finish both reads.

This principle of hiding latencies is crucial to extract as much performance as possible out of the DRAM, and fundamentally speaking, the data has to first be stored in a suitable fashion to help the controller hide the latencies.

But, banks and bank groups are not equal. Let’s examine the next two scenarios

• Scenario 3: Back-to-back reads to different banks in the same bank group

• Scenario 4: Back-to-back reads to different banks in different bank groups

In scenario 3, since the back-to-back reads go to different banks of the same bank group, the two reads have to be separated by the tCCD_L (Column-to-column-Long) timing parameter. (Which is 4 clock cycles in LPDDR5).

In scenario 4, the back-to-back reads go to different bank groups. In this case, the CAS (column-addres-strobe) commands only need to be separated by tCCD_S (Col-to-col-Short). So scenario 4 ends up being the best case for DRAM performance.

For LPDDR5 operating at 6400Mbps, tCCD_S is 2 clock cycles (2.5ns). That happens to be exactly the amount of time needed to read out the data from a single column. This detail is critical — if you want to keep the data bus fully occupied and extract maximum performance from the memory, two conditions have to be met.

1. Successive reads should target different bank groups.

2. A high portion of the accesses should be page hits.

With this setup, you can first issue a sequence of tRRD_S (row activation) commands to open rows across multiple bank groups. Once those pages are active, you can issue a burst of tCCD_S commands to stream data continuously from the open pages.

In this best-case scenario, the memory bus stays busy almost all the time, allowing you to capture roughly 93% of the theoretical bandwidth.

Summary so far …

• Timing parameters are a necessary evil. But it is possible to work around some of them by hiding the latencies.

• For the best performance, back-to-back reads should be issued to different Bank Groups. This is when the latencies introduced by the timing parameters are the least.

• The performance is worst when back-to-back reads are issued to different rows of the same bank.

• A basic understanding of timing parameters is necessary to develop an intuition of how good or bad the access pattern is for your workload. Take the time to study the illustrations above, there is a lot of detail in them.

• Here’s a convenient cheatsheet of all the timing parameters we discussed above.

Share

🔒 Meta, memory controllers, and estimating performance

In the members-only section,

• We’ll take a look at two more timing parameters - tFAW and tWTR, which play a crucial role in performance.

• We’ll take a brief look at how memory controllers work, and the structures at your disposal to help maximize throughput.

• How companies like Meta and Apple evaluate and study the performance of their DRAM architecture.

• Tools and strategies that are used to study performance.

• Common mistakes made during performance estimation.

Chiplog is a member supported newsletter. Right now memberships are 25% off, that’s $74/year(~$6/month).

Subscribe

Read more

So, in order to design a robust memory subsystem, make these memories run reliably, and squeeze out every bit of performance out of them, you need to understand 3 key dimensions:

1. Physical Structure - Understanding the physical structure is required to efficiently store and retrieve data from the memory. A good data storage architecture is critical to performance.

2. Timing Parameters - Reading and Writing to DRAM is like a carefully choreographed dance. We are bound by the physics of these devices. Timing parameters dictate the rules that have to be followed while accessing DRAM memories, i.e., how fast you can access the memory, how often they have to be refreshed to retain data, and such.

3. Initialization and Calibration - Over its lifetime, a DRAM device will be subject to variations in Process, Voltage and Temperatures (PVT). Memory controllers use special circuits and algorithms to periodically tune the interface so it runs reliably in all conditions.

In this post, we will focus on Dimension #1: the physical structure of DRAM. We’ll start from a single DRAM cell and work our way up to a complete package, while discussing ranks, banks, channels, page hits/miss and other terminology.

---

🔒 For members:

• We’ll take a side-by-side look at HBM3E and LPDDR5X — two of the most popular DRAM types that are shipping in volume right now

• Explore their structural differences

• Understand why HBM’s design delivers such staggering bandwidth compared to LPDDR5

Chiplog is a member supported newsletter. Right now memberships are 25% off. That’s $74/year (~$6/month).

Subscribe now

---

Structure of a DRAM

Cell

At the lowest level, a memory cell stores 1-bit of information. This cell is essentially a capacitor that holds the charge and a transistor acting as a switch. Since the capacitor discharges over time, the information eventually fades unless the capacitor is periodically REFRESHed. This is where the D in DRAM comes from - it refers to Dynamic as opposed to the Static in SRAM.

Rows, columns, banks

When you zoom out one level, you will see these memory cells arranged in a grid of Rows and Columns. One such grid is called a Bank.

In order to read or write to a specific location in the memory, an address has to be specified. In this address the Bank, Row and Column numbers are encoded.

A memory bank is like a book, where

• BANK address indicates which book you want to read or write to

• ROW address is the page number within that book

• COLUMN address is the line number within that page

There are 3 distinct phases in accessing a memory:

• ACTIVATE: First a Row Address Decoder ACTIVATEs the row number specified in the address, and brings the data into a structure called the Sense Amp. This is equivalent to opening the desired page of the book.

• COLUMN STROBE: Then a Column Address Decoder extracts data from a specified column and streams it out. This is the same as locating and reading the line in the open page.

• PRECHARGE: Once the access is complete, the Row has to be closed and data in the Sense Amps is returned back into the Row cells. This closing of the row is called PRECHARGE. This is the same as closing the book.

The GIF below shows a Memory READ operation in action.

### SIDE NOTE: Page hits and page miss

You may have heard of "Page Hits" and that it 
improves performance. This is indeed correct. 

Continuing with our analogy of the DRAM bank 
being a book ...

Page hit scenario:
If all the data you are trying to retrieve 
are present on the same page, then
+ You can open to the page (ROW address) once
+ Read all the lines (COL addresses) 
+ Close the page
 
Page miss scenario:
If the data you are trying to access are 
located in different pages, then you will have to
+ Open the first page
+ Read some lines 
+ Close that page
+ Open the next page
+ Read some more lines
+ Close that page
+ ... and so on

Needless to say, there is a significant penalty 
to page misses since there are many more ACTIVATEs 
and PRECHARGEs to be performed.

Memory die

A memory die is one physical wafer and is made up of a collection of banks. Memory vendors such as Micron, Samsung and SK Hynix make memory dies of various densities, depending upon the number of rows, columns and banks.

The table below shows the various densities LPDDR5 memories dies are made in. Notice how larger densities just have more number of Rows.

Memory channels

The interface to the Memory is like a pipe … a bundle of wires. This bundle has a name — Command-Address (CA) and Data (DQ) bus.

On one end of this pipe you have the brain (Processor/ASIC with the memory controller — usually called the logic die). At the other end is some arrangement of Memory dies. This pipe with a collection of memory dies at the end of it is called a “channel”, and the image below shows 3 different arrangements of memory dies.

Memory Package

If the memory is a stand-alone unit, like in LPDDR5 or DDR5, then it is packaged so that it can be soldered on the board, or within the main SoC package (Memory-on-Package). This package is made up of multiple-channels. Each channel is connected to a different memory controller and works independently.

Performance

I’ll wrap up the free portion of this article with a quick discussion on performance.

When deciding the memory architecture for an SoC and deciding which memory type to use (HBM, LPDDR, DDR or GDDR), you really only have 3 main levers to pull with regard to performance. (Of course, cost, power, and availability are important factors that guide this decision too.)

Using the analogy of the compute die being connected to the memory through a pipe, we can ask 3 questions:

1. How many pipes? (Number of memory channels)

If each memory channel delivers 10GB/s of bandwidth per channel, then you have to first decide how many channels are required for your SoC’s application. For instance, if you require 40GB/s of total bandwidth, then you will need four channels.

Each channel needs its own memory controller on the logic/compute die, which eats up both die area and beachfront. Beachfront is the Chip edge real estate where I/O interfaces live. A very precious resource, and something we’ll dive into in the next article (another reason to subscribe if you haven’t already!).

2. Size of each pipe? (Memory interface width)

The memory interface width dictates the number of data (DQ) bits that can be transmitted in parallel. The wider the DQ bus, the greater the bandwidth offered by the memory since more bits can be transported between the compute die and the memory in one clock cycle.

3. How fast does data flow through these pipes? (Memory frequency)

DRAMs can be operated at various frequencies, for example LPDDR5 can run at a maximum 6400Mbps, but also at 3200Mbps and 5400Mbps. But cranking the dial to the max isn’t always the feasible.

Higher frequencies draw more power, are harder to run reliably, and can introduce signal integrity headaches. That means your system design and cooling requirements also get more complex.

To balance performance with thermals and stability, techniques like Dynamic Frequency Scaling (DFS) are used to keep the TDP in check. But this complicates software design.

Page Hits & Data Architecture

Any discussion on performance is incomplete without talking about Data Storage Architecture.

Well-structured storage and retrieval patterns can significantly improve how well the memory bandwidth is utilized. Even with the best hardware, the wrong access patterns can leave much of the memory bandwidth untapped.

🔒 Comparing HBM3E versus LPDDR5X

The rest of this post is for members.

• We’ll take a side-by-side look at HBM3E and LPDDR5X — two of the most popular DRAM types that are shipping in volume right now

• Explore their structural differences

• Understand why HBM’s design delivers such staggering bandwidth compared to LPDDR5

Right now memberships are 25% off. That’s $74/year (~$6/month). With this membership, you’ll be supporting chiplog.io.

Subscribe

Read more

• I’ll first loosely define what an accelerator is and the scale on which the effectiveness of devices are measured.

• Then examine three real-world examples of hardware acceleration and the design patterns that power them.

Power, Performance and Area (PPA)

Let’s start with the proverbial PPA chart. In the world of hardware design, there's a fundamental trade-off that engineers constantly wrestle with:

• Power: How much energy does it consume to operate?

• Performance: How fast can it get the job done?

• Area/Cost: How much silicon real estate (and therefore money) does it take to fabricate? and how hard is it to make the system or is it even feasible?

Whether you're building a general-purpose CPU, a GPU, or a hyper-specialized ASIC, your design will ultimately be judged on this scale. The “true north” for any chip designer is achieving maximum performance with minimal power and area.

Why build accelerators?

One pattern that emerges over and over again in both human behavior and engineering is, when a task is repeated over-and-over again, someone will eventually build a tool to make it faster. Just as construction workers don’t rely on hand saws when they can use power tools, software engineers and hardware designers don’t keep performing the same slow computations in general-purpose environments if they can offload them to something more efficient.

Also, the term “accelerator” is not limited to hardware. As an obsessive gamer in the 90s and 00s, one software accelerator that caught our imagination was the Fast Inverse Square Root.

In 3D games, like Quake, it was required to compute 1/√x repeatedly for each point to determine lighting and reflections on surfaces. On early Intel processors, doing this calculation through repeated division operations was very slow. John Carmack famously replaced this code with the Fast Inverse Square Root, which was discovered by Quake III Arena fans when id software made the game open-source. This is the Gamer’s Accelerator.

3 Hardware Design Patterns that Accelerate

1. High-Speed Regular Expression Pattern Matching

In cybersecurity, devices known as networks firewalls perform deep packet inspection (DPI) on packets flowing over the network to detect malicious payloads and sensitive data leaks such as SSNs and credit card numbers.

The set of signatures to be identified are turned into regular expressions using deterministic and non-deterministic finite automata (DFA and NFA). For instance, suppose you are searching for patterns:

• abb

• aabb

• baaab

These patterns can be combined into the regular expression (a|b)*(abb|a+b) and when represented as a finite automata, it becomes the following state machine.

The firewall then maintains state for each individual user stream, using hash tables, and takes action if there is a match for any of these signatures. In a high throughput environment, such as a data center, there will be millions of concurrent traffic streams to manage.

Performing these hash calculations and pattern matches on every single packet is expensive. Purely software firewalls running on general-purpose processors will not scale beyond a certain bandwidth, say 20Gbps. Instead, an alternative is to offload such frequent computations to a FPGA, which can perform the same computation in a fraction of the time than software would take.1

Implementing such acceleration is now easily possible with services such as AWS F2 FPGA Instance. The system could be architected such that a pool of servers offload their most expensive computation to a single FPGA instance.

2. Data-Center Scale Cybersecurity

Let’s stick with cybersecurity and firewalls.

In places like a coffee shop or a small office, where there maybe 20 people connected to the WiFi network at a given time, the entire firewall software stack can run on a single processor. This is evident from the portfolio of hardware systems sold by Fortinet and Palo Alto Networks. You will see lower end systems with just one processor.

But at the scale of a cloud data center or ISP, things get more complex. You're handling terabits per second of encrypted HTTPS traffic. These high-end firewall systems have software running on multiple processors in a distributed fashion.

For this, an ASIC (Application Specific IC) is typically required to ingest the sheer volume of traffic. They analyze, pre-classify, split, and schedule massive network flows into smaller chunks for the downstream CPUs to process. 2

The image below is from the Palo Alto Networks website. It shows how a high-end system has an ASIC in the front-end to ingest traffic and a FPGA at the back-end to offload expensive compute off the processors running the firewall software.

So, when you scale up an application from one single processor to a collection of them, some specialized chips capable of performing specific computation is required to make that scale happen.

3. Accelerating Matrix Arithmetic with Systolic Arrays

The last two examples achieved acceleration by converting portions of the software into hardware circuits. This one accelerates by creating Domain Specific Architectures (DSA) rather than using general-purpose processors.

The conventional Von Neumann architecture, in which instructions are fetched and executed sequentially, combined with the fact that the processing pipeline is separated from memory by long paths and cache stages, can limit the amount of performance available from general purpose processors.

The primary benefit of domain-specific architectures, as you will see next, is that it unlocks the power of concurrency. However, parallelism in computation will only result in a moderate speedup. These designs also use custom memory architectures, such as co-locating small memories with the processing element, to achieve an order of magnitude increase in performance.

First published in 1978 by H.T.Kung, here below is the systolic array architecture. From a philosophical sense, one of the most elegant designs in computer architecture … it’s poetic. In these systems, data pulses through a grid of processing elements (PEs), each of which perform a small computation and passes results to the next.

This architecture is also the foundation of Google’s Tensor Processing Unit (TPU).

As reading a large SRAM uses much more power than arithmetic, the matrix unit uses systolic execution to save energy by reducing reads and writes of the Unified Buffer.

It relies on data from different directions arriving at cells in an array at regular intervals where they are combined. Figure 4 shows that data flows in from the left, and the weights are loaded from the top.

A given 256-element multiply-accumulate operation moves through the matrix as a diagonal wavefront. The weights are preloaded, and take effect with the advancing wave alongside the first data of a new block. Control and data are pipelined to give the illusion that the 256 inputs are read at once, and that they instantly update one location of each of 256 accumulators.

Jouppi et al., "In-Datacenter Performance Analysis of a Tensor Processing Unit", ISCA 2017

The beauty and poetry comes from the flexibility to create different geometries and control how data flows within the array mesh, and construct designs that are tailored to different problems and applications.

Final Thoughts

Those were 3 examples of how FPGAs and ASIC can be used to accelerate workloads. The main takeaways are:

1. Software design patterns some times work really well when implemented in hardware.

2. Domain specific architectures with application specific computation and memory structures can exploit concurrency and speed-up execution by orders of magnitude when compared to general-purposes CPUs or even GPUs.

That’s it for this edition of Chiplog. If you enjoyed this article and would like to support my work here at chiplog.io as well as systemverilog.io, please do consider becoming a member today.

Subscribe now

This article offers a 3-step guide on how to prepare and apply for your first, or next, job in hardware engineering.

🔒 For members, I’ve also included an extensive spreadsheet of 100 companies in hardware engineering to kickstart your search.

Step 1: Understanding the available opportunities

The journey of a chip or a hardware system, from conception to finished product, involves a number of sub-specializations in hardware engineering.

By accepting a job offer, you’re not just picking the company but also which specialization to focus your career on. So, the first step is to get a good understanding of the lay of the land.

While doing my graduate studies, I fell into this trap of applying only for jobs related to the subjects I had studied, because I felt these were the only interviews I could crack.

In hindsight, there were several opportunities in the neighborhood of my expertise that were worth considering. So, understand the lay of the land and confidently cast a wide net.

Step 2: Applying for the job

Once you have a grasp of the opportunities out there, the next step is figuring out how to reach them effectively.

Tailor your resume

• A single resume is insufficient. Create multiple versions of your resume tailored to different roles.

• Depending on the job description and what that company does, minor tweaks to your resume to highlight pertinent skills can significantly increase your chances of getting that call.

Referrals work

• Referrals are better than a blind application to a job listing. They often get your resume seen faster and by the right people.

• Leverage your network — friends, alumni, family, and acquaintances. Let people know you’re actively looking.

Cold emailing is okay

• Don’t hesitate to reach out to people on LinkedIn. A cold email is totally acceptable if it’s polite, direct, and genuine.

• The worst they can do is not reply. In the best-case, you get a referral or even an interview.

Step 3: Preparing for the interview

The mindset

• Your goal should be — if you get the interview, you should have a near 100% chance of converting it into an offer.

• You leave nothing on the table. The only reason why you should walk away without an offer is because they were just exploring to see if you would work and there was a genuine mismatch.

Know your work

• Almost always, the first question will be about your past experience. It sounds obvious, but the ability to clearly explain your past work is the first step towards success in your interview. This is where you make that first impression.

• You should rehearse your answers so that they are clear and concise. You’ll be anxious as it is, so don’t try to look for the right words during the interview.

I’ve made this mistake myself. I was once asked about a patent I had filed 8 years earlier. I hadn’t revisited the work beforehand and, in that moment, I blanked. It made my look unprepared and maybe even dishonest.

It is critical to revisit and rehearse your own experience before the interview.

Understand the surrounding 20%

• As a principle, you should strive to understand not just your own work but its surrounding 20% as well. While it is not practical to know everything, this is a good general approach to any problem or project that’s assigned to you.

• This extra 20% would help you understand the history behind this particular problem, studying other comparable designs, and what has and has not worked. This depth shows that you’re not just a doer but a thinker as well.

Standing out as a new college grad

As a new grad, your resume might look like a hundred others. So how do you stand out? How do you make a good first impression during your interview?

• Personality and communication skills: Personality is something that can be worked on. Things like how you speak and present yourself matter. Interviews are not just about your technical chops.

• Stay curious and well-read:

  • When you go through a panel of interviews, for a portion of the interview you can rely on your past work/project experience.

  • But, there will be technical questions which you haven’t directly worked on. Being well-read (i.e., knowing what companies in your industry are doing and the top papers published) will help you come up with a hypothesis and give a good thoughtful answer (even if its incorrect).

  • This also shows your willingness to learn and become good at your craft.

What interviewers are looking for

As someone who interviews candidates and helps build teams, I can tell you that my evaluation boils down to three things:

• Personality: Can I imagine working with this person for the next 3,5 or 8 years? Are they respectful, open, and articulate?

• Technical Aptitude: This isn’t about IQ. It’s about the quality of your past work, how well you know your industry, and can you apply your knowledge creatively to problems you haven’t exactly seen before.

• Willingness and Ambition: How much do you care about the quality of your work. How important is it for you to get better at your craft?

List of 100 companies

That’s it for this edition. To kickstart your job search, I’ve prepared a spreadsheet with 100 semiconductor companies based in North America.

This section is a token of thanks to members. You can support Chiplog through this link. There is purchasing price parity and student discounts. Please send an email to members@chiplog.io.

Subscribe now

Read more

This historic issue has been extensively analyzed and reported on. Here’s a brief summary:

• In 1994, a math professor discovered anomalies in calculations with Intel Pentium CPUs.

• Intel attempted to brush it under the rug, claiming it isn’t a big deal.

• However, the whole situation blew up, resulting in a recall that cost Intel ~$500 million.

The problem

To understand how a problem like this goes undetected before manufacturing, we must understand how chips are tested and verified before fabrication.

Pre-silicon verification (i.e., testing of chip designs before they are sent for fabrication) is primarily done through a method called simulation; this was true in the 90s and even today. This method involves writing tests that stimulate the design with all possible combinations of inputs and then observing if it works as expected.

Companies such as Apple, Intel, AMD, and Nvidia, spend considerable effort on simulating chip designs before tape-out. The memory, compute, and licenses for special tools in themselves cost several million dollars, and a fleet of machines regress through massive test suites 24x7x365 in the hopes of uncovering any design flaws.

But, even with such a rigorous process, bugs such as the FDIV have shown us that this method of testing is insufficient.

The issue is that simulation is inherently an incomplete process. It's analogous to using brute force to solve an issue.

This strategy works well for many designs, but for others, such as a Floating Point Unit (FPU), you will never cover all input combinations in a single lifetime.

To be fair, there is a reason why simulation is the primary method used in the industry for chip verification.

+ There is an ecosystem of mature industry tools and methodologies that have been used to successfully tape-out chips over the past decades.

+ During pre-silicon verification, chip designs are often decomposed into smaller modules, for which simulation generally works well.

The solution: Verification using Formal methods

Consider this equation,

There are two ways to verify if it is correct.

1. Brute force: If you take a simulation-like approach, you might try 10,000 random values for n. If it works, you will then have to optimistically conclude that the equation is right. But, if we’re being honest, you haven’t exhaustively proven that the equation is indeed correct.

2. Formal proof: You might recall from high school math that mathematical induction is the right way to exhaustively prove this equation.

   

The main takeaway from this section is that,

Simulation is not the only method to check if a circuit is doing what it is supposed to do. There is another approach called Formal Verification, which is capable of exhaustively proving that a design works.

The issue with Formal Verification

However, Formal isn’t without its own issues.

• Consider mathematical induction from the previous section — we conveniently used it to prove the equation, but how does it actually work? How can you simply assume something to be true for n=k and then use that assumption to construct a proof for n=k+1?

• Another example, consider Pythagoras’ or Euclid’s theorem. These problems use completely different mathematics and reasoning techniques to arrive at the proof.

The core issue with Formal methods is that understanding the mathematical basis for Formal proofs, which in case of hardware verification are LTL, CTL, Model Checking and Theorem Proving, is quite challenging, and coming up with fresh proof approaches is even more difficult.

Formal Verification has only recently (in the last ten years) broken away from being purely in the realm of Math and Computer Science PhDs and entered the toolkit of the average engineer.

We finally have a collection of mature tools, methodologies and proof techniques which can be learnt and applied within the constraints of aggressive tape-out schedules.

In the following section, we will look at how Intel contributed to this effort.

Formal Verification at Intel

Evolution

After the fallout from the FDIV problem in 1995, Intel knew it needed to do something different. At the time, there were no established alternatives to simulation. But, Intel correctly identified that the answer lies in the field of, what I will loosely call, Automated Reasoning.

The idea behind automated reasoning is to have a computer program prove mathematical theorems or, given a formal model of a system (such as a state transition diagram), have a program tell you if a certain equations are true in that system. Automated theorem proving first saw success in the 1950s. But further research in the field was largely motivated by the desire to improve solvers/proof assistants as well as software/systems reliability (See Edgar Dijkstra’s essay On Reliabilty of Programs). It was only in the 1980s when researchers began to explore the effectiveness of this approach in hardware verification.

Fast forward back to 1995, there were really three strategies worth exploring for hardware verification — Model Checking, SAT solvers and theorem provers. Ed Clarke, the co-inventor of Model Checking, said this in his 2007 Turing Award Lecture.

When Intel made the announcement that there was a real problem with the Pentium 1 chip, a former CMU student who had taken one of my classes, Manpreet Khaira (then a Principal at Intel), called me up and said “send me one of your students”.

I sent him Xudong Zhao, who was my best student at the time. Xudong spent 3-4 months at Intel and was able to show that his PhD thesis topic, Word Level Model Checking, could indeed have found the Pentium error and moreover he was able to show that the fix Intel had come up with did indeed fix the problem.

Ed Clarke, 2007 Turing Award Lecture

Subsequently, Intel committed time and money in developing Formal Verification. This chart summarizes Intel's evolution in this field.

People

When I scanned through the research published between 1995 and 2010, I noticed that practically every researcher from around the world who worked in the field of Model Checking or Automated Theorem Proving did a stint at Intel. By bringing these experts into the same cauldron new Formal techniques and methods were published at a brisk speed. (Calling it a cauldron seems appropriate given the heat Intel was experiencing at the time).

Techniques

Working with Intel gave these researchers the opportunity to apply their ideas to a real-world problem and, more importantly, iterate on these ideas until they matured and were robust enough for an industry setting.

For example, vanilla Model Checking has been known to have capacity issues. It only allows the user to reason with small designs, nowhere sufficient for something like the Intel Pentium processor. Several creative abstractions had to be constructed to boost the capacity of Model Checkers. The best idea to emerge from these academic collaborations was Symbolic Trajectory Evaluation (STE), which was the brainchild of Carl H. Seger.

But, even a venerable technique such as STE is only as good as the operator who uses it. Limor Fix and few others published seminal work on answering the question “Are we done, yet?”, that is, determining whether the Formal analysis of a design is indeed complete and there are no open holes.

Even here, there was a significant degree of engagement with academia, as seen in the papers below.

Tools

In order to Formally verify a chip design, which is typically represented as RTL, it has to first be synthesized into a suitable form, such as a Binary Decision Diagram (BDD). In 1993, while at the University of British Columbia, Carl Seger developed a tool called VOSS. After joining Intel in 1995, he collaborated with others like Tom Melham to developed its successor called Forte. It was designed specifically for industrial-scale circuits, with a unified interface to a BDD, a Model Checker, a SAT solver and a Theorem Prover [3, 6]. Making it seemless to verify a circuit by applying various techniques, such as STE.

Along with a tool like Forte, you require Formal semantics to write properties and ask questions about the design, such as, “if event-A occurs, then does event-B occur 3 clock cycles later?” For this they created a Formal Specification language called ForSpec.

ASIC/SoC engineers will be delighted to learn that Intel donated ForSpec to Accellera and major parts of it were adopted during the creation of the SystemVerilog Assertions language (SVA). SVA is the current industry standard using which Formal Verification is performed.

Methodology

In order to perform effective Formal Verification and produce high quality results, apart from tools and techniques, the Formal team at Intel also developed a robust methodology, spelling out how these tools should be applied. This was critical in democratizing Formal Verification within Intel and ensuring its adoption across projects in the company.

Here’s a snippet from “Making Formal Property Verification Mainstream at Intel” by M Achutha KiranKumar, et. al.

A 2-week dedicated FV training was conducted for all FV champions and interested designers. The training catered to meet the expectations of varied expertise level users. The training included topics such as FV basics, FV applications for specific verification problems, and working demos. The identified FV champions were given additional trainings on tackling convergence issues and on more complex areas. The response seen in trainings was very encouraging, with almost 85 engineers becoming formally literate. A round-the-clock formal assistance network from various resources was available after the training.

Conclusion

In some ways, the FDIV bug served as a blessing in disguise for the semiconductor industry. Intel helped bring an esoteric field from academic research into mainstream chip development. Today, Formal Verification has grow into a thriving micro-industry with the major three CAD companies (Cadence, Synopsys and Siemens) selling advanced tools akin to Forte and providing training as well. Now every chip company considers Formal Verification a mandatory part of their chip development process.

As a cherry on top, more recently, engineers from Intel’s FVCTO (Formal Verification Central Tech Office) which houses some of the best Formal Verification engineers, published the definitive book on Applied Formal Verification upon which engineers like me have come to rely - Formal Verification: An Essential Toolkit for Modern VLSI Design, Erik Seligman et, al.

If you want to get started with Formal Verification, I recommend these three in-depth tutorials I’ve written on systemverilog.io.

1. SystemVerilog Assertions Basics

2. A Gentle Introduction to Formal Verification

3. A Blueprint for Formal Verification

References

Many of the key references are included as images in earlier sections. This reference list offers a broader selection of interesting papers, along with brief commentary on a few. It’s also a small gesture of appreciation for the support from my patrons.

Read more

Distractions, in general, are an effective antidote to a stressful day at work or a hectic weekend with the family. However, not all types of distractions are equal.

Passive distractions such as Instagram, YouTube and Netflix are depleting. But there’s something about active distractions, like playing an instrument or cooking, they are rejuvenating.

Side projects are an active distraction. They transport you mentally and help you breakaway from your daily anxieties. They quench a certain thirst that you didn’t know needed quenching.

With side projects, you are doing something you have full ownership over. You pursue something for its own sake. There aren't many things in life that you can truly say that about.

side_projects === drug_free_therapy

Reason 1: Rejuvenation 

2. An antidote to diminishing attention span

Attention is the one true currency we have. Everyone is vying for it. Our Instagram and YouTube feeds have been meticulously engineered to capture our attention. Podcasts and blogs, including this Substack, all require your attention to ensure their success.

The volume of stimulus thrown at us every day has fractured our attention span. This fragmentation has affected children and grown-ups alike. Another therapeutic effect of side projects is that they are capable of healing our attention span.

I have friends who are hobby musicians and amateur woodworkers trying to produce original work on nights and weekends while supporting their families and raising children. Pulling this off demands a certain discipline, and side projects are the sandbox where your mind has the opportunity to sharpen this focus and discipline.

Reason 2: Honing your one true currency - attention

3. An antidote to retirement

In his book, James Carse purports that everything we do in life is either a finite or an infinite game.

Your 9-to-5 is a finite game that is often mistaken for an infinite one. When I meet someone new at kids birthday parties, the first question is always - “What do you do for work?” We as a society have subsumed into our collective consciousness that the day job is our identity.

In retirement, you will have the physical aspects figured out: morning walks, afternoon yoga, and a few classes. But it is the mental side of retirement that can be deadly. Pursuing side projects and making that a part of your DNA and your main identity is a better place to be.

However, becoming a hobbyist or tinkerer is not something that happens overnight. Just like any other muscle, it needs to be worked on. Side projects are the infinite game everyone should be playing.

Reason 3: Developing your infinite game

Bonus

Serendipity

These side endeavors could open new doors and opportunities that you might not have pursued otherwise.

Setting an example

Your tinkering never goes unnoticed.

I spent a big part of my teen years hanging around a friend’s place. His dad was an HR executive by day, but his shelves were full of programming books. I would always find him walking around with one. Even when driving, the book would be positioned between the driver and passenger seats, ready to read during traffic stops. Later in life he went on to start a tech company.

Observing him had a significant impact on many of my behaviors and interests.

Where do you begin?

1. If you have a few ideas for side projects but have been hesitant to get started, one way to beat the inertia is to commit to a 14-day sprint. Each session can be a short 15 minutes of tinkering, making a tiny bit of progress until the process becomes enjoyable and the habit becomes ingrained.

2. If you are still trying to figure out what side project to pursue, the best way to discover your next interest is to learn and imitate. Take an online course, or pick up a book and work through its examples. Starting your journey by imitating someone you admire removes the pressure to create something new and original.

A few of my favorite side projects

• arun.is

• Mohit Bhoite

• Vik’s Newsletter

That’s it for this session, see you in the next one.

This progression is one way to develop a relevant profile and become an indispensable engineer.

Apprentice → Specialist → Lieutenant → Leader 

Here are a few ideas on how to think about each point along this career pathway.

Apprentice [Years 0 to 5]

Fresh out of university, you are a blank slate. As an Electrical/Computer Engineer, you will graduate with a set of broad skills, almost never experiencing a proper chip tape-out, with the goal of working in some aspect of chip development (as an RTL designer, DV Engineer, Analog Design, Physical Design, etc.).

When you start that first job, you must be able to do a variety of tasks and become a jack of many trades. This will require you to hustle and learn quickly, but the benefit of this hustle is that you are demonstrating to your team leads that —

• You have the capacity to learn quickly and contribute to the project

• You can solve any problem that’s thrown your way

This helps you win the confidence of you peers and dramatically increases your chances of receiving high-quality problems to work on.

But, most importantly, becoming a jack of all trades allows you to learn how the sausage is made — the tools and methodologies used to design and validate a successful chip, how production code is written by experienced engineers, the various stages in SoC/ASIC development such as Performance Modeling, DFT, Physical Design, Timing Closure, Emulation, Verification, Post-Silicon Validation, and so on.

Specialist [Years 6 to 11]

At this point in your career, you’ve experienced a few tape-outs. So, as a natural progression you will be given the opportunity to own a meaningful chunk of the design. Use this to develop deep expertise in at least one area of your field.

Being regarded as the “go-to” person for a specific building block brings additional benefits, such as having to interact with cross-functional teams and become a proper stakeholder in meetings.

From my personal experience this is when I genuinely began to believe in my abilities to engineer, and began to feel like my contributions were making a meaningful difference.

Generally speaking, developing this feeling is a significant milestone. Your career shouldn't feel like a vast generic blob. Milestones like the one above will help you put stakes in the ground which you can look back on with confidence.

Lieutenant [Years 12 to 17]

As the lieutenant you are the linchpin of the team. You are responsible for executing several important functions. You will have to wear three different hats:

• You will continue to work as a specialist to deliver a specific complex portion of the chip, in the process producing a large amount of code.

• You will leverage your experience from being a jack of many trades to carry out auxiliary functions. For example, develop tools, methodologies, and workflows for the rest of the team to use.

• You will be entrusted by the team director to execute their vision, and sometimes filling in for them in their absence. You will be responsible not just for the work you produce but also for that produced by the team.

You are responsible for interviewing and building a team on behalf of your team manager, and make recommendations on how to get past technical challenges. At this stage you should be developing, based on your experience, your own philosophy of how things have to be done.

Regularly reflect on your experience, make notes, and refine your craft.

Leader [Years 18+]

As a leader, i.e., a Director, Architect or a Distinguished Engineer, you now have the opportunity employ the lessons from your experience, and build products and lead teams according to your vision.

Conclusion

Apprentice → Specialist → Lieutenant → Leader 

This, by no means, is the only way to think about your career. But, coming up with some framework like this and understanding what role you are playing in the team will help you set effective goals and get the most out of that point in your career.

That’s it for this session. See you in the next one.

Chip/SoC development cycles are long, typically taking 2-3 years to design, implement, verify, tape-out, and then bringup. Considering the complexity of chips these days, in order to stay competitive and ship a product in a reasonable amount of time, the team sizes even at chip startups are large (at least in the tens, if not hundreds).

From the individual engineer’s point of view, with the ever-shrinking project timelines, there’s enormous pressure to deliver your work at a high quality and in a short amount of time. This means that you will likely work in a silo on a small section of the design throughout the project.

In subsequent projects (for example, the next revision of the chip), you are likely to continue working on the next iteration of the same block. This is obviously beneficial for the company because they already have someone familiar with that block, but you may prefer this option as well because you now carry a certain weight being the expert or go-to person for that block, and since you slogged the last time, doing this again will allow you to take a little break and work at an easier pace (since the code you had written is reusable).

The problem really is that at the end of 6-7 years, when it comes time to sum up your experience at that job, you realize you’ve just worked on one thing. If you are thinking about changing jobs, your set of marketable skills may not be very impressive to potential employers.

The Question

How do you get the most out of your current job?

A Potential Solution

You can extract two different kinds of experiences from your work — implicit experience and explicit experience.

Explicit experience

Explicit experience is the work you’ve been assigned, the tasks that you perform day-to-day. Here are some ideas to make the most of it. Doing the following will demonstrate to your current and future employers that you have the aptitude, willingness, and capacity to own the work entrusted to you:

• History: Going a step farther and learning the history behind your particular block. If you are working with the GDDR6 memory interface, for instance, investigate further what distinguishes GDDR from LPDDR and HBM as well as the subtle differences between GDDR6 and its predecessor, GDDR5.

• Compare: Look into the most recent advancements in your immediate area of work and discover how other companies are doing the same thing. Papers from conferences like DVCon, Hot Chips, DesignCon and SNUG are the best resource for this.

  It’s easy to get caught up in the rush to meet deadlines and get hyper focused on the task at hand. All the exploratory work mentioned above might have to come at the cost of a few weeknights or weekends, but it will make your work more satisfying.

• Patent: Explore whether any aspect of the design (or a variation of it) is unique and patentable. Most businesses value this kind of contribution.

• Publish: Find an opportunity to publish and present your work at a conferences. In the process of preparing for a conference, you will pick up several important soft skills that will make you a well-rounded engineer. It also provides the opportunity to network with engineers from other organizations.

• Workflow: Take the time to develop a good workflow. Iterate and refine it. When a task is assigned to you how do you start? What tools do you use to maintain daily checklists? How proficient are you with diagramming tools, Vim/Emacs? Do you have keyboard shortcuts for frequent actions? These are little things that improve your style and mode of work.

  You must use your explicit experience to hone your craft and polish your tools so that when it is time to sculpt (i.e., write code), you will produce something you are proud of.

Implicit experience

Implicit experience is the experience you can derive from just being part of the project and a wider team. Here are a few ways to maximize your implicit experience.

• Fly on the wall: Being in a chip team means you have access to specialists in other domains such as, Static Timing Analysis (STA), emulation engineers, seasoned architects, signal integrity experts, and many more. By being a fly on the wall during meetings (and paying attention) you can experience the kind of problems people are facing in their own blocks and how they are working through it. During lunch or a walk, you can further probe your team mates to understand their domain a bit more.

  With implicit experience you are essentially collecting stories of others’ experience. Stories are important, because these stories will then influence how you think and design your blocks in the future. These stories are part of the experience you are developing as you transition from a junior to a senior engineer to a director.

• Finding your Minor: At university, there’s this concept of majoring in one subject and minoring in another. This very much translates to work as well. While you are “majoring” on the tasks assigned to you, pick a minor (some other part of the project which interests you).

• Seeking a mentor: This can be a whole article in itself. Mentors are not just for new college grads, you are never too old to have a mentor.

  Good mentorship allows us to stand on the shoulders of giants and look further ahead. Seek out a mentor, someone you aspire to become.

In a nutshell

If you approach your job with this mindset of implicit and explicit experience, when you sum up your work at that work place it will look different. You will genuinely feel like an experienced engineer and you will have developed a much richer tool chest. You will have a better framework to solve problems, and you will be able to borrow from a richer set of lessons.

That’s it for this session. See you in the next one.

I am a hardware engineer, yet there is nothing tangible in today’s 3nm semiconductor engineering. I feel like a software developer because I write code all day. Sure, it’s the type of code that turns into transistors, but still, for the most part I’m sitting at a computer staring at a screen. Mechanical watches with their intricate movements quench my thirst for tangible hardware engineering.

This insightful 5-minute video below is a conversation with craftsmen/women from Grand Seiko, a respected Japanese watchmaker. They are famous for inventing a revolutionary movement known as the Spring Drive.

Lesson 1

I think that being able to warm up the body before starting work is the best.

If the condition of the body isn’t good, then from starting in the morning until heading home, you can’t concentrate.

Minute 2:00

The level of creativity and quality of your work as an engineer is determined by more than just your intelligence, knowledge, and experience. For instance, one bad night’s sleep makes me operate like my kindergartner who was refused ice cream (cranky, stumbling, fumbling and incoherent), rather than demonstrating my 15+ years of experience.

Exercise feels like a super drug, the simplest method to break out of a rut when stuck on a problem.

Lesson 2

In making such a complex device, it is important that the next generation is able to service them since we are the ones that made them.

And, naturally, I will tell the younger workers 100% of what I know. Because I think that passing along these skills is such an important thing.

Minute 3:27

In my first job out of graduate school, one of my assignments was to help bring up the high speed SerDes interface on a chip that had just arrived in the lab. We were at least a week (or two?) into the bringup and these interfaces just wouldn’t function. I could feel the heat in the room, and the bosses' hourly update requests were nerve-wracking.

I was provided with TCL scripts, a stack of intimidating C-firmware, and a 600-page user guide of the SerDes PHY IP, with terms like DFE and FFE which I didn’t quite understand. I was a good student and graduated with honors, but no amount of books or classes can adequately prepare you for your first chip bringup under the watchful eye of your manager.

The only reason this episode had a happy ending was that I was fortunate to have a benevolent mentor who sat down with me, helped me comprehend the problem, but also gave me the space to learn, ask dumb questions, experiment, analyze and reach my own conclusions. When we finally got things working, I felt like I’d earned my moment in the spotlight.

As a senior engineer now playing the role of a mentor, I try not to forget this. It takes the right kind of mentorship to help a junior engineer overcome their impostor syndrome and gain confidence in their engineering abilities.

Lesson 3

This is the minimum requirement in making a beautiful watch: First, each individual part must be beautiful

Minute 0:35

This reminds me of a system architect I once worked with, a true 10X engineer. While the systems he architected were innovative, robust, and made the company a lot of money, I was always in awe of the code he produced. At the lowest level, even simple things like where he chose to place an empty line in a code block, his brief but self-explanatory variable and function names, and the premeditated flexibility he added to the code for future enhancements were stunning.

That’s it for this issue, see you in the next one!

This here is my habit tracker. I have it stuck to my kitchen pantry door. And you can bet that I visit the pantry at least a million times every day. It feels like this piece of paper is in front of my face all day everyday. This is the best way I've found for achieving a specific result.

As I’ve gotten older and my obligations at work and home have grown, I recognize my brain needs scaffolding. It requires a supporting structure to carry out day-to-day tasks. Without it, three things occur —

1. Losing focus: The important things tend to fade into the background and less important things that appear urgent, but are not, take their place. Fast-forward three months, I'm left asking, "Where has time gone? Why are all of these crucial tasks still where I left them months ago?”

2. Untrusting mind: When I’m in the shower, out for a run, or just before I fall asleep, this exhausting voice in my head begins rehearsing the list of tasks that must be completed, much like Arya Stark’s kill list. My best guess as to why this happens is that my mind is terrified of forgetting what needs to be done.

3. Defeat: Life feels overwhelming. A depressing sense of defeat, as though I can't keep up with life and fulfill my responsibilities.

Keeping focus

At work, I've come to rely on multiple pieces of scaffolding to get my tasks done — Post-its, a notebook and Obsidian

• It helps me always know what is important and stay on track with my goals.

• After working through an immediate issue it helps me find my way back to the previously incomplete task more easily, i.e., context switching.

• It helps me keep a paper trail so that I can periodically reflect on my work and evaluate if my processes need to change or if I need assistance from my team/manager.

Trusting mind

I've discovered that picking up a pen and writing down my thoughts is the best antidote to a noisy mind. This is the form of scaffolding my inner voice needs to be reassured that some action will be taken to address its concerns.

I usually find the need for a brain dump either in the morning when I arrive at work, or right before bed. I use post-its so that this list can travel with me.

Victory

When I check off a commitment from my habit tracker hanging on the pantry/fridge door or my Obsidian task list, I feel a sense of victory. Because it confirms my ability to be disciplined and not be captured by the waywardness of daily life.

Image Credit: Reddit

There are usually three aspects of good documentation: completeness, accuracy and clarity.

Software Engineering at Google, Tom Manshreck [Chapter 10]

The problem

Frustration with documentation usually falls into four tiers

• Tier 0: Where is the documentation?

  There is no documentation available, therefore you must rely on inspecting code or consulting with someone to figure things out.

• Tier 1: The Cheat Sheet

  Sparse, incomplete documentation, primarily written by the engineer for themselves.

• Tier 2: Rust

  Documents are present, however they have not been properly maintained or updated. Dangerous, may be misleading, and cause a loss of productivity.

• Tier 3: The 500-page monolith

  Documentation is all there but it is unorganized, difficult to parse and understand.

Why you should strive for good documentation

• This is the first line of defense. The document review is where the team gets involved and provides feedback. This is the best opportunity for you get your design vetted. The better your document is, the more your team will comprehend it and help you identify flaws.

• Your opportunity to truly showcase your work and demonstrate your clarity of thought. Good documentation demonstrates that you are a diligent engineer who pays attention to detail and can be counted on to own the block. It is an excellent indicator of the quality of your code.

• There is a self-reflective aspect to writing documentation. This is where you put pen to paper and distill your ideas into meaningful paragraphs. The process of writing has helped me weed out issues and reevaluate my decisions.

• Lastly, CYA — Fabricating a chip is expensive. You must maintain proper bookkeeping over the course of your deliverables and be able to explain why particular design or verification decisions were made. Were they reviewed? Who signed off on it? In short, a good document offers you insurance of sorts.

Potential solutions

Tiers 0, 1 & 2

The good news is that tiers 0, 1 and 2 can be easily solved by putting a process in place. The manager or project lead can then ensure that good documentation practices are upheld. Two eventualities that need to be addressed are:

• Engineers who believe their time is better spent doing other things should be nudged and reminded that the task is only complete if the documentation is completed.

• For engineers that are motivated to write great documentation but lack the time, the manager should set aside time and allow them to accomplish the work to a high standard.

Tier 3

Designs these days are complex, and specifications will be lengthy. There is no way around it, however here are some things to consider with regard to document organization and style of writing.

I. Know your audience

In semiconductor engineering, you can safely assume that people of at least three different nationalities will read your documentation. I’m a South Asian, and while English is my first language, my proficiency is a distant second to a native English speaker from the West. Add to that, people from different countries come with their own special flavor of english, not only in terms of accent but also in how we phrase our sentences to communicate technical ideas.

So, one of the secrets to good documentation in the hardware industry is, know your audience. Considering this one fact alone will bring you much appreciation from your team.

II. Understand how your document is used

Understanding what YOU look for in a document, will in turn help you become a better writer. I find myself using documents in two modes:

As a tutorial:

• When I’m assigned a block of work, I need to deeply understand how it works, so that I can accomplish my obligations.

• I rely on the document to be complete, with clear diagrams and necessary hyperlinks to other resources which I might need.

As a reference:

• In the course of my work, I often revisit sections of the specification to clarify my understanding of a feature, or quickly extract information.

• For this, I appreciate if key information is represented as a table, a flow chart, or pseudo-code instead of a wall of text.

III. Refactoring

• Documentation, like code, needs to be refactored to meet production quality standards.

• If you observe that numerous coworkers are asking similar questions, it is likely that something in the document needs to be explained differently.

• After writing a few paragraphs, I assess my writing in terms of effective bandwidth. The thing about writers I admire (and quite frankly, envy) such as Noahpinion and Wes Kao, is that they have a large vocabulary to dip into, which I lack. They can effectively explain ideas in a single sentence because of the language they utilize and how creatively they construct their sentences. I can't achieve what they do with just words, so I make significant use of diagrams and visualizations. As they say, a picture is worth a thousand words.

That’s it for this episode, stay tuned for the next one.

Opening Picture: A scene from Death Note

Over the last 30+ years I’ve seen what humans need to reach their potential, to disrupt what needs disrupting, to forge their own unorthodox path.

Build, Tony Fadell

The Problem

I’ve interviewed several engineers with 5 to 15 years of experience. The majority of them were excellent, but many, in my estimation, were yet to realize their full potential. They are smart, communicate clearly, and demonstrate the necessary aptitude and the will to tackle larger engineering problems. Unfortunately, their work experience hasn’t afforded them the opportunity to rise to their true potential.

The Baker’s Paradox

In some ways, corporate work is equivalent to waking up every day and making a hundred loaves of bread. Your baking must be uniform, resulting in similar-tasting bread of consistent shape, size, and quality.

While at school, you are taught to make bread but you haven’t had that many reps yet. You start working for a company and manufacture bread day after day, all day. You become skilled at creating one type of bread, your manager will regard you as a reliable employee, and you may even earn the title of Star Baker because the bread you produce is good and consistent.

However, you are still far from your true potential. You can only make one (or a few) kinds of bread exceptionally well. With time, you become complacent. They pay well, you receive regular promotions, you may even mentor/manage a few junior bakers, who will look up to you. After 10-15 years you are regarded a competant baker, but you haven’t really learnt any new skills in a long time.

In reality, you have progressed from a

Beginner → Advanced Beginner

and not from

Beginner → Expert

The Self-Learning Void

Another problem is the difficulty in self-learning. Semiconductor engineering does not lend itself well to weekend experimentation. It’s not like software engineering, where a hobby can be turned into a Minimum Viable SaaS Product or a Web App for little to no money before being taken to the market and evaluated for economically viable.

Cadence, Synopsys, and Siemens sell pricey chip-making tools. Open Source tools and the ability to tape-out small projects are in the works, but we’re still getting there. The Maker world with FPGAs, Arduinos, Raspberry Pi, sensors and breadboards, while exciting, are an insufficient proxy.

Another issue is, unlike in the field of software, in hardware engineering information is not readily shared. Companies are secretive and apprehensive about giving any edge to their competitors. There are a lot fewer blogs, YouTube channels, and activity on Stack Overflow is sparse.

The side-effect of a non-vibrant community is, engineers mostly learn on the job from their peers and mentors, and end up specializing only in that specific narrow silo that they work in.

Giving credit where it is due, there are a number of incredible semiconductor and computer engineering substacks. However, compared to the software industry, we as a community still have a long way to go.

Potential Solutions

The recipe to reach your true potential needs 4 ingredients —

1. Workshop: A place to experiment, learn and make mistakes at creating new recipes.

2. Inspiration: A mentor, someone you can ask questions to. Someone who can diagnose the issue with your sourdough by looking at it. Someone whom you aspire to become one day.

3. Community: 
   + Expose yourself to things you would not have considered. The chance to be amazed and wonder, "Holy sh*t you can really do that?" 
   + A bunch of equally passionate learners. A study group, just like in university, where you can discuss new recipes and exchange ideas.

4. Stage: An opportunity to showcase, present, and receive feedback on your work. 

Now, how do you apply these 4 ingredients to your own development?

• Workshop

  • Work is not the place to experiment. Semiconductor companies are risk-averse, for good reason. You are expected to use tried-and-true methods.

  • Today’s chips are complex, and the majority of ASIC/SoC engineers work on a sub-module of the chip. However, the code base and your teammates are at your disposal. Dig through other people’s code, study documentation, and ask questions during lunches, walks or any opportunity you get. Ask them why they made certain architectural decisions. Why was this memory type selected over the other? Why was this hash algorithm selected? Even if you only work on a portion of the chip, you must develop a sufficient understanding of the whole chip through this type of investigation.

  • Learn to compile and simulate other pieces of design you aren’t working on. You can of course use company resources and tools for this. Your company stands to benefit from your increased expertise.

  • If you’ve always inherited large code bases, learn to build toy designs from first principles. You may not be able to use company resources and tools for this, but I’ve found EDA Playground and my own Xilinx FPGA setup at home very handy for this type of exploration.

• Inspiration

  • Actively seek out a mentor, whether within or outside the company. Someone with whom you feel comfortable sharing your experiments or discussing ideas that excite you. Someone whose code you enjoy reading and whose concepts inspire you.

  • Don’t limit yourself to books and blogs about hardware engineering. The world of software engineering is a tremendoes source of inspiration, and you’ll be astonished at how useful concepts learned in software engineering may be applied to chip development.

  • Refactoring and The Pragmatic Programmer are two of my all time favorite books.

• Community

  • In 2012, one of my software engineering friends pulled me to PyCon, the Python conference. As a hardware engineer, I had no business being there. But the conference was just down the road from where I worked, so I tagged along with this friend. That one weekend forever changed the way I think about and write code. It exposed me to a whole new world of engineering, and even now 12 years later I’m still delving down rabbit holes that I started after that conference. This conference inspired me to build a set of skills that are uncommon among Hardware Engineers.

  • Conferences like DVCon and SNUG are the BEST places to learn what engineers at other companies have discovered. It is the best learning resources for semiconductor engineers. At every chance, ask your company to sponsor your attendance at these conferences.

• Stage

  • Presenting your work is important. It forces you to take a pause from all of your learning, reflect on it, and distill your ideas into coherent paragraphs and diagrams.

  • At least once in your career, you must present a paper at one of the conferences mentioned above.

  • This process will teach you some really good skills; you have to step out of your comfort zone, write a proposal, and perhaps get rejected. And if you are accepted, fight those nerves, overcome the impostor syndrome, and get up on that stage.

  • Start a blog or substack, its never been more easier to do this.

That’s all for this edition of Chip Log. If you enjoyed what you read, share it with your friends and colleagues. Subscribe to be notified of the next post.

Subscribe now

Image credit: Screen Rant

While solving a tough problem or writing an involved piece of code, my mind is entirely occupied in solving that problem. But, at least half the time I’m writing mundane code. Just implementing your average features, going through the motions and churning out blocks of code. During this time my mind wanders — often anxious about upcoming meetings at work, or over-analyzing previous conversations, or worrying about deadlines. My mind is not at peace.

This type of wandering is annoying. It makes me impatient. I can’t wait to get done with this immediate work in front of me. I’m in rush, enveloped by this feeling that this thing, this feature, is wasting my time. I have more important things to get to, sh*t the deadline is soon approaching ...

But now I recognize this mundane-ness while it is in progress and I remind myself to enjoy it. Turn that anxiety into a time to relax my mind. This work needs to get done too right? And it cannot be rushed anyway. Rushing is how mistakes are made and this simple mundane code could come back to bite me.

Music is my drug to facilitate enjoying mundane code, and may be a good cup of decaf to go along with it.