The rapid growth of AI models and KV cache requirements is pushing the limits of memory bandwidth and energy consumption in data centers. Although existing SRAM solutions provide high speeds, they remain constrained by scalability limitations and high costs. While HBM memory offers high capacity, it encounters a major power barrier as bandwidth increases. Seeking a solution to this problem, startup d-Matrix has announced its Raptor 3D DRAM technology, which eliminates the PHY layer and integrates memory beneath the logic chip.
Compared with conventional HBM4 solutions, the Raptor 3D DRAM architecture offers a radical difference in efficiency. In each chiplet, the logic layer manufactured using TSMC’s 4 nm process is positioned on top of the custom 3D DRAM layer using a 36-micron Face-to-Face vertical stacking method. This eliminates the PHY interface and physical boundary constraints. Providing 32 GB of capacity in a single stack layer, the system delivers 100 TB/s of bandwidth with energy consumption of 0.37 pJ/bit.
Considering that HBM4 memory consumes 2 to 3 pJ/bit to achieve 18 TB/s of bandwidth, the Raptor architecture delivers 5.6 times more bandwidth while consuming 5 to 8 times less power.
d-Matrix is not the only company operating in the field of vertical memory integration. Samsung’s three-stage zHBM architecture, which uses a similar vertical stacking method, also aims to boost performance and save power by placing memory layers directly on top of the processor. In contrast, d-Matrix positions the logic layer at the top of the Raptor architecture, facilitating heat dissipation through liquid cooling solutions. The system operates at a power density of 0.5 W/mm² per chiplet and was designed to withstand a junction temperature of 105 °C. To prevent leakage currents occurring at high temperatures, the memory refresh rate is increased 8 times, setting the refresh interval at once every 4 milliseconds. This process results in only a 1.37% loss in total memory bandwidth.
Special mappings were implemented at the architectural level to reduce the power consumed during data transfers.
The 256 tensor engines in the chiplet are directly mapped to the physical width of the DRAM banks. While the 3 banks on each channel provide 96 bytes of data, the 128-byte stream required by the tensor engines is supplied through a 4-way merging mechanism without causing bandwidth loss. Small micro-bank designs with 1,366 rows and a capacity of 5.33 megabytes are used to increase manufacturing yield. To guard against potential errors, the logic layer incorporates the Reed-Solomon T=2 error-correcting code (ECC), which can correct 2 symbol errors in 128 bytes.
In system-level tests, d-Matrix Raptor 3D DRAM provides 23.4 times more bandwidth per square millimeter than the NVIDIA Rubin R200 HBM4 platform at an effective bandwidth utilization rate of 83% to 85%. It also demonstrates 13.5 times higher power efficiency when measured in gigabytes per second per milliwatt.
Balancing high processing speed and energy efficiency, this architecture offers a high-performance alternative to the memory bottlenecks encountered particularly in the inference processes of large language models and autonomous AI agents.