FPGA DDR3

In late 2013 we were consulted to design a product which required a large amount of active memory capacity — of up to eight sticks of DDR3 RAM (computer memory) with four FPGA’s to interface to them (two DIMM’s per FPGA). There were a lot of unknowns and the project would require several months for research and development.

FPGA-DDR3 Close-up of Top View 3D Model

FPGA-DDR3 Close-up of Top View 3D Model

Unfortunately at that time we were still busy working on the stairlift system so we turned down the project. A few months later when the engineer they contracted could not complete the project, we agreed to take over and finish it, which we did by May 2014.

FPGA-DDR3 Bottom View 3D Model

FPGA-DDR3 Bottom View 3D Model

This is a fairly large 8-layer circuit board, 13 x 8.4 inches (330 x 213 mm).

Each of the eight DIMM sockets has 204 pins and each of the four FPGA’s is a 676-ball BGA. In addition, the board includes a 32-bit Atmel microcontroller in 144-ball BGA which has integrated USB, and an external Ethernet PHY chip, and many switching and linear voltage regulators.

The high-speed interface between the DIMM’s and FPGA’s requires equalized lengths and several of the signals are also differential pairs (one pair per 8-bit bank). Switchback routing was used to equalize the net lengths.

Comparison of Serpentine vs Switchback Length-matching

Comparison of Serpentine vs Switchback Length-matching

At the time Altium only supported automated serpentine which is a very nice and functional feature, except the central area of the FPGA-DIMM bus was too crowded. Technically serpentine is the use of many parallel switchbacks at right angles to the direction of signal flow, so switchback is simpler for parallel busses such as this, but not recommended when there are adjacent unrelated signals due to crosstalk.

Altium is so easy to use that manually routing the switchbacks was not very difficult. All the matched nets on the board have the same target length and you can see the length of any net just by pointing at it with the mouse. Adjusting the length of a switchback involves telescoping one U-shaped end like a trombone, whereas serpentines typically have many U-shapes which are preferred to be all the same length.

Due to the fact that all four FPGA blocks were identical, the majority of the routing was copy-pasted.

FPGA-DDR3 Sample Trace Routing

FPGA-DDR3 Sample Trace Routing

The engineer who worked on the design prior to us supposedly had a Master’s degree, but there were extensive mistakes throughout the design. He didn’t seem to have the expertise to design a board as complex as this.

Instead of using Altium’s reusable “REPEAT” schematic blocks, he had copy-pasted the sheets with global net identifier scope. By using the repeat feature we were able to reduce the schematic from 40 pages to just 11. Because he had not used strict heirarchical scope, he had to manually rename thousands of nets but forgot or misnumbered many netnames which would have caused crosslinking between the discrete FPGA blocks. Also, he made changes afterward which were not reflected identically on every block. Some sheets contained mixed elements from multiple blocks and had to be reorganized.

He had routed quite a lot of the PCB layout, however there were severe mistakes there as well. None of the through hole pads had holes in them. Many vias had no annular ring (the hole was the same size as the pad). The two rows of pads of the DIMM sockets weren’t properly offset which would have made the board completely unusable garbage (all eight DIMM’s were already routed this way). But worst of all, the high speed nets had been routed with no length equalization, and the differential pairs weren’t run in parallel. The default DRC rules had been disabled and none of the required length or clearance rules or net classes had been added.

FPGA-DDR3 Top View 3D Model

FPGA-DDR3 Top View 3D Model

Despite initially being told that the design was almost finished, we ended up completely tearing the design apart and putting it back together in just two months. A serious project like this requires an engineer with a broad range of technical experience and diligence, otherwise it will cost you far more money and time (more money) to spin the board design again and again. Don’t risk it. Come and talk to us.

At the present time, our client is still developing software applications for the board, still on our original spin.