RTC Magazine

Thanks to the universal nature of Ethernet, the 10 Gigabit Ethernet (10GbE) standard promises developers new levels of portability and easier software maintenance at a speed that has never been experienced before. This high-speed, ubiquitous technology has formed a new baseline, and many developers are looking for ways to introduce this technology into a number of high-performance real-time applications.

One such application—as an interconnect for high-bandwidth sensors—is a data plane domain where earlier generations of Ethernet were unable to compete against the likes of (serial) Front Panel Data Port (sFPDP) and Fibre Channel. Now, as 10GbE with its superior speed and facility for bidirectional, full-duplex communications pushes into these applications, the equipment used along with these sensor networks, such as high-speed real-time data record and playback systems, require 10GbE interfaces.

10GbE interfaces are somewhat more complex than earlier generations of Ethernet. Due to the interface’s sheer speed, processors would be entirely consumed running the 10GbE protocol stacks. Because of this, most 10GbE implementations use some form of a protocol offload coprocessor to achieve high data transfer rate performance and reduce processor utilization. The offload coprocessor, such as an ASIC or FPGA, performs the heavy lifting of the protocol, reducing the burden on the CPU. Integrating a 10GbE interface into a real-time record and playback system really involves integrating an offload coprocessor.

High-speed real-time data record and playback systems capture relatively large quantities of data from systems or sensors. The data typically is used for offline analysis or is replayed into a system for training, testing, simulation, or development. Figure 1 contains a block diagram of a typical system. The record and playback systems have some unique functional and performance requirements, which in turn impose some requirements on the 10GbE interface. Notably, these 10GbE interface requirements differ significantly in several key application-level and hardware-level areas from the requirements typical to 10GbE interfaces used in the large server-based markets.

Time-Stamping and Playback

A common requirement in high-speed recording systems, especially multichannel systems, is to accurately time-tag the data with a sufficient level of accuracy. The tagging is used during offline analysis, to enable the alignment of data recorded from multiple sensors. It is also of use during playback, to simulate the re-injection of data into the system with the same timing fidelity with which it was originally captured.

Accomplishing this time-tagging task presents two challenges. The first is to find a means of tagging the packets with sufficient accuracy and precision. When the time-stamp accuracy requirements are sufficiently high—a few microseconds or less—it is not satisfactory to stamp the packets at the application software layer, which is the most straightforward access layer. By the time they reach this layer, they have already made a trip through a PCI-X bus or PCIe fabric, processor memory, possibly a processor cache, and an operating system. This trip is subject to variable latency of a magnitude exceeding a few microseconds, making the solution intolerable.

Assuming recorded packets have been accurately time stamped, a second challenge hinges on meeting precise timing constraints during playback. A typical approach that relies on a processor pulling the data from system memory, flowing it up through the protocol stack, and sending it out over a 10GbE interface, suffers from the same non-deterministic latency that occurs when you time-stamp the application at the software level. This approach will not achieve the required precise playback timing.

Instead, the solution to the recording time-stamp precision/accuracy is to provide an interface to stamp the packets immediately after they arrive over the 10GbE wire. There they can be deterministically tagged before reaching any buses, fabrics, or processors. The playback precision challenge is solved by having an interface and memory located in the outbound path immediately before the 10GbE wire. Outbound packets can be moved from the recorder and staged in the memory ahead of time, and their release onto the wire precisely gated.

Note that ASICs designed for commodity 10GbE NIC cards generally do not support hardware interfaces for packet time-stamping or playback staging. This leaves access for tagging at the socket layer only, which is insufficiently accurate.