Zero Copy Components

When prototyping a new system, we aim at using the simplest available technology for performing communication. A solution that is offered on a broad variety of operating systems and hardware platforms is the IP protocol stack. Encapsulating the TCP protocol, the communication components become not more than a wrapper. Using a datagram oriented protocol[Ste98] like UDP however makes it necessary to provide a byte stream abstraction[Rit84]. Adding a ring buffer that allows data to be sent and received at arbitrary lengths makes the component more complex, but allows transparent replacement without any changes in the surrounding software system. Both the IP stack and the ring buffer require that the data are copied. This performance overhead can be described by giving the AlSyMe-tric properties of such a component. The algorithmic overhead is $O(N)$. The length of the basic transfer entity is fixed, hence the component has to iterate over the data that have to be sent and place them into several messages. When receiving datagrams it must put the messages into the ring buffer, from which the application can read them. The overhead of the system calls is therefore also $O(N)$, as the operating system is involved as often as the algorithm reads or writes data. Concerning the use of memory buffers, we can identify two buffers that are used. For incoming data a preallocated ring buffer is provided. In addition, the application programmer has to pass a second buffer with the message to be sent or for the message to be received to the component. Therefore the Me property is $O(2)$.

When moving from prototyping to a system integration phase, we leave the desktop environment behind us and let our software run on a real-time operating system. We provide another implementation for the communication component that is capable of making use of zero copy buffers that are available on the new platform. Both the ring buffer used for gathering incoming datagrams and the buffers provided by the user are just pointers to memory that is allocated by the operating system and provided to the application. When recalculating the properties of the component performance model, this only has an effect on the Me quantity. As depicted in figure 44 the data received by the component are not passed through all processing layers, but only their identifiers are forwarded. Therefore the Me property becomes $O(0)$. This difference can be seen when performing measurements of data transfer. From the metric properties we expect a growing performance difference between the ordinary implementation and the zero copy approach with increasing data sizes. This originates in the fact that the algorithmic overhead involves $O(N)$ data copy operations. The longer the messages are, the more time will be spent in a single data copy operation. Now these copies fall back to cheap pointer passing operations. The effect is indeed documented in the characteristic figure of the component (see figure 45). The benchmark consists of a roundtrip time measurement, performed at the sender side. The left plot depicts the increasing latency with growing data length, whereas the right one shows the effective bandwidth. All measurements have been carried out on PowerPC MV2304 platforms running the VxWorks 5.4 operating system. The network was a 100BASE-T Ethernet connected to a dedicated 3COM switch. The glitches at packet lengths equivalent to multiples of the Ethernet packet size stem from the high overhead of handling an additional package that has only very little payload.

The price that we have to pay for the increased flexibility obtained by object-oriented components is small. It can be best seen when performing low latency memory accesses with fine granularity. An example is shown in figure 46. It outlines the time needed to make a short data transfer from a memory module in a VME bus crate to main memory of a PCI bus based microprocessor board in the same VME crate. The set up is shown in the left part of the figure. A DRAM module is plugged into a VME bus crate. A microprocessor board (MVME2306) hosts a PowerPC processor and a Universe VME/PCI bridge that performs DMA data transfer from VME to the memory on the processor board. The smallest possible amount of data that can be transferred in the given configuration is 8 Bytes. The highest efficiency can be achieved when using the primitives provided by the VME-PCI bridge chip vendor directly. Some price has to be paid when providing a DMA copy operation as a C function call. Wrapping this call into an object oriented stream component adds only a little additional overhead. In total the difference between the bare interface to the chip and the high level stream component is 1 $\mu$sec. From the plot we can see that the overhead is constant for all data sizes.