Getting the Most from Network Function Virtualization

Virtualization is by now familiar in terms of CPUs and now virtualization concepts are coming into use for improving the efficiency and throughput of network architectures.



Network Function Virtualization (NFV) is the new technology that “virtualizes” network devices, such as firewalls, spam filters, intrusion detection systems and many other network devices that historically have been sold as a stand-alone device.  NFV has generated intense interest for the same reasons virtualization has come to dominate the server market: Flexibility to scale resources to the needs of the business, eliminate the burden of managing physical systems, and the obvious savings in capital investments and operational expenses.

The potential benefits are substantial enough to overcome the usual resistance to change in the data center. In addition, the market has generated a number of enticing applications typically required for a new technology to gain traction.

NFV Versus VMs


It may not be immediately apparent what the difference is between the technological requirements of virtual machines (VM), which are well-established in the market, and NFVs.  The main difference is that NFVs tend to be more network I/O dependent than general servers that follow a client-server model. To illustrate, Figure 1 provides an overview of the network environment inside the hypervisor where the two virtual technologies are used.


Figure 1
NFV versus VM .

The VM, which is typically a web server, serves requests from several clients.  It may require back-end servers for storage or databases, but in general it performs computations of some kind in the process.   Many factors, such as the number of CPU cores and the complexity of the service process, determine its overall performance.

When a client submits a request, or batch of requests, another request is generally not submitted until the first request is processed or times out. This process also applies to the packet aggregation of groups of clients.  This workflow tends to regulate the network flow in and out of the server, and is typically measured in terms of how many requests per second or the number of simultaneous clients.

In contrast, NFV will not be similarly “in control” of the volume of LAN traffic.  It is intended to replace a physical box that performs the same function on behalf of other servers that may be VMs or completely separate systems.  The performance of NFV is expected to exceed the performance of the servers for which it is processing network traffic (assuming that network traffic is not line-rate) because users do not want system performance limited by the firewall or other security device in front of it.   Its performance will tend to be measured in metrics such as packets-per-second and flows-per-second.  LAN interface performance is critical in the case of NFVs.

Hypervisor Software Switching


By default, NFVs are built with the same software systems and structures as servers (Figure 2).  In this example, we have two NFVs—a malware detection device and a spam filter—that are processing packets on behalf of a VM server (mail server).


Figure 2
Two NFVs and a server


The software switches in Figure 2 are programs executed in the hypervisor environment.  Software reads each incoming packet, identifies the destination MAC address, and copies it to its destination accordingly.  The NFVs and the VM server interface to the software switches with special virtual drivers.   A Linux kernal-based virtual machine (KVM) typically uses the virtio driver, while VMware uses vmxnet.


Figure 2
Two NFVs and a server

Although this technique is very expensive in terms of processor cycles, it is has the advantage of being very easy to set up and flexible to manage.  A common feature in modern VM environments is the ability to move VMs and NFVs from one physical machine to another without any loss of functionality.   This may be accomplished simply by moving the VM image files, then assigning the virtual network drivers to the appropriate software switches.  Standard L2 and L3 switching rules handle the rest.


The more NFVs in the system, the more network-related performance limits are apparent. The number of in-system transfers of packets increases quickly with each new NFV.   In the example shown in Figure 2, one packet will be transferred in and out of memory a minimum of 5 times, not counting the transfers the NFVs and the server do alone.   That calculation is for a one-way trip; the return path could add another 5 memory transfers or more.


Figure 2
Two NFVs and a server

The end result is that if the end server is handling 1Gbit of traffic in and 1Gbit of traffic out, the hardware may have to provide up to 10Gbit of data transfer capacity.   This drag is magnified even further when a single physical server hosts dozens of VMs.This situation has motivated the industry to develop solutions relieving the stress on the data busses of systems that implement VM and KVM technology.  One approach that is gaining popularity is single root I/O virtualization (SR-IOV).

Pros and Cons of SR-IOV

General purpose processors are not well optimized for L2/L3 data switching. Instead ASICs are typically better at this job, and have been introduced to the VM data flow by the SR-IOV industry standard.



The idea is to move the task of copying and switching data packets from a software switch running in the processor to the network interface (NIC) adapter itself.   When applied to the example in Figure 2, the data path structure is transformed as shown in Figure 3.

Figure 2
Two NFVs and a server


Figure 3
Two NFVs and a server with SR-IOV

Although this diagram is simplified (the presence of the hypervisor software driver has been omitted), the LAN adapter provides multiple state machines and register interfaces to simulate the presence of many LAN adapters instead of just one.  The Intel 82599 10 GbE controller, for example, implements up to 64 such interfaces for each Ethernet port.

Each SR-IOV Virtual Function (VF) interface can serve and be controlled by a separate SR-IOV driver instance.  These can be assigned to various VMs and NFVs, as needed.   The data arriving from the LAN or being transmitted to the LAN can transfer directly from buffers inside the virtual environment of the VM or NFV.  Although the number of data transfers that must occur to complete the data path is reduced only slightly (because there is no need to copy into the software switch in the hypervisor), the processor does not need to look up MAC or IP addresses and move the packets. This results in significantly less load on the system.

As a by-product of its design, the SR-IOV adapter can also serve as a switched data path between VMs and NFVs.  The data is transported out of one buffer, over the PCIe bus to the adapter’s NIC, then back over the PCIe bus to the new buffer.   SR-IOV VFs are a limited resource that must be considered in the management system.  This mode of operation does have its own considerations. For one thing, the PCIe bus can become overused resulting in a system-wide performance problem. And it is also true that methods of direct copy between VM structures may be faster. And finally, software management for setting up this mode of operation may be problematic, particularly when VMs and NFVs are moved from one hardware box to another.

A special problem with SR-IOV arises when an NFV is expected to perform a Layer 2 networking function.   All traffic on a particular LAN must be seen, not just those packets with the NFVs own MAC address. This can be achieved with an L2 switch or a network snooping device and is referred to as “promiscuous mode” operation.   In some models of SR-IOV silicon, promiscuous mode to a VF (the interface used by a VM or NFV) is not supported.

Performance for SR-IOV equipped VM/NFV systems is definitely better than software switched systems.  Data published by various vendors, including ZNYX Networks, show that a VM with an SR-IOV pipe can get as much as 80% of the throughput as the same software application running within a hypervisor.  This level of performance is not always easy to attain since it requires a fine degree of tuning. Tests indicate that carelessly installed NFVs can result in a performance loss of up to 80% — in other words an application will process only 20% of the traffic it could handle if were installed “bare metal” in the hypervisor environment.

Device Pass Through

Another approach to serving network traffic to VMs and NFVs is device pass through, which places a complete network adapter device under control of the VM and is not visible to the hypervisor. Complete LAN-to-VM memory transfers are achieved without overhead and the complexities of SR-IOV.  Applications that require promiscuous mode service must use Device Pass Through if the network device is to be run in a VM.


The disadvantage of device pass through is that, by definition, it allows only one VM or NFV to access the LAN channel.   Most VM hosts are limited in the number of LAN adapters they have to allocate.  In practice when device pass through is used, it is combined with the other methods of serving LAN traffic to VMs (Figure 4).


Figure 4
Use of Device Pass Through


In Figure 4, the intrusion detection NFV has full access to the LAN through its own dedicated adapter.  It is configured in monitor-mode only, where it just inspects all traffic and generates alerts or reports.  The NFV #2 is still acting as a filter for the mail server in the same system as before.


Figure 4
Use of Device Pass Through


device pass through is dedicated to a physical port on the system.  So in Figure 4, there will be a connector on the outside of the box labeled “Intrusion Detection.”  Similarly, each NFV that uses device pass through will have its own dedicated connector and label.  This attribute dilutes the benefit of NFV, which otherwise would leave all network connections fully up to software.


Figure 4
Use of Device Pass Through


A solution that ZNYX Networks has found is to integrate a fully functional Ethernet switch into the NFV platform, to which all of the LAN adapters are attached.  With the switch controlled by the software, any physical port on the outside of the box can be logically connected to any NFV, regardless of whether it is using device pass through or not (Figure 5).


Figure 5
Switching with NFV.

Optimal NFV Platform


The concept of NFVs allows them to run on the same processor systems as server VMs, and in many cases this will be the preferred mode.   Ideally, it does not matter where the NFV or chain of NVFs are relative to the servers, but in practice it is pretty clear that available LAN bandwidth will always be a factor. Large-scale NFV installations benefit from platforms with the features in Table 1.


Table 1

Simple in concept and with many benefits, there is little question that NFV is the next big thing in the data center.   The technology is not without its growing pains but these are being overcome by both open standards and proprietary efforts on every front.  For the current year, the recommendation is to use platforms that offer plenty of discrete LAN interfaces to match the processor capabilities, and to use SR-IOV and device pass through as much as possible.


ZNYX Networks
Fremont, CA
(800) 724-0911