Centralized network traffic monitoring at cloud hosted servers

We use centralized network traffic analysis tools to monitor performance of our applications. Transition some of the environments to the cloud data center created noticeable complications it their monitoring, as an existing approach doesn’t assume network traffic collection from distant hosts. In this article I want to share how I could resolve this problem doing a small workaround with a homebrew software.

GitHub URL: https://github.com/daniilkochetov/ivTAP

Existing monitoring approach

Traditionally, we capture network traffic from SPAN ports (or mirroring ports) at the edge switches of our local data center where we can see all the communications between end users and front-end servers (see Picture 1).

Picture 1

SPAN port reflects all the end users front-end traffic to the probe server. Probe server performs decoding and analysis of the collected raw data and sends aggregated statistics to the application performance analysis servers. These servers provides support teams with access to the collected statistics over various dashboards. They can also trigger an alert if monitored application suffers from unacceptable performance.

My approach for cloud hosted servers monitoring

The approach described above works well until you have physical access to the data center network equipment. You can take a patchcord and connect dedicated switch port to your probe device. You cannot do this if the monitored application migrates to the cloud. This is either technically impossible or it would be inappropriately expensive. Meanwhile, your application support teams prefer having a single performance monitoring system which would work similar at the internally and externally hosted servers.

I spent about two weeks of spare time for initial creation and launching of virtual network tap software. I called it – ivTAP: Intellectual Virtual TAP device. It is simple, stable and handy. It doesn’t consume noticeable system resources, but on top of that it can control the bandwidth of the reflected traffic flow.

Its working principle is illustrated at the Picture 2.

Picture 2

ivTAP clients must be installed at every virtual front-end server where we need to analyze network traffic. It does quite simple job. Using winpcap/libpcal library it captures network packets which matches predefined filtering condition and forwards them to the server part of the ivTAP software over UDP channel. Bandwidth generated by every UDP channel is under constant control, so the solution is not supposed to overload WAN link between Cloud hosting provider and our Data Center.

The server part of ivTAP software extracts those packets from UDP channel and injects them into physical SPAN port, like they captured “naturally”.

In the result, I have a solution which solves the task and satisfies my requirements:

• ivTAP doesn’t create an additional threat neither to the application front-end servers, nor to application performance monitoring solution:

o   It doesn’t consume noticeable system  resources (CPU<1%; RAM <=512Mb)

o   It doesn’t create uncontrollable bandwidth consumption spikes

• This method of capturing doesn’t add noticeable error to performance measurement. In every moment maximal error defined by network jitter, which is supposed to be low between two data centers
• ivTAP works stable during continuous time
• ivTAP client can be launched as an ordinary service (at Windows I prefer WINSW, but you can  select any)
• Application performance analysis server processes the ivTAP traffic with the same way as the one collected from physical SPAN port
• ivTAP software doesn’t breach any license agreement: all the manipulations with traffic happen at the operating system layer not touching existing probe server software
• During creation of ivTAP I used only those open source libraries which allows commercial usage – LGPL, MIT, Apache v2.

Here is the list of the tools and libraries I used for ivTAP creation:

•         Eclipse Java EE IDE;
•         Winpcap/libpcap;
•         Jnetpcap library;
•         kohsuke.args4j library;
•         Freeware service-wrapper for Windows – winsw;

ivTAP concept

Here is few the most important points explaining basic principles of ivTAP. I have to ask for your forgiveness in advance: I’m not a professional programmer. At least, I didn't own a cent working as a programmer for any project. Some of code fragments might trigger resentment for more experienced readers.

Picture 3 illustrates the general concept:

Picture 3

Client part of the software is based on the class :IPTAP. The first thing it does in void main() – it checks for single possible argument of the command line “-l”. This key supposed to be used only once, at the very first launch of the client.  It helps to list all the network adapter names at the given server. For further work we need to select the one which processes desired network traffic flows. The listing below represents a light modification of the example presented at jnetpcap web site (see Listing 1).

private static void listDevices() { List<PcapIf> alldevs = new ArrayList<PcapIf>(); // Will be filled with NICs  StringBuilder errbuf = new StringBuilder(); // For any error msgs int r = Pcap.findAllDevs(alldevs, errbuf); System.out.println("List available interfaces and exit"); if (r == -1 || alldevs.isEmpty()) {  System.err.printf("Can't read list of devices, error is %s", errbuf.toString());  return;  } Iterator<PcapIf> itrdev = alldevs.iterator(); while(itrdev.hasNext()) { PcapIf device = (PcapIf)itrdev.next(); StringBuilder sb = new StringBuilder(); sb.append(device.getName()); sb.append(";"); sb.append((device.getDescription() != null) ? device.getDescription() : "No description available"); sb.append("; MAC address:"); try { if (device.getHardwareAddress() != null) { byte[] mac = device.getHardwareAddress(); for (int i = 0; i < mac.length; i++) { sb.append(String.format("%02X%s", mac[i], (i < mac.length - 1) ? "-" : "")); } } else { sb.append("No MAC address"); } } catch (IOException e) { System.err.printf("Can't read MAC address"); } sb.append("; IP address(es):"); List<PcapAddr>  addrs = device.getAddresses(); Iterator<PcapAddr> itraddr = addrs.iterator(); while(itraddr.hasNext()) { PcapAddr pcapAddr = (PcapAddr)itraddr.next(); sb.append(pcapAddr.getAddr().toString()); } System.out.printf("%s\n", sb.toString()); }               return; }

Listing 1

Then, we have to specify this adapter in the client settings configuration file ivTAPclient.properties (see the example in Listing 2).

sIntName=\\Device\\NPF_{XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX} filterString=tcp port 3389 srvAddr=10.0.0.1 srvPort=20004 speedLimit=10000000 bandwidthCheckInterval=10 bandwidthBreachIntervals=6 maxIdleIntervals=4 maxRestarts=3 businessDaysNumbers=2,3,4,5,6

Listing 2

Apart of that, the client configuration file contains other important parameters:

• sIntName=\\Device\\NPF_{XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX} : This is a source device name. 
• filterString=tcp port 3389 : This is a BPF filter sting. When you launch the program winpcap/libpcap will be capturing only those packets which matches this filter. 
• srvAddr=10.0.0.1 : IP address of the host where you run the server part of ivTAP software. 
• srvPort=20004 : UDP port of the ivTAP server, where you send captured packets. 
• speedLimit=10000000 : Maximal bandwidth which ivTAP client allowed to use when sending captured packets to ivTAP server over UDP channel. Must be specified in bits per second.
• bandwidthCheckInterval=10 : Interval of bandwidth checks, specified in seconds. 
• bandwidthBreachIntervals=6 : Maximum number of sequential bandwidth check intervals with breached speedLimit. If bandwidth is always higher than allowed during bandwidthBreachIntervals in a row, ivTAP client will stop. This way we avoid unnecessary program terminations on rare bandwidth consumption spikes.
• maxIdleIntervals=40 : network capturing process will restart itself if it won't capture any packet during maxIdleIntervals. Practical usage in third party cloud environments shown that winpcap/libpcap might silently become unable to capture network traffic after some kind of manipulations at operating system level or at virtual machine environment. This happens without any exception so I find no better workaround.
• maxRestarts=3 : the program will terminate after maxRestarts number of restarts when no traffic is detected. 
• businessDaysNumbers=2,3,4,5,6 : there is no use to restart ivTAP client at the weekend even if it doesn't see any packet. Names of these days of the week can be verified in the ivTAP client log when it initiates (this is helpful if you are not sure what is the first day of the week number at the particular host). 

ivTAP client assumes creation of eternal loop of pcap capturing:

pcapIn.loop(-1, jpacketHandler, "ivTAP");

In pcapIn:Pcap instantiation I use timeout equal to 10 milliseconds (this is an additional error escribed in the Picture 3 above). Ideally, I would prefer having it zero, but this might conflict with basic principles of libpcap/winpcap. In the documentation it is said that zero timeout makes pcap filling all available buffer before it sends data to the process. The timeout I set might be looking questionable as it is taken rather from my understanding of how the system works as minimal and safe enough at the same time.

public class PHandler<T> implements PcapPacketHandler<String> { //Avoid excessive instantiations within endless loop private Tcp tcp = new Tcp(); // Preallocate a Tcp header private Ip4 ip = new Ip4(); // Preallocate a IP header private int size; @SuppressWarnings("unused") private T user; private static Logger log = Logger.getLogger(PHandler.class.getName()); public PHandler(T user) { this.setUser(user); } public void setUser(T user) { this.user = user; } @Override public void nextPacket(PcapPacket packet, String user) { if (packet.hasHeader(ip) && packet.hasHeader(tcp)) { if (log.isLoggable(Level.FINE)) { log.fine("Received packet len=" + String.valueOf(packet.getCaptureHeader().wirelen()) + " source_IP=" + FormatUtils.ip(ip.source()) + " source_port=" + String.valueOf(tcp.source()) + " destination_IP=" + FormatUtils.ip(ip.destination()) + " destination_port=" + String.valueOf(tcp.destination()));  } //preparing to send size = packet.size(); ByteBuffer byteBuffer = ByteBuffer.allocate(size); packet.transferTo(byteBuffer); IVTAP.bytesTransferred += size; if (IVTAP.bytesTransferred >= 8000000000000000000L) { IVTAP.bytesTransferred = 0L; } //sending UDP        byteBuffer.flip(); try { IVTAP.udpChannel.send(byteBuffer, IVTAP.dstaddr); } catch (IOException e) { log.log(Level.SEVERE, "Exception: ", e); } } } }

Listing 3

Before eternal pcap loop creation we should care about packet handler description (see Listing 3)

It works quite simple: we are reading the packet and sending it over UDP channel to the server side of ivTAP. ivTAP server receives this packet and sends it to the selected destination network adapter connected to physical SPAN port or just defined as a source interface in our application monitoring probe software (see Listing 4).

try { DatagramSocket listener = new DatagramSocket(bindsocketaddr); try { DatagramPacket udppacket = null; System.out.println("waitnig for packets..."); byte[] message = new byte[65536]; udppacket = new DatagramPacket(message, message.length); while (true) { try { listener.receive(udppacket); if (pcapOut.sendPacket(udppacket.getData(),0,udppacket.getLength()) == -1) { System.err.println(pcapOut.getErr()); if (pcapOut.getErr().equals("send: Message too long")) { System.err.println("Disable TCP segmentation offload at the source interface"); } return; } } catch (IOException e) { e.printStackTrace(); } } } finally { listener.close(); } } catch (SocketException e) { e.printStackTrace(); }

Listing 4

There is yet another important point we should consider. We have to create shutdown hook because the client and the server parts of the software use endless loop. If program terminates we have to close all the devices, threads and UDP channel.

The client part of the software controls bandwidth with dedicated thread. At regular intervals it checks amount of transmitted bytes and stops the program if the client generates too high network load.

Requirements and important considerations

· You must have administrative access rights to probe device operating system and to the hosts where you place the client part of ivTAP. Winpcap or libpcap must be installed at target hosts and at the probe device.

· You must disable the network offload feature at the machine where you deploy the client part of ivTAP.

· If you are going to send mirrored UDP flow over WAN link, you should consider available bandwidth and define safe thresholds in the client configuration.

· High network jitter between client and server parts of the software might introduce a noticeable error.

· When you sniff traffic directly from SPAN port it goes to receive (Rx) queue of the probe device network adapter. The server part of ivTAP can only put it into transmit (Tx) queue. Normally, probe device monitoring software doesn't care of it and takes all the traffic (at least it works well in my case). This might be an issue in the only case when your probe uses non-standard network driver. In this case a simple workaround might help: internal tunnel from one logical GRE interface to another one should solve the issue.

· If you use reverse-proxy technology between user and front-end server, you won’t be able to monitor real users’ network sessions. Eventually, you cannot measure round-trip time and packet loss rate (see Picture 4).

Picture 4

Security

Many times security related question was risen during our internal discussions. With ivTAP we actually duplicate traffic which, potentially, can contain sensitive information. Thus, if intruder can somehow listen to this traffic, all of the user’s data can be compromised. On the other hand, the ivTAP UDP data flow goes over our internal and protected links. If we assume the intruder has such access we must admit that all our environment already compromised and it already has much simpler and effective ways to get the desired information rather than inspecting UDP datagrams.

In any way, there are many relatively simple ways to encrypt data over UDP flow. They should be used if we for some reasons don’t trust our network links security.

Perspectives and outcomes

This experience helped me to better realize my capabilities. Sometimes we must create something new what suites our requirements the best. It doesn’t take much time and efforts but gives the best result and makes us learning.