Introduction

This article describes how to perform network traffic monitoring based on business criteria. First it describes the underlying technology, then it introduces some example business logic and finally the whole application is monitored using Wireshark network protocol analyser.

Background

The basic idea of the fine-grained business-oriented network traffic monitoring is to pair specific business logic with a specific port number.

Say, we want to distribute messages from the stock exchange to the traders. Let's simplify it a bit and consider only three distinct message feeds. Stock quotes are messages meant to inform trader about the current prices of the stock. Order confirmations let the trader know that his orders (to buy or sell the stock) were accepted by the exchange. Trades are notifications that the orders were executed (i.e. the stock was either bought or sold).

Each of these message feeds can be associated with a specific port number. In our example code stock quotes are transferred on port 33333, order confirmations on port 33334 and trades on port 33335.

IP port number being a part of TCP packet header is easy to monitor even on the low layers of the stack (either hardware of software). Monitoring program/device simply checks two bytes at exact offset in each TCP packet ("source port" field) and updates the statistics accordingly.

While this approach seems simple and self-evident, most business messaging solutions are incapable of it. For a discussion of traditional vs. ØMQ style of handling concurrent message feeds have a look at the conclusion of this article.

Business logic

To examine this functionality we are going to write two simple test programs simulating the communication between stock exchange and trader. monsend program plays the role of stock exchange and publishes three distinct feeds of messages. monrecv acts as a trader, receiving all the feeds. There can be several instances of monrecv (stock trader) running in parallel:

What follows is the monsend code. All the messages are 10 bytes long. We don't even care to fill in the message body as it's just an example so exact content of the message is irrelevant. Messages are sent at random, however, the program is tuned to send ~75% of stock quotes, ~20% of order confirmations and ~5% of trades. Also note the sleep period after each message (1ms). The intent is to get decent data flows to monitor rather then completely congested environment. 'host' command line argument specifies where zmq_server is running. 'out-interface' parameter specified which network interface should be used to send the data:

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

#include <zmq.hpp>
using namespace zmq;

int main (int argc, char *argv [])
{
    //  Parse the command line parameters.
    if (argc != 3) {
        printf ("usage: monsend <host> <out-interface>\n");
        return 1;
    }
    const char *host = argv [1];
    const char *out_interface = argv [2];

    //  Initialise 0MQ infrastructure.
    locator_t locator (host);
    dispatcher_t dispatcher (2);
    api_thread_t *api = api_thread_t::create (&dispatcher, &locator);
    i_thread *io = io_thread_t::create (&dispatcher);

    //  Set up the wiring.
    char buff [256];
    snprintf (buff, sizeof (buff), "%s:%d", out_interface, 33333);
    int quote_exchange = api->create_exchange (
        "Quotes", scope_global, buff, io, 1, &io);
    snprintf (buff, sizeof (buff), "%s:%d", out_interface, 33334);
    int confirmation_exchange = api->create_exchange (
        "Confirmations", scope_global, buff, io, 1, &io);
    snprintf (buff, sizeof (buff), "%s:%d", out_interface, 33335);
    int trade_exchange = api->create_exchange (
        "Trades", scope_global, buff, io, 1, &io);

    while (true) {

        //  Send up messages to different feeds based on 75%/20%/5% ratio.
        int r = rand () % 100;
        message_t msg (10);
        if (r < 75)
            api->send (quote_exchange, msg);
        else if (r < 95)
            api->send (confirmation_exchange, msg);
        else if (r < 100)
            api->send (trade_exchange, msg);

        //  Wait 1ms not to get into congestion.
        usleep (1000);
    }

    return 0;
}

As for the monrecv program, it's even simpler. It subscribes for all three message feeds. Then it retrieves the messages and drops them immediately as they arrive. 'host' command line argument specifies where zmq_server is running:

#include <stdio.h>
#include <stdlib.h>

#include <zmq.hpp>
using namespace zmq;

int main (int argc, char *argv [])
{
    //  Parse the command line parameters.
    if (argc != 2) {
        printf ("usage: monrecv <host>\n");
        return 1;
    }
    const char *host = argv [1];

    //  Initialise 0MQ infrastructure.
    locator_t locator (host);
    dispatcher_t dispatcher (2);
    api_thread_t *api = api_thread_t::create (&dispatcher, &locator);
    i_thread *io = io_thread_t::create (&dispatcher);

    //  Set up the wiring.
    int eid1 = api->create_queue ("Q");
    api->bind ("Quotes", "Q", io, NULL);
    api->bind ("Confirmations", "Q", io, NULL);
    api->bind ("Trades", "Q", io, NULL);

    //  Receive messages. No processing is done.
    while (true) {
        message_t msg;
        api->receive (&msg);
    }

    return 0;
}

Monitoring

First, run the application. Start zmq_server say at server001 box:

$ zmq_server

Run monsend application at server002 box, using eth0 (192.168.0.1) network interface to publish the feed:

$ monsend server001 eth0

Run monrecv application at server003 box:

$ monrecv server001

At this point the application is running, passing messages from monsend to monrecv. We can start monitoring the traffic.

Run wireshark at server002 box and start capturing packets on the network interface eth0.

Open the statistics window (Statistics | IO Graphs). Fill in appropriate filters to show the three feeds we are interested in - stock quotes at port 33333, order confirmations at port 33334, trades at port 33335:

Red line represents stock quotes, green line represents order confirmations, blue line represents trades. The statistics are charted using bytes per 0.1 second as a unit on y-axis (see the drop-down box in the bottom right corner of the window).

Now, let's start two more instances of monrecv at server002 box.

The I/O graph produced by Wireshark clearly shows that second instance was started at 10:17:53.8, while third instance began running at 10:17:57.8:

Results obtained by the monitoring can be used simply to be informed of the actual bandwidth requirements of the application. However, you can use them as well to analyse and improve overall design of the application data flow. For example, we may be concerned about the bandwidth consumed by stock quotes (red line). Results indicate that each running instance of monrecv requires approximately 50kB/0.1sec of bandwidth just to handle stock quotes. The fact can make us consider using PGM reliable multicast protocol instead of TCP for stock quotes - multicast would use constant amount of network bandwidth no matter how many instances of monrecv are running.

QoS and traffic shaping

The same principle applies to QoS and traffic shaping. Pairing business logic with ports gives you an opportunity to do traffic shaping based on business criteria. The big advantage is that it's done on networking level. Implmenting QoS on middleware level (as done in most messaging systems) proves inadequate once physical network issues are taken into account.

In our case, we can for example limit the amount of bandwidth assigned to stock quotes to 5120 kb/sec by configuring a network router (Cisco IOS):

access-list 100 permit tcp any any eq 33333                                     
class-map match-any port33333                                                   
  match access-group 100
policy-map port33333                                                            
  class port33333                                                               
   bandwidth 5120      
interface Ethernet0                                                             
 service-policy output port33333

This way stock quotes won't be able to overflow the network. Even if there's a trading peak, router itself will apply the QoS and prevent network outages experienced by the rest of your system.

Conclusion

The possibility of pairing business logic with port numbers gives you a powerful tool to monitor and analyse your business traffic.

Unfortunately, most traditional business messaging solutions don't offer similar feature. The problem is that in the traditional design all the applications speak to message broker using a single port (say port 5672 in the case of AMQP-based systems). While the implementations may be tweaked to use different ports, the tweaking wouldn't be easy. It would require the broker to listen on several ports, manage the listening ports in dynamic fashion, it would require clients to open several connections to the broker, it would require business applications to be rewritten to send messages to appropriate connections etc. With ØMQ all this is inherent part of design and available as a bonus.

Wireshark is available on most OS platforms. If it is not available on the platform you are using, you should be able to get an equivalent (though maybe not that potent) tool elsewhere.