Initialization

Of all the things in this manual, this is the one you should not forget :)

Before you can do anything with it, you need to initialize libnetdude by calling libnd_init(). This is necessary in order to initialize the various subsystems such as libpcapnav, preferences, several registries, etc.

Trace Files

Trace files are libnetdude's bread and butter. They are represented by instances of LND_Trace, which maintain state for a trace file the user is manipulating. State information consists of consistency management data, filter settings, packet iteration configuration, file modification status and more. See libnd_trace.h for details.

Instantiating a trace structure is done using libnd_trace_new(), passing it the path of the trace file to load or NULL when you want to create a new file. Saving is done using libnd_trace_save() and libnd_trace_save_as(), and releasing a trace is done using libnd_trace_free().

When you open a trace file, libnetdude does not load any packets. If you want to load any packets, from what part of the file, and how many packets, is entirely up to you. This is explained in detail in the section on trace parts below.

To allow applications to integrate libnetdude well, you can register trace observers. These observers will then receive notifications when certain events occur on a trace file. A good example are GUI-based applications — these can register an observer to update the GUI whenever the trace is modified, for example. Trace observers are created using libnd_trace_observer_new(). The different possible events are defined in the LND-TraceObserverOp enumeration. The actual observer structures contain function pointers to the various event callbacks. These pointers are initialized to safe defaults when a new observer is created. You then hook different implementations into this structure, and register it with a trace using libnd_trace_add_observer(). Events are pushed to the observers using libnd_trace_tell_observers().

To associate arbitrary data items with a trace, every trace comes with a simple key/value-based data storage, accessible through libnd_trace_get_data(), libnd_trace_set_data(), and libnd_trace_del_data(). libnetdude only stores and retrieves the data objects — you have to do any cleaning-up yourself before releasing a trace.

Protocols and Protocol Plugins

libnetdude protocols make packet handling easy. Each protocol encodes knowledge of the protocol structure found in a packet, such as the protocol header, protocol data length etc. The more protocols libnetdude can access, the more detailed the interpretation of the raw packet data in packets becomes. In libnetdude, protocol knowledge is provided in protocol plugins, to allow support for new protocols to be added in a modular fashion.

By default, libnetdude ships with protocol plugins for ARP, Ethernet, FDDI, ICMP, IP, Linux SLL, SNAP headers, TCP, and UDP. Everything else is interpreted as raw protocol data. More protocol plugins are always welcome! For details, see the chapter on how to write a protocol plugin below.

Protocol plugins get picked up, initialized and registered automatically when libnetdude is bootstrapped. Each protocol is registered in the protocol registry, from which you can retrieve protocols by providing the layer in the protocol stack that the protocol resides in (as defined through the LND_ProtocolLayer enumeration), and the magic value that the protocol is commonly known under. At the link level, these are the DLT_xxx values as defined in /usr/include/net/bpf.h, at the network layer the ETHERTYPE_xxx values of /usr/include/net/ethernet.h, at the transport layer the IPPROTO_xxx values of /usr/include/net/in.h, and at the application level, these are the well-known port numbers as found in /etc/services. We'll look at an example of how to use this registry once we have introduced libnetdude's packets.

Packets

Packets are instances of type LND_Packet. When requested, libnetdude initializes packets so that they contain detailed information about the nesting of the protocols contained in them. This depends on the number of protocols available to the library. libnetdude's packet type also contains the raw packet data and packet header as provided by libpcap.

You normally do not need to create and initialize packets manually. Typically, you will either iterate over the packets in a trace area and obtain the initialized current packet from a packet iterator, or you will ask libnetdude to load a number of packets into memory for you and then directly operate in that sequence of loaded and initialized packets. If you're interested in how packet initialization works, or if you want to add support for a new protocol, you should have a look at the chapter that explains how to write a protocol plugin.

Initialized packets make it easy to access protocol information of a given protocol at a given nesting level. You can easily obtain the first nested IP header from a packet, for example. Forget writing your own protocol-demuxer.

Now that protocols and packets are introduced, let's look at some code. Let's assume that we want to obtain the TCP header of a packet. We first retrieve the TCP protocol from the protocol registry.

#include <libnd.h> #include <netinet/in.h> #include <netinet/tcp.h> LND_Trace *trace; LND_Packet *packet; LND_Protocol *tcp; struct tcphdr *tcphdr; libnd_init(); /* Open a tracefile, load a few packets ... */ if (! (trace = libnd_trace_new("foo.trace"))) { /* Didn't work -- appropriate error handling. */ } libnd_tpm_load_packets(trace->tpm); /* Obtain the packet somehow: */ packet = libnd_trace_get_packets(trace); if (! (tcp = libnd_proto_registry_find(LND_PROTO_LAYER_TRANS, IPPROTO_TCP))) { /* Protocol not found -- handle accordingly. */ }

Then, we try to obtain the TCP header from a packet by using the TCP protocol just obtained. If %NULL is returned, we know that the packet does not contain TCP data.

if (! (tcphdr = (struct tcphdr *) libnd_packet_get_data(packet, tcp, 0))) { /* This packet does not contain TCP headers */ }

Similarly to traces, applications can register observers to be informed when certain operations are performed on a packet. These operations are enumerated in the LND_PacketObserverOp enum.

Since packets may be allocated and released quickly depending on the application, the memory management fir libnetdude's packets is wrapped by a packet recycler that keeps packets that are no longer used in sets of packets needing up to a certain number of bytes for their raw data. Packet allocation is then performed by first checking if an unused packet with enough storage capacity is around anyway. Packet deallocation is performed by putting the packet into the pool of unused packets of similar storage capacity. This is all hidden in libnetdude, so you do not need to take care of this yourself.

TODO: explain how to iterate over all protocol headers in a trace via packet->pd, and show how to print out TCP payloads as an example.

Trace Areas, Trace Parts and Trace Part Managers

As mentioned above, when opening a trace file libnetdude does not load the packets contained in the file into memory, because this approach is clearly bound to fail for larger traces -- and tcpdump traces often are hundreds of megabytes in size. It can load up to a configurable number of packets from a specified point in the file upon request, but typically you will just iterate over packets in a certain trace area, loading them one at a time to perform some operation on them. In code, trace areas are instances of LND_TraceArea. Every trace has an active trace area. At creation time of the trace, this trace area is initialized to the entire trace. Trace areas are specified in one of two ways; either in terms of start- and end timestamps (e.g., from 08:00:00 to 08:10:30 on August 4) or start- and end fractions (e.g., from 0.5 to 1.0, that is, from the middle of the file to its end). You can set and obtain this area using libnd_trace_set_area() and libnd_trace_get_area().

Iterating over the packets in a trace area and modifying them creates a trace part (in code: instances of type LND_TracePart) — a temporary overlay over the original trace with its extents defined by the trace area. When a multiple different trace areas are used, trace parts begin to pile up on the original input trace file, called the base file. Figure 1 illustrates the concepts of the base file, trace areas and trace parts.

Making sure that the trace parts are properly maintained, overlayed and disposed of is the job of the trace part manager, abbreviated "TPM". Each trace has one. In code, a TPM is an instance of type LND_TPM. TPMs organize the trace parts in such a way that the user always sees a consistent view of the modified trace, take care of properly sequencing the packets when a trace file that has developed multiple trace parts is saved back to disk (illustrated in Figure 2), and handle automatically saving trace part contents out to temporary files when necessary.

Intuitively, there are multiple ways to specify a location in a trace: global offsets simply give an offset in terms of a number of bytes from the start of the file. Timestamps specify the packet in the trace that is closest to a given timestamp. Fractional offsets are offsets relative to the entire size of the trace (for example, 0.5 is the middle of the trace). Keeping Figure 1 in mind, a location can also be defined as a trace part + local offset. The TPM provides a set of functions to convert from one representation to another. Examples of this API include libnd_tpm_map_loc_to_offset(), libnd_tpm_map_offset_to_loc(), libnd_tpm_map_fraction_to_loc(), and libnd_tpm_map_timestamp_to_loc().

Trace parts can grow and shrink — after all, without this feature it would be fine to simply mmap() parts of the trace file. When a trace area is iterated and packets are written to temporary storage, the user is free to not write out all packets or introduce new ones. The TPMs keeps track of where the trace areas begin and end, and by what amount trace parts grow or shrink, relative to their original size.

Packet Iterators

Iterating over packets in libpcap is done using either a callback mechanism that allows only little control over the iteration, or using pcap_next() to get a single packet. libnetdude provides a higher abstraction for iterating packets. The packet iterator API allows you to use common for (...) { } loops and provides one function for each of the three for-loop steps (initialization, exit checking and counter adjustment). Packet iterators are instances of type LND_PacketIterator and are statically initialized. Avoiding allocating and releasing an iterator structure every time an iteration is performed is easier for the programmer, given that there is no automatic garbage collection in C.

There are multiple ways you can iterate over packets in a trace. The iteration mode is specified using one of the LND_PacketIteratorMode constants when the iterator is initialized:

• LND_PACKET_IT_SEL_R assumes that you have loaded a number of packets into memory, starting at the current location in the trace (this is done using the libnd_tpm_load_packets() call). Within the loaded packets, you can select individual packets using libnd_tp_select_packet(). This iterator mode will only iterate over those packets that are currently in memory and have been selected, and assumes that no modifications to these packets are made.

• LND_PACKET_IT_SEL_RW is similar to LND_PACKET_IT_SEL_R but allows modifications to the iterated packets.

• LND_PACKET_IT_PART_R is used to request iteration of all packets currently loaded into memory and assumes that no modifications are made.

• LND_PACKET_IT_PART_RW is similar to LND_PACKET_IT_PART_R, but allows modifications to the iterated packets.

• LND_PACKET_IT_AREA_R is used to request iteration over packets in a trace area, without any modifications to the packets.

• LND_PACKET_IT_AREA_RW is similar to LND_PACKET_IT_AREA_R, but allows modifications to the iterated packets. The difference between the read-only and the read-write mode is bigger here compared to the other options, because iteration of a trace area in read-write mode will lead to the creation of a new trace area, whereas the other mode will not.

#include <libnd.h>

LND_Trace          *trace;
LND_PacketIterator  pit;   /* NOT a pointer */
LND_TraceArea       area;

libnd_init();

/* Open a tracefile: */
if (! (trace = libnd_trace_new("foo.trace")))
  {
     /* Didn't work -- appropriate error handling. */
  }

/* Now set the trace area to the second half of the trace.
 * The default iteration mode of a trace is LND_PACKET_IT_AREA_R,
 * so we can either set it for the trace or just for the iterator.
 * Here we do the latter.
 */
libnd_trace_area_init_space(&area, 0.5, 1.0);
libnd_trace_set_area(trace, &area);
        

for (libnd_pit_init_mode(&pit, trace, LND_PACKT_IT_AREA_RW); libnd_pit_get(&pit); libnd_pit_next(&pit))
  {
     /* Now just rely on the library and the protocol plugins to fix the packet
      * for us if it's currently broken:
      */
     libnd_packet_fix(libnd_pit_get(&pit));
  }
        

Note that if you have other conditions in your loop that may lead to breaking out of the loop, you MUST call libnd_pit_cleanup() when doing so. The packet iterators normally do this for you when they hit the end of the iteration, but when you stop the iteration before that, you have to take care of that.

Packet Filters

Packet iteration is nice but only helpful when you can skip the packets you don't care about. libnetdude provides a filtering framework that allows stateful packet filtering where the actual filtering condition is entirely up to you.

Packet filters in libnetdude are instances of type LND_Filter. Their implementation depends on function pointers you provide for initialization, filtering, and cleanup. Every filter has a name, and possible state-keeping data that are used during the filtering process. A new filter is created using libnd_filter_new().

Keep in mind that if you do not require some of the callbacks (like when your filtering is not stateful), you do not need to define and pass all of the callbacks.

A packet passes a filter when your filtering function returns TRUE for it.

#include <libnd.h>

LND_Trace          *trace;
LND_PacketIterator  pit;   /* NOT a pointer */
LND_TraceArea       area;

libnd_init();

/* Open a tracefile: */
if (! (trace = libnd_trace_new("foo.trace")))
  {
     /* Didn't work -- appropriate error handling. */
  }
        

#define PACKET_SIZE_LIMIT         100

gboolean
filter_length(LND_Filter *filter, LND_Packet *packet, void *filter_data)
{
  return (packet->ph.len <= PACKET_SIZE_LIMIT) ? TRUE : FALSE;
}
        

LND_Filter         *filter;

if (! (filter = libnd_filter_new("size filter",
                                 NULL, /* no initialization needed */
				 filter_length,
				 NULL, /* no cleanup needed */
				 NULL, /* no state, so no state cleanup needed */
				 NULL))) /* no statekeeping needed */
  {
     /* Didn't work -- appropriate error handling. */
  }

libnd_trace_add_filter(trace, filter);

for (libnd_pit_init(&pit, trace); libnd_pit_get(&pit); libnd_pit_next(&pit))
  printf("Packet with header length %i passed the filter\n", libnd_pit_get(&pit)->ph.len);
        

There are two other concepts for filtering: the filter registry stores created filters and can retrieve them by name. Filter factories add one level of indirection and provide a unique starting point for filter generation. They are still experimental and likely to change in the future.

Preferences Management

libnetdude does preferences management for you. Preferences are partitioned by a namespace system, where each set of common preference options is put in the same preferences domain. domain, you can define a set of apply callbacks that get called when preference modifications are applied (using libnd_prefs_apply()). You can save the current preference settings to disk at any time using libnd_prefs_save().

You create a new preferences domain using libnd_prefs_add_domain()). Besides a name for the new domain, this function is passed an array of preference settings. This array can be defined statically, using an array of instances of type LND_PrefsEntry. Here is how libnetdude defines its own preferences:

static LND_PrefsEntry prefs_entries_netdude[] = { { "tcpdump_path", LND_PREFS_STR, LND_UNUSED, LND_UNUSED, TCPDUMP }, { "tcpdump_resolve", LND_PREFS_INT, FALSE, LND_UNUSED, LND_UNUSED }, { "tcpdump_print_domains", LND_PREFS_INT, FALSE, LND_UNUSED, LND_UNUSED }, { "tcpdump_quick", LND_PREFS_INT, FALSE, LND_UNUSED, LND_UNUSED }, { "tcpdump_print_link", LND_PREFS_INT, FALSE, LND_UNUSED, LND_UNUSED }, { "tcpdump_print_timestamp", LND_PREFS_INT, FALSE, LND_UNUSED, LND_UNUSED }, { "workdir", LND_PREFS_STR, LND_UNUSED, LND_UNUSED, "/tmp" }, { "num_mem_packets", LND_PREFS_INT, 500, LND_UNUSED, LND_UNUSED }, };

Preferences are then queried and set using libnd_prefs_get_str_item()) and libnd_prefs_set_str_item()) (and corresponding calls for integer and floating point settings). To obtain the directory in which libnetdude stores its temporary data, you would use the following code:

char *workdir; if (! libnd_prefs_get_str_item(LND_DOM_NETDUDE, "workdir", &workdir)) { /* Error -- preference setting undefined. */ } printf("libnetdude's temporary storage area is %s\n", workdir);

The relationship between setting preferences, applying preferences, and saving preferences is as follows: preferences are modified in memory when you use one of the libnd_prefs_set_xxx() functions. Those modifications are lost when the application exits, unless you call libnd_prefs_save() to save them to disk. Calling libnd_prefs_apply() only causes the registered application callbacks to be called, so that other parts of the application can synchronize their operation to the new settings.

tcpdump Communication

libnetdude associates a tcpdump process with every trace. The programmer can send individual packets to this tcpdump instance and obtain the familiar tcpdump output for that packet (limited however to a single line -- this is one of the many occasions where the fact that some of tcpdump's protocol parsers output more than a single line per packet makes things a royal pain in the arsenal).

The function you want to look at for doing this is libnd_tcpdump_get_packet_line(). You can also specify the command line options passed to the tcpdump process when it is forked, using libnd_tcpdump_options_add(). and libnd_tcpdump_options_reset(). The default options that are always passed are -l -r. Depending on you preferences settings in the LND_DOM_NETDUDE domain, other flags are sent as well: -nnn depending on tcpdump_resolve, -N depending on tcpdump_domains, -q depending on tcpdump_quick, and -e depending onetcpdump_print_link.

For an example of the tcpdump interface being used, please have a look at the source code of the Netdude application that uses this feature to display a list of tcpdump output strings for the set of packets that are loaded into memory.

Feature Plugins

Feature plugins are libnetdude's mechanism to provide reusable packet manipulation modules that can contain arbitrary code, use other libraries, interface to other applications etc. Examples are advanced filter implementations (like "drop all incomplete TCP flows"), address mappers, or statistical analyzers. For details on writing a feature plugin, see the separate chapter on Feature Plugins.