RWPTOFLOW(1)

NAME
rwptoflow - Generate SiLK Flow records from packet data

SYNOPSIS
  rwptoflow [--plugin=PLUGIN [--plugin=PLUGIN ...]]
        [--active-time=YYYY/MM/DD:hh:dd:mm:ss.uuuuuu-YYYY/MM/DD:hh:dd:mm:ss.uuuuuu]
        [--flow-output=FLOW_PATH] [--packet-pass-output=PCKTS_PASS]
        [--packet-reject-output=PCKTS_REJECT]
        [--reject-all-fragments] [--reject-nonzero-fragments]
        [--reject-incomplete] [--set-sensorid=SCALAR]
        [--set-inputindex=SCALAR] [--set-outputindex=SCALAR]
        [--set-nexthopip=IP_ADDRESS] [--print-statistics]
        [--note-add=TEXT] [--note-file-add=FILE]
        [--compression-method=COMP_METHOD] TCPDUMP_INPUT

  rwptoflow [--plugin=PLUGIN ...] --help

  rwptoflow --version

DESCRIPTION
rwptoflow attempts to generate a SiLK Flow record for every IP packet in the pcap(3) (tcpdump(1)) capture file TCPDUMP_INPUT. TCPDUMP_INPUT must contain data captured from an Ethernet datalink.

rwptoflow does not attempt to reassemble fragmented packets or to combine multiple packets into a single flow record. rwptoflow is a simple program that creates one SiLK Flow record for every packet in TCPDUMP_INPUT. For packet reassembly and flow generation, consider using the rwp2yaf2silk(1) script. (That script requires that the yaf(1) program is installed on your system.)

rwptoflow will read from its standard input if TCPDUMP_INPUT is specified as "stdin". The SiLK Flow records are written to the specified flow-output file or to the standard output. The application will fail when attempting to read or write binary data from or to a terminal.

Packets outside of a user-specified active-time window can be ignored. Additional filtering on the TCPDUMP_INPUT can be performed by using tcpdump with an "expression" filter and piping tcpdump's output into rwptoflow.

In addition to generating flow records, rwptoflow can write pcap files containing the packets that it used to generate each flow, and/or the packets that were rejected. Note that packets falling outside the active-time window are ignored and are not written to the packet-reject-output.

Statistics of the number of packets read, rejected, and written can be printed.

rwptoflow will reject any packet that is not an IPv4 Ethernet packet and any packet that is too short to contain the Ethernet and IP headers. At the user's request, packets may be rejected when

•   they are fragmented---either the initial (zero-offset) fragment or a subsequent fragment

•   they have a non-zero fragment offset

•   they are not fragmented or they are the zero-fragment but the capture file does not contain enough information about the packet to set protocol-specific information---namely the ICMP type and code, the UDP source and destination ports, or the TCP source and destination ports and flags

Since the input packet formats do not contain some fields normally found in NetFlow data, rwptoflow provides a way to set those flow values in all packets. For example, it is possible to set the sensor-id manually for a tcpdump source, so that flow data can be filtered or sorted by that value later.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--plugin=PLUGIN
    Use the specified plug-in to ignore or reject packets or to modify the flow record that is generated from the packet. The switch may be repeated to load multiple plug-ins. See the "PLUG-IN SUPPORT" section below for details.

--active-time=YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]
--active-time=YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]-YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]
    Ignore all packets whose time falls outside the specified range. The times must be specified to at least day precision. The start time is required; when the end-time is not present, it is treated as infinite. The end-time will be rounded-up to instant before the next time unit; i.e., an end-time of 2006/08/31:15 is treated as 2006/08/31:15:59:59.999999.

--flow-output=FLOW_PATH
    Write the generated SiLK Flow records to the specified file at FLOW_PATH. When this switch is not provided, the flows are written to the standard output.

--packet-pass-output=PCKTS_PASS
    For each generated SiLK Flow record, write the packet that generated the flow to the pcap file specified by PCKTS_PASS. Use "stdout" to write the packets to the standard output.

--packet-reject-output=PCKTS_REJECT
    Write each packet that occurs within the active-time window but for which a SiLK Flow record was not generated to the pcap file specified by PCKTS_REJECT. Use "stdout" to write the packets to the standard output.

    The packets that get written to this file may include packets that were shorter than that required to get the IP header, non-IPv4 packets, and packets that get treated as "reject" packets by the following switches.

--reject-all-fragments
    Do not generate a SiLK Flow record for the packet when the packet is fragmented. This includes the initial (zero-offset) fragment and all subsequent fragments. If --packet-reject-output is specified, the packet will be written to that file.

--reject-nonzero-fragments
    Do not generate a SiLK Flow record for the packet when the packet is fragmented unless this is the initial fragment. That is, reject all packets that have a non-zero fragmentation offset. Normally flow records are generated for these packets, but the ports and TCP flag information is set to zero. If --packet-reject-output is specified, the packet will be written to that file.

--reject-incomplete
    Do not generate a SiLK Flow record for the packet when the packet's fragmentation-offset is zero yet the packet does not contain enough information to completely specify an ICMP, UDP, or TCP record (that is, the packet is too short to set the ICMP type and code, the UDP or TCP source or destination port, or the TCP flags). Normally, flow records are generated for these packets but the ports and TCP flag information is set to zero. This switch has no effect on packets where the protocol is not 1,6, or 17.

    This switch does not imply --reject-nonzero-fragments; to indicate that all generated flow records must have valid port and TCP flag information, specify --reject-nonzero-fragments --reject-incomplete.

--set-sensorid=SCALAR
    Set the sensor ID for all flows to SCALAR. SCALAR should be an integer value between 0 and 65534, inclusive. When not specified, the sensor ID is set to 65535.

--set-inputindex=SCALAR
    Set the input SNMP index value for all flows to SCALAR. SCALAR should be an integer value between 0 and 65535, inclusive. When not specified, the SNMP input is set to 0.

--set-outputindex=SCALAR
    Set the output SNMP index value for all flows to SCALAR. SCALAR should be an integer value between 0 and 65535, inclusive. When not specified, the SNMP output is set to 0.

--set-nexthopip=IP_ADDRESS
    Set the next-hop IP address for all flows to IP_ADDRESS; IP_ADDRESS may be in its canonical form or an integer. When not specified, the next-hop IP is set to 0.0.0.0.

--print-statistics
    Print a summary of the packets that were processed. This summary includes

    •   the total number of packets read

    •   the number that fell outside the time-window

    •   the number that were too short to get the IP header

    •   the number that were not IPv4

    •   the number that were discarded by a plug-in

    •   the total number of fragmented packets

    •   the number of fragments where the offset was zero

    •   the number of zero-offset packets that were incomplete

    •   the number of flows written to the output

--note-add=TEXT
    Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME
    Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--compression-method=COMP_METHOD
    Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

    none
        Do not compress the output using an external library

    zlib
        Use the zlib(3) library for compressing the output

    lzo1x
        Use the lzo1x algorithm from the LZO real time compression library for compression

    best
        Use whichever available method gives the "best" compression in general, though not necessarily the "best" for this particular output.

--help
    Print the available options and exit. Options that add fields can be specified before --help so that the new options appear in the output.

--version
    Print the version number and information about how SiLK was configured, then exit the application.

--dynamic library
    This switch is deprecated. It is an alias for --plugin.

PLUG-IN SUPPORT
rwptoflow allows the user to provide additional logic to ignore or reject packets, or to modify the flow record that is generated from the packet. To do this, the user creates a plug-in that gets loaded at run-time by giving rwptoflow the --plugin switch with the path to the plug-in as the parameter to the switch.

A plug-in is a shared object file (a.k.a. dynamic library) that is compiled from C source code. The plug-in should have four subroutines defined:

setup()
    is called when the object is first loaded. This is the place to initialize global variables to their default values. If the plug-in provides switches of its own, they must be registered in this subroutine.

initialize()
    gets called after all options have been processed but before any packets are read from the input. If this subroutine does not return 0, the application will quit.

ptoflow()
    will be called for every packet that rwptoflow is able to convert into a flow record just before the flow record is written. This subroutine will not see packets that are short or that are not IPv4; it will also not see fragmented packets if --reject-all-fragments is specified.

    The "ptoflow()" function is called with two parameters:

    •   a pointer to the "rwRec" object that rwptoflow created from the packet. The subroutine may modify the record as it sees fit.

    •   a void pointer that the function may cast to a pointer to the C structure:

          typedef struct _sk_pktsrc_t {
              /* the source of the packets */
              pcap_t                     *pcap_src;
              /* the pcap header as returned from pcap_next() */
              const struct pcap_pkthdr   *pcap_hdr;
              /* the packet as returned from pcap_next() */
              const u_char               *pcap_data;
          } sk_pktsrc_t;

        This structure gives the user access to all the information about the packet.

    The following return values from "ptoflow()" determines whether rwptoflow writes the flow and the packet:

    0   Write the flow record to the flow-output and the packet to the PCKTS_PASS unless another plug-in instructs otherwise.

    1   Write the flow record to the flow-output and the packet to the PCKTS_PASS immediately; do not call the "ptoflow()" routine on any other plug-in.

    2   Treat the packet as a reject: Do not write the flow record; write the packet to the PCKTS_REJECT immediately; do not call the "ptoflow()" routine on any other plug-in.

    3   Ignore the packet immediately: Do not write the flow record nor the packet; do not call the "ptoflow()" routine on any other plug-in.

    If "ptoflow()" returns any other value, the rwptoflow application will terminate with an error.

teardown()
    is called as the application exits. The user can use this routine to print results and to free() any data structures that were used.

rwptoflow uses the following rules to find the plug-in: When PLUGIN contains a slash ("/"), rwptoflow assumes the path to PLUGIN is correct. Otherwise, rwptoflow will attempt to find the file in $SILK_PATH/lib/silk, $SILK_PATH/share/lib, $SILK_PATH/lib, and in these directories parallel to the application's directory: lib/silk, share/lib, and lib. If rwptoflow does not find the file, it assumes the plug-in is in the current directory. To force rwptoflow to look in the current directory first, specify --plugin=./PLUGIN. When the SILK_PLUGIN_DEBUG environment variable is non-empty, rwptoflow prints status messages to the standard error as it tries to open each of its plug-ins.

EXAMPLES
Given the pcap capture file data.pcap, convert it to a SiLK flow file:

  rwptoflow data.pcap --packet-pass=good.pcap --flow-out=data.rw

Filter the SiLK flows---passing those records whose source IPs are found in the IPset file sip.set:

  rwfilter --sipset=sip.set --pass=filtered.rw  data.rw

Match the original pcap file against the filtered SiLK file, in effect generating a pcap file which has been filtered by sip.set:

  rwpmatch --flow-file=filtered.rw good.pcap > filtered.pcap

ENVIRONMENT

SILK_PLUGIN_DEBUG
    When set to 1, rwsort print status messages to the standard error as it tries to open each of its plug-ins.

SEE ALSO
rwpmatch(1), rwfilter(1), rwpdedupe(1), rwp2yaf2silk(1), yaf(1), tcpdump(1), pcap(3), mergecap(1)

NOTES
SiLK supports millisecond timestamps. When reading packets whose timestamps have finer precision, the times are truncated at the millisecond position.

The mergecap(1) or rwpdedupe(1) programs can be used to join multiple tcpdump capture files in order to convert into a single flow file.