RWCUT(1)

NAME
rwcut - Print selected fields of binary SiLK Flow records

SYNOPSIS
  rwcut [{--fields=FIELDS | --all-fields}]
        [--start-rec-num=START_NUM] [--end-rec-num=END_NUM]
        [--num-recs=REC_COUNT] [--dry-run] [--icmp-type-and-code]
        [--epoch-time] [{--integer-ips | --zero-pad-ips}]
        [--integer-sensors] [--integer-tcp-flags]
        [--no-titles] [--no-columns] [--column-separator=CHAR]
        [--no-final-delimiter] [{--delimited | --delimited=CHAR}]
        [--print-filenames] [--copy-input=PATH] [--output-path=PATH]
        [--pager=PAGER_PROG] [--site-config-file=FILENAME]
        [--ipv6-policy={ignore,asv4,mix,force,only}]
        [{--legacy-timestamps | --legacy-timestamps=NUM}]
        [--plugin=PLUGIN [--plugin=PLUGIN ...]]
        [--python-file=PATH [--python-file=PATH ...]]
        [--pmap-file=MAPNAME:PATH [--pmap-file=MAPNAME:PATH ...]]
        [--pmap-column-width=NUM]
        [FILES...]

  rwcut [--pmap-file=MAPNAME:PATH [--pmap-file=MAPNAME:PATH ...]]
        [--plugin=PLUGIN ...] [--python-file=PATH ...] --help

  rwcut --version

DESCRIPTION
rwcut reads binary SiLK Flow records from files listed on the command line or from the standard input and prints the records to the screen in a textual, bar ("|") delimited format. See the EXAMPLES section below for sample output.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--fields=FIELDS
    FIELDS contains the list of flow attributes (a.k.a. fields or columns) to print. The columns will be displayed in the order the fields are specified. Fields may be repeated. FIELDS is a comma separated list of field-names, field-integers, and ranges of field-integers; a range is specified by separating the start and end of the range with a hyphen (-). Field-names are case-insensitive. Example:

     --fields=stime,10,1-5

    If the --fields switch is not given, FIELDS defaults to:

     sIP,dIP,sPort,dPort,protocol,packets,bytes,flags,sTime,dur,eTime,sensor

    The complete list of built-in fields that the SiLK tool suite supports follows, though note that not all fields are present in all SiLK file formats; when a field is not present, its value is 0.

    sIP,1
        source IP address

    dIP,2
        destination IP address

    sPort,3
        source port for TCP and UDP, or equivalent

    dPort,4
        destination port for TCP and UDP, or equivalent

    protocol,5
        IP protocol

    packets,pkts,6
        packet count

    bytes,7
        byte count

    flags,8
        bit-wise OR of TCP flags over all packets

    sTime,9
        starting time of flow (millisecond resolution unless the --legacy-timestamps switch is specified)

    dur,10
        duration of flow (millisecond resolution unless the --legacy-timestamps switch is specified)

    eTime,11
        end time of flow (millisecond resolution unless the --legacy-timestamps switch is specified)

    sensor,12
        name or ID of sensor at the collection point

    class,20
        class of sensor at the collection point

    type,21
        type of sensor at the collection point

    sTime+msec,22
        starting time of flow including milliseconds (milliseconds are always displayed)

    eTime+msec,23
        end time of flow including milliseconds (milliseconds are always displayed)

    dur+msec,24
        duration of flow including milliseconds (milliseconds are always displayed)

    icmpTypeCode,25
        include two columns, "iType" and "iCode" that contain the ICMP type and code for ICMP flows; for non-ICMP flows, these columns are empty

    Many SiLK file formats do not store the following fields and their values will always be 0; they are listed here for completeness:

    in,13
        router SNMP input interface or vlanId if packing tools were configured to capture it (see sensor.conf(5))

    out,14
        router SNMP output interface or portVlanId

    nhIP,15
        router next hop IP

    SiLK can store flows generated by enhanced collection software that provides more information than NetFlow v5. These flows may support some or all of these additional fields; for flows without this additional information, the field's value is always 0.

    initialFlags,26
        TCP flags on first packet in the flow

    sessionFlags,27
        bit-wise OR of TCP flags over all packets except the first in the flow

    attributes,28
        flow attributes set by the flow generator:

        "F" flow generator saw additional packets in this flow following a packet with a FIN flag (excluding ACK packets)

        "T" flow generator prematurely created a record for a long-running connection due to a timeout. (When the flow generator yaf(1) is run with the --silk switch, it will prematurely create a flow and mark it with "T" if the byte count of the flow cannot be stored in a 32-bit value.)

        "C" flow generator created this flow as a continuation of long-running connection, where the previous flow for this connection met a timeout (or a byte threshold in the case of yaf).

        Consider a long-running ssh session that exceeds the flow generator's active timeout. (This is the active timeout since the flow generator creates a flow for a connection that still has activity). The flow generator will create multiple flow records for this ssh session, each spanning some portion of the total session. The first flow record will be marked with a "T" indicating that it hit the timeout. The second through next-to-last records will be marked with "TC" indicating that this flow both timed out and is a continuation of a flow that timed out. The final flow will be marked with a "C", indicating that it was created as a continuation of an active flow.

    application,29
        guess as to the content the flow. Some software that generates flow records from packet data, such as yaf, will inspect the contents of the packets that make up a flow and use traffic signatures to label the content of the flow. SiLK calls this label the application; yaf refers to it as the appLabel. The application is the port number that is traditionally used for that type of traffic (see the /etc/services file on most UNIX systems). For example, traffic that the flow generator recognizes as FTP will have a value of 21, even if that traffic is being routed through the standard HTTP/web port (80).

    The following fields provide a way to label the IPs or ports on a record. These fields require external files to provide the mapping from the IP or port to the label:

    sType,16
        for the source IP address, the value 0 if the address is non-routable, 1 if it is internal, or 2 if it is routable and external. Uses the mapping file specified by the SILK_ADDRESS_TYPES environment variable, or the address_types.pmap mapping file, as described in addrtype(3).

    dType,17
        as sType for the destination IP address

    scc,18
        for the source IP address, a two-letter country code abbreviation denoting the country where that IP address is located. Uses the mapping file specified by the SILK_COUNTRY_CODES environment variable, or the country_codes.pmap mapping file, as described in ccfilter(3). The abbreviations are those used by the Root-Zone Whois Index (see for example http://www.iana.org/cctld/cctld-whois.htm <http://www.iana.org/cctld/cctld-whois.htm>) or the following special codes: -- N/A (e.g. private and experimental reserved addresses); a1 anonymous proxy; a2 satellite provider; o1 other

    dcc,19
        as scc for the destination IP

    src-MAPNAME
        label determined by passing the source IP or the protocol/source-port to the user-defined mapping defined in the prefix map associated with MAPNAME. See the description of the --pmap-file switch below and the pmapfilter(3) manual page.

    dst-MAPNAME
        as src-MAPNAME for the destination IP or protocol/destination-port.

    sval
    dval
        These are deprecated field names created by pmapfilter that correspond to src-MAPNAME and dst-MAPNAME, respectively. These fields are available when a prefix map is used that is not associated with a MAPNAME.

    Finally, the list of built-in fields may be augmented by the run-time loading of PySiLK code or plug-ins written in C (also called shared object files or dynamic libraries), as described by the --python-file and --plugin switches.

--all-fields
    Instruct rwcut to print all known fields. This switch cannot be combined with the --fields switch. This switch suppresses error messages from the plug-ins.

--plugin=PLUGIN
    Augment the list of fields by using run-time loading of the plug-in (shared object) whose path is PLUGIN. The creation of these plug-ins is beyond the scope of this manual page. When PLUGIN contains a slash ("/"), rwcut assumes the path to PLUGIN is correct. Otherwise, rwcut will attempt to find the file in $SILK_PATH/lib/silk, $SILK_PATH/share/lib, $SILK_PATH/lib, and in these directories parallel to the application's directory: lib/silk, share/lib, and lib. If rwcut does not find the file, it assumes the plug-in is in the current directory. To force rwcut to look in the current directory first, specify --plugin=./PLUGIN. When the SILK_PLUGIN_DEBUG environment variable is non-empty, rwcut prints status messages to the standard error as it tries to open each of its plug-ins.

--start-rec-num=START_NUM
    Begin printing with the START_NUM'th record by skipping the first START_NUM-1 records. The default is 1; that is, to start printing at the first record; START_NUM must be a positive integer. If START_NUM is greater than the number of input records, the only output will be the title. When using multiple input files, records are treated as a single stream for the purposes of the --start-rec-num, --end-rec-num, and --num-recs switches. This switch does not affect the records written to the stream specified by --copy-input.

--end-rec-num=END_NUM
    Stop printing after the END_NUM'th record. When END_NUM is 0, the default, printing stops once all input records have been printed; that is, END_NUM is effectively infinity. If this value is non-zero, it must not be less than START_NUM. When using multiple input files, records are treated as a single stream for the purposes of the --start-rec-num, --end-rec-num, and --num-recs switches. This switch does not affect the records written to the stream specified by --copy-input.

--num-recs=REC_COUNT
    Print no more than REC_COUNT records. Specifying a REC_COUNT of 0 will print all records, which is the default. This switch is ignored either when both --start-rec-num and --end-rec-num are specified or when only --end-rec-num is given and END_NUM is less than REC_COUNT. When using multiple input files, records are treated as a single stream for the purposes of the --start-rec-num, --end-rec-num, and --num-recs switches. This switch does not affect the records written to the stream specified by --copy-input.

--dry-run
    Causes rwcut to print the column headers and exit. Useful for testing.

--icmp-type-and-code
    Unlike TCP or UDP, ICMP messages do not use ports, but instead have types and codes. Specifying this switch will cause rwcut to print, for ICMP records, the message's type and code in the sPort and dPort columns, respectively. The use of this switch is discouraged; use the icmpTypeCode field instead.

--epoch-time
    Print timestamps as epoch time (number of seconds since midnight GMT on 1970-01-01).

--integer-ips
    Print IPs as integers. By default, IP addresses are printed in their canonical form.

--zero-pad-ips
    Print IP addresses in their canonical form, but add zeros to the IP address so it fully fills the width of column. For IPv4, use three digits per octet, e.g, 127.000.000.001. For IPv6, use four digits per hexadectet and expand empty hexadectets, e.g.; "0000:0000:0000:0000:0000:FFFF:FF00:0001".

--integer-sensors
    Print the integer ID of the sensor rather than its name.

--integer-tcp-flags
    Print the TCP flag fields (flags, initialFlags, sessionFlags) as an integer value. Typically, the characters "F,S,R,P,A,U,E,C" are used to represent the TCP flags.

--no-titles
    Turn off column titles. By default, titles are printed.

--no-columns
    Disable fixed-width columnar output.

--column-separator=C
    Use specified character between columns and after the final column. When this switch is not specified, the default of '|' is used.

--no-final-delimiter
    Do not print the column separator after the final column. Normally a delimiter is printed.

--delimited
--delimited=C
    Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.

--print-filenames
    Print to the standard error the names of input files as they are opened.

--copy-input=PATH
    Copy all binary input to the specified file or named pipe. PATH can be "stdout" to print flows to the standard output as long as the --output-path switch has been used to redirect rwcut's ASCII output.

--output-path=PATH
    Determines where the output of rwcut (ASCII text) is written. If this option is not given, output is written to the standard output.

--pager=PAGER_PROG
    When output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the value of the pager is determined to be the empty string, no paging will be performed and all output will be printed to the terminal.

--ipv6-policy=POLICY
    Determine how IPv4 and IPv6 flows are handled when SiLK has been compiled with IPv6 support. When the switch is not provided, the SILK_IPV6_POLICY environment variable is checked for a policy. If it is also unset or contains an invalid policy, the POLICY is mixed. When SiLK has not been compiled with IPv6 support; IPv6 flows are always ignored, regardless of the value passed to this switch or in the SILK_IPV6_POLICY variable. The supported values for POLICY are:

    ignore
        Completely ignore IPv6 flows. Only IPv4 flows will be printed.

    asv4
        Convert IPv6 addresses to IPv4 if possible, otherwise ignore the IPv6 flows.

    mix Process the input as a mixture of IPv4 and IPv6 flows.

    force
        Force IPv4 flows to be converted to IPv6.

    only
        Only process flows that were marked as IPv6 and completely ignore IPv4 flows.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application's directory.

--legacy-timestamps
--legacy-timestamps=NUM
    Specify the format for human readable timestamps, either the default (new) style, "YYYY/MM/DDThh:mm:ss.sss", or the legacy style, "MM/DD/YYYY hh:mm:ss". When this switch is not present, the timestamps will be in the default format. When this switch is present and no argument is given, timestamps are in the legacy format. When an argument is supplied, timestamps will be in the new format if the argument begins with 0, and in the old format if the argument begins with 1. Any other argument to the switch is an error.

    This switch also controls whether fractional seconds are displayed in the sTime and eTime fields when --epoch-time is requested. If the --legacy-timestamps switch is present with no value or with a value of 1, milliseconds will not be displayed; when not present or specified with a value of 0, milliseconds will be displayed.

--help
    Print the available options and exit. Options that add fields can be specified before --help so that the new options appear in the output.

--version
    Print the version number and information about how SiLK was configured, then exit the application.

--dynamic-library=PLUGIN
    This switch is deprecated. It is an alias for --plugin.

--pmap-file=MAPNAME:PATH
--pmap-file=PATH
    Instruct rwcut to load the mapping file located at PATH and create the src-MAPNAME and dst-MAPNAME fields. When MAPNAME is provided explicitly, it will be used to refer to the fields specific to that prefix map. If MAPNAME is not provided, rwcut will check the prefix map file to see if a map-name was specified when the file was created. If no map-name is available, rwcut creates the fields sval and dval. Multiple --pmap-file switches are supported as long as each uses a unique value for map-name. The --pmap-file switch(es) must precede the --fields switch. For more information, see pmapfilter(3).

--pmap-column-width=NUM
    When printing a label associated with a prefix map, this switch gives the maximum number of characters to use when displaying the textual value of the field.

--python-file=PATH
    When the SiLK Python plug-in is used, rwcut reads the Python code from the file PATH to define additional fields for possible output. This file should call register_plugin_field() for each field it wishes to define. For details and examples, see the silkpython(3) and pysilk(3) manual pages.

EXAMPLES
A standard rwcut output will look like this (with the text wrapped for readability):

            sIP|            dIP|sPort|dPort|pro|\
    10.30.30.31|    10.70.70.71|   80|36761|  6|\

        packets|     bytes|          flags|\
              7|      3227|      FS PA    |\

                    sTime|      dur|                  eTime|senso|
  2003/01/01T00:00:14.625|    3.959|2003/01/01T00:00:18.584|EDGE1|

The first line of the output is the title line--this line shows what the selected fields are; the --no-titles switch will disable the printing of that line. The second line onwards will contain data.

The most basic use of rwcut is by being directly connected to rwfilter(1). For example, to see representative TCP traffic:

 rwfilter --start-date=2002/01/19:00 --end-date=2002/01/19:01 \
      --proto=6 --pass=stdout | rwcut

To see only limited field, use the --fields switch. For example, to see only the protocols, use:

 rwcut --fields=5

The silkpython(3) manual page provides examples that use PySiLK to create and print arbitrary fields for rwcut.

The order of the FIELDS is significant, and fields can be repeated. For example, here is a case where in addition to the default fields of 1-12, you also to prefix each row with an integer form of the destination IP and the start time to make processing by another tool easier. However, within the default fields of 1-12, you want to see dotted-decimal IP addresses.

 rwfilter ... --pass=stdout | \
       rwcut --integer-ip --fields=2,9,1-12 --epoch-time | \
       num2dot --ip-field=3,4

ENVIRONMENT

SILK_IPV6_POLICY
    This environment variable is used as the value for the --ipv6-policy when that switch is not provided.

SILK_PAGER
    When set to a non-empty string, rwcut automatically invokes this program to display its output a screen at a time. If set to an empty string, rwcut does not automatically page its output.

PAGER
    When set and SILK_PAGER is not set, rwcut automatically invokes this program to display its output a screen at a time.

PYTHONPATH
    This environment variable is used by Python to locate modules. When --python-file is specified, rwcut loads Python which in turn loads the PySiLK module which is comprised of several files (silk/pysilk_nl.so, silk/__init__.py, etc). If this silk/ directory is located outside Python's normal search path (for example, in the SiLK installation tree), it may be necessary to set or modify the PYTHONPATH environment variable to include the parent directory of silk/ so that Python can find the PySiLK module.

SILK_PYTHON_TRACEBACK
    When set, Python plug-ins will output traceback information on Python errors to the standard error.

SILK_COUNTRY_CODES
    This environment variable allows the user to specify the country code mapping file that rwcut will use when computing the scc and dcc fields. The value may be a complete path or a file relative to the SILK_PATH. If the variable is not specified, rwcut checks the following locations:

      $SILK_PATH/share/silk/country_codes.pmap
      $SILK_PATH/share/country_codes.pmap
      /usr/local/share/silk/country_codes.pmap
      /usr/local/share/country_codes.pmap

SILK_ADDRESS_TYPES
    This environment variable allows the user to specify the address type mapping file that rwcut will use when computing the sType and dType fields. The value may be a complete path or a file relative to the SILK_PATH. If the variable is not specified, rwcut checks the following locations:

      $SILK_PATH/share/silk/address_types.pmap
      $SILK_PATH/share/address_types.pmap
      /usr/local/share/silk/address_types.pmap
      /usr/local/share/address_types.pmap

SILK_CONFIG_FILE
    This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR
    When the --site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwcut looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH
    This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwcut checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share. These directories are also searched when any other configuration file is required (e.g., the country code map). In addition, rwcut looks for plug-ins in $SILK_PATH/lib/silk, $SILK_PATH/share/lib and $SILK_PATH/lib.

SILK_PLUGIN_DEBUG
    When set to 1, rwcut prints status messages to the standard error as it tries to open each of its plug-ins.

NOTES
The ordering of the field numbers in --fields is significant, specifying --fields="2,1" will print destination IP, then source IP.

If you are interested in only a few fields, use the --fields option to reduce the volume of data to be processed. For example, if you are checking to see which internal host got hit with the slammer worm (signature: UDP, destPort 1434, pkt size 404), then the following rwfilter, rwcut combination will be much faster than simply using default values:

  rwfilter --proto-17 --dport=1434 --bytes-per-packet=404-404 \
        | rwcut --fields=2

To get a mapping from the integer representing a sensor to its name, use the mapsid(1) command.

SEE ALSO
rwfilter(1), mapsid(1), num2dot(1), addrtype(3), ccfilter(3), pmapfilter(3), silkpython(3), pysilk(3), sensor.conf(5), yaf(1)