RWSET(1)

NAME
rwset - Generate binary IPset files of unique IP addresses

SYNOPSIS
  rwset [--sip-file=FILE] [--dip-file=FILE] [--nhip-file=FILE]
        [--print-filenames] [--copy-input=PATH]
        [--note-add=TEXT] [--note-file-add=FILE]
        [--compression-method=COMP_METHOD]
        [--site-config-file=FILENAME] [FILES...]

  rwset --help

  rwset --version

DESCRIPTION
rwset reads SiLK Flow data from the standard input, a named pipe, or named input files and generates binary IPset file(s) containing the list of the unique source IP addresses, destination IP addresses, and/or next-hop IP addresses.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

One of the following output switches is required:

--sip-file=FILE
    Store the unique source IP addresses in the binary IPset file FILE. rwset will write the IPset file to the standard output when FILE is "stdout" and the standard output is not a terminal.

--dip-file=FILE
    Store the unique destination IP addresses in the binary IPset file FILE. rwset will write the IPset file to the standard output when FILE is "stdout" and the standard output is not a terminal.

--nhip-file=FILE
    Store the unique next-hop IP addresses in the binary IPset file FILE. rwset will write the IPset file to the standard output when FILE is "stdout" and the standard output is not a terminal.

Only one of the above switches my use "stdout" as the name of the file.

rwset supports these additional switches:

--print-filenames
    Prints to the standard error the names of input files as they are opened.

--copy-input=PATH
    Copy all binary input to the specified file or named pipe. PATH can be "stdout" to print flows to the standard output as long as no IPset files are being written there.

--note-add=TEXT
    Add the specified TEXT to the header of every output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME
    Open FILENAME and add the contents of that file to the header of every output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--compression-method=COMP_METHOD
    Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

    none
        Do not compress the output using an external library

    zlib
        Use the zlib(3) library for compressing the output

    lzo1x
        Use the lzo1x algorithm from the LZO real time compression library for compression

    best
        Use whichever available method gives the "best" compression in general, though not necessarily the "best" for this particular output.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application's directory.

--help
    Print the available options and exit.

--version
    Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES
rwset is intended to work tightly with rwfilter. For example, consider generating two sets: one of all IP addresses which have 3 packets or less in their sessions, and one with four or more packets.

The first set, for TCP traffic on 03/01/2003 can be generated with:

  rwfilter --start-date=2003/03/01:00 --end-date=2003/03/01:23 \
        --proto=6 --packets=1-3 --pass=stdout | \
        rwset --sip-file=low_packet_tcp.set

The second set with:

  rwfilter --start-date=2003/03/01:00 --end-date=2003/03/01:23
        --proto=6 --packets=4- --pass=stdout | \
        rwset --sip-file=high_packet_tcp.set

ENVIRONMENT

SILK_CONFIG_FILE
    This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR
    When the --site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwset looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH
    This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwset checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share.

NOTES
rwset does not support IPv6 addresses. Any IPv6 data in the input will be ignored.

IPset files are in a binary format. To create a binary IPset file from a list of dotted decimal IP addresses, use rwsetbuild(1).

Use rwsetcat(1) to see the IP addresses in a binary set file.

rwsettool(1) allows you to perform set operations on binary IPset files.

The command:

  rwset --sip-file=stdout | rwsetcat

will be faster than rwuniq, but will cannot report total volume nor do the thresholding that rwuniq supports.

SEE ALSO
rwsetbuild(1), rwsetcat(1), rwsettool(1), rwsetmember(1), rwfilter(1), rwfileinfo(1), rwuniq(1)

BUGS
Currently there is no support for IPv6 addresses in IPsets.

When used in an IPv6 environment, rwset will attempt to convert any IPv6 addresses to IPv4. Records that can be converted will be processed, all other records will be silently ignored.