RWSPLIT(1)

NAME
rwsplit - Divide a SiLK file into a (sampled) collection of subfiles

SYNOPSIS
  rwsplit --basename=BASENAME
        { --ip-limit=LIMIT | --flow-limit=LIMIT
          | --packet-limit=LIMIT | --byte-limit=LIMIT }
        [--seed=NUMBER] [--sample-ratio=SAMPLE_RATIO]
        [--file-ratio=FILE_RATIO] [--file-limit=FILE_LIMIT]
        [--note-add=TEXT] [--note-file-add=FILE]
        [--compression-method=COMP_METHOD]
        [--site-config-file=FILENAME] [FILES]

  rwsplit --help

  rwsplit --version

DESCRIPTION
rwsplit reads SiLK Flow records from the standard input or from files named on the command line and writes the flows into a set of subfiles based on the partition specification. In its simplest form, rwsplit partitions the file, meaning that each input flow will appear in one (and only one) of the subfiles.

In addition to splitting the file, rwsplit can generate files containing sample flows. Sampling is specified by using the --sample-ratio and --file-ratio switches.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

The splitting criterion is defined using one of the limit specifiers; one and only one must be specified. They are:

--ip-limit=LIMIT
    Specifies the count of unique source and destination IPs at which to close the current subfile and begin a new subfile; next-hip-IPs do not count toward the limit. Note that LIMIT is approximate, since a record with source and destination IPs not seen previously will increase the unique IP count by 2.

--flow-limit=LIMIT
    Specifies the count of SiLK Flow records at which to close the current subfile and begin a new subfile.

--packet-limit=LIMIT
    Specifies the count of packets at which to close the current subfile and begin a new subfile. LIMIT is the lower threshold on the packet count for all files except the last.

--byte-limit=LIMIT
    Specifies the byte count at which to close the current subfile and begin a new subfile. LIMIT is the lower threshold on the byte count for all files except the last.

The other switches are:

--basename=BASENAME
    Specifies the basename of the output files; this switch is required. The flows are written sequentially to a set of subfiles whose names follow the format BASENAME.ORDER.rwf, where ORDER is an 8-digit zero-formatted sequence number (i.e., 00000000, 00000001, and so on). The sequence number will begin at zero and increase by one for every file written, unless --file-ratio is specified,

--seed=NUMBER
    Use NUMBER to seed the pseudo-random number generator for the --sample-ratio or --file-ratio switch. This can be used to put the random number generator into a known state, which is useful for testing.

--sample-ratio=SAMPLE_RATIO
    Writes one flow record, chosen at random, from every SAMPLE_RATIO flows that are read.

--file-ratio=FILE_RATIO
    Picks one subfile, chosen from random, out of every FILE_RATIO names generated, for writing to disk.

--file-limit=NUMBER
    Limits the number of files that are written to disk to NUMBER.

--note-add=TEXT
    Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME
    Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--compression-method=COMP_METHOD
    Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

    none
        Do not compress the output using an external library

    zlib
        Use the zlib(3) library for compressing the output

    lzo1x
        Use the lzo1x algorithm from the LZO real time compression library for compression

    best
        Use whichever available method gives the "best" compression in general, though not necessarily the "best" for this particular output.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application's directory.

--help
    Print the available options and exit.

--version
    Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES
Assume a source file source.rwf; to split that file into files that each contain about 100 unique IP addresses:

  rwsplit --basename=result --ip-limit=100 source.rwf

To split source.rwf into files that each contain 100 flows:

  rwsplit --basename=result --flow-limit=100 source.rwf

The following causes rwsplit to sample 1 out of every 10 records from source.rwf; i.e., rwsplit will read 1000 flow records to produce each subfile:

  rwsplit --basename=result --flow-limit=100 --sample-ratio=10 source.rwf

When --file-ratio is specified, the file names are generated as usual (e.g., base-00000000, base-00000001, ...); however, one of these names will be chosen randomly from each set of --file-ratio candidates, and only that file will be written to disk.

  $ rwsplit --basename=result --flow-limit=100 --file-ratio=5 source.rwf
  $ ls
  result-00000002.rwf
  result-00000008.rwf
  result-00000013.rwf
  result-00000016.rwf

LIMITATIONS
rwsplit can take exactly 1 partitioning switch per invocation.

Partitioning is not exact, rwsplit keeps appending flow records a file until it meets or exceeds the specified LIMIT. For example, if you specify --ip-limit=100, then rwsplit will fill up the file until it has 100 IP addresses in it; if the file has 99 addresses and a new record with 2 previously unseen addresses is received, rwsplit will put this in the current file, resulting in a 101-address file. Similarly, if you specify --byte-limit=2000, and rwsplit receives a 2GB flow record, that flow record will be placed in the current subfile.

The switches --sample-ratio, --file-ratio, and --file-limit are processed in that order. So, when you specify

  rwsplit --sample-ratio=10 --ip-limit=100 --file-ratio=10 --file-limit=20

then rwsplit will pick 1 out of every 10 flow records, write that to a file until it has 100 IP's per file, pick 1 out of every 10 files to write, and write up to 20 files. If there are 1000 records, each with 2 unique IPs in them, then rwsplit will write at most 1 file (it will write 200 unique IP addresses, but it may not pick one of the files from the set to write).

ENVIRONMENT

SILK_CONFIG_FILE
    This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR
    When the --site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwsplit looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH
    This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwsplit checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share.

SEE ALSO
rwfilter(1)

BUGS
When used in an IPv6 environment, rwsplit will process all flows unless the --ip-limit is requested. When limiting files to a certain number of IP addresses, rwsplit will attempt to convert any IPv6 addresses to IPv4. Records that can be converted will be processed, all other records will be silently ignored.