RWFILTER(1)

NAME
rwfilter - Choose which SiLK Flow records to process

SYNOPSIS
  rwfilter INPUT_ARGS OUTPUT_ARGS PARTITIONING_ARGS [MISC_ARGS]

Input switches or files on the command line are required:

  rwfilter ...
        {{ [--class=CLASS] [--type={all | TYPE[,TYPE ...]}]
           | [--flowtype=CLASS/TYPE[,CLASS/TYPE ...]] }
         [--data-rootdir=PATH] [--print-missing-filenames]
         [--sensors=SENSOR[,SENSOR ...]]
         [--start-date=YYYY/MM/DD[:HH] [--end-date=YYYY/MM/DD[:HH]]] }
        | [--input-pipe=INPUT_PATH]
        | [--xargs=INPUT_PATH]
        | [INPUT_PATH [INPUT_PATH...]]

One or more output switches are required:

  rwfilter ...
        [--all-destination=ALL_PATH [--all-destination=ALL_PATH ...]]
        [--fail-destination=FAIL_PATH [--fail-destination=FAIL_PATH ...]]
        [--pass-destination=PASS_PATH [--pass-destination=PASS_PATH ...]]
        [{ --print-statistics | --print-volume-statistics }]

One or more partitioning switches are required:

  rwfilter ...
        [--ack-flag=SCALAR] [--active-time=TIME_WINDOW]
        [{--any-address=IP_WILDCARD | --not-any-address=IP_WILDCARD}]
        [{--any-cidr=IP_OR_CIDR_LIST | --not-any-cidr=IP_OR_CIDR_LIST}]
        [{--anyset=IP_SET_FILENAME | --not-anyset=IP_SET_FILENAME}]
        [--aport=INTEGER_LIST] [--application=INTEGER_LIST]
        [--attributes=ATTRIBUTES_LIST]
        [--bytes=INTEGER_RANGE] [--bytes-per-packet=DECIMAL_RANGE]
        [--cwr-flag=SCALAR]
        [{--daddress=IP_WILDCARD | --not-daddress=IP_WILDCARD}]
        [--dcc=COUNTRY_CODE_LIST]
        [{--dcidr=IP_OR_CIDR_LIST | --not-dcidr=IP_OR_CIDR_LIST}]
        [{--dipset=IP_SET_FILENAME | --not-dipset=IP_SET_FILENAME}]
        [--dport=INTEGER_LIST] [--dtype=SCALAR]
        [--duration=DECIMAL_RANGE] [--ece-flag=SCALAR]
        [--etime=TIME_WINDOW] [--fin-flag=SCALAR]
        [--flags-all=HIGH_MASK_FLAGS_LIST]
        [--flags-initial=HIGH_MASK_FLAGS_LIST]
        [--flags-session=HIGH_MASK_FLAGS_LIST]
        [--icmp-code=INTEGER_LIST] [--icmp-type=INTEGER_LIST]
        [--input-index=INTEGER_LIST] [--ip-version=INTEGER_LIST]
        [--ippair-any=FILENAME] [--ipport-any=FILENAME]
        [{--next-hop-id=IP_WILDCARD | --not-next-hop-id=IP_WILDCARD}]
        [{--nhcidr=IP_OR_CIDR_LIST | --not-nhcidr=IP_OR_CIDR_LIST}]
        [{--nhipset=IP_SET_FILENAME | --not-nhipset=IP_SET_FILENAME}]
        [--output-index=INTEGER_LIST] [--packets=INTEGER_RANGE]
        [--pmap-file=MAPNAME:PATH [--pmap-file=MAPNAME:PATH ...]
         { [--pmap-src-MAPNAME=LABELS] [--pmap-dst-MAPNAME=LABELS]
           [--pmap-any-MAPNAME=LABELS] } ]
        [--protocol=INTEGER_LIST] [--psh-flag=SCALAR]
        [--python-expr=PYTHON_EXPR]
        [--python-file=FILENAME [--python-file=FILENAME ...]]
        [--rst-flag=SCALAR]
        [{--saddress=IP_WILDCARD | --not-saddress=IP_WILDCARD}]
        [--scc=COUNTRY_CODE_LIST]
        [{--scidr=IP_OR_CIDR_LIST | --not-scidr=IP_OR_CIDR_LIST}]
        [{--sipset=IP_SET_FILENAME | --not-sipset=IP_SET_FILENAME}]
        [--sport=INTEGER_LIST] [--stime=TIME_WINDOW] [--stype=SCALAR]
        [--syn-flag=SCALAR] [--tcp-flags=TCP_FLAGS]
        [--tuple-file=TUPLE_FILENAME { [--tuple-fields=FIELDS]
                                       [--tuple-direction=DIRECTION]
                                       [--tuple-delimiter=CHAR] } ]
        [--urg-flag=SCALAR]

Miscellaneous switches:

  rwfilter ...
        [--compression-method=COMP_METHOD] [--dry-run]
        [--max-fail-records=N] [--max-pass-records=N]
        [--note-add=TEXT] [--note-file-add=FILE]
        [--plugin=PLUGIN [--plugin=PLUGIN ...]]
        [--print-filenames] [--site-config-file=FILENAME]
        [--threads=N]

Help switches:

  rwfilter [--pmap-file=MAPNAME:PATH [--pmap-file=MAPNAME:PATH ...]]
        [--plugin=PLUGIN ...] [--python-file=PATH]
        [--data-rootdir=PATH] [--site-config-file=FILENAME]
        --help

  rwfilter --version

DESCRIPTION
rwfilter serves two purposes: (1) It acts as an interface to the data store to select which SiLK Flow records to process, and (2) it partitions those records into one or more pass and/or fail streams.

The "Selection Switches" let one choose flow records from the SiLK data store by specifying where the flow was collected (its sensor), the date of collection, and/or the flow's direction. The act of selecting records from the data store is sometimes called a "data pull".

The "Partitioning Switches" describe various types of traffic behavior (e.g., TCP traffic, or all traffic going to port 80). When a flow record matches all of the behaviors, it can be written to a pass stream (i.e., file). If a record fails to match any of these behavior predicates, it can be written to a fail stream. (You may also write every record rwfilter reads to an all stream.) These output streams from rwfilter are always binary SiLK Flow records. The output must be either written to a file or piped into another tool in the SiLK Suite, and rwfilter will complain if it determines you are attempting to send the stream to a terminal. To view the records, pipe the records into rwcut(1).

In addition to the partitioning switches built in to rwfilter, additional partitioning predicates can be created as C or PySiLK plug-ins, and these can be loaded into rwfilter using the --plugin and/or --python-file switches as described below.

Instead of using the selection switches to choose flow records from the data store, rwfilter can apply the partitioning switches to existing files of SiLK flow records---such as files generated by a previous invocation of rwfilter. To run rwfilter in this mode, you may

•   specify, on the command line, the files and/or named pipes from which rwfilter should read SiLK Flow records. Specifying "stdin" or "-" or the command line will cause rwfilter to read flow records from the standard input.

•   use the --input-pipe switch to specify a named pipe, or specify "stdin" or "-" as the argument to this switch to have rwfilter read flow records from the standard input.

•   use the --xargs switch to specify a file that contains the names of the input files to process. The --xargs switch requires an argument; specify "stdin" or "-" to read the names of the input files from the standard input. The name of each input file must appear on a single line.

When rwfilter is reading flow records from input files, some of the selection switches will act as partitioning switches. The remaining selection switches may not be specified when using the alternate forms of input, and it is an error to specify multiple types of input.

Unlike many other tools in the SiLK tool suite, rwfilter requires that you specify one or more "Output Switches" that tell rwfilter what types of output to produce.

Finally, there are "Miscellaneous Switches" that control other aspects of rwfilter.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

  Selection Switches
To read files from the data store, use the following options to specify which files to process. When rwfilter gets its input from files listed on the command line or from the --xargs or --input-pipe switches, the first four switches (--class, --type, --flowtypes, and --sensors) will act as partitioning switches, and specifying any other selection switch will produce an error.

--class=CLASS
    The --class switch is used to specify a group of data to process. Only a single class may be selected with the --class switch; for multiple classes, use the --flowtypes switch. Classes are defined in the silk.conf(5) site configuration file. If the --class option is not given, the default-class as specified in silk.conf is used. Use the --help option to see the list of available classes and the default class.

--data-rootdir=PATH
    Tell rwfilter to use PATH as the root of the data store directory, which overrides the location given in the SILK_DATA_ROOTDIR environment variable, which overrides the location that was compiled into rwfilter. The default data store directory will be shown when the --version option is given. It is an error to specify this switch when files are specified on the command line or "Non-Selection Input Switches" are given.

--flowtypes=CLASS/TYPE[,CLASS/TYPE ...]
    The --flowtype predicate provides an alternate way to specify class/type pairs. The --flowtypes switch allows a single rwfilter invocation to process data from multiple classes. The keyword "all" may be used for the CLASS and/or TYPE to select all classes and/or types.

--print-missing-files
    This option prints to the standard error file names that rwfilter's file selection switches expected to find but did not. This switch is useful for debugging, but the list of files it produces can be misleading. For example, suppose there is a decommissioned sensor that still appears in the silk.conf file; these data files will be missing even though their absence is expected. Use the output from this switch judiciously. It is an error to specify this switch when files are specified on the command line or "Non-Selection Input Switches" are given.

--sensors=SENSOR[,SENSOR ...]
    The --sensor switch is used to select data from specific sensors. The parameter is a comma separated list of sensor names, sensor IDs (integers), and/or ranges of sensor IDs. Sensors are defined in the silk.conf(5) site configuration file, and the mapsid(1) command can be used to print a mapping of sensor names to IDs and classes. When the --sensor switch is not specified, the default is to use all sensors which are valid for the specified class(es).

--start-date=YYYY/MM/DD[:HH]
--end-date=YYYY/MM/DD[:HH]
    The date predicates indicate which days and hours to consider when creating the list of files. The dates may be expressed as seconds since the UNIX epoch or in "YYYY/MM/DD[:HH]" format, where the hour is optional. A "T" may be used in place of the ":" to separate the day and hour. Whether the "YYYY/MM/DD[:HH]" strings represent times in UTC or the local timezone depend on how SiLK was compiled. To determine how your version of SiLK was compiled, see the "Timezone support" setting in the output from rwfilter --version.

    When both --start-date and --end-date are specified to hour precision, all hours within that time range are processed.

    When --start-date is specified to day precision, the hour specified in --end-date (if any) is ignored, and files for all dates between midnight on start-date and 23:59 on end-date are processed.

    When --end-date is not specified and --start-date is specified to day precision, files for that complete day are processed.

    When --end-date is not specified and --start-date is specified to hour precision, files for that single hour are processed.

    When neither --start-date nor --end-date is given, rwfilter processes all files for the current day.

    It is an error to specify --end-date without specifying --start-date, or to specify --start-date with the "Non-Selection Input Switches".

--type={"all" | TYPE[,TYPE]}
    The --type predicate further specifies data within the selected CLASS by listing the TYPEs of traffic to process. The switch takes a comma-separated list of types or the keyword "all" which specifies all types for the specified CLASS. Types are defined in silk.conf, they typically refer to the direction of the flow, and they may vary by class. When the --type switch is not specified, a list of default types will be used. The default-type list is determined by the value of CLASS, and the default types generally include only incoming traffic. Use the --help option to see the list of available types and the default types for each class.

  Non-Selection Input Switches
Instead of using the "Selection Switches" to read flow records from files in the data store, you can tell rwfilter to process files named on the command line or use one (and only one) of the following switches. To have rwfilter read flow records from the standard input, specify "-" as the name of an input file or use the --input-pipe switch.

--input-pipe=INPUT_PATH
    Specify a source for SiLK Flow records, where INPUT_PATH is a named pipe or the string "stdin" or "-" to represent the standard input. You do not need to use this switch, you can simply specify the named pipe or the strings "stdin" or "-" on the command line.

--xargs=INPUT_PATH
    Tell rwfilter to read file names from INPUT_PATH; the input should have one file name per line. rwfilter will open each file in turn and read records from it. Specify an INPUT_PATH of "stdin" or "-" to have rwfilter read the names of files from the standard input.

  Output Switches
At least one of the following output switches must be provided:

--all-destination=ALL_PATH
    Write every SiLK Flow record to ALL_PATH, where ALL_PATH refers to a file, a named pipe, or the strings "stdout" or "-" to refer to the standard output. This switch may be repeated to write all input records to multiple locations.

--fail-destination=FAIL_PATH
    Write SiLK Flow records that have failed ANY of the partitioning predicates to FAIL_PATH, where FAIL_PATH refers to a non-existent file, a named pipe, or the strings "stdout" or "-" to refer to the standard output. This switch may be repeated to write records that fail any predicate to multiple locations.

--pass-destination=PASS_PATH
    Write SiLK Flow records that have passed ALL of the partitioning predicates to PASS_PATH, where PASS_PATH refers to a non-existent file, a named pipe, or the strings "stdout" or "-" to refer to the standard output. This switch may be repeated to write records that pass every predicate to multiple locations.

--print-statistics
--print-statistics=PATH
    Print a one line summary specifying the number of files processed, the total number of records read, the number of records that passed all partitioning predicates, and the number of records that failed. If PATH is provided, the summary will be printed there; otherwise it is printed to the standard error. This switch cannot be mixed with --print-volume-statistics.

--print-volume-statistics
--print-volume-statistics=PATH
    Print a four line summary of rwfilter's processing. For each of all records, records that pass all the partitioning predicates, and records that fail, print the number of flow records and the number of packets and bytes represented by those flow records. The output also includes the number of files processed. If PATH is provided, the summary will be printed there; otherwise it is printed to the standard error. This switch cannot be mixed with --print-statistics.

  Partitioning Switches
rwfilter supports the following partitioning switches, at least one of which must be specified (unless the only Output Switch is --all-destination). The switches are AND'ed together; i.e., to pass the filter, the record must pass the test implied by each switch. Any record that does not pass will be sent to the fail-destination(s), if specified.

Partitioning Switches for IP Addresses

There are three families of switches that partition based on an IP address. Each family can partition by the source IP, the destination IP, the next hop IP, or either source or destination IP. Each family includes a --not-* variant to reverse the sense of the test.

The --*cidr-family takes as its argument an IP_OR_CIDR_LIST, which is a single IP address 10.1.2.3, a single CIDR block "FF01::/16", or a comma separated list of IPs and/or CIDR blocks "10.0.1.0/24,10.0.2.3,10.0.4.0/24". The IP_OR_CIDR_LIST supports IPv4 and IPv6 addresses.

The --*address-family (which includes --next-hop-id) takes as its argument an IP_WILDCARD. An IP_WILDCARD is a single IP address, a single CIDR block, or a single SiLK IP Wildcard. A SiLK IP Wildcard can represent multiple IPv4 or IPv6 addresses. An IP Wildcard contains an IP in its canonical form, except each part of the IP (where part is an octet for IPv4 or a hexadectet for IPv6) may be a single value, a range, a comma separated list of values and ranges, or the letter "x" to signify any value for that part of the IP (that is, "0-255" for IPv4). You may not specify a CIDR suffix when using the IP Wildcard notation. The following IP_WILDCARDs all represent the same value:

 ::ffff:0:0/112
 ::ffff:0:x
 ::ffff:0:aaab-ffff,aaaa,0-aaa9
 ::ffff:0.0.0.0/112
 ::ffff:0.0.128-254,0-126,255,127.x

The --*set-family requires that you store the IPs in a binary IPset file and pass the name of the file to the switch. IPset files are created from SiLK Flow records with rwset(1), or from textual input with rwsetbuild(1). Currently, IPsets only support IPv4 addresses.

The next hop address often has a value of 0.0.0.0 since the default configuration of SiLK does not store the next hop address in the data repository.

The address-partitioning switches are:

--scidr=IP_OR_CIDR_LIST
    Pass the record if its source IP address matches a value in IP_OR_CIDR_LIST, a comma separated list of IPs and/or CIDR blocks. See also --saddress and --sipset.

--dcidr=IP_OR_CIDR_LIST
    Pass the record if its destination IP address matches a value in IP_OR_CIDR_LIST. See also --daddress and --dipset.

--any-cidr=IP_OR_CIDR_LIST
    Pass the record if either its source or its destination IP address matches a value in IP_OR_CIDR_LIST. This switch does not consider the next hop IP address. See also --any-address and --anyset.

--nhcidr=IP_OR_CIDR_LIST
    Pass the record if its next hop IP address matches a value in IP_OR_CIDR_LIST. See also --next-hop-id and --nhipset.

--not-scidr=IP_OR_CIDR_LIST
    Pass the record if its source IP address does not match a value in IP_OR_CIDR_LIST, a comma separated list of IPs and/or CIDR blocks. See also --not-saddress and --not-sipset.

--not-dcidr=IP_OR_CIDR_LIST
    Pass the record if its destination IP address does not match a value in IP_OR_CIDR_LIST. See also --not-daddress and --not-dipset.

--not-any-cidr=IP_OR_CIDR_LIST
    Pass the record if neither its source nor its destination IP address matches a value in IP_OR_CIDR_LIST. See also --not-any-address and --not-anyset.

--not-nhcidr=IP_OR_CIDR_LIST
    Pass the record if its next hop IP address does not match a value in IP_OR_CIDR_LIST. See also --not-next-hop-id and --not-nhipset.

--saddress=IP_WILDCARD
    Pass the record if its source IP address is matched by the SiLK IP Wildcard IP_WILDCARD. To match on multiple IPs, use --scidr or create an IPset and use --sipset.

--daddress=IP_WILDCARD
    Pass the record if its destination IP address is matched by IP_WILDCARD, a SiLK IP Wildcard. See also --dcidr and --dipset.

--any-address=IP_WILDCARD
    Pass the record if either its source or its destination IP address is matched by IP_WILDCARD, a SiLK IP Wildcard. This switch does not consider the next hop IP address. See also --any-cidr and --anyset.

--next-hop-id=IP_WILDCARD
    Pass the record if its next hop IP address is matched by this IP_WILDCARD, a SiLK IP Wildcard. To match on multiple IPs, use --nhcidr or create an IPset and use --nhipset.

--not-saddress=IP_WILDCARD
    Pass the record if its source IP address is not matched by this IP_WILDCARD, a SiLK IP Wildcard. See also --not-scidr and --not-sipset.

--not-daddress=IP_WILDCARD
    Pass the record if its destination IP address is not matched by this IP_WILDCARD. See also --not-dcidr and --not-dipset.

--not-any-address=IP_WILDCARD
    Pass the record if neither its source nor its destination IP address is matched by this IP_WILDCARD. Does not consider the next hop address. See also --not-any-cidr and --not-anyset.

--not-next-hop-id=IP_WILDCARD
    Pass the record if its next hop IP address is not matched by this IP_WILDCARD. See also --not-nhcidr and --not-nhipset.

--sipset=IP_SET_FILENAME
    Pass the record if its source IP address is in the list of IPs contained in the binary set file IP_SET_FILENAME. See also --scidr.

--dipset=IP_SET_FILENAME
    As --sipset for the destination IP address. See also --dcidr.

--anyset=IP_SET_FILENAME
    Pass the record if either its source IP address or its destination IP address is in the list of IPs contained in the binary set file IP_SET_FILENAME. Does not consider the next hop IP. See also --any-cidr.

--nhipset=IP_SET_FILENAME
    As --sipset for the next-hop IP address. See also --nhcidr.

--not-sipset=IP_SET_FILENAME
    Pass the record if its source IP address is not in the list of IPs contained in the binary set file IP_SET_FILENAME. See also --not-scidr.

--not-dipset=IP_SET_FILENAME
    As --not-sipset for the destination IP address. See also --not-dcidr.

--not-anyset=IP_SET_FILENAME
    Pass the record if neither its source IP address nor its destination IP address is in the list of IPs contained in the binary set file IP_SET_FILENAME. Does not consider the next hop IP. See also --not-any-cidr.

--not-nhipset=IP_SET_FILENAME
    As --not-sipset for the next hop IP address. See also --not-nhcidr.

Partitioning Switches for Remainder of Five-Tuple

The following switches partition based on the protocol and source or destination port. The parameter to each of these switches is an INTEGER_LIST, which is a comma-separated list of individual non-negative integer values and ranges of those values. For example, "1,2,3,5-10,99-103". A range may be specified without an end value, such as "1-", in which case the ending value limit is set to the maximum value.

--sport=INTEGER_LIST
    Pass the record if its source port is in this INTEGER_LIST, possible values are 0-65535.

--dport=INTEGER_LIST
    Pass the record if its destination port is in this INTEGER_LIST, possible values are 0-65535

--aport=INTEGER_LIST
    Pass the record if its source port and/or its destination port is in this INTEGER_LIST, possible values are 0-65535. For example, use --aport=25 to see all SMTP conversions regardless or where they originated.

--protocol=INTEGER_LIST
    Pass the record if its IP Suite Protocol is in this INTEGER_LIST, possible values are 0-255.

--icmp-type=INTEGER_LIST
    Pass the record if its ICMP (or ICMPv6) type is in this INTEGER_LIST; possible values 0-255. This switch will also verify that the flow's protocol is 1 (or 58 if the flow is IPv6). It is an error to specify a --protocol that does not include 1 and/or 58.

--icmp-code=INTEGER_LIST
    Pass the record if its ICMP (or ICMPv6) code is in this INTEGER_LIST; possible values 0-255. This switch will also verify that the flow's protocol is 1 (or 58 if the flow is IPv6). It is an error to specify a --protocol that does not include 1 and/or 58.

Partitioning Switches for Time

These switches partition based on whether the time stamps on the flow record occur within the specified time window. The form of the argument is range of two dates, start-window and end-window, each in the form "YYYY/MM/DD[:HH[:MM[:SS[.ssssss]]]]", for example "2003/01/31:23:45:00.000-2003/01/31:23:59:59.999" represents the last fifteen minutes of Jan 31, 2003. (You may use a "T" in place of ":" to separate the day and hour; you may also specify the times as seconds since the UNIX epoch.) The start-window and end-window must be set to at least day precision. For the start-window, unspecified hour, minute, second, and millisecond values set to 0; for the end-window, those values are set to 23, 59, 59, and 999 respectively. Thus "2003/01/31:23-2003/01/31:23" will become "2003/01/31:23:00:00.000-2003/01/31:23:59:59.999". If an end-window is not given, it is set to the start-window, giving a window of a single millisecond. The date strings are considered to be in the timezone specified when SiLK was compiled, which you can determine from the output of rwfilter --version.

--active-time=TIME_WINDOW
    Pass the record if the record was active at ANY time during this TIME_WINDOW. If a single time is specified, pass the record if it was active at that instant.

--stime=TIME_WINDOW
    Pass the record if its starting time is in this TIME_WINDOW.

--etime=TIME_WINDOW
    As --stime for the ending time.

--duration=DECIMAL_RANGE
    Pass the record if its duration--that is, the record's end time minus its start time, as measured in seconds--is in this DECIMAL_RANGE. Use floating point numbers to specify millisecond values. The range should be specified as MIN-MAX; for example, "5.0-10.031". If a single value is given, the duration must match that value exactly. The upper limit may be omitted; for example, a range of "1.5-" passes records whose duration is at least 1.5 seconds.

Partitioning Switches for Volume

The following switches partition based on the volume of the flow; that is, the number of bytes or packets. For additional volume-related switches, load the flowrate plug-in as described in the flowrate(3) manual page.

These switches accept a range of non-negative integers or decimal values. If the upper limit is omitted, the volume must be at least that size. If the argument is a single value, the volume must match that value exactly.

--bytes=INTEGER_RANGE
    Pass the record if its byte count is in this INTEGER_RANGE.

--packets=INTEGER_RANGE
    Pass the record if its packet count is in this INTEGER_RANGE.

--bytes-per-packet=DECIMAL_RANGE
    Pass the record if its average bytes per packet count (bytes/packet) is in this DECIMAL_RANGE.

Partitioning Switches for TCP Flags

When a flow generator creates a flow record from TCP packets, it creates a field that is the bitwise OR of the TCP flags from all packets that comprise that flow record. Some flow generators, such as yaf(1), can export two TCP flag fields: one contains the flags on the first packet in the flow, and the second contains the bitwise OR of the remaining packets.

To partition records based on their TCP flags values, there is a recommended set of switches and legacy-supported switches. The switches accept the following letters to represent the named TCP flag: "F"=FIN; "S"=SYN; "R"=RST; "P"=PSH; "A"=ACK; "U"=URG; "E"=ECE; "C"=CWR.

The recommended set of switches take a comma separated list of pairs of TCP flags, where the pair is separated by a slash (/). The value to the left of the slash is the HIGH_SET and it must be a subset of the value to the right of the slash, which is the MASK_SET. For a record to pass the filter, the flags in the HIGH_SET must be on and the remaining flags in MASK_SET must be off. Flags not in MASK_SET may have any value. If a list of pairs is given, the record passes if any pair in the list matches. For example, "--flags-all=S/S,A/A" passes flows that have either the SYN or the ACK flag set, "--flags-all=S/SA" passes flow records where SYN is high and ACK is low, and "--flags-all=/F" passes flows where FIN is off. This list of flag pairs is called a HIGH_MASK_FLAGS_LIST.

The recommended switches for TCP flag partitioning are:

--flags-all=HIGH_MASK_FLAGS_LIST
    Pass the record if any of the HIGH_SET/MASK_SET pairs is true when looking at the bitwise OR of the TCP flags across all packets in the flow.

--flags-initial=HIGH_MASK_FLAGS_LIST
    As --flags-all, except this switch considers only the initial packet in the flow, for flow generators that can generate that field.

--flags-session=HIGH_MASK_FLAGS_LIST
    As --flags-all, except this switch considers the bitwise OR of the TCP flags across the second through the final packet in the flow; that is, ignoring the flags on the first packet.

The TCP-flag partitioning switches supported for legacy reasons are:

--tcp-flags=TCP_FLAGS
    Pass the record if, for any one of its packets, any of the specified TCP_FLAGS was on, where TCP_FLAGS contains the letters "F","S","R","P","A","U","E","C". For example, --tcp-flags=ASF passes records where ACK is set, or SYN is set, or FIN is set.

--ack-flag={0|1}
    Set to 0, only passes records where the ACK Flag is Low, Set to 1, only passes records where the ACK Flag is high.

--cwr-flag={0|1}
    As --ack-flag for the CWR Flag

--ece-flag={0|1}
    As --ack-flag for the ECE Flag

--fin-flag={0|1}
    As --ack-flag for the ACK Flag

--psh-flag={0|1}
    As --ack-flag for the PSH Flag

--rst-flag={0|1}
    As --ack-flag for the RST Flag

--syn-flag={0|1}
    As --ack-flag for the SYN Flag

--urg-flag={0|1}
    As --ack-flag for the URG Flag

Partitioning Switches for Other Flow Characteristics

Other than the --ip-version switch, the fields queried by the following switches may always be zero. The default configuration of SiLK does not store the fields that contain the SNMP values. The other fields are not present in NetFlow v5, and require use of properly-configured enhanced collection software, such as yaf(1)>, <http://tools.netsa.cert.org/yaf/>.

--ip-version={4|6|4,6}
    Passes the record if its IP Version is in the specified list. If SiLK does not have IPv6 support, the only legal value for this switch is 4.

--application=INTEGER_LIST
    Some flow generation software can inspect the contents of the packets that comprise a flow and use traffic signatures to label the content of the flow. SiLK calls this label the application; yaf refers to it as the appLabel (see the applabel(1) manual page in the yaf distribution). The application value is the port number that is traditionally used for that type of traffic (see the /etc/services file on most UNIX systems). For example, traffic that the flow generator recognizes as FTP will have a value of 21, even if that traffic is being routed through the standard HTTP/web port (80). The flow generator uses a value for 0 if the application cannot be determined. The --application switch passes the flow if the flow's application value is in the specified INTEGER_LIST, which is a comma separated list of integers from 0 to 65535 inclusive and ranges of those integers. The list of valid appLabels is determined by your site's yaf installation.

--attributes=ATTRIBUTES_LIST
    Currently, attributes field in SiLK Flow records contains information related to the closing of flow records. If a TCP session lasts longer than the flow generator's active timeout, the flow generator will close the flow record and open a new flow record. This can occur with a long-lived SSH session or a large HTTP download, for example. The valid attributes values are:

    "T" The flow generator prematurely created a record for a long-running connection due to a timeout. (Also, when yaf is run with the --silk switch, it will prematurely create a flow and mark it with "T" if the byte count of the flow cannot be stored in a 32-bit value.)

    "C" The flow generator created this flow as a continuation of long-running connection, where the previous flow for this connection met a timeout.

    "F" The flow generator saw additional packets in this flow following a packet with a FIN flag set (excluding ACK packets).

    For a long-lived connection spanning several flow records, the first flow record will be marked with a "T" indicating that it hit the active timeout. The second through next-to-last records will be marked with "CT" indicating that the flow is a continuation of a connection that timed out and that this flow also timed out. The final flow will be marked with a "C", indicating that it was created as a continuation of an active flow.

    The ATTRIBUTES_LIST argument is similar to the HIGH_MASK_FLAGS_LIST argument to the --flags-all switch. ATTRIBUTES_LIST is a comma separated list of up to 8 HIGH_ATTRIBUTES/MASK_ATTRIBUTES pairs, where HIGH_ATTRIBUTES and MASK_ATTRIBUTES are strings of the characters "F","T","C", and HIGH_ATTRIBUTES is a subset of MASK_ATTRIBUTES. rwfilter passes the record if, for any pair of attributes in the list, the attributes listed in HIGH_ATTRIBUTES are set and the remaining attributes in MASK_ATTRIBUTES are not-set.

--input-index=INTEGER_LIST
    Pass the record if its "in" field is in this INTEGER_LIST, which is a comma separated list of integers from 0 to 65535, inclusive, and ranges of those integers. When present, the "in" field normally contains the incoming SNMP interface, but it may contain the vlanId if the packing tools were configured to capture it (see sensor.conf(5)).

--output-index=INTEGER_LIST
    Pass the record if its "out" field is in this INTEGER_LIST. When present, the "out" field normally contains the outgoing SNMP interface, but it may contain the postVlanId if the packing tools were configured to capture it.

Selection Switches Acting as Partitioning Switches

The following four switches are normally file selection switches, that is they select which files rwfilter reads within the data repository. However, when rwfilter gets input without querying the data repository (that is, from files listed on the command line, from files specified by --xargs, or from the --input-pipe), these switches become partitioning switches and determine whether a record is written to the pass-destination or fail-destination.

--class=CLASS
    Pass the record if its class is CLASS and its type is listed in the --type switch, or its type is in the default type list for CLASS when --type is not specified. Use the --help option to see the list of available classes and types, and the defaults.

--flowtypes=CLASS/TYPE[,CLASS/TYPE ...]
    Pass the record its if class/type value is one of those listed. The keyword "all" may be used for the CLASS and/or TYPE to select all classes and/or types. This switch cannot be used when either --class or --type is used. Use the --help option to see the list of available classes and types.

--sensors=SENSOR[,SENSOR ...]
    Pass the record if its sensor is one of those listed. The parameter is a comma separated list of sensor names, sensor IDs (integers), and/or ranges of sensor IDs. Use the mapsid(1) command to see the list of sensors.

--type={"all" | TYPE[,TYPE]}
    Pass the record if its type is one of those listed and its class is specified by --class, or its class is the default class when the --class switch is not specified. Use the --help option to see the list of available classes and types, and the defaults.

Partitioning Switches that use Additional Mapping Files

Additional partitioning switches are available that allow one to partition flow records depending on a label, where the label is computed from an IP address or port on the record and an additional mapping file.

--pmap-file=MAPNAME:PATH
--pmap-file=PATH
    Instruct rwfilter to load the mapping file located at PATH and create new switches --pmap-src-MAPNAME, --pmap-dst-MAPNAME, and --pmap-any-MAPNAME. When MAPNAME is provided, it will be used to refer to the switches specific to that prefix map. If MAPNAME is not provided, rwfilter will check the prefix map file to see if a map-name was specified when the file was created. If no map-name is available, rwfilter creates legacy switches as described below. Multiple --pmap-file switches are supported as long as each uses a unique map-name. The --pmap-file switch(es) must precede all other --pmap-* switches. For more information, see pmapfilter(3).

--pmap-src-MAPNAME=LABELS
    If the prefix map associated with MAPNAME is an IP prefix map, this matches records with a source IPv4 address that maps to a label contained in the list of labels in LABELS.

    If the prefix map associated with MAPNAME is a proto-port prefix map, this matches records with a protocol and source port combination that maps to a label contained in the list of labels in LABELS.

--pmap-dst-MAPNAME=LABELS
    Similar to --pmap-src-MAPNAME, but uses the destination IP or the protocol and destination port.

--pmap-any-MAPNAME=LABELS
    If the prefix map associated with MAPNAME is an IP prefix map, this matches records with a source IP address or a destination IP address that maps to a label contained in the list of labels in LABELS.

    If the prefix map associated with MAPNAME is a port/protocol prefix map, this matches records with a protocol and source port or destination port combination that maps to a label contained in the list of labels in LABELS.

--pmap-saddress=LABELS
--pmap-daddress=LABELS
--pmap-any-address=LABELS
    These are deprecated switches created by pmapfilter that correspond to --pmap-src-MAPNAME, --pmap-dst-MAPNAME, and --pmap-any-MAPNAME, respectively. These switches are available when an IP prefix map is used that is not associated with a MAPNAME.

--pmap-sport-proto=LABELS
--pmap-dport-proto=LABELS
--pmap-any-port-proto=LABELS
    These are deprecated switches created by pmapfilter that correspond to --pmap-src-MAPNAME, --pmap-dst-MAPNAME, and --pmap-any-MAPNAME, respectively. These switches are available when a proto-port prefix map is used that is not associated with a MAPNAME.

--scc=COUNTRY_CODE_LIST
--dcc=COUNTRY_CODE_LIST
    Pass the record if the country code of its source (--scc) IP address or destination (--dcc) IP address is in the specified COUNTRY_CODE_LIST, where COUNTRY_CODE_LIST is a comma separated list of lowercase two-letter country codes---based on the Root-Zone Whois Index (see for example http://www.iana.org/cctld/cctld-whois.htm <http://www.iana.org/cctld/cctld-whois.htm>)---as well as the following special codes:

    "--"
        N/A (e.g. private and experimental reserved addresses)

    "a1"
        anonymous proxy

    "a2"
        satellite provider

    "o1"
        other

    For example: "cx,uk,kr,jp,--". To use this switch, the country code mapping file must be available in the default location, or in the location specified by the SILK_COUNTRY_CODES environment variable. See ccfilter(3) for details.

--stype={0|1|2|3}
--dtype={0|1|2|3}
    Pass a flow record depending on whether the IP address is internal, external, or non-routable. These switches use the mapping file specified by the SILK_ADDRESS_TYPES environment variable, or the address_types.pmap mapping file, as described in addrtype(3). When the parameter is 0, pass the record if its source (--stype) IP address or destination (--dtype) IP address is non-routable. When 1, pass if internal. When 2, pass if external (i.e., routable but not internal). When 3, pass if not internal (non-routable or external).

Partitioning Switches across Multiple Fields

The --tuple-* family of switches allows the user to partition flow records based on multiple values of the five-tuple.

--tuple-file=TUPLE_FILENAME
    This switch provides support for partitioning by arbitrary subsets of the basic five-tuple:

     {source-ip,destination-ip,source-port,destination-ip-port,protocol}

    A SiLK Flow record will pass the test when the record's fields match one of the tuples; if the SiLK record does not match any tuple, the record fails. The tuples are read from the text file TUPLE_FILENAME which must contain lines of delimited fields. The default delimiter is "|", but may be specified with the --tuple-delimiter switch. Each field contains one member of the tuple; the fields may appear in any order. The fields may represent any subset of the five-tuple, but each line in the file must define the same subset. A field that is present but has no value will generate an error. If you want the field to match any value, it is best that you not include that field in your input.

    In addition to the tuple-lines, TUPLE_FILENAME may contain blank lines and comments (which begin with "#" and continue to the end of the line). The first line of TUPLE_FILENAME may contain a title labeling the fields in the file. This title line will be ignored when the --tuple-fields switch is given.

    The IP fields may contain an IPv4 address, an integer, or a IP in CIDR block notation. Comma-separated lists ("80,443") and ranges ("0-1023,8080") are supported for the ports and protocol fields. NOTE: Currently the code is not clever in its support for CIDR notation and ranges in that each occurrence is fully expanded. When this occurs, the memory required to hold the search tree will quickly grow.

--tuple-fields=FIELDS
    FIELDS contains the list of fields (columns) to parse from the TUPLE_FILENAME in the order in which they appear in the file. When this switch is not provided, rwfilter will treat the first line in TUPLE_FILENAME as a title line and attempt to determine the fields (a la rwtuc(1)); rwfilter will exit if it cannot determine the fields.

    FIELDS is a comma separated list of field-names, field-integers, and ranges of field-integers; a range is specified by separating the start and end of the range with a hyphen (-). Names can be abbreviated to their shortest unique prefix. The field names and their descriptions are:

    sIP,sip,1
        source IP address

    dIP,dip,2
        destination IP address

    sPort,sport,3
        source port

    dPort,dport,4
        destination port

    protocol,5
        IP protocol

--tuple-direction=DIRECTION
    Allows you to change the comparison between the tuple and the SiLK Flow record. This switch allows one to look for traffic in the reverse direction (or both directions) without having to write all of the rules twice. The available directions are:

    forward
        The tuple's fields are compared against the corresponding fields on the flow; that is, sIP is compared with sIP, dIP with dIP, sPort with sPort, dPort with dPort, and protocol with protocol. This is the default.

    reverse
        The tuple's fields are compared against the opposite fields on the flow; that is, sIP is compared with dIP, dIP with sIP, sPort with dPort, dPort with sPort, and protocol with protocol.

    both
        Both of the above comparisons are performed.

--tuple-delimiter=CHAR
    Specifies the character separating the input fields. When the switch is not provided, the default of "|" is used.

--ippair-any=FILENAME
    Pass the record if the source IP and destination IP (in either order) match one of the IP-pairs listed in the text file FILENAME. Each line of FILENAME should contain two IP addresses separated by whitespace. This switch is equivalent to --tuple-file=FILENAME --tuple-fields=sIP,dIP --tuple-direction=both --tuple-delimiter=' '. You cannot use this switch in conjunction with --tuple-file or --ipport-any. This switch is deprecated and it exists for backward compatibility only; it may be removed in a future release.

--ipport-any=FILENAME
    Pass the record if either the source IP and port pair or the destination IP and port pair are listed in the text file FILENAME. Each line in FILENAME should contain an IP address and port list of interest for that IP separated by whitespace. This switch is equivalent to --tuple-file=FILENAME --tuple-fields=sIP,sPort --tuple-direction=both --tuple-delimiter=' '. You cannot use this switch in conjunction with --tuple-file or --ippair-any. This switch is deprecated and it exists for backward compatibility only; it may be removed in a future release.

Partitioning Switches that use the PySiLK Plug-in

The SiLK Python plug-in, silkpython.so, provides support for filtering by expressions or complex functions written in the Python programming language. See the silkpython(3) and pysilk(3) manual pages for information and examples for how to use Python to manipulate SiLK data structures. When multiple Partitioning Switches are given, the Python plug-in will be the next-to-last to be invoked. Only the code specified by the --plugin switch is called after the Python code.

--python-file=FILENAME
    Pass the record if the result of the processing the flow with the function named rwfilter() in FILENAME is true. The function should take a single silk.RWRec object as an argument. See silkpython(3) for details.

--python-expr=PYTHON_EXPRESSION
    Pass the record if the result of the processing the flow with the specified PYTHON_EXPRESSION is true. The expression is evaluated as if it appeared in the following context:

     from silk import *
     def rwfilter(rec):
         return (PYTHON_EXPRESSION)

Partitioning Switches that use the IP-Association Plug-In

The IPA plug-in, ipafilter.so, provides switches that can partition flows using data in an IP Association database. For this plug-in to be available, SiLK must be compiled with IPA support and IPA must be configured. See ipafilter(3) and <http://tools.netsa.cert.org/ipa/> for additional information.

--ipa-src-expr=IPA_EXPR
    Use IPA_EXPR to partition flows based on the source IP of the flow matching the IPA_EXPR expression.

--ipa-dst-expr=IPA_EXPR
    Use IPA_EXPR to partition flows based on the destination IP of the flow matching the IPA_EXPR expression.

--ipa-any-expr=IPA_EXPR
    Use IPA_EXPR to partition flows based on either the source or destination IP of the flow matching the IPA_EXPR expression.

  Miscellaneous Switches

--compression-method=COMP_METHOD
    Set the compression method of the output flow record files to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

    none
        Do not compress the output using an external library

    zlib
        Use the zlib(3) library for compressing the output

    lzo1x
        Use the lzo1x algorithm from the LZO real time compression library for compression

    best
        Use whichever available method gives the "best" compression in general, though not necessarily the "best" for this particular output.

--dry-run
    Perform a sanity check on the input arguments to check that the arguments are acceptable. In addition, prints to the standard output the names of the files that would be accessed (and the names of missing files if --print-missing is specified). rwfglob(1) can also be used to generate the lists of files that rwfilter will access.

--dynamic-library=PLUGIN
    This switch is deprecated. It is an alias for --plugin.

--help
    Print the available options and exit. Options that add fields (for example, options that load plug-ins, prefix maps, or PySiLK extensions) can be specified before the --help switch so that the new options appear in the output. The available classes and types will be included in output; you may specify a different root directory or site configuration file before --help to see the classes and types available for that site.

--max-fail-records=N
    Write N records to each --fail-destination. rwfilter will stop reading input once it has written these N records unless additional "Output Switches" were specified.

--max-pass-records=N
    Write N records to each --pass-destination. rwfilter will stop reading input once it has written these N records unless additional "Output Switches" were specified.

--note-add=TEXT
    Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME
    Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--plugin=PLUGIN
    Augment the partitioning switches by using run-time loading of the plug-in (shared object) whose path is PLUGIN. The switch may be repeated to load multiple plug-ins. The creation of plug-ins is described in the silk-plugin(3) manual page. When multiple partitioning switches are given, the code specified by the --plugin switch(es) will be last to be invoked. When PLUGIN contains a slash ("/"), rwfilter assumes the path to PLUGIN is correct. Otherwise, rwfilter will attempt to find the file in $SILK_PATH/lib/silk, $SILK_PATH/share/lib, $SILK_PATH/lib, and in these directories parallel to the application's directory: lib/silk, share/lib, and lib. If rwfilter does not find the file, it assumes the plug-in is in the current directory. To force rwfilter to look in the current directory first, specify --plugin=./PLUGIN. When the SILK_PLUGIN_DEBUG environment variable is non-empty, rwfilter prints status messages to the standard error as it tries to open each of its plug-ins.

--print-filenames
    Print the names of input files as they are read. This can be useful feedback for a long-running rwfilter process.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the root of the data directory (see --data-rootdir); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application's directory.

--threads=N
    Invoke rwfilter with N threads reading the input files. When this switch is not provided, the value in the SILK_RWFILTER_THREADS environment variable is used. If that variable is not set, rwfilter runs with a single thread. Using multiple threads, performance of rwfilter is greatly improved for queries that look at many files but return few records. Preliminary testing has found that performance peaks around four threads per CPU, but performance will vary depending on the type of query and the number of records returned.

--version
    Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES
In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash ("\") is used to indicate a wrapped line.

The most basic filtering involves looking at specific traffic over a specific time. For example:

 $ rwfilter --start-date=2003/02/19:00 --end-date=2003/02/19:23 \
       --proto=6 --pass-destination=tcp-in.rwf

will create a file, tcp-in.rwf containing all incoming TCP traffic on February 19, 2003. The tcp-in.rwf file contains SiLK Flow data in a binary format. To examine the contents, use the command rwcut(1). This query only selects incoming traffic because the silk.conf(5) configuration file at most sites tells rwfilter to look at incoming traffic unless an explicit --type switch is given. The following query gets all TCP traffic (for the default class) for February 19, 2003.

 $ rwfilter --type=all --start-date=2003/02/19  \
       --proto=6 --pass-destination=alltcp.rwf

Note the addition of --type=all. This query also relies on the default behavior of --start-date to consider a full day's worth of data when no hour is specified.

The above query gets all traffic for the default class. If your silk.conf file has a single class, that query will capture all of it. For silk.conf files that specify multiple classes, the following will get all TCP traffic for February 19, 2003:

 $ rwfilter --flowtypes=all/all --start-date=2003/02/19  \
       --proto=6 --pass-destination=alltcp.rwf

To get all non-TCP traffic, there are two approaches. rwfilter does not supply a way to choose a negatate set of protocols, but you can choose all protocols other than TCP:

 $ rwfilter --start-date=2003/02/19:00 --end-date=2003/02/19:23  \
       --proto=0-5,7-255 --pass-destination=non-tcp.rwf

The other approach is to use the --fail-destination switch:

 $ rwfilter --start-date=2003/02/19:00 --end-date=2003/02/19:23  \
       --proto=6 --fail-destination=non-tcp.rwf

To print information about the number of flow records that pass a filter, use --print-volume-statistics. This can be combined with other output switches.

 $ rwfilter --start-date=2003/02/19:00 --end-date=2003/02/19:23    \
      --proto=6 --print-volume-stat --pass-destination=tcp-in.rwf
      |        Recs|     Packets|         Bytes|  Files|
 Total|      515359|     2722887|    1343819719|    180|
  Pass|      512071|     2706571|    1342851708|       |
  Fail|        3288|       16316|        968011|       |

If you want to see the number of records in a file produced by rwfilter, or to remind yourself how a file was created, use rwfileinfo(1):

 $ rwfileinfo tcp-in.rwf
 tcp-in.rwf:
   format(id)          FT_RWGENERIC(0x16)
   version             16
   byte-order          littleEndian
   compression(id)     lzo1x(2)
   header-length       208
   record-length       52
   record-version      5
   silk-version        2.4.0
   count-records       512071
   file-size           8576160
   command-lines
       1  rwfilter --start-date=2003/02/19:00 --end-date=2003/02/19:23 \
            --proto=6 --print-volume-stat --pass-destination=tcp-in.rwf

Once a file is written, rwfilter can process the file again. Traffic on port 25 is most likely email (SMTP) traffic. To split the email traffic from the other traffic, use:

 $ rwfilter --aport=25 --pass=mail.rwf --fail=not-mail.rwf tcp-in.rwf

This command puts traffic where the source or destination port was 25 into the file mail.rwf, and all other traffic into the file not-mail.rwf. The --fail-destination is an effective way to reverse the sense of a test. For example, to remove traffic on port 80 from the not-mail.rwf file, run the command:

 $ rwfilter --aport=80 --fail=not-mail-web.rwf not-mail.rwf

To verify that the not-mail-web.rwf file does not contain any traffic on ports 25 or 80, you can use the --print-statistics switch and see that 0 records pass:

 $ rwfilter --aport=25,80 --print-stat not-mail-web.rwf
 Files     1.  Read    54641.  Pass        0. Fail     54641.

The file maintains a history of the commands that created it:

 $ rwfileinfo not-mail-web.rwf
 not-mail-web.rwf:
   format(id)          FT_RWGENERIC(0x16)
   version             16
   byte-order          littleEndian
   compression(id)     lzo1x(2)
   header-length       364
   record-length       52
   record-version      5
   silk-version        2.4.0
   count-records       54641
   file-size           762875
   command-lines
       1  rwfilter --start-date=2003/02/19:00 --end-date=2003/02/19:23 \
            --proto=6 --print-volume-stat --pass-destination=tcp-in.rwf
       2  rwfilter --aport=25 --pass=mail.rwf --fail=not-mail.rwf      \
            tcp-in.rwf
       3  rwfilter --aport=80 --fail=not-mail-web.rwf not-mail.rwf

The following finds all outgoing traffic from February 19, 2003, going to an external email server. Traffic going to a server will contact that server on its well-known port, and the flow record's destination port should hold that well-known port:

 $ rwfilter --type=out --start-date=2003/02/19 --print-volume-stat \
       --dport=25 --proto=6

To limit the result to completed connections, select flow records that contain at least three packets, use the --packets switch with an open-ended range:

 $ rwfilter --type=out --start-date=2003/02/19 --print-volume-stat \
       --dport=25 --proto=6 --packets=3-

To limit the search to a particular internal CIDR block, 10.1.2.0/24, there are three different IP-partitioning switches you can use. The final approach uses rwsetbuild(1) to create an IPset file from textual input.

 $ rwfilter --type=out --start-date=2003/02/19 --print-volume-stat \
       --dport=25 --proto=6 --packets=3- --scidr=10.1.2.0/24

 $ rwfilter --type=out --start-date=2003/02/19 --print-volume-stat \
       --dport=25 --proto=6 --packets=3- --saddress=10.1.2.x

 $ echo "10.1.2.0/24" | rwsetbuild > my-set.set
 $ rwfilter --type=out --start-date=2003/02/19 --print-volume-stat \
       --dport=25 --proto=6 --packets=3- --sipset=my-set.set

rwfilter does not have to output its records to a file. The output from rwfilter can be piped into a another SiLK tool. To get the IPs of the external email servers that the monitored network contacted, pipe the rwfilter output into rwset(1), and tell rwset to store the destination addresses:

 $ rwfilter --type=out --start-date=2003/02/19 --print-volume-stat \
       --dport=25 --proto=6 --packets=3- --scidr=10.1.2.0/24       \
   | rwset --dip-file=external-mail-servers.set

rwfilter can also pipe input into another rwfilter command, which allows them to be chained together:

 $ rwfilter --type=out,outweb --start-date=2003/02/19   \
        --scidr=10.1.2.0/24 --pass=stdout               \
   | rwfilter --input-pipe=stdin --proto=17             \
        --pass=udp.rwf --fail=stdout                    \
   | rwfilter --input-pipe=stdin --proto=6              \
        --pass=stdout --fail=non-tcp-udp.rwf            \
   | rwfilter --input-pipe=stdin --aport=25             \
        --pass=mail.rwf --fail=stdout                   \
   | rwfilter --input-pipe=stdin --aport=80,443         \
        --pass=web.rwf --fail=tcp-non-web-mail.rwf

This chain of commands looks at outgoing traffic on February 19, 2003, coming from the internal net-block 10.1.2.0/24, creates the following files:

udp.rwf
    Outgoing UDP traffic

non-tcp-udp.rwf
    Outgoing traffic that is neither TCP nor UDP

mail.rwf
    Outgoing TCP traffic on port 25, most of which is probably email (SMTP). Since the query looks at outgoing traffic and the --aport switch was used, this file represents email going from the internal 10.1.2.0/24 to external mail servers, and the responses from any internal mail servers that exist in the 10.1.2.0/24 net-block to external clients.

web.rwf
    Outgoing TCP traffic on ports 80 and 443, most of which is probably web traffic (HTTP,HTTPS). As with the mail.rwf file, this file represents queries to external web servers and responses from internal web servers.

tcp-non-web-mail.rwf
    Outgoing TCP traffic other than that on ports 25, 80, and 443

Expert users can create even more complicated chains of rwfilter commands using named pipes.

ENVIRONMENT

SILK_RWFILTER_THREADS
    The number of threads to use while reading input files or files selected from the data store.

PYTHONPATH
    This environment variable is used by Python to locate modules. When --python-file or --python-expr is specified, rwfilter loads Python which in turn loads the PySiLK module which is comprised of several files (silk/pysilk_nl.so, silk/__init__.py, etc). If this silk/ directory is located outside Python's normal search path (for example, in the SiLK installation tree), it may be necessary to set or modify the PYTHONPATH environment variable to include the parent directory of silk/ so that Python can find the PySiLK module. For information on using Python from within rwfilter, see pysilk(3).

SILK_PYTHON_TRACEBACK
    When set, Python plug-ins will output traceback information on Python errors to stderr.

SILK_COUNTRY_CODES
    This environment variable allows the user to specify the country code mapping file that the --scc and --dcc switches use. The value may be a complete path or a file relative to the SILK_PATH. If the variable is not specified, rwcut checks the following locations:

      $SILK_PATH/share/silk/country_codes.pmap
      $SILK_PATH/share/country_codes.pmap
      /usr/local/share/silk/country_codes.pmap
      /usr/local/share/country_codes.pmap

SILK_ADDRESS_TYPES
    This environment variable allows the user to specify the address type mapping file that the --stype and --dtype switches use. The value may be a complete path or a file relative to the SILK_PATH. If the variable is not specified, rwcut checks the following locations:

      $SILK_PATH/share/silk/address_types.pmap
      $SILK_PATH/share/address_types.pmap
      /usr/local/share/silk/address_types.pmap
      /usr/local/share/address_types.pmap

SILK_CONFIG_FILE
    This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR
    When set, overrides the compiled-in value for the location of the directory tree containing the files of SiLK Flow records collected and stored by the packing system (rwflowpack(8)). In addition, when the --site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwfilter looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH
    This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwfilter checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share. These directories are also searched when any other configuration file is required (e.g., the country code map). In addition, rwfilter looks for plug-ins in $SILK_PATH/lib/silk, $SILK_PATH/share/lib and $SILK_PATH/lib.

SILK_PLUGIN_DEBUG
    When set to 1, rwfilter prints status messages to the standard error as it tries to open each of its plug-ins.

SILK_LOGSTATS
    When set to a non-empty value, rwfilter will treat the value as the path to an external program to execute with information about this rwfilter invocation. If the value in SILK_LOGSTATS does not contain a slash or if it references a file that does not exist, is not a regular file, or is not executable, the SILK_LOGSTATS value is silently ignored. The arguments to the external program are:

    •   The application name, i.e., "rwfilter". Note that "rwfilter" is always used as this argument, regardless of the name of the executable.

    •   The version number of this command line, currently "v0001".

    •   The start time of this invocation, as seconds since the UNIX epoch.

    •   The end time of this invocation, as seconds since the UNIX epoch.

    •   The number of data files opened for reading.

    •   The number of records read.

    •   The number of records written.

    •   A variable number of arguments that are the complete command line used to invoke rwfilter, including the name of the executable.

SILK_LOGSTATS_RWFILTER
    If set, this environment variable overrides the value specified in SILK_LOGSTATS.

SILK_LOGSTATS_DEBUG
    If the environment variable is set to a non-empty value, rwfilter will print messages to the standard error about the SILK_LOGSTATS value being used and either the reason why the value cannot be used or the arguments to the external program being executed.

NOTES
rwfilter is the most commonly used application in the suite. It provides access to the data files and performs all the basic queries.

rwfilter supports a variety of I/O options - in addition to reading from the data store, rwfilter results can be chained together with named pipes to output results to multiple files simultaneously. An introduction to named pipes is outside the scope of this document, however.

Two often underused options are --dry-run and --print-statistics. --dry-run performs a sanity check on the arguments and can be used, especially for complicated arguments, to check that the arguments are acceptable. --print-statistics used without --pass-destination or --fail-destination simply prints aggregate statistics to the standard error on a single line, and it can be used to do a quick pass through the data to get aggregate counts before going in deeper into the phenomenon being investigated.

--print-filename can be used as a progress meter; during long jobs, it shows which file is currently being read by rwfilter. --print-filename will not provide meaningful feedback with piped input.

Filters are applied in the order given on the command line. It is best to apply the biggest filters first.

The rwfilter command line will be written into the header of the output file(s). You may use the rwfileinfo(1) command to see this information.

SEE ALSO
rwcount(1), rwcut(1), rwfglob(1), rwfileinfo(1), rwset(1), rwsort(1), rwstats(1), rwtotal(1), rwuniq(1), rwtuc(1), rwsetbuild(1), mapsid(1), addrtype(3), ccfilter(3), pmapfilter(3), pysilk(3), silkpython(3), silk-plugin(3), silk.conf(5), sensor.conf(5), silk(7), rwflowpack(8), yaf(1), zlib(3), Analysts' Handbook: Using SiLK for Network Traffic Analysis