### Home Classes Methods

### In Files

- ext/r_analysis.c

### Parent

Object

### Methods

- ::new
- #<=>
- #end
- #end=
- #pos_inc
- #pos_inc=
- #start
- #start=
- #text
- #text=
- #to_s

### Included Modules

- Comparable

### Files

- MIT-LICENSE
- README
- TODO
- TUTORIAL

### Class/Module Index

Quicksearch

- Ferret
- Ferret::Analysis
- Ferret::Analysis::Analyzer
- Ferret::Analysis::AsciiLetterAnalyzer
- Ferret::Analysis::AsciiLetterTokenizer
- Ferret::Analysis::AsciiLowerCaseFilter
- Ferret::Analysis::AsciiStandardAnalyzer
- Ferret::Analysis::AsciiStandardTokenizer
- Ferret::Analysis::AsciiWhiteSpaceAnalyzer
- Ferret::Analysis::AsciiWhiteSpaceTokenizer
- Ferret::Analysis::HyphenFilter
- Ferret::Analysis::LetterAnalyzer
- Ferret::Analysis::LetterTokenizer
- Ferret::Analysis::LowerCaseFilter
- Ferret::Analysis::MappingFilter
- Ferret::Analysis::PerFieldAnalyzer
- Ferret::Analysis::RegExpAnalyzer
- Ferret::Analysis::RegExpTokenizer
- Ferret::Analysis::StandardAnalyzer
- Ferret::Analysis::StandardTokenizer
- Ferret::Analysis::StemFilter
- Ferret::Analysis::StopFilter
- Ferret::Analysis::Token
- Ferret::Analysis::TokenStream
- Ferret::Analysis::WhiteSpaceAnalyzer
- Ferret::Analysis::WhiteSpaceTokenizer
- Ferret::BlankSlate
- Ferret::BoostMixin
- Ferret::Browser
- Ferret::Browser::Controller
- Ferret::Browser::Delegator
- Ferret::Browser::DocumentController
- Ferret::Browser::ErrorController
- Ferret::Browser::HelpController
- Ferret::Browser::HomeController
- Ferret::Browser::TermController
- Ferret::Browser::TermVectorController
- Ferret::Browser::ViewHelper
- Ferret::Document
- Ferret::Field
- Ferret::FieldSymbol
- Ferret::FieldSymbolMethods
- Ferret::FileNotFoundError
- Ferret::Index
- Ferret::Index::FieldInfo
- Ferret::Index::FieldInfos
- Ferret::Index::Index
- Ferret::Index::IndexReader
- Ferret::Index::IndexWriter
- Ferret::Index::LazyDoc
- Ferret::Index::LazyDoc::LazyDocData
- Ferret::Index::TermDocEnum
- Ferret::Index::TermEnum
- Ferret::Index::TermVector
- Ferret::Index::TermVector::TVOffsets
- Ferret::Index::TermVector::TVTerm
- Ferret::ParseError
- Ferret::QueryParser
- Ferret::QueryParser::QueryParseException
- Ferret::Search
- Ferret::Search::BooleanQuery
- Ferret::Search::BooleanQuery::BooleanClause
- Ferret::Search::ConstantScoreQuery
- Ferret::Search::Explanation
- Ferret::Search::Filter
- Ferret::Search::FilteredQuery
- Ferret::Search::FuzzyQuery
- Ferret::Search::Hit
- Ferret::Search::MatchAllQuery
- Ferret::Search::MultiSearcher
- Ferret::Search::MultiTermQuery
- Ferret::Search::PhraseQuery
- Ferret::Search::PrefixQuery
- Ferret::Search::Query
- Ferret::Search::QueryFilter
- Ferret::Search::RangeFilter
- Ferret::Search::RangeQuery
- Ferret::Search::Searcher
- Ferret::Search::Sort
- Ferret::Search::SortField
- Ferret::Search::Spans
- Ferret::Search::Spans::SpanFirstQuery
- Ferret::Search::Spans::SpanMultiTermQuery
- Ferret::Search::Spans::SpanNearQuery
- Ferret::Search::Spans::SpanNotQuery
- Ferret::Search::Spans::SpanOrQuery
- Ferret::Search::Spans::SpanPrefixQuery
- Ferret::Search::Spans::SpanTermQuery
- Ferret::Search::TermQuery
- Ferret::Search::TopDocs
- Ferret::Search::TypedRangeFilter
- Ferret::Search::TypedRangeQuery
- Ferret::Search::WildcardQuery
- Ferret::StateError
- Ferret::Store
- Ferret::Store::Directory
- Ferret::Store::FSDirectory
- Ferret::Store::Lock
- Ferret::Store::Lock::LockError
- Ferret::Store::RAMDirectory
- Ferret::Utils
- Ferret::Utils::BitVector
- Ferret::Utils::MultiMapper
- Ferret::Utils::PriorityQueue
- WEBrick
- WEBrick::FerretBrowserHandler
- Date
- DateTime
- FileUtils
- Float
- Integer
- String
- Symbol
- Time

No matching classes.

# Ferret::Analysis::Token

## Summary

A Token is an occurrence of a term from the text of a field. It consists of a term's text and the start and end offset of the term in the text of the field;

The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC (KeyWord In Context) display, etc.

### Attributes

text | the terms text which may have been modified by a Token Filter or Tokenizer from the text originally found in the document
start | is the position of the first character corresponding to this token in the source text
end | is equal to one greater than the position of the last character corresponding of this token Note that the difference between @end_offset and @start_offset may not be equal to @text.length(), as the term text may have been altered by a stemmer or some other filter.

### Public Class Methods

new(text, start, end, pos_inc = 1) → new Token click to toggle source

Creates a new token setting the text, start and end offsets of the token and the position increment for the token.

The position increment is usually set to 1 but you can set it to other values as needed. For example, if you have a stop word filter you will be skipping tokens. Let's say you have the stop words "the" and "and" and you parse the title "The Old Man and the Sea". The terms "Old", "Man" and "Sea" will have the position increments 2, 1 and 3 respectively.

Another reason you might want to vary the position increment is if you are adding synonyms to the index. For example let's say you have the synonym group "quick", "fast" and "speedy". When tokenizing the phrase "Next day speedy delivery", you'll add "speedy" first with a position increment of 1 and then "fast" and "quick" with position increments of 0 since they are represented in the same position.

The offset set values `start` and `end` should be byte offsets, not character offsets. This makes it easy to use those offsets to quickly access the token in the input string and also to insert highlighting tags when necessary.

text | the main text for the token.
start | the start offset of the token in bytes.
end | the end offset of the token in bytes.
pos_inc | the position increment of a token. See above.
return | a newly created and assigned Token object

    static VALUE
    frb_token_init(int argc, VALUE *argv, VALUE self)
    {
        RToken *token;
        VALUE rtext, rstart, rend, rpos_inc, rtype;
        GET_TK(token, self);
        token->pos_inc = 1;
        switch (rb_scan_args(argc, argv, "32", &rtext, &rstart,
                             &rend, &rpos_inc, &rtype)) {
            case 5: /* type gets ignored at this stage */
            case 4: token->pos_inc = FIX2INT(rpos_inc);
        }
        token->text = rb_obj_as_string(rtext);
        token->start = FIX2INT(rstart);
        token->end = FIX2INT(rend);
        return self;
    }

### Public Instance Methods

cmp(other_token) → bool click to toggle source

Used to compare two tokens. Token is extended by Comparable so you can also use +<+, +>+, +<=+, +>=+ etc. to compare tokens.

Tokens are sorted by the position in the text at which they occur, ie the start offset. If two tokens have the same start offset, (see pos_inc=) then, they are sorted by the end offset and then lexically by the token text.

    static VALUE
    frb_token_cmp(VALUE self, VALUE rother)
    {
        RToken *token, *other;
        int cmp;
        GET_TK(token, self);
        GET_TK(other, rother);
        if (token->start > other->start) {
            cmp = 1;
        } else if (token->start < other->start) {
            cmp = -1;
        } else {
            if (token->end > other->end) {
                cmp = 1;
            } else if (token->end < other->end) {
                cmp = -1;
            } else {
                cmp = strcmp(rs2s(token->text), rs2s(other->text));
            }
        }
        return INT2FIX(cmp);
    }

end → integer click to toggle source

End byte-position of this token

    static VALUE
    frb_token_get_end_offset(VALUE self)
    {
        RToken *token;
        GET_TK(token, self);
        return INT2FIX(token->end);
    }

end = end → integer click to toggle source

Set end byte-position of this token

    static VALUE
    frb_token_set_end_offset(VALUE self, VALUE rend)
    {
        RToken *token;
        GET_TK(token, self);
        token->end = FIX2INT(rend);
        return rend;
    }

pos_inc → integer click to toggle source

Position Increment for this token

    static VALUE
    frb_token_get_pos_inc(VALUE self)
    {
        RToken *token;
        GET_TK(token, self);
        return INT2FIX(token->pos_inc);
    }

pos_inc = pos_inc → integer click to toggle source

Set the position increment. This determines the position of this token relative to the previous Token in a TokenStream, used in phrase searching.

The default value is 1.

Some common uses for this are:

- Set it to zero to put multiple terms in the same position. This is useful if, e.g., a word has multiple stems. Searches for phrases including either stem will match. In this case, all but the first stem's increment should be set to zero: the increment of the first instance should be one. Repeating a token with an increment of zero can also be used to boost the scores of matches on that token.

- Set it to values greater than one to inhibit exact phrase matches. If, for example, one does not want phrases to match across removed stop words, then one could build a stop word filter that removes stop words and also sets the increment to the number of stop words removed before each non-stop word. Then exact phrase queries will only match when the terms occur with no intervening stop words.

    static VALUE
    frb_token_set_pos_inc(VALUE self, VALUE rpos_inc)
    {
        RToken *token;
        GET_TK(token, self);
        token->pos_inc = FIX2INT(rpos_inc);
        return rpos_inc;
    }

start → integer click to toggle source

Start byte-position of this token

    static VALUE
    frb_token_get_start_offset(VALUE self)
    {
        RToken *token;
        GET_TK(token, self);
        return INT2FIX(token->start);
    }

start = start → integer click to toggle source

Set start byte-position of this token

    static VALUE
    frb_token_set_start_offset(VALUE self, VALUE rstart)
    {
        RToken *token;
        GET_TK(token, self);
        token->start = FIX2INT(rstart);
        return rstart;
    }

text → text click to toggle source

Returns the text that this token represents

    static VALUE
    frb_token_get_text(VALUE self)
    {
        RToken *token;
        GET_TK(token, self);
        return token->text;
    }

text = text → text click to toggle source

Set the text for this token.

    static VALUE
    frb_token_set_text(VALUE self, VALUE rtext)
    {
        RToken *token;
        GET_TK(token, self);
        token->text = rtext;
        return rtext;
    }

to_s → token_str click to toggle source

Return a string representation of the token

    static VALUE
    frb_token_to_s(VALUE self)
    {
        RToken *token;
        char *buf;
        GET_TK(token, self);
        buf = alloca(RSTRING_LEN(token->text) + 80);
        sprintf(buf, "token[\"%s\":%d:%d:%d]", rs2s(token->text),
                token->start, token->end, token->pos_inc);
        return rb_str_new2(buf);
    }

[Validate]

Generated with the Darkfish Rdoc Generator 2.