### Home Classes Methods

### In Files

- ext/r_analysis.c

### Namespace

- CLASS Ferret::Analysis::Analyzer
- CLASS Ferret::Analysis::AsciiLetterAnalyzer
- CLASS Ferret::Analysis::AsciiLetterTokenizer
- CLASS Ferret::Analysis::AsciiLowerCaseFilter
- CLASS Ferret::Analysis::AsciiStandardAnalyzer
- CLASS Ferret::Analysis::AsciiStandardTokenizer
- CLASS Ferret::Analysis::AsciiWhiteSpaceAnalyzer
- CLASS Ferret::Analysis::AsciiWhiteSpaceTokenizer
- CLASS Ferret::Analysis::HyphenFilter
- CLASS Ferret::Analysis::LetterAnalyzer
- CLASS Ferret::Analysis::LetterTokenizer
- CLASS Ferret::Analysis::LowerCaseFilter
- CLASS Ferret::Analysis::MappingFilter
- CLASS Ferret::Analysis::PerFieldAnalyzer
- CLASS Ferret::Analysis::RegExpAnalyzer
- CLASS Ferret::Analysis::RegExpTokenizer
- CLASS Ferret::Analysis::StandardAnalyzer
- CLASS Ferret::Analysis::StandardTokenizer
- CLASS Ferret::Analysis::StemFilter
- CLASS Ferret::Analysis::StopFilter
- CLASS Ferret::Analysis::Token
- CLASS Ferret::Analysis::TokenStream
- CLASS Ferret::Analysis::WhiteSpaceAnalyzer
- CLASS Ferret::Analysis::WhiteSpaceTokenizer

### Files

- MIT-LICENSE
- README
- TODO
- TUTORIAL

### Class/Module Index

Quicksearch

- Ferret
- Ferret::Analysis
- Ferret::Analysis::Analyzer
- Ferret::Analysis::AsciiLetterAnalyzer
- Ferret::Analysis::AsciiLetterTokenizer
- Ferret::Analysis::AsciiLowerCaseFilter
- Ferret::Analysis::AsciiStandardAnalyzer
- Ferret::Analysis::AsciiStandardTokenizer
- Ferret::Analysis::AsciiWhiteSpaceAnalyzer
- Ferret::Analysis::AsciiWhiteSpaceTokenizer
- Ferret::Analysis::HyphenFilter
- Ferret::Analysis::LetterAnalyzer
- Ferret::Analysis::LetterTokenizer
- Ferret::Analysis::LowerCaseFilter
- Ferret::Analysis::MappingFilter
- Ferret::Analysis::PerFieldAnalyzer
- Ferret::Analysis::RegExpAnalyzer
- Ferret::Analysis::RegExpTokenizer
- Ferret::Analysis::StandardAnalyzer
- Ferret::Analysis::StandardTokenizer
- Ferret::Analysis::StemFilter
- Ferret::Analysis::StopFilter
- Ferret::Analysis::Token
- Ferret::Analysis::TokenStream
- Ferret::Analysis::WhiteSpaceAnalyzer
- Ferret::Analysis::WhiteSpaceTokenizer
- Ferret::BlankSlate
- Ferret::BoostMixin
- Ferret::Browser
- Ferret::Browser::Controller
- Ferret::Browser::Delegator
- Ferret::Browser::DocumentController
- Ferret::Browser::ErrorController
- Ferret::Browser::HelpController
- Ferret::Browser::HomeController
- Ferret::Browser::TermController
- Ferret::Browser::TermVectorController
- Ferret::Browser::ViewHelper
- Ferret::Document
- Ferret::Field
- Ferret::FieldSymbol
- Ferret::FieldSymbolMethods
- Ferret::FileNotFoundError
- Ferret::Index
- Ferret::Index::FieldInfo
- Ferret::Index::FieldInfos
- Ferret::Index::Index
- Ferret::Index::IndexReader
- Ferret::Index::IndexWriter
- Ferret::Index::LazyDoc
- Ferret::Index::LazyDoc::LazyDocData
- Ferret::Index::TermDocEnum
- Ferret::Index::TermEnum
- Ferret::Index::TermVector
- Ferret::Index::TermVector::TVOffsets
- Ferret::Index::TermVector::TVTerm
- Ferret::ParseError
- Ferret::QueryParser
- Ferret::QueryParser::QueryParseException
- Ferret::Search
- Ferret::Search::BooleanQuery
- Ferret::Search::BooleanQuery::BooleanClause
- Ferret::Search::ConstantScoreQuery
- Ferret::Search::Explanation
- Ferret::Search::Filter
- Ferret::Search::FilteredQuery
- Ferret::Search::FuzzyQuery
- Ferret::Search::Hit
- Ferret::Search::MatchAllQuery
- Ferret::Search::MultiSearcher
- Ferret::Search::MultiTermQuery
- Ferret::Search::PhraseQuery
- Ferret::Search::PrefixQuery
- Ferret::Search::Query
- Ferret::Search::QueryFilter
- Ferret::Search::RangeFilter
- Ferret::Search::RangeQuery
- Ferret::Search::Searcher
- Ferret::Search::Sort
- Ferret::Search::SortField
- Ferret::Search::Spans
- Ferret::Search::Spans::SpanFirstQuery
- Ferret::Search::Spans::SpanMultiTermQuery
- Ferret::Search::Spans::SpanNearQuery
- Ferret::Search::Spans::SpanNotQuery
- Ferret::Search::Spans::SpanOrQuery
- Ferret::Search::Spans::SpanPrefixQuery
- Ferret::Search::Spans::SpanTermQuery
- Ferret::Search::TermQuery
- Ferret::Search::TopDocs
- Ferret::Search::TypedRangeFilter
- Ferret::Search::TypedRangeQuery
- Ferret::Search::WildcardQuery
- Ferret::StateError
- Ferret::Store
- Ferret::Store::Directory
- Ferret::Store::FSDirectory
- Ferret::Store::Lock
- Ferret::Store::Lock::LockError
- Ferret::Store::RAMDirectory
- Ferret::Utils
- Ferret::Utils::BitVector
- Ferret::Utils::MultiMapper
- Ferret::Utils::PriorityQueue
- WEBrick
- WEBrick::FerretBrowserHandler
- Date
- DateTime
- FileUtils
- Float
- Integer
- String
- Symbol
- Time

No matching classes.

# Ferret::Analysis

## Summary

The Analysis module contains all the classes used to analyze and tokenize the data to be indexed. There are three main classes you need to know about when dealing with analysis; Analyzer, TokenStream and Token.

## Classes

### Analyzer

Analyzers handle all of your tokenizing needs. You pass an Analyzer to the indexing class when you create it and it will create the TokenStreams necessary to tokenize the fields in the documents. Most of the time you won't need to worry about TokenStreams and Tokens, one of the Analyzers distributed with Ferret will do exactly what you need. Otherwise you'll need to implement a custom analyzer.

### TokenStream

A TokenStream is an enumeration of Tokens. There are two standard types of TokenStream; Tokenizer and TokenFilter. A Tokenizer takes a String and turns it into a list of Tokens. A TokenFilter takes another TokenStream and post-processes the Tokens. You can chain as many TokenFilters together as you like but they always need to finish with a Tokenizer.

### Token

A Token is a single term from a document field. A token contains the text representing the term as well as the start and end offset of the token. The start and end offset will represent the token as it appears in the source field. Some TokenFilters may change the text in the Token but the start and end offsets should stay the same so (end - start) won't necessarily be equal to the length of text in the token. For example using a stemming TokenFilter the term "Beginning" might have start and end offsets of 10 and 19 respectively ("Beginning".length == 9) but Token#text might be "begin" (after stemming).

### Constants

ENGLISH_STOP_WORDS
EXTENDED_ENGLISH_STOP_WORDS
FULL_DANISH_STOP_WORDS
FULL_DUTCH_STOP_WORDS
FULL_ENGLISH_STOP_WORDS
FULL_FINNISH_STOP_WORDS
FULL_FRENCH_STOP_WORDS
FULL_GERMAN_STOP_WORDS
FULL_HUNGARIAN_STOP_WORDS
FULL_ITALIAN_STOP_WORDS
FULL_NORWEGIAN_STOP_WORDS
FULL_PORTUGUESE_STOP_WORDS
FULL_RUSSIAN_STOP_WORDS
FULL_SPANISH_STOP_WORDS
FULL_SWEDISH_STOP_WORDS

[Validate]

Generated with the Darkfish Rdoc Generator 2.