BufferedTokenStream (Solr 3.6.2 API)
Keywords: org.apache.solr.analysis.BufferedTokenStream class

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.solr.analysis
Class BufferedTokenStream

    java.lang.Object
      org.apache.lucene.util.AttributeSource
          org.apache.lucene.analysis.TokenStream
              org.apache.lucene.analysis.TokenFilter
                  org.apache.solr.analysis.BufferedTokenStream

All Implemented Interfaces:
  Closeable

Deprecated. This class does not support custom attributes. Extend TokenFilter instead, using `AttributeSource.captureState()` and `AttributeSource.restoreState(State)` which support all attributes.

    @Deprecated
    public abstract class BufferedTokenStream

extends TokenFilter

Handles input and output buffering of TokenStream

     // Example of a class implementing the rule "A" "B" => "Q" "B"
     class MyTokenStream extends BufferedTokenStream {
       public MyTokenStream(TokenStream input) {super(input);}
       protected Token process(Token t) throws IOException {
         if ("A".equals(t.termText())) {
           Token t2 = read();
           if (t2!=null && "B".equals(t2.termText())) t.setTermText("Q");
           if (t2!=null) pushBack(t2);
         }
         return t;
       }
     }

     // Example of a class implementing "A" "B" => "A" "A" "B"
     class MyTokenStream extends BufferedTokenStream {
       public MyTokenStream(TokenStream input) {super(input);}
       protected Token process(Token t) throws IOException {
         if ("A".equals(t.termText()) && "B".equals(peek(1).termText()))
           write((Token)t.clone());
         return t;
       }
     }

NOTE: BufferedTokenStream does not clone() any Tokens. This is instead the responsibility of the implementing subclass. In the "A" "B" => "A" "A" "B" example above, the subclass must clone the additional "A" it creates.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
`AttributeSource.AttributeFactory, AttributeSource.State`

Field Summary

Fields inherited from class org.apache.lucene.analysis.TokenFilter
`input`

Constructor Summary
`BufferedTokenStream(TokenStream input)` Deprecated.

Method Summary
`boolean` | `incrementToken()` Deprecated.
`protected Iterable<Token>` | `output()` Deprecated. Provides direct Iterator access to the buffered output stream.
`protected Token` | `peek(int n)` Deprecated. Peek n tokens ahead in the buffered input stream, without modifying the stream.
`protected abstract Token` | `process(Token t)` Deprecated. Process a token.
`protected void` | `pushBack(Token t)` Deprecated. Push a token back into the buffered input stream, such that it will be returned by a future call to `read()`
`protected Token` | `read()` Deprecated. Read a token from the buffered input stream.
`void` | `reset()` Deprecated.
`protected void` | `write(Token t)` Deprecated. Write a token to the buffered output stream

Methods inherited from class org.apache.lucene.analysis.TokenFilter
`close, end`

Methods inherited from class org.apache.lucene.util.AttributeSource
`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

Constructor Detail

### BufferedTokenStream

    public BufferedTokenStream(TokenStream input)

  Deprecated.

Method Detail

### process

    protected abstract Token process(Token t)
                              throws IOException

  Deprecated.
  Process a token. Subclasses may read more tokens from the input stream, write more tokens to the output stream, or simply return the next token to be output. Subclasses may return null if the token is to be dropped. If a subclass writes tokens to the output stream and returns a non-null Token, the returned Token is considered to be at the head of the token output stream.

  Throws:
    `IOException`

### incrementToken

    public final boolean incrementToken()
                                 throws IOException

  Deprecated.
  Specified by:
    `incrementToken` in class `TokenStream`
  Throws:
    `IOException`

### read

    protected Token read()
                  throws IOException

  Deprecated.
  Read a token from the buffered input stream.

  Returns:
    null at EOS
  Throws:
    `IOException`

### pushBack

    protected void pushBack(Token t)

  Deprecated.
  Push a token back into the buffered input stream, such that it will be returned by a future call to `read()`

### peek

    protected Token peek(int n)
                  throws IOException

  Deprecated.
  Peek n tokens ahead in the buffered input stream, without modifying the stream.

  Parameters:
    `n` - Number of tokens into the input stream to peek, 1 based ... 0 is invalid
  Returns:
    a Token which exists in the input stream, any modifications made to this Token will be "real" if/when the Token is `read()` from the stream.
  Throws:
    `IOException`

### write

    protected void write(Token t)

  Deprecated.
  Write a token to the buffered output stream

### output

    protected Iterable<Token> output()

  Deprecated.
  Provides direct Iterator access to the buffered output stream. Modifying any token in this Iterator will affect the resulting stream.

### reset

    public void reset()
               throws IOException

  Deprecated.
  Overrides:
    `reset` in class `TokenFilter`
  Throws:
    `IOException`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD