Field tokenStream should be usable with stored fields. [LUCENE-1699]

[asfimport]

Field.tokenStream should be usable for indexing even for stored values. Useful for many types of pre-analyzed values (text/numbers, etc)
http://search.lucidimagination.com/search/document/902bad4eae20bdb8/field_tokenstreamvalue

---

Migrated from LUCENE-1699 by Yonik Seeley (@yonik), 1 vote, resolved Jun 22 2009
Attachments: LUCENE-1699.patch (versions: 2)
Linked issues:

• Add NumericField, make plain text numeric parsers public in FieldCache, move trie parsers to FieldCache [LUCENE-1701] #2775

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

incomplete patch for feedback - needs tests and further work to allow binary field values with token streams (I assume this would be desirable for numeric fields).

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

Another 2.9 huh? You own it then :)

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

Watch out - there's a new sheriff in town!

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

This group could develop hard for the rest of the year no problem by the looks. Someones got to herd these cats. Since we last semi decided it would be a good idea to release 2.9, we have rewritten half the core code base :)

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Patch looks good:

• Can you make sure CHANGES describes this new behavior (Field is
  allowed to have both a tokenStream and a String/Reader/binary
  value)?

• The javadoc for readerValue is wrong (copy/paste from stringValue)

• Can you spell out more clearly in the javadocs that even when a
  tokenStream value is set, one of String/Reader/binary may also be
  set, or, not, and if so, that "other" value is only used for
  stored fields. Eg, explain why one would use setTokenStream
  instead of setValue(TokenStream value).

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

Patch attached.

I've attempted to clean up some of the semantics of Fieldable - it's really a reasonable interface, with just a few redundancies (isBinary/getBinary). I've purposely avoided messing with Field more than necessary (adding yet more constructors, constants, etc).

This change is back compatible as it just removes some unnecessary restrictions. I'll commit in a few days.

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

committed.