Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fetch raw document #937

Merged
merged 2 commits into from
Jan 7, 2020
Merged

Fetch raw document #937

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ad8dc20
Add method to fetch raw document
zeynepakkalyoncu Jan 6, 2020
f93be9b
Handle error cases in fetching raw doc
zeynepakkalyoncu Jan 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions src/main/java/io/anserini/index/IndexReaderUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,18 @@ public static Map<String, Long> getDocumentVector(IndexReader reader, String doc
return docVector;
}

/**
* Returns the raw document given its collection docid.
* @param reader index reader
* @param docid collection docid
* @return the raw document given its collection docid
* @throws IOException if error encountered during document fetching
*/
public static String getRawDocument(IndexReader reader, String docid) throws IOException {
Document rawDoc = reader.document(convertDocidToLuceneDocid(reader, docid));
return rawDoc.get("raw");
}

/**
* Computes the BM25 weight of a term (prior to analysis) in a particular document.
* @param reader index reader
Expand Down
10 changes: 10 additions & 0 deletions src/test/java/io/anserini/index/IndexReaderUtilsTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,16 @@ public void testDocumentVector() throws Exception {
assertEquals(Long.valueOf(1), documentVector.get("test"));
}

@Test
public void testRawDoc() throws Exception {
Directory dir = FSDirectory.open(tempDir1);
IndexReader reader = DirectoryReader.open(dir);

assertEquals("here is some text here is some more text", IndexReaderUtils.getRawDocument(reader, "doc1"));
assertEquals("more texts", IndexReaderUtils.getRawDocument(reader, "doc2"));
assertEquals("here is a test", IndexReaderUtils.getRawDocument(reader, "doc3"));
}

@Test
public void testDocidConversion() throws Exception {
Directory dir = FSDirectory.open(tempDir1);
Expand Down