org.terrier.structures.integer

Class IntegerCodingPostingOutputStream

java.lang.Object

org.terrier.structures.AbstractPostingOutputStream

org.terrier.structures.integer.IntegerCodingPostingOutputStream

public class IntegerCodingPostingOutputStream
extends AbstractPostingOutputStream

This class is used to store a posting list to the disk, compressing it using some integer coding technique. It can be used to save IterablePosting with (optionally) fields and/or blocks. Note: this class writes the posting list in chunks, whose size is specified by chunkSize.

Since:

4.0

Author:

Matteo Catena

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	bfs
protected int[]	blocks
protected IntegerCodec	blocksCodec
protected int	chunkSize Number of posting in a chunk
protected int[][]	fields
protected IntegerCodec	fieldsCodec
protected int	fieldsCount
protected int	hasBlocks
protected int[]	ids
protected IntegerCodec	idsCodec
protected int	lastDocid
protected static org.slf4j.Logger	logger The logger used
protected int	maxBlocks
protected ByteOut	output
protected int[]	tfs
protected IntegerCodec	tfsCodec

Constructor Summary

Constructors

Constructor and Description
IntegerCodingPostingOutputStream(ByteOut output, int chunkSize, int fieldsCount, int hasBlocks, int maxBlocks, IntegerCodec idsCodec, IntegerCodec tfsCodec, IntegerCodec fieldsCodec, IntegerCodec blocksCodec)
IntegerCodingPostingOutputStream(String filename, int chunkSize, int fieldsCount, int hasBlocks, int maxBlocks, IntegerCodec idsCodec, IntegerCodec tfsCodec, IntegerCodec fieldsCodec, IntegerCodec blocksCodec)

Method Summary

Modifier and Type	Method and Description
void	close() close this object.
int	getLastDocidWritten()
BitFilePosition	getOffset() What is current offset?
Class<? extends IterablePosting>	getPostingIteratorClass() Returns the IterablePosting class to use for reading structure written by this class
protected void	init(int fieldsCount, int hasBlocks, int maxBlocks, int chunkSize, IntegerCodec idsCodec, IntegerCodec tfsCodec, IntegerCodec fieldsCodec, IntegerCodec blocksCodec)
protected void	write(int i, int cnt) compress and write down the current chunk of postings
BitIndexPointer	writePostings(int[][] postings, int startOffset, int Length, int firstId)
BitIndexPointer	writePostings(IterablePosting postings) Write out the specified postings.
BitIndexPointer	writePostings(IterablePosting postings, int previousId)
BitIndexPointer	writePostings(Iterator<Posting> iterator)
BitIndexPointer	writePostings(Iterator<Posting> iterator, int previousId)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

The logger used

output

protected ByteOut output

lastDocid

protected int lastDocid

chunkSize

protected int chunkSize

Number of posting in a chunk

fieldsCount

protected int fieldsCount

hasBlocks

protected int hasBlocks

maxBlocks

protected int maxBlocks

ids

protected int[] ids

tfs

protected int[] tfs

fields

protected int[][] fields

bfs

protected int[] bfs

blocks

protected int[] blocks

idsCodec

protected IntegerCodec idsCodec

tfsCodec

protected IntegerCodec tfsCodec

fieldsCodec

protected IntegerCodec fieldsCodec

blocksCodec

protected IntegerCodec blocksCodec

Constructor Detail

IntegerCodingPostingOutputStream

public IntegerCodingPostingOutputStream(String filename,
                                        int chunkSize,
                                        int fieldsCount,
                                        int hasBlocks,
                                        int maxBlocks,
                                        IntegerCodec idsCodec,
                                        IntegerCodec tfsCodec,
                                        IntegerCodec fieldsCodec,
                                        IntegerCodec blocksCodec)
                                 throws IOException

Parameters:

filename - the file where to write

chunkSize - the chunk size (in term of number of posting)

fieldsCount - the number of fields, if any. 0 otherwise

hasBlocks - 0:no blocks, 1:positions, 2:blocks of any size!=1

maxBlocks - 0:no limit, >0 limited.

idsCodec - the IntegerCodec to use to compress ids

tfsCodec - the IntegerCodec to use to compress tfs

fieldsCodec - the IntegerCodec to use to compress fields (null if there are no fields)

blocksCodec - the IntegerCodec to use to compress blocks (null if there are no blocks)

Throws:

IOException

IntegerCodingPostingOutputStream

public IntegerCodingPostingOutputStream(ByteOut output,
                                        int chunkSize,
                                        int fieldsCount,
                                        int hasBlocks,
                                        int maxBlocks,
                                        IntegerCodec idsCodec,
                                        IntegerCodec tfsCodec,
                                        IntegerCodec fieldsCodec,
                                        IntegerCodec blocksCodec)
                                 throws IOException

Parameters:

output - the output channel

chunkSize - the chunk size (in term of number of posting)

fieldsCount - the number of fields, if any. 0 otherwise

hasBlocks - 0:no blocks, 1:positions, 2:blocks of any size!=1

maxBlocks - 0:no limit, >0 limited.

idsCodec - the IntegerCodec to use to compress ids

tfsCodec - the IntegerCodec to use to compress tfs

fieldsCodec - the IntegerCodec to use to compress fields (null if there are no fields)

blocksCodec - the IntegerCodec to use to compress blocks (null if there are no blocks)

Throws:

IOException

Method Detail

init

protected void init(int fieldsCount,
                    int hasBlocks,
                    int maxBlocks,
                    int chunkSize,
                    IntegerCodec idsCodec,
                    IntegerCodec tfsCodec,
                    IntegerCodec fieldsCodec,
                    IntegerCodec blocksCodec)

getPostingIteratorClass

public Class<? extends IterablePosting> getPostingIteratorClass()

Returns the IterablePosting class to use for reading structure written by this class

Specified by:

getPostingIteratorClass in class AbstractPostingOutputStream

writePostings

public BitIndexPointer writePostings(IterablePosting postings)
                              throws IOException

Write out the specified postings.

Specified by:

writePostings in class AbstractPostingOutputStream

Parameters:

postings - IterablePosting postings accessed through an IterablePosting object

Throws:

IOException

write

protected void write(int i,
                     int cnt)
              throws IOException

compress and write down the current chunk of postings

Parameters:

i - number of postings

cnt - number of blocks

Throws:

IOException

close

public void close()

close this object. suppresses any exception

Specified by:

close in class AbstractPostingOutputStream

getOffset

public BitFilePosition getOffset()

What is current offset?

Specified by:

getOffset in class AbstractPostingOutputStream

writePostings

public BitIndexPointer writePostings(int[][] postings,
                                     int startOffset,
                                     int Length,
                                     int firstId)
                              throws IOException

Specified by:

writePostings in class AbstractPostingOutputStream

Throws:

IOException

writePostings

public BitIndexPointer writePostings(IterablePosting postings,
                                     int previousId)
                              throws IOException

Specified by:

writePostings in class AbstractPostingOutputStream

Throws:

IOException

writePostings

public BitIndexPointer writePostings(Iterator<Posting> iterator,
                                     int previousId)
                              throws IOException

Specified by:

writePostings in class AbstractPostingOutputStream

Throws:

IOException

writePostings

public BitIndexPointer writePostings(Iterator<Posting> iterator)
                              throws IOException

Specified by:

writePostings in class AbstractPostingOutputStream

Throws:

IOException

getLastDocidWritten

public int getLastDocidWritten()

Specified by:

getLastDocidWritten in class AbstractPostingOutputStream