[[TOC(inline)]]

== Introduction to Bi-gram in Xapian ==

 === Bigrams ===
   Bi-gram indexing will be implemented in Xapian for use in Bigram Language Model Implementation ,collocations.All the indexed data while indexing in stored in Btree table.So to easy access bigram will be stored as a bi-gram term.

For a Document with content:

{{{
Read a book about the history of read.
}}}

Bi-gram will be (with stop words removed) :
given are bigram with bigramid
{{{
  Bi-gram      bgid(bigramid)
"read book"  - 881
"book about" - 882
"about history" - 883
"history read" - 884
}}}

=== Term List Data Storage ===
Term list table for the document (Lets say did = 88 and make key for terms will add uni and for bi-gram will add bi):

''Although these data is stored with several optimization like reusing previous term etc.i only mean to depict a overview of how (not exact) data is stored.''

{{{

88uni 5 4 read 2 book 1 about 1history 1
88bi 4 4 read^book 1 book^about 1 about^history 1 history^read 1
}}}

=== Posting List Data Storage ===
Posting List Table for the Document(document ids are assumed and key will add uni for unigram terms bi for bigram terms and coll for collocation additions:

{{{

BRASSPostTable Entry

term          termfreq  collfreq firstdid islast firstdocid lastdocid  docid freq  docid freq  docid  freq
readuni        3          8         98      1       98            88    98     8     78    9      88   1

BRASSBigramPostListTable Entry

bigram        bigramfreq collfreq firstdid islast firstdocid lastdocid  docid freq  docid freq
read^bookbi    2          2         67      1       67            74    67     1     74    1

BigramCollocationPostListTable Entry

             uniqbigramfreq               totalbigramfreq        firstbigramid islast firstbigramid  lastbigramid bigramid  freq    bigramid   freq
readcoll       2(with first gram read)    4(with first gram read)  871           1      871           981            871     2     981        2

}}}

I am Suggesting a postinglist type entry of unigram and bigramids so that it is possible to use this bigrams for collocation. Otherwise i wasnot able to foresee the use of indexed bi-grams for collocation.

== TermGenerator Class(Indexing) ==

Currently index_text function of termgenerator_internal.cc index the terms in the text.But since not we want to also receive Bigrams and Unigram Both or either.

=== In-place Tokenization ===

One option could be to store the previous tokenization and merge the previous and current term here in the index_text function.

=== Tokenization Class[Prefered One] ===

Two classes UnigramTokenization and BigramTokenization to be implemented .
UnigramTokenization.next() will return the current term and BigramTokenization.next() will return current bigram.

'''This will make framework better for if we want to increase grams for indexing or make changes to implementation of Bigram creation i.e removing restriction of keeping bigrams just as consecutive terms etc.Moreover code would be more redable and easy to understand.'''

== DocumentBigramTerm ==

Since the final storage of bi-gram is similar to normal terms of Xapian. I am planning to DocumentBigramTerm similar to OmDocumentTerm class.

I am planning DocumentBigramTerm with following:

string bigram
int wdf
string term1;
string term2;

functions:

inc_wdf(termcount)

get_wdf()

get_description()

'''
Doubts:'''
 * '''Since these are similar to OmDocumentTerm should it be inherited by OmDocument term or not?'''
 * '''  should we store bigrams in database with term1 term2 or term1-term2 (since we do a lot of optimization there and reuse previous term just wondering would having space between term will make diffrence due to optimization)? '''

== Document class changes ==

This class will have two more storage maps to store bigrams and collocation terms:

map<String bigramterm,BigramDocumentTerm>

map<String unigramterm,map<bigramid bgid,bigramterm>>

Methods to support the reading of these storage maps like bigramlist_begin() and to fill them back from database on open_document
change open_term_list() implementation to support the reading of bigram from database.

using open_term_list(bool isbigram) in document class

'''Doubts:'''

 * '''I want to generate bigramsid for all the bigrams mainly to store bigram in back-end for easy access for collocation.How can we generate such uniqueid ... docid is self generated how is it generated ? '''
 * '''What is use and in what all cases document is open from Database using open_document function in brass_backend?'''
 * '''Generation of bigramid is fine but i will also need to access bigram corresponding to bigramid.Where can i easily store and acess those (bigramid,bigram) pair in backend '''
 * ''' Do you think it will be a good idea implement and integrate collocation in this way.if you can think of better way to access collocation please suggest.(This is really a low priority task now i think it more sane to first implement BigramLMWeight and make that functional than including all functionality of bigrams) '''

== Database changes ==

=== changes in function function add_document_() ===

Add document function transfer the term list to termlist table and stores posting list changes (i.e did need to be add to the term in document for postlist).

Now in addition it will transfer the bigrams to the inverter class to store postlist changes for these bigrams and bigram list to termlist table to store bigrams.

This function will also add the collocation data to the postlist changes.
and finally calls flush function on both all the posting list doclength, unigram post list , bigram post list,collocation postlist

=== changes in open_term_list() ,open_post_list() ===

Now open_term_list function will support the bigram and will open BrassTermList or BrassBigramList based on its unigram or bigram

open_term_list(did,isbigram)

Now open_post_list function will support the bigram and will open BrassPostList or BrassBigramPostList based on its unigram or bigram

open_post_list(term,isbigram)

== Bigram Termlist ==

Bi-gram Termlist will be added similarly to the terms with a diffrent key.

'''Doubts:'''

 * ''' I have noticed that term list is used to open the document i.e we read term list and add terms to the document.Is there any other use of maintaing term list ?'''

== Brass BiGram List Changes ==

Opens bigram list in termlist tablenin a way similar to BRASSTermList to support reading.

== Brass Bigram PostList Changes ==

Opens the posting list of Bigram stored in  postlist table and provide access to Bigrams post-list .  open_post_list(bigram,1(isbigram)) will return object of this class.


== Brass Collocation PostList Changes ==

This class will not be implemented as it is reuired for collocation use and BigramLMWeight currently make no use of collocation.

== Query Parser Changes ==

Need to intergrate BiGram Term creation in the query with a flag set. Need to make changes to calling infrastructure .QueryParser and query internal object makes call to the open_post_list function to get postlist of term.Need to tweak it to call BrassBigramPostList in case term is bigram.

Rest every thing will be similar as BrassBigramPostList will be similar to BrassPostList .