Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- At the start, the following HashMaps are created:
- 1. HashMap<String, ArrayList<Integer>> indexDocID = new HashMap<String, ArrayList<Integer>>();
- 2. HashMap<String, ArrayList<Integer>> indexTF = new HashMap<String, ArrayList<Integer>>();
- 3. HashMap<Integer, Integer> indexTC = new HashMap<Integer, Integer>();
- 4. HashMap<Integer, String> indexRealDocumentName = new HashMap<Integer, String>();
- 5. HashMap<String, HashMap<Integer, Integer>> indexWithDocIDandTF = new HashMap<String, HashMap<Integer, Integer>>();
- IndexDoCID stores the term, and the documents that term is stored in.
- IndexTF stores the term, and the term frequency as they correspond to each document. (e.g. position 0 in IndexDoCID and IndexTF's ArrayList match).
- indexTC stores the docid, and the total token count in that document.
- indexRealDocumentName stores the docid, and the name of that document.
- indexWithDocIDandTF stores the term, and then in an internal HashMap, the docid and its term frequency.
- I then use indexWithDocIDandTF to generate a HashMap with the following structure: HashMap<Integer, HashMap<String, Double>> called TermTF.
- The integer holds the docid, the internal HashMap's String value is the term, and its Double is its TF. hese values are obtained from TermTF.
- I then calculate the TermIDF by obtaining the total documents in the corpus, and dividing it by the amount of documents that have a term.
- TF-IDF is calculating with the following HashMapHashMap<Integer, HashMap<String, Double>> called TermTFIDF.
- Terms are normalized into the Double by dividng the termfrequency by the total amount of terms in that document.
- Then, that value is multiplied by thge IDF of that term to get the TF-IDF.
- The TF-IDF vector for the query is calculated by: 1 / MaxFrequency.
- Where MaxFrequency is the frequency a term appears in a document. (e.g. a query like: "new new happy", would have MaxFrequency of 2.)
- Then the true length of the documents and the query is calculated by obtaining the TFIDF for a term in a document, then multiplying it by itself, then adding it to the existing value for the document length.
- Query length is calculated in the same way.
- Lastly, cosine similarity is calculated by multiplying each TF-IDF by itself, then adding it to the next value.
- This number is then divided by the length of the document and the query.
- This product is the cosine similarity of the document to the query.
Advertisement
Add Comment
Please, Sign In to add comment