itsjube

Untitled

Nov 20th, 2016
143
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 2.40 KB | None | 0 0
  1. At the start, the following HashMaps are created:
  2.  
  3. 1. HashMap<String, ArrayList<Integer>> indexDocID = new HashMap<String, ArrayList<Integer>>();
  4. 2. HashMap<String, ArrayList<Integer>> indexTF = new HashMap<String, ArrayList<Integer>>();
  5. 3. HashMap<Integer, Integer> indexTC = new HashMap<Integer, Integer>();
  6. 4. HashMap<Integer, String> indexRealDocumentName = new HashMap<Integer, String>();
  7. 5. HashMap<String, HashMap<Integer, Integer>> indexWithDocIDandTF = new HashMap<String, HashMap<Integer, Integer>>();
  8.  
  9.  
  10. IndexDoCID stores the term, and the documents that term is stored in.
  11. IndexTF stores the term, and the term frequency as they correspond to each document. (e.g. position 0 in IndexDoCID and IndexTF's ArrayList match).
  12. indexTC stores the docid, and the total token count in that document.
  13. indexRealDocumentName stores the docid, and the name of that document.
  14. indexWithDocIDandTF stores the term, and then in an internal HashMap, the docid and its term frequency.
  15.  
  16. I then use indexWithDocIDandTF to generate a HashMap with the following structure: HashMap<Integer, HashMap<String, Double>> called TermTF.
  17. The integer holds the docid, the internal HashMap's String value is the term, and its Double is its TF. hese values are obtained from TermTF.
  18.  
  19.  
  20. I then calculate the TermIDF by obtaining the total documents in the corpus, and dividing it by the amount of documents that have a term.
  21.  
  22.  
  23. TF-IDF is calculating with the following HashMapHashMap<Integer, HashMap<String, Double>> called TermTFIDF.
  24.  
  25. Terms are normalized into the Double by dividng the termfrequency by the total amount of terms in that document.
  26. Then, that value is multiplied by thge IDF of that term to get the TF-IDF.
  27.  
  28. The TF-IDF vector for the query is calculated by: 1 / MaxFrequency.
  29. Where MaxFrequency is the frequency a term appears in a document. (e.g. a query like: "new new happy", would have MaxFrequency of 2.)
  30.  
  31. Then the true length of the documents and the query is calculated by obtaining the TFIDF for a term in a document, then multiplying it by itself, then adding it to the existing value for the document length.
  32.  
  33. Query length is calculated in the same way.
  34.  
  35. Lastly, cosine similarity is calculated by multiplying each TF-IDF by itself, then adding it to the next value.
  36. This number is then divided by the length of the document and the query.
  37.  
  38. This product is the cosine similarity of the document to the query.
Advertisement
Add Comment
Please, Sign In to add comment