itsjube

Untitled

Nov 7th, 2016
216
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.62 KB | None | 0 0
  1. ---- Stop List ----
  2. the
  3. of
  4. and
  5. in
  6. to
  7. a
  8. is
  9. for
  10. as
  11. by
  12. that
  13. are
  14. with
  15. on
  16. from
  17. be
  18. or
  19. was
  20. it
  21. an
  22. at
  23. which
  24. this
  25. has
  26. can
  27. have
  28. also
  29. used
  30. such
  31. more
  32. its
  33. not
  34. than
  35. may
  36. into
  37. but
  38. most
  39. some
  40. they
  41. there
  42. will
  43.  
  44.  
  45. ---- Note ----
  46. This is just a small sample to illustrate a point. A full stop list would be much larger.
  47.  
  48. ---- Choosing cut off values ----
  49. We choose the cut off values for our stop list based on frequently used terms that had little or nothing to the context of the query.
  50. Further, removing them would cause little to no meaning to be lost.
  51. In the explanation section is an example.
  52.  
  53. ---- Explanation ----
  54. Looking at the Unigram data, I see a ton of characters that we could have ignored.
  55. We could easily ignore all hashtags and # signs as they do not add anything to the results.
  56. I think we could also safely strip "$" from numbers and mathematical notations like: kw·h
  57. This would grately reduce the size of the unigram list, consider the following example:
  58.  
  59. 500 -- 218
  60. 500% -- 3
  61. 500+ -- 1
  62.  
  63. Had we stripped mathematical notations, these 3 records would just be one, grately reducing the length.
  64.  
  65. Now, when dealing with common words, we could easily remove several of them:
  66.  
  67. "the", "of" are examples of the top 3 words, we could easily remove them without losing the context. For example:
  68.  
  69. The Spanish Coast -> Spanish Coast
  70. The Coast of Spain -> Coast Spain
  71. The meaning is retained, yet the index size is grately reduced.
  72. Of course, anything we do to reduce the size of the index, we must also do to the query.
  73. If we remove these terms from the index, we must also remove them from the query.
Advertisement
Add Comment
Please, Sign In to add comment