Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- ---- Stop List ----
- the
- of
- and
- in
- to
- a
- is
- for
- as
- by
- that
- are
- with
- on
- from
- be
- or
- was
- it
- an
- at
- which
- this
- has
- can
- have
- also
- used
- such
- more
- its
- not
- than
- may
- into
- but
- most
- some
- they
- there
- will
- ---- Note ----
- This is just a small sample to illustrate a point. A full stop list would be much larger.
- ---- Choosing cut off values ----
- We choose the cut off values for our stop list based on frequently used terms that had little or nothing to the context of the query.
- Further, removing them would cause little to no meaning to be lost.
- In the explanation section is an example.
- ---- Explanation ----
- Looking at the Unigram data, I see a ton of characters that we could have ignored.
- We could easily ignore all hashtags and # signs as they do not add anything to the results.
- I think we could also safely strip "$" from numbers and mathematical notations like: kw·h
- This would grately reduce the size of the unigram list, consider the following example:
- 500 -- 218
- 500% -- 3
- 500+ -- 1
- Had we stripped mathematical notations, these 3 records would just be one, grately reducing the length.
- Now, when dealing with common words, we could easily remove several of them:
- "the", "of" are examples of the top 3 words, we could easily remove them without losing the context. For example:
- The Spanish Coast -> Spanish Coast
- The Coast of Spain -> Coast Spain
- The meaning is retained, yet the index size is grately reduced.
- Of course, anything we do to reduce the size of the index, we must also do to the query.
- If we remove these terms from the index, we must also remove them from the query.
Advertisement
Add Comment
Please, Sign In to add comment