Advertisement
Guest User

the main part of tokenize

a guest
Nov 26th, 2014
190
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.53 KB | None | 0 0
  1. tokens = []
  2. for t in line.split():
  3. if t != ' ':
  4. #print(t)
  5. #print(tokens)
  6. if t in all_brackets:
  7. #print("holla " +t)
  8. tokens.append(t)
  9. #print(tokens)
  10.  
  11. if coerce_to_number(t) !=None:
  12. tokens.append(coerce_to_number(t))
  13. # print(brackets[('(',')')])
  14. if coerce_to_number(t) ==None and t not in all_brackets:
  15. raise ValueError("invalid token " + t)
  16. return tokens
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement