Guest User

Untitled

a guest
May 25th, 2018
81
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.21 KB | None | 0 0
  1. import spacy
  2.  
  3. # Use English model
  4. spacy_en = spacy.load('en')
  5.  
  6. # create a tokenizer function
  7. def tokenizer(text):
  8. text = text.replace("<br />", " ")
  9. return [tok.text for tok in spacy_en.tokenizer(text)]
Add Comment
Please, Sign In to add comment