Advertisement
Guest User

Untitled

a guest
Aug 20th, 2019
106
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.27 KB | None | 0 0
  1. from pytorch_transformers import BertTokenizer
  2.  
  3. tokenizer = BertTokenizer.from_pretrained(pretrained_model_name_or_path='bert-base-uncased')
  4. input_text = "this is a piece of feedback"
  5. tokenized = tokenizer.tokenize(input_text)
  6. tokenizer.convert_tokens_to_ids(tokenized_text)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement