Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- from pytorch_transformers import BertTokenizer
- tokenizer = BertTokenizer.from_pretrained(pretrained_model_name_or_path='bert-base-uncased')
- input_text = "this is a piece of feedback"
- tokenized = tokenizer.tokenize(input_text)
- tokenizer.convert_tokens_to_ids(tokenized_text)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement