Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- tokenizer = tensorflow_text.WhitespaceTokenizer()
- tokens = tokenizer.tokenize(['everything not saved will be lost.', u'Sad☹'.encode('UTF-8')])
- print(tokens.to_list())
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
