SHARE
TWEET

Untitled

a guest Sep 20th, 2019 77 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. class RobertaTokenizeProcessor(TokenizeProcessor):
  2.     def __init__(self, tokenizer):
  3.          super().__init__(tokenizer=tokenizer, include_bos=False, include_eos=False)
  4.          
  5. class RobertaNumericalizeProcessor(NumericalizeProcessor):
  6.     def __init__(self, *args, **kwargs):
  7.         super().__init__(*args, vocab=fastai_roberta_vocab, **kwargs)
  8.        
  9. def get_roberta_processor(tokenizer:Tokenizer=None, vocab:Vocab=None):
  10.     return [RobertaTokenizeProcessor(tokenizer=tokenizer), NumericalizeProcessor(vocab=vocab)]
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top