betoesquivel

tokenizer in perl

Nov 9th, 2015
416
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Perl 0.27 KB | None | 0 0
  1. while (<BR>) {
  2.   # Only punctuation that is not part of a word is a token on its own.
  3.   # This only works on this text, because one flaw it has is decimal numbers
  4.   # and currency.
  5.   s/([,"$%\^\&*()+=\[\]!?])/ $1 /g;
  6.   s/[^A-Z.](\.)\s+/ $1 /g;
  7.   s/\s+/\n/g;
  8.   print $_;
  9. }
Advertisement
Add Comment
Please, Sign In to add comment