Advertisement
Guest User

Untitled

a guest
Jun 22nd, 2017
58
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Perl 0.49 KB | None | 0 0
  1. use utf8;
  2. use strict;
  3. use warnings;
  4. use open qw<:encoding(utf8) :std>;
  5. use Data::Dumper;
  6. use Data::Section -setup;
  7. use Hailo::Tokenizer::Words;
  8.  
  9. my $self = bless {} => __PACKAGE__;
  10. my $text = ${ $self->section_data("UTF-8 encoded sample plain-text file") };
  11. print $text;
  12.  
  13. my $toke = Hailo::Tokenizer::Words->new();
  14. my $parsed = $toke->make_tokens($text);
  15. print Dumper($parsed);
  16.  
  17. __DATA__
  18. __[ UTF-8 encoded sample plain-text file ]__
  19.   STARGΛ̊TE SG-1, a ==, a⃑ ⊥ b⃑
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement