> use (load "mew-meow/tokenizer.mew"); def tokenize (make-tokenizer)
^[[A^[[A
> display (tokenize "def f { x ~ ys : print \"Testing!\"; [ * x (+ 1 2 3) ] }")
Name : [def] @ line 1 column 1
Name : [f] @ line 1 column 5
.open-function @ line 1 column 7
Name : [x] @ line 1 column 9
.tilde @ line 1 column 11
Name : [ys] @ line 1 column 13
.colon @ line 1 column 16
Name : [print] @ line 1 column 18
String : Testing! @ line 1 column 25
.semi-colon @ line 1 column 25
.open-list @ line 1 column 27
Name : [*] @ line 1 column 29
Name : [x] @ line 1 column 31
.open-call @ line 1 column 33
Name : [+] @ line 1 column 34
Integer : 1 @ line 1 column 36
Integer : 2 @ line 1 column 38
Integer : 3 @ line 1 column 40
.close-call @ line 1 column 41
.close-list @ line 1 column 43
.close-function @ line 1 column 45
>