Liber Primus 5-grams Found in Google Data
6/5/2017

INTRODUCTION:
Lists of tagged 5-grams from google data that have word lengths that match the 5-grams
in the Liber Primus that can be used as the raw data for crib-finding.
http://cicada3301.boards.net/thread/48/words-liber-primus-ii-grams

DATASET:
http://storage.googleapis.com/books/ngrams/books/datasetsv2.html
files: googlebooks-eng-all-5gram-20120701-XX_.gz

CUTS:
Selected 5-grams that Match LP 5-gram word lengths, with and without punctutation
For 5-grams with quotation marks, have added versions with additonal commas
e.g. ...,"..." even though LP has ..."..."
for hints and tips on how the cutting was done see "example_python_n-gram_parser.py"
, ; : assumed punctuation for "3-dots"
. ? ! assumed punctuation for "4-dots"

FILENAME:
all words are     in master list "XX_master.txt"
all words are NOT in master list "XX_non_master.txt"
XX is the first two letters of the bigram
master_word_list is available at https://pastebin.com/2tpcEwTs

FORMAT:
Space delimited data
word1 word2 word3 word4 word5 POS1 POS2 POS3 POS4 POS5 LENGTH1 LENGTH2 LENGTH3 LENGTH4 LENGTH5 MASTER_TAG COUNTS
wordX is word or punctuation Letter cases (upper/lower) has been preserved, Capitalsed first letters
(probably) indicate start of sentence. Punctuation . ! ? (probably) indicates end of sentence
POS = Part Of Speech, (noun, verb, etc.)
LENGTHX is the length of the word *in runes*
MASTER_TAG are all the words in our master_list
COUNT number of entries for phrase in google data

TAG CODE:
_NOUN = N
_VERB = V
_ADJ  = J
_ADV  = D
_PRON = P
_DET  = E
_ADP  = A
_NUM  = U
_CONJ = C
_PRT  = R
_X    = X
_.    = . (punctuation)
NO_POS_TAG = _
ALL_IN_MASTER_LIST = M
ALL_NOT_IN_MASTER_LIST = S

EXAMPLE:
back to the asylum with D R E N A 4 2 2 6 3 S 46
resolves to:
back_ADV to_PRT the_DET asylum_NOUN with_ADP
word-length-in-runes 4 2 2 6 3
NOT all words in master list
46 counts

SAMPLE:
bid her unsay all again _ _ _ _ _ 3 3 5 3 5 S 73
bidding him use it for _ _ _ _ _ 5 3 3 2 3 S 95
bier ye can not fashion _ _ _ _ _ 4 2 3 3 6 S 47
big and bright and crazy _ _ _ _ _ 3 3 6 3 5 S 66
big basket that had hints _ _ _ _ _ 3 6 3 3 5 S 46
big dining room table with J N N N A 3 4 4 5 3 S 48
big steamer sink beneath the J N N A E 3 6 4 5 2 S 74
bigger man might find ample J N V V J 6 3 5 4 5 S 129
biggest and jovialest man of _ _ _ _ _ 7 3 8 3 2 S 49
...
parents when they need to N D P V R 7 4 3 4 2 M 46
part almost as well as _ _ _ _ _ 4 6 2 4 2 M 55
part both mad and wicked _ _ _ _ _ 4 3 3 3 6 M 47
part had been played out _ _ _ _ _ 4 3 4 6 3 M 59
part not to take the _ _ _ _ _ 4 3 2 4 2 M 40
part of all the operations _ _ _ _ _ 4 2 3 2 9 M 62
part of all the participants _ _ _ _ _ 4 2 3 2 12 M 434
part of an agency permit _ _ _ _ _ 4 2 2 6 6 M 174
part of an ongoing and _ _ _ _ _ 4 2 2 4 3 M 599
part of her body has _ _ _ _ _ 4 2 3 4 3 M 139
part of her had believed _ _ _ _ _ 4 2 3 3 8 M 44


MOAR:
2,3,4 grams next ... ? :)

HELP:
http://webchat.freenode.net/?channels=cicadasolvers