Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- from __future__ import print_function
- import re
- import nltk
- from nltk import word_tokenize
- from nltk.util import ngrams
- from collections import Counter
- from nltk.tokenize import RegexpTokenizer
- sample_text = """
- 1609
- THE SONNETS
- by William Shakespeare
- 1
- From fairest creatures we desire increase,
- That thereby beauty's rose might never die,
- But as the riper should by time decease,
- His tender heir might bear his memory:
- But thou contracted to thine own bright eyes,
- Feed'st thy light's flame with self-substantial fuel,
- Making a famine where abundance lies,
- Thy self thy foe, to thy sweet self too cruel:
- Thou that art now the world's fresh ornament,
- And only herald to the gaudy spring,
- Within thine own bud buriest thy content,
- And tender churl mak'st waste in niggarding:
- Pity the world, or else this glutton be,
- To eat the world's due, by the grave and thee.
- 2
- When forty winters shall besiege thy brow,
- And dig deep trenches in thy beauty's field,
- Thy youth's proud livery so gazed on now,
- Will be a tattered weed of small worth held:
- Then being asked, where all thy beauty lies,
- Where all the treasure of thy lusty days;
- To say within thine own deep sunken eyes,
- Were an all-eating shame, and thriftless praise.
- How much more praise deserved thy beauty's use,
- If thou couldst answer 'This fair child of mine
- Shall sum my count, and make my old excuse'
- Proving his beauty by succession thine.
- This were to be new made when thou art old,
- And see thy blood warm when thou feel'st it cold.
- 3
- Look in thy glass and tell the face thou viewest,
- Now is the time that face should form another,
- Whose fresh repair if now thou not renewest,
- Thou dost beguile the world, unbless some mother.
- For where is she so fair whose uneared womb
- Disdains the tillage of thy husbandry?
- Or who is he so fond will be the tomb,
- Of his self-love to stop posterity?
- Thou art thy mother's glass and she in thee
- Calls back the lovely April of her prime,
- So thou through windows of thine age shalt see,
- Despite of wrinkles this thy golden time.
- But if thou live remembered not to be,
- Die single and thine image dies with thee.
- 4
- Unthrifty loveliness why dost thou spend,
- Upon thy self thy beauty's legacy?
- Nature's bequest gives nothing but doth lend,
- And being frank she lends to those are free:
- Then beauteous niggard why dost thou abuse,
- The bounteous largess given thee to give?
- Profitless usurer why dost thou use
- So great a sum of sums yet canst not live?
- For having traffic with thy self alone,
- Thou of thy self thy sweet self dost deceive,
- Then how when nature calls thee to be gone,
- What acceptable audit canst thou leave?
- Thy unused beauty must be tombed with thee,
- Which used lives th' executor to be.
- 5
- Those hours that with gentle work did frame
- The lovely gaze where every eye doth dwell
- Will play the tyrants to the very same,
- And that unfair which fairly doth excel:
- For never-resting time leads summer on
- To hideous winter and confounds him there,
- Sap checked with frost and lusty leaves quite gone,
- Beauty o'er-snowed and bareness every where:
- Then were not summer's distillation left
- A liquid prisoner pent in walls of glass,
- Beauty's effect with beauty were bereft,
- Nor it nor no remembrance what it was.
- But flowers distilled though they with winter meet,
- Leese but their show, their substance still lives sweet.
- """
- if __name__ == '__main__':
- line = raw_input()
- word = nltk.word_tokenize(line)
- length = len(nltk.word_tokenize(line))
- print(word)
- if length == 2:
- #unigrams
- tokenizer = RegexpTokenizer(r'\w{3,}')
- unigrams = tokenizer.tokenize(sample_text.lower())
- fdistU = nltk.FreqDist(unigrams)
- for k,v in fdistU.items():
- print(k, v)
- elif length == 3:
- #bigrams
- tokenizer = RegexpTokenizer(r'\w+')
- unigrams = tokenizer.tokenize(sample_text.lower())
- bigrams = nltk.bigrams(unigrams)
- fdistB = nltk.FreqDist(bigrams)
- for k,v in fdistB.items():
- print(k, v)
- else:
- print("invalid input")
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement