Advertisement
nanorocks

document_classification_lab3_ex

May 9th, 2018
342
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 17.34 KB | None | 0 0
  1. # -*- coding: utf-8 -*-
  2.  
  3. """
  4. За секоја прочитан документ од стандарден влез да се испечатат зборовите кои се употребуваат за класификација, класата и
  5. веројатноста на зборот да е од таа класа (заокружено на 4 децимали), како и тежинската веројатност на зборот да припаѓа
  6. на класата (заокружено на 4 децимали). На крај да се испечати предвидената класа на документот и логаритам со основа
  7. 2 од веројатноста со која се предвидува (заокружено на 4 децимали).
  8.  
  9. """
  10.  
  11.  
  12.  
  13. import re
  14. import math
  15.  
  16. train_data = [
  17.     ("""What Are We Searching for on Mars?
  18. Martians terrified me growing up. I remember watching the 1996 movie Mars Attacks! and fearing that the Red Planet harbored hostile alien neighbors. Though I was only 6 at the time, I was convinced life on Mars meant little green men wielding vaporizer guns. There was a time, not so long ago, when such an assumption about Mars wouldn’t have seemed so far-fetched.
  19. Like a child watching a scary movie, people freaked out after listening to “The War of the Worlds,” the now-infamous 1938 radio drama that many listeners believed was a real report about an invading Martian army. Before humans left Earth, humanity’s sense of what—or who—might be in our galactic neighborhood was, by today’s standards, remarkably optimistic.
  20. """,
  21.      "science"),
  22.     ("""Mountains of Ice are Melting, But Don't Panic (Op-Ed)
  23. If the planet lost the entire West Antarctic ice sheet, global sea level would rise 11 feet, threatening nearly 13 million people worldwide and affecting more than $2 trillion worth of property.
  24. Ice loss from West Antarctica has been increasing nearly three times faster in the past decade than during the previous one — and much more quickly than scientists predicted.
  25. This unprecedented ice loss is occurring because warm ocean water is rising from below and melting the base of the glaciers, dumping huge volumes of additional water — the equivalent of a Mt. Everest every two years — into the ocean.
  26. """,
  27.      "science"),
  28.     ("""Some scientists think we'll find signs of aliens within our lifetimes. Here's how.
  29. Finding extraterrestrial life is the essence of science fiction. But it's not so far-fetched to predict that we might find evidence of life on a distant planet within a generation.
  30. "With new telescopes coming online within the next five or ten years, we'll really have a chance to figure out whether we're alone in the universe," says Lisa Kaltenegger, an astronomer and director of Cornell's new Institute for Pale Blue Dots, which will search for habitable planets. "For the first time in human history, we might have the capability to do this."
  31. """,
  32.      "science"),
  33.     ("""'Magic' Mushrooms in Royal Garden: What Is Fly Agaric?
  34. Hallucinogenic mushrooms are perhaps the last thing you'd expect to find growing in the Queen of England's garden.
  35. Yet a type of mushroom called Amanita muscaria — commonly known as fly agaric, or fly amanita — was found growing in the gardens of Buckingham Palace by the producers of a television show, the Associated Press reported on Friday (Dec. 12).
  36. A. muscaria is a bright red-and-white mushroom, and the fungus is psychoactive when consumed.
  37. """,
  38.      "science"),
  39.     ("""Upcoming Parks : 'Lost Corner' Finds New Life in Sandy Springs
  40. At the corner of Brandon Mill Road, where Johnson Ferry Road turns into Dalrymple Road, tucked among 24 forested acres, sits an early 20th Century farmhouse. A vestige of Sandy Springs' past, the old home has found new life as the centerpiece of Lost Forest Preserve. While the preserve isn't slated to officially debut until some time next year, the city has opened the hiking trails to the public until construction begins on the permanent parking lot (at the moment the parking lot is a mulched area). The new park space includes community garden plots, a 4,000-foot-long hiking trail and an ADA-accessible trail through the densely wooded site. For Atlantans seeking an alternate escape to serenity (or those who dig local history), it's certainly worth a visit.
  41. """,
  42.      "science"),
  43.     ("""Stargazers across the world got a treat this weekend when the Geminids meteor shower gave the best holiday displays a run for their money.
  44. The meteor shower is called the "Geminids" because they appear as though they are shooting out of the constellation of Gemini. The meteors are thought to be small pieces of an extinct comment called 3200 Phaeton, a dust cloud revolving around the sun. Phaeton is thought to have lost all of its gas and to be slowly breaking apart into small particles.
  45. Earth runs into a stream of debris from 3200 Phaethon every year in mid-December, causing a shower of meteors, which hit its peak over the weekend.
  46. """,
  47.      "science"),
  48.     ("""Envisioning a River of Air
  49. By the classification rules of the world of physics, we all know that the Earth's atmosphere is made of gas (rather than liquid, solid, or plasma). But in the world of flying it's often useful to think
  50. """,
  51.      "science"),
  52.     ("""Following Sunday's 17-7 loss to the Seattle Seahawks, the San Francisco 49ers were officially eliminated from playoff contention, and they have referee Ed Hochuli to blame. OK, so they have a lot of folks to point the finger at for their 7-7 record, but Hochuli's incorrect call is the latest and easiest scapegoat.
  53. """
  54.      , "sport"),
  55.     ("""Kobe Bryant and his teammates have an odd relationship. That makes sense: Kobe Bryant is an odd guy, and the Los Angeles Lakers are an odd team.
  56. They’re also, for the first time this season, the proud owners of a three-game winning streak. On top of that, you may have heard, Kobe Bryant passed Michael Jordan on Sunday evening to move into third place on the NBA’s all-time scoring list.
  57. """
  58.      , "sport"),
  59.     ("""The Patriots continued their divisional dominance and are close to clinching home-field advantage throughout the AFC playoffs. Meanwhile, both the Colts and Broncos again won their division titles with head-to-head wins.The Bills' upset of the Packers delivered a big blow to Green Bay's shot at clinching home-field advantage throughout the NFC playoffs. Detroit seized on the opportunity and now leads the NFC North.
  60. """
  61.      , "sport"),
  62.     ("""If you thought the Washington Redskins secondary was humbled by another scintillating performance from New Yorks Giants rookie wide receiver sensation Odell Beckham Jr., think again.In what is becoming a weekly occurrence, Beckham led NFL highlight reels on Sunday, collecting 12 catches for 143 yards and three touchdowns in Sunday's 24-13 victory against an NFC East rival.
  63. """
  64.      , "sport")
  65.     , ("""That was two touchdowns and 110 total yards for the three running backs. We break down the fantasy implications.The New England Patriots' rushing game has always been tough to handicap. Sunday, all three of the team's primary running backs put up numbers, and all in different ways, but it worked for the team, as the Patriots beat the Miami Dolphins, 41-13.
  66. """
  67.        , "sport"),
  68.     ("""General Santos (Philippines) (AFP) - Philippine boxing legend Manny Pacquiao vowed to chase Floyd Mayweather into ring submission after his US rival offered to fight him next year in a blockbuster world title face-off. "He (Mayweather) has reached a dead end. He has nowhere to run but to fight me," Pacquiao told AFP late Saturday, hours after the undefeated Mayweather issued the May 2 challenge on US television. The two were long-time rivals as the "best pound-for-pound" boxers of their generation, but the dream fight has never materialised to the disappointment of the boxing world.
  69. """
  70.      , "sport"),
  71.     ("""When St. John's landed Rysheed Jordan, the consensus was that he would be an excellent starter.
  72. So far, that's half true.
  73. Jordan came off the bench Sunday and tied a career high by scoring 24 points to lead No. 24 St. John's to a 74-53 rout of Fordham in the ECAC Holiday Festival.
  74. ''I thought Rysheed played with poise,'' Red Storm coach Steve Lavin said. ''Played with the right pace. Near perfect game.''
  75. """
  76.      , "sport"),
  77.     ("""Five-time world player of the year Marta scored three goals to lead Brazil to a 3-2 come-from-behind win over the U.S. women's soccer team in the International Tournament of Brasilia on Sunday. Carli Lloyd and Megan Rapinoe scored a goal each in the first 10 minutes to give the U.S. an early lead, but Marta netted in the 19th, 55th and 66th minutes to guarantee the hosts a spot in the final of the four-team competition.
  78. """
  79.      , "sport"),
  80. ]
  81.  
  82.  
  83. def getwords(doc):
  84.     splitter = re.compile('\\W*')
  85.     words = set()
  86.     for word in splitter.split(doc):
  87.         if 2 < len(word) < 20:
  88.             words.add(word.lower())
  89.     return words
  90.  
  91. class classifier:
  92.     def __init__(self,getfeatures,filename=None):
  93.         # Broj na parovi atribut/kategorija (feature/category)
  94.         self.fc={}
  95.         # Broj na dokumenti vo sekoja kategorija
  96.         self.cc={}
  97.         # funkcija za dobivanje na atributite (zborovite) vo dokumentot
  98.         self.getfeatures=getfeatures
  99.         # Zgolemuvanje na brojot na parovi atribut/kategorija
  100.  
  101.     def incf(self,f,cat):
  102.         self.fc.setdefault(f,{})
  103.         self.fc[f].setdefault(cat,0)
  104.         self.fc[f][cat]+=1
  105.         # Zgolemuvanje na brojot na predmeti(dokumenti) vo kategorija
  106.  
  107.     def incc(self,cat):
  108.         self.cc.setdefault(cat,0)
  109.         self.cc[cat]+=1
  110.         # Dobivanje na brojot kolku pati
  111.         # odreden atribut se ima pojaveno vo odredena kategorija
  112.  
  113.     def fcount(self,f,cat):
  114.         if f in self.fc and cat in self.fc[f]:
  115.             return float(self.fc[f][cat])
  116.         return 0.0
  117.  
  118.     # Treniranje na klasifikatorot
  119.     # Noviot predmet (dokument) item pripagja na kategorijata cat
  120.     def train(self, item, cat):
  121.         # Se zemaat atributite (zborovite) vo predmetot(dokumentot)
  122.         features = self.getfeatures(item)
  123.         # Se zgolemuva brojot na sekoj atribut vo ovaa kategorija
  124.         for f in features:
  125.             self.incf(f, cat)
  126.         # Se zgolemuva brojot na predmeti
  127.         # (dokumenti) vo ovaa kategorija
  128.         self.incc(cat)
  129.  
  130.     # Dobivanje na brojot kolku pati
  131.     # odreden atribut se ima pojaveno vo odredena kategorija
  132.     def getFeatureCountsPerCategory(self, currentFeature, currentCategory):
  133.         if currentFeature in self.fc and currentCategory in self.fc[currentFeature]:
  134.             v= float(self.fc[currentFeature][currentCategory])
  135.         else:
  136.             v= 0.0
  137.         #print('getFeatureCountsPerCategory',currentFeature, currentCategory, v)\n",
  138.         return v
  139.  
  140.     # Dobivanje na brojot na predmeti(dokumenti) vo kategorija
  141.     def getCategoryCount(self,currentCategory):
  142.         if currentCategory in self.cc:
  143.             return float(self.cc[currentCategory])
  144.         return 0
  145.  
  146.     # Dobivanje na vkupniot broj na predmeti
  147.     def getTotalCount(self):
  148.         v=sum(self.cc.values())
  149.         # print('Dobivanje na vkupniot broj na dokumenti:', v)
  150.         return v
  151.  
  152.     def categories(self):
  153.         return self.cc.keys()
  154.  
  155.     def fprob(self, f, cat):
  156.         if self.getCategoryCount(cat) == 0: return 0
  157.         # Verojatnosta e vkupniot broj na pati koga
  158.         # ovoj atribut f (zbor) se pojavil vo ovaa
  159.         # kategorija podeleno so vkupniot broj na
  160.         # predmeti (dokumenti) vo ovaa kategorija
  161.         return self.fcount(f, cat) / self.getCategoryCount(cat)
  162.  
  163.     def weightedprob(self, f, cat, prf, weight=1.0, ap=0.5):
  164.         # Presmetaj ja osnovnata verojatnost
  165.         basicprob = prf(f, cat)
  166.         # Izbroj kolku pati se ima pojaveno ovoj atribut (zbor)
  167.         # vo site kategorii
  168.         totals = sum([self.fcount(f, c) for c in self.categories()])
  169.         # Presmetaj ja tezinski usrednetata verojatnost
  170.         bp = ((weight * ap) + (totals * basicprob)) / (weight + totals)
  171.         return bp
  172.  
  173.  
  174. class naivebayes(classifier): # nasleduva od classifier
  175.  
  176.     def __init__(self, getfeatures):
  177.         classifier.__init__(self, getfeatures)
  178.         self.thresholds = {}
  179.  
  180.     def setthreshold(self, cat, t):
  181.         self.thresholds[cat] = t
  182.  
  183.     def getthreshold(self, cat):
  184.         if cat not in self.thresholds: return 1.0
  185.         return self.thresholds[cat]
  186.  
  187.     def docprob(self,item,cat):
  188.         # zemi gi zborovite vo dokumentot item
  189.         features=self.getfeatures(item)
  190.         # pomnozi gi verojatnostite na site zborovi
  191.         p=1
  192.         for f in features:
  193.             p*=self.weightedprob(f,cat,self.fprob)
  194.  
  195.         return p
  196.  
  197.     def totalcount(self):
  198.         v=sum(self.cc.values())
  199.         #print('Dobivanje na vkupniot broj na dokumenti:', v)
  200.         return v
  201.  
  202.     def prob(self, item, currentCategory):
  203.         catprob = self.getCategoryCount(currentCategory) / self.getTotalCount()
  204.         caclulateDocumentProbabilityInClass = self.docprob( item, currentCategory)
  205.         # Bayes Theorem
  206.         return caclulateDocumentProbabilityInClass * catprob / (1.0 / self.getTotalCount())
  207.  
  208.     # klasificiranje na dokument
  209.     def classifyDocument(self, item, default=None):
  210.         probs={}
  211.         max=0.0
  212.         for cat in self.categories():
  213.             probs[cat]=self.prob(item, cat)
  214.             #print(probs[cat], cat)
  215.             if probs[cat]>max:
  216.                 max=probs[cat]
  217.                 best=cat
  218.  
  219.         # proveri dali verojatnosta e pogolema od
  220.         # threshold*next best (sledna najdobra)
  221.         for cat in probs:
  222.             if cat==best: continue
  223.             if probs[cat]*self.getthreshold(best)>probs[best]:
  224.                 return default
  225.         return best, format(math.log(max,2) ,'.4f') # max
  226.  
  227. if __name__=='__main__':
  228.     cl = naivebayes(getwords)
  229.     for i in train_data:
  230.         cl.train(i[0], i[1])
  231.  
  232.  
  233.     data = ["""Japan on Tuesday ordered the slaughter of some 4,000 chickens after officials confirmed bird flu at a poultry farm in the southwest of the country. DNA tests confirmed the H5 strain of the virus at a farm in Miyazaki after its owner reported more
  234.    than 20 sudden deaths among his poultry on Sunday and Monday, the agriculture ministry said.""",
  235.             """Mayweather can get the amount he wants. As early as January this year, I challenged him to a charity fight. Until now, he has not agreed to it. So, money is not the issue in our fight," Pacquiao said on his official website. "This fight is about legacy, this is about making the fans happy and, above all, this is for the good of boxing.""",
  236.             """Joe Philbin arrived at work at 5:50 a.m. Monday, anxious to get the Miami Dolphins headed in the right direction. He's running out of time. Consecutive dismal defeats have left the Dolphins all but eliminated from the playoffs with two games to go, and there's widespread speculation Philbin will be fired after three seasons as coach. Predictably, Philbin deflected questions about his future and said he's only focused on the upcoming game Sunday against Minnesota.""",
  237.             """ Former Arsenal and French striker Thierry Henry announced Tuesday he was retiring from football after a glittering two decades long career and will become a TV analyst for Sky Sports in England."After 20 years in the game I have decided to retire from professional football," Henry said in a statement on his Facebook page. Henry, 37, a world and European champion with France, and the scorer of 228 goals with Arsenal including 175 English Premier League goals for the club, finished his playing career at New York Red Bulls just days ago.""",
  238.             """Liverpool had its chances, but David De Gea was monstrous in goal and his attack justified his strength between the sticks as Manchester United sunk the leaky visitors 3-0 at Old Trafford. De Gea has grown into one of the world’s best goalkeepers, and his best was on display today as Liverpool couldn’t find a way through despite great chances early and often for Raheem Sterling and Mario Balotelli. Liverpool began the brightest, with a testing corner inside five minutes, and a solid run from Adam Lallana down the flank which drew a very early yellow for Marouane Fellaini.""",
  239.             """HONOLULU (AP) — Lava from a volcano on Hawaii's Big Island is on course to reach a shopping center with a gas station and a supermarket in seven to 10 days, officials said Monday.""",
  240.             """To say that Osservoort's odyssey took five years wouldn't be accurate, but it has been quite a journey. She left her home town near Amsterdam in 2005, driving her tractor across Europe, and Africa and in an expedition that took four years. """,
  241.             """Just last week, preservationists at the Old Pejeta animal sanctuary in Kenya conceded that their one male and two female northern white rhinos will not reproduce naturally. The animals were flown from the Czech zoo to the Kenyan conservancy in December 2009 in hopes that the natural environment could be easier for them to breed there than in captivity."""
  242.             ]
  243.  
  244.     for ins in data:
  245.         words=getwords(ins)
  246.         category = cl.categories()
  247.  
  248.         for j in words:
  249.             for k in category:
  250.                 print j, k, round(cl.fprob(j,k),4), round(cl.weightedprob(j,k,cl.fprob),4)
  251.         x,y = cl.classifyDocument(ins)
  252.         print x,y
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement