Advertisement
unmeshabiju

Technical

Aug 7th, 2014
211
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.16 KB | None | 0 0
  1. book 6
  2. book,eraser 2
  3. book,pen 4
  4. book,pen,pencil 3
  5. book,pencil 5
  6. chalk 3
  7. chalk,eraser 2
  8. chalk,eraser,pen 2
  9. chalk,pen 2
  10. eraser 6
  11. eraser,pen 5
  12. eraser,pen,pencil 2
  13. eraser,pencil 3
  14. pen 8
  15. pen,pencil 5
  16. pencil 7
  17.  
  18. This will be my input to Mapper 1 and Mapper 2
  19. In Mapper 1 I will be generating rules with this input file
  20.  
  21. //OUTPUT
  22. Rule 1: book => eraser 2
  23. Rule 0: eraser => book 2
  24. Rule 1: book => pen 4
  25. Rule 0: pen => book 4
  26.  
  27. Mapper2 emits the same input file
  28.  
  29.  
  30. What calculation I have to do is
  31. P(A union B)/P(B)
  32.  
  33. ie for
  34. book => eraser = 2/6
  35. eraser => book 2/6
  36.  
  37. book => pen 4/6
  38. pen => book 4/8
  39.  
  40. So my logic is
  41. in mapper1
  42. context.write(book,book => eraser 2)
  43. context.write(eraser,eraser => book 2) etc
  44. AND
  45. in mapper 2
  46. context.write(book,6)
  47. context.write(book,eraser,2)
  48.  
  49. So in Reducer we get
  50.  
  51. key -- book
  52. values---book => eraser 2
  53. 6
  54.  
  55. similary for
  56. key -- eraser
  57. values----eraser => book 2
  58. 6
  59.  
  60. So i can clavulate as explained in my Stackoverflow question
  61.  
  62. So while doing secondary sorting I have to emit a composite key right?
  63. so if i am emitting composite key ,I will not be able to get values in reducer as expected
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement