Advertisement
Guest User

Untitled

a guest
Oct 24th, 2016
63
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.07 KB | None | 0 0
  1.  
  2. object MainCode extends App {
  3. val conf = new SparkConf
  4. conf.setMaster("local[*]").setAppName(this.getClass.getName)
  5. val sc = new SparkContext(conf)
  6.  
  7. // the RDD instantiation goes here
  8. val rdd = ...
  9.  
  10. // instead of
  11. // val thisWontWork = new ShouldBeLocal
  12. // rdd.mapPartitions(_.map(thisWontWork.theMethod))
  13. // we'll do the following...
  14. rdd.mapPartitions(_.map(ShouldBeLocalWrapper.theMethodWrapper))
  15. // then of course some action taken
  16. ...
  17.  
  18. sc.stop
  19. }
  20.  
  21. // Creating a singleton wrapper will make the contained codes
  22. // available to all executors. This is useful if creating ShouldBeLocal
  23. // instance a very expensive operation. For non thread-safe, we can also
  24. // implement object pooling of the said instance, e.g. for DB connectivities
  25. object ShouldBeLocalWrapper {
  26.  
  27. // Usage of ShouldBeLocal is assumed to be thread-safe.
  28. lazy val inst: ShouldBeLocal = new ShouldBeLocal
  29.  
  30. // Let's say we need to use an arbitary class/object ShouldBeLocal
  31. // with function/method theMethod to parse the data..
  32. def theMethodWrapper(s: String) = inst.theMethod(s)
  33. }
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement