Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- object MainCode extends App {
- val conf = new SparkConf
- conf.setMaster("local[*]").setAppName(this.getClass.getName)
- val sc = new SparkContext(conf)
- // the RDD instantiation goes here
- val rdd = ...
- // instead of
- // val thisWontWork = new ShouldBeLocal
- // rdd.mapPartitions(_.map(thisWontWork.theMethod))
- // we'll do the following...
- rdd.mapPartitions(_.map(ShouldBeLocalWrapper.theMethodWrapper))
- // then of course some action taken
- ...
- sc.stop
- }
- // Creating a singleton wrapper will make the contained codes
- // available to all executors. This is useful if creating ShouldBeLocal
- // instance a very expensive operation. For non thread-safe, we can also
- // implement object pooling of the said instance, e.g. for DB connectivities
- object ShouldBeLocalWrapper {
- // Usage of ShouldBeLocal is assumed to be thread-safe.
- lazy val inst: ShouldBeLocal = new ShouldBeLocal
- // Let's say we need to use an arbitary class/object ShouldBeLocal
- // with function/method theMethod to parse the data..
- def theMethodWrapper(s: String) = inst.theMethod(s)
- }
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement