a guest Aug 18th, 2019 81 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. import org.apache.spark.{SparkConf, SparkContext}
  2.     import org.apache.spark.SparkContext._
  3.     println("Creating Context")
  4.     val conf = new SparkConf().setMaster("local").setAppName("Test")
  5.     val sc = new SparkContext(conf)
  6.     val lines = sc.textFile("/home/lbali/example.txt")
  7.     val pandas = lines filter(line => line.contains("pandas"))
  8.     println("+++++ number of lines: " + lines.count()) // this works ok.
  9.     println("+++++ number of lines with pandas: " + pandas.count()) // This does not work
  10.     sc.stop()
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand