Advertisement
Guest User

Untitled

a guest
Aug 18th, 2019
103
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.52 KB | None | 0 0
  1. import org.apache.spark.{SparkConf, SparkContext}
  2. import org.apache.spark.SparkContext._
  3. println("Creating Context")
  4. val conf = new SparkConf().setMaster("local").setAppName("Test")
  5. val sc = new SparkContext(conf)
  6. val lines = sc.textFile("/home/lbali/example.txt")
  7. val pandas = lines filter(line => line.contains("pandas"))
  8. println("+++++ number of lines: " + lines.count()) // this works ok.
  9. println("+++++ number of lines with pandas: " + pandas.count()) // This does not work
  10. sc.stop()
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement