Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- // create the Spark configuration and context
- SparkConf conf = new SparkConf().setAppName("Wordcount").setMaster("local[*]");
- JavaSparkContext sc = new JavaSparkContext(conf);
- // load data and create an RDD of string
- JavaRDD<String> tweets = sc.textFile("path_To_File")
- JavaPairRDD<String, Integer> wordcount = tweets.flatMap(line -> Arrays.asList(line.toString().split(" ")))
- // mapper step
- .mapToPair(word -> new Tuple2<>(word, 1))
- // reducer step
- .reduceByKey((x, y) -> x + y);
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement