Advertisement
Guest User

Untitled

a guest
Jan 17th, 2020
102
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.44 KB | None | 0 0
  1. val london_crime_records_DS = spark.read.format("org.apache.spark.csv").
  2. option("header", true).option("inferSchema", true).
  3. csv("C:\\Users\\Luk3r\\Desktop\\london-crime\\CityofLondonPoliceRecords.csv").
  4. cache();
  5.  
  6. london_crime_records_DS.union(metropolitan_crime_records_DS).dropDuplicates("Crime type").withColumn("id", functions.monotonically_increasing_id).select("id","Crime type").collect().foreach(x => println(x))
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement