Advertisement
Guest User

Untitled

a guest
Jul 29th, 2016
54
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.60 KB | None | 0 0
  1. //Get all the calls for the last 24 hours for each MSISDN in the hour
  2. val sCallsPlaced = (grouped24HourCallsPlaS).join(distinctMSISDNs)
  3. val oCallsPlaced = (grouped24HourCallsPlaO).join(distinctMSISDNs)
  4. val sCallsReceived = grouped24HourCallsRecS.join(distinctMSISDNs)
  5. val oCallsReceived = grouped24HourCallsRecO.join(distinctMSISDNs)
  6.  
  7. val callsToProcess = sCallsPlaced.union(oCallsPlaced).union(sCallsReceived).union(oCallsReceived)
  8.  
  9. spark.driver.memory=16g
  10. spark.driver.cores=1
  11. spark.driver.maxResultSize=2g
  12.  
  13. spark.executor.memory=24g
  14. spark.executor.cores=10
  15.  
  16. spark.default.parallelism=256
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement