SHARE
TWEET

Untitled

a guest Aug 17th, 2019 71 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. # Define date range
  2. START_DATE = dt.datetime(2019,8,15,20,30,0)
  3. END_DATE = dt.datetime(2019,8,16,15,43,0)
  4.  
  5. # Generate date range with pandas
  6. timerange = pd.date_range(start=START_DATE, end=END_DATE, freq='15min')
  7. # Convert to timestamp
  8. timestamps = [int(x) for x in timerange.values.astype(np.int64) // 10 ** 9]
  9.  
  10. # Create pyspark dataframe from the above timestamps
  11. (spark.createDataFrame(dates, IntegerType())
  12.     .withColumn('value_date', sf.from_unixtime('value'))
  13.     .drop('value')
  14.     .withColumnRenamed('value_date', 'date').show())
  15.      
  16. +-------------------+
  17. |               date|
  18. +-------------------+
  19. |2019-08-15 20:30:00|
  20. |2019-08-15 20:45:00|
  21. |2019-08-15 21:00:00|
  22. |2019-08-15 21:15:00|
  23. |2019-08-15 21:30:00|
  24. |2019-08-15 21:45:00|
  25. |2019-08-15 22:00:00|
  26. |2019-08-15 22:15:00|
  27. |2019-08-15 22:30:00|
  28. |2019-08-15 22:45:00|
  29. |2019-08-15 23:00:00|
  30. |2019-08-15 23:15:00|
  31. |2019-08-15 23:30:00|
  32. |2019-08-15 23:45:00|
  33. |2019-08-16 00:00:00|
  34. |2019-08-16 00:15:00|
  35. |2019-08-16 00:30:00|
  36. |2019-08-16 00:45:00|
  37. |2019-08-16 01:00:00|
  38. |2019-08-16 01:15:00|
  39. +-------------------+
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!
 
Top