Advertisement
Guest User

Untitled

a guest
Jun 26th, 2019
82
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.80 KB | None | 0 0
  1. temp_tab_name = "mytablename";
  2. x = spark.sql("select * from " +temp_tab_name +" limit 10");
  3. x = x.persist()
  4. x.count() #action to activate all the above steps
  5. x.show() #x should have been persisted in memory here, DAG evaluated, no going back to "select..." whenever referred to
  6. x.is_cached #True
  7. spark.sql("drop table "+ temp_tab_name);
  8. x.is_cached #Still true!!
  9. x.show() # Error, table not found here
  10.  
  11. # df method
  12. df = spark.range(10)
  13. df.count() # action to materialize df object in ram
  14. df.unpersist() # remove df object from ram
  15.  
  16. # temp table method
  17. df.createOrReplaceTempView("df_sql")
  18. spark.catalog.cacheTable("df_sql")
  19. spark.sql("select * from df_sql").count() # action to materialize temp table in ram
  20. spark.catalog.uncacheTable("df_sql") # remove temp table from ram
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement