Guest User

Untitled

a guest
Apr 30th, 2018
95
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.97 KB | None | 0 0
  1. **ERROR : java.lang.OutOfMemoryError java heap space**
  2.  
  3. **ERROR : java.lang.OutOfMemoryError: GC overhead limit exceeded**
  4.  
  5. spark.driver.memory 1g
  6.  
  7. spark.executor.memory 1g
  8. spark.executor.extraJavaOptions Xmx1024m
  9. spark.dirver.maxResultSize 2g
  10.  
  11. **PSQL ERROR : org.postgresql.util.psqlexception ran out of memory retrieving query results**
  12.  
  13. from pyspark.sql import SparkSession
  14. from pyspark import SparkContext
  15. from pyspark.sql import SQLContext
  16. from pyspark.sql import HiveContext
  17.  
  18. from pyspark.sql import DataFrameReader
  19.  
  20.  
  21. sc = SparkContext()
  22.  
  23. sqlContext = SQLContext(sc)
  24. sqlContext = HiveContext(sc)
  25.  
  26. input_media_source = " Times of India"
  27.  
  28. # Connecting to news_mentions table
  29. df1 = sqlContext.read.format("jdbc").option("url", "jdbc:postgresql://localhost:5432/testdb").option("dbtable", "news_mentions").option("user", "postgres").load()
  30. df1.createOrReplaceTempView("news_mentions")
  31. news_mentions_DF = sqlContext.sql("SELECT * FROM news_mentions")
  32.  
  33. news_mentions_DF.show()
Add Comment
Please, Sign In to add comment