Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- **ERROR : java.lang.OutOfMemoryError java heap space**
- **ERROR : java.lang.OutOfMemoryError: GC overhead limit exceeded**
- spark.driver.memory 1g
- spark.executor.memory 1g
- spark.executor.extraJavaOptions Xmx1024m
- spark.dirver.maxResultSize 2g
- **PSQL ERROR : org.postgresql.util.psqlexception ran out of memory retrieving query results**
- from pyspark.sql import SparkSession
- from pyspark import SparkContext
- from pyspark.sql import SQLContext
- from pyspark.sql import HiveContext
- from pyspark.sql import DataFrameReader
- sc = SparkContext()
- sqlContext = SQLContext(sc)
- sqlContext = HiveContext(sc)
- input_media_source = " Times of India"
- # Connecting to news_mentions table
- df1 = sqlContext.read.format("jdbc").option("url", "jdbc:postgresql://localhost:5432/testdb").option("dbtable", "news_mentions").option("user", "postgres").load()
- df1.createOrReplaceTempView("news_mentions")
- news_mentions_DF = sqlContext.sql("SELECT * FROM news_mentions")
- news_mentions_DF.show()
Add Comment
Please, Sign In to add comment