API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jun 27th, 2017
549
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 26.22 KB | None | 0 0
raw download clone embed print report
  1. {
  2. "nbformat": 4,
  3. "cells": [
  4. {
  5. "execution_count": 8,
  6. "source": "# The code was removed by DSX for sharing.",
  7. "cell_type": "code",
  8. "metadata": {
  9. "collapsed": true
  10. },
  11. "outputs": []
  12. },
  13. {
  14. "source": "## This is testing for gist push",
  15. "cell_type": "markdown",
  16. "metadata": {}
  17. },
  18. {
  19. "source": "## Ingest load data from dashDB",
  20. "cell_type": "markdown",
  21. "metadata": {}
  22. },
  23. {
  24. "execution_count": 9,
  25. "source": "from ingest.Connectors import Connectors\n\nsourceOptions = { \n Connectors.DASHDB.HOST : credentials_1[\"host\"],\n Connectors.DASHDB.DATABASE : credentials_1[\"db\"],\n Connectors.DASHDB.USERNAME : credentials_1[\"username\"],\n Connectors.DASHDB.PASSWORD : credentials_1[\"password\"],\n Connectors.DASHDB.SOURCE_TABLE_NAME : 'DASH107803.Test',\n}\n\ndf = sqlContext.read.format('com.ibm.spark.discover').options(**sourceOptions).load()\ndf.show()\ndf.printSchema()",
  26. "cell_type": "code",
  27. "metadata": {},
  28. "outputs": [
  29. {
  30. "name": "stdout",
  31. "text": "+---+----------------+-----+\n| id| text|label|\n+---+----------------+-----+\n| 0| a b c d e spark| 1.0|\n| 3|hadoop mapreduce| 0.0|\n| 1| b d| 0.0|\n| 0| a b c d e spark| 1.0|\n| 3|hadoop mapreduce| 0.0|\n| 3|hadoop mapreduce| 0.0|\n| 1| b d| 0.0|\n| 1| b d| 0.0|\n| 1| b d| 0.0|\n| 2| spark f g h| 1.0|\n| 1| b d| 0.0|\n| 0| a b c d e spark| 1.0|\n| 2| spark f g h| 1.0|\n| 0| a b c d e spark| 1.0|\n| 0| a b c d e spark| 1.0|\n| 3|hadoop mapreduce| 0.0|\n| 2| spark f g h| 1.0|\n| 0| a b c d e spark| 1.0|\n| 3|hadoop mapreduce| 0.0|\n| 2| spark f g h| 1.0|\n+---+----------------+-----+\nonly showing top 20 rows\n\nroot\n |-- id: long (nullable = true)\n |-- text: string (nullable = true)\n |-- label: double (nullable = true)\n\n",
  32. "output_type": "stream"
  33. }
  34. ]
  35. },
  36. {
  37. "source": "## Ingest save dataframe to a table in dashDB",
  38. "cell_type": "markdown",
  39. "metadata": {}
  40. },
  41. {
  42. "execution_count": 10,
  43. "source": "targetOptions = { \n Connectors.DASHDB.HOST : credentials_1[\"host\"],\n Connectors.DASHDB.DATABASE : credentials_1[\"db\"],\n Connectors.DASHDB.USERNAME : credentials_1[\"username\"],\n Connectors.DASHDB.PASSWORD : credentials_1[\"password\"],\n Connectors.DASHDB.TARGET_TABLE_NAME : 'DASH106358.OutPutTable',\n}\n\ndf.write.format('com.ibm.spark.discover').options(**sourceOptions).save()",
  44. "cell_type": "code",
  45. "metadata": {},
  46. "outputs": [
  47. {
  48. "evalue": "An error occurred while calling o285.save.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 10 times, most recent failure: Lost task 0.9 in stage 5.0 (TID 32, yp-spark-dal09-env5-0030): org.apache.spark.SparkException: CDICO2018E: Table name null is not in [schema_name.]table_name format.\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:98)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:25)\n\tat org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:218)\n\tat org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:218)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:922)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:922)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1913)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1913)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:86)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.lang.Thread.run(Thread.java:785)\nCaused by: com.ibm.iis.scapi.SCAPIException: CDICO2018E: Table name null is not in [schema_name.]table_name format.\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcTableNameHelper.<init>(JdbcTableNameHelper.java:160)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.getTableType(JdbcOutputInteraction.java:1176)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.handleTableAction(JdbcOutputInteraction.java:1121)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.<init>(JdbcOutputInteraction.java:186)\n\tat com.ibm.iis.scapi.dw.jdbc.dashdb.DashdbOutputInteraction.<init>(DashdbOutputInteraction.java:39)\n\tat com.ibm.iis.scapi.dw.jdbc.dashdb.DashdbConnector.getOutputInteraction(DashdbConnector.java:152)\n\tat com.ibm.is.drs.spark.connector.ConnectorUtils.getOutputInteraction(ConnectorUtils.java:837)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertMapFunction$RowIterator.hasNext(ConnectorInsertMapFunction.java:164)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:81)\n\t... 13 more\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1461)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1449)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1448)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1448)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:812)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:812)\n\tat scala.Option.foreach(Option.scala:257)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:812)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1674)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1629)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat java.lang.Thread.getStackTrace(Thread.java:1117)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:633)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1887)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1900)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1913)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1927)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:922)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:920)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\n\tat org.apache.spark.rdd.RDD.withScope(RDD.scala:378)\n\tat org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:920)\n\tat org.apache.spark.api.java.JavaRDDLike$class.foreachPartition(JavaRDDLike.scala:218)\n\tat org.apache.spark.api.java.AbstractJavaRDDLike.foreachPartition(JavaRDDLike.scala:45)\n\tat com.ibm.is.drs.spark.rdd.connector.DataFrameWriter.writeDataFrame(DataFrameWriter.java:114)\n\tat com.ibm.is.drs.spark.connector.RDDFactory.writeDataFrame(RDDFactory.java:63)\n\tat com.ibm.is.drs.spark.connector.ConnectorRelation.insert(ConnectorRelation.java:139)\n\tat com.ibm.is.drs.spark.connector.DefaultSource.createRelation(DefaultSource.scala:40)\n\tat org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:457)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)\n\tat com.ibm.spark.discover.DefaultSource.createRelation(DefaultSource.scala:77)\n\tat org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:457)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)\n\tat java.lang.reflect.Method.invoke(Method.java:507)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:280)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:214)\n\tat java.lang.Thread.run(Thread.java:785)\nCaused by: org.apache.spark.SparkException: CDICO2018E: Table name null is not in [schema_name.]table_name format.\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:98)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:25)\n\tat org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:218)\n\tat org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:218)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:922)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:922)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1913)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1913)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:86)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\t... 1 more\nCaused by: com.ibm.iis.scapi.SCAPIException: CDICO2018E: Table name null is not in [schema_name.]table_name format.\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcTableNameHelper.<init>(JdbcTableNameHelper.java:160)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.getTableType(JdbcOutputInteraction.java:1176)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.handleTableAction(JdbcOutputInteraction.java:1121)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.<init>(JdbcOutputInteraction.java:186)\n\tat com.ibm.iis.scapi.dw.jdbc.dashdb.DashdbOutputInteraction.<init>(DashdbOutputInteraction.java:39)\n\tat com.ibm.iis.scapi.dw.jdbc.dashdb.DashdbConnector.getOutputInteraction(DashdbConnector.java:152)\n\tat com.ibm.is.drs.spark.connector.ConnectorUtils.getOutputInteraction(ConnectorUtils.java:837)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertMapFunction$RowIterator.hasNext(ConnectorInsertMapFunction.java:164)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:81)\n\t... 13 more\n",
  49. "traceback": [
  50. "\u001b[0;31m\u001b[0m",
  51. "\u001b[0;31mPy4JJavaError\u001b[0mTraceback (most recent call last)",
  52. "\u001b[0;32m<ipython-input-10-a2fa638148c0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 7\u001b[0m }\n\u001b[1;32m 8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'com.ibm.spark.discover'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0msourceOptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
  53. "\u001b[0;32m/usr/local/src/spark20master/spark/python/pyspark/sql/readwriter.py\u001b[0m in \u001b[0;36msave\u001b[0;34m(self, path, format, mode, partitionBy, **options)\u001b[0m\n\u001b[1;32m 543\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 544\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mpath\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 545\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jwrite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 546\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 547\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jwrite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
  54. "\u001b[0;32m/usr/local/src/spark20master/spark/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1131\u001b[0m \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1132\u001b[0m return_value = get_return_value(\n\u001b[0;32m-> 1133\u001b[0;31m answer, self.gateway_client, self.target_id, self.name)\n\u001b[0m\u001b[1;32m 1134\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1135\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtemp_arg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtemp_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
  55. "\u001b[0;32m/usr/local/src/spark20master/spark/python/pyspark/sql/utils.py\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m 61\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mdeco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 62\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 63\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 64\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mpy4j\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPy4JJavaError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 65\u001b[0m \u001b[0ms\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjava_exception\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
  56. "\u001b[0;32m/usr/local/src/spark20master/spark/python/lib/py4j-0.10.3-src.zip/py4j/protocol.py\u001b[0m in \u001b[0;36mget_return_value\u001b[0;34m(answer, gateway_client, target_id, name)\u001b[0m\n\u001b[1;32m 317\u001b[0m raise Py4JJavaError(\n\u001b[1;32m 318\u001b[0m \u001b[0;34m\"An error occurred while calling {0}{1}{2}.\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 319\u001b[0;31m format(target_id, \".\", name), value)\n\u001b[0m\u001b[1;32m 320\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 321\u001b[0m raise Py4JError(\n",
  57. "\u001b[0;31mPy4JJavaError\u001b[0m: An error occurred while calling o285.save.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 10 times, most recent failure: Lost task 0.9 in stage 5.0 (TID 32, yp-spark-dal09-env5-0030): org.apache.spark.SparkException: CDICO2018E: Table name null is not in [schema_name.]table_name format.\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:98)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:25)\n\tat org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:218)\n\tat org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:218)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:922)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:922)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1913)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1913)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:86)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.lang.Thread.run(Thread.java:785)\nCaused by: com.ibm.iis.scapi.SCAPIException: CDICO2018E: Table name null is not in [schema_name.]table_name format.\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcTableNameHelper.<init>(JdbcTableNameHelper.java:160)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.getTableType(JdbcOutputInteraction.java:1176)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.handleTableAction(JdbcOutputInteraction.java:1121)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.<init>(JdbcOutputInteraction.java:186)\n\tat com.ibm.iis.scapi.dw.jdbc.dashdb.DashdbOutputInteraction.<init>(DashdbOutputInteraction.java:39)\n\tat com.ibm.iis.scapi.dw.jdbc.dashdb.DashdbConnector.getOutputInteraction(DashdbConnector.java:152)\n\tat com.ibm.is.drs.spark.connector.ConnectorUtils.getOutputInteraction(ConnectorUtils.java:837)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertMapFunction$RowIterator.hasNext(ConnectorInsertMapFunction.java:164)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:81)\n\t... 13 more\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1461)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1449)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1448)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1448)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:812)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:812)\n\tat scala.Option.foreach(Option.scala:257)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:812)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1674)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1629)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat java.lang.Thread.getStackTrace(Thread.java:1117)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:633)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1887)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1900)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1913)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1927)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:922)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:920)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\n\tat org.apache.spark.rdd.RDD.withScope(RDD.scala:378)\n\tat org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:920)\n\tat org.apache.spark.api.java.JavaRDDLike$class.foreachPartition(JavaRDDLike.scala:218)\n\tat org.apache.spark.api.java.AbstractJavaRDDLike.foreachPartition(JavaRDDLike.scala:45)\n\tat com.ibm.is.drs.spark.rdd.connector.DataFrameWriter.writeDataFrame(DataFrameWriter.java:114)\n\tat com.ibm.is.drs.spark.connector.RDDFactory.writeDataFrame(RDDFactory.java:63)\n\tat com.ibm.is.drs.spark.connector.ConnectorRelation.insert(ConnectorRelation.java:139)\n\tat com.ibm.is.drs.spark.connector.DefaultSource.createRelation(DefaultSource.scala:40)\n\tat org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:457)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)\n\tat com.ibm.spark.discover.DefaultSource.createRelation(DefaultSource.scala:77)\n\tat org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:457)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)\n\tat java.lang.reflect.Method.invoke(Method.java:507)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:280)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:214)\n\tat java.lang.Thread.run(Thread.java:785)\nCaused by: org.apache.spark.SparkException: CDICO2018E: Table name null is not in [schema_name.]table_name format.\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:98)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:25)\n\tat org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:218)\n\tat org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:218)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:922)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:922)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1913)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1913)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:86)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\t... 1 more\nCaused by: com.ibm.iis.scapi.SCAPIException: CDICO2018E: Table name null is not in [schema_name.]table_name format.\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcTableNameHelper.<init>(JdbcTableNameHelper.java:160)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.getTableType(JdbcOutputInteraction.java:1176)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.handleTableAction(JdbcOutputInteraction.java:1121)\n\tat com.ibm.iis.scapi.dw.jdbc.JdbcOutputInteraction.<init>(JdbcOutputInteraction.java:186)\n\tat com.ibm.iis.scapi.dw.jdbc.dashdb.DashdbOutputInteraction.<init>(DashdbOutputInteraction.java:39)\n\tat com.ibm.iis.scapi.dw.jdbc.dashdb.DashdbConnector.getOutputInteraction(DashdbConnector.java:152)\n\tat com.ibm.is.drs.spark.connector.ConnectorUtils.getOutputInteraction(ConnectorUtils.java:837)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertMapFunction$RowIterator.hasNext(ConnectorInsertMapFunction.java:164)\n\tat com.ibm.is.drs.spark.rdd.connector.ConnectorInsertFunction.call(ConnectorInsertFunction.java:81)\n\t... 13 more\n"
  58. ],
  59. "ename": "Py4JJavaError",
  60. "output_type": "error"
  61. }
  62. ]
  63. },
  64. {
  65. "execution_count": null,
  66. "source": "",
  67. "cell_type": "code",
  68. "metadata": {
  69. "collapsed": true
  70. },
  71. "outputs": []
  72. }
  73. ],
  74. "metadata": {
  75. "language_info": {
  76. "pygments_lexer": "ipython2",
  77. "codemirror_mode": {
  78. "version": 2,
  79. "name": "ipython"
  80. },
  81. "mimetype": "text/x-python",
  82. "name": "python",
  83. "file_extension": ".py",
  84. "version": "2.7.11",
  85. "nbconvert_exporter": "python"
  86. },
  87. "kernelspec": {
  88. "language": "python",
  89. "display_name": "Python 2 with Spark 2.0",
  90. "name": "python2-spark20"
  91. }
  92. },
  93. "nbformat_minor": 1
  94. }
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!