API tools faq
paste
Guest User

20200528_no_overwrite

a guest
May 28th, 2020
123
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 38.83 KB | None | 0 0
raw download clone embed print report
  1. (pipeline)root@pre-openedx:/edx/analytics# remote-task InsertToMysqlCourseActivityTask --host localhost --user root --remote-name analyticstack --skip-setup --wait --local-scheduler --end-date 2020-05-28 --weeks 52 --n-reduce-tasks 1 --overwrite-n-days 365
  2. Parsed arguments = Namespace(branch='release', extra_repo=None, host='localhost', job_flow_id=None, job_flow_name=None, launch_task_arguments=['InsertToMysqlCourseActivityTask', '--local-scheduler', '--end-date', '2020-05-28', '--weeks', '52', '--n-reduce-tasks', '1', '--overwrite-n-days', '365'], log_path=None, override_config=None, package=None, private_key=None, python_version=None, remote_name='analyticstack', repo=None, secure_config=None, secure_config_branch=None, secure_config_repo=None, shell=None, skip_setup=True, sudo_user='hadoop', user='root', vagrant_path=None, verbose=False, virtualenv_extra_args=None, wait=True, wheel_url=None, workflow_profiler=None)
  3. Running commands from path = /edx/analytics/pipeline/share/edx.analytics.tasks
  4. Remote name = analyticstack
  5. Running command = ['ssh', '-tt', '-o', 'ForwardAgent=yes', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PasswordAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', 'localhost', "sudo -Hu hadoop /bin/bash -c 'cd /var/lib/analytics-tasks/analyticstack/repo && . $HOME/.bashrc && . /var/lib/analytics-tasks/analyticstack/venv/bin/activate && launch-task InsertToMysqlCourseActivityTask --local-scheduler --end-date 2020-05-28 --weeks 52 --n-reduce-tasks 1 --overwrite-n-days 365'"]
  6. Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
  7.  
  8.  
  9.  ================================================================
  10.  Esta maquina solo es para usuarios autorizados.
  11.  Si usted no esta autorizado, por favor, no intente acceder.
  12.  Todos los accesos son monitorizados y comprobados.
  13.  ================================================================
  14.  This machine is only for authorized users.
  15.  If you are not authorized, please, do not try to access.
  16.  All the accesses are logged and are verified.
  17.  ================================================================
  18.  
  19.  
  20. No handlers could be found for logger "luigi-interface"
  21. DEBUG:stevedore.extension:found extension EntryPoint.parse('sqoop-import = edx.analytics.tasks.common.sqoop:SqoopImportFromMysql')
  22. DEBUG:stevedore.extension:found extension EntryPoint.parse('run-vertica-sql-script = edx.analytics.tasks.warehouse.run_vertica_sql_script:RunVerticaSqlScriptTask')
  23. DEBUG:stevedore.extension:found extension EntryPoint.parse('obfuscation = edx.analytics.tasks.export.obfuscation:ObfuscatedCourseTask')
  24. DEBUG:stevedore.extension:found extension EntryPoint.parse('enrollment_validation = edx.analytics.tasks.monitor.enrollment_validation:CourseEnrollmentValidationTask')
  25. INFO:luigi-interface:Loaded ['/etc/luigi/client.cfg', 'client.cfg']
  26. DEBUG:stevedore.extension:found extension EntryPoint.parse('test-vertica-sqoop = edx.analytics.tasks.common.vertica_export:VerticaSchemaToBigQueryTask')
  27. DEBUG:stevedore.extension:found extension EntryPoint.parse('problem_response = edx.analytics.tasks.insights.problem_response:LatestProblemResponseDataTask')
  28. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-warehouse-bigquery = edx.analytics.tasks.warehouse.load_warehouse_bigquery:LoadWarehouseBigQueryTask')
  29. DEBUG:stevedore.extension:found extension EntryPoint.parse('push_to_vertica_lms_courseware_link_clicked = edx.analytics.tasks.warehouse.lms_courseware_link_clicked:PushToVerticaLMSCoursewareLinkClickedTask')
  30. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-internal-active-users = edx.analytics.tasks.warehouse.load_internal_reporting_active_users:LoadInternalReportingActiveUsersToWarehouse')
  31. DEBUG:stevedore.extension:found extension EntryPoint.parse('video = edx.analytics.tasks.insights.video:InsertToMysqlAllVideoTask')
  32. DEBUG:stevedore.extension:found extension EntryPoint.parse('ed_services_report = edx.analytics.tasks.warehouse.financial.ed_services_financial_report:BuildEdServicesReportTask')
  33. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-internal-database = edx.analytics.tasks.warehouse.load_internal_reporting_database:ImportMysqlToVerticaTask')
  34. DEBUG:snowflake.connector.ssl_wrap_socket:Injecting ssl_wrap_socket_with_ocsp
  35. DEBUG:snowflake.connector.auth:cache directory: /edx/app/hadoop/.cache/snowflake
  36. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-insights = edx.analytics.tasks.warehouse.load_warehouse_insights:LoadInsightsTableToVertica')
  37. DEBUG:stevedore.extension:found extension EntryPoint.parse('export-student-module = edx.analytics.tasks.export.database_exports:StudentModulePerCourseAfterImportWorkflow')
  38. DEBUG:stevedore.extension:found extension EntryPoint.parse('calendar = edx.analytics.tasks.insights.calendar_task:CalendarTableTask')
  39. DEBUG:stevedore.extension:found extension EntryPoint.parse('snowflake-load = edx.analytics.tasks.common.snowflake_load:SnowflakeLoadTask')
  40. DEBUG:stevedore.extension:found extension EntryPoint.parse('affiliate_window = edx.analytics.tasks.warehouse.financial.fees:LoadFeesToWarehouse')
  41. DEBUG:stevedore.extension:found extension EntryPoint.parse('orders = edx.analytics.tasks.warehouse.financial.orders_import:OrderTableTask')
  42. DEBUG:stevedore.extension:found extension EntryPoint.parse('cybersource = edx.analytics.tasks.warehouse.financial.cybersource:DailyPullFromCybersourceTask')
  43. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-user = edx.analytics.tasks.warehouse.load_internal_reporting_user:LoadInternalReportingUserToWarehouse')
  44. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-google-sheet-warehouse = edx.analytics.tasks.warehouse.load_google_sheet_to_warehouse:LoadGoogleSpreadsheetsToWarehouseWorkflow')
  45. DEBUG:stevedore.extension:found extension EntryPoint.parse('location-per-course = edx.analytics.tasks.insights.location_per_course:LastCountryOfUser')
  46. DEBUG:stevedore.extension:found extension EntryPoint.parse('payment_reconcile = edx.analytics.tasks.warehouse.financial.reconcile:ReconcileOrdersAndTransactionsTask')
  47. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-warehouse = edx.analytics.tasks.warehouse.load_warehouse:LoadWarehouseWorkflow')
  48. DEBUG:stevedore.extension:found extension EntryPoint.parse('engagement = edx.analytics.tasks.insights.module_engagement:ModuleEngagementDataTask')
  49. DEBUG:stevedore.extension:found extension EntryPoint.parse('events_obfuscation = edx.analytics.tasks.export.events_obfuscation:ObfuscateCourseEventsTask')
  50. DEBUG:stevedore.extension:found extension EntryPoint.parse('dump-student-module = edx.analytics.tasks.export.database_exports:StudentModulePerCourseTask')
  51. DEBUG:stevedore.extension:found extension EntryPoint.parse('export-events-by-course = edx.analytics.tasks.export.event_exports_by_course:EventExportByCourseTask')
  52. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-ga-permissions = edx.analytics.tasks.warehouse.load_ga_permissions:LoadGoogleAnalyticsPermissionsWorkflow')
  53. DEBUG:stevedore.extension:found extension EntryPoint.parse('noop = edx.analytics.tasks.monitor.performance:ParseEventLogPerformanceTask')
  54. DEBUG:stevedore.extension:found extension EntryPoint.parse('course_blocks = edx.analytics.tasks.insights.course_blocks:CourseBlocksApiDataTask')
  55. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-events = edx.analytics.tasks.warehouse.load_internal_reporting_events:TrackingEventRecordDataTask')
  56. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-certificates = edx.analytics.tasks.warehouse.load_internal_reporting_certificates:LoadInternalReportingCertificatesToWarehouse')
  57. DEBUG:stevedore.extension:found extension EntryPoint.parse('user-activity = edx.analytics.tasks.insights.user_activity:InsertToMysqlCourseActivityTask')
  58. DEBUG:stevedore.extension:found extension EntryPoint.parse('tags-dist = edx.analytics.tasks.insights.tags_dist:TagsDistributionPerCourse')
  59. DEBUG:stevedore.extension:found extension EntryPoint.parse('bigquery-load = edx.analytics.tasks.common.bigquery_load:BigQueryLoadTask')
  60. DEBUG:stevedore.extension:found extension EntryPoint.parse('run-vertica-sql-scripts = edx.analytics.tasks.warehouse.run_vertica_sql_scripts:RunVerticaSqlScriptTask')
  61. DEBUG:stevedore.extension:found extension EntryPoint.parse('paypal = edx.analytics.tasks.warehouse.financial.paypal:PaypalTransactionsByDayTask')
  62. DEBUG:stevedore.extension:found extension EntryPoint.parse('grade-dist = edx.analytics.tasks.data_api.studentmodule_dist:GradeDistFromSqoopToMySQLWorkflow')
  63. DEBUG:stevedore.extension:found extension EntryPoint.parse('database-import = edx.analytics.tasks.insights.database_imports:ImportAllDatabaseTablesTask')
  64. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-course-catalog = edx.analytics.tasks.warehouse.load_internal_reporting_course_catalog:PullDiscoveryCoursesAPIData')
  65. DEBUG:stevedore.extension:found extension EntryPoint.parse('enrollments = edx.analytics.tasks.insights.enrollments:ImportEnrollmentsIntoMysql')
  66. DEBUG:stevedore.extension:found extension EntryPoint.parse('event-type-dist = edx.analytics.tasks.warehouse.event_type_dist:PushToVerticaEventTypeDistributionTask')
  67. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-internal-course-structure = edx.analytics.tasks.warehouse.load_internal_reporting_course_structure:LoadCourseBlockRecordToVertica')
  68. DEBUG:stevedore.extension:found extension EntryPoint.parse('enterprise_enrollments = edx.analytics.tasks.enterprise.enterprise_enrollments:ImportEnterpriseEnrollmentsIntoMysql')
  69. DEBUG:stevedore.extension:found extension EntryPoint.parse('export-events = edx.analytics.tasks.export.event_exports:EventExportTask')
  70. DEBUG:stevedore.extension:found extension EntryPoint.parse('financial_reports = edx.analytics.tasks.warehouse.financial.finance_reports:BuildFinancialReportsTask')
  71. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-warehouse-snowflake = edx.analytics.tasks.warehouse.load_warehouse_snowflake:LoadWarehouseSnowflakeTask')
  72. DEBUG:stevedore.extension:found extension EntryPoint.parse('data_obfuscation = edx.analytics.tasks.export.data_obfuscation:ObfuscatedCourseDumpTask')
  73. DEBUG:stevedore.extension:found extension EntryPoint.parse('course_list = edx.analytics.tasks.insights.course_list:CourseListApiDataTask')
  74. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-user-course = edx.analytics.tasks.warehouse.load_internal_reporting_user_course:LoadUserCourseSummary')
  75. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-country = edx.analytics.tasks.warehouse.load_internal_reporting_country:LoadInternalReportingCountryToWarehouse')
  76. DEBUG:stevedore.extension:found extension EntryPoint.parse('overall_events = edx.analytics.tasks.monitor.overall_events:TotalEventsDailyTask')
  77. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-f-user-activity = edx.analytics.tasks.warehouse.load_internal_reporting_user_activity:LoadInternalReportingUserActivityToWarehouse')
  78. DEBUG:stevedore.extension:found extension EntryPoint.parse('enterprise_user = edx.analytics.tasks.enterprise.enterprise_user:ImportEnterpriseUsersIntoMysql')
  79. DEBUG:stevedore.extension:found extension EntryPoint.parse('paypal-report = edx.analytics.tasks.warehouse.financial.paypal_ftpreport:LoadPayPalCaseReportToVertica')
  80. DEBUG:stevedore.extension:found extension EntryPoint.parse('answer-dist = edx.analytics.tasks.insights.answer_dist:AnswerDistributionPerCourse')
  81. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-vertica-schema-snowflake = edx.analytics.tasks.warehouse.load_vertica_schema_to_snowflake:VerticaSchemaToSnowflakeTask')
  82. DEBUG:stevedore.extension:found extension EntryPoint.parse('student_engagement = edx.analytics.tasks.data_api.student_engagement:StudentEngagementTask')
  83. DEBUG:stevedore.extension:found extension EntryPoint.parse('insert-into-table = edx.analytics.tasks.common.mysql_load:MysqlInsertTask')
  84. DEBUG:stevedore.extension:found extension EntryPoint.parse('all_events_report = edx.analytics.tasks.monitor.total_events_report:TotalEventsReportWorkflow')
  85. DEBUG:edx.analytics.tasks.launchers.local:Loading override configuration 'override.cfg'...
  86. /var/lib/analytics-tasks/analyticstack/venv/src/luigi/luigi/parameter.py:261: UserWarning: Parameter "input_format" with value "None" is not of type string.
  87.   warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
  88. /var/lib/analytics-tasks/analyticstack/venv/src/luigi/luigi/parameter.py:261: UserWarning: Parameter "pool" with value "None" is not of type string.
  89.   warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
  90. /var/lib/analytics-tasks/analyticstack/venv/src/luigi/luigi/parameter.py:261: UserWarning: Parameter "effective_user" with value "None" is not of type string.
  91.   warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
  92. /var/lib/analytics-tasks/analyticstack/venv/src/luigi/luigi/parameter.py:261: UserWarning: Parameter "namenode_host" with value "None" is not of type string.
  93.   warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
  94. 2020-05-28 14:07:08,652 INFO 81536 [luigi-interface] worker.py:501 - Informed scheduler that task   InsertToMysqlCourseActivityTask__edx_etc_edx_ana_reports__Y_m_d_62276a52c4   has status   PENDING
  95. 2020-05-28 14:07:29,407 INFO 81536 [luigi-interface] worker.py:501 - Informed scheduler that task   CourseActivityPartitionTask__Y_m_d_2020_05_28_0_w_2_d_0_h_0_m__52dfacc26c   has status   PENDING
  96. 2020-05-28 14:07:29,408 INFO 81536 [luigi-interface] worker.py:501 - Informed scheduler that task   CalendarTableTask_2012_01_01_2020__hdfs___localhost_c96ea87e42   has status   DONE
  97. 2020-05-28 14:07:29,409 INFO 81536 [luigi-interface] worker.py:501 - Informed scheduler that task   UserActivityTableTask_2020_05_28__Y_m_d_0_w_2_d_0_h_0_m__38e39789a6   has status   DONE
  98. 2020-05-28 14:07:29,410 INFO 81536 [luigi-interface] worker.py:501 - Informed scheduler that task   CourseActivityTableTask_hdfs___localhost_c8b98d8e7b   has status   DONE
  99. 2020-05-28 14:07:29,410 INFO 81536 [luigi-interface] worker.py:501 - Informed scheduler that task   ExternalURL__edx_etc_edx_ana_6be258ccf9   has status   DONE
  100. 2020-05-28 14:07:29,410 INFO 81536 [luigi-interface] interface.py:206 - Done scheduling tasks
  101. 2020-05-28 14:07:29,411 INFO 81536 [luigi-interface] worker.py:1070 - Running Worker with 1 processes
  102. 2020-05-28 14:07:29,412 INFO 81536 [luigi-interface] worker.py:159 - [pid 81536] Worker Worker(salt=091093078, workers=1, host=pre-openedx, username=hadoop, pid=81536, sudo_user=root) running   CourseActivityPartitionTask(source=["hdfs://localhost:9000/data/"], expand_interval=0 w 2 d 0 h 0 m 0 s, pattern=[".*tracking.log.*"], date_pattern=%Y%m%d, warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/, end_date=2020-05-28, weeks=52)
  103. 2020-05-28 14:07:51,672 INFO 81536 [luigi-interface] hive.py:370 - ['hive', '-f', '/tmp/tmpD3_kXG', '--hiveconf', "mapred.job.name='CourseActivityPartitionTask__Y_m_d_2020_05_28_0_w_2_d_0_h_0_m__52dfacc26c'", '--hiveconf', 'mapred.reduce.tasks=1']
  104. 2020-05-28 14:07:51,674 INFO 81536 [luigi-interface] hadoop.py:306 - hive -f /tmp/tmpD3_kXG --hiveconf mapred.job.name='CourseActivityPartitionTask__Y_m_d_2020_05_28_0_w_2_d_0_h_0_m__52dfacc26c' --hiveconf mapred.reduce.tasks=1
  105. 2020-05-28 14:07:53,693 INFO 81536 [luigi-interface] hadoop.py:339 - SLF4J: Class path contains multiple SLF4J bindings.
  106. 2020-05-28 14:07:53,694 INFO 81536 [luigi-interface] hadoop.py:339 - SLF4J: Found binding in [jar:file:/edx/app/hadoop/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  107. 2020-05-28 14:07:53,695 INFO 81536 [luigi-interface] hadoop.py:339 - SLF4J: Found binding in [jar:file:/edx/app/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  108. 2020-05-28 14:07:53,695 INFO 81536 [luigi-interface] hadoop.py:339 - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  109. 2020-05-28 14:07:53,697 INFO 81536 [luigi-interface] hadoop.py:339 - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  110. 2020-05-28 14:07:54,286 INFO 81536 [luigi-interface] hadoop.py:339 - Logging initialized using configuration in jar:file:/edx/app/hadoop/apache-hive-2.1.1-bin/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
  111. 2020-05-28 14:07:57,861 INFO 81536 [luigi-interface] hadoop.py:339 - OK
  112. 2020-05-28 14:07:57,863 INFO 81536 [luigi-interface] hadoop.py:339 - Time taken: 0.717 seconds
  113. 2020-05-28 14:08:00,047 INFO 81536 [luigi-interface] hadoop.py:339 - WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  114. 2020-05-28 14:08:00,049 INFO 81536 [luigi-interface] hadoop.py:339 - Query ID = hadoop_20200528140757_14173601-5004-4bf9-bb63-8b9cc586f353
  115. 2020-05-28 14:08:00,050 INFO 81536 [luigi-interface] hadoop.py:339 - Total jobs = 1
  116. 2020-05-28 14:08:02,979 INFO 81536 [luigi-interface] hadoop.py:339 - SLF4J: Class path contains multiple SLF4J bindings.
  117. 2020-05-28 14:08:02,981 INFO 81536 [luigi-interface] hadoop.py:339 - SLF4J: Found binding in [jar:file:/edx/app/hadoop/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  118. 2020-05-28 14:08:02,981 INFO 81536 [luigi-interface] hadoop.py:339 - SLF4J: Found binding in [jar:file:/edx/app/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  119. 2020-05-28 14:08:02,982 INFO 81536 [luigi-interface] hadoop.py:339 - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  120. 2020-05-28 14:08:02,983 INFO 81536 [luigi-interface] hadoop.py:339 - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  121. 2020-05-28 14:08:03,928 INFO 81536 [luigi-interface] hadoop.py:339 - 2020-05-28 14:08:03    Starting to launch local task to process map join;  maximum memory = 119537664
  122. 2020-05-28 14:08:05,451 INFO 81536 [luigi-interface] hadoop.py:339 - 2020-05-28 14:08:05    Dump the side-table for tag: 0 with group count: 88 into file: file:/tmp/hadoop/6051d279-f8a3-4a20-9457-467a20d3adcb/hive_2020-05-28_14-07-57_864_4903091427798724067-1/-local-10003/HashTable-Stage-2/MapJoin-mapfile00--.hashtable
  123. 2020-05-28 14:08:05,482 INFO 81536 [luigi-interface] hadoop.py:339 - 2020-05-28 14:08:05    Uploaded 1 File to: file:/tmp/hadoop/6051d279-f8a3-4a20-9457-467a20d3adcb/hive_2020-05-28_14-07-57_864_4903091427798724067-1/-local-10003/HashTable-Stage-2/MapJoin-mapfile00--.hashtable (12559 bytes)
  124. 2020-05-28 14:08:05,483 INFO 81536 [luigi-interface] hadoop.py:339 - 2020-05-28 14:08:05    End of local task; Time Taken: 1.555 sec.
  125. 2020-05-28 14:08:06,259 INFO 81536 [luigi-interface] hadoop.py:339 - Execution completed successfully
  126. 2020-05-28 14:08:06,260 INFO 81536 [luigi-interface] hadoop.py:339 - MapredLocal task succeeded
  127. 2020-05-28 14:08:06,273 INFO 81536 [luigi-interface] hadoop.py:339 - Launching Job 1 out of 1
  128. 2020-05-28 14:08:06,275 INFO 81536 [luigi-interface] hadoop.py:339 - Number of reduce tasks not specified. Defaulting to jobconf value of: 1
  129. 2020-05-28 14:08:06,275 INFO 81536 [luigi-interface] hadoop.py:339 - In order to change the average load for a reducer (in bytes):
  130. 2020-05-28 14:08:06,276 INFO 81536 [luigi-interface] hadoop.py:339 - set hive.exec.reducers.bytes.per.reducer=<number>
  131. 2020-05-28 14:08:06,276 INFO 81536 [luigi-interface] hadoop.py:339 - In order to limit the maximum number of reducers:
  132. 2020-05-28 14:08:06,277 INFO 81536 [luigi-interface] hadoop.py:339 - set hive.exec.reducers.max=<number>
  133. 2020-05-28 14:08:06,277 INFO 81536 [luigi-interface] hadoop.py:339 - In order to set a constant number of reducers:
  134. 2020-05-28 14:08:06,277 INFO 81536 [luigi-interface] hadoop.py:339 - set mapreduce.job.reduces=<number>
  135. 2020-05-28 14:08:07,396 INFO 81536 [luigi-interface] hadoop.py:339 - Starting Job = job_1580815239440_0092, Tracking URL = http://pre-openedx:8088/proxy/application_1580815239440_0092/
  136. 2020-05-28 14:08:07,397 INFO 81536 [luigi-interface] hadoop.py:339 - Kill Command = /edx/app/hadoop/hadoop-2.7.2/bin/hadoop job  -kill job_1580815239440_0092
  137. 2020-05-28 14:08:11,601 INFO 81536 [luigi-interface] hadoop.py:339 - Hadoop job information for Stage-2: number of mappers: 0; number of reducers: 1
  138. 2020-05-28 14:08:11,662 INFO 81536 [luigi-interface] hadoop.py:339 - 2020-05-28 14:08:11,653 Stage-2 map = 0%,  reduce = 0%
  139. 2020-05-28 14:08:16,909 INFO 81536 [luigi-interface] hadoop.py:339 - 2020-05-28 14:08:16,909 Stage-2 map = 0%,  reduce = 100%, Cumulative CPU 2.18 sec
  140. 2020-05-28 14:08:17,959 INFO 81536 [luigi-interface] hadoop.py:339 - MapReduce Total cumulative CPU time: 2 seconds 180 msec
  141. 2020-05-28 14:08:17,981 INFO 81536 [luigi-interface] hadoop.py:339 - Ended Job = job_1580815239440_0092
  142. 2020-05-28 14:08:17,998 INFO 81536 [luigi-interface] hadoop.py:339 - Loading data to table default.course_activity partition (dt=2020-05-28)
  143. 2020-05-28 14:08:18,312 INFO 81536 [luigi-interface] hadoop.py:339 - MapReduce Jobs Launched:
  144. 2020-05-28 14:08:18,312 INFO 81536 [luigi-interface] hadoop.py:339 - Stage-Stage-2: Reduce: 1   Cumulative CPU: 2.18 sec   HDFS Read: 8946 HDFS Write: 63 SUCCESS
  145. 2020-05-28 14:08:18,313 INFO 81536 [luigi-interface] hadoop.py:339 - Total MapReduce CPU Time Spent: 2 seconds 180 msec
  146. 2020-05-28 14:08:18,313 INFO 81536 [luigi-interface] hadoop.py:339 - OK
  147. 2020-05-28 14:08:18,317 INFO 81536 [luigi-interface] hadoop.py:339 - Time taken: 20.45 seconds
  148. 2020-05-28 14:08:18,684 INFO 81536 [luigi-interface] worker.py:206 - [pid 81536] Worker Worker(salt=091093078, workers=1, host=pre-openedx, username=hadoop, pid=81536, sudo_user=root) done      CourseActivityPartitionTask(source=["hdfs://localhost:9000/data/"], expand_interval=0 w 2 d 0 h 0 m 0 s, pattern=[".*tracking.log.*"], date_pattern=%Y%m%d, warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/, end_date=2020-05-28, weeks=52)
  149. 2020-05-28 14:08:18,686 INFO 81536 [luigi-interface] worker.py:501 - Informed scheduler that task   CourseActivityPartitionTask__Y_m_d_2020_05_28_0_w_2_d_0_h_0_m__52dfacc26c   has status   DONE
  150. 2020-05-28 14:08:18,687 INFO 81536 [luigi-interface] worker.py:159 - [pid 81536] Worker Worker(salt=091093078, workers=1, host=pre-openedx, username=hadoop, pid=81536, sudo_user=root) running   InsertToMysqlCourseActivityTask(source=["hdfs://localhost:9000/data/"], expand_interval=0 w 2 d 0 h 0 m 0 s, pattern=[".*tracking.log.*"], date_pattern=%Y%m%d, warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/, database=reports, credentials=/edx/etc/edx-analytics-pipeline/output.json, end_date=2020-05-28, weeks=52)
  151. 2020-05-28 14:08:21,831 INFO 81536 [edx.analytics.tasks.common.mysql_load] mysql_load.py:336 - YAGO ----- column_names: course_id,interval_start,interval_end,label,count, row count: 0, value_list: []
  152. 2020-05-28 14:08:21,841 INFO 81536 [luigi-interface] worker.py:206 - [pid 81536] Worker Worker(salt=091093078, workers=1, host=pre-openedx, username=hadoop, pid=81536, sudo_user=root) done      InsertToMysqlCourseActivityTask(source=["hdfs://localhost:9000/data/"], expand_interval=0 w 2 d 0 h 0 m 0 s, pattern=[".*tracking.log.*"], date_pattern=%Y%m%d, warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/, database=reports, credentials=/edx/etc/edx-analytics-pipeline/output.json, end_date=2020-05-28, weeks=52)
  153. 2020-05-28 14:08:21,844 INFO 81536 [luigi-interface] worker.py:501 - Informed scheduler that task   InsertToMysqlCourseActivityTask__edx_etc_edx_ana_reports__Y_m_d_62276a52c4   has status   DONE
  154. 2020-05-28 14:08:21,886 INFO 81536 [luigi-interface] worker.py:401 - Worker Worker(salt=091093078, workers=1, host=pre-openedx, username=hadoop, pid=81536, sudo_user=root) was stopped. Shutting down Keep-Alive thread
  155. 2020-05-28 14:08:21,898 INFO 81536 [luigi-interface] interface.py:208 -
  156. ===== Luigi Execution Summary =====
  157.  
  158. Scheduled 6 tasks of which:
  159. * 4 present dependencies were encountered:
  160.     - 1 CalendarTableTask(warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/, interval=2012-01-01-2020-01-01)
  161.     - 1 CourseActivityTableTask(warehouse_path=hdfs://localhost:9000/edx-analytics-pipeline/warehouse/)
  162.     - 1 ExternalURL(url=/edx/etc/edx-analytics-pipeline/output.json)
  163.     - 1 UserActivityTableTask(...)
  164. * 2 ran successfully:
  165.     - 1 CourseActivityPartitionTask(...)
  166.     - 1 InsertToMysqlCourseActivityTask(...)
  167.  
  168. This progress looks :) because there were no failed tasks or missing external dependencies
  169.  
  170. ===== Luigi Execution Summary =====
  171.  
  172. Connection to localhost closed.
  173. Exiting with status = 0
  174. (pipeline)root@pre-openedx:/edx/analytics# remote-task InsertToMysqlCourseActivityTask --host localhost --user root --remote-name analyticstack --skip-setup --wait --local-scheduler --end-date 2020-05-28 --weeks 52 --n-reduce-tasks 1 --overwrite-n-days 365 --overwrite-hive
  175. Parsed arguments = Namespace(branch='release', extra_repo=None, host='localhost', job_flow_id=None, job_flow_name=None, launch_task_arguments=['InsertToMysqlCourseActivityTask', '--local-scheduler', '--end-date', '2020-05-28', '--weeks', '52', '--n-reduce-tasks', '1', '--overwrite-n-days', '365', '--overwrite-hive'], log_path=None, override_config=None, package=None, private_key=None, python_version=None, remote_name='analyticstack', repo=None, secure_config=None, secure_config_branch=None, secure_config_repo=None, shell=None, skip_setup=True, sudo_user='hadoop', user='root', vagrant_path=None, verbose=False, virtualenv_extra_args=None, wait=True, wheel_url=None, workflow_profiler=None)
  176. Running commands from path = /edx/analytics/pipeline/share/edx.analytics.tasks
  177. Remote name = analyticstack
  178. Running command = ['ssh', '-tt', '-o', 'ForwardAgent=yes', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PasswordAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', 'localhost', "sudo -Hu hadoop /bin/bash -c 'cd /var/lib/analytics-tasks/analyticstack/repo && . $HOME/.bashrc && . /var/lib/analytics-tasks/analyticstack/venv/bin/activate && launch-task InsertToMysqlCourseActivityTask --local-scheduler --end-date 2020-05-28 --weeks 52 --n-reduce-tasks 1 --overwrite-n-days 365 --overwrite-hive'"]
  179. Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
  180.  
  181.  
  182.  ================================================================
  183.  Esta maquina solo es para usuarios autorizados.
  184.  Si usted no esta autorizado, por favor, no intente acceder.
  185.  Todos los accesos son monitorizados y comprobados.
  186.  ================================================================
  187.  This machine is only for authorized users.
  188.  If you are not authorized, please, do not try to access.
  189.  All the accesses are logged and are verified.
  190.  ================================================================
  191.  
  192.  
  193. No handlers could be found for logger "luigi-interface"
  194. DEBUG:stevedore.extension:found extension EntryPoint.parse('sqoop-import = edx.analytics.tasks.common.sqoop:SqoopImportFromMysql')
  195. DEBUG:stevedore.extension:found extension EntryPoint.parse('run-vertica-sql-script = edx.analytics.tasks.warehouse.run_vertica_sql_script:RunVerticaSqlScriptTask')
  196. DEBUG:stevedore.extension:found extension EntryPoint.parse('obfuscation = edx.analytics.tasks.export.obfuscation:ObfuscatedCourseTask')
  197. DEBUG:stevedore.extension:found extension EntryPoint.parse('enrollment_validation = edx.analytics.tasks.monitor.enrollment_validation:CourseEnrollmentValidationTask')
  198. INFO:luigi-interface:Loaded ['/etc/luigi/client.cfg', 'client.cfg']
  199. DEBUG:stevedore.extension:found extension EntryPoint.parse('test-vertica-sqoop = edx.analytics.tasks.common.vertica_export:VerticaSchemaToBigQueryTask')
  200. DEBUG:stevedore.extension:found extension EntryPoint.parse('problem_response = edx.analytics.tasks.insights.problem_response:LatestProblemResponseDataTask')
  201. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-warehouse-bigquery = edx.analytics.tasks.warehouse.load_warehouse_bigquery:LoadWarehouseBigQueryTask')
  202. DEBUG:stevedore.extension:found extension EntryPoint.parse('push_to_vertica_lms_courseware_link_clicked = edx.analytics.tasks.warehouse.lms_courseware_link_clicked:PushToVerticaLMSCoursewareLinkClickedTask')
  203. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-internal-active-users = edx.analytics.tasks.warehouse.load_internal_reporting_active_users:LoadInternalReportingActiveUsersToWarehouse')
  204. DEBUG:stevedore.extension:found extension EntryPoint.parse('video = edx.analytics.tasks.insights.video:InsertToMysqlAllVideoTask')
  205. DEBUG:stevedore.extension:found extension EntryPoint.parse('ed_services_report = edx.analytics.tasks.warehouse.financial.ed_services_financial_report:BuildEdServicesReportTask')
  206. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-internal-database = edx.analytics.tasks.warehouse.load_internal_reporting_database:ImportMysqlToVerticaTask')
  207. DEBUG:snowflake.connector.ssl_wrap_socket:Injecting ssl_wrap_socket_with_ocsp
  208. DEBUG:snowflake.connector.auth:cache directory: /edx/app/hadoop/.cache/snowflake
  209. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-insights = edx.analytics.tasks.warehouse.load_warehouse_insights:LoadInsightsTableToVertica')
  210. DEBUG:stevedore.extension:found extension EntryPoint.parse('export-student-module = edx.analytics.tasks.export.database_exports:StudentModulePerCourseAfterImportWorkflow')
  211. DEBUG:stevedore.extension:found extension EntryPoint.parse('calendar = edx.analytics.tasks.insights.calendar_task:CalendarTableTask')
  212. DEBUG:stevedore.extension:found extension EntryPoint.parse('snowflake-load = edx.analytics.tasks.common.snowflake_load:SnowflakeLoadTask')
  213. DEBUG:stevedore.extension:found extension EntryPoint.parse('affiliate_window = edx.analytics.tasks.warehouse.financial.fees:LoadFeesToWarehouse')
  214. DEBUG:stevedore.extension:found extension EntryPoint.parse('orders = edx.analytics.tasks.warehouse.financial.orders_import:OrderTableTask')
  215. DEBUG:stevedore.extension:found extension EntryPoint.parse('cybersource = edx.analytics.tasks.warehouse.financial.cybersource:DailyPullFromCybersourceTask')
  216. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-user = edx.analytics.tasks.warehouse.load_internal_reporting_user:LoadInternalReportingUserToWarehouse')
  217. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-google-sheet-warehouse = edx.analytics.tasks.warehouse.load_google_sheet_to_warehouse:LoadGoogleSpreadsheetsToWarehouseWorkflow')
  218. DEBUG:stevedore.extension:found extension EntryPoint.parse('location-per-course = edx.analytics.tasks.insights.location_per_course:LastCountryOfUser')
  219. DEBUG:stevedore.extension:found extension EntryPoint.parse('payment_reconcile = edx.analytics.tasks.warehouse.financial.reconcile:ReconcileOrdersAndTransactionsTask')
  220. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-warehouse = edx.analytics.tasks.warehouse.load_warehouse:LoadWarehouseWorkflow')
  221. DEBUG:stevedore.extension:found extension EntryPoint.parse('engagement = edx.analytics.tasks.insights.module_engagement:ModuleEngagementDataTask')
  222. DEBUG:stevedore.extension:found extension EntryPoint.parse('events_obfuscation = edx.analytics.tasks.export.events_obfuscation:ObfuscateCourseEventsTask')
  223. DEBUG:stevedore.extension:found extension EntryPoint.parse('dump-student-module = edx.analytics.tasks.export.database_exports:StudentModulePerCourseTask')
  224. DEBUG:stevedore.extension:found extension EntryPoint.parse('export-events-by-course = edx.analytics.tasks.export.event_exports_by_course:EventExportByCourseTask')
  225. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-ga-permissions = edx.analytics.tasks.warehouse.load_ga_permissions:LoadGoogleAnalyticsPermissionsWorkflow')
  226. DEBUG:stevedore.extension:found extension EntryPoint.parse('noop = edx.analytics.tasks.monitor.performance:ParseEventLogPerformanceTask')
  227. DEBUG:stevedore.extension:found extension EntryPoint.parse('course_blocks = edx.analytics.tasks.insights.course_blocks:CourseBlocksApiDataTask')
  228. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-events = edx.analytics.tasks.warehouse.load_internal_reporting_events:TrackingEventRecordDataTask')
  229. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-certificates = edx.analytics.tasks.warehouse.load_internal_reporting_certificates:LoadInternalReportingCertificatesToWarehouse')
  230. DEBUG:stevedore.extension:found extension EntryPoint.parse('user-activity = edx.analytics.tasks.insights.user_activity:InsertToMysqlCourseActivityTask')
  231. DEBUG:stevedore.extension:found extension EntryPoint.parse('tags-dist = edx.analytics.tasks.insights.tags_dist:TagsDistributionPerCourse')
  232. DEBUG:stevedore.extension:found extension EntryPoint.parse('bigquery-load = edx.analytics.tasks.common.bigquery_load:BigQueryLoadTask')
  233. DEBUG:stevedore.extension:found extension EntryPoint.parse('run-vertica-sql-scripts = edx.analytics.tasks.warehouse.run_vertica_sql_scripts:RunVerticaSqlScriptTask')
  234. DEBUG:stevedore.extension:found extension EntryPoint.parse('paypal = edx.analytics.tasks.warehouse.financial.paypal:PaypalTransactionsByDayTask')
  235. DEBUG:stevedore.extension:found extension EntryPoint.parse('grade-dist = edx.analytics.tasks.data_api.studentmodule_dist:GradeDistFromSqoopToMySQLWorkflow')
  236. DEBUG:stevedore.extension:found extension EntryPoint.parse('database-import = edx.analytics.tasks.insights.database_imports:ImportAllDatabaseTablesTask')
  237. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-course-catalog = edx.analytics.tasks.warehouse.load_internal_reporting_course_catalog:PullDiscoveryCoursesAPIData')
  238. DEBUG:stevedore.extension:found extension EntryPoint.parse('enrollments = edx.analytics.tasks.insights.enrollments:ImportEnrollmentsIntoMysql')
  239. DEBUG:stevedore.extension:found extension EntryPoint.parse('event-type-dist = edx.analytics.tasks.warehouse.event_type_dist:PushToVerticaEventTypeDistributionTask')
  240. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-internal-course-structure = edx.analytics.tasks.warehouse.load_internal_reporting_course_structure:LoadCourseBlockRecordToVertica')
  241. DEBUG:stevedore.extension:found extension EntryPoint.parse('enterprise_enrollments = edx.analytics.tasks.enterprise.enterprise_enrollments:ImportEnterpriseEnrollmentsIntoMysql')
  242. DEBUG:stevedore.extension:found extension EntryPoint.parse('export-events = edx.analytics.tasks.export.event_exports:EventExportTask')
  243. DEBUG:stevedore.extension:found extension EntryPoint.parse('financial_reports = edx.analytics.tasks.warehouse.financial.finance_reports:BuildFinancialReportsTask')
  244. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-warehouse-snowflake = edx.analytics.tasks.warehouse.load_warehouse_snowflake:LoadWarehouseSnowflakeTask')
  245. DEBUG:stevedore.extension:found extension EntryPoint.parse('data_obfuscation = edx.analytics.tasks.export.data_obfuscation:ObfuscatedCourseDumpTask')
  246. DEBUG:stevedore.extension:found extension EntryPoint.parse('course_list = edx.analytics.tasks.insights.course_list:CourseListApiDataTask')
  247. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-user-course = edx.analytics.tasks.warehouse.load_internal_reporting_user_course:LoadUserCourseSummary')
  248. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-d-country = edx.analytics.tasks.warehouse.load_internal_reporting_country:LoadInternalReportingCountryToWarehouse')
  249. DEBUG:stevedore.extension:found extension EntryPoint.parse('overall_events = edx.analytics.tasks.monitor.overall_events:TotalEventsDailyTask')
  250. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-f-user-activity = edx.analytics.tasks.warehouse.load_internal_reporting_user_activity:LoadInternalReportingUserActivityToWarehouse')
  251. DEBUG:stevedore.extension:found extension EntryPoint.parse('enterprise_user = edx.analytics.tasks.enterprise.enterprise_user:ImportEnterpriseUsersIntoMysql')
  252. DEBUG:stevedore.extension:found extension EntryPoint.parse('paypal-report = edx.analytics.tasks.warehouse.financial.paypal_ftpreport:LoadPayPalCaseReportToVertica')
  253. DEBUG:stevedore.extension:found extension EntryPoint.parse('answer-dist = edx.analytics.tasks.insights.answer_dist:AnswerDistributionPerCourse')
  254. DEBUG:stevedore.extension:found extension EntryPoint.parse('load-vertica-schema-snowflake = edx.analytics.tasks.warehouse.load_vertica_schema_to_snowflake:VerticaSchemaToSnowflakeTask')
  255. DEBUG:stevedore.extension:found extension EntryPoint.parse('student_engagement = edx.analytics.tasks.data_api.student_engagement:StudentEngagementTask')
  256. DEBUG:stevedore.extension:found extension EntryPoint.parse('insert-into-table = edx.analytics.tasks.common.mysql_load:MysqlInsertTask')
  257. DEBUG:stevedore.extension:found extension EntryPoint.parse('all_events_report = edx.analytics.tasks.monitor.total_events_report:TotalEventsReportWorkflow')
  258. DEBUG:edx.analytics.tasks.launchers.local:Loading override configuration 'override.cfg'...
  259. /var/lib/analytics-tasks/analyticstack/venv/src/luigi/luigi/parameter.py:261: UserWarning: Parameter "input_format" with value "None" is not of type string.
  260.   warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
  261. /var/lib/analytics-tasks/analyticstack/venv/src/luigi/luigi/parameter.py:261: UserWarning: Parameter "pool" with value "None" is not of type string.
  262.   warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
  263. /var/lib/analytics-tasks/analyticstack/venv/src/luigi/luigi/parameter.py:261: UserWarning: Parameter "effective_user" with value "None" is not of type string.
  264.   warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
  265. /var/lib/analytics-tasks/analyticstack/venv/src/luigi/luigi/parameter.py:261: UserWarning: Parameter "namenode_host" with value "None" is not of type string.
  266.   warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
  267. 2020-05-28 14:13:56,677 INFO 83000 [luigi-interface] worker.py:501 - Informed scheduler that task   InsertToMysqlCourseActivityTask__edx_etc_edx_ana_reports__Y_m_d_62276a52c4   has status   DONE
  268. 2020-05-28 14:13:56,678 INFO 83000 [luigi-interface] interface.py:206 - Done scheduling tasks
  269. 2020-05-28 14:13:56,678 INFO 83000 [luigi-interface] worker.py:1070 - Running Worker with 1 processes
  270. 2020-05-28 14:13:56,682 INFO 83000 [luigi-interface] worker.py:401 - Worker Worker(salt=599835503, workers=1, host=pre-openedx, username=hadoop, pid=83000, sudo_user=root) was stopped. Shutting down Keep-Alive thread
  271. 2020-05-28 14:13:56,683 INFO 83000 [luigi-interface] interface.py:208 -
  272. ===== Luigi Execution Summary =====
  273.  
  274. Scheduled 1 tasks of which:
  275. * 1 present dependencies were encountered:
  276.     - 1 InsertToMysqlCourseActivityTask(...)
  277.  
  278. Did not run any tasks
  279. This progress looks :) because there were no failed tasks or missing external dependencies
  280.  
  281. ===== Luigi Execution Summary =====
  282.  
  283. Connection to localhost closed.
  284. Exiting with status = 0
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!