2012-09-27 09:22:05,771 Loading VCSRegistry
2012-09-27 09:22:15,993 Loading triggers
2012-09-27 09:22:16,203 0 triggers has been loaded to 0 classes
2012-09-27 09:22:17,178 [fm.correlator] Running scheduler
2012-09-27 09:22:17,209 UNHANDLED EXCEPTION (2012-09-27 09:22:17.180006)
Working directory: /opt/noc
<type 'exceptions.KeyError'>
'schedule'
START OF TRACEBACK
------------------------------------------------------------------------
File: /opt/noc/lib/scheduler/scheduler.py (Line: 328)
Function: run_pending
  321                     # Invalid job class. Park job to FAIL state
  322                     self.error("Invalid job class: %s" % jcls)
  323                     self.set_job_status(job_data[self.ATTR_CLASS],
  324                         job_data[self.ATTR_KEY], self.S_FAIL)
  325                     continue
  326                 job = jcls(self,
  327                     job_data[self.ATTR_KEY], job_data[self.ATTR_DATA],
  328 ==>                 job_data[self.ATTR_SCHEDULE]
  329                 )
  330                 self.run_job(job)
  331                 n += 1
  332             return n
  333
  334         def run(self):
Variables:
                jcls = <class 'noc.fm.correlator.jobs.performance_report.PerformanceReportJob'>
           throttled = set([])
                self = <noc.fm.correlator.scheduler.CorrelatorScheduler object at 0x41e9f10>
            job_data = {u'runs': 40457, u'last': datetime.datetime(2012, 9, 24, 11, 26, 0, 709000),
u'log': [], u'jcls': u'performance_report', u'ldur': 0.0004208087921142578, u'ts': datetime.datetime
(2012, 9, 24, 11, 27), u'tb': None, u's': u'W', u'ls': u'S', u'key': u'report', u'_id': ObjectId('50
38a18e4a5e6844e97eade2'), u'data': {u'_noc': {u'interval': 60, u'offset': 0}}}
                   n = 0
                   q = {'s': 'W', 'ts': {'$lte': datetime.datetime(2012, 9, 27, 9, 22, 17, 178983)}}
------------------------------------------------------------------------
File: /opt/noc/lib/scheduler/scheduler.py (Line: 345)
Function: run
  338                 self.collection.update({
  339                     self.ATTR_STATUS: self.S_RUN
  340                 }, {
  341                     "$set": {self.ATTR_STATUS: self.S_WAIT}

------------------------------------------------------------------------
File: /opt/noc/fm/correlator/daemon.py (Line: 384)
Function: run
  377                 self.stat_success_count += 1
  378
  379         def run(self):
  380             """
  381             Main daemon loop
  382             """
  383             self.reset_stats()
  384 ==>         self.scheduler.run()

Variables:
                self = <noc.fm.correlator.daemon.Correlator object at 0x37fc710>

------------------------------------------------------------------------
File: /opt/noc/lib/daemon.py (Line: 364)
Function: guarded_run
  357
  358         def guarded_run(self):
  359             """
  360             Run daemon and catch common exceptions
  361             :return:
  362             """
  363             try:
  364 ==>             self.run()
  365             except KeyboardInterrupt:
  366                 pass
  367             except MemoryError:
  368                 logging.error("Out of memory. Exiting.")
  369             except SystemExit:
  370                 logging.info("Exiting")
Variables:
                self = <noc.fm.correlator.daemon.Correlator object at 0x37fc710>
------------------------------------------------------------------------
END OF TRACEBACK
2012-09-27 09:22:17,210 Removing pidfile: /var/run/noc/noc-correlator.pid
2012-09-27 09:22:17,210 STOP