Discover

2017-07-04 10:34:45.002280  [box] Starting at e7cab67f-f689-48f9-84a7-52ec2bffc0d1[10.0.4.111:19006] (Lag 167.01ms)
2017-07-04 10:34:45.008482 [discovery|box|DeviceName|profile]  Checking profile accordance
2017-07-04 10:34:45.008738 [discovery|box|DeviceName|profile]  Compiling "Profile Check rules"
2017-07-04 10:34:45.106206 [discovery|box|DeviceName|profile]  Use snmp_v2c_get for request
2017-07-04 10:34:45.110898 [discovery|box|DeviceName|profile]  Set path: None
2017-07-04 10:34:45.185049 [discovery|box|DeviceName|profile]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.112488)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff19ec1cad0>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff19ec1cad0>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff19f54dfa0>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff1a35f23d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff19ec1cad0>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = u'activator-default'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff19ec1cad0>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff1a35f23d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff19ec1cad0>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff1a360b7d0>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body =
'{"params":["10.12.0.8","communityName","1.3.6.1.2.1.1.2.0"],"method":"snmp_v2c_get","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f702bd0>
                args = (u'10.12.0.8', u'communityName', '1.3.6.1.2.1.1.2.0')
                  t0 = 1499153685.109118
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg =
{'id': 0,
 'method': 'snmp_v2c_get',
 'params': [u'10.12.0.8', u'communityName', '1.3.6.1.2.1.1.2.0']}
              method = 'snmp_v2c_get'
           make_call = <function make_call at 0x7ff19f293488>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff1a360b7d0>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff1a360bb50>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (u'10.12.0.8', u'communityName', '1.3.6.1.2.1.1.2.0')
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f702bd0>
              kwargs = {}
                item = 'snmp_v2c_get'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19f564320>)]
                  ev = <threading._Event object at 0x7ff19f6ee250>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f702bd0>
                args = (u'10.12.0.8', u'communityName', '1.3.6.1.2.1.1.2.0')
                item = 'snmp_v2c_get'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19f564320>)]
                  ev = <threading._Event object at 0x7ff19f6ee250>
               _call = <function _call at 0x7ff19f2930c8>
------------------------------------------------------------------------
File: services/discovery/jobs/box/profile.py (Line: 178)
Function: check_snmp_v2c_get
  171                 return open_sync_rpc(
  172                     "activator",
  173                     pool=self.object.pool.name,
  174                     calling_service="discovery"
  175                 ).__getattr__(self.snmp_version_def)(
  176                     self.object.address,
  177                     snmp_ro,
  178 ==>                 param
  179                 )
  180             except RPCError as e:
  181                 self.logger.error("RPC Error: %s", e)
  182                 return None
  183
  184         def check_http_get(self, param):
Variables:
                self =
<noc.services.discovery.jobs.box.profile.ProfileCheck object at 0x7ff1a256fc10>
             snmp_ro = u'communityName'
               param = '1.3.6.1.2.1.1.2.0'
                caps =
{u'Cisco | IOS | Syntax | IP SLA': 0,
 u'DB | Interfaces': 28,
 u'Network | CDP': True,
 u'Network | IPv6': True,
 u'Network | LACP': True,
 u'Network | LLDP': True,
 u'Network | STP': True,
 u'SNMP': True,
 u'SNMP | Bulk': True,
 u'SNMP | IF-MIB': True,
 u'SNMP | IF-MIB | HC': True,
 u'SNMP | v1': True,
 u'SNMP | v2c': True,
 u'SNMP | v3': False}
------------------------------------------------------------------------
File: services/discovery/jobs/box/profile.py (Line: 138)
Function: do_check
  131                 self.logger.debug("Using cached value")
  132                 return self.result_cache[method, param]
  133             h = getattr(self, "check_%s" % method, None)
  134             if not h:
  135                 self.logger.error("Invalid check method '%s'. Ignoring",
  136                                   method)
  137                 return None
  138 ==>         result = h(param)
  139             self.result_cache[method, param] = result
  140             return result
  141
  142         def check_snmp_v2c_get(self, param):
  143             """
  144             Perform SNMP v2c GET. Param is OID or symbolic name
Variables:
                   h =
<bound method ProfileCheck.check_snmp_v2c_get of <noc.services.discovery.jobs.box.profile.ProfileCheck object at 0x7ff1a256fc10>>
                self =
<noc.services.discovery.jobs.box.profile.ProfileCheck object at 0x7ff1a256fc10>
              method = u'snmp_v2c_get'
               param = u'SNMPv2-MIB::sysObjectID.0'
------------------------------------------------------------------------
File: services/discovery/jobs/box/profile.py (Line: 61)
Function: get_profile
   54             snmp_result = ""
   55             http_result = ""
   56             message = "Cannot detect profile"
   57             fatal = False
   58             for ruleset in self.get_rules():
   59                 for (method, param), actions in ruleset:
   60                     try:
   61 ==>                     result = self.do_check(method, param)
   62                         if not result:
   63                             continue
   64                         if "snmp" in method:
   65                             snmp_result = result
   66                         if "http" in method:
   67                             http_result = result
Variables:
                self =
<noc.services.discovery.jobs.box.profile.ProfileCheck object at 0x7ff1a256fc10>
               param = u'SNMPv2-MIB::sysObjectID.0'
         http_result = ''
             ruleset =
[((u'snmp_v2c_get', u'SNMPv2-MIB::sysObjectID.0'),
  [(u'eq',
    u'1.3.6.1.4.1.890.1.5.11.13',
    u'match',
    u'Zyxel.MSAN',
    u'Zyxel | MSAN | IES1248-51 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.2011.10.1.232',
    u'match',
    u'Huawei.VRP',
    u'Huawei | VRP | S5628F-HI sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.890.1.5.8.30',
    u'match',
    u'Zyxel.ZyNOS',
    u'Zyxel | ZyNOS | ES-2226 sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.76.12',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DGS-1210-10P sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.76.15',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DGS-1210-28 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.25506.1.33',
    u'match',
    u'H3C.VRP',
    u'H3C | VRP | S3600-52P-SI sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.6486.800.1.1.2.1.7.1.10',
    u'match',
    u'Alcatel.AOS',
    u'Alcatel | AOS | OS6850-U24X'),
   (u'eq',
    u'1.3.6.1.4.1.2011.2.123',
    u'match',
    u'Huawei.MA5600T',
    u'Huawei | MA | MA5603T sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.1795.1.14.9.14.5.3',
    u'match',
    u'Zhone.Bitstorm',
    u'Zhone | Bitstorm | 8820-A2-xxx sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.43.1.8.42',
    u'match',
    u'3Com.SuperStack3',
    u'3Com | SuperStack3 | 3250 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.3955.6.9.224.1',
    u'match',
    u'Linksys.SPS2xx',
    u'Lynksis | SPS2xx | SPS224G4 sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.75.13',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DES-1210-08P sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.2011.10.1.43',
    u'match',
    u'Huawei.VRP',
    u'Huawei | VRP | S5624P sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.34300.1.6',
    u'match',
    u'Eltex.DSLAM',
    u'Eltex | DSLAM | MXA24 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.259.10.1.22.101',
    u'match',
    u'EdgeCore.ES',
    u'EdgeCore | ES | ES3528MV2 sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.2011.2.80.1',
    u'match',
    u'Huawei.VRP3',
    u'Huawei | MA | MA5605 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.890.1.5.5.2',
    u'match',
    u'Zyxel.MSAN',
    u'Zyxel | MSAN | IES-1000 sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.76.14',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DGS-1210-20 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.835457.3.4',
    u'match',
    u'Alcatel.TIMOS',
    u'Alcatel | TIMOS | 7750 SR-7'),
   (u'eq',
    u'1.3.6.1.4.1.43.1.8.41',
    u'match',
    u'3Com.SuperStack3',
    u'3Com | SuperStack3 | 3226 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.2011.2.23.96',
    u'match',
    u'Huawei.VRP',
    u'Huawei | VRP | S5328C-EI-24S sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.31926',
    u'match',
    u'Siklu.EH',
    u'Siklu | EH | EH-1200L.v700 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.2011.2.78',
    u'match',
    u'Huawei.MA5600T',
    u'Huawei | HONET | UA5000 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.6486.800.1.1.2.1.7.1.48',
    u'match',
    u'Alcatel.AOS',
    u'Alcatel | AOS | OS6850E-U24X'),
   (u'eq',
    u'1.3.6.1.4.1.2011.2.6.6.1',
    u'match',
    u'Huawei.MA5300',
    u'Huawei | MA | MA5300 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.35265.1.28',
    u'match',
    u'Eltex.DSLAM',
    u'Eltex | DSLAM | MXA32 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.3320.1.263.0',
    u'match',
    u'BDCOM.xPON',
    u'BDCOM | xPON | P3608 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.6527.1.15.1',
    u'match',
    u'Alcatel.TIMOS',
    u'Alcatel | TIMOS | 7950 XRS'),
   (u'eq',
    u'1.3.6.1.4.1.14988.1',
    u'match',
    u'MikroTik.RouterOS',
    u'MikroTik | RouterOS | sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.231.7.1.2.2.1.101.1.1',
    u'match',
    u'NSN.hiX56xx',
    u'NAG | NSN | hiX5622G2002GE sysObjectID.0'),
   (u'eq',
    u'1.3.6.1.4.1.890.1.5.8.26',
    u'match',
    u'Zyxel.ZyNOS',
    u'Zyxel | ZyNOS | ES-3124-4F sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.75.5',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DES-1210-28 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.4874.1.1.1.6.1',
    u'match',
    u'Juniper.JUNOSe',
    u'Juniper | JUNOSe | E320 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.193.81.1.1.3',
    u'match',
    u'Ericsson.SEOS',
    u'Ericsson | MINI-LINK | CN510 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.259.10.1.27.102',
    u'match',
    u'EdgeCore.ES',
    u'EdgeCore | ES | ECS3510-52T sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.40418.7.2',
    u'match',
    u'NAG.SNR',
    u'NAG | SNR | SNR-S2940 sysObjectID.0'),
   (u'eq',
    u'1.3.6.1.4.1.25506.1.34',
    u'match',
    u'H3C.VRP',
    u'H3C | VRP | S3600-28TP-SI sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.75.19.1',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DES-1210-28P V4 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.2011.2.6.5.21',
    u'match',
    u'Huawei.VRP3',
    u'Huawei | MA | MA5105 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.40248.1',
    u'match',
    u'Vitesse.VSC',
    u'Vitesse | VSC | VSC7460 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.890.1.5.11.9',
    u'match',
    u'Zyxel.MSAN',
    u'Zyxel | ZyNOS | IES1248-71 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.2011.10.1.12',
    u'match',
    u'Huawei.VRP',
    u'Huawei | VRP | S3928TP-SI sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.4249.1.114.1',
    u'match',
    u'Nateks.FlexGain',
    u'Nateks | FlexGain | FG-ACE120 sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.75.6',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DES-1210-28P sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.75.7',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DES-1210-52 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.890.1.5.13.8',
    u'match',
    u'Zyxel.MSAN',
    u'Zyxel | ZyNOS | IES-6000 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.27514.1.1.1.48',
    u'match',
    u'Qtech.QSW2800',
    u'Qtech | QSW | QSW-2800-10T-AC sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.38838.1.10',
    u'match',
    u'Angtel.Topaz',
    u'Angtel | Topaz | Topaz-2O-16E'),
   (u'eq',
    u'.1.3.6.1.4.1.30982.1.1',
    u'match',
    u'TFortis.PSW',
    u'TFortis | PSW | PSW-2G6F+'),
   (u'eq',
    u'1.3.6.1.4.1.27514.1.1.1.203',
    u'match',
    u'Qtech.QSW',
    u'Qtech | QSW | QSW-3400-28T-AC sysObjectID 2'),
   (u'eq',
    u'1.3.6.1.4.1.890.1.5.8.16',
    u'match',
    u'Zyxel.ZyNOS',
    u'Zyxel | ZyNOS | ES-2024A sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.35265.1.22',
    u'match',
    u'Eltex.LTP',
    u'Eltex | LTP | LTP sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.76.10',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DGS-1210-24 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.3320.1.228.0',
    u'match',
    u'BDCOM.xPON',
    u'BDCOM | xPON | P3310B sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.76.16',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DGS-1210-28P sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.1795.1.14.17.1.3',
    u'match',
    u'Zhone.Bitstorm',
    u'Zhone | Bitstorm | 4229-A1-520 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.637.61.1',
    u'match',
    u'Alcatel.7302',
    u'Alcatel | 7302 | 7302 sysObjectID.0'),
   (u'eq',
    u'1.3.6.1.4.1.27514.1.1.1.49',
    u'match',
    u'Qtech.QSW2800',
    u'Qtech | QSW | QSW-2800-28T sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.27514.1.1.1.221',
    u'match',
    u'Qtech.QSW',
    u'Qtech | QSW | QSW-3400-28T-AC sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.21696',
    u'match',
    u'Generic.Host',
    u'KeyMile | MileGate | 2510 sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.94.5',
    u'match',
    u'DLink.DGS3100',
    u'DLink | DGS3100 | DGS-3100-24TG sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.3320.1.294.0',
    u'match',
    u'BDCOM.xPON',
    u'BDCOM | xPON | P3310C sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.3955.7.2.2000.1',
    u'match',
    u'Alstec.ALS',
    u'Alstec | ALS | ALS-62000'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.76.11',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DGS-1210-48 sysObjectID'),
   (u'eq',
    u'.1.3.6.1.4.1.1332.1.1.1',
    u'match',
    u'Iskratel.VOIP',
    u'Iskratel | VOIP | ISKRATEL VOIP-1 sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.76.9',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DGS-1210-16 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.4874.1.1.1.6.2',
    u'match',
    u'Juniper.JUNOSe',
    u'Juniper | JUNOSe | E120 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.25506.1.37',
    u'match',
    u'H3C.VRP',
    u'H3C | VRP | S3600-28P-EI sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.8886.6.140',
    u'match',
    u'Raisecom.ROS',
    u'Raisecom | ROS | ISCOM2128EA-MA-AC sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.890.1.5.8.12',
    u'match',
    u'Zyxel.ZyNOS',
    u'Zyxel | ZyNOS | ES-3124 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.890.1.5.11.11',
    u'match',
    u'Zyxel.MSAN',
    u'Zyxel | ZyNOS | IES-612 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.27514.1.1.2.59',
    u'match',
    u'Qtech.QSW2800',
    u'Qtech | QSW | QSW-8200-28F-AC-DC sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.2011.2.248',
    u'match',
    u'Huawei.MA5600T',
    u'Huawei | MA | MA5608T sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.890.1.5.13.7',
    u'match',
    u'Zyxel.MSAN',
    u'Zyxel | ZyNOS | IES-5005 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.27514.1.3.26.7',
    u'match',
    u'Qtech.QSW',
    u'Qtech | QSW | QSW-2910-28F sysObjectID'),
   (u'contains',
    u'.1.3.6.1.4.1.1332.1.24.3',
    u'match',
    u'Iskratel.ESCOM',
    u'Iskratel | ESCOM | SI3000 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.30803',
    u'match',
    u'Vyatta.Vyatta',
    u'Vyatta | Vyatta | sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.2011.2.23.92',
    u'match',
    u'Huawei.VRP',
    u'Huawei | VRP | S2326TP-EI sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.27514.1.1.1.228',
    u'match',
    u'Qtech.QSW2800',
    u'Qtech | QSW | QSW-3450-28T-AC sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.35265.1.21.202.1',
    u'match',
    u'Eltex.LTE',
    u'Eltex | LTE | LTE sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.890.1.5.11.6',
    u'match',
    u'Zyxel.MSAN',
    u'Zyxel | ZyNOS | IES-1000 AAM1008-61 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.2011.10.1.200',
    u'match',
    u'Huawei.VRP',
    u'Huawei | VRP | S2403TP-EA sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.27514.1.1.1.281',
    u'match',
    u'Qtech.QSW',
    u'Qtech | QSW | QSW-3470-28T-AC sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.259.6.10.94',
    u'match',
    u'EdgeCore.ES',
    u'EdgeCore | ES | ES3528M sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.835457.6.3',
    u'match',
    u'Alcatel.TIMOS',
    u'Alcatel | TIMOS | 7450 ESS-7'),
   (u'eq',
    u'1.3.6.1.4.1.4413',
    u'match',
    u'Alstec.24xx',
    u'Alstec | ALS24300 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.2011.2.169',
    u'match',
    u'Huawei.MA5600T',
    u'Huawei | MA | MA5616 sysObjectID'),
   (u'contains',
    u'1.3.6.1.4.1.171.10.76.17',
    u'match',
    u'DLink.DxS_Smart',
    u'DLink | DxS_Smart | DGS-1210-52 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.27514.1.1.1.51',
    u'match',
    u'Qtech.QSW2800',
    u'Qtech | QSW | QSW-2800-10T-DC sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.3955.6.1.2016.1',
    u'match',
    u'Linksys.SRW',
    u'Linksys | SRW | SRW2016 sysObjectID'),
   (u'eq',
    u'1.3.6.1.4.1.11.2.3.7.11.63',
    u'match',
    u'HP.ProCurve',
    u'HP | ProCurve | 2810-24G sysObjectID')]),
 ((u'snmp_v2c_get', u'SNMPv2-MIB::sysDescr.0'),
  [(u'contains',
    u'ALS24100LVT',
    u'match',
    u'Alstec.24xx',
    u'Alstec | ALS24100 sysDescr.0')])]
         snmp_result = ''
             message = 'Cannot detect profile'
               fatal = False
              method = u'snmp_v2c_get'
             actions =
[(u'eq',
  u'1.3.6.1.4.1.890.1.5.11.13',
  u'match',
  u'Zyxel.MSAN',
  u'Zyxel | MSAN | IES1248-51 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.2011.10.1.232',
  u'match',
  u'Huawei.VRP',
  u'Huawei | VRP | S5628F-HI sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.890.1.5.8.30',
  u'match',
  u'Zyxel.ZyNOS',
  u'Zyxel | ZyNOS | ES-2226 sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.76.12',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DGS-1210-10P sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.76.15',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DGS-1210-28 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.25506.1.33',
  u'match',
  u'H3C.VRP',
  u'H3C | VRP | S3600-52P-SI sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.6486.800.1.1.2.1.7.1.10',
  u'match',
  u'Alcatel.AOS',
  u'Alcatel | AOS | OS6850-U24X'),
 (u'eq',
  u'1.3.6.1.4.1.2011.2.123',
  u'match',
  u'Huawei.MA5600T',
  u'Huawei | MA | MA5603T sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.1795.1.14.9.14.5.3',
  u'match',
  u'Zhone.Bitstorm',
  u'Zhone | Bitstorm | 8820-A2-xxx sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.43.1.8.42',
  u'match',
  u'3Com.SuperStack3',
  u'3Com | SuperStack3 | 3250 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.3955.6.9.224.1',
  u'match',
  u'Linksys.SPS2xx',
  u'Lynksis | SPS2xx | SPS224G4 sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.75.13',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DES-1210-08P sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.2011.10.1.43',
  u'match',
  u'Huawei.VRP',
  u'Huawei | VRP | S5624P sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.34300.1.6',
  u'match',
  u'Eltex.DSLAM',
  u'Eltex | DSLAM | MXA24 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.259.10.1.22.101',
  u'match',
  u'EdgeCore.ES',
  u'EdgeCore | ES | ES3528MV2 sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.2011.2.80.1',
  u'match',
  u'Huawei.VRP3',
  u'Huawei | MA | MA5605 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.890.1.5.5.2',
  u'match',
  u'Zyxel.MSAN',
  u'Zyxel | MSAN | IES-1000 sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.76.14',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DGS-1210-20 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.835457.3.4',
  u'match',
  u'Alcatel.TIMOS',
  u'Alcatel | TIMOS | 7750 SR-7'),
 (u'eq',
  u'1.3.6.1.4.1.43.1.8.41',
  u'match',
  u'3Com.SuperStack3',
  u'3Com | SuperStack3 | 3226 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.2011.2.23.96',
  u'match',
  u'Huawei.VRP',
  u'Huawei | VRP | S5328C-EI-24S sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.31926',
  u'match',
  u'Siklu.EH',
  u'Siklu | EH | EH-1200L.v700 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.2011.2.78',
  u'match',
  u'Huawei.MA5600T',
  u'Huawei | HONET | UA5000 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.6486.800.1.1.2.1.7.1.48',
  u'match',
  u'Alcatel.AOS',
  u'Alcatel | AOS | OS6850E-U24X'),
 (u'eq',
  u'1.3.6.1.4.1.2011.2.6.6.1',
  u'match',
  u'Huawei.MA5300',
  u'Huawei | MA | MA5300 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.35265.1.28',
  u'match',
  u'Eltex.DSLAM',
  u'Eltex | DSLAM | MXA32 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.3320.1.263.0',
  u'match',
  u'BDCOM.xPON',
  u'BDCOM | xPON | P3608 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.6527.1.15.1',
  u'match',
  u'Alcatel.TIMOS',
  u'Alcatel | TIMOS | 7950 XRS'),
 (u'eq',
  u'1.3.6.1.4.1.14988.1',
  u'match',
  u'MikroTik.RouterOS',
  u'MikroTik | RouterOS | sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.231.7.1.2.2.1.101.1.1',
  u'match',
  u'NSN.hiX56xx',
  u'NAG | NSN | hiX5622G2002GE sysObjectID.0'),
 (u'eq',
  u'1.3.6.1.4.1.890.1.5.8.26',
  u'match',
  u'Zyxel.ZyNOS',
  u'Zyxel | ZyNOS | ES-3124-4F sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.75.5',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DES-1210-28 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.4874.1.1.1.6.1',
  u'match',
  u'Juniper.JUNOSe',
  u'Juniper | JUNOSe | E320 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.193.81.1.1.3',
  u'match',
  u'Ericsson.SEOS',
  u'Ericsson | MINI-LINK | CN510 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.259.10.1.27.102',
  u'match',
  u'EdgeCore.ES',
  u'EdgeCore | ES | ECS3510-52T sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.40418.7.2',
  u'match',
  u'NAG.SNR',
  u'NAG | SNR | SNR-S2940 sysObjectID.0'),
 (u'eq',
  u'1.3.6.1.4.1.25506.1.34',
  u'match',
  u'H3C.VRP',
  u'H3C | VRP | S3600-28TP-SI sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.75.19.1',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DES-1210-28P V4 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.2011.2.6.5.21',
  u'match',
  u'Huawei.VRP3',
  u'Huawei | MA | MA5105 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.40248.1',
  u'match',
  u'Vitesse.VSC',
  u'Vitesse | VSC | VSC7460 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.890.1.5.11.9',
  u'match',
  u'Zyxel.MSAN',
  u'Zyxel | ZyNOS | IES1248-71 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.2011.10.1.12',
  u'match',
  u'Huawei.VRP',
  u'Huawei | VRP | S3928TP-SI sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.4249.1.114.1',
  u'match',
  u'Nateks.FlexGain',
  u'Nateks | FlexGain | FG-ACE120 sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.75.6',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DES-1210-28P sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.75.7',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DES-1210-52 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.890.1.5.13.8',
  u'match',
  u'Zyxel.MSAN',
  u'Zyxel | ZyNOS | IES-6000 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.27514.1.1.1.48',
  u'match',
  u'Qtech.QSW2800',
  u'Qtech | QSW | QSW-2800-10T-AC sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.38838.1.10',
  u'match',
  u'Angtel.Topaz',
  u'Angtel | Topaz | Topaz-2O-16E'),
 (u'eq',
  u'.1.3.6.1.4.1.30982.1.1',
  u'match',
  u'TFortis.PSW',
  u'TFortis | PSW | PSW-2G6F+'),
 (u'eq',
  u'1.3.6.1.4.1.27514.1.1.1.203',
  u'match',
  u'Qtech.QSW',
  u'Qtech | QSW | QSW-3400-28T-AC sysObjectID 2'),
 (u'eq',
  u'1.3.6.1.4.1.890.1.5.8.16',
  u'match',
  u'Zyxel.ZyNOS',
  u'Zyxel | ZyNOS | ES-2024A sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.35265.1.22',
  u'match',
  u'Eltex.LTP',
  u'Eltex | LTP | LTP sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.76.10',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DGS-1210-24 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.3320.1.228.0',
  u'match',
  u'BDCOM.xPON',
  u'BDCOM | xPON | P3310B sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.76.16',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DGS-1210-28P sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.1795.1.14.17.1.3',
  u'match',
  u'Zhone.Bitstorm',
  u'Zhone | Bitstorm | 4229-A1-520 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.637.61.1',
  u'match',
  u'Alcatel.7302',
  u'Alcatel | 7302 | 7302 sysObjectID.0'),
 (u'eq',
  u'1.3.6.1.4.1.27514.1.1.1.49',
  u'match',
  u'Qtech.QSW2800',
  u'Qtech | QSW | QSW-2800-28T sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.27514.1.1.1.221',
  u'match',
  u'Qtech.QSW',
  u'Qtech | QSW | QSW-3400-28T-AC sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.21696',
  u'match',
  u'Generic.Host',
  u'KeyMile | MileGate | 2510 sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.94.5',
  u'match',
  u'DLink.DGS3100',
  u'DLink | DGS3100 | DGS-3100-24TG sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.3320.1.294.0',
  u'match',
  u'BDCOM.xPON',
  u'BDCOM | xPON | P3310C sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.3955.7.2.2000.1',
  u'match',
  u'Alstec.ALS',
  u'Alstec | ALS | ALS-62000'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.76.11',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DGS-1210-48 sysObjectID'),
 (u'eq',
  u'.1.3.6.1.4.1.1332.1.1.1',
  u'match',
  u'Iskratel.VOIP',
  u'Iskratel | VOIP | ISKRATEL VOIP-1 sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.76.9',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DGS-1210-16 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.4874.1.1.1.6.2',
  u'match',
  u'Juniper.JUNOSe',
  u'Juniper | JUNOSe | E120 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.25506.1.37',
  u'match',
  u'H3C.VRP',
  u'H3C | VRP | S3600-28P-EI sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.8886.6.140',
  u'match',
  u'Raisecom.ROS',
  u'Raisecom | ROS | ISCOM2128EA-MA-AC sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.890.1.5.8.12',
  u'match',
  u'Zyxel.ZyNOS',
  u'Zyxel | ZyNOS | ES-3124 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.890.1.5.11.11',
  u'match',
  u'Zyxel.MSAN',
  u'Zyxel | ZyNOS | IES-612 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.27514.1.1.2.59',
  u'match',
  u'Qtech.QSW2800',
  u'Qtech | QSW | QSW-8200-28F-AC-DC sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.2011.2.248',
  u'match',
  u'Huawei.MA5600T',
  u'Huawei | MA | MA5608T sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.890.1.5.13.7',
  u'match',
  u'Zyxel.MSAN',
  u'Zyxel | ZyNOS | IES-5005 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.27514.1.3.26.7',
  u'match',
  u'Qtech.QSW',
  u'Qtech | QSW | QSW-2910-28F sysObjectID'),
 (u'contains',
  u'.1.3.6.1.4.1.1332.1.24.3',
  u'match',
  u'Iskratel.ESCOM',
  u'Iskratel | ESCOM | SI3000 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.30803',
  u'match',
  u'Vyatta.Vyatta',
  u'Vyatta | Vyatta | sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.2011.2.23.92',
  u'match',
  u'Huawei.VRP',
  u'Huawei | VRP | S2326TP-EI sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.27514.1.1.1.228',
  u'match',
  u'Qtech.QSW2800',
  u'Qtech | QSW | QSW-3450-28T-AC sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.35265.1.21.202.1',
  u'match',
  u'Eltex.LTE',
  u'Eltex | LTE | LTE sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.890.1.5.11.6',
  u'match',
  u'Zyxel.MSAN',
  u'Zyxel | ZyNOS | IES-1000 AAM1008-61 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.2011.10.1.200',
  u'match',
  u'Huawei.VRP',
  u'Huawei | VRP | S2403TP-EA sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.27514.1.1.1.281',
  u'match',
  u'Qtech.QSW',
  u'Qtech | QSW | QSW-3470-28T-AC sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.259.6.10.94',
  u'match',
  u'EdgeCore.ES',
  u'EdgeCore | ES | ES3528M sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.835457.6.3',
  u'match',
  u'Alcatel.TIMOS',
  u'Alcatel | TIMOS | 7450 ESS-7'),
 (u'eq',
  u'1.3.6.1.4.1.4413',
  u'match',
  u'Alstec.24xx',
  u'Alstec | ALS24300 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.2011.2.169',
  u'match',
  u'Huawei.MA5600T',
  u'Huawei | MA | MA5616 sysObjectID'),
 (u'contains',
  u'1.3.6.1.4.1.171.10.76.17',
  u'match',
  u'DLink.DxS_Smart',
  u'DLink | DxS_Smart | DGS-1210-52 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.27514.1.1.1.51',
  u'match',
  u'Qtech.QSW2800',
  u'Qtech | QSW | QSW-2800-10T-DC sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.3955.6.1.2016.1',
  u'match',
  u'Linksys.SRW',
  u'Linksys | SRW | SRW2016 sysObjectID'),
 (u'eq',
  u'1.3.6.1.4.1.11.2.3.7.11.63',
  u'match',
  u'HP.ProCurve',
  u'HP | ProCurve | 2810-24G sysObjectID')]
------------------------------------------------------------------------
File: services/discovery/jobs/box/profile.py (Line: 36)
Function: handler
   29         name = "profile"
   30
   31         _rules_cache = cachetools.TTLCache(10, ttl=60)
   32         snmp_version_def = None
   33
   34         def handler(self):
   35             self.logger.info("Checking profile accordance")
   36 ==>         profile = self.get_profile()
   37             if not profile:
   38                 return  # Cannot detect
   39             if profile == self.object.profile_name:
   40                 self.logger.info("Profile is correct: %s", profile)
   41             else:
   42                 self.logger.info(
Variables:
                self =
<noc.services.discovery.jobs.box.profile.ProfileCheck object at 0x7ff1a256fc10>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self =
<noc.services.discovery.jobs.box.profile.ProfileCheck object at 0x7ff1a256fc10>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.191432 [discovery|box|DeviceName|version]  Checking version
2017-07-04 10:34:45.199680 [discovery|box|DeviceName|version]  Set path: None
2017-07-04 10:34:45.222728 [discovery|box|DeviceName|version]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.200823)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff1a8558780>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff1a35e40d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff1a35e40d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff1a35e4f90>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_version",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a360b710>
                args = (13, 'get_version', {}, None)
                  t0 = 1499153685.198205
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg = {'params': [13, 'get_version', {}, None], 'method': 'script', 'id': 0}
              method = 'script'
           make_call = <function make_call at 0x7ff19f29ac08>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff1a35e4f90>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff1a35e4810>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_version', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a360b710>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19ec48368>)]
                  ev = <threading._Event object at 0x7ff1a35e4450>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a360b710>
                args = (13, 'get_version', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19ec48368>)]
                  ev = <threading._Event object at 0x7ff1a35e4450>
               _call = <function _call at 0x7ff19f54f668>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_version'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff1a360ba10>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/version.py (Line: 23)
Function: handler
   16         """
   17         name = "version"
   18         required_script = "get_version"
   19
   20         def handler(self):
   21             self.logger.info("Checking version")
   22             old_platform = self.object.platform
   23 ==>         result = self.object.scripts.get_version()
   24             r = {}
   25             for k in result:
   26                 v = result[k]
   27                 if k == "attributes":
   28                     for kk in v:
   29                         r[kk] = v[kk]
Variables:
                self =
<noc.services.discovery.jobs.box.version.VersionCheck object at 0x7ff19f29d910>
        old_platform = u'Cisco C2960'
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self =
<noc.services.discovery.jobs.box.version.VersionCheck object at 0x7ff19f29d910>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.224183 [discovery|box|DeviceName|caps]  Checking capabilities
2017-07-04 10:34:45.226001 [discovery|box|DeviceName|caps]  Set path: None
2017-07-04 10:34:45.248176 [discovery|box|DeviceName|caps]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.227091)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff1a8558140>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff1a35e4ad0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff1a35e4ad0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff1a35e4190>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_capabilities",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a3665910>
                args = (13, 'get_capabilities', {}, None)
                  t0 = 1499153685.224653
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg =
{'id': 0, 'method': 'script', 'params': [13, 'get_capabilities', {}, None]}
              method = 'script'
           make_call = <function make_call at 0x7ff1a2571050>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff1a35e4190>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff1a35e4990>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_capabilities', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a3665910>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19f5acb90>)]
                  ev = <threading._Event object at 0x7ff1a35e4c50>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a3665910>
                args = (13, 'get_capabilities', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19f5acb90>)]
                  ev = <threading._Event object at 0x7ff1a35e4c50>
               _call = <function _call at 0x7ff1a25719b0>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_capabilities'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff1a3665ad0>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/caps.py (Line: 24)
Function: handler
   17         Version discovery
   18         """
   19         name = "caps"
   20         required_script = "get_capabilities"
   21
   22         def handler(self):
   23             self.logger.info("Checking capabilities")
   24 ==>         result = self.object.scripts.get_capabilities()
   25             self.logger.debug("Received capabilities: \n%s",
   26                               ujson.dumps(result, indent=4))
   27             self.update_caps(result, source="caps")
Variables:
                self =
<noc.services.discovery.jobs.box.caps.CapsCheck object at 0x7ff19f29d910>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self =
<noc.services.discovery.jobs.box.caps.CapsCheck object at 0x7ff19f29d910>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.251094 [discovery|box|DeviceName|interface]  Checking interfaces
2017-07-04 10:34:45.252637 [discovery|box|DeviceName|interface]  Set path: None
2017-07-04 10:34:45.272130 [discovery|box|DeviceName|interface]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.253560)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff19f53be60>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f351050>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f351050>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f5b4490>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_interfaces",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a35e4a50>
                args = (13, 'get_interfaces', {}, None)
                  t0 = 1499153685.251442
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg =
{'id': 0, 'method': 'script', 'params': [13, 'get_interfaces', {}, None]}
              method = 'script'
           make_call = <function make_call at 0x7ff19f37fed8>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f5b4490>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f5b4f50>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_interfaces', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a35e4a50>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19f6f65f0>)]
                  ev = <threading._Event object at 0x7ff1a35e4510>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a35e4a50>
                args = (13, 'get_interfaces', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19f6f65f0>)]
                  ev = <threading._Event object at 0x7ff1a35e4510>
               _call = <function _call at 0x7ff19f37f410>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_interfaces'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff1a35e4610>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/interface.py (Line: 33)
Function: handler
   26
   27         def __init__(self, *args, **kwargs):
   28             super(InterfaceCheck, self).__init__(*args, **kwargs)
   29             self.get_interface_profile = InterfaceClassificationRule.get_classificator()
   30
   31         def handler(self):
   32             self.logger.info("Checking interfaces")
   33 ==>         result = self.object.scripts.get_interfaces()
   34             self.seen_interfaces = []
   35             # Process forwarding instances
   36             for fi in result:
   37                 # Apply forwarding instance
   38                 forwarding_instance = self.submit_forwarding_instance(
   39                     name=fi["forwarding_instance"],
Variables:
                self =
<noc.services.discovery.jobs.box.interface.InterfaceCheck object at 0x7ff19f29d910>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self =
<noc.services.discovery.jobs.box.interface.InterfaceCheck object at 0x7ff19f29d910>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.273384 [discovery|box|DeviceName|id]  Checking chassis id
2017-07-04 10:34:45.274947 [discovery|box|DeviceName|id]  Set path: None
2017-07-04 10:34:45.294507 [discovery|box|DeviceName|id]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.275877)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff19f5b1320>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f351790>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f351790>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f351250>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_discovery_id",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f5b4d50>
                args = (13, 'get_discovery_id', {}, None)
                  t0 = 1499153685.273782
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg =
{'id': 0, 'method': 'script', 'params': [13, 'get_discovery_id', {}, None]}
              method = 'script'
           make_call = <function make_call at 0x7ff1a2571b90>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f351250>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f351110>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_discovery_id', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f5b4d50>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a45f60e0>)]
                  ev = <threading._Event object at 0x7ff19f3516d0>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f5b4d50>
                args = (13, 'get_discovery_id', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a45f60e0>)]
                  ev = <threading._Event object at 0x7ff19f3516d0>
               _call = <function _call at 0x7ff19ec29398>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_discovery_id'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff1a35e4250>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/id.py (Line: 23)
Function: handler
   16         Version discovery
   17         """
   18         name = "id"
   19         required_script = "get_discovery_id"
   20
   21         def handler(self):
   22             self.logger.info("Checking chassis id")
   23 ==>         result = self.object.scripts.get_discovery_id()
   24             cm = result.get("chassis_mac")
   25             if cm:
   26                 cm = ", ".join(
   27                     "%s - %s" % (m["first_chassis_mac"], m["last_chassis_mac"])
   28                     for m in cm
   29                 )
Variables:
                self = <noc.services.discovery.jobs.box.id.IDCheck object at 0x7ff19f29d910>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self = <noc.services.discovery.jobs.box.id.IDCheck object at 0x7ff19f29d910>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.295806 [discovery|box|DeviceName|config]  Checking config
2017-07-04 10:34:45.297369 [discovery|box|DeviceName|config]  Set path: None
2017-07-04 10:34:45.317250 [discovery|box|DeviceName|config]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.298312)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff19f253c30>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f3513d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f3513d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f351410>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_config",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f351610>
                args = (13, 'get_config', {}, None)
                  t0 = 1499153685.296176
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg = {'params': [13, 'get_config', {}, None], 'method': 'script', 'id': 0}
              method = 'script'
           make_call = <function make_call at 0x7ff19f37fc08>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f351410>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f351190>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_config', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f351610>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19f2c7a28>)]
                  ev = <threading._Event object at 0x7ff19f3512d0>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f351610>
                args = (13, 'get_config', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19f2c7a28>)]
                  ev = <threading._Event object at 0x7ff19f3512d0>
               _call = <function _call at 0x7ff19f377f50>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_config'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff19f351710>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/config.py (Line: 22)
Function: handler
   15         Version discovery
   16         """
   17         name = "config"
   18         required_script = "get_config"
   19
   20         def handler(self):
   21             self.logger.info("Checking config")
   22 ==>         result = self.object.scripts.get_config()
   23             self.object.save_config(result)
Variables:
                self =
<noc.services.discovery.jobs.box.config.ConfigCheck object at 0x7ff19f29d910>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self =
<noc.services.discovery.jobs.box.config.ConfigCheck object at 0x7ff19f29d910>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.322443 [discovery|box|DeviceName|asset]  Checking assets
2017-07-04 10:34:45.324652 [discovery|box|DeviceName|asset]  Set path: None
2017-07-04 10:34:45.343824 [discovery|box|DeviceName|asset]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.325597)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff1a8558fa0>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f6f4550>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f6f4550>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f4dd0>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_inventory",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f351250>
                args = (13, 'get_inventory', {}, None)
                  t0 = 1499153685.322955
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg = {'params': [13, 'get_inventory', {}, None], 'method': 'script', 'id': 0}
              method = 'script'
           make_call = <function make_call at 0x7ff19f54f500>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f4dd0>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f351810>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_inventory', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f351250>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a45f6440>)]
                  ev = <threading._Event object at 0x7ff19f351590>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f351250>
                args = (13, 'get_inventory', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a45f6440>)]
                  ev = <threading._Event object at 0x7ff19f351590>
               _call = <function _call at 0x7ff1a3601578>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_inventory'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff19f351050>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/asset.py (Line: 49)
Function: handler
   42             self.stack_member = {}  # object -> stack member numbers
   43             self.managed = set()  # Object ids
   44             self.unk_model = {}  # name -> model
   45             self.lost_and_found = self.get_lost_and_found(self.object)
   46
   47         def handler(self):
   48             self.logger.info("Checking assets")
   49 ==>         result = self.object.scripts.get_inventory()
   50             self.find_managed()
   51             # Submit objects
   52             for o in result:
   53                 self.logger.debug("Submit %s", str_dict(o))
   54                 self.submit(
   55                     type=o["type"], number=o.get("number"),
Variables:
                self =
<noc.services.discovery.jobs.box.asset.AssetCheck object at 0x7ff19f29d910>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self =
<noc.services.discovery.jobs.box.asset.AssetCheck object at 0x7ff19f29d910>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.345028 [discovery|box|DeviceName|vlan]  Checking vlans
2017-07-04 10:34:45.348970 [discovery|box|DeviceName|vlan]  Set path: None
2017-07-04 10:34:45.366055 [discovery|box|DeviceName|vlan]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.349798)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff19f253b40>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f6e16d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f6e16d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6e1050>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_vlans",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a360b250>
                args = (13, 'get_vlans', {}, None)
                  t0 = 1499153685.347559
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg = {'params': [13, 'get_vlans', {}, None], 'method': 'script', 'id': 0}
              method = 'script'
           make_call = <function make_call at 0x7ff19ec29398>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6e1050>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6e14d0>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_vlans', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a360b250>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a3585170>)]
                  ev = <threading._Event object at 0x7ff1a360b850>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff1a360b250>
                args = (13, 'get_vlans', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a3585170>)]
                  ev = <threading._Event object at 0x7ff1a360b850>
               _call = <function _call at 0x7ff1a3601938>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_vlans'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff1a35f23d0>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/vlan.py (Line: 28)
Function: handler
   21         def handler(self):
   22             self.logger.info("Checking vlans")
   23             if not self.object.vc_domain:
   24                 self.logger.info(
   25                     "No vc domain. Skipping"
   26                 )
   27                 return
   28 ==>         result = self.object.scripts.get_vlans()
   29             for v in result:
   30                 self.submit(
   31                     vc_domain=self.object.vc_domain,
   32                     l1=v["vlan_id"],
   33                     name=v.get("name")
   34                 )
Variables:
                self =
<noc.services.discovery.jobs.box.vlan.VLANCheck object at 0x7ff19f29d910>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self =
<noc.services.discovery.jobs.box.vlan.VLANCheck object at 0x7ff19f29d910>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.367193 [discovery|box|DeviceName|lacp]  Checking lacp topology
2017-07-04 10:34:45.368881 [discovery|box|DeviceName|lacp]  Set path: None
2017-07-04 10:34:45.396375 [discovery|box|DeviceName|lacp]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.369750)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff19f5b1320>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f3514d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f3514d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f3518d0>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_lacp_neighbors",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f6e1a50>
                args = (13, 'get_lacp_neighbors', {}, None)
                  t0 = 1499153685.367538
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg =
{'id': 0, 'method': 'script', 'params': [13, 'get_lacp_neighbors', {}, None]}
              method = 'script'
           make_call = <function make_call at 0x7ff1a36b19b0>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f3518d0>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f702310>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_lacp_neighbors', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f6e1a50>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a3583d88>)]
                  ev = <threading._Event object at 0x7ff19f6e10d0>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f6e1a50>
                args = (13, 'get_lacp_neighbors', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a3583d88>)]
                  ev = <threading._Event object at 0x7ff19f6e10d0>
               _call = <function _call at 0x7ff1a36b1488>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_lacp_neighbors'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff19f6e1450>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/lacp.py (Line: 22)
Function: iter_neighbors
   15         CDP Topology discovery
   16         """
   17         name = "lacp"
   18         required_script = "get_lacp_neighbors"
   19         required_capabilities = ["Network | LACP"]
   20
   21         def iter_neighbors(self, mo):
   22 ==>         result = mo.scripts.get_lacp_neighbors()
   23             for lag in result:
   24                 for n in lag["bundle"]:
   25                     self.set_interface_alias(mo, n["interface"], n["local_port_id"])
   26                     # We yield local port id instead of real name
   27                     # as we set interface alias
   28                     # Real port name will be set during clean_interface
Variables:
                  mo = <ManagedObject: DeviceName>
                self =
<noc.services.discovery.jobs.box.lacp.LACPCheck object at 0x7ff1a360b7d0>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 593)
Function: handler
  586         def handler(self):
  587             self.logger.info("Checking %s topology", self.name)
  588             # remote object -> [(local, remote), ..]
  589             candidates = defaultdict(set)
  590             loops = {}  # first interface, second interface
  591             problems = {}
  592             # Check local side
  593 ==>         for li, ro, ri in self.iter_neighbors(self.object):
  594                 # Resolve remote object
  595                 remote_object = self.get_neighbor(ro)
  596                 if not remote_object:
  597                     problems[li] = "Remote object '%s' is not found" % str(ro)
  598                     self.logger.info(
  599                         "Remote object '%s' is not found. Skipping",
Variables:
               loops = {}
          candidates = defaultdict(<type 'set'>, {})
                self =
<noc.services.discovery.jobs.box.lacp.LACPCheck object at 0x7ff1a360b7d0>
            problems = {}
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self =
<noc.services.discovery.jobs.box.lacp.LACPCheck object at 0x7ff1a360b7d0>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.397723 [discovery|box|DeviceName|udld]  Object hasn't required capability 'Network | UDLD'. Skipping
2017-07-04 10:34:45.397968 [discovery|box|DeviceName|lldp]  Checking lldp topology
2017-07-04 10:34:45.399667 [discovery|box|DeviceName|lldp]  Set path: None
2017-07-04 10:34:45.420801 [discovery|box|DeviceName|lldp]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.400644)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff19f596190>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f6f4f50>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f6f4f50>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f4890>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_lldp_neighbors",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f6e1390>
                args = (13, 'get_lldp_neighbors', {}, None)
                  t0 = 1499153685.398464
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg =
{'id': 0, 'method': 'script', 'params': [13, 'get_lldp_neighbors', {}, None]}
              method = 'script'
           make_call = <function make_call at 0x7ff1a2571c08>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f4890>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f4790>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_lldp_neighbors', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f6e1390>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a255fcb0>)]
                  ev = <threading._Event object at 0x7ff1a360b7d0>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f6e1390>
                args = (13, 'get_lldp_neighbors', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a255fcb0>)]
                  ev = <threading._Event object at 0x7ff1a360b7d0>
               _call = <function _call at 0x7ff1a2571d70>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_lldp_neighbors'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff19f702490>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/lldp.py (Line: 35)
Function: iter_neighbors
   28         PORT_SUBTYPE_ALIAS = 1
   29         PORT_SUBTYPE_MAC = 3
   30         PORT_SUBTYPE_NAME = 5
   31         PORT_SUBTYPE_LOCAL = 7
   32         PORT_SUBTYPE_UNSPECIFIED = 128
   33
   34         def iter_neighbors(self, mo):
   35 ==>         result = mo.scripts.get_lldp_neighbors()
   36             self.n_cache = {}  # (chassis_id, chassis_subtype) -> object
   37             for n in result:
   38                 if len(n["neighbors"]) == 1:
   39                     nn = n["neighbors"][0]
   40                     yield n["local_interface"], nn, nn
   41
Variables:
                  mo = <ManagedObject: DeviceName>
                self =
<noc.services.discovery.jobs.box.lldp.LLDPCheck object at 0x7ff19f6e1550>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 593)
Function: handler
  586         def handler(self):
  587             self.logger.info("Checking %s topology", self.name)
  588             # remote object -> [(local, remote), ..]
  589             candidates = defaultdict(set)
  590             loops = {}  # first interface, second interface
  591             problems = {}
  592             # Check local side
  593 ==>         for li, ro, ri in self.iter_neighbors(self.object):
  594                 # Resolve remote object
  595                 remote_object = self.get_neighbor(ro)
  596                 if not remote_object:
  597                     problems[li] = "Remote object '%s' is not found" % str(ro)
  598                     self.logger.info(
  599                         "Remote object '%s' is not found. Skipping",
Variables:
               loops = {}
          candidates = defaultdict(<type 'set'>, {})
                self =
<noc.services.discovery.jobs.box.lldp.LLDPCheck object at 0x7ff19f6e1550>
            problems = {}
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self =
<noc.services.discovery.jobs.box.lldp.LLDPCheck object at 0x7ff19f6e1550>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.422218 [discovery|box|DeviceName|cdp]  Checking cdp topology
2017-07-04 10:34:45.423831 [discovery|box|DeviceName|cdp]  Set path: None
2017-07-04 10:34:45.444269 [discovery|box|DeviceName|cdp]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.424806)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff1a8558140>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f6f48d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f6f48d0>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f44d0>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_cdp_neighbors",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f713390>
                args = (13, 'get_cdp_neighbors', {}, None)
                  t0 = 1499153685.422671
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg =
{'id': 0, 'method': 'script', 'params': [13, 'get_cdp_neighbors', {}, None]}
              method = 'script'
           make_call = <function make_call at 0x7ff19f29a2a8>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f44d0>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f4810>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_cdp_neighbors', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f713390>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19f5464d0>)]
                  ev = <threading._Event object at 0x7ff19f7136d0>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f713390>
                args = (13, 'get_cdp_neighbors', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff19f5464d0>)]
                  ev = <threading._Event object at 0x7ff19f7136d0>
               _call = <function _call at 0x7ff19f54f320>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_cdp_neighbors'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff19f713b10>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/cdp.py (Line: 25)
Function: iter_neighbors
   18         name = "cdp"
   19         required_script = "get_cdp_neighbors"
   20         required_capabilities = ["Network | CDP"]
   21
   22         RESERVED_NAMES = set(["Switch", "Router", "MikroTik"])
   23
   24         def iter_neighbors(self, mo):
   25 ==>         result = mo.scripts.get_cdp_neighbors()
   26             for n in result["neighbors"]:
   27                 device_id = n["device_id"]
   28                 if device_id in self.RESERVED_NAMES and n.get("remote_ip"):
   29                     device_id = n["remote_ip"]
   30                 yield (
   31                     n["local_interface"],
Variables:
                  mo = <ManagedObject: DeviceName>
                self = <noc.services.discovery.jobs.box.cdp.CDPCheck object at 0x7ff19f6ee250>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 593)
Function: handler
  586         def handler(self):
  587             self.logger.info("Checking %s topology", self.name)
  588             # remote object -> [(local, remote), ..]
  589             candidates = defaultdict(set)
  590             loops = {}  # first interface, second interface
  591             problems = {}
  592             # Check local side
  593 ==>         for li, ro, ri in self.iter_neighbors(self.object):
  594                 # Resolve remote object
  595                 remote_object = self.get_neighbor(ro)
  596                 if not remote_object:
  597                     problems[li] = "Remote object '%s' is not found" % str(ro)
  598                     self.logger.info(
  599                         "Remote object '%s' is not found. Skipping",
Variables:
               loops = {}
          candidates = defaultdict(<type 'set'>, {})
                self = <noc.services.discovery.jobs.box.cdp.CDPCheck object at 0x7ff19f6ee250>
            problems = {}
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self = <noc.services.discovery.jobs.box.cdp.CDPCheck object at 0x7ff19f6ee250>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.445635 [discovery|box|DeviceName|stp]  Checking stp topology
2017-07-04 10:34:45.447470 [discovery|box|DeviceName|stp]  Set path: None
2017-07-04 10:34:45.475260 [discovery|box|DeviceName|stp]  UNHANDLED EXCEPTION (2017-07-04 10:34:45.448721)
BRANCH: feature/microservices TIP: 0db95855ccf8
PROCESS: ./services/discovery/service.py
ERROR FINGERPRINT: 63d23b53-e459-5cf6-a8b0-b3cf3621d591
WORKING DIRECTORY: /opt/noc
EXCEPTION: <type 'exceptions.TypeError'> unsupported type for timedelta seconds component: datetime.timedelta
START OF TRACEBACK
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 225)
Function: resolve
  218         @tornado.gen.coroutine
  219         def resolve(self, hint=None, wait=True, timeout=None, full_result=False):
  220             metrics["dcs.resolver.requests"] += 1
  221             if wait:
  222                 # Wait until service catalog populated
  223                 try:
  224                     yield self.ready_event.wait(
  225 ==>                     timeout=datetime.timedelta(seconds=timeout or self.dcs.DEFAULT_SERVICE_RESOLUTION_TIMEOUT)
  226                     )
  227                 except tornado.gen.TimeoutError:
  228                     metrics["dcs.resolver.errors"] += 1
  229                     raise ResolutionError()
  230             if not wait and not self.ready_event.is_set():
  231                 raise ResolutionError()
Variables:
                self = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
                hint = None
             timeout = None
         full_result = False
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 307)
Function: wrapper
  300                     # Inline the first iteration of Runner.run.  This lets us
  301                     # avoid the cost of creating a Runner when the coroutine
  302                     # never actually yields, which in turn allows us to
  303                     # use "optional" coroutines in critical path code without
  304                     # performance penalty for the synchronous case.
  305                     try:
  306                         orig_stack_contexts = stack_context._state.contexts
  307 ==>                     yielded = next(result)
  308                         if stack_context._state.contexts is not orig_stack_contexts:
  309                             yielded = TracebackFuture()
  310                             yielded.set_exception(
  311                                 stack_context.StackContextInconsistentError(
  312                                     'stack_context inconsistency (probably caused '
  313                                     'by yield within a "with StackContext" block)'))
Variables:
                func = <function resolve at 0x7ff1b91302a8>
                args = (<noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>,)
             yielded = None
    replace_callback = True
              future = None
              result = <generator object resolve at 0x7ff19f5b14b0>
 orig_stack_contexts = ((), None)
              kwargs = {'wait': True, 'full_result': False, 'timeout': None, 'hint': None}
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f6f4210>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: core/dcs/base.py (Line: 114)
Function: resolve
  107
  108         @tornado.gen.coroutine
  109         def resolve(self, name, hint=None, wait=True, timeout=None,
  110                     full_result=False):
  111             resolver = yield self.get_resolver(name)
  112             r = yield resolver.resolve(
  113                 hint=hint, wait=wait, timeout=timeout,
  114 ==>             full_result=full_result
  115             )
  116             raise tornado.gen.Return(r)
  117
  118         @tornado.gen.coroutine
  119         def expire_resolvers(self):
  120             with self.resolvers_lock:
Variables:
                name = 'sae'
                hint = None
                self = <noc.core.dcs.consuldcs.ConsulDCS object at 0x7ff1ae99a150>
            resolver = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
         full_result = False
             timeout = None
                wait = True
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
                self = <tornado.gen.Runner object at 0x7ff19f6f4210>
               value = <noc.core.dcs.consuldcs.ConsulResolver object at 0x7ff1a4537e90>
             yielded = None
              future = None
 orig_stack_contexts = ((), None)
            exc_info = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f4610>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 148)
Function: _call
  141             # Get services
  142             response = None
  143             for t in self._service.iter_rpc_retry_timeout():
  144                 # Resolve service against service catalog
  145                 if self._hints:
  146                     svc = random.choice(self._hints)
  147                 else:
  148 ==>                 svc = yield self._service.dcs.resolve(self._service_name)
  149                 response = yield make_call(
  150                     "http://%s/api/%s/" % (svc, self._api),
  151                     body
  152                 )
  153                 if response:
  154                     break
Variables:
                body = '{"params":[13,"get_spanning_tree",{},null],"method":"script","id":0}'
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f6f46d0>
                args = (13, 'get_spanning_tree', {}, None)
                  t0 = 1499153685.446104
            response = None
                 tid = 0
                   t = 0.1
              kwargs = {}
                 msg =
{'id': 0, 'method': 'script', 'params': [13, 'get_spanning_tree', {}, None]}
              method = 'script'
           make_call = <function make_call at 0x7ff19f37f410>
           is_notify = False
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1063)
Function: run
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
 1062                             try:
 1063 ==>                             yielded = self.gen.throw(*exc_info)
 1064                             finally:
 1065                                 # Break up a reference to itself
 1066                                 # for faster GC on CPython.
 1067                                 exc_info = None
 1068                         else:
 1069                             yielded = self.gen.send(value)
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f4610>
              future = None
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/concurrent.py (Line: 238)
Function: result
  231             before the `Future` is done, so the ``timeout`` is never used.
  232             """
  233             self._clear_tb_log()
  234             if self._result is not None:
  235                 return self._result
  236             if self._exc_info is not None:
  237                 try:
  238 ==>                 raise_exc_info(self._exc_info)
  239                 finally:
  240                     self = None
  241             self._check_done()
  242             return self._result
  243
  244         def exception(self, timeout=None):
Variables:
                self = None
             timeout = None
------------------------------------------------------------------------
File: lib/python2.7/site-packages/tornado/gen.py (Line: 1055)
Function: run
 1048                         return
 1049                     self.future = None
 1050                     try:
 1051                         orig_stack_contexts = stack_context._state.contexts
 1052                         exc_info = None
 1053
 1054                         try:
 1055 ==>                         value = future.result()
 1056                         except Exception:
 1057                             self.had_exception = True
 1058                             exc_info = sys.exc_info()
 1059                         future = None
 1060
 1061                         if exc_info is not None:
Variables:
            exc_info = None
                self = <tornado.gen.Runner object at 0x7ff19f6f43d0>
              future = None
                   e = StopIteration()
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 63)
Function: _call
   56                 result = yield self._call(item, *args, **kwargs)
   57                 raise tornado.gen.Return(result)
   58
   59             def sync_wrapper(*args, **kwargs):
   60                 @tornado.gen.coroutine
   61                 def _call():
   62                     try:
   63 ==>                     r = yield self._call(item, *args, **kwargs)
   64                         result.append(r)
   65                     except tornado.gen.Return as e:
   66                         result.append(e.value)
   67                     except Exception:
   68                         error.append(sys.exc_info())
   69                     finally:
Variables:
                args = (13, 'get_spanning_tree', {}, None)
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f6f46d0>
              kwargs = {}
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a8554998>)]
                  ev = <threading._Event object at 0x7ff19f6ee250>
------------------------------------------------------------------------
File: core/service/rpc.py (Line: 78)
Function: sync_wrapper
   71
   72                 ev = threading.Event()
   73                 result = []
   74                 error = []
   75                 self._service.ioloop.add_callback(_call)
   76                 ev.wait()
   77                 if error:
   78 ==>                 six.reraise(*error[0])
   79                 else:
   80                     return result[0]
   81
   82             if item.startswith("_"):
   83                 return self.__dict__[item]
   84             elif self._sync:
Variables:
              kwargs = {}
                self = <noc.core.service.rpc.RPCProxy object at 0x7ff19f6f46d0>
                args = (13, 'get_spanning_tree', {}, None)
                item = 'script'
              result = []
               error =
[(<type 'exceptions.TypeError'>,
  TypeError('unsupported type for timedelta seconds component: datetime.timedelta',),
  <traceback object at 0x7ff1a8554998>)]
                  ev = <threading._Event object at 0x7ff19f6ee250>
               _call = <function _call at 0x7ff19f37f320>
------------------------------------------------------------------------
File: sa/mtmanager.py (Line: 30)
Function: run
   23             """
   24             Run SA script and wait for result
   25             """
   26             if "." in script:
   27                 # Leave only script name
   28                 script = script.split(".")[-1]
   29             return open_sync_rpc("sae", calling_service="MTManager").script(
   30 ==>             object.id, script, params, timeout
   31             )
   32
   33
   34     # Run single instance
   35     MTManager = MTManagerImplementation()
Variables:
                self = <noc.sa.mtmanager.MTManagerImplementation object at 0x7ff1aa721f10>
              object = <ManagedObject: DeviceName>
              params = {}
             timeout = None
              script = 'get_spanning_tree'
------------------------------------------------------------------------
File: sa/models/managedobject.py (Line: 1172)
Function: __call__
 1165     class ScriptsProxy(object):
 1166         class CallWrapper(object):
 1167             def __init__(self, obj, name):
 1168                 self.name = name
 1169                 self.object = obj
 1170
 1171             def __call__(self, **kwargs):
 1172 ==>             return MTManager.run(self.object, self.name, kwargs)
 1173
 1174         def __init__(self, obj):
 1175             self._object = obj
 1176             self._cache = {}
 1177
 1178         def __getattr__(self, name):
Variables:
                self = <noc.sa.models.managedobject.CallWrapper object at 0x7ff19ec49b50>
              kwargs = {}
------------------------------------------------------------------------
File: services/discovery/jobs/box/stp.py (Line: 49)
Function: get_root_ports
   42                         )
   43
   44         def get_root_ports(self):
   45             """
   46             Returns a dict of
   47             remote_object_id -> set((local_interface, remote_port_id)
   48             """
   49 ==>         result = self.object.scripts.get_spanning_tree()
   50             roots = defaultdict(set)  # Neighbor -> [(local iface, remote_id)]
   51             for i in result["instances"]:
   52                 for iface in i["interfaces"]:
   53                     if iface["role"] in ("root", "alternate"):
   54                         roots[iface["designated_bridge_id"]].add((
   55                             iface["interface"],
Variables:
                self = <noc.services.discovery.jobs.box.stp.STPCheck object at 0x7ff19f6ee110>
------------------------------------------------------------------------
File: services/discovery/jobs/box/stp.py (Line: 25)
Function: handler
   18         """
   19         name = "stp"
   20         required_script = "get_spanning_tree"
   21         required_capabilities = ["Network | STP"]
   22
   23         def handler(self):
   24             self.logger.info("Checking %s topology", self.name)
   25 ==>         roots = self.get_root_ports()
   26             for ro in roots:
   27                 remote_object = self.get_neighbor(ro)
   28                 if not remote_object:
   29                     self.logger.debug(
   30                         "Remote object '%s' is not found. Skipping",
   31                         ro
Variables:
                self = <noc.services.discovery.jobs.box.stp.STPCheck object at 0x7ff19f6ee110>
------------------------------------------------------------------------
File: services/discovery/jobs/base.py (Line: 382)
Function: run
  375                                      self.required_script)
  376                     return
  377                 # Check required capabilities
  378                 if not self.has_required_capabilities():
  379                     return
  380                 # Run check
  381                 try:
  382 ==>                 self.handler()
  383                 except RPCRemoteError as e:
  384                     self.logger.error(
  385                         "RPC Remote error (%s): %s",
  386                         e.remote_code, e)
  387                     self.set_problem(
  388                         alarm_class=self.error_map.get(e.remote_code),
Variables:
                self = <noc.services.discovery.jobs.box.stp.STPCheck object at 0x7ff19f6ee110>
                   e =
TypeError('unsupported type for timedelta seconds component: datetime.timedelta',)
------------------------------------------------------------------------
END OF TRACEBACK
2017-07-04 10:34:45.476734 [discovery|box|DeviceName]  Completed. Status: SUCCESS (474.53ms)
2017-07-04 10:34:45.476945 [discovery|box|DeviceName]  Timings: profile = 178.07ms, version = 32.70ms, caps = 25.43ms, interface = 22.24ms, id = 22.32ms, config = 22.68ms, asset = 22.56ms, vlan = 22.08ms, lacp = 30.41ms, udld = 0.23ms, lldp = 24.14ms, cdp = 23.32ms, stp = 31.03ms
2017-07-04 10:34:45.477331 [discovery|box|DeviceName]  Updating alarm statuses
2017-07-04 10:34:45.503490 [discovery|box|DeviceName]  Opening umbrella alarm 595b45154a5e68ae813df106 (Discovery | Job | Box)
2017-07-04 10:34:45.504572 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.509372 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df107  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.509557 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.513257 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df108  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.513441 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.517081 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df109  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.517264 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.520934 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df10a  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.521120 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.524780 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df10b  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.524963 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.529155 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df10c  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.529341 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.533059 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df10d  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.533244 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.536940 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df10e  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.537124 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.540802 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df10f  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.540985 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.544612 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df110  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.544801 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.548670 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df111  (Discovery | Error | Unhandled Exception)
2017-07-04 10:34:45.548855 [discovery|box|DeviceName]  Create detail alarm to path
2017-07-04 10:34:45.552488 [discovery|box|DeviceName]  Opening detail alarm 595b45154a5e68ae813df112  (Discovery | Error | Unhandled Exception)