2015-11-19 15:44:22,809 [script] [Cisco.IOS.get_version] [10.31.10.115] Running. Input arguments: {}, timeout 120
2015-11-19 15:44:22,809 [noc.core.ioloop.snmp] [10.31.10.115] SNMP GET ['1.3.6.1.2.1.1.1.0']
2015-11-19 15:44:22,813 [noc.core.ioloop.snmp] [10.31.10.115] GET result: Cisco IOS Software, C3560 Software (C3560-IPSERVICESK9-M), Version 15.0(2)SE4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Wed 26-Jun-13 02:28 by prod_rel_team
2015-11-19 15:44:22,813 [script] [Cisco.IOS.get_version] [10.31.10.115] Result: {'platform': 'C3560', 'version': '15.0(2)SE4', 'vendor': 'Cisco', 'attributes': {'image': 'C3560-IPSERVICESK9-M'}}
2015-11-19 15:44:22,814 [script] [Cisco.IOS.get_version] [10.31.10.115] Complete (4.87ms)
2015-11-19 15:44:22,814 [script] [Cisco.IOS.get_interfaces] [10.31.10.115] Open ssh CLI
2015-11-19 15:44:22,815 [script] [Cisco.IOS.get_interfaces] [10.31.10.115] Setup session
2015-11-19 15:44:22,815 [script] [Cisco.IOS.get_interfaces] [10.31.10.115] Disable paging
2015-11-19 15:44:22,815 [script] [Cisco.IOS.get_interfaces] [10.31.10.115] [ssh] Connecting (u'10.31.10.115', 22)
2015-11-19 15:44:22,819 [script] [Cisco.IOS.get_interfaces] [10.31.10.115] [ssh] Connected
2015-11-19 15:44:22,819 [script] [Cisco.IOS.get_interfaces] [10.31.10.115] [ssh] Startup ssh session
2015-11-19 15:44:22,821 [tornado.general] error on read
Traceback (most recent call last):
  File "/opt/noc/local/lib/python2.7/site-packages/tornado/iostream.py", line 644, in _handle_read
    pos = self._read_to_buffer_loop()
  File "/opt/noc/local/lib/python2.7/site-packages/tornado/iostream.py", line 614, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/opt/noc/local/lib/python2.7/site-packages/tornado/iostream.py", line 726, in _read_to_buffer
    chunk = self.read_from_fd()
  File "/opt/noc/core/script/cli/ssh.py", line 63, in read_from_fd
    return self.channel.read(self.read_chunk_size)
AttributeError: 'NoneType' object has no attribute 'read'
2015-11-19 15:44:22,835 [noc.lib.debug] UNHANDLED EXCEPTION (2015-11-19 15:44:22.823595)
BRANCH: feature/microservices TIP: 92a686c04ea1
PROCESS: ./services/activator/service.py
ERROR FINGERPRINT: 654fafad-835b-5899-a59e-2f152916f9c5
WORKING DIRECTORY: /opt/noc
EXCEPTION: <class '_libssh2.Error'> SSH startup: Failed getting banner
START OF TRACEBACK
------------------------------------------------------------------------
File: local/lib/python2.7/site-packages/libssh2/session.py (Line: 230)
Function: startup
  223
  224             @param sock: a connected socket object
  225             @type sock: socket._socketobject
  226
  227             @return: 0 on success or negative on failure
  228             @rtype: int
  229             """
  230 ==>         self._session.startup(sock)
  231
  232         def userauth_authenticated(self):
  233             """
  234             Returns authentification status for the given session.
  235
  236             @return: non-zero if authenticated or 0 if not
Variables:
                self = <libssh2.session.Session object at 0x7fb7b21ec750>
                sock = <socket._socketobject object at 0x7fb7b2247750>
------------------------------------------------------------------------
File: core/script/cli/ssh.py (Line: 38)
Function: startup
   31         @tornado.gen.coroutine
   32         def startup(self):
   33             """
   34             SSH session startup
   35             """
   36             user = self.script.credentials["user"]
   37             self.logger.debug("Startup ssh session")
   38 ==>         self.session.startup(self.socket)
   39             host_hash = self.session.hostkey_hash(2)  # SHA1
   40             self.logger.debug("Connected. Host fingerprint is %s",
   41                               host_hash.encode("hex"))
   42             auth_methods = self.session.userauth_list(user).split(",")
   43             self.logger.debug("Supported authentication methods: %s",
   44                               ", ".join(auth_methods))
Variables:
                self = <noc.core.script.cli.ssh.SSHIOStream object at 0x7fb7b21ec410>
                user = u'cisco'
------------------------------------------------------------------------
File: local/lib/python2.7/site-packages/tornado/gen.py (Line: 215)
Function: wrapper
  208
  209             if replace_callback and 'callback' in kwargs:
  210                 callback = kwargs.pop('callback')
  211                 IOLoop.current().add_future(
  212                     future, lambda future: callback(future.result()))
  213
  214             try:
  215 ==>             result = func(*args, **kwargs)
  216             except (Return, StopIteration) as e:
  217                 result = getattr(e, 'value', None)
  218             except Exception:
  219                 future.set_exc_info(sys.exc_info())
  220                 return future
  221             else:
Variables:
    replace_callback = True
              future = <tornado.concurrent.Future object at 0x7fb7b21ec890>
                func = <function startup at 0x7fb7b21b9d70>
                args = (<noc.core.script.cli.ssh.SSHIOStream object at 0x7fb7b21ec410>,)
              kwargs = {}
------------------------------------------------------------------------
File: local/lib/python2.7/site-packages/tornado/concurrent.py (Line: 215)
Function: result
  208             `concurrent.futures.Future` but it is an error to call it
  209             before the `Future` is done, so the ``timeout`` is never used.
  210             """
  211             self._clear_tb_log()
  212             if self._result is not None:
  213                 return self._result
  214             if self._exc_info is not None:
  215 ==>             raise_exc_info(self._exc_info)
  216             self._check_done()
  217             return self._result
  218
  219         def exception(self, timeout=None):
  220             """If the operation raised an exception, return the `Exception`
  221             object.  Otherwise returns None.
Variables:
                self = <tornado.concurrent.Future object at 0x7fb7b21ec890>
             timeout = None
------------------------------------------------------------------------
File: local/lib/python2.7/site-packages/tornado/gen.py (Line: 870)
Function: run
  863                         return
  864                     self.future = None
  865                     try:
  866                         orig_stack_contexts = stack_context._state.contexts
  867                         exc_info = None
  868
  869                         try:
  870 ==>                         value = future.result()
  871                         except Exception:
  872                             self.had_exception = True
  873                             exc_info = sys.exc_info()
  874
  875                         if exc_info is not None:
  876                             yielded = self.gen.throw(*exc_info)
Variables:
                self = <tornado.gen.Runner object at 0x7fb7b21eca90>
               value = <noc.core.script.cli.ssh.SSHIOStream object at 0x7fb7b21ec410>
             yielded = <tornado.concurrent.Future object at 0x7fb7b21ec890>
              future = <tornado.concurrent.Future object at 0x7fb7b21ec890>
 orig_stack_contexts = ((), None)
            exc_info =
(<class '_libssh2.Error'>,
 Error('SSH startup: Failed getting banner',),
 <traceback object at 0x7fb7ade25fc8>)
------------------------------------------------------------------------
File: core/script/cli/base.py (Line: 75)
Function: submit
   68                 address = (
   69                     self.script.credentials.get("address"),
   70                     self.script.credentials.get("port", self.default_port)
   71                 )
   72                 self.logger.debug("Connecting %s", address)
   73                 yield self.iostream.connect(address)
   74                 self.logger.debug("Connected")
   75 ==>             yield self.iostream.startup()
   76             # Perform all necessary login procedures
   77             if not self.is_started:
   78                 self.on_start()
   79                 yield self.read_until_prompt()
   80                 self.is_started = True
   81             # Send command
Variables:
                self = <noc.core.script.cli.ssh.SSHCLI object at 0x7fb7ade33710>
             address = (u'10.31.10.115', 22)
------------------------------------------------------------------------
File: local/lib/python2.7/site-packages/tornado/gen.py (Line: 876)
Function: run
  869                         try:
  870                             value = future.result()
  871                         except Exception:
  872                             self.had_exception = True
  873                             exc_info = sys.exc_info()
  874
  875                         if exc_info is not None:
  876 ==>                         yielded = self.gen.throw(*exc_info)
  877                             exc_info = None
  878                         else:
  879                             yielded = self.gen.send(value)
  880
  881                         if stack_context._state.contexts is not orig_stack_contexts:
  882                             self.gen.throw(
Variables:
                self = <tornado.gen.Runner object at 0x7fb7b21eca90>
               value = <noc.core.script.cli.ssh.SSHIOStream object at 0x7fb7b21ec410>
             yielded = <tornado.concurrent.Future object at 0x7fb7b21ec890>
              future = <tornado.concurrent.Future object at 0x7fb7b21ec890>
 orig_stack_contexts = ((), None)
            exc_info =
(<class '_libssh2.Error'>,
 Error('SSH startup: Failed getting banner',),
 <traceback object at 0x7fb7ade25fc8>)
------------------------------------------------------------------------
File: local/lib/python2.7/site-packages/tornado/concurrent.py (Line: 215)
Function: result
  208             `concurrent.futures.Future` but it is an error to call it
  209             before the `Future` is done, so the ``timeout`` is never used.
  210             """
  211             self._clear_tb_log()
  212             if self._result is not None:
  213                 return self._result
  214             if self._exc_info is not None:
  215 ==>             raise_exc_info(self._exc_info)
  216             self._check_done()
  217             return self._result
  218
  219         def exception(self, timeout=None):
  220             """If the operation raised an exception, return the `Exception`
  221             object.  Otherwise returns None.
Variables:
                self = <tornado.concurrent.Future object at 0x7fb7b21ec610>
             timeout = None
------------------------------------------------------------------------
File: local/lib/python2.7/site-packages/tornado/ioloop.py (Line: 445)
Function: run_sync
  438             if timeout is not None:
  439                 timeout_handle = self.add_timeout(self.time() + timeout, self.stop)
  440             self.start()
  441             if timeout is not None:
  442                 self.remove_timeout(timeout_handle)
  443             if not future_cell[0].done():
  444                 raise TimeoutError('Operation timed out after %s seconds' % timeout)
  445 ==>         return future_cell[0].result()
  446
  447         def time(self):
  448             """Returns the current time according to the `IOLoop`'s clock.
  449
  450             The return value is a floating-point number relative to an
  451             unspecified time in the past.
Variables:
                self = <tornado.platform.epoll.EPollIOLoop object at 0x7fb7b21ec250>
                 run = <function run at 0x7fb7ade62758>
                func =
<bound method SSHCLI.submit of <noc.core.script.cli.ssh.SSHCLI object at 0x7fb7ade33710>>
             timeout = None
         future_cell = [<tornado.concurrent.Future object at 0x7fb7b21ec610>]
------------------------------------------------------------------------
File: core/script/cli/base.py (Line: 60)
Function: execute
   53                     socket.IPPROTO_IP, socket.IP_TOS, self.tos
   54                 )
   55             return self.iostream_class(s, self)
   56
   57         def execute(self, cmd):
   58             self.buffer = ""
   59             self.command = cmd
   60 ==>         self.ioloop.run_sync(self.submit)
   61             return self.result
   62
   63         @tornado.gen.coroutine
   64         def submit(self):
   65             # Create iostream and connect, when necessary
   66             if not self.iostream:
Variables:
                self = <noc.core.script.cli.ssh.SSHCLI object at 0x7fb7ade33710>
                 cmd = 'terminal length 0\n'
------------------------------------------------------------------------
File: core/script/base.py (Line: 566)
Function: cli
  559             Execute CLI command and return a result.
  560             if list_re is None, return a string
  561             if list_re is regular expression object, return a list of dicts (group name -> value),
  562                 one dict per matched line
  563             """
  564             command_submit = command_submit or self.profile.command_submit
  565             stream = self.get_cli_stream()
  566 ==>         r = stream.execute(cmd + command_submit)
  567             # Check for syntax errors
  568             if not ignore_errors:
  569                 # Check for syntax error
  570                 if (self.profile.rx_pattern_syntax_error and
  571                         self.profile.rx_pattern_syntax_error.search(r)):
  572                     raise self.CLISyntaxError(r)
Variables:
              nowait = False
              stream = <noc.core.script.cli.ssh.SSHCLI object at 0x7fb7ade33710>
              cached = False
          bulk_lines = None
                 cmd = 'terminal length 0'
      command_submit = '\n'
                file = None
             list_re = None
       ignore_errors = True
                self = <sa.profiles.Cisco.IOS.get_interfaces.Script object at 0x7fb7ade332d0>
------------------------------------------------------------------------
File: core/script/base.py (Line: 615)
Function: get_cli_stream
  608                     self.profile.setup_session(self)
  609                 self.to_shutdown_session = bool(self.profile.shutdown_session)
  610                 # Disable pager when nesessary
  611                 if self.to_disable_pager:
  612                     self.logger.debug("Disable paging")
  613                     self.to_disable_pager = False
  614                     self.cli(self.profile.command_disable_pager,
  615 ==>                          ignore_errors=True)
  616             return self.cli_stream
  617
  618         def has_snmp(self):
  619             """
  620             Check wherher equipment has SNMP enabled
  621             """
Variables:
                self = <sa.profiles.Cisco.IOS.get_interfaces.Script object at 0x7fb7ade332d0>
            protocol = u'ssh'
------------------------------------------------------------------------
File: core/script/base.py (Line: 565)
Function: cli
  558
  559             Execute CLI command and return a result.
  560             if list_re is None, return a string
  561             if list_re is regular expression object, return a list of dicts (group name -> value),
  562                 one dict per matched line
  563             """
  564             command_submit = command_submit or self.profile.command_submit
  565 ==>         stream = self.get_cli_stream()
  566             r = stream.execute(cmd + command_submit)
  567             # Check for syntax errors
  568             if not ignore_errors:
  569                 # Check for syntax error
  570                 if (self.profile.rx_pattern_syntax_error and
  571                         self.profile.rx_pattern_syntax_error.search(r)):
Variables:
              nowait = False
              cached = False
          bulk_lines = None
                 cmd = 'show vlan brief'
      command_submit = '\n'
                file = None
             list_re = None
       ignore_errors = False
                self = <sa.profiles.Cisco.IOS.get_interfaces.Script object at 0x7fb7ade332d0>
------------------------------------------------------------------------
File: sa/profiles/Cisco/IOS/get_interfaces.py (Line: 141)
Function: execute
  134             if self.match_version(uBR):
  135                 # uBR series
  136                 pvm = self.get_ubr_pvm()
  137             else:
  138                 vlans = None
  139                 for cmd in ("show vlan brief", "show vlan-switch brief"):
  140                     try:
  141 ==>                     vlans = self.cli(cmd)
  142                     except self.CLISyntaxError:
  143                         continue
  144                 if vlans:
  145                     for sp in self.scripts.get_switchport():
  146                         switchports[sp["interface"]] = (
  147                             sp["untagged"] if "untagged" in sp else None,
Variables:
               vlans = None
                 cmd = 'show vlan brief'
                 pvm = {}
                self = <sa.profiles.Cisco.IOS.get_interfaces.Script object at 0x7fb7ade332d0>
         switchports = {}
------------------------------------------------------------------------
File: core/script/base.py (Line: 179)
Function: run
  172                     result = self.get_cache(self.name, self.args)
  173                     self.logger.debug("Cache hit")
  174                     cache_hit = True
  175                 except KeyError:
  176                     pass
  177             # Execute script
  178             if not cache_hit:
  179 ==>             result = self.execute()
  180                 if self.to_shutdown_session:
  181                     self.logger.debug("Shutdown session")
  182                     self.profile.shutdown_session(self)
  183             # Clean result
  184             result = self.clean_output(result)
  185             self.logger.debug("Result: %s", result)
Variables:
           cache_hit = False
                self = <sa.profiles.Cisco.IOS.get_interfaces.Script object at 0x7fb7ade332d0>
------------------------------------------------------------------------
File: services/activator/api/activator.py (Line: 56)
Function: script
   49                 credentials=credentials,
   50                 capabilities=capabilities,
   51                 version=version,
   52                 timeout=timeout,
   53                 name=name
   54             )
   55             try:
   56 ==>             result = script.run()
   57             except script.ScriptError, why:
   58                 raise APIError("Script error: %s", why.__doc__)
   59             return result
Variables:
                name = u'Cisco.IOS.get_interfaces'
              script = <sa.profiles.Cisco.IOS.get_interfaces.Script object at 0x7fb7ade332d0>
        script_class = <class 'sa.profiles.Cisco.IOS.get_interfaces.Script'>
                self = <api.activator.ActivatorAPI object at 0x7fb7ade33bd0>
        capabilities = {}
             version = {u'platform': u'C3560', u'version': u'15.0(2)SE4', u'vendor': u'Cisco'}
             timeout = None
         credentials =
{u'address': u'10.31.10.115',
 u'cli_protocol': u'ssh',
 u'password': u'cisco',
 u'path': None,
 u'snmp_ro': u'fsonetro',
 u'snmp_version': u'v2c',
 u'super_password': u'cisco',
 u'user': u'cisco'}
------------------------------------------------------------------------
File: local/lib/python2.7/site-packages/concurrent/futures/thread.py (Line: 55)
Function: run
   48             self.kwargs = kwargs
   49
   50         def run(self):
   51             if not self.future.set_running_or_notify_cancel():
   52                 return
   53
   54             try:
   55 ==>             result = self.fn(*self.args, **self.kwargs)
   56             except BaseException:
   57                 e, tb = sys.exc_info()[1:]
   58                 self.future.set_exception_info(e, tb)
   59             else:
   60                 self.future.set_result(result)
   61
Variables:
                   e = Error('SSH startup: Failed getting banner',)
                self = <concurrent.futures.thread._WorkItem object at 0x7fb7ade33110>
                  tb = <traceback object at 0x7fb7ade27dd0>
------------------------------------------------------------------------
File: local/lib/python2.7/site-packages/concurrent/futures/_base.py (Line: 396)
Function: result
  389                     timeout.
  390                 Exception: If the call raised then that exception will be raised.
  391             """
  392             with self._condition:
  393                 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
  394                     raise CancelledError()
  395                 elif self._state == FINISHED:
  396 ==>                 return self.__get_result()
  397
  398                 self._condition.wait(timeout)
  399
  400                 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
  401                     raise CancelledError()
  402                 elif self._state == FINISHED:
Variables:
                self = <Future at 0x7fb7ade33c10 state=finished raised Error>
             timeout = None
------------------------------------------------------------------------
File: local/lib/python2.7/site-packages/tornado/gen.py (Line: 870)
Function: run
  863                         return
  864                     self.future = None
  865                     try:
  866                         orig_stack_contexts = stack_context._state.contexts
  867                         exc_info = None
  868
  869                         try:
  870 ==>                         value = future.result()
  871                         except Exception:
  872                             self.had_exception = True
  873                             exc_info = sys.exc_info()
  874
  875                         if exc_info is not None:
  876                             yielded = self.gen.throw(*exc_info)
Variables:
            exc_info =
(<class '_libssh2.Error'>,
 Error('SSH startup: Failed getting banner',),
 <traceback object at 0x7fb7ade27680>)
                self = <tornado.gen.Runner object at 0x7fb7ade33b10>
              future = <Future at 0x7fb7ade33c10 state=finished raised Error>
 orig_stack_contexts = ((), None)
------------------------------------------------------------------------
File: core/service/api.py (Line: 86)
Function: post
   79                     # Threadpool version
   80                     executor = self.service.get_executor(h.executor)
   81                     result = executor.submit(h, *params)
   82                 else:
   83                     # Serialized version
   84                     result = h(*params)
   85                 if tornado.gen.is_future(result):
   86 ==>                 result = yield result
   87                 self.write(json.dumps({
   88                     "id": id,
   89                     "error": None,
   90                     "result": result
   91                 }))
   92             except APIError, why:
Variables:
                 req =
{u'id': 82,
 u'method': u'script',
 u'params': [u'Cisco.IOS.get_interfaces',
             {u'address': u'10.31.10.115',
              u'cli_protocol': u'ssh',
              u'password': u'cisco',
              u'path': None,
              u'snmp_ro': u'fsonetro',
              u'snmp_version': u'v2c',
              u'super_password': u'cisco',
              u'user': u'cisco'},
             {},
             {u'platform': u'C3560',
              u'vendor': u'Cisco',
              u'version': u'15.0(2)SE4'},
             None]}
              kwargs = {}
                   h =
<bound method ActivatorAPI.script of <api.activator.ActivatorAPI object at 0x7fb7ade33bd0>>
                 why = Error('SSH startup: Failed getting banner',)
     calling_service = 'sae'
                args = ()
                  id = 82
                self = <noc.core.service.api.APIRequestHandler object at 0x7fb7ade33d50>
                 api = <api.activator.ActivatorAPI object at 0x7fb7ade33bd0>
              params =
[u'Cisco.IOS.get_interfaces',
 {u'address': u'10.31.10.115',
  u'cli_protocol': u'ssh',
  u'password': u'cisco',
  u'path': None,
  u'snmp_ro': u'fsonetro',
  u'snmp_version': u'v2c',
  u'super_password': u'cisco',
  u'user': u'cisco'},
 {},
 {u'platform': u'C3560', u'vendor': u'Cisco', u'version': u'15.0(2)SE4'},
 None]
              result = <Future at 0x7fb7ade33c10 state=finished raised Error>
            executor = <concurrent.futures.thread.ThreadPoolExecutor object at 0x7fb7b05ac650>
              method = u'script'
------------------------------------------------------------------------
END OF TRACEBACK