ERROR SSH forwarding error: bind: Cannot assign requested

Hi! I would to explain in details my problem using juju. I have a Diablo installation, over Ubuntu 11.10 Oneiric. All working fine with nova so I decided to try the juju. Installed from repos, configured to access my private cloud. When I run "juju bootstrap" it creates a new instance and I can see that everything is working on it (I can see using euca-get-console-output that the instance installs a lot of packages, required by cloud-init).

 

When the instance achieve the running state I ran "juju status". The first time I ran that command I got "ERROR SSH forwarding error: bind: Cannot assign requested address" but I could see the return, showing my instance in machine section "0: {dns-name: 172.16.0.2, instance-id: i-0000001d}". The next times I run the command, I always get the error, showing no more the status.

 

So, I've been doing some tests/debug.

 

Although the error says something with bind one address, when I run the command with -v it outputs the port to be used. This port is not in use. I tried the command a lot of times and it changes the port, but the error continues. I could test this using the same port but with the ssh command line with "ssh -L port:host:2181 172.16.0.2" and it worked! So, with ssh (cli) I can log in to the instance but when juju try to use ssh I get the error.

 

I understood that juju try to open a ssh tunnel from the host to the instance, using a local port forwarded to port 2181 of the instance. Once I can log in using cli ssh, I went to the instance and checked the zookeeper. It was always running, everything fine.

 

So, I decided to run "tcptrack" and "tcpdump" on the instance to show me the connections attempts. When I run "juju status" I can see a connection, from the hosts where I ran this command to the instance, using port 22. This connection is very fast, Syn, some packages and then Close. I think this is the problem!! It occurs intermittently. The instance can be closing the ssh connection.

 

Analysing the /var/log/auth.log at the instance I could get two examples. The first is when the juju status got error. The second is when it worked.

 

172.16.0.1 -> host running juju command.

 

When didn't work:

Jan 27 08:42:06 server-29 sshd[14780]: Accepted publickey for ubuntu from 172.16.0.1 port 58905 ssh2

Jan 27 08:42:06 server-29 sshd[14780]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)

Jan 27 08:42:07 server-29 sshd[14780]: pam_unix(sshd:session): session closed for user ubuntu <<== 1 second after connected, get closed!!!

 

When it has worked:

Jan 27 08:39:43 server-29 sshd[14701]: Accepted publickey for ubuntu from 172.16.0.1 port 58873 ssh2

Jan 27 08:39:43 server-29 sshd[14701]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)

Jan 27 08:39:43 server-29 sshd[14701]: pam_unix(sshd:session): session closed for user ubuntu <<== first attempt, ERROR.

Jan 27 08:39:48 server-29 sshd[14704]: Accepted publickey for ubuntu from 172.16.0.1 port 58885 ssh2

Jan 27 08:39:48 server-29 sshd[14704]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)

Jan 27 08:39:49 server-29 sshd[14762]: Received disconnect from 172.16.0.1: 11: disconnected by user <<==2=

Jan 27 08:39:49 server-29 sshd[14704]: pam_unix(sshd:session): session closed for user ubuntu

 

We can see that the first attempt got error but the second worked! See in the second indication (<<==2=) that who closes the connection is 172.16.0.1, in other words, the juju host. I understood that in this case it connected, ran the necessary commands and disconnected, as should be.

 

The question is... why does the instance close the connection a lot of times?

 

When this occurs I can see the localhost reseting the connection:

 

 

08:42:01.284071 IP 127.0.0.1.44357 > 127.0.0.1.58873: Flags [S], seq 197321408, win 32792, options [mss 16396,sackOK,TS val 18036057 ecr 0,nop,wscale 5], length 0

08:42:01.284094 IP 127.0.0.1.58873 > 127.0.0.1.44357: Flags [R.], seq 0, ack 197321409, win 0, length 0

08:42:01.785143 IP 127.0.0.1.44359 > 127.0.0.1.58873: Flags [S], seq 310817086, win 32792, options [mss 16396,sackOK,TS val 18036182 ecr 0,nop,wscale 5], length 0

08:42:01.785166 IP 127.0.0.1.58873 > 127.0.0.1.44359: Flags [R.], seq 0, ack 310817087, win 0, length 0

 

In my installation there is a node with the nova services, like api, network, compute and so on. This host has the IP 172.16.0.1 that is the gateway to the instances network. So, the connectivity from this host to the instances is direct.

 

Any suggestions?

 

Thanks.