gnome-keyring doesn't work as an ssh-agent correctly

you can hard-code

export SSH_AUTH_SOCK="$XDG_RUNTIME_DIR/keyring/ssh"
in ~/.bashrc to avoid the problem.

Also make sure the name of the key is user@hostname. In my case hostname did not much my current hostname because I had imported my key from another installation and gnome-keyring was refusing to use it.