How do I remove the passphrase for the SSH key without having to create a new key?

Short answer:

$ ssh-keygen -p [-P old_passphrase] [-N new_passphrase] [-f keyfile]

However, you might want to consider using ssh-agent, which can cache the passphrase for a time.

The latest versions of gpg-agent also support the protocol that is used by ssh-agent.