From the RVM support site:

RVM installs everything into ~/.rvm. To remove RVM from your system run `rm -rf ~/.rvm`. You may have one additional config file in `~/.rvmrc` and of course the RVM hook in your `bash/zsh` startup files.

So, just go to the command line and type:
`rm -rf ~/.rvm`

All the installed gems are in the ~/.rvm folders, so doing the above will remove the gems and installed rubies in one go.

Gems you added pre-RVM with the default ruby install can be removed by typing this at the command prompt:

`for x in gem list --no-versions; do gem uninstall $x -a -x -I; done`