I've encountered this exact problem myself.

First, in most of the guides, you usually do the following:

Specify a monitor resolution, then provide it to gtf:
gtf 1024 768 60 (get the Modeline information for 1024x768 resolution at 60Hz).
In my case, it yields:

#1024x768 @ 60.00 Hz (GTF) hsync: 47.70 kHz; pclk: 64.11 MHz Modeline "1024x768_60.00" 64.11 1024 1080 1184 1344 768 769 772 795 -HSync +Vsync

Create the new mode:
xrandr --newmode "1024x768_60.00"  64.11  1024 1080 1184 1344  768 769 772 795 (DO NOT include the trailing -HSync +Vsync . Some guides tell you to do this, but it will break the configuration for some reason).

Now you should be able to add the mode to a new display:
xrandr --addmode VBOX0 1024x768_60.00

Set the new mode for the device: xrandr --output VBOX0 --mode 1024x768_60.00

If step 3 failed still (these steps work for my laptop screen which is 1680x1050, but for some reason not for my external monitor which supports 1600x1200. These steps do work for resolutions up to 1280x1024 for my external monitor though. Weird), you can still try letting xrandr use auto mode. In my case, it allowed me to have my laptop screen and external monitor working perfectly. The script I use is attached below: