For recording I use VirtualDub (http://www.virtualdub.org/). After recording I render the clips with VirtualDub: deinterlacing with yadif (with double framerate, top first), cropping out the black edges (null transform > crop) and resizing to 1280x720 (lanczos3).

For both recording and rendering you need the Huffyuv codec (http://neuron2.net/www.math.berkeley.edu/benrg/huffyuv.html).

The file you get when you render a recording is very big and cannot be uploaded to youtube (well it can, but it will take a very long time), so finally I use Sony Vegas to put the clips together and do the final render.

Vegas project settings:

Width: 1280
Height: 720
Field order: None
Pixel aspect ratio: 1,0000 (Square)
Frame rate: 29,970 (NTSC)
Pixel format: 32-bit floating point (full range)
Compositing gamma: 2,222 (Video)
Full-resolution rendering quality: Best
Deinterlace method: None
Untick [ ] Adjust source media to better match project or render settings

Vegas render settings (Sony AVC):

Video tab:
Video format: AVC
Frame size: High definition (1280x720)
Frame rate: 29,970 (NTSC)
Field order: None
Pixel aspect ratio: 1,0000
Bit rate (bps): 15 000 000

Project tab:
Video rendering quality: Best


For the best quality you also need a S-Video cable, but it still looks pretty good without it.

I also disable resample for all clips I drop in Sony Vegas (right-click on a clip > switches > disable resample).

DaCerebro