Uninstall CamStudio, no matter the version, as well as any lossless codec you may have. Remove any ffdshow codecs you have as well, and install these old ones: http://sourceforge.net/projects/ffdshow-tryout/files/Old%20releases/generic%20builds/ffdshow_beta6_rev2527_20081219.exe/download

Then, re-install CamStudio 2.0, don't install 2.6 (any of them, they all will not display ffdshow.)

Then, when in video settings, select ffdshow, and then click configure. You will want to have the MSMPEGv4 selected from the bigger dropdown. Set the slider for ffdshow to be based on quality, not quantizer or bitrate, and slide the slider to 100.

Also, use the chart at the top of this thread: http://www.aeriagames.com/forums/en/viewtopic.php?t=1269325

For what the system requirements are for recording high-resolutions.