Fuzzyfireball: Thank you for your help, but it seems both programs really hate Dosbox.
How do I record with DosBox? I hit Ctrl+Alt+F5 but I can't tell if it's recording. Can't find any files. Do I need the latest version of DosBox?
FRAPS won't record unless the game uses OpenGL. Most GOG games come with a video settings tool where you can choose OpenGL as the renderer. This is a requirement for FRAPS to recognize the stream, however it may or may not work for any game on an individual basis. I couldn't get it to work with Blood, which annoys me.
You can specify the folder DosBox saves captures to in the dosbox.conf file. Scroll down to the part that has something to do with video or capture.
Basically, for DosBox games your only options are 1) the built-in video capture feature or 2) run with OpenGL as renderer and use FRAPS.