If no video files are being created by VBOX Sim, there are a number of steps that can be taken to try and rectify the issue;
Ensure PC Based or External Device has been selected.
If using an external video capture card, check that the capture card software has been installed correctly.
If using an external video capture card, check that the external video capture card is correctly plugged into the computer via USB and HDMI.
If using the PC-based built-in screen capture function, ensure the correct display has been selected.