We have succesfully integrated Ranorex with Jenkins. But I think your problem is the SUT -> System under test. We have a Jenkins Slave running on a VM, with exactly the same settings as the end user would have on his System. This is our 'test client' or SUT. The test Client has a 'test user' and the Desktop should not have a screensaver or a locked Screen. The Desktop and the application should be not be in a 'shadow mode' as you mention. To recognise the controls, Ranorex needs the application to be run and the controls should be displayed on the screen.
The Jenkins job actally calls a batch file located on the slave.
The Jenkins Slave is a Windows Service and there is a autologon for the test user on the VM because incase we have windows updates then the Workstations are all automatically restarted. When there is no autologon then the tests don't run.
Ask general questions here.