I think I found the solution.
I was doing the initialization of XR manually directly after the project launched instead of activating it in the Projectsettings in XR Plug-in Management.
I have now checked the Initialize XR on Startup checkbox and have removed my code for starting XR manually.
I needed this in the past because I am developing a second app within the same unity-project. It just had different build settings but shared all the code and prefabs etc. Unfortunately the other project is no VR project and thus I added so code that would activate VR if needed
Thanks for the help