Off-resonance regimes in nonlinear quantum Rabi models