That is exactly the problem. I have found that if I wait a second after the handoff to take control of the RB, it eliminates most of those jukes.
I wish they would remove the animation from the left thumbstick and simply make it directional in control.....