Switching to non text panels like the git sidebar is something I've just not figured out well. I can get there with a keybind, but focus is never in the right place to allow me to navigate the git sidebar with the Vim controls it already supports.
If this could be solved I think I would make the permanent switch.