My biggest mistake with Colemak was actually to completely switch to it on all keyboards, I find. I now use it (with some mods for Dutch) on my columnar keyboards, but keep Qwerty around on the built-in MacBook staggered keyboard and the old keyboard I had. It works fine: my hands feel the keys on a different place and just remember Qwerty or Colemak (including ALL my very bad Qwerty habits I am not going to fix anymore).

By using Qwerty on the side, I actually have an out when work is on fire, while doing more relaxed work on Colemak. The first time around I just had to quit Colemak because I was not fast enough, but with this duo setup the second time around I could make the switch just fine, and now I am around my normal typing speed again on both.

As for Vim... I did not change any bindings and even moved the arrow keys on the navigation layer to the new places of HJKL. My brain just adapted, despite their weird placements.