Found out about this super useful git command for checking out the previous branch:

git checkout -

Not sure how I went this long without knowing that, but I'll be using it all the time now :D