With some extra bracing on the press, I still could not take the pitman off, so I wrestled the box out, with the pitman attached. Using a combination of hammer and press the pitman came off very easy.
Turns out the previous owner also installed a reman box at some point.
Now...where's my steering box postal service ?