I don't think you will get it off without pulling the whole column. Just did a 67 Barracuda. It was a floor shift car but the concept is the same. The tube that goes through the center of the column was press fitted into that housing - did not see anyway of separating the 2 without damage but there was no reason to. Not sure what holds the tube at the bottom of the column (in engine compartment) but I suspect something does and that is preventing you from pulling it out.
At least that is how it was on the Barracuda column.
If you have a copy of the Factory Service Manual for your Charger, there is a complete section regarding the removal and installation of the steering column assembly.
Richard
For some unknown reason, nobody seems to take me seriously............