How "small" panel? Converter wires are probably #12, so... you can wire controller to converter output, if panel is 50-60W. 100W panel (which is still "small", in my books) will work fine most of the time, but sometimes will cause too much voltage drop (6-8ft to converter + another 15-20ft to battery), and then your battery won't get a good charge.
This all should concern you less than the fact that controller-converter path usually doesn't have easy and/or obvious means to disconnect the controller before disconnecting the battery. If you forget, controller might get a heart attack - or it might survive. Better not to take chances.
You might want to run the numbers here:
Voltage drop
