You will likely ruin the charge controller. I have not seen a controller that does not warm to not have a battery hooked up before connecting the solar panel.
That said, I have accidentally hooked up the panel with no battery with no damage to the controller. I do not know what the issue is or what can go wrong, but imagine without a battery there that the voltage levels can spike too high. The battery is able to absorb any stray voltages or odd voltages into its storage capacity rather than all that spike go into the electronics.
You will have to either put the switch between the panel and the controller, or not put any switches in.
Another option is to put cardboard or plywood over the panel (give it a few mins to depower) and then it should be safe to disconnect, or do this at night when there is no sun to power the panel. Don't let the cover blow off though.
I have a truck camper with the batteries on the truck, not the camper. When I would remove the camper then there were no batteries, so I put in a small exit sign battery that I could switch over to that the panel had a battery still connected, this was a post-controller switch.
Since then I have replaced the little battery with a regular marine battery, but still use the switch for the panel that is not usually connected to that camper battery.
