Are you asking about doing damage to the battery? Or damage to the camper? Or damage to the solar charge controller?
Battery should be fine, if the 100 watt panel gets at least some good direct sun throughout the day then I would have left the battery installed because the 100 watt panel would be fulfilling its purpose. The presumably healthy battery would be getting topped off daily by the sun/solar. The controller would prevent overcharging the battery.
Camper should be fine.
Solar charge controller may not be happy though, the protocol is almost always connect the controller to the battery first to power it on and then connect solar input to the controller. When disconnected the protocol is reversed, disconnect solar input to controller and then disconnect controller from the battery. Right now it sounds like you have the solar input hooked up to the controller but no battery. I don't know that this means it gets immediately fried or what, but a charge controller is not meant to have solar input connected with no battery to direct the current to.