It is not a good idea to "back feed" (which you are really not doing)
Here are some reasons why.
Many controllers state that they will burn out if they are connected to the panels but not connected to the battery bank.
When in storage you would want to disconnect the battery bank from the RV. Wiring directly to the bank allows it to be charged while in storage mode.
Wire from the controller to the battery bank may be larger than what is used from the bank to the distribution box.
The charge controller ought to be as close to the battery bank as possible.