If all the panels are around the same output voltage, you will be fine.
WHat I did is hook up my pair of 45 watt panels and 75 watt panel to one #10 gauge wire going to the controller, and the new pair of 120 watt panels to another #10 wire going to the panels. This way if there is a slight difference in the voltage output, it all comes together at the controller. I have a SB 50 amp controller that is MPPT.
So if one is say 21.2 open circuit and the other is within 1/2 volt of that, you will be fine.
If one set of panels is say 21.2 volts, and the other is say 42.0 volts, then a separate controller on each voltage panels is required, but still both can be hooked up to the same batteries.
Fred.