If the generator is running and someone turns on the inverter they will both feed the same bus and the smoke comes out of one or both.
Yes you could switch some breakers manually but then why bother with the ATS.
If you just need one branch circuit then it could come right off the ATS.
Might get by with two circuits off the ATS depending on loads.
Have you mapped all five circuits?
-------
As for wire I would just use the max that the terminals are rated. I recommend battery cable over speaker wire.
I get my provisions here:
http://www.genuinedealz.com/marine-wire/marine-battery-cableBest value is to have them put the correct battery lug on one end and sealed with heat shrink.