The 12 volt outlet in the TV plate is not a high current device, Normal current to that switch is on the order of LESS THAN ONE AMP and many are thus wired with fairly small wire and a very small fuse, With a 100 watt inverter, i'm surprised you did not blow said fuse, i'd fuse it at about half that if I were designing.
Suggestion.
Replace the wall plate with a Sensar Pro (Way better at sucking in ATSC broadcast TV, and put in a dedicated 20 amp line (at least 10ga wire) and outlet. Run the 10 GA all the way to your fuse box and put in a fuse just for it, or if the box (unlike mine) is fully populated, run it to the battery and put in a fuse there.