Skip the Hamptons. Just getting to Long Island will ruin your vacation. You'll want to give th NYC metro area a wide berth. We head northwest out of Fredricksburg VA and pick up I-81 to at least Harrisburg PA then east. You can do I-78 or I-80 to I-287 around the metro then across the Tappn Zee or Newburg bridge to pick up I-84, or continue up I-81 to Scranton to I-84. Do not do I-95 through New Haven. There's been road construction there for 20 years. Also, I-95 east of New Haven is still 2 lanes and very congested.
My favorite way to get to Mystic is to get off I-84 at I-691 to CT66 and then south on CT 9 to Old Lyme. Aces High in East Lyme is a nice park and close to Mystic.