I've gone thru several times without an issue but if they wanted, I'd say come on in as long as your Ok with four dogs nipping at your ankles.
I'd guess in general most of us recognize these Border agents/Agricultural agents have a job to do and whether you allow a search or don't, that's your choice, your no more nor less an American for doing what's right for you. Having a Law enforcement background, I can say with certainty, refusing a cursory search for fruits and vegetables would cause one to think, "what are you hiding"...that's just human nature...but once again, do what is right for you!