I called a bunch of banks directly, and none of them would offer me a better interest rate on an RV loan then the dealer got me. I was a bit annoyed by this, as I never had an issue with my banks matching or beating car loan numbers before.
My loan is with a bank, the dealer doesn't "hold the papers", the bank does.
That said, I don't like the $1k deposit that they keep if you decide you don't like the interest rate they end up getting.
That sounds real shady, like they aren't going to try to get you a good rate because you will feel committed to getting it and cave to a worse rate so you don't lose your deposit.