I'm no expert, but I'd say it might be a matter of time. A man is only going to be on the ladder briefly, his weight being distributed over it for a relatively brief period of time. If you strap Jimmy Hoffa (in his pre-deteriorated state :D) to a ladder and head off for 5 hours, would it hold?
A kayak, even if only 50#, over a prolonged period of time might cause a failure. Especially if not secure and bouncing.
Just a thought. Again, I'm no expert. I was wondering the same thing. I have two kayaks, both around 70#. I was wondering how RVers take those things with them if not on a trailer. I certainly don't like the idea of lugging one up and down a ladder!! LOL