I'm going to second that it may be a weak wifi receiver in the TV. I couldn't use wifi to stream on my PS3 at my last house. I bought a powerline ethernet hub and that solved the problem. My router is now next to the entertainment center and wifi works.
Are you doing the speed test on a browser on the TV or another device? Try a speed test from the browser on the TV - of course it still comes down to a reliable signal.
Another possibility is that the TV has a low cache which doesn't affect quality during non-peak but does hit you during peak.