Do you have anything else to use to stream netflix as a test? It could be the netflix app in the tv. I normally use a PS3 but also a Roku. For 5 years I was DSL with maybe 10 down and didn't have buffer issues as a general rule. I'm 100 down now occasionally I can see the quality dip for a few minutes in the evening.
I do think the number of folks on NETFLIX is part of it with 50MB service, you shouldn't have a regular issue. Have you done a speed test to see what your actual download speed is?