What is holding Google back from taking it all?

A lot of the desktop licenses are sold to corporations. And those desktops need to be managed. Until Android comes up with a good enterprise/corporate story in that arena corporate IT depts aren't going to convert. And then if you change the OS you have to update the applications, which won't be cheap or easy. Not to speak of having to do training.

There are lots that have tried chrome and hate it since it is a resource hog.
I know I can see the browser that people use to access my blog and the vast majority hands down is IE with Firefox 2nd. As for mobile phone type access hands down is Android with iPad/iPhone a distant 2nd.

And there are privacy and security concerns as well

They'd have to make it work on non-ARM chips for one thing. While there is an Android x86 project it's not Google's.

I've never quite understood Chrome OS either.