Removing JAVA is probably one of the single best security things you can do. I removed JAVA from my home PCs and have not run into any issues. I'm working on a project at work right now to remove JAVA from 1200 PCs. We've been testing and so far no ill effects.
If you find you do need JAVA, uninstall all versions, then only install the latest secure version. Then keep up with it. Use
Secunia's free software to scan for vulnerable software and help keep JAVA up to date and secure.
You may also want to remove Acrobat Reader and replace it with
PDF Xchange Viewer. While it does occasionally has vulnerabilities, Tracker Software is quick to patch them.