If you unplugged it an everything else still works, you should be fine if you are satisfied with the sound. In all likelyhood, it was not run off of a separate amp either, so it probably didn't have a draw when the sound system was off. But you may still need to find if it had a separate amp if you are worried about it drawing power. If it has a separate amp and that amp is powered when the system is off (unlikely, but possible if the person wiring it wired the turn-on trigger wire and the power wire together), just disconnecting the speaker won't stop that power draw. If it has a separate amp, it could be anywhere, but I'd check the compartment the speaker is in and the radio location first. When you find it, you should be able to just remove the fuse and don't have to remove the whole amp.
If the radio is reasonably easy to pull out, pull it and see if there is a subwoofer pre-amp output. If there is and there is an RCA type cord plugged into it, there is an amp somewhere for sure. If there is an output and it is not being used, the sub was probably running off of the internal amp in the radio.