You need a sufficiently well calibrated/trusted voltage source.
A normal AA cell is not that precise. Wall voltage varies as well, within limits. Perhaps the best simple method for a quick check is to find someone who has a good, trusted meter and compare readings--preferably someone with one that has been calibrated and certified by a proper calibration lab, if absolute accuracy is required.
If you want to build a calibrator yourself, you might start with a precision voltage reference chip. Something like
this TI LM4040 variant looks promising; that, plus a current limiting resistor plus a voltage source > 10V would give a 10V reference that's within 0.1% or so under any reasonable operating condition, quite sufficient to calibrate or check inexpensive and not overly precise multimeters. This is not at all NIST-level precision.
There are many precision voltage reference devices available at varying costs, outputs, degrees of precision, etc.
For many jobs, absolute accuracy is not essential; the actual values of 12.3V or 12.4V might not matter as much as the fact that there's a 0.1V change between them. For other jobs, of course, the actual true value of a reading may be rather critical.