7-14 psi turbo boost when engine compression brake engaged

I notice the same on my Powerstroke with no exhaust brake.
But on long grades, when I am not touching gas pedal for several minutes, the boost keeps going down.
It is at 10 psi on top of the hill, but after couple of minutes it drops to 4 psi.
I figured out it is the heat that air pumps from the engine into exhaust.
Since there is no burning fuel we don't have increase in coming gases to spin the turbine, but they do expand due to warming up.
Would be interesting if somebody can make other theory.