If the exhaust brake is a function of a VGT turbo then it is normal. A VGT acheives exhaust back pressure by closing the variable vanes all the way creating a back pressure between the pistons and the turbine wheel. The pressure is released over the turbine wheel and that spins the turbine wheel which in turn spins the compressor wheel and you get intake airflow and thus boost.
On an exhaust brake that is post turbo you will not build boost as the pressure is the same on both sides of the turbine wheel. One of the reason for the activation delay is to allow the turbo to slow down before the airflow decreases.
A compression release brake shouldn't build boost either as the exhaust flow shouldn't be sufficient enough to spin the turbo fast enough to build boost. The exhaust flow brake on is the same as brake off. If you turn the brake on and gain boost something is wrong.
So more info is needed. What engine/brake configuration.
