What are your EGTs at idle?
A boost leak is unlikely as airflow at idle is very very low. I have seen truck run without the turbo or intercooler hooked up at idle and you cannot even tell. EGTs might be a little higher but most likely not 150°-200°.
Any white smoke?
Inter cooler should have metal ends. The plastic ends was a 05 thing that was gone by 05.5.
A cracked exhaust manifold is possible, but it shouldn't effect the idle.
The smell is making me thing it is piston or injector related.
If you remove the oil cap and set it back on the hole does it rattle and have a bunch or pressure going by? Would be in pulses.