Personally, I like communication.
I would go up to the person who took the bricks, be friendly, and tell them that it's fine that they have the bricks, I didn't want them, but I feel a little weird that they did not ask me.
A couple of times I have been the person who wanted something that was apparently up for grabs but I was not 100% sure (stuff left NEAR the curb, but not right BY it). What I did: I took the stuff, and left a note on the door with my phone number, saying I would bring it back if they did want it. Nobody called me.