Also if the rivet is too long for the application you'll also have to slide the tool down the nail at least once after a couple of pulls. Just open the handles as wide as they go and slide it tight against the pop rivet and start to pump again.
Best thing to do is just practice "popping" a pop rivet. Or even just punch a hole in a can and then put a pop rivet into that hole and practice "popping" the rivet.
Sorry your first attempt didn't work out. Now that the hole is enlarged you'll need to use one of those smaller molly bolts in it's place. Use pop rivets for the rest.
Also the pop rivet size should be no longer they 1/8-1/4 past the void. If it is a very long pop rivet the mushroom part will not occur correctly.
Good luck.