I see no evidence that freezing hurts the memory foam, but I do know that memory foam has much thermal mass, and can be slow to warm and soften up. On our house Tempur-Pedics, they were delivered cold from a warehouse, and the delivery crew was spot on when they said it would take days for the mattress to warm and soften. Googling just now, I saw an RVer say their 3" pad was taking 5-6 hours to soften from rock hard.
--SiletzSpey