The SSN is the property of the SSA, and remains so. No assignment of the number can be treated as legally valid, if you have not accepted that assignment knowingly, thoughtfully, intentionally and voluntarily. Therefore the actions of the SSA do not affect your declaration of an unconscionable contract.
If the SSA’s failure or refusal to adjust its records to reflect that you have rescinded, and its failure or refusal ever causes you any damage, it may be held liable for said damage.