Brexit vote will not be delayed, say ministers, as critics urge the PM to renegotiate with the EU.