But the ex-Brexit secretary thinks that could prompt a better deal being negotiated by the UK and EU.