Ex-chancellor says Tories were wrong over immigration – which helped lose the 2016 referendum.