Downing Street says the PM had to hold cross-party talks or risk letting Brexit “slip through our fingers”.