The ex-cabinet minister says Downing Street’s words “legitimise” aggression, a claim it denies.