The former foreign secretary stands by his remarks despite the Tory chairman telling him to apologise.