Calls for the Conservatives to take further action as the ex-foreign secretary stands by his comments.