The upcoming elections are going to be difficult for the Conservatives, says the education secretary.