Minorities often feel “like they are hitting a brick wall” at work, the prime minister says.