Ministers are urged to intervene amid rising Covid infection numbers at the Swansea office.