Cheryl, 31, Nick Grimshaw, 30, and 24-year-old Rita Ora have been ganging up on 55-year-old Simon – with a little help from new presenters Olly Murs and Caroline Flack.
Olly, 31, told the Daily Star: "The girls are giving Simon a really hard time. In one session at the auditions they actually made him change his mind on a few of the acts. Grimmy has been giving him loads of stick as well."
Singer Olly admitted he and Caroline, 35, have been joining – and joked Simon will probably fire them if they carry on.
He continued: "Me and Caroline call him Uncle Simon backstage. I'm not sure he likes it very much. We might get sacked before the lived shows."
It seems Cheryl and Rita are definitely breaking the tradition of the female judges feuding.
Rita said: "It's been hilarious ganging up on Simon. It's been great having Cheryl by my side."
While her bestie added: "We've all been going for Simon. He's less of a big brother now and more of an old uncle."
Poor Simon – something tells us it won't be long before he's getting his own back…