The 30-year-old is hotly tipped to replace Dermot, 41, when the show returns later this year, after he quit last week.
Speaking at the launch of his waxwork at Madame Tussauds in Blackpool, Olly said: “It’s sad to see him leave.
“Dermot was there from day one. Dermot is a personal friend of mine and is someone I get on really well with.
“I have known him since 2009 when I did the show as a contestant. It is obviously a massive shame, he has done an incredible eight years but I think it is now time for a new chapter.”
While Olly, who is one of the show’s most successful acts, refused to confirm that he’ll be taking over from Dermot, he did hint that there is some truth behind them.
He said: “It’s rumours. Everyone is speculating about it. I have presented the Xtra Factor before with Caroline [Flack] and I loved it.”
Last week Dermot took to Twitter to announce his departure from the ITV show, writing: “Good afternoon. After eight wonderful years on @TheXFactor it’s time for me to move on.
“I’d like to thank ITV, Simon, The X Factor family and particularly the viewers, all of whom have been a big part of my life for so long.”
It is thought that if Olly does present the show, he will be joined by Strictly champ Caroline.