After presenting the first season in 2011, Steve, 38, was dropped in favour of Khloe Kardashian and Mario Lopez, who co-hosted the second season.
Now Steve has insisted Simon, 55, hasn’t attempted to keep things civil between them since his axing.
He told The Sun: “I don’t have a relationship with Cowell. Once you stop working with him, that’s it.
“I didn’t have much of a relationship with Simon when I was working with him. It would have been easier to be bitter about it, but it’s not who I am. I handled myself with grace.”
Despite having no regrets about his attempt at cracking the American market, Steve suggested he’s happier on home soil.
He added: “In TV things come and go and, when it went, I was left high and dry. It’s a case of, ‘be careful what you wish for’.”
While hosting The X Factor USA, Steve met his wife Phylicia Jackson – so it wasn’t all bad.
Discussing life with Phylicia, Steve told Digital Spy: “I feel calmer now I’m married.
“I love married life, if I knew then what I do now I might have done it before. But then again, that means I wouldn’t be married to Phylicia. But I love it.”