Mike Richards, who was to be the new host of popular game show Jeopardy!, apologized for his jokes in a 2013 podcast. Still, the Anti-Defamation League (ADL) is calling for an investigation into his “pattern” of comments nonetheless. But before any investigation could be launched, Richards announced that he’s stepping down from the game show hosting job.

On Thursday, the Washington Post published an article reviewing the content of Richards’ podcasts released beginning in 2013, many of which featured the longtime Jeopardy! producer making off-color jokes and comments. Immediately after the Post’s expose was published, the ADL posted a statement on Twitter blasting Richards.

“New ‘Jeopardy!’ host Mike Richards’ disparaging remarks about Jews, women & Asians are no laughing matter,” the statement read. “Stereotyping is an entry point to hate, and his apology lacks acknowledgment of its harm. This reported pattern warrants an investigation.”

During his 41-episode The Randumb Show podcast released between 2013 and 2014, Richards often lapsed into lowbrow joking with his co-host Beth Triffon. His jokes included comments about women’s bodies, Jewish stereotypes, and other uncouth quips.

For instance, in one episode he called Triffon a “booth slut” for a modeling job she took. In another, he made a crude joke based on the stereotype of Jewish noses. In another bit, he made fun of women’s bodies in a photo of his co-host and her friends in bathing suits, calling them “frumpy and overweight,” according to The Wrap.

Richards was named the game show’s permanent replacement for beloved host Alex Trebeck just last week. But in response to questions about the decision process, former Big Bang actress Mayim Bialik was also named a host.

The producer initially apologized for the “terribly embarrassing” old podcasts:

“It is humbling to confront a terribly embarrassing moment of misjudgment, thoughtlessness, and insensitivity from nearly a decade ago. Looking back now, there is no excuse, of course, for the comments I made on this podcast, and I am deeply sorry. The podcast was intended to be a series of irreverent conversations between longtime friends who had a history of joking around. Even with the passage of time, it’s more than clear that my attempts to be funny and provocative were not acceptable, and I have removed the episodes. My responsibilities today as a father, husband, and a public personality who speaks to many people through my role on television means I have substantial and serious obligations as a role model, and I intend to live up to them.”

The apology did not seem to molly very many people. Late Show host Stephen Colbert, for one, tore into Richards on Thursday. Colbert scoffed about how coincidental it was that the producer of the game show was named its new host and flayed Richards for his jokes on the eight-year-old podcast.

Watch below: 

One of the off-color comments Richards made, for instance, was when he asked his female co-host if she took naked selfies, or as he called them, “booby pictures.” Colbert mocked Richards for this.

“‘Booby pictures?'” Colbert said. “Is this the man about to become the host of America’s most beloved quiz show or a 12-year-old boy trying to sneak into an R-rated movie?”

Colbert also dismissed Richards’ apology statement, insisting that the posting was like saying, “I know I drove drunk and plowed through your rose garden, and to prove I’m sorry, I recycled my empty liquor bottles.”

By Friday, it appears that the pressure had become too much, and Richards announced that he would not be taking the job as Jeopardy! host.

“It has become clear that moving forward as host would be too much of a distraction for our fans and not the right move for the show,” Richards wrote in a statement. “I want to apologize to each of you for the unwanted negative attention that has come to Jeopardy! over the last few weeks and for the confusion and delays this is now causing. I know I have a lot of work to do to regain your trust and confidence,” he added.

Production on the show was halted on Friday in light of the controversy. Richards also announced that a search for a permanent host would resume. It appears he intends to stay on as producer.

Follow Warner Todd Huston on Facebook at: facebook.com/Warner.Todd.Huston.