House Speaker John Boehner’s top allies are racing out the door to retirement, Bloomberg News reported on Thursday.

Top Boehner allies that are retiring in 2014 include four committee chairmen: Reps. Mike Rogers (R-MI) of the Intelligence Committee, Buck McKeon (R-CA) of the Armed Services Committee, Doc Hastings (R-WA) of the Natural Resources Committee and Dave Camp (R-MI) of the Ways and Means Committee. One of Boehner closest friends, Rep. Tom Latham (R-IA), is retiring this year, and another–Rep. Mike Simpson (R-ID)–may lose his seat in a primary in Idaho against conservative Bryan Smith.

“In a Capitol where accomplishments are built on relationships, some of Boehner’s most trusted confidantes won’t return next year, when lawmakers must again address the U.S. debt ceiling and possibly grapple with immigration policy,” Bloomberg’s Michael Bender and Derek Wallbank wrote. “Replenishing his support system will be crucial for Boehner as the limited-government Tea Party movement — which often equates deal-making and compromise with weakness — has complicated management of the House Republican majority.”

Notably, Simpson, a top Boehner confidant, suggested Boehner has yet to decide whether he will come back to Congress next year.

“That’s a challenge for Boehner if he decides to run for speaker again,” Simpson said about how will Boehner address the exit of many of his top supporters.

Some of Boehner’s other key allies do plan to remain in Congress next year. “Team Boehner is larger than it’s ever been,” Rep. Devin Nunes (R-CA) said, adding that those talking about Boehner’s retirement or departure as Speaker are “exotic members” who “want to be a spectacle and hurt the Republican Party.”

“He’s been able to win the confidence of this conference when we were in the majority, keep it after we lost the majority, and keep it during the creation of a new majority,” Rep. Tom Cole (R-OK) said of Boehner. “It suggests someone with enormous staying power on his part.”

But conservatives who want Boehner gone say they will succeed in showing him the door the easy way or the hard way. “We’re going to have a new speaker,” Rep. Tim Huelskamp (R-KS), a conservative who helped spearhead the failed coup against Boehner at the beginning of the 113th Congress. “Looks like a whole bunch of folks leaving who are key to him staying as speaker.”

Huelskamp was one of 12 Republicans who voted for someone other than Boehner at the beginning of this Congress in January 2013. Those seeking to execute the coup fell just five votes short of forcing a deadlocked vote on the first ballot, which would have set off a chaotic scramble.

Since then, Boehner has made a number of decisions that angered the right flank of the GOP conference, including pushing through the budget deal from House Budget Committee chairman Rep. Paul Ryan (R-WI) and Senate Budget Committee chairwoman Sen. Patty Murray (D-WA) that raised spending and increased the national debt, as well as with his focus on immigration. Other Republicans have expressed dissatisfaction with the House’s oversight efforts on the Obama administration, especially as it relates to the terrorist attack in Benghazi, Libya, on Sept. 11, 2012.

“I am slowly but surely developing the opinion that John Boehner does not intend to run for Speaker,” Rep. Mo Brooks (R-AL), someone who voted for Boehner both times he had an opportunity to since being elected in 2010, told Breitbart News in response to Boehner’s immigration principles. “Or, if he does, then it will be quite a challenge for him to accumulate the 218 votes needed to be elected Speaker. He’s just not acting like a candidate for Speaker who needs and wants 95 percent of the Republican conference vote to get to that 218 needed to be elected. It’s one thing to get a majority vote of the Republican conference. It’s another thing to get the 218 votes you need to actually be elected. That’s the number you need under the Constitution. He almost was not elected last time. Remember, these remarks are coming from somebody who voted for John Boehner twice.”

“My sense at the present time is that the Speaker doesn’t have the support of the conference,” South Carolina Rep. Jeff Duncan recently told Breitbart News.

Boehner also recently purchased a luxury condo in Marco Island, FL, prompting a new round of speculation about the issue by GOP insiders. Meanwhile, there is considerable chatter about whether Boehner’s top lieutenant Majority Leader Eric Cantor will stage a run for the Speakership in 2014.

Rory Cooper, a Cantor spokesman, recently told Breitbart News, “the Speaker of the House is John Boehner, period, and the Majority Leader is proud to call him a friend and have him leading House Republicans.”

Boehner has vowed both publicly and privately to return and has even filed re-election papers to run again.