The judge overseeing an investigation into the early-morning robbery of U.S. swimming champion Ryan Lochte and three colleagues during the Rio de Janeiro Summer Olympics has banned the Americans from leaving Brazil and called for their passports to be confiscated. The order comes one day after Lochte flew back to the United States, ending his Olympic trip early.

Brazilian magazine Veja reports that the judge, Keyla Blanc, also issued warrants for the search of the Olympic Village residences of Lochte and fellow American swimmer James Feigen, who reported to crime to police. It is unclear what police will find there, as Lochte is no longer in the country and Feigen moved out of the Olympic Village to a hotel shortly after the incident. The newspaper O Globo cites authorities stating that the main reason for the warrants is to find and sift through Feigen’s mobile phone for evidence of where the swimmers were on Saturday night, to see whether they can match Feigen and Lochte’s testimonies with concrete evidence of their whereabouts.

News broke on Sunday morning through Lochte’s mother, Ileana, that her son and three other swimmers – Feigen, Gunnar Bentz, and Jack Conger – had been held up at gunpoint on their way back from a party in Rio’s Laguna neighborhood. According to Lochte, robbers wearing police badges demanded they get out of the car and on the floor, and put a gun to his head when he refused. They took hundreds of dollars from the swimmers, but no personal belongings.

The International Olympic Commission (IOC) and it’s US subsidiary both denied the story, with the USOC claiming to quote Lochte. They later apologized when Lochte himself confirmed the robbery.

Brazilian officials, meanwhile, have consistently downplayed the story. Officials tell the Associated Press they have found no evidence to back up the swimmers’ claims. Police claim the swimmers gave contradictory testimony – Lochte claiming they were pulled over, while Feigen says the taxi pulled over before the “police” arrived – and that their inebriated state casts doubt on the story.

With video surfacing of the athletes returning to the Olympic Village that night, in which they appear fully functional as they pass through security, authorities are now saying the opposite – that the swimmers were too sober and acting drunker in interviews than they actually were.

Brazil’s Minister of Sports said that security at the Olympics had been “efficient” and the incident occurred because of the “inappropriate” time that the swimmers were outside of the Olympic Village. He did not address the multiple assaults that have occurred inside the Olympic Village, including an attempted robbery against the head of Olympic security.

There is no evidence that Brazilian police have investigated local taxi companies or have any leads regarding who drove the swimmers home, or that they are engaging in an internal investigation. Brazilian police – especially in Rio de Janeiro, where they are often not paid their salaries for months – have been known to use their uniforms and badges to rob foreigners, so there is no guarantee the assailants in this case were police impersonators.

USOC spokesman Patrick Sandusky is now claiming that reports that Brazilian officials are trying to detain the swimmers are false. “There was no effort to detain anyone, but police did have further questions this a.m. It is a matter for our consulate and U.S. citizen services and we will continue to cooperate with all involved,” he said in a statement. Lochte’s attorney has confirmed he is in the United States, and Lochte himself issued a statement to USA Today saying that the swimmers initially only told family about the situation, and not police, because “we were afraid we’d get in trouble.”