Westerners flocking to join jihadist groups in Syria are a growing concern for Western intelligence agencies – and U.S. intelligence officials in particular.

Recent U.S. intelligence estimates suggest that more than 100 Americans have joined the jihadist cause, The Daily Beast‘s Eli Lake reports. As many as a dozen of them have returned to the United States, and intelligence officials are having trouble keeping track.

“We know where some are,” a U.S. intelligence official told The Daily Beast. “The concern is the scale of the problem we are dealing with.”

At least two Americans are known to have been killed fighting alongside the jihadists. Last July, a Pittsburgh man, identified as Amir Farouk Ibrahim, was reportedly killed fighting in Syria. Nicole Lynn Mansfield, a white convert to Islam from the Detroit area, was killed while fighting in June.

Some such as Nicholas Teusant, who was charged by the FBI in March of plotting to fight alongside the Islamic State of Iraq and Syria (ISIS), have been intercepted before they reached the battlefield.

It’s not a problem limited to the United States. An estimated 3,000 people from an assortment of European nations are believed to have joined the jihadists in their fight against the Assad regime.

National security officials worry about what happens when these people return home from the battlefield.

In February, Homeland Security Secretary Jeh Johnson warned that American jihadists returning from Syria posed a serious threat to homeland security.

U.S. intelligence officials particularly worry that jihadists who are traveling on U.S. passports or those issued by other Western nations could slip through the cracks, Lake reports. There are simply too many people for intelligence agencies to track them all.

The story points to Faisal Shahzad as an example. Shahzad, a Pakistani-American who received training in Taliban training camps in Pakistan, later attempted to detonate a car bomb in Times Square in 2010.

A federal court charged former Army Private Eric Harroun with firing a rocket-propelled grenade alongside Al-Qaida’s Syrian affiliate, Jabhat al-Nusra in June 2013, after he posted videos of himself with fighting alongside the terrorist group.