They are among about 55 Australians, believed to have links to overseas militants, who are forbidden to travel.
However ASIO and the federal government have not gone as far as issuing more onerous "control orders" - which include a curfew and movement restrictions - such as those imposed on Jack Thomas and David Hicks in 2006 and 2007 respectively.
Both orders were lifted after 12 months, with former Taliban fighter Thomas cleared of terrorism charges in 2008 and the federal government choosing to drop the condition on Hicks a year after he walked free from an Adelaide prison following his stint in Guantanamo Bay.
Passports were confiscated from a number of Australians and dual nationals on the basis of suspected espionage, a security source said.
Many of the suspects came from, or had travelled to, Afghanistan, Somalia, Yemen or Pakistan.
Australian National University's Dr Michael McKinley said, given it was a criminal offence to be a member of a proscribed terror group, the challenge for authorities was identifying people sympathising with those organisations and exhibiting the potential to act.
"There's an intelligence input into those assessments from police and intelligence organisations and that would include foreign intelligence organisations who will identify someone who is a risk," Dr McKinley said.
He said taking their passport let people know authorities were watching.
Of particular concern to Australian authorities are people with links to splinter al-Qaeda groups and to the Somali-based terror group al-Shabaab and Lashkar e-Toiba, based in Pakistan.
It comes a year after The Daily Telegraph revealed a grandmother with arthritis, an English teacher, a Lakemba cleric and a surfie who converted to Islam were on a secret alert list drawn up by the US embassy in Canberra.