Garang told Uganda's state-owned newspaper, The New Vision after arriving at Entebbe International Airport on Friday that he was going to deal firmly with the militias operating in southern part of his country.
The LRA rebels, based in southern Sudan, have fought the Ugandan government in their 19-year rebellion, killing tens of thousands civilians and displaced over 1.4 million people in northern Uganda.
"Kony won't be hiding there for long. It is not only Kony, but also all the militias who have been operating in the area. We need to provide peace, security and stability, so the militias including those that were formerly supported by the government, must be disbanded," Garang said.
Garang arrived in Uganda for a brief visit during which he is expected to hold talks with President Yoweri Museveni.
Garang, formerly rebel leader of the Sudan People's Liberation Movement/Army was recently sworn-in with the government after 20 years of civil war.
He said the priority of his government was to resettle displaced people and remove camps of the internally displaced people and return the Sudanese refugees.
He said there were between 3 to 4 million Sudanese refugees outside the country who need to be returned and resettled.