That changed in 2006, when the Forces nationales de libération (FNL) - the country's remaining rebel movement - signed a ceasefire agreement with the government, effectively opening up the province.
Residents say their lives have improved and thousands living in internally displaced people's (IDP) camps have returned home.
Not everyone is applauding yet, however. Most of the thousands of refugees returning from neighbouring countries and IDPs forced out of their homes by more than a decade of civil war have found they lack one vital life-sustaining tool: suitable land.
"We live in very poor conditions here," Moise Barekezabe, who also acts as the chief in a camp in the commune of Rukaramu, told IRIN. "The greatest handicap is that we have no land to cultivate; we have to beg from the local communities just to survive."
The camp, which hosts both returnees and IDPs, has 100 or so families living off 20x18m plots of land next to their two-roomed houses. On these little plots, they grow garden crops such as maize, sweet potatoes and vegetables.
In yet another camp, in the western province of Cibitoke, 32-year-old Emmanuel Niyonzima, said: "All I need is a piece of land to cultivate. I have three children who will soon be in school; how will I feed them if I have no land?"
Two years after democratic elections, more and more IDPs and refugees have returned to Burundi. There was already considerable pressure on the limited land - 27,830 sqkm for a population of about eight million - but the returnees have exacerbated the crunch.
According to a 2005 census of the displaced population conducted by the UN Office for the Coordination of Humanitarian Affairs (OCHA), there were at least 116,000 IDPs in the country. But no significant movement to their original homes has occurred since then.
According to a briefing document by the UN Refugee Agency (UNHCR), most IDPs in southern Burundi have expressed their willingness to return to their homes as soon as they have the means, but those in the north fear that doing so could revive ethnic tensions. UNHCR has facilitated the repatriation of at least 342,500 Burundians since 2002.
Reverien Simbarakiye, director of reinsertion of IDPs and refugees in the Ministry of National Solidarity, Human Rights and Gender, said Bubanza province had the highest number of IDPs, totalling 7,325; followed by Bujumbura Rural with 4,141 and Cibitoke with 3,381.
However, Simbarakiye told IRIN that these figures date from April 2005. "We have not been able to carry out another census for lack of financial means," he said.
The ministry, he added, was planning another census of IDPs this year but had noted that many also feared going back to their areas of origin because of security concerns.
"They tell us the FNL has not yet completely laid down weapons; others say their former neighbours who killed their relatives are still there and could kill them as well if they returned home," he said.
Others, he added, simply stayed on in the camps because they have no homes to return to.
According to Simbarakiye, Bujumbura Rural, Cibitoke and Bubanza provinces suffered the most from the civil war. The different armed groups, including the now ruling Conseil national pour la défense de la démocratie-Forces pour la défense de la démocratie (CNDD-FDD) and the FNL operated in these provinces for many years, subjecting residents to frequent displacement.
Facing up to the challenges
Besides land, the other main concerns for most IDPs and returnees are access to healthcare, lack of water, sanitation, lack of food and schooling for their children.
In the commune of Nyabiraba in Bujumbura Rural, Deogratias Ntikazohera, the commune's secretary, explained: "The problems they encountered are the same problems that all the people here are facing: the banana disease that is affecting our harvests and the disease that is also causing the spoiling of sweet potatoes."
According to Simbarakiye, the government plans to resettle IDPs and other war-affected people and those with land would be helped to build homes. Those without anywhere to go or who feared returning to their villages would be resettled in reinsertion sites.
"The plan is to resettle IDPs, returnees and those who remain in the villages together," he said.
One proposed resettlement site is Muyange in Bubanza province, where the governmental Project d'Appui au Rapatriement et à la Reintegration des Sinistrés (PARESI) has started clearing the ground for IDPs and returnees to build homes. Another site has been identified at Buhomba in Bujumbura Rural.
Simbarakiye, however, said the plan would not be easily implemented as it requires significant financing - and land.
There are at least 350,000 Burundian refugees in Tanzania and another 17,000 in the Democratic Republic of Congo (DRC). It is expected that most will return home, compounding the land pressure.
Burundi also hosts thousands of refugees from neighbouring countries. The majority of Congolese are from eastern DRC but there are others from Rwanda, for instance.
According to UNHCR, two camps for Congolese refugees have reached capacity. These include the camp at Gasorwe in the northeastern province of Muyinga, with at least 8,000 refugees, and Gihinga, in the central province of Mwaro, with about 2,400.
"Many of those transferred to these camps are urban refugees - who have been in the country for a while - and are seeking to be transferred to camps," the agency's associate external relations officer, Andreas Kirchhof, said.
The numbers of those seeking transfer had increased lately, he added, because of false information that they would be relocated to industrialised countries.
This rumour, Kirchhof added, was sparked by a two-month operation that ended in mid-May whereby 550 Congolese refugees were relocated to the United States. This group comprised mostly ethnic Banyamulenge Congolese, who survived an attack on Gatumba refugee camp in August 2004. The camp is in western Burundi near the DRC border. The FNL claimed responsibility for the attack.
"Unfortunately, UNHCR has limited offers like this and this relocation is now closed," Kirchhhof said. "What we are experiencing now is that many other refugees hope that by going to the camps they can get to relocate to an industrialised country."