Last Updated Jun 4, 2014 5:11 PM EDT
A petition on the White House website calling for him to be court-martialed for walking away from his post in 2009 had up to 15,000 names on it Wednesday morning.
In Belgium, Defense Secretary Chuck Hagel said Wednesday that it would be unfair to rush to judgment on Bergdahl, who spent nearly five years as a Taliban prisoner of war.
CBS News' Jan Crawford reports the Army is saying the priority right now is to get Bergdahl back to health before it formally investigates and, if necessary, disciplines him.
But there's growing pressure to punish Bergdahl, much of it from the soldiers who served with him and from families who say they lost their sons because of his actions.
"It appears that he defected, based on the evidence," said Reesa Doebbler, who says that after Bergdahl vanished the Taliban stepped up its attacks, which she believes led to the death of her son, Staff Sgt. Clayton Bowen.
"After he disappeared, they had more strikes against them, and they expected them to be looking for him, so I think it endangered the soldiers that were there," Doebbler said.
Doebbler wants Bergdahl punished for desertion, but a court-martial could be complicated.
Merely showing that Bergdahl left his post is not enough, according to former military prosecutor Eugene Fidell.
"Maybe he had a change of heart and the Taliban prevented him from returning the day after he defected," said Fidell. "We don't know."
Investigators will also want to know more about videos of Bergdahl taped by the Taliban and used as propaganda to make sure he wasn't aiding the enemy.
There is precedent for prosecuting former prisoners of war.
Charles Jenkins deserted his post in South Korea in 1965, only to be captured by the North Koreans.
In 2004, after they finally let him leave, Jenkins turned himself in to U.S. forces.
He was court-martialed and sentenced to 30 days confinement.
Gen. Martin Dempsey, the chairman of the Joint Chiefs, suggested Tuesday that nothing was off the table: "Our Army's leaders will not look away from misconduct if it occurred."
Spc. Cody Full, who was in Bergdahl's platoon, said if Bergdahl escapes punishment it would send a dangerous message.
"Once you go to a combat zone, you're supposed to stay there and protect your brothers and be with your brothers and upheld your mission, and for whatever reason if you don't like what's going on you can just walk off and you won't get in trouble at all," Full said.
If Bergdahl was court-martialed and convicted Fidell said the maximum penalty would be five years in prison.
CBS News' Bigad Shaban reports the backlash overshadowed plans to celebrate Bergdahl's return in his hometown of Hailey, Idaho.
Late Wednesday, the city canceled the event planned for June 28, saying it was done "in the interest of public safety" because of the high number of supporters and protesters expected to turn out.
"Hailey, a town of 8,000, does not have the infrastructure to support an event of the size this could become," city officials said in a statement.
The controversy had come as an unwelcome surprise to the mountain town, where the celebration over the 28-year-old soldier's release hasn't waned and neither have the calls suggesting he was deserter.
Heather Dawson is the city administrator of Hailey, where hundreds of angry calls and emails have been received at City Hall.
"Bergdahl is not a hero ... He should be executed," one caller said.
Dawson said, "Some of them said, 'Beware, be wary of what you're doing. Think through it carefully.'"
What they were doing was planning a celebration to honor Bergdahl and his family.
"We were so unprepared for the negativity," said Dayle Ohlau, a friend of the Bergdahls who worked on the event.
One email to City Hall threatened, "If your community decides to celebrate this deserter and ignore the known deaths caused by your deserter ... Idaho will suffer economically by having boycotts by military (wounded warriors) and tea party groups."
The backlash also extends to the soldier's father, Bob Bergdahl, who used social media to try and speak to his son's captors.
"I was so impressed by how he immersed himself in the culture, how he learned the language and spoke Pashto so that in some way, in some miraculous way, if he put those letters and conversations and interviews out there, somehow Bowe might get it," Ohlau said.
Last week, he apparently tweeted a message to a Taliban spokesman saying, "I am still working to free all Guantanamo prisoners. God will repay for the death of every Afghan child."
That tweet has since been deleted.
"I think he feels a real kinship with the people of Afghanistan and the kind of faith-based people they are, and I think that resonated with him," said Ohlau. "But being a Taliban sympathizer? No, I never got that from Bob."
Bob Bergdahl and his wife, Jani, remain in their secluded home in Hailey. A military spokesman assigned to them said they have no plans to speak publicly or address the mounting criticism facing them and their son.