The 150 experts -- including former White House science adviser Dr. Philip Rubin -- issued an open letter to the U.N. health agency, calling for the games to be delayed or relocated "in the name of public health."
The letter cited recent scientific evidence that the Zika virus causes severe birth defects, most notably babies born with abnormally small heads. In adults, it can cause neurological problems, including a rare syndrome that can be fatal or result in temporary paralysis.
The authors also noted that despite increased efforts to wipe out the mosquitoes that spread Zika, infections in Rio have gone up rather than down.
Several public health academics have previously warned that having hundreds of thousands of people head to the Aug. 5-21 games in Brazil will inevitably lead to the births of more brain-damaged babies and speed up the virus' global spread. Most people infected by Zika suffer only minor symptoms including fever, a rash and muscle or joint pain.
WHO declared the Zika epidemic to be a global emergency in February and in its latest assessment this week, said it "does not see an overall decline in the outbreak."
"The fire is already burning, but that is not a rationale not to do anything about the Olympics," said Amir Attaran, a professor at the University of Ottawa and one of the letter's authors. "It is not the time now to throw more gasoline onto the fire."
WHO has already advised pregnant women not to go to Rio and says other travelers should avoid poor and overcrowded parts of the city. The U.N. agency also predicted the Zika risk in August would drop since it will be the south American winter and there should be fewer mosquitoes.
Zika can also be spread via sex in some cases; WHO recommends that pregnant women abstain or practice safe sex with partners returning from Zika-affected areas.
WHO Director-General Dr. Margaret Chan said earlier this month that the U.N. health agency is increasingly worried about Zika but stopped short of recommending the Rio Olympics be moved or postponed. Chan, who is not of child-bearing age, noted that she herself would be attending the games.
Among the letter's signatories are experts from more than two dozen countries in fields including public health, bioethics and pediatrics. The letter also noted a potential conflict of interest, highlighting the decades-long collaboration between WHO and the International Olympic Committee.
The authors said the "overly close" relationship "was last affirmed in 2010 at an event where the Director-General of WHO and president of the IOC signed a memorandum of understanding, which is secret because neither has disclosed it."
They also pointed to a group that WHO established to help cities not only with health advice, but to potentially help them bid for major events including the Olympics.
"WHO cannot credibly assess the public health risks of Zika and the Olympics when it sets neutrality aside," the letter stated.
WHO did not immediately respond to a request for comment Friday.
In an email to the AP, the IOC said it would "always consult the WHO for guidance and advice on health matters."
Concerns over Zika have prompted USA Swimming to move its pre-Olympic training camp from Puerto Rico to Atlanta and Major League Baseball also scrapped a series of games that were going to be held in San Juan.
No Olympic Games have ever been moved from their host city due to medical concerns, but in 2003, FIFA decided to switch the Women's World Cup soccer tournament from China to the United States on short notice due to the threat posed by the respiratory virus SARS.