Amnesty International, Liberty and Privacy International filed a legal complaint with the court today.
The scale of the surveillance carried out by GCHQ has been revealed by US whistleblower Edward Snowden.
A similar legal challenge mounted in the UK last year saw judges rule that the spying did not breach human rights.
Secret court
"The UK government's surveillance practices have been allowed to continue unabated and on an unprecedented scale, with major consequences for people's privacy and freedom of expression," said Nick Williams, legal counsel for Amnesty in a statement.
The three organisations claim that the surveillance carried out by GCHQ breaches the European Convention on Human Rights that enshrines certain freedoms in law.
The surveillance carried out by GCHQ has been subject to a series of legal challenges since National Security Agency documents provided by Edward Snowden started to appear in the media.
In December, the Investigatory Powers Tribunal that oversees the work of the intelligence services ruled that GCHQ's spying did not violate Britons' human rights and was a legitimate way to gather intelligence.
In February, a separate ruling by the IPT found that the spy agency's surveillance programme was unlawful because the processes governing how GCHQ gathered and shared information were not public enough.
Amnesty acknowledged these rulings in its statement but said the "secretive" nature of IPT hearings meant there was little transparency about the way GCHQ was being policed. This, it said, undermined the faith people had in official oversight of the agency.
Information that had come to light in the last 12 months showed, said Amnesty, that there were flaws in the oversight system. One revelation concerned arrangements GCHQ has with its US counterparts to get at data it would be difficult for the UK agency to get permission to acquire.
There were also loopholes in UK laws governing surveillance being exploited by GCHQ to expand its spying abilities, it said.