What does a poll really ask you?

Just a few days before transport ministers met last month (6-8 June) to discuss the future of the beleaguered Galileo satellite navigation project, a poll was released by the European Commission claiming that "an overwhelming majority consider that Europe should set up an independent navigation system even if this involves securing additional public funds".

Such polls are a regular feature of Brussels policy debates. "If you see all the opinion polls, the large majority of European citizens, including the British, every day highlight as a top priority more Europe on granting our security," said Franco Frattini, the commissioner in charge of justice, security and freedom, arguing that the UK should give up its veto on justice and security. ‘More Europe’ seems to be what everyone is clamouring for in Commission-funded Eurobarometer polls.

The polls turn out such anomalously pro-EU results that a cynic might wonder whether the results have more to do with the way the poll is structured than the actual attitudes of respondents. "Eurobarometer was the only poll to ever show support for the [EU] constitution in the UK," said Neil O’Brien of Open Europe, a Eurosceptic think-tank. O’Brien referred to a Eurobarometer poll published in 2004 claiming 48% support for the EU constitution in the UK and 14% opposition. In stark contrast, an ICM research poll from 2004 found 25% support and 55% opposition.

Such high levels of support could have to do with the way Eurobarometer questions are asked. The poll on justice and security referred to by Frattini asked whether, in areas of security, "more decision-making should take place at a European level or on the contrary that less decision-making should take place at a European level"?

"They don’t ask people to make the real choice between the EU level and the national level but instead more or less Europe, as if the EU existed in a vacuum," said O’Brien, arguing that many of the polls did not give adequate choice. Referring to a special Eurobarometer poll on "the future of Europe" last year asking what changes people would like to see in the future, "all of the options they asked are further integration - there are no anti-integrationist options", he said. But such issues with phrasing nuances are nothing compared to the question that got 80% of Europeans to sign up to the Galileo project. That question included the following explanation: "The US owns and controls GPS, which is primarily for military use, but also provided for civilian use, however, without quality of service guarantees. Russia and China are working on setting up their alternative navigation systems. These are mostly military systems as well. According to your opinion, should Europe set up its own navigation system, or should Europe rely on American, Russian or Chinese systems?"

Robert Manchin, chairman of Gallup Europe, said that the question "was debated quite a bit". Eurobarometer and Flash Eurobarometer polls are ordered by the Commission and carried out by Gallup Europe under a €50 million four-year contract. The Commission often proposes a questionnaire and Gallup carries it out - in the case of a Flash Eurobarometer poll, with 27,000 phone interviews.

Manchin said that Gallup eventually felt the question was justified because it placed the issue in a "visionary, long-term strategic context". The results, he said, could only be interpreted that 80% of the public supports Galileo if the issue is presented to them a certain way. "It can be misleading if it’s presented in a different light," he said.

The reason the issue required such framing is that only about 40% claimed to know beforehand what the Galileo project was and only 20% had ever used a satellite navigation device. Manchin said that if an issue has to be framed for an audience that is not familiar with it, results "can change in a range of up to ten to 15 percentage points" based on the wording of the question. "Eighty percent would never be 40%," he said. "It was always a majority."

But the idea that question wording could only change results by 15 percentage points is "nonsense", according to George Bishop, guest professor at the Mannheim-based Centre for Survey Research and Methodology (ZUMA) and author of the book ‘The Illusion of Public Opinion’. Bishop said that wording could change results anywhere from 5-50%, as when people are unfamiliar with something many of them "generate opinions off the top of their head on the spot".

Bishops calls the Galileo poll "an excellent example of how public opinon can be readily manufactured by how questions about current issues are framed and asked of a public that is not particularly well informed about the topic".

Results that stem from such "mini courses" do not, he said, mean that the public has "real attitudes" about something. "They pick up on a cue or phrase" contained in the question, he added.

The problem with public opinion polling is that the results are presented to the public as scientifically valid, said Bishop. In contrast to scientific studies that undergo a "rigorous process of peer review", poll results go "straight to an unprotected public", he added. Numbers such as the stated margin of error give such polls the "spurious impression of accuracy". The Galileo poll stated a margin of error of 1.3%.

Manchin acknowledged that there was a risk in polls being used for publicity purposes. He held up the positive example of Eurobarometer polls used by the European Investment Bank to uncover problems with the introduction of the euro in Slovenia. But, he said, "if you have politicians who want to use [a poll] to boost their own projects, it can happen and it’s out of your hands".

Manchin said he was concerned that the Commission was "doing more surveys than they are able to digest". One way to guard against spurious conclusions, he said, would be for experts to spend two or three weeks comparing the poll results with other surveys. Another way would be to "provide funding for research parties to come and use the data". At present the poll results are deposited into a hard-to-access data account, but Gallup is developing an online platform that would make it easier for the public to gain access. With such a system, everyone could see the survey data, read the question in their own language and draw their own conclusions.

Just a few days before transport ministers met last month (6-8 June) to discuss the future of the beleaguered Galileo satellite navigation project, a poll was released by the European Commission claiming that "an overwhelming majority consider that Europe should set up an independent navigation system even if this involves securing additional public funds".

• Primary Source