Explainer
How opinion polls work: sampling, questions and weighting
An opinion poll interviews a sample of people chosen to represent a larger population and uses their answers to estimate what the whole population thinks.[1] In a probability sample everyone has a known chance of selection, which lets pollsters calculate a margin of error; in opt-in polls respondents choose themselves, and that error cannot be estimated in the same way.[2][3] Question wording, question order and the way samples are weighted can all change results, which is why AAPOR asks pollsters to publish their methods.[4][5][6]
Polls estimate what millions of people think by asking a few hundred or a few thousand of them. Whether that works depends on who is asked, what they are asked and how the answers are adjusted. This page walks through each step, using the methods published by Pew Research Center and the standards of the American Association for Public Opinion Research (AAPOR).[7][6]
Choosing who to ask
A sample is a subset of a population chosen to represent the whole, and its answers are used to estimate the characteristics of everyone.[1] The key distinction is how people get into the sample. In a probability, or random, sample every person has a chance of selection that can be calculated, which is what makes population estimates possible.[2] In a non-probability sample, such as a convenience, quota or volunteer sample, some people have no chance of selection or their chance cannot be determined, so sampling error cannot be estimated.[3]
Online opt-in polls fall in the second group. Their respondents are not selected at random but recruited from online sources such as social media and search ads, reward websites or panels people sign up to themselves.[8] AAPOR asks researchers to state explicitly which kind of sample they used.[9]
A probability sample works like a lottery draw from a list of everyone: each person’s chance of being picked is known. An opt-in poll is more like a survey link that anyone can click, so the people who answer may differ from those who do not in ways the pollster cannot measure.[2][3]
Panels, addresses and phones
Many pollsters now use survey panels, groups of people who have agreed to take surveys on an ongoing basis.[10] Pew’s American Trends Panel is a probability-based panel of roughly 10,000 US adults selected at random; people cannot volunteer, and since 2018 it has recruited by address-based sampling.[7] That method draws a random sample of residential addresses from a master list such as the one kept by the US Postal Service.[11] Since mid-2024 people joining the panel have been able to choose between answering online or by phone, which Pew says ensures that adults who do not use the internet are represented.[12]
The industry has changed quickly. A 2023 Pew study of 78 national pollsters found that 61% of those active in both 2016 and 2022 had changed their methods.[13] Only 10% used live telephone interviews as their sole method in 2022, though 32% still used live phone alongside other methods.[14] AAPOR estimated, according to Pew, that 56% of national polls shortly before the 2016 election used online methods, rising to 84% in 2020.[15] Pew noted that most polls had understated Republican support in both the 2016 and 2020 presidential elections, and that new methods were no guarantee the problem had been fixed.[16]
Writing the questions
Small differences in wording can substantially change answers.[4] In a January 2003 Pew survey, 68% favoured military action in Iraq, but only 43% did when the question added that US forces might suffer thousands of casualties.[4] The format matters too: after the 2008 election, 58% picked the economy as their top issue when it was offered as an option, but only 35% named it when asked in their own words.[17]
Earlier questions can provide context that shapes answers to later ones, and Pew randomises many lists of answer options to spread out these order effects.[5] People also tend to give socially acceptable answers: research shows respondents understate alcohol and drug use and tax evasion, and overstate church attendance and their likelihood of voting.[18]
Adjusting the results
Raw samples rarely match the population. Without adjustment, polls tend to over-represent people who are easier to reach, so pollsters weight their data to match the population, usually on several demographic measures.[19] How surveys are weighted covers the methods. Weighting reduces bias but widens the margin of error, an effect known as the design effect.[20]
Sources of error
The margin of error measures only sampling error, the chance difference between a sample and the full population.[21] For a probability sample of 1,067 people it is about plus or minus 3 points.[22] Other errors can arise at any stage and are hard to measure: coverage gaps, people who do not respond, inaccurate answers, and interviewer or processing mistakes.[23] Margin of error explains how to read the plus-or-minus figure.
What a poll should tell you
AAPOR’s Code of Professional Ethics and Practice asks researchers who release results to disclose who sponsored and conducted the research, the exact question wording, the population, how the sample was drawn, the dates, the sample sizes and how the data were weighted. AAPOR says the Code applies to all public opinion researchers, whether or not they are members.[6] The standards also ask probability surveys to say whether their reported margin of error accounts for the design effect of weighting.[24]
Read next
Next in the crash course
Margin of error: what the plus-or-minus in a poll means
What a poll's margin of error measures, why a lead must be about twice it to be clear, why subgroups are less precise, and the errors it leaves out.
Questions readers ask
How can a poll of 1,000 people represent a whole country?
A sample is a subset of a population chosen to represent the whole. In a probability sample every person has a chance of selection that can be calculated, which makes it possible to produce population estimates. A simple random sample of 1,067 people has a margin of error of plus or minus 3 percentage points.[1][2][22]
What is the difference between a probability poll and an opt-in poll?
In a probability sample each person's chance of selection is known. Online opt-in polls recruit respondents from sources such as ads, reward websites or self-enrolment, so they are not selected at random and their sampling error cannot be estimated in the same way.[2][8][3]
Can I volunteer to take part in a Pew poll?
No. Pew Research Center's American Trends Panel is made up of roughly 10,000 US adults selected at random, and only people who are selected can join. Since 2018 it has recruited by sampling residential addresses.[7]
Does question wording really change poll results?
Yes. In a January 2003 Pew survey, 68% favoured military action in Iraq, but only 43% did when the question added that US forces might suffer thousands of casualties.[4]
Do people answer polls honestly?
Not always. Research has shown that respondents understate things such as alcohol and drug use and tax evasion, and overstate church attendance and the likelihood that they will vote.[18]
What should a poll tell you about its methods?
AAPOR's Code asks researchers to disclose who sponsored and conducted the poll, the exact question wording, the population, how the sample was drawn, the dates, sample sizes and how the data were weighted. AAPOR says the Code applies to all public opinion researchers, not just its members.[6]
Sources
· 24 claims from 3 publishers
Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.
- [1]
A sample is a subset of a population selected to represent the whole, and information from it is used to estimate characteristics of the entire population. confirmedas of 2026-11-14
- Census and sample · Australian Bureau of Statistics · Primary source · Census and sample (retrieved 2026-11-14)
- Census and sample · Australian Bureau of Statistics · Primary source · Census and sample (retrieved 2026-11-14)
- [2]
In a probability (random) sample every unit in the population has a chance of selection that can be calculated, which makes it possible to produce population estimates. confirmedas of 2026-11-14
- Census and sample · Australian Bureau of Statistics · Primary source · Random sampling (retrieved 2026-11-14)
- Census and sample · Australian Bureau of Statistics · Primary source · Random sampling (retrieved 2026-11-14)
- [3]
In a non-probability sample, such as a convenience, quota or volunteer sample, some people have no chance of selection or their chance cannot be determined, so sampling error cannot be estimated. confirmedas of 2026-11-14
- Census and sample · Australian Bureau of Statistics · Primary source · Non-random sampling (retrieved 2026-11-14)
- Census and sample · Australian Bureau of Statistics · Primary source · Non-random sampling (retrieved 2026-11-14)
- [4]
Small differences in question wording can substantially change answers; in a January 2003 Pew survey, 68% favoured military action in Iraq, but only 43% did when the question added that US forces might suffer thousands of casualties. confirmedas of 2026-11-14
- Writing Survey Questions · Pew Research Center · Primary source · Question wording (retrieved 2026-11-14)
- Writing Survey Questions · Pew Research Center · Primary source · Question wording (retrieved 2026-11-14)
- Writing Survey Questions · Pew Research Center · Primary source · Question wording (retrieved 2026-11-14)
- [5]
The order of questions can influence answers, because earlier questions can provide context for later ones; Pew randomises many lists of answer options for this reason. confirmedas of 2026-11-14
- Writing Survey Questions · Pew Research Center · Primary source · Question order (retrieved 2026-11-14)
- Writing Survey Questions · Pew Research Center · Primary source · Open- and closed-ended questions (retrieved 2026-11-14)
- [6]
AAPOR's Code of Professional Ethics and Practice asks researchers who release results to disclose who sponsored and conducted the research, the exact question wording, the population, how the sample was drawn, the dates, sample sizes and how the data were weighted, and AAPOR says the Code applies to all public opinion researchers, members or not. confirmedas of 2026-11-14
- Disclosure Standards · American Association for Public Opinion Research · Primary source · Section III, A (retrieved 2026-11-14)
- Disclosure Standards · American Association for Public Opinion Research · Primary source · Section III, A (retrieved 2026-11-14)
- Disclosure Standards · American Association for Public Opinion Research · Primary source · FAQ (retrieved 2026-11-14)
- [7]
Pew Research Center's American Trends Panel is a probability-based panel of roughly 10,000 US adults selected at random; people cannot volunteer to join, and since 2018 it has recruited by address-based sampling. confirmedas of 2026-11-14
- The American Trends Panel · Pew Research Center · Primary source · introduction (retrieved 2026-11-14)
- The American Trends Panel · Pew Research Center · Primary source · How are panelists selected (retrieved 2026-11-14)
- The American Trends Panel · Pew Research Center · Primary source · How are panelists selected (retrieved 2026-11-14)
- [8]
Pew describes online opt-in polls as convenience samples whose respondents are not selected at random but recruited from online sources such as social media or search ads, reward websites or self-enrolment in a panel. confirmedas of 2026-11-14
- How Public Polling Has Changed in the 21st Century · Pew Research Center · Primary source · 2023-04-19 · Terminology, online opt-in polls (retrieved 2026-11-14)
- [9]
AAPOR's disclosure standards ask researchers to state explicitly whether a sample was drawn using probability-based methods or non-probability methods such as opt-in or volunteer sources. confirmedas of 2026-11-14
- Disclosure Standards · American Association for Public Opinion Research · Primary source · Section III, A, sample (retrieved 2026-11-14)
- [10]
A survey panel is a group of people who have agreed to take surveys on an ongoing basis. confirmedas of 2026-11-14
- How Public Polling Has Changed in the 21st Century · Pew Research Center · Primary source · 2023-04-19 · Terminology, survey panel (retrieved 2026-11-14)
- [11]
Address-based sampling recruits survey takers by drawing a random sample of residential addresses from a master list such as the one kept by the US Postal Service. confirmedas of 2026-11-14
- How Public Polling Has Changed in the 21st Century · Pew Research Center · Primary source · 2023-04-19 · Terminology, address-based sampling (retrieved 2026-11-14)
- [12]
Since mid-2024 people joining the American Trends Panel have been given a choice between taking surveys online or by phone, which Pew says ensures that adults who do not use the internet are represented. confirmedas of 2026-11-14
- The American Trends Panel · Pew Research Center · Primary source · How surveys are conducted (retrieved 2026-11-14)
- The American Trends Panel · Pew Research Center · Primary source · How surveys are conducted (retrieved 2026-11-14)
- [13]
Pew's 2023 study of 78 national pollsters found that 61% of those that released national polls in both 2016 and 2022 used different methods in 2022 than in 2016. confirmedas of 2026-11-14
- How Public Polling Has Changed in the 21st Century · Pew Research Center · Primary source · 2023-04-19 · overview (retrieved 2026-11-14)
- How Public Polling Has Changed in the 21st Century · Pew Research Center · Primary source · 2023-04-19 · overview (retrieved 2026-11-14)
- [14]
In Pew's study, 10% of national pollsters used live telephone interviews as their only method in 2022, while 32% used live phone alone or with other methods. confirmedas of 2026-11-14
- How Public Polling Has Changed in the 21st Century · Pew Research Center · Primary source · 2023-04-19 · overview, key findings (retrieved 2026-11-14)
- [16]
Pew noted in 2023 that most polls understated Republican support in both the 2016 and 2020 presidential elections, and that changes to methods were no guarantee the problem had been fixed. confirmedas of 2026-11-14
- How Public Polling Has Changed in the 21st Century · Pew Research Center · Primary source · 2023-04-19 · overview, key findings (retrieved 2026-11-14)
- How Public Polling Has Changed in the 21st Century · Pew Research Center · Primary source · 2023-04-19 · overview, key findings (retrieved 2026-11-14)
- [17]
Whether a question is open-ended or closed-ended can change results; after the 2008 election, 58% chose the economy as their top issue when it was offered as an option, but only 35% volunteered it in the open-ended version. confirmedas of 2026-11-14
- Writing Survey Questions · Pew Research Center · Primary source · Open- and closed-ended questions (retrieved 2026-11-14)
- [19]
Without adjustment, polls tend to over-represent people who are easier to reach and under-represent those who are harder to interview, so pollsters weight their data to match the population, usually on several demographic measures. confirmedas of 2026-11-14
- 5 key things to know about the margin of error in election polls · Pew Research Center · Primary source · 2016-09-08 · section 5 (retrieved 2026-11-14)
- 5 key things to know about the margin of error in election polls · Pew Research Center · Primary source · 2016-09-08 · section 5 (retrieved 2026-11-14)
- [20]
Weighting helps avoid biased results but also makes the margin of error larger; statisticians call this increase in variability the design effect, and a margin of error that ignores it claims more precision than the survey warrants. confirmedas of 2026-11-14
- 5 key things to know about the margin of error in election polls · Pew Research Center · Primary source · 2016-09-08 · section 5 (retrieved 2026-11-14)
- 5 key things to know about the margin of error in election polls · Pew Research Center · Primary source · 2016-09-08 · section 5 (retrieved 2026-11-14)
- 5 key things to know about the margin of error in election polls · Pew Research Center · Primary source · 2016-09-08 · section 5 (retrieved 2026-11-14)
- Margin of Sampling Error/Credibility Interval · American Association for Public Opinion Research · Primary source · Sample Size and MOSE (retrieved 2026-11-14)
- [21]
Pew Research Center explains that because a survey talks only to a sample, its result will probably not exactly match the true population value, and the margin of sampling error describes how close the result can reasonably be expected to fall to that value. confirmedas of 2026-11-14
- 5 key things to know about the margin of error in election polls · Pew Research Center · Primary source · 2016-09-08 · section 1 (retrieved 2026-11-14)
- Margin of Sampling Error/Credibility Interval · American Association for Public Opinion Research · Primary source · page 1, first paragraph (retrieved 2026-11-14)
- [22]
A simple random sample of 1,067 people has a margin of error of plus or minus 3 percentage points for estimates of overall support for a candidate. confirmedas of 2026-11-14
- 5 key things to know about the margin of error in election polls · Pew Research Center · Primary source · 2016-09-08 · section 4 (retrieved 2026-11-14)
- [23]
Non-sampling errors such as coverage error, non-response, inaccurate answers and interviewer or processing mistakes can occur at any stage of a survey and are not easily identified or measured. confirmedas of 2026-11-14
- Types of error · Australian Bureau of Statistics · Primary source · Non-sampling error (retrieved 2026-11-14)
- Types of error · Australian Bureau of Statistics · Primary source · Non-sampling error (retrieved 2026-11-14)
- Types of error · Australian Bureau of Statistics · Primary source · Non-sampling error (retrieved 2026-11-14)
- Types of error · Australian Bureau of Statistics · Primary source · Non-sampling error (retrieved 2026-11-14)
- [24]
AAPOR's disclosure standards ask that probability surveys report their sampling error and say whether it has been adjusted for the design effect of weighting, clustering or other factors. confirmedas of 2026-11-14
- Disclosure Standards · American Association for Public Opinion Research · Primary source · Section III, A, sample sizes (retrieved 2026-11-14)
Revision history (1)
- Page created.
Created Nov 14, 2026. Last reviewed by an editor on Nov 14, 2026. Next scheduled review: May 14, 2027.
Cite this page
"How opinion polls work: sampling, questions and weighting." ContentLora, updated Nov 14, 2026. https://contentlora.com/explain/how-polls-work
Spotted an error? Suggest a correction or emailcorrections@contentlora.com.