Answers to Exercise

9.1 Examples of poor question wording

How strongly do you agree that smoking is harmful to health?

o Not at all

o Slightly

o Quite strongly

o Very strongly

The problem – this is a leading question which assumes that smoking is harmful. The response scale is also skewed – it has more responses on the ‘agree’ end than on the ‘disagree’ end.

The question could be rewritten in a number of ways in order to make it more neutral. The first example below has more neutral wording in the question, but retains the lack of symmetry in the response alternatives. The second example is also more neutral in question wording, and has symmetrical response alternatives. It is difficult to imagine that smoking could be seen as beneficial to health, given what we know about its effects.

What is your opinion about the effect of smoking on health?

o Not at all harmful

o Slightly harmful

o Quite harmful

o Very harmful

How much do you agree or disagree with the following statement: smoking is harmful to health?

o Strongly disagree

o Disagree

o Not sure

o Agree

o Strongly agree

How good is your voting record in local elections?

o Not at all

o Quite good

o Very good

o Excellent

There are several problems with this question as it is worded. To begin with, it assumes that all respondents know what ‘local elections’ are, and this could be dealt with by a sentence which explained that the question refers to voting in elections for local government representatives (rather than national or other kinds of elections). Furthermore, this question does not specify anything about when the respondent did or did not vote. Given that most of us have rather fallible memories, especially for events which are not deeply significant for us, it would be better to ask simply about the most recent local election. The question is also rather leading in its wording, since it assumes that the respondent has a good voting record. The response alternatives are poorly worded, because it is impossible to know what any respondents considers to be ‘good’ or ‘excellent’.

The question could be replaced with one or more questions such as:

Did you vote in the last local election?

o Yes

o No

o don’t know / can’t remember

How often have you voted in local elections over the last ten years?

o Never

o Hardly ever

o Sometimes

o Almost always

o Always

If you wanted to express your opinion about genetically modified foods would you consider taking part in a boycott of your local supermarket?

o Not at all

o Probably not

o Not sure

o Probably

o Definitely

There are several problems with this question. First, it expresses more than one idea. It is also hard to interpret any specific response. If a respondent says ‘not at all’, it is impossible to say whether the person has no opinion about GM foods, is in favour of GM foods (and therefore would not boycott supermarkets that sell them), does not agree with boycotts in principle, or agrees with boycotts but it is too inconvenient to boycott their local supermarket. It could also be that the respondent never shops in their local supermarket anyway.

The key to rewriting the question is to untangle what it is that the researcher wants to get at. If it is general favourability towards GM foods, then a possible question would be something like:

What is your opinion about genetically modified foods?

o Strongly against

o Slightly against

o Not sure

o Slightly in favour

o Strongly in favour

If the focus of attention is on attitudes to boycotts as a means of expressing an opinion, then the question could be something like:

What is your opinion about boycotting shops which sell GM food?

o Strongly against

o Slightly against

o Not sure

o Slightly in favour

o Strongly in favour

If the researcher wants to assess whether the respondent would exercise a boycott, then a gating question is needed to establish personal strength of opinion towards GM foods. This would then be followed by a question about boycotting only for those who are against GM foods.

How much do you agree or disagree with the following: Politicians never keep the promises they make before an election, once they are in office.

o Strongly disagree

o Disagree

o Not sure

o Agree

o Strongly agree

The problem with this question is the negative wording. The question is also somewhat leading, since its wording expresses a populist cynicism about the integrity of politicians. The response alternatives are hard to interpret (what does it mean to say that I ‘disagree’ with the statement?). Suppose I think that only some politicians never keep their promises, while most politicians do keep their promises. How would I express that opinion?

There are different ways to re-write the question, depending on the researcher’s focus of attention. For example:

How many politicians keep the promises they make before an election, once they are in office?

o None of them

o A few of them

o About half of them

o Most of them

o All of them

How often do politicians keep the promises they make before an election, once they are in office?

o Never

o Hardly ever

o Sometimes

o Most of the time

o All the time

How much do you agree with the following: My training supervisor is dynamic and well organized.

o Strongly disagree

o Disagree

o Not sure

o Agree

o Strongly agree

This question is actually two questions. The researcher would be unable to tell what any response alternative means, other than a generalized attitude of favourability towards the training supervisor.

This should be replaced with two questions:

How much do you agree with the following: My training supervisor is:

Dynamic

o Strongly disagree

o Disagree

o Not sure

o Agree

o Strongly agree

Well organized

o Strongly disagree

o Disagree

o Not sure

o Agree

o Strongly agree

                   

How much do you agree with the following: I am not satisfied with the progress of my research.

o Strongly disagree

o Disagree

o Not sure

o Agree

o Strongly agree

The problem here is that the question uses complex expressions and negatives.

It would be much simpler to remove the negative in the statement and re-write the response alternatives in terms of degree of satisfaction:

How satisfied are you with the progress of your research?

o Very dissatisfied

o Slightly dissatisfied

o Not sure

o Slightly satisfied

o Very satisfied

How much do you agree with the following: The presence of humorous literary allusions is conducive to an accessible presentation mode in academic pedagogy.

o Strongly disagree

o Disagree

o Not sure

o Agree

o Strongly agree

The silly wording here is an example of jargon as well as complex expressions. ‘Humorous literary allusions’ are jokes and ‘academic pedagogy’ means lecturing. It is harder to interpret what ‘accessible presentation mode’ means, though perhaps it means that the lecture is easier to follow.

A possible wording for the question could be:

How much do you agree with the following: Jokes make lectures easier to follow.

o Strongly disagree

o Disagree

o Not sure

o Agree

o Strongly agree

9.2 Types of questions that experimental and observational studies can answer

1 The first study rules out the possibility of a randomized controlled trial because the policy has already been introduced. This leaves the researcher with different options of observational studies. The second research question lends itself to quasi-experimental design. Provided the data captures labour market information before and after the policy intervention, researchers can estimate whether the policy elevated women’s chances for promotion. Quasi-experimental research design assumes the use of secondary labour market data, but researchers might struggle to find an appropriate measurement of promotion in existing datasets which would then require primary data collection. Primary data collection can be done through disseminating a questionnaire, and researchers will have to decide the target population (the whole of the labour market, a single organization, a specific industry, etc.). It is also possible to use firm-level administrative data so long as it measures promotions. In summary, though quasi-experimental research design based on secondary data analysis is a preferred approach, much depends on whether we can get the data we need, so researchers need to have back-up options,

2 The second study is a classic situation for experimental research design. Researchers can pilot a new AI technology in a sample of randomly selected outlets using other stores as a control group. This, however, depends on whether the outlets are comparable with each other. It is plausible that some stores are located in rural areas or serve particular types of customers (e.g. older people, women). The product range and types of services provided can also vary. If there is no way of making sure that in an experimental study, we will be comparing likes for likes, we might be better off collecting primary survey or administrative data before and after the company rolls out a new technology and then using an observational study. If the company uses social networks (e.g. Facebook, Twitter), we can scrape customer feedback from social network and use it as data in its own right or addition to an experimental/observational study.