Call us toll-free: 1 800-645-9850    


The presence of disengaged or inauthentic respondents is one of the greatest concerns for online research. The easiest way to identify these respondents within a dataset is to include measures inside a survey that can indicate aberrant or illogical behavior. In most cases, these indicators are nothing more than suggestive evidence of an underlying issue, and the determination of whether an individual is truly of 'low quality' cannot be made based on any one with great certainty. However, when a respondent accumulates many such faults across the entirety of a survey, we can identify disengaged respondents with greater confidence. Mktg Inc. has a battery of metrics that can be deployed on appropriate surveys in order to make these identifications.

The 5 indicators of quality metrics :  
2. Open-Ends
3. Straight Liners
4. Trap Questions
5. Inconsistencies

Speeding - It makes sense that some respondents take longer to complete a survey than others. In addition to differences in reading comprehension and other distractions in the respondent's environment, skip patterns and different concepts can drastically change even the length of the survey he or she sees. However, when comparing sections of a survey where all respondents see the same (or nearly the same) questions on the same topics, we have good reason to believe those who complete portions of the survey in less than half the median time may not be giving sufficient thought to their answers, or may not be reading directions thoroughly.
Open-Ends - Open-ended questions require the most effort of any question format. As such, they can represent a more stringent test of engagement. While strongly disengaged respondents may not even enter sentences composed of actual words, those who are only mildly disengaged may give responses that have less meaningful content than the norm (as measured by number of words). Respondents giving less than half the median number of words per response are marked for a quality fault.
Trap Questions - Trap questions are questions to which only a single response is acceptable. A ''directions trap'' directs respondents to provide a specific answer. Failure to do so indicates a failure to read the text in the question. A ''factual trap'' directs a respondent to rate or indicate ownership of a product that does not exist. If they indicate that they have ever used the item in question, it suggests they are not giving truthful answers, possibly to avoid being screened out of the survey.
Inconsistencies - Through the insertion of questions that ask a respondent to answer the exact same question in two opposite ways, we can tell whether he is sufficiently engaged to represent a coherent body of thought. Where possible, such efforts are disguised to minimize the annoyance to the respondent. On a scale question, providing responses in the bottom two or top two categories for both of a pair of logically contradictory questions causes us to mark a respondent as inconsistent.

In the below example of an inconsistency test, respondents who answer a 6 or 7 to both questions, or alternately, a 1 or a 2, would be marked as inconsistent.

Please indicate your level of agreement or disagreement with the following statements
Straight-Lining - Long grid or matrix questions provide a tempting opportunity to easily complete a portion of a survey without proper consideration of the attributes within them. In most cases, straight-lining is marked for those respondents who provide identical responses for all attributes on a grid with at least a 5-point scale. However, in instances where respondents are asked to evaluated a combination of negative and positive attributes, we would not expect them to give the same answer throughout. In these cases, responses with less than half the average standard deviation (in other words, an unusually tightly grouped response pattern) are marked as straight-liners. This method ensures that the criteria for straight-lining is sensitive to the likelihood that a tightly-grouped but not identical pattern of responses represents a legitimate body of opinions.

Quality Control

Copyright © 2017 Mktg, Inc. All rights reserved.  |  200 Carleton Ave, East Islip, NY 11730 | 631.277.7000 |

Privacy Policy | Terms of Use Policy

The Sample You Need Telephone Data Collection Capabilities and Services Quality Training Supervision Consulting: Sample Source Auditors Consistent Track Certification Program Real ID Grand Mean Standard Optimum Blending Solutions Online Data Collection Surveys OnLine Interview Design Sample Management Online Report Interface Crop Duster Digital Fingerprinting Online Quality Controls