Research Article

Adolescent Sexual Behavior, Drug Use, and Violence: Increased Reporting with Computer Survey Technology

See allHide authors and affiliations

Science  08 May 1998:
Vol. 280, Issue 5365, pp. 867-873
DOI: 10.1126/science.280.5365.867

Abstract

Surveys of risk behaviors have been hobbled by their reliance on respondents to report accurately about engaging in behaviors that are highly sensitive and may be illegal. An audio computer-assisted self-interviewing (audio-CASI) technology for measuring those behaviors was tested with 1690 respondents in the 1995 National Survey of Adolescent Males. The respondents were randomly assigned to answer questions using either audio-CASI or a more traditional self-administered questionnaire. Estimates of the prevalence of male-male sex, injection drug use, and sexual contact with intravenous drug users were higher by factors of 3 or more when audio-CASI was used. Increased reporting was also found for several other risk behaviors.

A small number of national surveys using large, representative samples of the United States and other populations have attempted to assess the sexual and drug-using behaviors that risk transmission of the human immunodeficiency virus (HIV) that causes acquired immunodeficiency syndrome (AIDS) (1). Because sexual activity is typically initiated in adolescence or early adulthood and because that period for many young people is characterized by greater amounts of experimentation, partner change, and risk taking than in later years, research programs with a focus on the behaviors of adolescents and young adults are of particular importance (2). AIDS is not, however, the only threat faced by young people. Interpersonal violence represents an even greater threat of morbidity and mortality (3); consequently, interpersonal violence has also received greater scrutiny in recent surveys of the adolescent and young adult population. Yet concerns have surfaced regarding the quality of survey measurements of such sensitive behaviors (4, 5). The concern most often cited is that respondents may be reluctant to report accurately about sensitive or stigmatized behaviors in which they have engaged. (Reservations are particularly strong in situations in which teenage respondents are asked to tell an adult interviewer whether, for example, they have had a certain sexual experience or used a certain drug.) In one study (5), it was estimated that survey measurements that relied upon women's responses to questions about their abortion history during a face-to-face interview captured only 35% of the abortions performed in the United States (a conclusion based on aggregate data supplied by abortion providers for the 1984–1987 period).

The level of privacy that an interviewing mode affords a respondent can dramatically affect survey measurements of sensitive behaviors (5-7). Traditionally, surveys have attempted to encourage more accurate reporting of sensitive behaviors by allowing respondents to complete a paper-and-pencil self-administered questionnaire (paper SAQ), which they can seal in an envelope and return to the interviewer. Paper SAQs, however, require that respondents be sufficiently literate to complete a written questionnaire. In addition, because a respondent's answers together with their identification number are recorded on a paper form, some respondents may remain suspicious about the privacy of their responses. Another criticism of paper SAQs is that extensive use of contingent questioning (that is, branching or skip patterns) is difficult. Even literate respondents may have trouble following instructions for navigating through a complex self-administered form (8).

Audio-CASI technology. A computer-driven technology has been developed that can administer complex survey questionnaires in an audio format and record respondents' answers without the direct participation of a survey interviewer (9). This approach, audio computer-assisted self-interviewing (audio-CASI), allows respondents to listen over headphones to spoken questions that have been digitally recorded and stored on a laptop computer. To answer, respondents press numbered keys on the computer keyboard. Questions are also displayed on the computer's screen, and respondents may respond to the visual presentation of the question rather than waiting until the audio reading has been completed. Respondents can thus answer questions in complete privacy, even if their reading ability is limited. Because survey data are stored on the drives of the computer, they are less vulnerable to inadvertent disclosure to interviewers or others. Audio-CASI also provides (10) (i) a completely standardized measurement system—every respondent (in a given language) hears the same question asked in exactly the same way; (ii) computer-controlled branching through complex questionnaires and automated consistency and range checking; and (iii) efficient multilingual administration of surveys (10, 11).

Here we report the use of this technology in a large-scale national survey, the 1995 National Survey of Adolescent Males (NSAM), which included a randomized experiment to assess the effect of this technology on the measurement of sensitive behaviors. This study was begun in January 1995 and it measured HIV-risk behaviors, drug use, and interpersonal violence among young men in the United States.

Since 1988, NSAM has tracked the sexual, contraceptive, HIV-risk, and other behaviors of a national probability sample of young men who were 15 to 19 years of age in 1988 (12). Past data have presented some intriguing methodological puzzles. A case in point is the measurement of male-male sexual contacts. Because of the potential sensitivity of reporting such behavior, the relevant NSAM questions were presented in a paper SAQ instead of being administered by the interviewer. We expected that use of this more private mode of survey administration would encourage more accurate reporting of the behavior. We found, however, that the proportion of males reporting such contacts in 1988 (2.2%) was lower than would be expected on the basis of surveys of adult men, which have recorded retrospective reports of adolescent male-male sexual behaviors of 4 to 9% (13). Further evidence of the difficulty of making such measurements was provided by the rescissions that occurred in these NSAM reports between 1988 and 1991. Eleven of the 30 young men who reported having had an oral or anal male-male contact in the 1988 NSAM survey reported that they had never had such contacts when they were interviewed again in the 1991 survey. [Paper SAQs were used in prior rounds of the survey (14).] These considerations and our desire to increase the actual and perceived privacy of the 1995 NSAM interview context motivated us to use audio-CASI technology for measurements of the most sensitive behaviors among the new cohort of 15- to 19-year-olds recruited in the 1995 round of the survey.

Survey sample and experiment. We drew a multistage area probability sample from the population of males ages 15 to 19 who were living in households in the continental United States between February and November 1995 (15). Black and Hispanic males were oversampled. To obtain the final sample, we screened 54,265 housing units out of the 56,199 that were eligible for screening. The process identified 2240 eligible males; we completed interviews with 1729 of them. After adjusting for the 3.4% of housing units that could not be screened, we had an overall response rate of 75% (16). (The major reasons for nonresponse were refusal by the respondent, refusal by the parent of the potential respondent, and unavailability of the potential respondent after repeated visits.) Of the 1729 young men who completed the interview, 1690 (97.7%) also completed a self-administered interview containing questions on topics that were potentially quite sensitive (including same-gender sex, drug use, and violent behaviors).

Sample weights were used to adjust for the different initial probabilities of sample selection and subsequent nonresponse. Poststratification adjustments were applied to align the NSAM sample estimates with 1995 population estimates from the Bureau of the Census (17). To represent accurately the effect of this sample weighting and the stratified and clustered design used in drawing the 1995 NSAM sample, we used statistical software (18) that adjusted variances for our sample estimates to reflect the complex design. Our statistical inferences are intended to generalize to the universe of (noninstitutionalized) U.S. males ages 15 to 19 (Table 1).

Table 1

Selected social and demographic characteristics of the 1995 NSAM respondents who completed self-administered interviews. (Wgt. %, weighted percent.)

View this table:

By design, the sample included almost as many Hispanic and black respondents as white ones (Table 1). That pattern reflects our oversampling of those minorities and permits a relatively precise characterization of any differences in reported risk behaviors that may exist among the groups or in their reaction to the technology being tested in our experiment. (All subsequent tabulations use weighted data.)

All respondents initially completed an interviewer-administered survey that contained a range of questions on their relations with their peers; the sources of their education and information on sexuality and contraception; their perceptions of and attitudes toward condoms, contraceptives, and the consequences of unintended pregnancy; their heterosexual and contraceptive experiences and their knowledge and perceptions of sexually transmitted diseases (STDs); and their social background and demographic characteristics. The interviewer-administered portion of the survey lasted an average of 68 min.

Next, respondents completed a self-administered interview. This interview contained 99 of the most sensitive questions being asked in the survey, including questions on the respondent's male-male sexual experiences, drug use, STD history, experiences with violence and weapons, heterosexual oral and anal sex, and sexual orientation (19). To assess the effects of audio-CASI versus more traditional paper SAQs on the reporting of such information, we embedded an experiment in this part of the survey. Participants were randomly assigned (at a ratio of 4:1) to a condition in which questions were administered with audio-CASI (N = 1361) or to a condition that used a paper SAQ (N = 368). Of the 1729 eligible survey respondents, 1690 completed the experiment. Because of computer problems and interviewer mistakes, a small number of those respondents (N = 18) were not interviewed in the mode that had been randomly assigned. These cases were excluded from our analyses. Thus, the base N for our analyses is 1672 (20).

Statistical analyses. In estimating the prevalence of reported behaviors, we focused on variations between modes of interviewing (audio-CASI versus paper SAQ). Because respondents were randomly assigned to the two interviewing conditions, the crude ratio of the odds of reporting a particular behavior in each condition provides an unbiased measure of the effect of audio-CASI on such reports relative to the more traditional paper SAQs. Our target behaviors were dichotomous variables (for example, did versus did not have male-male sex); therefore, we used log-linear modeling and logistic regression procedures to gauge the statistical significance of any effect that was observed and to test for variations in effect across subpopulations (21). Because we wished to make inferences about the likelihood that a given effect would hold in the population at large, our analyses used the sampling weights described earlier, and our statistical tests took account of weighting and the NSAM's complex sample design (22). The P values shown in Tables through 5 test the null hypothesis that given the odds ratio observed in this experiment, the “true” odds ratio would have been 1.0 (that is, no effect of audio-CASI) if the same experiment was repeated among an infinite number of similarly designed samples of the national population (23).

Although the crude odds ratio is an unbiased indicator of the effect of the survey technology, more precise measures were also calculated. Adjusted odds ratios may correct for random perturbations across experimental conditions in the distribution of factors correlated with the targeted risk behavior (24).

Estimated prevalence of risk behaviors. As shown in Table2 (25, 26), the mode of survey administration made little difference in estimates of the prevalence of male-female sexual contacts. The only significant effect was for reporting of sexual contact with a prostitute. Respondents who were interviewed with audio-CASI were 3.6 times as likely (crude odds ratio) to report such contact as respondents who were interviewed with the paper SAQ (2.5% versus 0.7%) (27).

Table 2

Alternate estimates of prevalence of male-female and male-male sexual behaviors among 1995 NSAM respondents obtained by using different methods of questioning. For Tables to 5 the odds ratio (OR) is adjusted (Adj.) for covariates [race (white, black, other as residual category), whether they have health insurance, age, whether they currently attend school, and whether they had sexual intercourse with a female], as reported in the interviewer-administered portion of the survey.

View this table:

Although the use of audio-CASI had little effect on reporting of male-female adolescent sexual behavior, it produced highly significant increases in reports of male-male sex. Overall, respondents were almost four times as likely to report some type of male-male sex in the audio-CASI mode compared with the paper SAQ (5.5% versus 1.5%). Estimated odds ratios varied from 1.85 to 7.91 across the six measurements of passive and active masturbation and oral and anal sex with males. The individual results are not all significant, but the pattern is consistent; that is, audio-CASI was more likely than the paper SAQ to elicit a report that the respondent had engaged in male-male sex. In addition, the observed fourfold increase in the reported prevalence of male-male contact among adolescents is statistically significant (P < 0.001), and the estimated prevalence of such contact (5.5%) is more consistent with prevalences derived from retrospective reports provided by adult men reporting on their sexual behaviors during adolescence (4 to 9%) (13).

Table 3 presents parallel results for the reporting of drug use alone and as an adjunct to sexual behaviors. For the drug use behaviors that are most commonly associated with transmission of HIV—that is, injection of illicit drugs and use of crack (a refined, smokable form of cocaine) or cocaine—the audio-CASI mode elicited substantially more reporting. Overall, the audio-CASI measurements yielded an estimate of the prevalence of injection drug use (5.2%) that is substantially higher than any prior population estimate. The estimated prevalence derived from the more traditional paper SAQ measurements was 1.4% (28).

Table 3

Alternate estimates of prevalence of drug use, per se, and drug use during sex derived by using different methods of questioning. Results are from the 1995 NSAM.

View this table:

Behaviors that commingled drug use and sex are another category that evinced a strong effect from the use of the audio-CASI technology (29). Respondents assigned to the audio-CASI condition who reported that they were sexually active were more likely than their counterparts in the SAQ condition to report that they or their partners were drunk or high when they last had heterosexual intercourse (odds ratio = 2.95); they were also more likely to report that they were “always” or “often” drunk or high when they had heterosexual sex during the past year (odds ratio = 5.5). In the audio-CASI mode, respondents who reported having sex were much more likely than SAQ respondents to report that they had had sex with someone who injected drugs (odds ratio = 13.8).

The pattern of higher estimates of prevalence in the audio-CASI mode is repeated in the NSAM measurements of violent behaviors (Table4). Respondents in the audio-CASI mode were more likely to report that they had carried a gun in the past 30 days (12.4 versus 7.9% in the SAQ mode) (30) and to report that they had carried a knife or razor in the same period (27.1 versus 19.4%). Similar increases were found in reports of threats of violence, both those made by the respondent and those made against him.

Table 4

Alternate estimates of prevalence of violent behaviors derived by using different questioning methods. Data are from the 1995 NSAM.

View this table:

Variations in effects across social and demographic subgroups. To determine whether the increase in reporting we observed with audio-CASI might vary across race-ethnicity or age groups, we fit a hierarchical series of log-linear models (31) to the four-way tables for each behavior reported in Tables through 4 for which there was a significant difference (P < 0.05) in estimated prevalence across modes. The four-way table broke down the estimated prevalence of each risk behavior by three variables: mode of interview (audio-CASI versus paper SAQ), age (15 to 16 versus 17 to 19), and race-ethnicity (black, Hispanic, white, or other) (32). These analyses revealed little evidence of significant variation in the effect of the survey mode across those age and race-ethnicity groups.

Although we did not measure respondent literacy directly, we were able to identify respondents who were “not at grade level” in school or who had dropped out of high school without graduating. Respondents were coded as “at approximate grade level” if they were age 15 and had completed 8th grade or higher; age 16 and had completed 9th grade or higher; age 17 and had completed 10th grade or higher; age 18 and had completed 11th grade; or any age and had completed 12th grade. All other respondents were coded as “below grade level.” To detect variations in how audio-CASI influenced reporting in the latter group, we focused on the reporting of drug use, the combination of drug use and sex, male-male sex, and violence.

For most behaviors, no substantial differences were observed in the effect of audio-CASI across these two education strata. However, in four instances (reporting of any male-male sex, injection drug use, daily marijuana use in the past year, and crack or cocaine use in the past year), there were noteworthy differences in the effect of the survey mode. For reporting of the three drug use behaviors, audio-CASI had a strong effect (odds ratios = 2.8 to 14.1) for the subpopulation of respondents who were high school graduates or who were at their approximate grade level (Table5). No such effect was found for the subpopulation who had dropped out of school or were still in school but seriously behind their normal grade level. For the three drug use behaviors, the paper SAQ yielded estimates that were about equivalent to those obtained with audio-CASI technology (odds ratios = 0.5 to 1.5). For the reporting of male-male sex, there were parallel increases in reporting when audio-CASI was used, but the effect was more pronounced for males who were behind in school. No male who was behind in school reported any of the six male-male sexual behaviors when completing the paper SAQ. In the audio-CASI condition, 6.2% of respondents not at grade level reported one (or more) types of male-male sexual contact. For males who were at grade level a more attenuated effect was observed [1.8% in the paper SAQ versus 5.3% in audio-CASI (33)].

Table 5

Selected estimates of prevalence of male-male sex and drug use by mode of interview and whether respondent was at “grade level.” Respondent is coded as “at grade level” if by age 15 they have completed grade 8, completed 9th grade by 16, 10th grade by 17, 11th grade by 18, or completed high school. All other respondents and those that have not completed 8 years of school are “below grade level.”

View this table:

For reports of serious drug use, this pattern of differential sensitivity to interview mode might imply that the segment of the population that is “on track” educationally may perceive that they have more to lose by discovery of their past drug use. Alternatively, there may be differences across groups in peer norms about the acceptability of such behavior. Similar differences in peer norms could account for a differential sensitivity to reporting of male-male sexual contacts. Data from representative national samples of adults surveyed in 1990 to 1994 indicate that tolerance of same-gender sex increases with the educational attainment of the respondent (34). Although such speculations have some intuitive appeal, they do not account for the failure to find parallel results for questions other than the four noted here.

Other results. Nonresponse to individual questions has been a persistent problem in self-administered interviews that use paper SAQs (35). Respondents in the audio-CASI mode in our experiment were less likely than respondents in the SAQ mode to use the “don't know” or “refuse to answer” options. (The audio-CASI technology provided labeled keys for those responses.) Of the 58 questions that were designed to be answered by all respondents, an average of only 0.5% of respondents in the audio-CASI mode did not answer them, compared with an average of 2.3% of respondents in the paper SAQ mode.

Discussion. The results of this large-scale experiment indicate that respondents in the 1995 NSAM's new cohort, a probability sample of U.S. males aged 15 to 19, were much more likely to report risky behaviors when they were interviewed with audio-CASI measurement technology than when interviewed with the more traditional paper SAQ. The magnitude of the observed effect was substantial for many behaviors, particularly those that were illicit or highly stigmatized. Thus, the estimated prevalence of past-year use of crack or cocaine almost doubled, from 3.3 to 6.0%, and the estimated proportion of adolescent males who reported ever taking street drugs with a needle more than tripled, from 1.4 to 5.2%. Although we have some concern about the interpretation of the injection drug use result (36), overall the audio-CASI estimates were considerably higher for illicit and stigmatized behaviors. These NSAM audio-CASI estimates were also higher than estimates derived from the federal government's 1995 National Household Survey on Drug Abuse. Estimates derived from that survey are that the prevalence of reported crack or cocaine use during the past year among males aged 15 to 19 was 2.6% and that the prevalence of reported injection drug use in the same group was 0.3% (37).

Varying the interview mode also affected the reporting of the co-occurrence of drug use and sexual behavior. The proportion of sexually active adolescent males who reported that they or their partner had been drunk or high at last intercourse more than doubled in the audio-CASI mode (from 15.3% in the paper SAQ condition to 34.8%). In addition, indicators of interpersonal violence increased markedly.

In planning this experiment, we were aware of prior research (5-7) showing that a private mode of survey response such as paper SAQs encouraged more complete reporting of sensitive, stigmatized, and illicit behaviors. We anticipated that the additional benefits of computerization and an audio format for the self-administered interview would bring only modest improvements in reporting. The large differences we found were surprising. If this measurement technology provides a more accurate picture of patterns of sexual behaviors, drug use, and violence among adolescent males in the United States, then the risks they are encountering are substantially greater than was previously supposed. This conclusion assumes, of course, that the observed increase in reporting of these behaviors is, in fact, more accurate reporting. That assumption follows standard practice in much social measurement research (38); nonetheless, it remains an assumption. Yet in the present instance, there is empirical evidence to support it. The large increases in the reporting of male-male sexual contacts under audio-CASI (to 5.5% for any contact) make these measurements more compatible with estimates derived from adults' reports of their adolescent behaviors (39).

The fact that audio-CASI both increases reporting of male-male contacts and makes our estimates more consistent with those obtained from retrospective reports of adults encourages the conclusion that this technology is reducing the underreporting bias known to affect such measurements. In addition, the technology appears to have a more pronounced effect on the reporting of behaviors that are particularly sensitive, stigmatized, or subject to serious legal sanctions, compared with less sensitive areas of conduct. Thus, our analyses of the NSAM data found few reliable effects of audio-CASI on the reporting of male-female sexual contacts; the sole exception was for reporting of sexual contact with prostitutes. That behavior is arguably a more sensitive topic for adolescents than are other male-female sexual contacts. Similarly, audio-CASI substantially increased reporting of all types of male-male sexual contacts and drug use. Here again, exceptions occurred for reports of more common and, we suspect, less sensitive behaviors such as the use of alcohol and marijuana.

Those results and complementary evidence emerging from smaller studies using local samples (40) lead us to conclude that the estimates presented here are more accurate than previous estimates derived from data collected in less private interview contexts. These estimates present a disturbing picture of the biological and social risks that confront young males in the United States at the end of the 20th century.

NOTES AND REFERENCES

View Abstract

Navigate This Article