To ensure valid results without incurring the time and cost required to interview all members of the target population, researchers developed protocols for drawing a sample of individuals that adequately represented the entire population and from which conclusions about the sample could be drawn with a known level of reliability. The simplest of these protocols is a uniformly distributed random sampling in which the sample members are selected randomly from the target population. Indeed, to have a statistical basis for making statements based on the sample about the population, one must use a random sample; one can not extrapolate to the population using non-random (i.e., non-probability) samples. Convenient or accidental sampling of patrons within the library is not random sampling. Variations on random sampling such as cluster sampling (applied in the undergraduate sample design) and stratified sampling (applied in the graduate and faculty sample designs) add flexibility and cost savings and tend to reduce standard errors.
A particular method for implementing the sampling should maximize reliability and validity subject to cost and feasibility constraints. Given these criteria, campus mail was the chosen mode of administration for both graduate students and faculty and, for undergraduates, in-class administration. Regular mail was not a valid or feasible option for undergraduates. In previously published library user studies, it was shown that undergraduates responded poorly to mail questionnaires. Given the mobility of the undergraduate population, the Registrar's list of undergraduate home addresses and phone numbers is never current and in many instances contains the parents home address rather than a local address for the student.
In the Undergraduate sample design, the study was designed to yield a sample of 2,000 class enrollees from a population of unique class enrollees, i.e., each student in each class constituted a unique population element. The term class enrollee is more accurate than student as any given student may be enrolled in more than one selected class and therefore has, given this sampling frame design, a probability of selection proportional to the number of classes in which he or she is enrolled. Because a given student can be found in multiple sample points, the class enrollee, rather than the student, was the sampling unit. However, the sample design begins with a consideration of classes.
All eligible classes were divided into distinct subpopulations or strata and, from each stratum, a sample of classes in that stratum was selected. This stratification reduces standard error and, as each subsample can be considered a sample of that particular population stratum, allows extrapolation from the strata subsamples to their respective population strata. In addition to allowing one to make statements about the population as a whole, it allows greater confidence in making statements about the characteristics of a given population stratum. The sample design established ten strata by crossing two dimensions: class level (i.e., lower- or upper-division) and field grouping (i.e., Arts, Engineering, Humanities, Physical Sciences, and Social Sciences).
The class list (the 1996 Spring Quarter enrollment file ), was provided by the UCSD Office of the Registrar. The list was divided into lower-division undergraduate classes with course numbers 1-99 and upper-level undergraduate classes with course numbers 100-199. Classes with other course numbers, e.g., graduate courses, were discarded. The undergraduate classes were also assigned to one of the five field groupings listed above. Table 1 identifies the specific classes included in each group. Of the classes identified in each of these five groups, some had to be discarded for the following reasons: a) either they had duplicative enrollment with other classes (i.e., the discussion sections and labs tied to lectures) or b) types of classes which posed logistic difficulties for in-class administration (i.e., field work, independent study, and practicum classes). The percentages in Table 1 are based on the enrollment in eligible classes.
| Lower- Division | Upper- Division |
|
|---|---|---|
| Arts (Music; Theatre; Visual Arts) | 6.32% | 3.32% |
| Engineering (AMES; CSE; ECE; Bioengineering) | 3.04% | 5.85% |
| Humanities (History; Literature; Philosophy; Humanities; IRPS; Internal Relations and Pacific Studies; Third World Studies; Classical, Japanese, Judiac, Latin, and Chinese Studies; ESL; Dimensions of Culture; Muir and Warren College Writing Programs; Making of the Modern World) | 10.98% | 7.02% |
| Physical Sciences (Biology; Chemistry and Biochemistry; Mathematics; Physics; Earth Sciences) | 18.86% | 14.90% |
| Social Science (Anthropology; Cognitive Science; Communications; Contemporary Issues; Economics; Ethnic Studies; Human Development; Linguistics; Political Science; Psychology; Science, Technology, and Public Affairs; Social Science; Sociology; Urban Studies & Planning) | 11.95% | 17.76% |
For each stratum, classes were drawn randomly with probabilities of selection proportional to class enrollment until the class list for that stratum was exhausted. Subject to permission by the class instructor and administrative convenience, each consecutive class was incorporated into the sample sequentially until the threshold for a sufficient sample in the respective stratum was satisfied. For each stratum, that threshold was based on the number of class enrollees representing that stratum's proportion of class enrollees out of the entire population of class enrollees (e.g., the proportion of the population in the lower-division/Science and Math stratum was approximately 1/5 hence, this stratum contained 1/5 of the total sample).
Table 2 exhibits the classes included in the sample; the four classes, Visual Arts 14, Music 13, Music 95, and CSE 14, were treated as ineligible (and are discussed further in the response rate section below). As the number of enrollees varied considerably across classes and the class selection was not terminated in a given stratum until the threshold was reached, the resulting sample size was inevitably greater than 2,000.
| Stratum | Sequence Number | Class | Completes | Cumulative Completes | Cumulative Percentage | Target Completes | Target Percentage |
|---|---|---|---|---|---|---|---|
| LD-ART | 1 | MUS 11 | 92 | ||||
| 5 | MUS 14 | 97 | 189 | 6.61% | 126 | 6.32% | |
| UD-ART | 1 | VIS 155 | 103 | 103 | 3.60% | 66 | 3.32% |
| LD-ENG | 1 | CSE 21 | 108 | 108 | 3.77% | 61 | 3.04% |
| UD-ENG | 1 | EEC 117 | 58 | ||||
| 3 | AMES 121c | 67 | 125 | 4.37% | 117 | 5.85% | |
| LD-HUM | 1 | DOC 3 | 242 | ||||
| 2 | HILD 7c | 177 | 419 | 14.65% | 220 | 10.98% | |
| UD-HUM | 1 | HIEU 157 | 105 | ||||
| 2 | HIEU 131 | 89 | 194 | 6.78% | 141 | 7.02% | |
| LD-SCI | 1 | BILD 2 | 234 | ||||
| 2 | PHYSI 1c | 214 | 448 | 15.66% | 377 | 18.86% | |
| UD-SCI | 1 | BIPN 102 | 217 | ||||
| 2 | BIBC 100 | 409 | 626 | 21.88% | 298 | 14.90% | |
| LD-SOC | 1 | ETHNIC 1c | 227 | 227 | 7.93% | 239 | 11.95% |
| UD-SOC | 1 | PSYC 169 | 236 | ||||
| 2 | POLI 104b | 181 | 417 | 14.58% | 335 | 17.75% | |
| Cases missing class information: | 5 | 5 | 0.17% | 0 | 0.00% | ||
| TOTAL | 2861 | 2861 | 100.0% | 2000 | 100.0% | ||
The result is a cluster sample, so called because the selected classes are clusters of class-enrollees. Class enrollee is the secondary sampling unit; in each class, all class enrollees were selected as part of the sample.
As noted above, in the sample design for this study, the fraction of the sample selected from each population stratum was equal to the fraction of the population within that stratum. Furthermore, within each of the strata, each class has a probability of selection equal to its purported enrollment. (Note that the higher probability of selection of large classes gave rise to the possibility that a sample stratum might be represented by a single class that poorly represents that stratum. As shown in Table 2, in three of ten strata, a sampling stratum consisted of a single class--UD-Art, LD-Eng, LD-Soc.) Each class enrollee had an equal probability of selection for inclusion in the sample of 1/N, N being the total number of class enrollees.
The graduate and faculty sampling frames were designed to yield a sample of 600 graduate and School of Medicine (SOM) students, and a second group of 600 faculty and campus researchers, from a target populations of 2,942 and 3,779 respectively. The list of medical and graduate students was provided by the Office of Graduate Studies & Research (OGSR) and the School of Medicine (SOM) and included only those students registered for the 1995 Winter Quarter. The Personnel Office provided a list of academic staff, from which the Libraries User Survey Team identified all non-faculty staff to be excluded from the sampling frame.
Each graduate student and faculty listed with a department affiliation was assigned a random number, as well as each represented department. The lists were sorted by the department-level random number and, within each department, by the individual-level random number. Based on the sizes of the populations and desired samples, a sampling interval was calculated and the samples generated. This random procedure resulted in a graduate/SOM student and a faculty/researcher sample stratified by department. Both samples were stratified in this fashion because attitudes toward the library were thought likely to differ among the departments, and, in such cases, stratification may reduce the standard error.
The undergraduate questionnaire was administered in-person during the first ten minutes of class time in weeks two to four of the 1996 Spring Quarter. In-class administration maximized the response rate and minimized self-selection bias while avoiding the usual financial impact attendant to telephone and in-person interviews. The sample included class enrollees from 22 classes with a total enrollment of approximately 4,910, from which we had 2,861 respondents. The remaining students could not be included for the following reasons: 1) access was not granted by the class instructors or the class was excluded for administrative convenience and 2) class enrollees in the sampled classes were not present while the survey was being administered, arrived late or though in attendence, chose not to complete the survey.(Since surveys were administered during the early weeks of the quarter, when students are known to be still "shopping" for classes, accurate class enrollment figures were not available for weeks two through four. Consequently, the class enrollment figures discussed below and used in the calculation of the response rate are taken from week-one class enrollment files.)
Statistically, individuals in the non-response group were likely to be similar to the students who took the survey. The sampling plan assumed that the four unsurveyed classes (one instructor refused, two instructors could never be contacted -one of whom taught two of the selected classes- and one class excluded for administrative convenience) and their enrollees were outside the sample design. Strictly speaking, this means the sample as drawn can only be extrapolated to those students in classes which could have been reached. While this distinction is probably minor, it is not trivial. It is possible that whether an instructor allows or refuses survey administration in a class, could be related to the likely responses of students in that class. Enrollment in the four non-surveyed classes was 1,221.
As for the group of students who did not participate in the surveyed classes, while it is unlikely that any student knew of the survey in advance, it seems improbable it played a substantial role in the decision to come to class. Thus, the probability of intentional self-selection seems minimal. However, their non-attendance could be related in less direct ways to the answers they would have given, i.e., non-class attendees may have different attitudes toward the library. The administration of the survey early in the quarter was intended to reduce such losses through non-attendance. Non-respondents who did not take or complete the survey because they were late for class are subject to this same analysis. More problematic are persons who did not complete the survey despite being in attendance since the reasons could be related in some way to the measures taken in the survey. Based on the number of completes and the enrollment in the 17 classes in which surveys were collected, the number of non-participating but enrolled students was 828.
As noted above, the total number of completions was 2,861 and the number of eligible respondents was 3,689 (4,910 enrolled in all 22 classes less 1,221 enrolled in the four classes which were treated as ineligible). The response rate was 78 percent, calculated as 2,861 completes divided by the 3,689 eligible respondents. The number of undergraduate completes yielded a confidence interval of 95±1.8%.
Students have until the end of the fourth week of the quarter to drop classes without receiving a "Withdrawal" on their transcripts. Some students, though technically enrolled in a class, may not attend that class if they intend to drop it before this deadline. Thus, the enrollment figures used in the response rate calculation (week one) are probably an overestimate of the enrollment at the time of the survey administration (weeks two through four); consequently, the 78 percent response rate should be considered a lower bound on the actual response rate.
During any survey administration, several factors may cause the collected sample to deviate in relevant particulars from the sample design. As noted above, some individuals selected to be in the sample may be discovered to have been ineligible, and some individuals may not be collected because they refuse to participate or because they are not available. Clustering may also contribute to such deviations. In addition, in this sample design, the available clusters were of widely varying sizes.
The customary manner of dealing with such deviations is that of weighting the data so that it better approximates the intent of the sampling plan. Each person in the sample is given a weight such that when looking at the percentages across the sample, that sample more closely approximates the population from which it was drawn. The weight is given by the formula Fn/fN, where F is the frequency of class enrollees in the population stratum, N is the size of the population, f is the frequency of class enrollees in the sample stratum, and n is the size of the sample. A weight of 1 was assigned to five completed surveys missing class information. While weights in certain circumstances can also be constructed to take into account factors other than sample design, the weights used in Table 3 were merely to adjust the stratum proportions in the sample to the stratum proportions in the undergraduate population. (The population percentage used in the weighting calculation is based again on the week one enrollment figures.) Analysis revealed the undergraduate sampling weights had a insignificant impact on the results; hence, the survey findings presented below are based on the unweighted data.
| Group | Division | Population % | Sample % | Weight |
|---|---|---|---|---|
| Arts | Lower | 7.05% | 6.62% | 1.0651 |
| Upper | 3.47% | 3.61% | 0.9623 | |
| Engineering | Lower | 2.97% | 3.78% | 0.7845 |
| Upper | 5.97% | 4.38% | 1.3650 | |
| Humanities | Lower | 10.80% | 14.67% | 0.7359 |
| Upper | 7.08% | 6.79% | 1.0425 | |
| Physical Sciences | Lower | 18.81% | 15.69% | 1.1991 |
| Upper | 14.42% | 21.92% | 0.6579 | |
| Social Science | Lower | 11.65% | 7.95% | 1.4660 |
| Upper | 17.78% | 14.60% | 1.2178 |
Both the graduate and faculty questionnaires were administered through the UCSD campus mail network. All surveys were mailed during the first week of the 1996 Spring Quarter, the only exception applying to third year medical school students. Representatives from the School of Medicine indicated to the survey team, that having completed their on-campus classes, third year medical school students rarely came to campus and hence did not received their mail on campus. Based on that information, the survey team and the consultant decided to mail the questionnaires and other survey-related correspondence, to medical students' home address. To maximize response rates and minimize later non-response conversion efforts, of all mailed surveys, a raffle ticket was also enclosed. The ticket was to be returned by a specified date-- midway through the 11-week field period--to qualify for the $250 University Bookstore gift certificate drawing. The idea for an incentive had been well-received by both the graduate and faculty focus group participants.
Exactly one week after the initial mailing, everyone being surveyed, received a postcard thanking them for their response and reminding them to complete and return the survey if they had not already done so. Two weeks later, a second letter with a replacement questionnaire was mailed. The third and final non-response conversion effort entailed phoning non-respondents which was implemented in two stages during the latter part of the field period. In stage one, members of departments with response rates below the overall response rate were targets, and in stage two,members of the remaining departments were contacted.
Of the 600 selected graduate/SOM students, two were nurses who should have been excluded up-front, leaving 598 graduate students selected. Another 14 ineligibles were discovered to have graduated, transferred to another school, or taken a leave of absence, leaving 584 eligible students of which there were 415 completions. Thus, the response rate was 71%. The number of graduate student responses yielded a confidence interval of 95±4.8%.
Of the 600 faculty members and researchers, four were later identified as staff. Of the remaining 596, another 83 proved to be ineligibles because they were either retired, no longer at UCSD, or on leave from UCSD, leaving 513 eligible cases. Of these, 368 completed the survey, giving a response rate of 72 percent and yielding a confidence interval of 95±5.1%.
In principle, stratification would allow one to make more reliable statements about the characteristics of the individual departments. However, the relatively small sizes (and thus number of completions) in some departments preclude any meaningful statements about those departments. The graduate sample included 36 different departments, with departmental sample sizes ranging from 1 to 83 with an average of approximately 16. The faculty sample included 69 different departments, with departmental sample sizes ranging from 1 to 93, with an average of approximately 7. Similarly, sample size diminished the usefulness of weights to adjust for differential non-response among the various departments.