 |
 |
 |
 |
 |
||||||||||||||||||||||||||||||||
|
|
|
||||||||||||||||||||||||||||||||
........... |
ABSTRACT The National Survey of Student Engagement (NSSE) is a U.S. survey of first and fourth-year students based on known “best practices” in teaching and learning. The survey is operated by an independent entity based at Indiana University and is supported by user fees paid by individual participating institutions. Since the survey’s inception in 2000, some 840 institutions have participated with over 630,000 student responses compiled. To date, twelve state systems of higher education have participated in NSSE, with the resulting information used as part of state accountability requirements for public higher education. NSSE information is also frequently featured by participating institutions in their required accreditation reviews. The survey was originally developed to generate information on academic program quality in response to the growing influence of annual media rankings of institutions. But voluntary participation and institutional desires to keep results confidential have limited its impact in this arena. Through an aggressive media effort of its own, however, NSSE has had a significant impact on public perceptions of how “quality” should be construed in higher education. At the same time, it has given participating institutions an important diagnostic tool for improving their instruction and their educational environments. NSSE’s concept and approach is readily portable to other national higher education contexts, where more proactive government sponsorship or support might well result in greater policy impact. Indeed, the survey is similar to Australia’s Student Course Experience Questionnaire (SCEQ), which is administered under government auspice. INTRODUCTION The National Survey of Student Engagement (NSSE) is somewhat unusual as a “policy instrument” in this series because it was conceived outside government and is operated by an independent third-party agency. Its origins, however, are rooted in policy because its original intent was to provide a means for students, citizens, elected officials, and taxpayers to better assess the quality of higher education institutions. A secondary objective was to create a tool for the oversight bodies of publicly-funded institutions to monitor the quality of higher education provision at the institutions for which they are responsible. While the NSSE has to date not completely fulfilled this first rather lofty original expectation, its design and implementation hold many lessons for other non-governmental approaches to quality assurance. At the same time, growing state-level use of NSSE in publicly-supported colleges and universities provides useful lessons about how to create and manage systems based on more general public reporting of institutional performance. Higher Education Context The policy environment for higher education in the United States within which NSSE evolved is complex, decentralized, and in many ways unique. There is no national ministry governing public higher education. Governance and support of public higher education is instead a responsibility of fifty individual states, which differ markedly in how they approach the task. Some are organized as systems with centralized policies with respect to administration, finance, and curriculum. Others comprise individual institutions that are connected to one another only in that they receive public funds. Complicating the picture, about half of all degree-granting colleges and universities are private, non-profit entities governed by independent lay boards and funded largely by income from tuition and fees. Although these institutions tend to be small—enrolling only about 20% of the total undergraduate student population in the country—their numbers (and, to a certain extent, their prestige) make them a formidable presence in the nation’s higher education system. Despite their independence, moreover, private institutions receive a good deal of indirect public support through federal student scholarship and loan programs and, in many cases, through parallel state financial aid programs. Partly as a result of these conditions, the U.S. higher education system is strongly conditioned by classic market forces. Private institutions charge what the market will bear, and the two hundred or so of the most selective and prestigious “national institutions” can command tuition “sticker prices” of over $20,000 per year. Private institutions farther down the prestige ladder still have tuition charges of over $10,000 per year. For these institutions, moreover, maintaining tuition revenue is a matter of life or death, as typically over 90% of their costs must be covered through tuition charges. For public institutions, market forces are buffered somewhat by public subsidy, but tuition still accounts for about a third of institutional revenues. Recently, due to substantial shortfalls in state revenues, public institutions in most states have engaged in record-breaking increases in tuition. All of this means that the behaviors of colleges and universities in the U.S. are shaped substantially by the market. Attracting a sufficient number of students to pay the bills is fundamental for most. And for those whose endowments and reputations allow them a bit more flexibility, attracting an ever-more-selective student body remains a priority. Under these conditions, the factors that influence student choices about where to attend affect institutions far more than government regulation or steering. To be sure, public institutions in the U.S. are subject to various forms of regulatory control in realms such as finance and procedural accountability. More recently, most of the states have established systems of performance reporting, based largely on efficiency measures and in a substantial minority, institutional performance on such indicators is consequential (Burke 2002, Burke and Minnassians 2003). At the same time, a purportedly voluntary institutional accreditation system, loosely-regulated by the federal government, requires all institutions—both public and private—to undergo a periodic comprehensive review that examines resources, organizational structures, instructional processes, and (most recently) student learning outcomes. But all of these public and quasi-public government regulatory and steering mechanisms operate on the margins of an enterprise that is shaped heavily by the marketplace of student choice. The factors influencing this marketplace, therefore, are fundamental to higher education policy in the U.S. A major force shaping institutional behavior in this environment is media rankings of institutions. The U.S. News and World Report annual survey of “America’s Best Colleges” was the first such venture in the world, with its inaugural edition launched in 1983. Since that time, additional media rankings have emerged in the U.S. including a “value for money” review by Money Magazine and a burgeoning industry in college guides. And for better or worse, much of the world seems to be following this “league table” phenomenon with examples ranging from McLean’s Magazine in Canada, through The Times in the United Kingdom, to Der Spiegel in Germany. Research in the U.S. suggests that such publications exert very little leverage over actual student choices, although they can sometimes noticeably affect admissions markets in the short term for institutions in the most selective tier (McDonough, Antonio, Walpole and Perez 1997). More important are their indirect—and often substantial—effects on institutional behaviors, which have been repeatedly documented (Machung 1998). The institutions whose admissions pools might actually be affected by the changes in fashion reported by the national media quite naturally attempt to improve their rankings. And the vast majority, although their admissions markets are local, follow these leaders in a continuing attempt to move up the ladder of prestige. All of this might be considered beneficial if the metrics of “quality” underlying media rankings faithfully represented institutional capacity and performance. Market forces, as in any other field, would automatically induce institutions toward ever-increasing “quality” (at least as perceived by the customer), thus serving public purposes. Indeed, this popular policy logic has been used increasingly to steer institutional behavior in the U.S. since at least 1989 when Congress first required colleges and universities to disclose graduation rates to prospective students. But the problem with media rankings in the eyes of most critics is that they are based on a badly-flawed metric of “quality” driven essentially by institutional resources and reputation. The U.S. News measures, for instance, began as a reputational ranking done by college presidents, and only gradually added such measures as dollars spent per-student, admissions selectivity, and alumni loyalty as measured by financial contributions. While these factors produce a familiar list of “winners” drawn from the nation’s best-known colleges every year, they say nothing about the question really being asked: what do institutions do to enhance student learning and how well do they do it? In framing the policy problem that NSSE was originally intended to address, moreover, it is important to emphasize that moving the metrics of quality away from resources and reputation toward student outcomes was part of a larger undergraduate reform movement and a consequent change in the way governments approached accountability for higher education in the U.S. Part of the impetus for this arose from the academy itself, stimulated by reformers worried about growing lack of coherence in the undergraduate curriculum (AACU 1985). But part of it also came from state governments, reflecting a new view of higher education as a “public good” connected directly with such statewide benefits as economic development and functional citizenship (NGA 1986, Ewell 1997). By the late 1980s, many states had enacted requirements for institutions to assess student learning and report publicly on the results and by 1989, the federal government mandated institutional accrediting bodies to adopt such requirements as well. The following year, state and federal actors came together to proclaim a set of “National Education Goals” to guide educational policy for the coming decade. Although mostly about elementary/secondary education, these goals included an explicit commitment to “increase substantially” the ability of the nation’s college graduates to “think critically, communicate effectively, and solve problems.” The implied promise to develop the metrics needed to track progress on these elusive qualities was one of the many roots of NSSE because it stimulated thinking about how to examine them indirectly by looking at what institutions did to promote them (Ewell and Jones 1993). Not only would such an approach be less intrusive and expensive than launching a massive national testing program, but it could also be built on a solid tradition of research about effective student learning environments in the U.S. using the proven technology of survey research. With this as a backdrop, the Pew Charitable Trusts—a charitable foundation with considerable visibility and influence in the U.S.—launched a multifaceted program to stimulate quality improvement in undergraduate education in the mid-1990s. The bulk of this effort comprised grants to individual colleges and universities intended to support promising innovations in teaching and learning. But some of it was designed to influence institutional behavior indirectly by re-shaping the structure of regulations and incentives within which colleges and universities must operate. And a prominent negative element in this environment, at least for those at the Pew Trusts, were media rankings that rewarded institutions for the wrong things and reinforced the public’s image that institutional “quality” was simply a matter of money and selectivity. To attack this perceived problem, leaders at Pew convened a meeting of higher education leaders concerned about the rankings in the spring of 1998. One conclusion was that the Trusts should underwrite a new survey of college student perceptions and behaviors, based on the kinds of indirect indicators of “good practice” suggested earlier as an approach to assessing the National Education Goals. DESIGN AND DEVELOPMENT The NSSE is a national survey that focuses on specific undergraduate student experiences and features of the educational environment (Kuh 2001, 2003, http://nsse.iub.edu). The concept of “engagement” that constitutes its core reflects the results of at least two decades of research in the U.S. identifying specific factors of both experiences and environment that are associated with high learning gain (e.g. Astin 1978, 1993, Pace 1979, Pascarella and Terenzini 1991). These factors are embodied in the five “benchmarks” scales around which NSSE results are reported:
Items on the survey were specifically selected for inclusion in the survey only if there was a clear empirical case in the literature on college student learning and development that the factor represented could be associated with learning gain. Indeed, given the initiative’s origins, documenting the relationship between the “engagement” concept and actual learning has been crucial to its implementation. This has been accomplished in several ways. First, those responsible for NSSE stressed this relationship from the outset through documents describing the instrument’s conceptual and empirical foundations (e.g. http://nsse.iub.edu/pdf/conceptual_framework_2003.pdf, http://nsse.iub.edu/html/psychometric_framework_2002.htm). Second, the survey was extensively validated through two major field tests (see below) which involved student focus group work to both refine item content and to collect external evidence of links between particular item responses and actual student experiences (Ouimet, Bunnage, Carini, Kuh, and Kennedy 2004). Finally, NSSE has engaged in ongoing attempts to directly validate the link between survey items and direct measures of student learning through the cross-administration of NSSE with a number of cognitive assessment measures (Carini and Kuh 2004, NCHEMS 2003). This unusual level of conceptual and empirical documentation was seen by NSSE’s founders as important in gaining public credibility for an indirect approach to examining academic quality, and is frequently cited as a factor in its success. NCHEMS began the task of designing the survey by convening a team of recognized experts on college student survey research and higher education quality. With an initial design in place, a successful pilot study involving twelve institutions was undertaken in the spring of 1999 to test the instrument itself. This was followed in the fall by a sixty-institution field study to test survey administration procedures at different kinds of institutions. Both pilots were administered under subcontract to Indiana University’s Survey Research Center, which was then chosen to house and administer the survey under a competitive RFP-based selection process. NSSE was launched on a national basis in the spring of 2000 supported by a three-year grant from the Pew Trusts with the understanding that the survey would be self-supporting via user fees by the end of this period—a goal which has since been accomplished. A “sister” survey targeted at two-year institutions, the Community College Survey of Student Engagement (CCSSE), was launched in 2003 also underwritten by the Pew Charitable Trusts, and is housed at the University of Texas Austin. How the survey worksThe NSSE is administered to samples of students at the end of their first year of study and just before they are expected to receive a baccalaureate degree. Sample sizes are based on the size of the institution and range from 450 to 1000 students (or up to 3000 in Web-based administration). A substantial advantage is the fact that the survey is administered to students at all institutions directly by Indiana University’s Center for Survey Research using state-of-the art survey research techniques which require little work by participating campuses. This approach not only helps maximize response rates, but also helps ensure that data are comparable across campuses because administration procedures are standardized. Participating institutions are asked to send an electronic list of all students qualified to be chosen as part of the designated sample. NSSE staff then select a random group of students to be surveyed from this list, and administer the survey directly. The survey is available in both Web-administered form and as a paper questionnaire sent through the mail, with the proportion between these two modes of completion shifting markedly toward the former: in recent administrations, more than two-thirds of all respondents completed the survey on line. Response rates for both versions have averaged 43% and while there is individual variation in response rates across institutions, this national average response rate has been maintained within one percentage point for five years (and was also obtained by the pilots). Volume of Participation. Institutional participation is voluntary and the enterprise is at this point entirely supported by user fees. Nevertheless, numbers have steadily increased over five years, with the latest administration involving more than 480 institutions and over 550,000 students. The total number of institutions that have participated in NSSE since its launch in the spring of 2000 is about 840 with more than 630,000 students responding. Looking at volume another way, institutions that have participated in NSSE now represent more than 60% of the total number of students enrolled at four-year institutions in the U.S. Reporting. NSSE results are reported nationally in the form of an annual report issued by Indiana University (e.g. www.iub.edu/~nsse/2003_annual_report/), summarized in terms of the five “benchmark” indicators, which represent reliable aggregates of individual survey questions. Although institutions typically choose to participate individually, to this point some twelve state systems of public colleges and universities have administered NSSE on a statewide basis with data reported publicly. NSSE does not release scores or benchmark results for individual institutions without their consent, but CCSSE results for individual institutions are posted publicly on the survey’s website (www.ccsse.org ). Individual institutions receive comprehensive reports on the responses of their own students, as well as electronic files to support further local analyses of the survey data. These institutional reports contain two features not typically found in standard survey reports. First, benchmark scores are reported comparatively in the form of deciles, so an institution can see immediately where it stands comparatively with respect to other institutions of its type. Second, additional indicators, based on a multiple regression model, are provided that show how well the institution performed in the light of how it should have been expected to perform given its student and institutional characteristics. Finally, for an additional modest cost, institutions can receive specially-constructed peer comparison reports that show their results against other groups of institutions that they select. Feedback from institutions suggest that the information supplied by NSSE reports is far superior in form and content to commercial or other national surveys in which they have participated. COSTS The original development of NSSE, including the design of the survey and two field tests, was underwritten by grants from the Pew Charitable Trusts totaling about $650,000. Pew further supported the first three years of implementation of the survey with a grant to Indiana University totaling $3.3 million. Currently, NSSE is self-supporting with an operating budget exceeding $2 million. Based on four years of operation, survey costs approximate $6-7 per student surveyed. Direct costs to institutions in the form of user fees vary by size, and range from $1500 to $7000. Other institutional costs are almost exclusively incurred in the form of staff time to compile the lists of qualified students and their associated contact information to create the sample pool that is sent to Indiana University. IMPLEMENTATION ISSUES In order to fulfill the Pew Trusts’ original notion of mounting a direct challenge to media rankings, it was originally envisioned that comparative NSSE results for institutions would be reported publicly—and perhaps even be incorporated in some way into the U.S. News ranking methodology. NSSE’s sponsors expected the higher education community to welcome this proposal—both because institutional leaders visibly and vocally opposed existing media rankings and because the content of the survey addressed precisely the elements of academic quality that faculty felt such media reports lacked. But NSSE’s leaders were appropriately cautious about immediately proposing such a position and engaged in a number of deliberate efforts to tap community opinion on this matter. One line of inquiry consisted of open meetings held at major national meetings representing various institutional constituencies. Another was the full-scale field test at sixty institutions held in the fall of 1999, which was consciously structured to include participation by various types of institutions—small independent colleges, large public research universities, and regional public universities—chosen in consultation with the national associations that represent them. As these conversations unfolded, institutional opposition to the planned public release of survey results became increasingly vocal. This was especially the case among the high-prestige independent colleges that were currently at the top of the media rankings. Such institutions, though they were intrigued by NSSE’s content and continued to maintain a public posture of vehement criticism of actors like U.S. News, were not ready to risk losing their dominant position as “America’s Best” by participating in a process whose results were uncertain. Public institutions, already prominently in the public eye by this point due to numerous public-reporting mandates imposed by states, were less uncomfortable with the notion of public reporting. But early opposition by an influential sector of institutions quickly revealed the policy dilemma embedded in the enterprise from the outset. To leverage quality through the market, NSSE needed everybody to participate. But why would institutions pay for a voluntary data-gathering effort that might yield adverse public information about their own performances and undermine their markets? After some consideration, the architects of NSSE decided to implement the survey on a confidential basis. The resulting data, according to a carefully-worded institutional participation agreement, would be the joint property of the institution and NSSE and the agreement stated explicitly that “results specific to each institution and identified as such will not be made public except by mutual agreement.” While this wording protected institutional confidentiality where it was desired, it also did not preclude the release of comparative results for any public institutions that participated in NSSE as a system, in which case the “client” was a state governing board instead of an individual institution. As a result, some twelve state systems of higher education have administered NSSE at all of their campuses, with results reported publicly by institution. Significant examples here have included Kentucky, North Carolina, and South Dakota. Kentucky’s use of the survey is typical, in that a report on survey results was used to generate cross-campus discussions about how to improve quality, and campus-by-campus results were posted on the state agency’s website (www.cpe.state.ky.us/facts/facts_researchbriefs.asp ). Kentucky plans to administer NSSE at all public institutions on a biennial basis. Other states, for example, Virginia and Missouri, have encouraged institutions to adopt NSSE items as part of their response to state-mandated performance measures where institutions are allowed to choose the array of statistics to report. In these cases, NSSE has given participating states a cost-effective policy tool to monitor the quality of teaching and learning environments that they would otherwise have had to invent and run themselves. Given the prominent position of highly-selective independent institutions in the U.S., though, the confidentiality provision was essential to obtain widespread and broadly-representative institutional participation in NSSE early in the implementation process. Were this decision not taken, the initiative might have failed to achieve self-sufficiency altogether or be confined almost exclusively to public institutions. But this necessary decision forced a change of NSSE’s main strategy from direct to indirect with respect to inducing quality improvement. Instead of creating a highly-visible and authoritative competitor to the media rankings that might directly influence institutional behavior, NSSE would instead provide institutions with a tool that could be used to guide their own internal improvement efforts and would continue to supply states with a ready-made quality measure should they choose to participate. At the same time, NSSE would work consistently through the media to change perceptions of institutional quality among opinion leaders and the public at large. Opportunities to pursue this latter “quasi-public” agenda were exploited from the outset, and kept NSSE a policy initiative instead of just another consortium-based institutional data-gathering effort. A first such opportunity arose in the summer of 2000 at the point when NSSE was to issue its first national report. Since the effort was new and had been much heralded in the higher-education press, this report was the first significant chance to promulgate the “engagement” view of quality in a public forum. NSSE leaders knew that the report’s punch (and its consequent press appeal), would be greatly enhanced if specific colleges and universities that exemplified particular characteristics were explicitly named. Naming institutions, all recognized, would have to be confined to top performers—both because such institutions would be unlikely to withhold permission to be named and because illustrating “best performance” was an effective way to communicate NSSE’s more basic message about the nature of institutional quality. But naming institutions publicly, even with their permission, might upset the delicate perceptual balance embodied in the institutional participation agreement. By implication, not naming other participating institutions automatically implied they were of lesser quality. And losing institutional trust at this early stage—even if the letter of the agreement with respect to the particular institutions being named was honored—would be a significant setback. The decision was made to go forward with this strategy and four institutions were named in the national report as top performers on all five benchmarks. All four were selective private liberal arts colleges—though they were not necessarily the top performers in the U.S. News ranking. Although this action produced some grumbling about “broken promises,” it did not noticeably affect NSSE’s institutional participation rate in subsequent years (although it is not known whether or not it influenced the decisions of particular institutions—or even institutions of particular types—to participate). This episode also illustrated the double-edged nature of participants’ self-imposed ban on disclosing individual results. Some top performers, after discovering that they were such, quickly proclaimed this result in their recruitment materials and websites—illustrating precisely but unintentionally the ability to leverage institutional behaviors through student choice that NSSE’s sponsors originally hoped for. And by implication, those not named, might either have been spurred toward further efforts to improve engagement or, equally plausibly, might have decided not to risk another encounter. The relative numbers making each decision, of course, are unknown. But after five years of operation, two things are clear. First, with only a few exceptions, institutions placing in the top fifty in the U.S. News rankings have not participated in NSSE. And second, NSSE has up to now refrained from naming “top performers” in its national report, choosing instead to illustrate findings (with permission) drawn from the entire range of institutional types. A second revealing incident in NSSE’s early implementation resulted from the survey’s systematic and highly proactive media strategy. Working with a nationally-known media consultant, NSSE quickly became positioned as a premier press source about both college quality and other higher education matters. But part of the way this was accomplished was by continually questioning the U.S. News rankings or, more accurately, allowing others to take on U.S. News themselves. The most prominent of these early attacks occurred in 2000 when the Washington Monthly published a lengthy criticism of both the magazine and its methods, co-authored by a former U.S. News insider. In it, NSSE was prominently featured as an example of the kind of data source that U.S. News could use if it was serious and sincere. Subsequently picked up by The New York Times, this attack quickly got the attention of the editors of U.S. News. U.S. News, of course, cannot use NSSE data in its rankings even if it wanted to because not all institutions participate in the survey—a fact that its editors have repeatedly (and appropriately) claimed in their defense. But they needed some kind of response to what had become a considerable body of negative press about their methods. Their answer was to try to publish NSSE data in some form in the upcoming college rankings issue for the fall of 2002. When approached directly by U.S. News to provide the needed data, NSSE staff honored the institutional participation agreement and chose not to do so. NSSE staff then took the further step of declining to help U.S. News choose a subset of survey items that the magazine could ask participating institutions to supply directly, on the grounds that any substantive contact with the magazine might be seen by participants as collusion. In a letter to all participants, NSSE made its position clear: NSSE had nothing to do with the U.S. News request (though the magazine had a perfect right to make it); and it was up to each institution to decide how to respond. More than a hundred institutions chose to supply U.S. News with requested data about selected questions on the survey (about a quarter of then-current NSSE participants). This request was repeated in 2004 with somewhat a somewhat lower level of response. Again, participation rates in NSSE seemed unaffected as a record number of almost 500 institutions chose to participate in the survey in its most recent year. But this episode illustrates the delicacy of using a voluntary reporting process—regardless of government or private sponsorship—to serve public accountability purposes. IMPACT From the standpoint of providing information useful to institutions to guide internal improvement, NSSE has clearly had a major positive impact. The four national reports issued to date list scores of examples of how individual institutions have harnessed NSSE data for local planning and quality-enhancement purposes. Further examples quoted in these reports strongly demonstrate the utility of NSSE information in informing “soft” accountability processes that rely on institution-chosen measures of effectiveness (Kuh 2003). For example, some institutions have made indicators constructed from the NSSE survey the centerpiece of their response to state requests for information about performance. Many more have featured NSSE data in their institutional accreditation reviews (with no small number reporting such data on publicly-accessible websites). In fact, this latter use of the NSSE data has become so prominent that NSSE is in the midst of constructing an “accreditation toolkit” to guide institutions in preparing and displaying data explicitly for this purpose. More explicitly, though admittedly far less frequently, NSSE data have been displayed in a format that allows direct comparisons of performance as originally envisioned. And, as noted earlier, some twelve states have to date arranged for all public institutions to administer the survey as part of their accountability programs, with states like North Carolina and Kentucky featuring comparative institutional results on their websites. At a different level of analysis, the National Center for Public Policy in Higher Education (an independent policy agency) has twice used aggregated NSSE data to compare states in its prominent fifty-state “report card” on higher education (NCPPHE 2002, 2004). With respect to the NSSE’s sponsors’ original intentions, however, neither the survey nor its associated view of quality have directly displaced media rankings as public markers of quality. And the lack of participation by highly-selective “national” institutions means that its impact in shaping the marketplace of college choice has so far been limited. But NSSE’s “semi-public” strategy of aggressive media relations—together with its growing use as a state reporting tool—continues to attract considerable press and political attention. Seeing a potential opportunity to compete with U.S. News in the lucrative college choice market, for example, both Newsweek and Time magazines have published fall issues featuring stories based on NSSE. More recently last fall, the prestigious Atlantic Monthly magazine parodied the rankings mania and highlighted NSSE as posing the kinds of questions that parents and prospective students ought to be asking about the colleges they are considering. Indeed, The Atlantic has approached NSSE for a “preferred” relationship with respect to reporting information for its fall 2004 college issue, though it is as yet unclear what this might mean. Meanwhile, NSSE continues its campaign to affect public perceptions of quality through its own initiatives. One of the most interesting of these is the production of a pocket-sized brochure on student engagement entitled College: What You Need to Know Before You Go that outlines the questions that prospective students should ask when they visit a college. Although impacts of this campaign are as yet unknown, several hundred thousand of these brochures have to date been distributed to high school guidance counselors and others charged with helping students select colleges to attend. Finally, NSSE has clearly had an impact on campuses themselves in stimulating conversations about academic quality improvement among faculties. Indeed, a faculty version of the instrument, the Faculty Survey of Student Engagement (FSSE) was developed precisely to help stimulate such conversations (http://nsse.iub.edu/fsse/index.cfm). Teaching staff complete this survey, which consists of parallel items on the NSSE, before they know what their students’ responses look like in order to provide a point of contextual comparison. Many institutions have used FSSE and NSSE together as part of their required accreditation reviews, and this appears to be a process that could be employed effectively in other quality review settings based on academic audit or similar mechanisms. COMPARISON WITH RELATED POLICIES The closest counterpart to NSSE as a policy instrument is the Australian Student Course Experience Questionnaire (SCEQ) which was first administered as a census to all enrolled undergraduate students in 1999, and which has subsequently been administered annually to stratified random samples of undergraduate students. Questions on this survey address various aspects of the student experience including multiple items on the quality of teaching and learning. The survey is administered centrally by the Teaching Evaluation Enhancement Service and results are disseminated publicly as part of an institutional quality indicators system. The SCEQ was preceded by the Course Experience Questionnaire (CEQ), which asks similar questions of university graduates and has been administered since 1993. Both surveys are based on the Quality in Courses survey originally developed in the U.K. by Paul Ramsden. (Further information about these surveys and their policy applications can be obtained at www.itl.usyd.edu.au/Teval.) The primary difference between the Australian approach and NSSE is the fact that both Australian surveys are administered by a public agency, participation is mandatory, and results are presented publicly. As a growing number of U.S. states participate in NSSE, it will be interesting to see if the leverage on institutional behavior they obtain parallels the Australian experience. CONCLUSION Although NSSE has fallen short of the direct impacts on student choice and institutional behavior that the Pew Trusts originally hoped for, the survey’s indirect impact on public perceptions of quality and on institutional actions to improve practice appear to be substantial and growing. Continued media presence and widespread voluntary acceptance by institutions are indeed gradually “changing the public conversation about quality.” But the continuing problems that surround public release of NSSE results illustrates the difficulty of using voluntary processes to decisively steer institutions in desired directions if government stands on the sidelines. This difficulty is faced by institutional accreditation mechanisms in the U.S. as well. In a different public policy environment, a well-conceived and constructed survey like NSSE that addresses the quality of instructional provision directly might have a notably different impact, as is beginning to be demonstrated by growing state-level participation in NSSE. Yet even when governments decide to use such information to leverage institutional behavior, policymakers face a major dilemma about how to proceed. If, is as typically the case for survey-generated information, they display it as a performance indicator and expect institutions to strive toward improvement simply to improve their public posture, impacts will likely be limited because institutional consequences are equally limited. But choosing to make performance on such measures consequential through approaches like performance funding may be equally inappropriate because of the very nature of survey-based evidence. Unlike direct measures of performance such as enrollments or completion rates, survey-derived statistics are subject to multiple uncertainties because of factors like sampling error and response bias. As a result, they usually lack sufficient precision to appropriately ground formal performance funding schemes or other high-stakes decisions (Ewell 1999). As a result, government’s best bet may be to either advance such statistics more aggressively as aids to student choice—thus harnessing the market—or to use them proactively as benchmarks to trigger deeper questions about teaching and learning for institutions whose performance remains consistently low. Both lines of action, of course, demand universal institutional participation and active government advocacy.
RESOURCES FOR POLICY MAKERS Although many links to web pages and other resources are noted in the body of the text, policymakers may find the following summary of resources on NSSE and information-based quality assurance efforts helpful a) Live Internet Links on NSSE REFERENCES Astin, Alexander W. (1978). Four Critical Years: Effects of College on Beliefs, Attitudes, and Knowledge. San Francisco: Jossey-Bass. Astin, Alexander W. (1993). What Matters in College: Four Critical Years Revisited. San Francisco: Jossey-Bass. Burke, Joseph C. and Associates (2002). Funding Public Colleges and Universities for Performance: Popularity, Problems, and Prospects. Albany, NY: Rockefeller Institute of Government, State University of New York. Carini, Robert and Kuh, George D. (2004). 2002 NSSE-RAND Cross Validation Study: Some Insights into the Role of Student Engagement to Student Learning. Bloomington, IN: Indiana University, Center for Postsecondary Research. Ewell, Peter T. (1999). Linking Performance Measures to Resource Allocation: Exploring Unmapped Terrain. In Quality in Higher Education, 5, 3, 191-209. Ewell, Peter T. (1997). Accountability and Assessment in a Second Decade: New Looks or Same Old Story. In Assessing Impact: Evidence and Action. Washington DC: American Association for Higher Education (AAHE), 7-21. Ewell, Peter T. and Jones, Dennis P. (1997). Actions Matter: The Case for Indirect Measures in Assessing Higher Education’s Progress on the National Education Goals. Journal of General Education, 42, 2, 123-148. Kuh, George D. (2001). Assessing What Really Matters to Student Learning: Inside the National Survey of Student Engagement. Change Magazine, 33 (3), 10-17. Kuh, George D. (2003). What We’re Learning About Student Engagement from NSSE. Change Magazine, 35 (2), 24-32. Machung, Anne (1998). The Rankings Game. Change Magazine, 30 (4), 12-16. McDonough, Patricia M.; Antonio, Anthony Lising; Walpole, Mary Beth; and Perez, Leonor (1997). College Rankings: Who Uses Them and with What Impact? Los Angeles, CA: Graduate School of Education and Information Studies, The University of California at Los Angeles (UCLA). National Center for Higher Education Management Systems (2003). An Analysis of Relationships Between NSSE and Selected Student Outcomes Measures for Seniors Attending Public Institutions in South Dakota. Boulder, CO: National Center for Higher Education Management Systems (NCHEMS). National Center for Public Policy in Higher Education (2002, 2004). Measuring Up: The Fifty-State Report Card for Higher Education. San Jose, CA: National Center for Public Policy in Higher Education. National Governors Association (1986). Time for Results: The Governors’ 1991 Report on Education. Washington, DC: National Governors Association (NGA). Ouimet, Judy; Bunnage, B.; Carini, Robert M.; Kuh, George D.; and Kennedy, John (2004). Using Focus Groups to Establish the Validity and Reliability of a College Student Survey. Research in Higher Education, 45, 233-50. Pace, C. Robert (1979). Measuring Outcomes of College: Fifty Years of Findings and Recommendations. San Francisco: Jossey-Bass. Pascarella, Ernest T. and Terenzini, Patrick T. (1991). How College Affects Students: Findings and Insights from Twenty Years of Research. San Francisco: Jossey-Bass.
|
