.............

The Government of Hong Kong made substantial investments in higher education during the decade beginning in the mid-1980s. These investments more than doubled the fraction of school-levers attending postsecondary institutions, to just under 20%, and the number of institutions grew accordingly. The two traditional universities, The Hong Kong University and the Chinese University of Hong Kong , broadened and deepened their degree offerings. The Region's two polytechnics increased their production of bachelor's degrees, reduced sub-degree enrollments, and eventually achieved university status as the Hong Kong Polytechnic University and the City University of Hong Kong. The newly founded University of Science and Technology, opened in the early nineties, soon became a force to be reckoned with across Asia . Hong Kong 's liberal arts colleges became full-fledged universities: Hong Kong Baptist University and Lingnan University . With the advent of the Institute of Education , the Hong Kong University Grants Committee (UGC) was responsible for eight postsecondary institutions by the year 2000.

Hong Kong 's universities are self-accrediting. As such, they can set their own standards and curricula without outside intervention. Absent self accrediting status, institutions must get their courses approved by the Hong Kong Council for Academic Accreditation. Achievement of self accrediting status emancipates an institution from detailed regulation and makes it a substantially autonomous entity. Each UGC institution has its own Council, manages its own finances, procurement, and physical plant, and employs its own academic and non-academic staff outside the civil service system.

Funding from Government comes as a block grant whose size is determined by the UGC, with most remaining money coming from tuition. UGC funding, which comprises about 80% of funding, is built up from notional allocations for teaching (68%), research (22%), and performance and role related factors (10%), The teaching component depends on a model driven by student numbers differentiated by field of study, level (bachelors, masters, etc.), and mode of attendance (part-time v. full-time). Tuition rates and student numbers have historically been regulated, but the degree of regulation is declining. The research component is determined mainly by a Research Assessment Exercise (RAE), which will be described presently. The UGC reserves the right to adjust its funding allocations according to judgment and does so regularly—for example, the results of audit are said to “inform funding,” though not in a formulaic way.

Prior to the nineteen nineties, the UGC's approach to quality assurance consisted of institutional visitations in which a broad range of university operations was reviewed during a two- or three-day period. The visits were not unlike institutional accreditation visits in the United States as they were being conducted at the time. The reviewers, which generally included most or all UGC members, sought to familiarize themselves with the institution's governance, priorities for use of resources, quality of faculty and staff, research and scholarship, and academic standards. But while the agenda was broad the evidence obtained was not particularly deep. UGC members were able to form impressionistic conclusions but it was hard to drill down into particular areas – especially the quality of education as actually delivered to students. The institutional visits' shortcomings became increasingly apparent as the number and variety of institutions grew, and mitigating these shortcomings became an important objective for the UGC.

The rise of research in Hong Kong exposed additional shortcomings. Research was viewed as important for the Region's economic development and, also, as crucial for the development of top-flight universities. Research growth was spurred by the Research Grants Council, which the UGC created and funded circa 1990. All the UGC institutions sought to appoint and promote research-active academic staff, who in turn demanded investments in research infrastructure, increased numbers of students taught by research, and often reduced teaching loads.

The large research investments made measuring research activity and outcomes a high priority for the UGC. This led to the triennial Research Assessment Exercise (RAE), which was implemented circa 1993 and continues to this day. The RAE measures the publications and other scholarly work-products of academic staff and assesses the degree to which each staff member is “research active”. Because research activity as measured by RAE drives more than 80% of the UGC's notional research allocation, it became an enormously important incentive for both the institutions and staff members. 

THE POLICY PROBLEM 

The growth of postsecondary education in Hong Kong coincided with the rise of academic quality assurance around the world. Country after country came to realize that the traditions upon which universities had relied for centuries to assure quality could not cope with dramatic increases in participation rates and huge investments in research. The UGC was quick to recognize this problem. It understood the need for QA in both teaching and research from the outset, but moved first to establish the RAE because it needed to direct its investments and also because the task appeared more tractable.[1]

Quality assurance for teaching and learning emerged as a top priority as the RAE's influence on academic priorities became apparent. The UGC joined the International Network of Quality Assurance Agencies for Higher Education (INQAAHE) and in 1994 this author, the UGC member who had headed the original RAE, was asked to research QA for teaching and learning and make recommendations about the way forward.

Stripped to their essentials, the available approaches fell into three categories. The first, rooted in US-style accreditation, sought to determine whether an institution's governance processes and resources were sufficiently robust for it to be capable of educating students at degree level. The UGC believed that its institutions passed this test: after all, as funding agency it was already analyzing the schools' finances and making institutional visitations. The second approach, practiced in Denmark , the Netherlands , and in the Higher Education Funding Council of England's subject level assessments, used external assessors to evaluate the delivered quality of education (“external assessment”). The third approach, developed by the UK 's Academic Audit Unit and practiced in New Zealand and Sweden , viewed quality assurance as an institutional obligation and audited the degree to which institutions were discharging their responsibilities (“academic audit”).

The UGC's policy problem was how to discharge its obligation to Government and the public to assure the quality of teaching and learning without disempowering the institutions, infringing their autonomy, or spending too much in relation to the results achieved. However, the Committee wanted to do more than assure traditional academic standards: it wanted to use the QA process to spur improvement in teaching and learning. The policy problem's urgency was underscored by institutional diversity, which meant that “quality” had to be defined differently in different places, and evidence that the RAE was diverting staff attention from teaching and learning at all institutions.

The Committee made its decision based on the principle that quality assurance is intertwined with quality improvement, which is unquestionably an institutional responsibility. Furthermore, institutional autonomy and the Committee's history of collegial interaction with the universities favored the “light touch” represented by audit over the more intrusive interventions needed for external quality assessment. Finally, committee members, including this author, were concerned about the high cost of external assessment and doubted whether good evaluations could in fact be made. US-style accreditation had been ruled out for the reasons given above, which left academic audit as the method of choice. Two rounds of audit have been conducted since this decision and ways of integrating a further round into a general institutional review framework now are being considered.[2]

Rather than adopt the UK 's original audit approach, which was judged to be insufficiently improvement-oriented, the UGC set out to invent its own methodology. (We knew little about the Swedish and New Zealand approaches, which in any case were in their infancy.) This paper describes academic audit as developed and used in Hong Kong . The method was named “Teaching and Learning Quality Process Review” (TLQPR) to avoid perceived negative connotations associated with the word “audit,” but I have come to prefer the term “Education Quality Audit.”

The Hong Kong experience has led to implementations in Missouri and Tennessee and the descriptions that follow will draw upon this experience. [3] It also is worth noting that different lineages of academic audit are developing around the world. They differ from the Hong Kong audits in scope (e.g., education quality or all academic operations) and the definition of “quality work” (described in the next section), but not in fundamental approach.[4] Referring to the Hong Kong , Missouri , and Tennessee implementations as education quality audits calls out their lineage while differentiating them from the other types of academic audit.

Because the RAE measures number of academic staff whose work meets preset quality standards, it can be viewed as combining QA with measurement of the amount of activity. For teaching, the analogous quantity measure is student numbers. One needs a separate QA exercise for teaching because the relation between student numbers and quality standards is not automatic. For more discussion on the RAE see French, et. al, (1999, 2001).

CONTENT OF THE POLICY INSTRUMENT 

Education quality audits can best be understood using the flowchart in Figure 1. The chart consists of three elements: inputs, teaching and learning processes, and learning outcomes. The forward-facing arrows depict how inputs energize teaching and learning processes, which then produce learning outcomes. But what is most relevant to audit are the backward-facing or “feedback” arrows. To produce education quality, teachers must consistently measure the quality of outcomes, contrast it with their objectives, and then adjust the processes as needed to fix problems or effect improvements (arrow A). Process adjustments also can result from self-reflection and comparisons with best practice inside and outside the university (arrow B). Finally, process adjustments may trigger changes in the type, amount, and quality of needed inputs (arrow C). The performance of processes without feedback, which are said to run “open loop,” is sure to degrade over time. Decades of experience in quality assurance in a wide variety of fields demonstrate that feedback is essential for maintaining quality.

Figure 1 
The Production of Quality Education 

In a complex environment like education, getting and interpreting the feedback and acting on it requires more than casual effort. Faculty who assess learning carefully and apply what they've learned to improve their teaching do better than those who don't. Likewise, faculty who spend time reflecting on their teaching and thinking about how to improve it tend to produce more learning than their colleagues. The same is true for departments. Those that stress learning assessment and reflection on teaching processes generally produce better teaching. Furthermore, they build a “culture of quality” that triggers a self-perpetuating cycle of improvement. Reflective and evidence-rich feedback processes also help departments optimize their use of inputs and, where necessary, to make the case for additional resources.

The Audited Activity: “Education Quality Work” (EQW)

Feedback is the key to effective quality assurance. For example, one can measure learning outcomes and then take corrective action if quality falls below standard. Or one can measure perceptions about teaching and learning processes, as in student course evaluations, and then take corrective action if the evaluations are unsatisfactory. No feedback means no corrective action and thus no QA. And to get ahead of our story slightly, feedback without a goal or standard to compare against is largely useless.

The examples also illustrate the tight connection between quality assurance and improvement: in each case the corrective action represents an effort to improve. The activities required to set standards, assess outcomes, and take corrective action – in other words, to create and use the feedback loops – have come to be called “Education Quality Work” or EQW for short.[5] We shall see that EQW gets performed at the department level, at the level of a school or faculty, and at the level of a campus or institution.

EQW can be defined as:

Organized activities dedicated to improving and assuring educational quality . They systematize a university's approach to quality instead of leaving it mainly to unmonitored individual initiative. They provide what higher education quality pioneers David Dill and Frans van Vught call “…a framework for quality management in higher education…drawn from insights in Deming's approach, but grounded in the context of academic operations.[6]

Hong Kong 's approach to education quality audit examines EQW rather than the inputs, teaching and learning processes, and learning outcomes that most observers view as being the only determinants of education quality. The auditors determine whether systematic feedback processes exist and, if so, what kind. They ask whether the processes make systematic use of evidence, and whether the evidence is robust or circumstantial. They ask whether faculty members and departments compare the evidence with policy objectives and their own clearly-stated goals and, if so, whether they act promptly and decisively to correct discrepancies. Education quality audits evaluate the maturity of institutions' EQW. Thinking broadly, all lineages of academic audit can be said to evaluate the maturity of “quality work” somewhere in the institution.

Audit's focus on quality work has positive implications for institutional autonomy and academic freedom. For example, auditors do not substitute their judgments about the quality and quantity of inputs or the appropriateness of teaching and learning processes for those of institutional leaders and faculty. What they do is ask whether those judgments are characterized by careful reasoning and informed by good evidence. Nor do they try to measure learning outcomes. They ask whether the local academics are measuring outcomes adequately and whether they use the information systematically to improve teaching. Getting a good audit score depends on having evidence, including evidence from learning assessments, and then using it systematically. However, getting a good score does not depend on matching the reviewers' preconceptions about educational content, teaching methods, or the “right way” to assess learning. It is sufficient that the respondent's judgments flow logically from evidence, that they take account of established policy, and people have exercised due diligence in making them.

The proposition that audits of EQW are sufficient for education quality assurance depends upon two fundamental assumptions.

•  Most professors want to teach well. Unless stymied by resource constraints or driven by incentives that discourage investment of time in teaching, they will use feedback to effect improvement—especially if the feedback has been produced by a collegial process.

•  Most professors have only sketchy knowledge of EQW and, therefore, of how to generate and use feedback. They are trained as content experts, and while most have acquired an understanding of conventional teaching and assessment methods they have little experience with organized quality improvement and assurance activities.

While exceptions can be found at both the individual and institutional level, informed and objective observers generally agree that these assumptions do in fact characterize modern universities.

The above implies that better EQW will pay off for teaching: i.e., that new tools for the improvement of teaching will in fact be put to good use. Furthermore, because EQW includes student learning assessment, better assessments will improve the stock of information about education quality— information that is eagerly sought by external quality assurance agencies and the public. Audit spurs better EQW and vets EQW maturity. It also can vet the efficacy of information about education quality supplied by the institution to the public. The bottom line here is that external quality assessment isn't the only way to get good information about education quality to Government and the public. Audit also can do that job better and, as argued later, there is reason to believe more effectively.

A primer on education quality work as audited in Hong Kong , Missouri , and Tennessee can be found in the Appendix. It provides brief descriptions of the “focal areas” of EQW (the subjects that audit should cover), some principles by which the efficacy of a respondent's EQW can be judged, and a maturity scale for summarizing the audit results. The material is important because understanding the education quality audit as a policy instrument requires an understanding of EQW itself. Agencies that adopt education quality audit may wish to substitute their own materials, but having some kind of standards to audit against is essential.

All academic audits involve two basic steps: (1) the entity being audited prepares a self-evaluation of its quality work; and (2) the audit panel reads the self-evaluation, visits the entity, and prepares a report. These two steps do not differ from most other types of evaluation. What is different is the content of the self-evaluation and of the conversations that take place during the audit visit. The differences reflect audit's improvement orientation as well as its focus on quality work.

Consistent with its improvement orientation, education quality audit elicits structured conversations among auditors and auditees about how the EQW quality principles described in the Appendix are being applied across the five focal areas. “Conversations” are important because the complex issues of teaching and learning quality are best addressed through dialog. “Structure” is important because the auditors must cover all the relevant topics, gauge quality process maturity, and produce a meaningful report.

But however structured audit's basic design, the conversations themselves are free-flowing and collegial. Respondents are encouraged to “come as you are” and standup presentations are held to a minimum. This approach has several advantages. First, the auditors and auditees learn about EQW from each other, which spurs improvement. Second, the auditors learn whether the auditees' descriptions of their EQW activities are “for real”—the gloss one can sometimes hide behind in a PowerPoint or document breaks down in deep conversation. Third, conversation blurs the distinction between accountability and improvement. Auditees learn that what they are accountable for is a sincere effort to improve, not adherence to rigid standards. They want to hold up their end of the conversation, and this introduces a degree of self-accountability.

Approaching audit through structured conversation mitigates the problem attributed to the original UK academic audits: that their focus on formal policies and documentation resulted in a bureaucratic “paper exercise.” Policy statements and other written materials should be present in the audit rooms for reference before, during, or after the audit conversations. However, the dialog should be more concerned with respondents' attitudes, behavior, and command of quality processes and principles—i.e., their EQW maturity—than written policies and paper trails. The Hong Kong auditors stressed that “it's what you're doing that matters,” not the precision of your documentation. 

IMPLEMENTATION 

Hong Kong 's education quality audits involved six distinct steps: (i) the initial design process; (ii) onsite briefings or workshops for prospective auditees; (iii) the auditees' self-studies; (iv) the audit visits; (v) preparation of the audit reports; and (vi) a meeting to debrief the exercise and share exemplary practice after the audit round was completed. Steps (iii)-(v) encompass the two “basic steps” introduced above. However, all six steps are crucial for a successful implementation.

Initial Design

The exercise began with a detailed design for how EQW concepts would be introduced, what would be included in the self-study, how the audit visit should be conducted, and how the report would be written and promlogated. The UGC felt the auditee institutions should participate in the design process, so each campus appointed members to a Consultative Committee that worked with UGC members and staff through both the first and second audit rounds. The Committee included people responsible for quality assurance and improvement on their respective campuses. They contributed valuable insights about emergent design ideas and provided a reality check on the result. Most became enthusiastic supporters of quality improvement and of education quality audit, and they helped transmit this enthusiasm to their colleagues within the institutions.

Onsite Briefings

Each institution's introduction to EQW, before the first audit round, began with a two to three hour briefing by the chair of the UGC's audit team and a few of his colleagues. The briefing occurred about nine months before the audit visit. It described quality process concepts and principles, the institutional self-study, and the audit visit and report. The session was open to all faculty and staff, and participation often numbered in the hundreds. In addition to launching the self-study, the briefing sought to focus attention on EQW and initiate self-reflection and improvement.

The UGC secretariat followed up on the briefing with written guidance notes describing the self-study and arrangements for the audit visit. The team chair and a member of the Secretariat visited each campus a second time about four months before the audit to finalize the arrangements. The briefing and follow-up visits were omitted in the second audit round because people were familiar with the exercise and the requisite activities already were being conducted on the campuses.

Self-Study

Doing the self-studies stimulated institutions to reflect on their EQW and begin working on improvements prior to the audit team's arrival. As in other quality assurance regimens, the self-study reports helped orient the audit team before its arrival on campus. Members could request additional information and/or supporting documentation before the audit visit.

Initially the institutions were free to structure the self-study reports as they wished and include appendices of any length—which soon overwhelmed the audit team. The UGC responded by putting a twenty-page limit on the self-study reports and discouraging voluminous appendices. Lists of relevant documents were included, however, so team members could conveniently request the ones they wanted.

Audit Visit

The visits were conducted by intact teams of eighteen members in Round 1 and ten members in Round 2.[7] The large size in Round 1 was due to inclusion of one member of the Consultative Committee from each institution. Eight UGC overseas academics also served, along with two overseas academic quality experts who were not UGC members. The second-round team was similar except that it included the consultative committee chair but not other members. Ten is still a fairly large number of auditors, but the size was dictated by the need for division into subgroups as described below.

The audit visits lasted between 1½ and 2 days depending on the size of the institution. More time might have been desirable, but the limit was dictated by the availability of the overseas UGC members and experts. In the event, the amount of time available did prove sufficient.

A typical visit schedule follows.

Day 1

•  Executive session (60 minutes). Team members compared notes on the self-study, look at documents, and plan their queries.

•  Opening plenary with the institution's president, chief academic officer, and other senior officers (45 minutes). The president gave an opening presentation not to exceed fifteen minutes. Questioning by team members generally addressed institutional priorities and policy issues raised by the self-study.

•  Plenary with the institution's Quality Assurance or equivalent committee (45 minutes). There was no opening presentation. Questions generally involved institution-level QA policies and procedures

•  Plenary with students (30 minutes). The group often consisted of representatives from student government and/or institutional student-faculty committees. Questions addressed perceptions about education quality, whether students were involved in quality assurance, whether the problems they identified were addressed promptly, and whether prompt feedback on resolution was forthcoming.

•  First set of small-group sessions as described below (90 minutes)

•  Second set of small-group sessions (90 minutes)

•  Executive session to recap the day (30 minutes)

Day 2

•  Third set of small-group sessions (90 minutes)

•  Plenary with the deans of schools (60 minutes). Questions generally addressed the deans' familiarity with education quality processes and principles, and their role in the institution's self-regulation of quality.

•  Executive session to recap the morning and plan the audit report (90 minutes)

•  Exit conference with the opening plenary group (30 minutes)

•  Executive session to recap the exit conference (15 minutes)

The time allocations varied depending on institutional size and complexity.

Panel size allowed for six replications in each of the three small-group sets: for a total of eighteen separate meetings. (The sub-panels had three people in Round 1 and two in Round 2—two members proved sufficient.) About two-thirds of the sessions were with departments; the rest were with schools and special-purpose entities like educational technology and teacher development units. Most respondents were faculty, but students always were included. Numbers ranged from half a dozen to as many as twenty people. The ninety-minute sessions were divided three ways: about seventy minutes with the whole group, ten minutes with the students separately, and ten minutes in an executive session. The students tended to be fairly quiet in the general session but opened up when asked separately, “You heard the faculty—is this how things really look to you?”

The small-group sessions were the most important part of the audit visit. Conversations at the grass-roots level allowed panelists to get past the formalities of policies and procedures and find out what was really happening on the ground. The multiple replications also provided data about interdepartmental and interschool variance—which often contradicted the positive face put on by institutional leaders and quality assurance committees. Moreover, the grass-roots conversations proved almost impossible to fake. Faculty in departments that had embraced quality processes would back up their remarks with a rich mosaic of examples, whereas those whose experience was limited to lip service would soon sputter into generalities. The subgroups noted good and bad examples of quality work and assigned capability-maturity scores for subsequent discussion with the full audit panel.

The auditors also tested quality processes further up in the institution's academic hierarchy. For example, they quizzed deans and their associates about EQW in their schools and, in particular, what they were doing to improve weak-performing departments. The teams observed considerable variation in the deans' knowledge and attitudes. Some deans were aware that certain of their departments needed improvement and were working to achieve that, whereas others didn't know and still others knew but didn't believe they were responsible for effecting change. Such observations were usefully provocative in our subsequent plenary sessions with the deans and institution-level leadership. One of the points pressed by audit is that everyone in the hierarchy, from president to individual professors, should take education quality seriously. Deans, provosts, and presidents should join with quality assurance committees in reinforcing the quality message at every opportunity. They should take all needed steps to assure and improve departmental EQW.

The desire to detect variance conditioned the selection of which departments and schools to visit. The institution made nominations, but the panel chair and UGC Secretariat always added their own selections. Sometimes these were based on hunch or insider knowledge, sometimes simply by a desire to span a range of disciplines while visiting multiple departments within a given school. The selections were announced about a month before the audit visit. This meant all departments and schools had to participate in the institution's preparation for audit and that the ones selected could not over-prepare. Selected units were asked to table a one- or two-page “talking paper” to guide discussion of their quality processes but otherwise no special preparation was required.

Audit Report

The reports described each institution's education quality processes and, importantly, what it was doing to improve them. They did not grade or rank the institutions, but careful reading does reveal a rough ranking. (Links to the reports for both rounds can be found at www.ugc.edu.hk.) The team chair wrote all the reports in Round 1 but workload dictated that a professional secretary (a retired professor at one of the institutions) do the job in Round 2. The Secretariat sent the report drafts to the institutions for correction of significant factual errors, but no attempt was made to vet the draft with the individual units visited. Hence examples of good and bad practices at the grass-roots level were not identified as to unit. The reports were written in non-technical language in order to make them as accessible as possible.

The UGC viewed the institutions as owning the reports but required publication in both English and Chinese along with whatever comments the university wished to make. The press took a keen interest, and some reports turned out to be lightening rods for discussion. This was positive on the whole, since it highlighted the importance of education quality and quality work for the general public as well as for the institutions.

Debriefing Session

The Consultative Committee convened a Region-wide meeting to air comments on the Round 1 exercise and share exemplary practices identified during the audits. The meeting was attended by several hundred faculty and staff from the eight institutions. The testimony was mainly positive, and many exemplary practices were described. [8] Presentation of such practices served a dual purpose: to propagate the specific practices and to illustrate quality process concepts through the vehicle of examples. 

IMPACT 

Four kinds of evidence about impact can be identified: (i) testimony from auditors and, especially, auditees; (ii) changed institutional behavior; (iii) external evaluation of the audit process; and (iv) evidence from a subsequent audit.

Testimony from people involved in the audits

The testimony from participants tended to be positive. The debriefing session after Hong Kong 's Round 1 included favorable testimonials from a cross section of university respondents. The closer a person was to the audit processes, whether as an auditor or an auditee, the more likely it was that his or her opinion would be favorable. There were concerns about the number of different reviews being conducted by the UGC, but few people having first-hand experience with audit complained that the time spent was not worthwhile.

The UGC received numerous reports to the effect that “we should have been asking ourselves these questions all along” and “at last someone cares about education quality.” One should recognize that these comments came from converts, but the participants' conversion was by no means preordained. Most heartening was confirmation that the structured conversations were in fact meaningful to auditors and auditees alike.

Changed institutional behavior

While the Hong Kong audits did not rank the institutions, the UGC was sensitive to whether the institutions were taking their quality work seriously. It had been made clear from the beginning that the audit results might “inform funding” but not in a formulaic way. Fortunately the linkage hardly ever had to be demonstrated, but when it was the reason was poor performance and a lack of demonstrated willingness to improve. Removing a small increment of funding solved the problem in short order. Having made its point and obtained reports that the situation was improving, the UGC restored funding to its previous level. The problem did not reoccur.

The example teaches an important lesson about audit's capacity to further an agency's accountability and improvement goals simultaneously. The Hong Kong audits are oriented primarily toward improvement, but their role as an accountability tool also was recognized from the beginning. Because the concept of quality work was largely undefined prior to the first exercise, the UGC did not penalize institutions for EQW immaturity. However, it acted decisively in when good-faith improvement efforts were not forthcoming. Exemplary EQW might well be rewarded, and conversely, in future audit rounds when the nature and importance of EQW has become clear to all.

External evaluation

As a quasi-governmental agency accountable to the public, the UGC commissioned a formal review of the first audit round. The review was performed by the Center for Higher Education Policy Studies (CHEPS) at the University of Twente , NL, which fielded a team of international experts including the head of the PPAQ project.

The CHEPS team interviewed audit participants in all the UGC institutions to ascertain their views on the exercise and its impact. The team's major conclusions follow. (Recall that Hong's name for audit is “TLQPR”.)

•  The overall major conclusion is that ‘TLQPR was the right instrument at the right time', because the review was a positive stimulus to institutional attention to teaching.

•  There were clear achievements with respect to the first goal of TLQPR - to focus on teaching and learning as the primary mission of higher education institutions . The signal given by TLQPR that teaching and learning were as important as research was seen by many as the prime impact. [The team also noted that the substantial funds associated with the Research Assessment Exercise continued to drive incentives in that area.]

•  With respect to the second goal of TLQPR - to assist higher education institutions in their efforts to improve teaching and learning quality assurance processes - there appeared to be institutionalisation of quality management procedures in some institutions, especially those that did not have such procedures in place before. A number of examples of innovation in existing quality procedures were also found. Sustainment of present efforts and new initiatives would be helped by the signal that there would be a second round of external reviews of teaching and learning processes [a signal that was given].

•  The TLQPR has certainly contributed to achieving the third goal - accountability of the UGC and the higher education institutions to society - firstly, through the review process itself and secondly through the publication of the reports and the institutional progress reports. However, in the eyes of the higher education community, the press coverage of the reports was not seen as a balanced reflection of the process.[9]

Evidence from a subsequent audit

The most telling evidence about the impact of education quality audit came from Hong Kong 's second audit round. Once again an audit team visited each of the Region's institutions for an in-depth review of education quality work. Team members who participated in both rounds concluded unanimously that the institutions had made great progress. All demonstrated systematic EQW with substantial traction at every level from departments to the central administration. Some institutions had gone so far as to institute “internal education quality audits” to maintain momentum between the UGC's visits. Several with poor results in Round 1 scored well in Round 2 and no institution regressed. The degree of progress across the Region surpassed the UGC's most optimistic expectations. While improved quality processes don't guarantee improvements in delivered educational quality, it is hard to believe that such improvements are failing to materialize. The one disappointment from the second round concerned student learning assessment. While progress had been made, most departments still had a long way to go. The evaluation and improvement of student learning assessment remains a high priority for subsequent exercises.  

COSTS 

The numbers of audit visits and the types and numbers of visitors are the main cost drivers for any onsite quality evaluation program. Audit offers significant advantages on both dimensions. A quality assurance agency need mount only one audit per institution as opposed to the separate evaluations of each department needed for subject-level evaluations: for n institutions and an average of m departments per institution, this means only n as opposed to n x m visits. Second, auditors need not be expert in any particular discipline, which greatly simplifies team selection.

The external cost of audit depends on local circumstances and design details, but a sample calculation of the staffing requirements for Hong Kong-style audits is possible.

Per-visit cost drivers

•  Audit teams usually run in the range of 6 to 10 members, with larger numbers for more complex venues that require more small-group sessions.

•  An audit requires from 1 to 2 days on site and about an equal number of days for visit preparation and reviewing the draft report.

•  Team chairs require 2 or 3 additional days for report writing and first-time auditors require a day of training.

Sample calculation (Assume 5 institutions with 2-day visits and an intact team of 8 auditors.)

Institutional costs are harder to calculate because it's never clear which activities should be attributed specifically to audit and which to things that should be done anyway. For example, faculty should attend quality process workshops and strive for improvement whether they will be audited or not. The fact that the audit stimulates such behavior should be viewed as a benefit, not a cost. The same is true for the reflection that informs a self-study. The actual writing up of the self-study, direct preparation for the audit visit, and the visit itself might reasonably be counted as a cost of audit, yet even these activities confer benefits. The key point is that audit addresses practical issues associated with important day-to-day activities rather than requiring large amounts of bureaucratic make-work.

COMPARISONS 

While detailed comparisons of the Hong Kong education quality audits with other quality assurance methods are beyond the scope of this paper, a brief summary may prove useful.

Variants of Audit

Academic audit started with the UK 's Academic Audit Unit (AAU), which was organized by the Committee of Vice Chancellors and Principles circa 1990—in part as a counterweight to the Higher Education Funding Council of England's external assessment initiatives. The emphasis was on quality assurance policies and processes and not so much on improvement. Two criticisms were leveled at the AAU's approach, both of which were discussed earlier in this paper. The first was that the audits were overly concerned with formalities and documentation. The second was that audit doesn't provide external assessments of education quality. Citing duplication of QA effort in the UK , the government merged audit with external assessment in a new Quality Assurance Agency (QAA) in the mid-1990s. The QAA adopted external assessment as its primary methodology but changed to audit circa 2001, Both audit and assessment play a role currently, and the situation continues to evolve.

Another academic audit lineage runs from the UK through New Zealand 's Academic Audit Unit to the Australian Universities Quality Agency (AUQA). Australia 's audits are comprehensive. They go beyond education quality to include governance, research, management, access, and support services among other things. The preparation and review processes are very well organized and the Agency staff provides strong support to the audit teams. The reports are taken seriously by the institutions and by the public. The Australian process provides a good model for agencies that wish to pursue a comprehensive audit approach [www.auqa.edu.au]. AUQA also is developing a good practices database.

Denmark 's new audit program was t riggered by legislation in 2003 that requires universities “systematically to develop and improve the quality of their processes for teaching and learning. ” The approach draws on UK , Australia , and Hong Kong as well as the Danish Evaluation Agency's own rich history of subject-level external assessment.[10] Agency staff again play a strong role: for example, they gloss the self-evaluation documents, provide auditors with suggested questions, and draft the panel's report. The University of Copenhagen and The Technical University of Denmark received audit visits during 2004 and reports have been issued [ www.eva.dk] .

The first education quality audit application in the United States was begun in 2001 by the University of Missouri System. [provost.missouri.edu/faq/academic-audit.html] The approach was to apply audit at the department level. One department on each of the University's four campuses was audited in 2003 with good results, and a successful second round has just been completed. The second U.S. application was by the Tennessee Board of Regents, which audited department-level EQW on 13 of its 19 campuses during 2005 [ www.tbr.state.tn.us/academic_affairs/acadaudit/audit.htm ] . This project also appears to have been successful. The methodology used in Missouri and Tennessee is a direct lineal descendent of the one developed Hong Kong .

Other variants of audit also are emerging in the U.S. . The Senior College Commission of the Western Association of Schools and Colleges (“WASC,” a regional accreditor) piloted institution-level audit at the California State University at Fullerton in 1999 but has yet to roll out the method in a major way [ www.fullerton.edu/wasc/Newsletter4.htm] . Other regional accreditors have adopted elements of education quality audit, as have subject-level accreditors like the Teacher Evaluation Council (TEAC) and the Association of Collegiate Schools of Business (AACSB). While not related to higher education, it is interesting to note that the Education Commission of the States (ECS) adopted education quality audit as the core concept for its second-generation accountability proposal for primary and secondary education. [11]

External Assessment

The strongest and most recurring criticism of audit is that it doesn't provide an “objective third-party assessment” of education quality. We heard this criticism from people in the Hong Kong Government and the Hong Kong Council for Academic Accreditation, for example, and one runs into it frequently in discussing academic quality assurance around the world. The Research Assessment Exercises conducted in the UK and Hong Kong represent strong forms of external quality assessment, so why not do the same for education?

The answer flows from the difficulty of assessing education quality. Research and scholarship can be judged by artifacts, especially peer-reviewed publications, which external assessors can review at whatever length they desire. Even this is not easy, given that there is not just one but four kinds of scholarship [12] and the observed differences in what's considered “quality” across disciplines,[13] but it is vastly easier than assessing the delivered quality of education.

Traditional external assessments focus on the quality of faculty, staff, and infrastructure, and the degree to which the breadth and depth of curricula meet generally accepted standards for the degree being granted. But while these are necessary conditions for quality, they are not sufficient. A curriculum that passes the “generally accepted” test may not fit the needs of the student segments served by a particular institution, or it may be poorly taught. Hence quality assurance agencies are focusing more and more on outcomes measures.

But measures of what? Decided by whom? How implemented? How used to effect improvement? Programs in which third parties assess education quality are costly and inspire institutional resentment. Furthermore, the history of the so-called “assessment movement” in the United States , which demands that institutions assessment learning outcomes but ignore the other aspects of EQW, does not inspire confidence. I have written extensively about these matters and will offer only a brief summary here.[14]

External assessment compares the “educational production function” (the three boxes in Figure 1) against a predetermined standard for what's good in the circumstances. Education quality audit compares EQW (the feedback arrows in Figure 1) against a standard. Both methods consider learning outcomes. External assessment makes its own outcomes quality determination whereas audit reviews the institution's measurements. This might seem like a small difference but it has profound consequences.

First, external assessment is inherently confrontational. Hence it is difficult to approach quality in a collegial way, let alone combine quality assurance and improvement goals in the same exercise. [15] Education quality audits as defined in this paper are inherently collegial. They are designed from the ground up to combine improvement with accountability.

Second, external assessors bear the burden of proof for getting valid quality measures. Institutions can hardy be expected to search out and lay bare their shortcomings. That's the assessor's job. The complexity and subtlety of higher learning makes this a difficult burden to bear, all the more so when quality is defined in terms of the institution's own goals—goals they can interpret retrospectively when their quality is challenged.

Auditors, on the other hand, need only ascertain whether institutions and faculty are doing good job of EQW. Because the standards for effective EQW don't vary across institutions and disciplines, the auditors' task is far simpler. The burden of proof shifts to the respondent—for example, to convince the auditors that effective feedback loops exist and are being used? Countless examples will be brought to the auditors' attention if the answer is “yes.” However, a negative answer means no examples will be forthcoming and respondents will stumble repeatedly in their dialog with the auditors.

It is my strong conviction that university-level education is too complex and subtle—and the opportunities for institutions to go into a “compliance mode” and withhold, distort, or simply not produce evidence are too great— to make external assessment the method of first choice for QA agencies. (Assessments may be useful in special circumstances. The UK , for instance, relies mainly on audit but reverts to assessment when an institution fails the audit.) The danger is that, when propagated on a large scale, external assessment will oversimplify the definition of quality and drive higher education toward its lowest common denominator. It also will be very costly. In my view, those who demand external assessment regardless of circumstances are seeking a magic bullet that simply doesn't exist.

My concern about external quality assessment applies with much less force to assessments conducted within the institutions, particularly for departmental majors and organized general education programs. What is difficult or impossible when propagated at the level of a higher education system becomes manageable at the program level. The development of robust and meaningful outcomes-based performance indicators, managed at the local level as part of EQW, is not too much to hope for.

In closing, I must admit that critics who argue that universities cannot be trusted to perform their own quality assessments may have an element of history on their side. However, it is precisely this history that education quality audit is designed to change. EQW as described herein embeds learning outcomes assessment in a complete program of quality assurance and improvement. Education quality audits spur the development of EQW and, after allowing a reasonable period for maturity to develop, they can be used to hold institutions accountable for good EQW. In the long run, when locally-based assessment methods have matured and become generally accepted, institutions also can be held accountable for results as measured by locally generated, audited, performance indicators. The road to accountability through local action is longer than through preemption of quality assessment by external agencies, but the results will be more meaningful.

To test the efficacy of this assertion, I suggest that a robust external assessment system be given its own education quality audit. Let the assessors have structured conversations with experienced auditors about the assessment's conclusions and the evidence that supports them, the methods used to obtain the evidence, and the difficulties, if any, encountered during the assessment process. I predict that the external assessors will get less complete evidence and draw less valid conclusions about education quality than faculty within institutions characterized by mature EQW—faculty whose work is, of course, audited.
 

RESOURCES FOR POLICY MAKERS

General links

University Grants Committee www.ugc.edu.hk

 

Review Template: Second Round Teaching and Learning Quality Process Reviews http://www.ugc.edu.hk/eng/doc/ugc/publication/prog/tlqpr/2nd_tlqpr_review_template_e.pdf

One example of TLQPR documents

Hong Kong University of Science and Technology (from http://publish.ust.hk/vpaao/tlq/report/eng_version/index.html )

 

TLQPR report

Self-evaluation documents

 

Main document

Supplements (postgraduate education, continuing education)

University's response

Review methodology

Review templates

All institutional audit reports

REFERENCES

Boyer, Ernest L. (1991), Scholarship Reconsidered: Priorities of the Professoriate. Princeton , NJ: Carnegie Foundation for the Advancement of Teaching. 
Dill, David D. (2000). “Designing Academic Audit” lessons learned in Europe and Asia .” Quality in Higher Education , Col. 6, No. 3, pp. 187-207. 
Dill, David D., (1992), Quality By Design: Toward a Framework for Academic Quality Management. Higher Education: Handbook of Theory and Research, pp. 37-83. New York : Agathon Press, Inc. 
ECS (2003), “Accountability – Next Generation Models: Quality Improvement Model.” Denver , CO : Education Commission of the States. 
French, Nigel J. , Ping K. Ko, William F. Massy, Helen F.H.  Sui, and Kenneth Young (1999), “Research Assessment in Hong Kong .” Journal of International Education, Vol. 10, No. 1 (Spring), pp. 46-53. 
French, Nigel J., William F. Massy, and Kenneth Young (2001), “Research Assessment in Hong Kong .” Higher Education (Vol. 42, No. 1, July). 
Harvey, Lee (1999), “Quality in Higher Education”. Paper at the Swedish Quality Conference, Goteborg (November). 
Massy, W. F., and N. J. French (1999b), “Teaching and Learning Quality Process Review: What Has the Programme Achieved in Hong Kong ?.”  Presented at the Fifth Conference of the International Network for Quality Assurance Agencies in Higher Education (Santiago de Chile, May 2-5). 
Massy, William F. (1997), “Teaching Learning Quality Process Review: The Hong Kong Programme,” Quality in Higher Education, 3(3): 249-262. 
Massy, William F. (2000), “Energizing Quality Work: Higher Education Quality Evaluation in Sweden and Denmark .” Technical Report: National Center for Postsecondary Improvement, Stanford University . 
Massy, William F. (2003), Honoring the Trust: Quality and Cost Containment in Higher Education. Bolton, MA: Anker Publishing Company, Inc. 
Massy, William F. (2004), “Academic Audit for Accountability and Improvement.” in Joseph C. Burke (Ed.), The Many Faces of Accountability: Holding Higher Education Responsible for Performance (working title), Jossey-Bass (forthcoming). 
Massy, William F. and Nigel J. French (1999a), “Teaching and Learning Quality Process Review: what the program has achieved in Hong Kong .” Quality in Higher Education, Vol. 7, No, 1 (April), pp. 33-45. 
Meade, P., and D. Woodhouse (2000), “Evaluating the Effectiveness of the New Zeeland Academic Audit Unit: Review and Outcomes.” Quality in Higher Education, April, pp. 19-30. 
Trow, Martin (1994), Academic Reviews and the Culture of Excellence. Stockholm : Kanslersãmbetet. 
van Vught, Frans (1994), “The New Context for Academic Quality.” Center for Higher Education Policy Studies, University of Twente , Enschede, Holland . Paper for Symposium: “University and Society”, Vienna (June 9-10). 
Wahlén, Staffan (1998), “Is there a Scandinavian model of evaluation of higher education?” Higher Education Management, Vol. 10, No. 3, pp. 27-41. 
Westerheijden, D., Brennan, J., Dill, D., Shah, T. and Verkleij, A. (1999), A Campaign for Quality: Hong Kong Teaching and Learning Quality Process Review. Hong Kong : University Grants Committee. 

APPENDIX: PRIMER ON EDUCATION QUALITY WORK (EQW) [16]

Quality Focal Areas

Education quality addresses five key “focal areas” of education quality. [17] In the first focal area the questions center on learning objectives . What should students whom the department has taught know and be able to do? (Goals and standards are essential for the use of feedback.) How do the students' educational experiences contribute to their employment success, their capacities as citizens, and their quality of life? Are the specified learning objectives based on the needs of enrolled students rather than the ideal student most faculty want teach?

The next focal area deals with the curriculum and co-curriculum. [18] How does the curriculum relate to the program's learning objectives? What is being taught, in what order, and from what perspective? Does the curriculum build cumulatively on the students' prior knowledge and capacity? To what extent does the co-curriculum, those organized experiences outside the classroom, support the curriculum? Does the curriculum meet accepted standards for the degree, to the extent such standards have been articulated by accreditors or other external bodies?

The third focal area centers on teaching and learning processes. For example, what methods are employed for introducing students to new materials, for interpreting those materials and answering student questions, for stimulating student involvement, and for providing feedback on each student's work? Is learning active? Is technology being used, and if so, is it being exploited effectively?

The fourth area focuses on the assessment of student learning . What measures are used to assess student learning? Are they aligned with the learning objectives? Do they compare beginning and ending performance to ascertain value added? Who is actually responsible for student learning assessment—each individual faculty member? A department committee? Members of the administrative staff, either within or outside the department?

The fifth and final area focuses on the institution's or department's processes for assuring educational quality. Can faculty and administrators assure first themselves and subsequently the audit team that the designs for curricula, teaching and learning activities, and student assessments are being implemented as intended? Can they be certain, in short, that curricula and teaching are subjects of robust evaluations?

Quality Principles

To achieve rigor, academic auditors need principles against which to judge activities in each of the focal areas. The principles are still evolving, but the ( U.S. ) National Center for Postsecondary Improvement (NCPI) developed a starting set that was used successfully in Hong Kong . The principles have their roots in business, health care, and government quality work, but they have been adapted for use in academe. They are analogous to the “Generally Accepted Accounting Principles” (GAAP) used in financial audits. As in finance, academic auditors can ascertain whether respondents know about and practice the principles, and then follow up on shortfalls.

The first principle is obvious but often overlooked: define educational quality in terms of outcomes . The quality of student learning, not teaching per se, is what ultimately matters. The outcomes should pertain to what is or will become important for the students enrolled in the program. Outcomes mandated by institutional or oversight-agency policy should of course be observed, but the entity should have its own outcome goals in any case.

The second principle calls for a focus on process—on how things get done. It becomes important to know, in some detail, how teachers teach, how students learn, and how each approaches the task of assessment.

Third, quality should be everyone's business. Faculty need to demonstrate collegiality in teaching, just as they do in research. The department, as the organizing unit, needs to encourage faculty members to work together, to hold one another accountable, and to bring a broad array of talent to bear on difficult problems. The goal of such teamwork is to make the institution or department a learning organization with respect to education and EQW as well as disciplinary content.

The fourth principle calls for decisions to be based on evidence . Faculty should collect data on student preparation, learning styles, and, where relevant, probable requirements for employment. The data—testimonies by current and former students, and perhaps by employers or the faculty members who taught them in graduate or professional school, along with the numeric data culled from the institution's student record system—need to be analyzed carefully in light of disciplinary standards along with the faculty's own professional experiences. The results should have a direct and demonstrable impact on curricula, learning processes, and assessment methods. [19]

Fifth, coherence should be viewed as a virtue. The goal is to have departments see learning through the lens of the student's entire educational experience. In an ideal curriculum courses build upon one another to provide the desired depth and breadth, and students' educational portfolios should reflect the same coherence.

Sixth, there is a paramount need to identify and learn from best practice . Institutions and departments should seek out examples of good practice and adapt the best to their own circumstances. They should compare well- versus average or poorly-performing methods and students, assess the causes of the differences, and seek ways to minimize the variation.

Seventh and finally, continuous improvement should be viewed as not just an important but an attainable priority . Quality should be everybody's business all of the time. While faculty will continue to place strong emphasis on research, they should spend enough discretionary time on educational quality to keep the improvement process moving. The department's as well as the institution's personnel committees need to make the results of such work, along with teaching and research performance, a criterion for promotion and tenure.

The quality principles are couched in terms of departmental performance—that is, what a department should be doing to improve and assure educational quality. Translation to multi-disciplinary programs like general education is straightforward: the committee or other entity responsible for program quality should apply the quality principles. What may seem less straightforward is how the principles apply to a school (a faculty in most places outside the United States ) or to the institution as a whole. But the answer is clear. Deans, and the quality committees in their areas, need to make sure that the departments under their purview apply the principles effectively. Chief academic officers should make sure the deans take education quality seriously and hold their departments accountable. Systemwide administrators and external quality agencies need to make sure campus leaders do the same. The audit methodology, described under “Implementation,” is designed to help the responsible parties at each level perform these tasks.

EQW Maturity Ratings

Having established the focal areas to be addressed by audit and principles of good practice in each area, the last requirement is a language by which auditors can describe the performance of an institution or department. Fortunately, by the time of the second Hong Kong audit round, Carnegie-Mellon University had developed its capability maturity model for tracking the prowess of software development teams. Somewhat unexpectedly, the categories and definitions, along with the model's conceptual approach, helped the Hong Kong auditors gauge the relative maturity and hence systematization of the quality processes in its eight universities. The Hong Kong auditors used the capability maturity scale in their internal deliberations but did not make their ratings public. Since then, however, the UGC has begun experimenting with maturity ratings for institutions and focal areas based on the published audit reports. Departmental maturity levels were discussed publicly in Missouri and they are integral to both the self-study and auditing processes in Tennessee .

The capability maturity scale's zero point is no effort at all. The department or institution being evaluated does not have organized educational quality processes. Quality and quality assurance remain in the hands of individual professors. Next comes firefighting. The entity responds to problems, but mostly with ad hoc methods. The five focal areas described above are not covered systematically, and the quality principles receive little attention.

Mid-point on the scale is occupied by informal effort. The entity can report individual initiatives and experimentation with the principles in one or more focal areas. Coverage remains spotty, however, and the entity had yet to become a learning organization with respect to its educational quality processes.

The fourth point on the scale is reached when the entity's quality process show evidence of organized effort. The entity plans and tracks quality process initiatives in all five focal areas. Emergent norms encourage investment in the quality principles. Methods for gauging performance are under development.

The true winners are departments or institutions that have reached the scale's terminus: mature effort . The quality principles have become embedded in the entity's culture, and the idea of regular improvement in all five focal areas has become an accepted way of life. The entity recognizes the planning, tracking, and performance evaluation of quality processes as important elements of peer accountability and collegiality, and it has developed appropriate and feasible performance indicators.

The Hong Kong experience suggests that auditors, and also most chairs, deans, and provosts for that matter, can use the scale to evaluate the maturity of an entity's EQW. Departments, schools, and institutions with immature education quality work can be encouraged or spurred to do better. Fully mature entities can be celebrated—one can say that education quality has become “job one” when an entity reaches the mature end of the scale.
 

FOOTNOTES 

 [1] Because the RAE measures number of academic staff whose work meets preset quality standards, it can be viewed as combining QA with measurement of the amount of activity. For teaching, the analogous quantity  measure is student numbers. One needs a separate QA exercise for teaching because the relation between student numbers and quality standards is not automatic. For more discussion on the RAE see French, et. al, (1999, 2001). 

[2] Massy (1997); Massy and French (1999a, b).

[3] Education quality audit was significantly improved during the second Hong Kong round, and again in the Missouri and Tennessee implementations. Because this paper is a policy analysis and not a case study, I will describe the current state of the art rather that the method as originally implemented. Areas where the method has changed materially will be noted, however.

[4] See, for example, Harvey (1999), Dill (2000); Meade and Woodhouse (2000), Massy (2000), Wahlén (1998).

[5] Sweden 's National Agency for Higher Education coined a term for describing the subject matter of academic audit. The term translates to English roughly as “Education Quality Work” (EQW). Massy (2003, 2004) uses the term “Education Quality Processes” (EQP). However, designating EQW as EQP invites confusion with teaching and learning processes..

[6] Massy (2003). The references within the quotation are van Vught (1994) and Dill (1992).

[7] In Round 2, additional two-person sub-panels addressed research post-graduate programs and continuing education.

[8] Massy and French (1999a).

[9] Westerheijden, et. al. (1999), “Major Conclusions” section.

[10] See Massy (2000).

[11] ECS (2003).

[12] Boyer (1991).

[13] French, et al (1989, 2001)

[14] Massy (2003, 2004)

[15] Trow (1994).

[16] This Appendix describes what the author considers to be current best practice. A comprehensive statement (including sample audit questions), developed for the Tennessee Board of Regents, can be found at www.tbr.state.tn.us/academic_affairs/acadaudit/audit.htm .

[17] The focal area definitions used by the UGC, called “domains” rather than focal areas, differed slightly from the ones presented here. Those domain definitions were curriculum, teaching and learning processes, student learning assessment, quality assurance, and resources devoted to quality processes. The current best-practice definitions evolved subsequent to (and a a result of) the first UGC audit round. However, given everyone's familiarity with the original definitions, the UGC chose not to adopt the change for the second round.

[18] While the Hong Kong audits did not explicitly address the co-curriculum, co-curricular issues did arise frequently.

[19] The Tennessee audits make use of principles for the use of evidence in teaching and learning prepared by the Senior College Division of The Western Association of Schools and Colleges (WASC); see “Resources for Policymakers.”