Appendix C

IOM Committee on Health Professions Education: A Bridge to Quality­Abbreviated Executive Summary

Committee on the Health Professions Education Summit
Board on Health Care Services

Ann C. Greiner, M.C.P.

Elisa Knebel, M.H.S.

Editors

COMMITTEE ON THE HEALTH PROFESSIONS EDUCATION SUMMIT

EDWARD M. HUNDERT (Co-Chair), President, Case Western Reserve University

MARY WAKEFIELD (Co-Chair), Director, Center for Rural Health, School of and Health Sciences, University of North Dakota

J. LYLE BOOTMAN, Dean and Professor, College of Pharmacy, Arizona Sciences Center, University of Arizona

CHRISTINE K. CASSEL, Dean, School of Medicine, Oregon Health & Science University

WILLIAM CHING, Joint M.D.-Ph.D. Student, School of Medicine, New York University

MARILYN P. CHOW, Vice President, Patient Care Services, Kaiser Permanente

STEPHEN N. COLLIER, Director and Professor, Center for Health Policy Workforce Research, Towson University

JOHN D. CROSSLEY, Vice President for Operations and Nursing Practice, Head, Division of Nursing, M.D. Anderson Cancer Center, University of Texas

ROBERT S. GALVIN, Director Global Health, General Electric

CARL J. GETTO, Senior Vice President for Medical Affairs, University of Wisconsin Hospital, and Associate Dean for Hospital Affairs, University of Medical School

ROBIN ANN HARVAN, Director, Office of Education, University of Health Sciences Center

POLLY JOHNSON, Executive Director, North Carolina Board of Nursing

ROBERT L. JOHNSON, Professor and Interim Chair of Pediatrics, Professor of Psychiatry, Director, Division of Adolescent and Young Adult Medicine, New Jersey Medical School at UMDNJ

DAVID LEACH, Executive Director, Accreditation Council for Graduate Medical Education

JUDY GOFORTH PARKER, Professor, Department of Nursing, East Central University

JOSEPH E. SCHERGER, Dean, College of Medicine, Florida State University

JOAN SHAVER, Dean and Professor, College of Nursing, University of Illinois at Chicago

DAVID SWANKIN, President and CEO, Citizen Advocacy Center

Abbreviated Executive Summary

Abstract

The 2001 Institute of Medicine report Crossing the Quality Chasm: A New Health System for the 21st Century recommended that an interdisciplinary summit be held to develop next steps for reform of health professions education lead to enhancement of patient care quality and safety. In June 2002, the IOM convened this summit, which included 150 participants across disciplines and occupations. This follow-up report focuses on integrating a core set of competencies­patient-centered care, interdisciplinary teams, evidence-based practice, quality improvement and informatics‹into health professions education.

The report's recommendations include a mix of approaches related to oversight processes, the training environment, research, public reporting, and leadership. The recommendations targeting oversight organizations include integrating core competencies into accreditation, and credentialing processes across the professions. The goal is an outcome-based education system that better prepares clinicians to meet both the needs of patients and the requirements of a changing health system.

The summit and follow up report were supported by the Health Resources and Services Administration (HRSA), the Agency for Healthcare Research and Quality (AHRQ), the ABIM Foundation, and the California HealthCare Foundation (CHCF).

Education for the health professions is in need of a major overhaul. Clinical education simply has not kept pace with or been responsive enough to shifting patient demographics and desires, changing health system expectations, evolving practice requirements, new information, a focus on improving quality, or new technologies (Institute of Medicine, 2001):

• Health professionals are not adequately prepared to address shifts in the nation's patient population, including increased prevalence of chronic conditions (Cantillon and Jones, 1999; Council on Graduate Medical Education, 1999; Davis et al., 1999; Grantmakers in Health, 2001; Halpern et al., 2001; Health Resources and Services Administration, 1999; Pew Health Professions Commission, 1995, (Calabretta, 2002; Frosch and Kaplan, 1999; Gerteis et al., 1993; Mansell et al., 2000; Mazur and Hickam, 1997; Wu and Green, 2000). This changing landscape requires that clinicians be skilled in providing ongoing patient management; deliver and coordinate care across teams, settings, and time frames; and support patients' endeavors to change behavior and lifestyle­training for which is in short supply in today¹s clinical education settings (Calabretta, 2002).
• Once in practice, health professionals are asked to work in interdisciplinary teams, often to support those with chronic conditions, yet they are not trained in team-based skills.
• These same clinicians are confronted with a rapidly expanding evidence base, but are not schooled in how to evaluate and apply this evidence base to practice (American Association of Medical Colleges, 1999; Detmer, 1997; Green, 2000; Shell, 2001).
• Although there is a spotlight on the serious mismatch between what we know to be good-quality care and the care that is actually delivered, students and health professionals have few opportunities to learn how to analyze the root causes of quality problems and to design systemwide fixes (Baker et al., 1998; Buerhaus and Norman, 2001).
• While clinicians are trained to use an array of cutting-edge technologies related to care delivery, they often are not provided a basic foundation in informatics (Gorman et al., 2000; Hovenga, 2000).
• While there are notable pockets of innovation, these are by and large exceptions to the rule.

Building a Bridge to Cross the Quality Chasm

Numerous recent studies have led to the conclusion that "there is abundant evidence that serious and extensive quality problems exist throughout American medicine, resulting in harm to many Americans" (Schuster et al., 1998). Crossing the Quality Chasm: A New Health System for the 21st Century (Institute of Medicine, 2001) emphasizes that safety and quality problems exist largely because of system problems, and that browbeating health professionals to just try harder is not the answer to addressing the system's flaws and future challenges. The report concludes that change around the edges will not work and sets forth an ambitious redesign agenda. Further, it provides initial guidance on what kinds of competencies clinicians would need to carry out this agenda and emphasizes additional examination to better understand how the workforce should be educated, how it should be deployed, and how it should be held accountable.

Specifically, the Quality Chasm report recommends that a multidisciplinary summit of leaders within the health professions be held to discuss and develop strategies for restructuring clinical education across the full educational continuum. The Committee on the Health Professions Education Summit was convened to plan and hold this summit‹which was held on June 17­18, 2002 and to produce a follow-up report. Summit participants were asked to develop proposed strategies and actions for addressing the five competency areas recommended by the committee (described below) for health professions education. The committee reviewed the ideas proposed by summit participants as part of its deliberations.

A New Vision for Health Professions Education

With the ideal health care system described in the Quality Chasm report as a backdrop, the committee developed a new, overarching vision for clinical education in the health professions that is centered on a commitment to, first and foremost, meeting patients' needs.

All health professionals should be educated to deliver patient-centered care as members of an interdisciplinary team, emphasizing evidence-based practice, quality improvement approaches, and informatics.

This vision is apparent in selected institutions around the country, but is not incorporated into the basic fabric of health professions education, nor is it supported by oversight processes or financing arrangements. Accordingly, the committee proposes a set of five core competencies that all clinicians should possess, regardless of their discipline, to meet the needs of the 21st-century health system. Competencies are defined here as the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice (Hundert et al., 1996).

• Provide patient-centered care­respect patients' differences, values, and expressed needs; communicate with and educate patients; share decision making and management; and continuously advocate disease prevention and promotion of healthy lifestyles.
• Work in interdisciplinary teams­cooperate, collaborate, communicate, and integrate care in teams to ensure that care is continuous and reliable.
• Employ evidence-based practice­integrate best research with clinical expertise and patient values for optimum care, and participate in learning and research activities to the extent feasible.
• Apply quality improvement­identify errors and hazards in care; continually measure quality of care; and design and test interventions to change processes and systems of care in order to improve quality.
• Utilize informatics­communicate, manage knowledge, mitigate error, and support decision making using information technology.

To formulate the core competencies, the committee examined the skills outlined in the Quality Chasm report, various health professions' efforts to define competencies, and relevant literature (ABIM Foundation, 2002; Accreditation Council for Graduate Medical Education, 1999; American Association of Medical Colleges, 2001; Brady et al., 2001; Center for the Advancement of Pharmaceutical Education [CAPE] Advisory Panel on Educational Outcomes, 1998; Halpern et al., 2001; O'Neil and the Pew Health Professions Commission, 1998). The five competencies are meant to be core, overlap with other health professions' efforts to define competencies, and should not be viewed as an exhaustive list. The committee also acknowledges that the core competencies will differ in application across the disciplines and across educational settings, e.g., didactic versus training.

Next Steps

With some notable exceptions, most current and past reform efforts have focused within a particular profession (Bellack and O'Neil, 2000; Christakis, 1995; Harmening, 1999; Jablonover et al., 2000), however the committee believes the time has come for leaders across the professions to work together on the cross- cutting changes that must occur to affect comprehensive reform in clinical education and related training environments.

The committee believes that integrating a core set of competencies‹one that is shared across the professions‹into the health professions oversight spectrum would provide the most leverage. The committee also recommends pursing other leverage points‹such as enhanced information, e.g., performance metrics, and improved training environments­but the preponderance of its 10 recommendations are directed at oversight bodies, which include accrediting, licensing and certifying organizations. Health professions oversight processes, such as accreditation and certification, function at the national level and thereby afford a mechanism for systemwide change.

The call for accrediting and certifying organizations to move toward a competency-based approach to education is in response to growing concerns about patient safety (Institute of Medicine, 2000), the persistent and substantial variation in patient care across geographic settings unrelated to patient characteristics (O'Connor et al., 1996; Wennberg, 1998), and the desire on the part of public payers and consumers for increased accountability (Leach, 2002; Lenburg et al., 1999). Competency-based education focuses on making the learning outcomes for courses explicit and on evaluating how well students have mastered these outcomes or competencies (Harden, 2002). The evidence base on the efficacy of various educational approaches is slim. However, the limited evidence that does exist points to improvements, such as better performance on licensing exams, for outcome-based educational approaches (Carraccio et al., 2002)

A competency-based approach to education could result in better quality because educators would begin to have information on outcomes, which could ultimately lead to better patient care. Defining a core set of competencies across educational oversight processes could also reduce costs as a result of better communication and coordination, with processes being streamlined and redundancies reduced. Integrating core competencies into oversight processes would likely provide the impetus for faculty development, curricular reform, and leadership activities.

Common Language and Adoption of Core Competencies

Before steps can be taken to integrate a core set of competencies into oversight processes, an interdisciplinary group will need to define common terms. A number of studies have shown that any collective movement to reform education must begin by defining a shared language (Halpern et al., 2001; Harden, 2002). Such an effort can help set in motion a process focused on achieving a threshold level of consensus across the disciplines around a core set of competencies.

Recommendation 1: DHHS and leading foundations should support an interdisciplinary effort focused on developing a common language, with the ultimate aim of achieving consensus across the health professions on a core set of competencies that includes patient-centered care, interdisciplinary teams, evidence-based practice, quality improvement, and informatics.

Integrating Competencies into Oversight Processes

Recommendations 2-5 address further integration of competencies into existing oversight processes varies. Such efforts would be strengthened if predicated on a core a set of competencies shared across the professions. During the last decade, competencies have begun to redefine accreditation, particularly in pharmacy and medicine, and such competencies overlap with the core competencies recommended by the committee (American Council on Pharmaceutical Education, 2002) (Accreditation Council for Graduate Medical Education, 2002). Until they are fully incorporated and evaluated, it remains to be seen what effect these competencies will have on pharmacy and medical education. In nursing, the two accrediting organizations also have defined competencies‹which do not fully overlap with the core competencies defined here‹but differ in whether they require demonstration of such competencies (Commission on Collegiate Nursing Education, 2002; National League for Nursing Accrediting Commission, 2002).

The competency movement, however, does not have as much of a foothold in licensure and certification processes. Requirements for maintaining a license vary considerably, as do requirements for those who pursue recognition of clinical excellence. Further, research has raised questions about the efficacy of continuing education courses, the most common way to demonstrate ongoing competency (Cantillon and Jones, 1999; Davis et al., 1999).

Efforts to incorporate a core set of competencies across the professions into the full oversight framework­accreditation, licensing, and certification­would need to occur on the national, state, and local levels; coordinate both public- and private-sector oversight organizations; and solicit broad input. Again, the involvement of DHHS, and specifically the Health Resources and Services Administration, would be important in getting this effort off the ground. It is imperative to have linkages among accreditation, certification, and licensure; it would mean very little, for example, if accreditation standards set requirements for educational programs, and these requirements were not then reinforced through licensing exams.

Recommendation 2: DHHS should provide a forum and support for a series of meetings involving the spectrum of oversight organizations across and within the disciplines. Participants in these meetings would be charged with developing strategies for incorporating a core set of competencies into oversight activities, based on definitions shared across the professions. These meetings would actively solicit the input of health professions associations and the education community.

Strategies for incorporating the competencies into oversight processes would necessarily differ across the oversight framework based on history, regulatory approach, and structure. In all cases, the oversight bodies should proceed with deliberation. The experiences of ACPE and ACGME are instructive, with both organizations undertaking decade-long efforts to reform their processes (Byrd, 2002).(Batalden et al., 2002). What has not yet occurred is coordination across various professional accrediting bodies. Such coordination would obviate the need for each to reinvent the wheel, promote synergies, and enable better communication and working relationships, as well as more consistent integration of the core competencies across schools.

Recommendation 3: Building upon previous efforts, accreditation bodies should move forward expeditiously to revise their standards so that programs are required to demonstrate­through process and outcome measures­that they educate students in both academic and continuing education programs in how to deliver patient care using a core set of competencies. In so doing, these bodies should coordinate their efforts.

With the exception of patient-centered care, which is consistently included in examinations across the professions, licensing exams for health professionals vary considerably in whether they test for competency in the core areas (National Association of Boards of Pharmacy, 2002; National Council of State Boards of Nursing, 2001; United States Medical Licensing Exam, 2002). This situation also needs to be addressed and could be the focus of a subset of the oversight organizations described in recommendation 2. In addition, geographic restrictions on licensure and separate and sometimes conflicting scope-of-practice acts need to be examined to determine whether they are a serious barriers to the full integration of the core competencies into practice, and if so, how to modify them so that all clinicians can practice to the fullest extent of their technical training and ability. Although beyond the scope of this report, the committee believes that this matter deserves further examination because licensure and scope of practice influence how clinicians are deployed, which in turn affects decisions about education.

Finally, the committee believes that there should be a focused effort to integrate a core set of competencies into oversight processes focused on practicing clinicians. Such an effort would require coordination among an array of public- and private-sector licensing and certification organizations, within which there is a currently little uniformity in approach across the professions or within a given profession across the states. At present, many boards require only a fee for license renewal (Swankin, 2002b; Yoder-Wise, 2002), and many others view continuing education courses as evidence of competence, even though this has not been shown to be a reliable measure of such ability (Davis et al., 2000; O'Brien et al., 2001).

To begin with, state legislatures would need to require state licensing boards to insist through rigorous means that their licensees demonstrate competence, not just pay a license renewal fee, to maintain their authority to practice. Licensing boards also would need to consider clinician competency at varying career stages. The committee believes that all health professions boards need to require demonstration of continued competency, and that they should move toward adopting rigorous tests for this purpose. Beyond licensure examinations, there is evidence to suggest that structured direct observation using standardized patients, peer assessments, and case­and essay-based questions are reliable ways to assess competency (Epstein and Hundert, 2002; Murray et al., 2000)

Recommendation 4: All health professions boards should move toward requiring licensed health professionals to demonstrate periodically their ability to deliver patient care­as defined by the five competencies identified by the committee­through direct measures of technical competence, patient assessment, evaluation of patient outcomes, and other evidence-based assessment methods. These boards should simultaneously evaluate the different assessment methods.

There is more uniformity among certifying organizations as compared with professional boards, in that nearly all require some means of demonstrating continuing competence. The vast majority allow for two or more approaches, and many also consider competency at various career stages. Moreover, in response to the paucity of evidence that taking continuing education courses improves practice outcomes, some certifying organizations are beginning to emphasize alternative measures that are more evidence based (American Board of Medical Specialties, 2000; American Nurses Association/ NursingWorld.Org, 2001; Bashook et al., 2000; Board of Pharmaceutical Specialties, 2002; Federation of State Medical Boards, 2002; Finocchio et al., 1998; National Council of State Boards of Nursing, 1997-2000; Swankin, 2002a). Certification bodies should recognize continuing education courses as a valid method of maintaining competence if there is an evidence-based assessment of such courses; if clinicians select courses based on an assessment of their individual skills and knowledge; and if clinicians then demonstrate, through testing or other methods, that they have learned the course content. The committee recognizes that there is a monetary and human resource cost to moving to evidence-based assessment, whether it is related to licensure or certification. Consequently, such assessments may need to be phased in, or less costly assessment methods identified.

Recommendation 5: Certification bodies should require their certificate holders to maintain their competence throughout the course of their careers by periodically demonstrating their ability to deliver patient care that reflects the five competencies, among other requirements.

Training Environments

Education does not occur in a vacuum; indeed, much of what is learned lies outside of formal academic coursework. A "hidden curriculum" of observed behavior, interactions, and the overall norms and culture of a student's training environments are extremely powerful in shaping the values and attitudes of future health professionals. Often, this hidden curriculum contradicts what is taught in the classroom (Ferrill et al., 1999; Hafferty, 1998; Maudsley, 2001).

Consequently, the committee believes that initial support should be provided for existing exemplary practice organizations that partner with educational institutions, and are already providing the interdisciplinary education and training necessary for staff to consistently deliver care that incorporates the core competencies. These learning centers could test various approaches for incorporating the core competencies into education for students, clinicians, and faculty, and provide guidance to practice and educational organizations about key operational issues. Is problem-based learning the best approach to teaching these competencies? Should the teaching of these competencies be infused into other courses, or should they be stand-alone? These learning centers should also consider how, after an initial investment, they could become self-sustaining in 3-5 years.

Recommendation 6: Foundations, with support from education and practice organizations, should take the lead in developing and funding regional demonstration learning centers, representing partnerships between practice and education. These centers should leverage existing innovative organizations and be state-of-the art training settings focused on teaching and assessing the five core competencies. There are many barriers to incorporating the five competencies into the practice environment, where medical residents and new graduates in allied health, nursing, and pharmacology obtain initial training that leaves an important imprint on their future practice (Partnership for Solutions, 2002). In addition to the barriers of time constraints, oversight restrictions, resistance from the professions, and absence of political will, the health care financing system is a large impediment to integrating the core competencies into practice settings. Therefore, the committee believes steps must be taken to explore alternative ways of paying clinicians to foster such integration.

As the largest payer, Medicare has a major effect on the system when it innovates (Institute of Medicine, 2002). Moreover, the committee believes that patients with chronic conditions‹a sizable proportion of whom are covered by Medicare‹would benefit greatly from integration of the five competencies into practice. There are a number of different options that could serve as models for these payment experiments, including capitation, bundled payments, bonuses, withholds, and various ways to share risk and responsibility between clinicians and payers (Bailit Health Purchasing, 2002; Guyatt et al., 2000). The committee encourages other payers to follow suit.

Recommendation 7: Through Medicare demonstration projects, the Centers for Medicare and Medicaid Services (CMS) should take the lead in funding experiments that will enable and create incentives for health professionals to integrate interdisciplinary approaches into educational or practice settings, with the goal of providing a training ground for students and clinicians that incorporates the five core competencies.

Research and Information

Along with oversight changes and supportive training environments, the committee believes that evidence of the efficacy of an educational intervention can be a catalyst for change. To this end, evidence related to the link between clinical education and health care quality needs to be better developed, as does evidence about various teaching approaches.

In a review of 117 trials in continuing education, fewer than 20 percent were found to use health care outcomes as their measure of effectiveness (Davis et al., 2000). Teaching itself is dominated by intuition and tradition, which do not always hold up when submitted to empirical verification (Tanenbaum, 1994; van der Vleuten et al., 2000). The committee believes the time has come to focus energy and resources on developing a more robust and compelling evidence base about what matters in patient care and what works in teaching clinicians so that educators, payers, and regulators can assess objectively what needs to be emphasized in the health professions curricula and what should be eliminated. The research should also span disciplines.

Recommendation 8: The Agency for Healthcare Research and Quality (AHRQ) and private foundations should support ongoing research projects addressing the five core competencies and their association with individual and population health, as well as research related to the link between the competencies and evidence-based education. Such projects should involve researchers across two or more disciplines.

The committee believes that incorporation of education-related measures into quality-reporting efforts and ongoing monitoring will be required to realize the vision articulated in this report. The lack of standardized information about the quality of clinical education makes the job of leaders seeking to reform such education more difficult. The lack of standardized measures also sets clinical education apart from the broader health care quality movement. A focused effort to develop education-related measures must begin now, given the amount of time required to develop and test prospective measures before they can be incorporated into report cards. The committee recognizes that initially there will be a small number of measures ready for public reporting.

Recommendation 9: AHRQ should work with a representative group of health care leaders to develop measures reflecting the core set of competencies, set national goals for improvement, and issue a report to the public evaluating progress toward these goals. AHRQ should issue the first report, focused on clinical educational institutions, in 2005 and produce annual reports thereafter.

Providing Leadership

Significant reform in health professions education is a challenge to say the least. The oversight framework is a morass of different organizations with differing requirements and philosophies, now under considerable pressure to demonstrate greater accountability (Batalden et al., 2002; Finocchio et al., 1998; Leach, 2002; O'Neil and the Pew Health Professions Commission, 1998). In academia, deans, department chairs, residency directors, and other leaders face a stream of requests for adding new elements to a curriculum that is already overcrowded. Shortages of key professionals are another significant challenge. Moreover, funding for some academic health centers has been under pressure, and states trimming education budgets (Griner and Danoff, 2000).

When change happens in health professions education, it does not happen overnight. Multiyear processes are required to develop, review, and achieve consensus on new requirements before they can be implemented. Given this environment, the committee believes that reform of clinical education will be possible only with the skill and commitment of a broad range of health care leaders. A recent analysis and synthesis of 44 curriculum reform efforts revealed that leadership is the factor most often cited as affecting curriculum change (Bland et al., 2000).

Recommendation 10: Beginning in 2004, a biennial interdisciplinary summit should be held involving health care leaders in education, oversight processes, practice, and other areas. This summit should focus on both reviewing progress against explicit targets and setting goals for the next phase with regard to the five competencies and other areas necessary to prepare professionals for the 21st-century health system.

Conclusion

The committee has set forth 10 major recommendations for reforming health professions education to enhance quality and meet the evolving needs of patients. The staging of these recommendations is important. The first step is to articulate common terms. Once the disciplines have agreed on a core set of competencies, oversight bodies can consider how to incorporate such competencies into their processes. The committee believes that the development of common language and definition of core competencies should happen as rapidly as possible and by no later than 2004, given that oversight process changes take considerable time. As this work proceeds, the efforts of leading practice organizations to integrate the core competencies into care delivery should be fostered through regional demonstration learning centers and Medicare demonstration projects. Simultaneously with these efforts, AHRQ and private foundations should provide support for research focused on the efficacy of the competencies and competency education and, most importantly, develop an initial set of measures reflecting the core set of competencies, along with national goals for improvement. Finally, the committee believes that biennial summits of health care leaders who control and shape education­starting in 2004­will be an important mechanism for integrating and furthering the efforts of reform minded leaders.

Building a bridge to cross the quality chasm in health care cannot be done in isolation. The committee hopes that this report will jump start other efforts to reform clinical education, both individually and collectively, so that it focuses on continually reducing the burden of illness, injury, and disability, with the ultimate aim of improving the health status, functioning, and satisfaction of the American people (President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry, 1998b). The public deserves nothing less.

References

ABIM Foundation. 2002. Medical professionalism in the new millennium: a physician charter. Annals of Internal Medicine 136 (3):243-6.

Accreditation Council for Graduate Medical Education. 1999. "General Competencies."

Accreditation Council for Graduate Medical Education. "ACGME Outcome Project."

Agency for Health Care Research Quality. 2002. "NHQR Preliminary Measure Set."

American Association of Medical Colleges. 1999. Evidence Based Medicine Instruction. Vol 2, No.3 edition Washington, DC: AAMC. 2001. "Medical School Objectives Project."

American Board of Medical Specialties. 2000. 2000 ABMS Annual Report and Reference Handbook.

American Council on Pharmaceutical Education. 2002. "ACPE Web site."

American Nurses Association/NursingWorld.Org. 2001. "On-line Health and Safety Survey: Key Findings."

Bailit Health Purchasing. 2002. Provider Incentive Models for Improving Quality of Care. Washington DC: National Health Care Purchasing Institute.

Baker, G.R., S. Gelmon, L. Headrick, M. Knapp, L. Norman, D. Quinn, and D. Neuhauser. 1998. Collaborating for improvement in health professions education. Quality Management in Health Care 6 (2):1-11.

Bashook, P.G., S.H. Miller, J. Parboosingh, and S.D. Horowitz. 2000. "Credentialing Physician Specialists: A World Perspective."

Batalden, P., D. Leach, S. Swing, H. Dreyfus, and S. Dreyfus. 2002. General competencies and accreditation in graduate medical education. Health Affairs 21 (5):103-11.

Bellack, J.P., and E.H. O'Neil. 2000. Recreating nursing practice for a new century: recommendations and implications of the pew health professions commission's final report. Nursing & Health Care Perspectives 21 (1):14-21.

Bland, C.J., S. Starnaman, L. Wersal, L. Moorhead- Rosenberg, S. Zonia, and R. Henry. 2000. Curricular change in medical schools: How to succeed. Academic Medicine 75 (6):575-94.

Board of Pharmaceutical Specialties. 2002. "Recertification."

Brady, M., Leuner J.D., Bellack J.P., Loquist R.S., Cipriano P.F., and O'Neil E.H. 2001. A Proposed Framework for Differentiating the 21 Pew Competencies by Level of Nursing Education. Nursing & Health Care Perspectives 22 (1):30-35.

Buerhaus, P.I., and L. Norman. 2001. It's time to require theory and methods of quality improvement in basic and graduate nursing education. Nursing Outlook 49 (2):67-9.

Byrd, G. 2002. Can the profession of pharmacy serve as a model for health informationist professionals? Journal of Medical Library Association 90 (1):68-75.

Calabretta, N. 2002. Consumer-driven, patient-centered health care in the age of electronic information. Journal of Medical Library Association 90 (1):32-7.

Cantillon, P., and R. Jones. 1999. Does continuing medical education in general practice make a difference? British Medical Journal 318 (7193):1276-79.

Carraccio, C., S.D. Wolfsthal, R. Englander, K. Ferentz, and C. Martin. 2002. Shifting

paradigms: From flexner to competencies. Academic Medicine 77 (5):361-67.

Center for the Advancement of Pharmaceutical Education [CAPE] Advisory Panel on Educational Outcomes. 1998. "Educational Outcomes."

Chassin, M.R., R.W. Galvin, and the National Roundtable on Health Care Quality. 1998. The urgent need to improve health care quality. Journal of the American Medical Association 280 (11):1000-1005.

Christakis, N.A. 1995. The similarity and frequency of proposals to reform US medical education: constant concerns. Journal of American Medical Association 274 (9):706-11.

Collier, S. March 2002. Workforce Shortages. Personal communication to (Ann Greiner).

Commission on Collegiate Nursing Education. 2002. "CCNE Accreditation."

Council on Graduate Medical Education. 1999. Physician Education for a Changing Health Care Environment. Rockville, MD: Health Resources and Services Administration.

Counsell, S., R. Kennedy, P. Szwabo, N. Wadsworth, and C. Wohlgemuth. 1999. Curriculum recommendations for resident training in geriatrics interdisciplinary team care. Journal of the American Geriatrics Society 47 (9):1145-48.

Davis, D., M.A. O'Brien, N. Freemantle, F.M. Wolf, P. Mazmanian, and A. Taylor-Vaisey. 1999. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? Journal of American Medical Association 282 (9):867-74.

Davis, D., M.A. Thomson O'Brien, and N. Freemantle. 2000. Review: Interactive, but not didactic, continuing medical education is effective in changing physician performance. Database of Abstracts of Reviews of Effectiveness Volume 132 (2):75.

Detmer, D.E. 1997. Knowledge: a mountain or a stream? Science 275 (5308):1859.

Epstein, R.M., and E.M. Hundert. 2002. Defining and assessing professional competence. Journal of the American Medical Association 287 (2):226-35.

Federation of State Medical Boards. 2002. "Post-Licensure Assessment System."

Ferrill, M.J., L.L. Norton, and S.J. Blalock. 1999. Determining the statistical knowledge of pharmacy practitioners: A survey and review of the literature 1. American Journal of Pharmaceutical Education 63 (3).

Finocchio, L. J., C. M. Dower, N. T. Blick, C. M. Gragnola, and the Taskforce on Health Care Workforce Regulation. 1998. Strengthening Consumer Protection: Priorities for Health Care Workforce Regulation. San Francisco, CA: Pew Health Professions Commission.

Frosch, D.L., and R.M. Kaplan. 1999. Shared decision making in clinical medicine: past research and future directions. American Journal of Preventive Medicine 17 (4):285-94.

Gerteis, M., S. Edgman-Levitan, J. Daley, and T. Delbanco, Editors. 1993. Through the Patient Eyes. Vol. San Francisco, CA: Josey-Bass.

Gifford, A.L., Laurent D. D., V.M. Gonzales, et al. 1998. Pilot randomized trial of education to improve self-management skills of men with symptomatic HIV/AIDS. Journal of Acquired Immune Deficiency Syndromes and Human Retrovirology 18 (2):136-44.

Gorman, P.J.M., A.H.M. Meier, C. Rawn, and T.M.M. Krummel. 2000. The future of medical education is no longer blood and guts, it is bits and bytes. American Journal of Surgery 180 (5):353-56.

Grantmakers in Health. 2001. Training the Health Workforce of Tomorrow. Washington, DC: Grantmakers In Health .

Green, M.L. 2000. Evidence-based medicine training in internal medicine residency programs a national survey. Journal of General Internal Medicine 15 (2):129-33.

Griner, P.F.M., and D.M. Danoff. 2000. Sustaining Change in Medical Education. Journal of American Medical Association 283 (18):2429-31.

Guyatt, G. 1992. Evidence-based medicine. A new approach to teaching the practice of medicine. Evidence-Based Medicine Working Group. Journal of American Medical Association 268 (17):2420-5.

Guyatt, G.H., R.B. Haynes, R.Z. Jaeschke, D.J. Cook, L. Green, C.D. Naylor, M. Wilson, and W.S. Richardson. 2000. User's guide to the medical literature: XXV. Evidence-based medicine: Principles for applying the user's guides to patient care. Journal of American Medical Association 284 (10):1290-1296.

Hafferty, F. 1998. Beyond curriculum reform: confronting medicine's hidden curriculum. Academic Medicine 73 (4):403-7.

Halpern, J. 1996. The Measurement of Quality of Care in the Veterans Health Administration. Medical Care 34 (3):55-68.

Halpern, R., M.Y. Lee, P.R. Boulter, and R.R. Phillips. 2001. A synthesis of nine major reports on physicians' competencies for the emerging practice environment. Academic Medicine 76 (6):606-15.

Harden, R.M. 2002. Developments in outcome-based education. Medical Teacher 24 (2): 117-20.

Harmening, D.M. 1999. "Pioneering Allied Health Clinical Education Reform. A National Consensus Conference."

Health Resources and Services Administration. 1999. Building the Future of Allied Health: Report of the Implementation Task Force of the National Commission on Allied Health. Rockville, MD: Health Resources and Services Administration.

Hovenga, E.J. 2000. Global health informatics education. Studies in Health Technology & Informatics 57:3-14.

Hundert, E.M., F. Hafferty, and D. Christakis. 1996. Characteristics of the informal curriculum and trainees' ethical choices. Academic Medicine 71 (6):624-42.

Hyde, R.S., and J.M. Vermillion. 1996. Driving quality through Hoshin planning. Joint Commission Journal on Quality Improvement 22 (1):27-35.

Ingersoll, G. 2000. Evidence-based nursing: What it is and what it isn't. Nursing Outlook 48:151-2.

Institute of Medicine. 2000. To Err Is Human: Building a Safer Health System. Linda T. Kohn, Janet M. Corrigan, and Molla S. Donaldson, eds. Washington, DC: National Academy Press. 2001. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press.

Institute of Medicine. 2002. Leadership By Example. Washington, DC: National Academies Press.

Jablonover, R.S., D.J. Blackman, E.B. Bass, G. Morrison, and A.H. Goroll. 2000. Evaluation of a national curriculum reform effort for the medicine core clerkship. Journal of General Internal Medicine 15 (7):484-91.

Jordan, S. 2000. Educational input and patient outcomes: Exploring the gap. Journal of Advanced Nursing 31 (2):461-71.

Lavin, M.A., I. Ruebling, R. Banks, L. Block, M. Counte, G. Furman, P. Miller, C. Reese, V. Viehmann, and J. Holt. 2001. Interdisciplinary health professional education: A historical review. Advances in Health Sciences Education 6 (1):25-47.

Leach, D.C. 2002. Competence is a habit. Journal of the American Medical Association 287 (2):243-4.

Lenburg, C., R. Redman, and P. Hinton. 1999. "Competency Assessment: Methods for Development and Implementation in Nursing Education."

Mansell, D., R.M. Poses, L. Kazis, and C.A. Duefield. 2000. Clinical factors that influence patients' desire for participation in decisions about illness. Archives of Medicine 160:2991-96.

Marwick, C. 2000. Will evidence-based practice help span gulf between medicine and law? Journal of American Medical Association 283 (21):2775-76.

Maudsley, G. 2001. What issues are raised by evaluating problem-based undergraduate medical curricula? Making healthy connections across the literature. [Review] [93 refs]. Journal of Evaluation in Clinical Practice 7 (3):311-24.

Mazur, D.J. and D.H. Hickam. 1997. Patients' preferences for risk disclosure and role in decision making for invasive medical procedures . Journal of General Internal Medicine 12:114-17.

Mazurek, B. 2002. Strategies for overcoming barriers in implementing evidence-based practice. Pediatric Nursing 28 (2):159-61.

Mitchell, G. 1999. Evidence-based practice: Critique and alternative view. Nursing Science Quarterly Vol. 12, No. 1:30-35.

Murray, E., L. Gruppen, P. Catton, R. Hays, and J.O. Woolliscroft. 2000. The accountability of clinical education: its definition and assessment. Medical Education 34 (10):871-79.

National Association of Boards of Pharmacy. 2002. "Examinations ­ NAPLEX."

National Committee for Quality Assurance. 2002. "What Does NCQA Review When It Accredits and HMO?"

National Council of State Boards of Nursing. 2001. "NCLEX - RN@ Examination: Test Plan for the National Council Licensure Examination for Registered Nurses."

National Council of State Boards of Nursing, I. 1997-2000. "Nursing Regulation: Examination Pass Rates & Licensure Statistics."

National League for Nursing Accrediting Commission. 2002. "National League for Nursing Accreditation Commission Website ."

O'Brien, T., N. Freemantle, A.D. Oxman, F. Wolf, D.A. Davis, and J. Herrin. 2001. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database System Review (2):CD003030.

O'Connor, G.T., S.K. Plume, E.M. Olmstead, J.R. Morton, C.T. Maloney, W.C. Nugent, F. Hernandez Jr, R. Clough, B.J. Leavitt, L.H. Coffin, C.A. Marrin, D. Wennberg, J.D. Birkmeyer, D.C. Charlesworth, D.J. Malenka, H.B. Quinton, and J.F. Kasper. 1996. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardio-vascular Disease Study Group. Journal of the American Medical Association 275 (11):841-6.

O'Neil, E. H. and the Pew Health Professions Commission. 1998. Recreating health professional practice for a new century­The fourth report of the pew health professions Commission. San Francisco, CA: Pew Health Professions Commission.

Partnership for Solutions. 2002. "Physician Concerns: Caring for People with Chronic Conditions."

Pew Health Professions Commission. 1995. Critical Challenges: Revitalizing the Health Professions for the Twenty-First Century. San Francisco, CA: UCSF Center for the Health Professions.

Phillips, R.L. Jr, D.C. Harper, M. Wakefield, L.A. Green, and G.E. Fryer Jr. 2002. Can nurse practitioners and physicians beat parochialism into plowshares? Health Affairs 21 (5):133-42.

Platt, D., and C. Laird. 1995. CQI: using the Hoshin planning system to design an orientation process. Radiology Management 17 (2):42-50.

Pomeroy, W.M., and I. Philp. 1994. Healthcare teams: An interdisciplinary workshop for undergraduates. Medical Teacher:6p.

President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry. 1998a. "Quality First: Better Health Care for All Americans."

Satya-Murti, S. 2000. Evidence-based clinical practice: concepts and approaches. Journal of the American Medical Association 282 (17):2306-7.

Schuster, M.A., E.A. McGlynn, and R.H. Brook . 1998. How good is the quality of health care in the United States?. Milbank Quarterly 76 (4):517-63, 509.

Shell, R. 2001. Perceived barriers to teaching for critical thinking by BSN nursing faculty. Nursing & Health Care Perspectives 22 (6):286-91.

Superio-Cabuslay, E., M.M. Ward, and K.R. Lorig. 1996. Patient education interventions in osteoarthritis and rheumatoid arthritis: A meta-analytic comparison with nonsteroidal antinflammatory drug treatment. Arthritis Care Research 9 (4):292-301.

Swankin, D. 30 May 2002a . Continuing Competence. Personal communication to Elisa Knebel.

Swankin, D.S. 2002b. Results of Survey of Selected State Health Licensing Boards and Health Voluntary Certification Agencies Concerning their Continuing Competence Programs and Requirements. Washington, DC: Citizen Advocacy Center.

Tanenbaum, S.J. 1994. Knowing and acting in medical practice: the epistemological politics of outcomes research. J Health Polit Policy Law 19 (1):27-44.

U.S. News and World Report. "Latest Hospital Rankings."

United States Medical Licensing Exam. 2002. "United States Medical Licesning Examination - Steps 1, 2, 3."

van der Vleuten, C.M., D.M. Dolmans, and A.A. Scherpbier. 2000. The need for evidence in education. Medical Teacher 22 (3):246-50.

Von Korff, M., J.E. Moore, K.R. Lorig, et al. 1998. A randomized trial of a lay person-led self-management group intervention for back pain patients in primary care. Spine 23 (23):2608-51.

Wagner, E.H., R.E. Glasgow, C. Davis, A.E. Bonomi, L. Provost, D. McCulloch, P. Carver, and C. Sixta. 2001. Quality improvement in chronic illness care: a collaborative approach. Joint Commission Journal on Quality Improvement 27 (2):63-80.

Wass, V., C. Van der Vleuten, J. Shatzer, and R. Jones. 2001. Assessment of clinical competence. Lancet 357 (9260):945-9.

Weed, L.L. and L. Weed. 1999. Opening the black box of clinical judgment. Part II: consumer protection and the patient's Role. British Medical Journal. November 13

Wennberg, J.H. 1998. The Dartmouth Atlas of Health Care 1998. Hanover, NH: Center for the Evaluation Clinical Sciences, Dartmouth University.

Woolf, S.H. 2000. Taking critical appraisal to extremes: The need for balance in the evaluation of evidence. Journal of Family Practice 49 (12):1081-85.

Wu, S., and A. Green. 2000. Projection of Chronic Illness Prevalence and Cost Inflation. California: RAND Health.

Yoder-Wise, P.S. 2002. State and association/certifying boards: CE requirements. Journal of Continuing Education in Nursing 33 (1):3-11.