ACTPCMD October 2004 Meeting Minutes

Advisory Committee on Training in Primary Care Medicine and Dentistry

MINUTES OF MEETING, OCTOBER 21-22, 2004

Approved on February 10, 2005

Advisory Committee Members Present
Gregory Strayhorn, MD, PhD, Chair
David P. Asprey, PhD, PA-C, Vice Chair
Man Wai Ng, DDS, MPH, Vice Chair
Margaret I. Aguwa, DO, MPH, Member
Rodolfo R. Burquez, DDS, Member
Tina L. Cheng, MD, MPH, Member
Alan K. David, MD, Member
Michael W. Donohoo, DDS, Member
Sanford J. Fenton, DDS, MDS, Member
Charles H. Griffith III, MD, MSPH, Member
Michelle Hauser, PA-C, Member
Bonnie Head, MD, Member
Warren A. Heffron, MD, Member
Christopher M. Howard, MD, Member
Matilde M. Irigoyen, MD, Member
Rubens P. Pamies, MD, Member
Eugene C. Rich, MD, Member
Joseph L. Price, PhD, Member
Raymond J. Tseng, Member
Craig D. Whiting, DO, FACFP, Member

Others Present
Kerry Paige Nesseler, RN, MS, Associate Administrator for Health Professions
Tanya Pagan Raggio, MD, MPH, Director of Division of Medicine and Dentistry and Executive Secretary of the ACTPCMD
Jerilyn K. Glass, MD, PhD, Deputy Executive Secretary of the ACTPCMD
O’Neal A. Walker, PhD, Chief, Dental and Special Projects Branch, Division of Medicine and Dentistry

Thursday, October 21, 2004

The Advisory Committee on Training in Primary Care Medicine and Dentistry (Advisory Committee) convened at 8:38 a.m. in Salons A, B, and C of The Hilton in Gaithersburg, 620 Perry Parkway, Gaithersburg, Maryland 20877. Gregory Strayhorn, MD, PhD, Chair, opened the meeting and asked Committee members to introduce themselves.

Dr. Strayhorn introduced Kerry Paige Nesseler, RN, MS, Associate Administrator for the Bureau of Health Professions, who brought greetings from Dr. Duke who was unable to attend. She thanked the members for their time and expertise in service to the Advisory Committee. She reviewed the reorganization of HRSA. A new office is the Office of Performance Review (OPR), which is the organization which will do an on-site performance review of each grantee in HRSA’s ten regions at least once every five years. Captain Nesseler introduced new staff including Tanya Pagan Raggio, MD, MPH, director of the Division of Medicine and Dentistry, O’Neal A. Walker, PhD, chief of the Dental and Special Projects Branch, and P. Preston Reynolds, MD, PhD, chief of the Primary Care Medical Education Branch. She announced the theme of the all-grantee meeting scheduled for June 1-3, 2005 in Washington, D.C.--“Health Professions: The Lifeline to America’s Health.” The aim of the Bureau is to have people see that health professionals are the individuals who make the difference in the health of America, and our programs train and educate them to be able to provide the services.

Captain Nesseler stated that the Bureau has at least 40 different programs, each developing a logic model reflecting legislative intent, short-term goals, and long-term goals. Five major concepts predominate across programs--quality, distribution, diversity, infrastructure, and choice of primary care career. Every program in the Bureau that has a performance measure on diversity, for example, will use the same performance measure, thus allowing aggregation of data across programs. Captain Nesseler pointed out that grantees will not be asked to show direct impact on a National health outcome. Rather, National data sets will be used to link performance measures to National level health outcome measures. The Bureau plans to complete the development of performance and outcome measures, develop data collection tools, conduct a pilot with a group of grantees, and finally send the package for Office of Management and Budget (OMB) approval.

Dr. Strayhorn introduced Dr. Raggio who underscored the connection between community health centers and the health professionals who provide the services. She announced that the deadline for grant applications has been extended to December 6. She reviewed the number of grants and cooperative agreements funded in FY 2004, including additional resources for dentistry. Dr. Raggio informed the Advisory Committee about the Council on Graduate Medical Education’s two reports in process, one on physician workforce policy guidelines and the other on minorities and medicine. The Division also has graduate psychology and geropsychology education training grant programs. She concluded with a description of OPR’s role in evaluating Agency-funded grant programs. The Advisory Committee requested that a representative from OPR come to a future meeting to describe the site-visit evaluation process.

Dr. Strayhorn introduced the first of three speakers on the topic for this meeting: how to evaluate outcomes of Title VII, section 747 programs. Dr. Thomas Ricketts from the Cecil G. Sheps Center for Health Services Research at the University of North Carolina at Chapel Hill, acknowledged the need for funding accountability. He stated that while the Bureau’s programs are authorized under different sections of legislation, there is probably more commonality than difference. One over-arching theme is quality, a component shared by all programs. He also suggested that one might measure whether the programs reinforce each other.

With an emphasis on placing practitioners in underserved areas, Dr. Ricketts stated that changing the workforce can be difficult because the decision on practice location has many factors. He presented the concept of a policy space which provides guidance for what to evaluate depending on what can be changed and how long it takes to see an effect. For example, you cannot change society or market influences, but you can change selection criteria for students into professional schools and institutional policies. Curricula in medical and dental schools can have large effects, but changes may take a generations. Dr. Ricketts said the process of training practitioners requires many little transitions and one should ask what is known about these transitions He urged the Advisory Committee to sharpen and integrate the intent of Title VII, section 747 within the context of the Federal workforce family. He suggested an evidence-based review of the effectiveness of workforce programs, an examination of multiple evaluation models, and consideration of logic models.

Dr. Cynthia Olney, the next speaker from the University of Texas Health Science Center at San Antonio, stressed measuring education outcomes. One type is a system outcome which, in our case, is implementation of a primary care curriculum. Has a curriculum changed and is it better? She discussed the alignment of learning objectives, instructional activities, and evaluation activities--the three basic elements of an educational curriculum. She listed recent educational trends: problem-based learning, working with simulated patients prior to working with live patients, integration of preclinical and clinical training, ambulatory and community-based training, and use of technology for content delivery. Measures of system change may focus on evidence or self-report that shows clinical integration of curricula, enhanced content delivery, increased use of technology, and improved evaluation methods.

Another outcome type involves learner knowledge, skills, and attitudes. Dr. Olney suggested that the education of skills and attitudes related to professional competency are very hard to teach and hard to measure. She presented Miller’s Pyramid of Competence, cited in a 2001 article in The Lancet, which depicts how learners develop knowledge and skill. First they know; then they know how to do something; then they are able to show how to do it; and then they are able to show it in day-to-day practice. How we evaluate students as they work to the top of the pyramid will be critical. Objective type tests may be appropriate at the lowest level but more case-based testing, essays on working through clinical problems, standardized patient exams, and evaluation in the practice setting may be more appropriate at higher levels.

Dr. Strayhorn reviewed a study done by the Robert Graham Center for Policy Studies looking at the association between Title VII funding to departments of family medicine and choice of physician specialty and practice location. The study group was 180,000 physicians who graduated from U.S. medical schools between 1981 and 1993. The study was restricted to non-federal allopathic and osteopathic U.S. physicians in direct patient care in the year in 2000. The focus was on 1) the number of primary care physicians, and 2) the number of physicians practicing in rural and under served areas. The study examined where physicians trained and whether their school received any Title VII funding during their four years of training. For about 1/3 of physicians who were in medical school that year, the department of family medicine at the school where they trained received no Title VII funding, no department funding, no faculty development funding, and no predoctoral funding. For about 2/3 of physicians, their schools did receive some amount of funding. A statistically significant finding was that those attending schools receiving Title VII funding were more likely to become primary care physicians, especially family physicians, as compared to attendees at schools not receiving such funding. Those trained at schools receiving Title VII funding, were also significantly more likely to practice in a rural area or a health prevention shortage area.

Dr. Ronald Markert, the next speaker from Tulane University School of Medicine, defined an educational outcome as a measurable effect resulting from student learning or from a program. He said that what people produce is of greater interest than what they do because production is measurable. He made the point that planning has to be results-oriented if one is to get satisfactory results. Learning outcomes need to be made explicit and communicated to students, faculty, funders, institutions, and so forth. A focus should be on deadlines and deliverables, and decisions should be based on data. Dr. Markert described the difference between research, which is based on theory and has hypotheses, and evaluation, which is based on goals and determines whether objectives have been met. The focus of the Advisory Committee’s fifth report is program evaluation. He asked whether grants have learning methods that will assure achievement of educational outcomes.

Dr. Markert discussed Gagne’s conditions of learning, namely that learning needs to be experience-based, problem-centered, operate in a supportive environment, and have active participation with feedback. He presented the concept of reflective practice wherein clinicians provide explanation as they demonstrate care. Whether participants have changed as a result of our training/education is an important outcome. In terms of program evaluation, one should ask whether the program is unique or like another program. He stressed that program evaluation must assure internal validity, identify and control confounders, and minimize sources of bias. For a specific grant, one should ask whether the results are generalizable to a broader setting. Captain Nesseler said the Bureau is considering a scale that reflects the recommendations of the Institute of Medicine and others as to what needs to be in a medical curriculum; grantees would have to indicate their progress on the various criteria.

Janet Schiller, EdD from the Bureau defined a logic model as a simplified visual representation of how a program is expected to work to achieve intended results. Key questions are: What are you doing? Why? What resources do you have? How are you going to do it? We may be able to use grantee self-scored measures which show the degree to which we are doing all the things that we think are important. Then we can use the literature to help make the case that those things are linked to National health outcomes of interest. She clarified the difference between 1) performance measure, which is a snapshot at one point in time that allows for a quick understanding of one aspect of a program, and 2) more in-depth evaluation that, for example, determines where graduates working in underserved areas were trained.

Describing features of Title VII, section 747 grants, Dr. Reynolds discussed the needs assessment, baseline and post-intervention assessments, the use of pre and post knowledge assessments, and the use of pre and post clinical skills assessments. She described a recent review of a grantee by OPR site visitors from the field. The review reflected a different, and potentially useful perspective on evaluation. The Bureau plans to interact more with OPR reviewers prior to their site visits.

Ms. Erica Froyd, a legislative analyst for the Association of American Medical Colleges, gave an update on Title VII, section 747. She stated that Health Professions Program appropriations for FY2005 were not complete. A Senate committee passed a bill requesting an increase of funding for Title VII bringing it almost back to 2003 levels. It is likely that Congress will come back after the election and pass an omnibus appropriations bill. The hope, too, is that there will be movement on the re-authorization of these programs which expired in September 2002, not too atypical a situation. She passed out a Congressional Research Service document on Title VII re-authorization, dated August 2004. Dr. Donohoo urged the membership to review it carefully, especially the section on Title VII effectiveness.

Man Wai Ng, DDS, MPH and Eugene C. Rich, MD reviewed the Report Writing Group’s August 20, 2004 meeting. The group said that the report should make clear its purpose to educate Congress, OMB, and other constituents about Title VII programs. The report should acknowledge and respond to past and present criticism and should present a fair and balanced discussion of the issues. It should explain 1) what primary care is and the need it fulfills and 2) a new vision for primary care training. A conceptual framework for Title VII programs, with logical linkages to broader goals and an explanation of its contribution to the health professions pipeline would be helpful. A conceptual framework in the context of public policy would also be beneficial. The program needs to have a fair and balanced systematic evaluation within the context of other Bureau programs. The report should contain a history of Title VII and a history of how these programs have been evaluated. There needs to be consideration of alternative models for evaluation, including the current proposal within HRSA for site evaluations by OPR. It may be useful to take some recommendations from previous Advisory Committee reports and explore alternative evaluation approaches. The group suggested an outline for the report which included: 1) background/evolution, 2) competing goals/ideas, 3) synthesis of goals/ideas, 4) logic model, and 5) evaluation.

Dr. Strayhorn urged that the report be forward looking rather than a presentation of past successes. The report should reflect what we feel these programs are about and where they are likely to have influence. Dr. Pamies stressed how quickly the workforce, through these programs, can be re-educated or mobilized in times of emergency.

The next order of business was the annual election of officers. Prior to the election process, Dr. Strayhorn, Chair, asked that his name be withdrawn from consideration. The newly elected Chair was Dr. Rich. Dr. Ng and David P. Asprey, PhD, PA-C were re-elected Vice Chairs.

Dr. Rich, Chair, then asked the Advisory Committee to finalize several issues pertinent to the fourth report on the future of primary care, specifically, the first recommendation and the discipline-specific definitions.

During the period for public comment, Ms. Hope Wittenberg from the Organizations of Academic Family Medicine said that Congress receives numerous requests each year and really wants to hear the Advisory Committee’s recommendations.

The meeting adjourned at 4:23 p.m.

Friday, October 22, 2004

At 8:00 a.m. the Advisory Committee met in three workgroups to discuss the following areas for the fifth report: background, policy, training.

The Advisory Committee re-convened in plenary session at 10:15 a.m. Tina L. Cheng, MD, MPH gave the report for the “background workgroup.” The report should provide an overview of the legislative intent of Title VII, section 747 and should acknowledge this program as the only Federal program that addresses training of primary care professionals who are responsible to the public in their capacity as first contact health providers. The report should state a clear purpose for these programs, distinguish these programs from nursing programs, establish their position in the training pipeline, and point out they have led to educational innovation. The group pointed out some of the difficulties associated with evaluation: cost, time, tracking trainees, program burden and distraction, quality of measures, and diversity of programs. Dr. Cheng presented the group’s diagram of the training pipeline and its effect on the health of the Nation. Michael W. Donohoo, DDS suggested that the National Health Service Corps be incorporated into the diagram.

Matilde M. Irigoyen, MD added that Title VII programs play a catalyst role in promoting higher standards, creating champions who will then develop new programs. She urged a new definition of “underserved” which goes beyond geography and ethnicity and not favoring grantees who already have a grant award track record when making the awards..

Dr. Ng gave the report from the “policy workgroup.” The report should present a conceptual framework in the context of policy and examine how other Federal programs are evaluated, especially the National Health Service Corps and the Bureau of Primary Health Care which intersect with Title VII. Primary care needs to be defined not just for the underserved but for the entire public. Associations should be drawn to National goals as found in Healthy People 2010. The group feared that with a restrictive definition of “medically underserved area,” current data on service to the underserved represents serious undercounting. The report should include examples of success like the Title VII-funded dental program at the University of Tennessee, described by Sanford J. Fenton, DDS, MDS, that has been replicated within and beyond the state.

Raymond J. Tseng suggested an important outcome to measure is the number of applicants who view service in the Indian Health Service or the National Health Service Corps as a viable, realistic career option. Outcome measures should reflect what we take responsibility for. Is it and increase in the number of qualified minority clinicians? Is it increased cultural competence of clinicians we graduate? Dr. Ng said the report should state that we have limited funds and, therefore, are limited in what is under our control.

Dr. Asprey gave the report from the “training workgroup.” The report should discuss answers to questions like: What is the effect of placing a health care provider in an underserved area? What type of curriculum is effective in causing change? What can influence students career choice? How are institutions affected by these programs? The report should identify measures that are not too narrow that they miss capturing the heterogeneity of the program. It would be useful to look at training competencies developed by other groups and show how our programs help healthcare providers achieve, attain, and use competencies that we have collectively agreed are desirable. The report should acknowledge linkages our training programs have to workforce, quality of care, and patient outcomes. Evaluation is stronger when uniform measures are used to report programs collectively.

Dr. Irigoyen said the report should include the innovations made by Title VII, section 747 and how they have been disseminated. Dr. Ng urged the Bureau to consider a mechanism for collecting long-term data. Dr. Strayhorn said that besides standard methods to collect information, narratives documenting grantee successes could be put into a database. There should be a mechanism to disseminate exemplary curricula and best practices each year. Charles H. Griffith III, MD, MSPH said the report should stress the heterogeneity of programs within Title VII, section 747, list the numerous ways to evaluate outcomes, and provide examples.

Joseph L. Price, PhD raised the question as to whether evaluation of individual programs adds up to the total. He questioned the usefulness of Agency-developed measures when applied at the local level, which harkens back to a point made by Dr. Markert that validity requires that instruments be tested in the setting of interest, in this case, Title VII grantee sites.

Dr. Schiller commented that these programs are not just about distributing people–but finding them and training them first. She saw these programs advancing the teaching and practice of primary care and additionally, pushing other parts of medicine. We might propose a 5-10 point agenda for evaluation in primary care teaching and indicate the need for funds to do the evaluation.

Dr. Rich thanked staff and recognized that the Writing Group has work to do before the February meeting. There was no public comment.

The meeting was adjourned at 12:20 p.m.