Graduate Program Review

Procedures for Graduate Program Review

Division of Research and Graduate Studies

April 8, 2002

Rationale and Background

The general mission of East Carolina University as outlined in the Strategies for Distinction, University Directions 1995-2000 states that ECU “is a public institution committed to rich and distinctive undergraduate and graduate education, exemplary teaching, research and scholarship, public service, and human and intellectual diversity.”  Goal Two under Strategies for Distinction directs that ECU will offer outstanding and distinctive graduate programs and lists four objectives: A) to develop new doctoral programs that capitalize on institutional strengths, respond to societal needs, and address national shortages of faculty and other professionals in selected areas; B) to increase the number of candidates and the graduation rates in existing doctoral programs; C) to initiate and implement procedures for the periodic evaluation of graduate programs; and D) to improve our competitiveness for attracting and retaining top graduate students.

In meeting Goal 2c (listed above) it is noted that effective graduate program review at most institutions consists of a self-study conducted by each program, followed by a site visit by academicians from both within and outside the university.  The usual outcomes for such reviews are the identification of program strengths and weaknesses with a determination of overall program quality and specific recommendations for improvement. Peer-review, when properly done, is a very effective way to maintain and improve program quality if the university responds to the recommendations for improvement as suggested by the review teams.   In addition, the periodic review could be used as a basis for a more effective allocation of resources such as graduate student support funds (e.g. Graduate Assistantships) by using the program quality as indicated by the review and the undergraduate teaching needs as criteria for distribution.  In summary, it is anticipated that the outcomes will strengthen all of the graduate programs. 

Overview of Review Process

All graduate programs are subject to the review process, although it is recognized that some programs and/or academic units at East Carolina University hold accreditation reviews that include a review of the graduate program. In these instances, the accreditation self study will largely substitute for the required self-study of the Graduate Program Review, and whenever possible the Graduate School will add one external and one internal reviewer to the accreditation team to evaluate those items not covered in the accreditation review. Finally, the vice chancellor for research and graduate studies, after consultation with the appropriate dean and vice chancellor for academic affairs or health sciences, may determine that the findings of the accreditation review will suffice completely for the Graduate Program Review.

Each review will consist of four major components:

1. With the understanding that some units have more than one graduate program, each graduate program will prepare and submit to the Graduate School a self-study that is reviewed and approved by the appropriate chairperson and dean. 

2. The dean, chairperson, and graduate program director will work with the graduate program review coordinator (vice chancellor for research and graduate studies or his/her designee) to prepare a list of potential reviewers (both internal and external), to plan the site visit, and to select a date for the visit.

3. The site visit team will:  prepare a report describing the strengths and weaknesses of the program based upon the criteria noted below; make recommendations when appropriate for program improvement; and provide opinions on overall program strength. 

4. Following receipt of the site-visit report, the graduate program director, the chairperson, and the appropriate dean will meet with the vice chancellor for research and graduate studies, the appropriate vice chancellor for academic affairs or health sciences, and the chair of the Graduate Administrative Board to respond to this report.    A response plan will be developed that prioritizes the review's recommendations, and outlines steps for their implementation, including budgetary implications and responsibilities.  The response plan will then be approved and signed by the appropriate departmental chairperson, dean, and vice chancellor, following which the Plan will be forwarded to the vice chancellor for research and graduate studies, and the Graduate Administrative Board for final approval.

Each program will be reviewed on a predetermined cycle as outlined under “Timetable for Review Visits”. 

Participating Units

All reviews will be conducted under the auspices of the Division of Research and Graduate Studies and, as appropriate, the Division of Academic Affairs or the Division of Health Sciences in cooperation with the individual graduate programs under review and the dean of the College or School in which that program is offered. The review process will be jointly funded with the necessary EPA faculty, SPA staff, and operating costs by the deans, Division of Research and Graduate Studies, the Division of Academic Affairs, the Division of Health Sciences, and the Chancellor’s Office.  The roles of each participating unit are described below:

Division of Research and Graduate Studies

The vice chancellor for research and graduate studies or his/her designee will function as the coordinator for the review process and will be aided by the Graduate School staff.  As coordinator, the dean or his/her designee will be responsible for coordinating all aspects of the review process.  He or she will receive the self-study reports from the graduate program under review and distribute them to the vice chancellor for academic affairs or health sciences, the Graduate Administrative Board, and the vice chancellor for research and graduate studies.  In coordinating the individual program reviews, the coordinator will:  conduct the initial planning sessions with the chairperson, graduate program director, dean (and/or his/her designee), and the faculty of the program under review (if requested by the chairperson); ensure that the self-study is initiated in a timely manner; and help identify site visitors.  The coordinator, working with the dean, chairperson and graduate program director, is responsible for establishing each review team’s schedule.  After the visit, the coordinator will receive the report from the review team and coordinate the response plan meeting and the follow-up progress report. 

The College or School

The dean of the College or School (or designee) will participate in the initial planning of the external review, including the approval of possible internal and external reviewers. The dean will meet with the review team during its visit and participate in the exit interview. Following completion of the review, the dean will receive a copy of the review team’s report from the coordinator and meet with the departmental chairperson, the graduate program director, the appropriate vice chancellors, and the chair of the Graduate Administrative Board to develop a response plan for implementing the recommendations of the review team.  The dean and the appropriate vice chancellor for academic affairs or Health must approve the response plan prior to its implementation and provide the necessary resource requirements as outlined in the response plan.

Unit Offering the Graduate Program

The chairperson of the unit is responsible for completing the program self-study and forwarding copies of the report through the appropriate dean to the Graduate School. The chairperson will participate in planning the external review by:  recommending, through the dean, a list of suggested internal and external reviewers; establishing the review dates and coordinating the dates with accreditation reviews if possible; and arranging the schedule of the site-visit.  The chairperson will meet with the review team during the visit, and after the review report is received, the chairperson and the graduate program director will meet with the appropriate dean, vice chancellors, and the chair of the Graduate Administrative Board to develop a response plan.

In interdisciplinary graduate programs, the program director should take responsibility for the actions listed above but should coordinate these actions with the appropriate chairs and deans.

The Review Team

The review team will include two internal reviewers and two external reviewers nominated by the program under review. The actual membership of the review team will be decided jointly by the vice chancellor for research and graduate studies, the participating dean, and the vice chancellor for academic affairs or health sciences and may include members that were not nominated by the program if deemed necessary. The internal reviewers will be selected from disciplines closely related to the program under review, but not from the same department/school. If no closely related discipline exists, then someone with extensive experience in graduate education will be chosen.  External reviewers will be selected from lists provided by the chairperson of the graduate program being reviewed and the coordinator. The recommended external reviewers of both masters and doctoral programs should be from similar or more advanced programs in the same field of study.   In addition, the reviewers of doctoral programs must come from other doctoral-granting programs. 

The review team should review the self-study document before the site visit and identify the need for any additional information.  All team members should participate in the two full-day site visit. On the afternoon of the second full day, the review team will be given approximately three hours to prepare a detailed outline of its report.  At the end of the second day, the team will discuss its major findings and recommendations with: 1. The chairperson and the graduate program director at a “wrap-up” session, 2. and the  appropriate dean, vice chancellor,  the vice chancellor for research and graduate studies, the chair of the Graduate Administrative Board, and the review coordinator at an exit interview.  (Designees may be appointed as substitutes for the administrative officers if necessary although such substitution should be avoided if possible.) The team will  leave a draft copy of its detailed outline with the dean and vice chancellors and agree upon a date when the full final report will be submitted to the coordinator.

External reviewers will be reimbursed for all related expenses and provided with an honorarium for their participation in the review process.  This honorarium will be provided by the chancellor or, as appropriate, the vice chancellor for academic affairs or the vice chancellor for health sciences.

Graduate Administrative Board

The Graduate Administrative Board will receive a copy of each self-study, the review team’s report, and the approved response plan. After discussion, the Graduate Administrative Board will make a recommendation to the vice chancellor for research and graduate studies regarding the final approval or disapproval of the response plan. Any disapproval of the response plan should be forwarded to the vice chancellor for research and graduate studies along with suggested changes.  The changes will then be discussed with the appropriate chair, dean, and vice chancellor for revision. The revisions will then be submitted to the Graduate Administrative Board for final approval. 

Central Administration

The Division of Research and Graduate Studies, in conjunction with the Division of Academic Affairs or Health Sciences and the Chancellor, will provide staffing and funds for expenses and honoraria of the review team.  In addition, they will work with the graduate program to develop a response plan that addresses the recommendations made by the review team and establish a timeline for the implementation of the response plan.  The response plan is subject to the final approval of the Graduate Administrative Board.     

Criteria for Review

Self-Study

The purpose of the self-study is to produce a report containing sufficient information to allow the review  team to  make a preliminary evaluation of program quality prior to the site- visit. The self-study will be prepared by the program’s academic unit and will be distributed by the chairperson to the appropriate dean and  the review coordinator at least four weeks prior to the site-visit.  The coordinator, in turn, will distribute the self-study to the review team, vice chancellor for academic affairs or health sciences, and the vice chancellor for research and graduate studies. The following outline is designed to provide guidelines to assist the unit in conducting its self-study. While not limited to the following, each self-study must include the following information:

1. General Program Characteristics:  A description of the degree program, including the educational objectives and curriculum, should be provided.  There should be a discussion of how this degree program enhances departmental, School or College, and institutional objectives, including those aspects of the program, which make it unique, and those, which are interdisciplinary.  NOTE:  The review team should compare program characteristics and, where appropriate, contrast them to those of similar or more renowned programs at other Doctoral / Research Intensive (Carnegie Classification) universities and/or to the standards of an appropriate accrediting agency for the program.  These same standards should be applied to all the following categories: facilities, faculty, applicant pool, student performance, and any other measures of quality.

2. Facilities:  Description of the facilities used for the graduate program as well as the areas used by faculty for their research and scholarly activity should be provided.  Discuss the adequacy of the support facilities including libraries, computer networks, graduate student support, support for pursuing grants and contracts, etc.  Include a brief description of facility utilization; how many graduate courses are offered each term; how often are key courses offered; what is the average class size?  NOTE:  Program facilities should be compared and, where appropriate, contrasted to those of similar or more renowned programs at other Doctoral / Research Intensive (Carnegie Classification)  universities and/or to the standards of an appropriate accrediting agency for the program.

3. Faculty:   provide a current vitae on all graduate faculty in the program and the following five-year summarized data in a single table: a)  total grant, contract, or fellowship extramural support (to be obtained from unit annual reports or from the Office of Sponsored Programs)
   • peer-reviewed or juried publications, exhibits and shows (to be obtained from unit annual reports or from the Office of Sponsored Programs)
   • total invited presentations, exhibits and shows total number of theses and dissertations supervised
   • on editorial boards
   • leadership roles in local, state, regional, national, and international professional organizations; service on peer review boards, selection committees, panels, and/or judge on state, national, or international levels.

4. Quality of student applicant pool:  Provide a description of the criteria used for admission into the program and include a table listing the following student information for the past five years.  NOTE: Some programs may choose to use summaries of more than five years and a few programs may be new and lack this historical data.   Include any information considered appropriate, but at least the following, if available:
   • number of program applicants
   • number of students admitted, and the percent matriculating
   • an assessment of the quality of matriculants as indicated by grade-point average, standardized test scores or other appropriate measures (to be provided by the Graduate School Office)

5. Student performance measures: Include a table that summarizes the following student information for the past five years (or less if necessary):
   • student presentations and publications
   • awards received by students for teaching, research or performance
   • when appropriate, percentage of graduates who attain licensure or certification
   • placement of program graduates, describing how many are appropriately employed in their professional  field of study ( i.e. using education), gone on to further education, are employed in non-profession related jobs or  unemployed.  Doctoral programs are required to provide a  list of graduates and their current professional status.  Masters programs should provide a list of graduates and their current professional status if possible
   • time to degree, percent retention.

6. Other measures of quality:  Please present a description of the following activities.
   • program awards such as national rankings, extramural (state, national or foundation) programmatic support, etc.
   • attraction of postdoctoral students, interns, apprentices, etc.
   • visiting and adjunct professors/scholars

Review Team Responsibilities

Based on the self-study report and the on-campus visit, the review team should address the specific areas enumerated below in making its evaluations of program strengths, weakness, and overall quality, and in making its recommendations for improvement.  A document entitledCriteria for Graduate Program Review  that outlines specific points to be evaluated will be provided to the review team as a template for the report (See Appendix. Template for Reporting Criteria for Graduate Program Review  at East Carolina University)

NOTE:  Each enumerated item should be compared and, where appropriate, contrasted to national standards or norms as reflected in similar or more renowned programs at other Doctoral /Research Intensive (Carnegie Classification) universities and/or in accrediting agency guidelines.

In general, the following items will be assessed:

1. Program quality:
   • program goals, objectives, and rationale
   • status of accreditation (if appropriate)
   • quality of curriculum and instruction
   • quality of graduate faculty
   • quality of graduate students
   • admission standards and procedures
   • appropriateness of training as measured by regional/state/national need for graduates and placement of program graduates
   • the plan established for assessing learning outcomes and the use of assessment results to improve the program

2. Program characteristics:
   • unique characteristics of program
   • interdisciplinary characteristics
   • program productivity (graduation numbers, attrition, enrollment trends)
   • student productivity (publications and presentations at local, state, national, or international meetings, shows, exhibits)

3. Program support:
   • number, use, and remuneration of graduate teaching assistants
   • number, use, and remuneration of graduate assistants and graduate research assistants
   • extramural support--individual and training grants
   • number of graduate faculty involved

4. Facilities:
   • laboratory and studio facilities (where applicable)
   • equipment (including instruments)
   • library resources
   • computer services
   • office and classroom space
   • overall intellectual environment

Timetable for the Graduate Program Review

The following describes a timetable for the review visits,  the selection of the reviewers, the schedule for the visit, and the generation of the final report.

1. Review cycle: Each program should be reviewed on a predetermined cycle of once every six years. If possible, the date of review will coincide with the date of any accreditation review required for the graduate program.  The details of this cycle (month and year of site visit) will be predetermined following consultation with the deans, and vice chancellors for academic affairs and health sciences.  The deans, chairs or vice chancellors may call special reviews.

2. Initial planning meeting: The coordinator will meet with the chair, the graduate program director, the dean (or his/her designee), and the faculty of the program (if requested by the chair) to discuss the purpose and procedures of the review and to respond to their questions and concerns.  This should be done in the semester preceding the semester of the site-visit.<

3. Date of the site-visit and list of prospective reviewers: The specific date for the site-visit will be established by agreement between the coordinator, the dean, the chair and the graduate program director. The dates should be determined at least four months prior to the site-visit and every attempt should be made to find times agreeable to the appropriate deans and vice chancellors. At the same time a list of prospective reviewers will be submitted from the chair   to the coordinator who will then present the list to the appropriate dean and Vice-chancellors for prioritization. Invitations will be issued to the chosen reviewers.

4. Submission of the self-study: The graduate program submits a self-study document to the dean and the coordinator at least four weeks prior to the scheduled site-visit.  The coordinator distributes the self-study to the vice chancellor for research and graduate studies, the vice chancellor for academic affairs or health sciences, and the review team.

5. The site-visit agenda and schedule: A draft schedule for the review will be developed approximately four weeks prior to the site-visit by the coordinator, the appropriate chairperson and the program director following the general outline below:
   • Day 1: Review team meets over dinner with the vice chancellor for research and graduate studies, the vice chancellor for academic affairs or health sciences, the dean, the chair, the graduate program director, the chair of the Graduate Administrative Board, and the coordinator.  This is an introductory orientation meeting.
   • Day 2:  Review team meets with the chair, the graduate program director, each graduate faculty member, and selected advanced graduate students.  A tour of the facilities is usually held in the morning.
   • Day 3:
     • Morning: Review team continues meeting  with the faculty and/or graduate program support personnel, central administration and anyone not available on Day 2. In many cases the team will have breakfast with the graduate students.
     • Afternoon: Review team holds  a “wrap-up” session with the chair and the graduate program director.  In addition, the review team prepares a rough draft of the review report and holds an “exit” interview with the appropriate vice chancellors, dean, chair of the Graduate Administrative Board and the review coordinator.
   • NOTE: The coordinator handles the lodging arrangements, and payment and reimbursement of the external consultants.  The academic unit under review provides local transportation.

6. Submission of final report by the review team: The review team will submit a final report of its review  within one month of the site visit, and the coordinator will distribute copies of this report to the dean, chairperson, graduate program director, the chair of the Graduate Administrative Board, and the appropriate vice chancellors.

7. Development of response plan: Within one month after receipt of the final report, the coordinator will schedule a meeting of the chair, the graduate program director, the appropriate dean, the vice chancellors, and the chair of the Graduate Administrative Board to develop a response plan.  The response plan will outline those recommendations made by the review team that will be acted upon and a timetable for the implementation of the response plan.  The chairperson, the appropriate dean, and vice chancellor must approve the response plan. 

8. Approval of response plan by the Graduate Administrative Board:   Following approval of the response plan as described above in point 7,  copies of the self-study, the review team report, and the response plan will be sent to the Graduate Administrative Board for its review and recommendation for final approval. The recommendation will then be submitted to the vice chancellor for research and graduate studies.

9. Progress report and follow-up procedures: The effectiveness of graduate program reviews is dependent on follow-up procedures that are timely and meaningful. Thus, three years after completion of the review, the coordinator will ask the chairperson to prepare a document outlining progress toward implementing the agreed upon recommendations. The appropriate vice chancellors, vice chancellor for research and graduate studies and his/her designee, dean, chair and graduate program director will then meet to review this progress report.