ACADEMIC PEDIATRIC ASSOCIATION

By: L Chandran, CD Baldwin, and ME Gusic
Download file in MS Word

RATINGS
The Analysis tool includes 18 qualitative and 18 quantitative items.

• For quantitative items (e.g. item 6-8), put scores in the third column.
• For qualitative items (e.g. items 1-3), put scores in the fourth column, using the scale shown at the top of that column. Add comments to explain each qualitative score.

Qualitative Ratings are: 1= Novice,   2= Intermediate,  3= Expert
These ratings are based on a developmental perspective. For a new faculty member, some items might appropriately be in the novice, whereas for a full professor, more items would be in the expert category. Also note that individual expertise varies, so ratings of faculty members would be expected to differ between SECTIONS of the portfolio. In fact, some faculty, especially those early in their careers, may have no data to enter in some sections of their EP.

SECTIONS
The sections in the analysis tool follow the components of the Educator Portfolio template, available
at http://www.academicpeds.org/education/education_faculty_dev_form.cfm

INTRODUCTORY SECTIONS
Educational Philosophy
Five-Year Goals

FIVE DOMAINS OF EDUCATIONAL ACTIVITY
I.    Teaching
II.   Learner Assessment
III.  Curriculum Development
IV.  Mentoring/Advising
V.   Educational Leadership/Administration

EVIDENCE OF SCHOLARLY ACTIVITY (rated across all domains)
VI.  Scholarly Approach To Education
VII. Products Of Educational Scholarship

If the EP includes no information in a particular section, check the box: Check if no entries in this section q. Within sections, enter NA (not available) if the faculty member fails to include data needed to evaluate an item. Throughout the EP, insufficient data earns the same score as no entered data.

Rating of Evidence of Scholarly Activity requires an overall assessment of the faculty member’s activities across all five domains. The EP does not provide a separate section for documenting scholarly activity. 

COLUMNS
The first column in the analysis table shows the specific items that are to be evaluated under each EP section. The second column gives a descriptive specification of what should be evaluated in selecting a score. For quantitative items, the method for calculating a numerical score is described.  Qualitative items are all rated on a scale of 1=Novice, 2=Intermediate, 3=Expert, and the specifications define an INTERMEDIATE rating (2). Quantitative scores are placed in the third column. Qualitative scores go in the fourth column, where addition of explanatory comments is required.

INSTRUCTIONS FOR RATING EACH SECTION

EDUCATIONAL PHILOSOPHY

We recommend rating this section after reviewing all sections of the EP.

Item 1: Self-reflection, self-appraisal (QUAL): Does the written philosophy describe an approach to education and the principles that underlie the faculty member’s teaching in a way that reflects his/her ability to learn from previous teaching experiences?

Item 2: Philosophy both rooted in theory or principle, and applied to experience (QUAL):  To what extent has the statement integrated sound educational theory with the faculty’s own experiential learning as a teacher? This theory may be taken from the literature or developed individually. The statement should spring from careful reflection on the faculty member’s educational strengths and practice, and include examples from his/her teaching experience.

Item 3: Evidence of philosophy applied throughout EP (QUAL):  Is the philosophy presented reflected in the activities described in the rest of the EP? 

 

FIVE YEAR GOALS

The evaluator is looking for a balance between a sound educational vision (item 4) and specific and realistic goals (item 5) to achieve that vision.

Item 4: Goals set bar appropriately high (QUAL):  Are the goals sufficiently ambitious to stretch the faculty member and help him/her grow and develop?

Item 5: Focused and realistic plan (QUAL):  Is there evidence that the five-year plan has been carefully thought out, with interim steps articulated? Are learning strategies described that will help the faculty member realistically meet the goals set forth?

I. TEACHING

Note: To score items 6, 7 & 8, assume that the faculty member works 8 hr/day, 48 wk/yr or 11 mo/yr, unless he/she states otherwise.

Item 6: Total Learner Number/yr (QUANT):  For each activity, the number of learners per year that participated in that teaching activity is given in the EP template. If the activity is given multiple times and the learner numbers vary, an average number of learners for that activity per year would be indicated. The total learner number score is the sum of all learner numbers per year for ALL activities.

Item 7: Total Teaching Hours/yr (QUANT):   The EP asks the faculty member to state the number of teaching hours per activity per year. Enter for Item 7 a sum of the total teaching hours of all activities for the year.

Item 8: Total Teaching Impact Score (QUANT):  This score quantifies the geographic impact of the faculty member’s teaching activities in a year. The geographic impact weight is  Department=1, Institutional =3, Regional= 5, National or international= 10. For each activity that is listed in the Teaching Activities Grid, multiply the geographic impact weight to obtain a teaching impact index. The sum of all the teaching impact indices will be the Total Teaching Impact Index. Examples are shown with the Worksheet at end of Instructions.

Item 9: Variety of teaching strategies (QUAL):  This item evaluates the variety of strategies employed by the teacher that go beyond the traditional lecture (which typically uses a passive transfer of knowledge). In the Teaching Activities Grid, look for evidence of teaching that is interactive and/or engages trainees in participatory or experiential learning. A rating of Intermediate (2) would be given to a faculty member who uses at least 3 teaching methods to create active or interactive learning, such as interactive techniques used during didactic presentations, case discussions, workshops, problem based learning sessions, evidence-based medicine activities, journal clubs, etc. 

Item 10: Teaching evaluation score (QUANT): This item is a quantitative comparison of the educator’s teaching evaluations with those of the local peers, with potential scores ranging from 0 to 5. Most institutions use such metrics to provide feedback to individual faculty members on how the learners evaluate their teaching. If the teaching of the faculty member is on par with the peer average, then a score of 1 would be assigned. A superior rating compared to the peer ratings would receive a score of 5.

Item 11: Multiple sources and types of teaching evaluations (QUAL): To receive a rating of Intermediate (2), the faculty member should use 2 or more types of evaluations of his/her teaching (e.g., live feedback, surveys of learner satisfaction, peer rating based on a checklist of validated items). In addition, this evaluation data should come from 2 or more different sources (learners, colleagues, peers, supervisors, education experts).

Item 12: Teaching Award Score (QUANT):  Teaching awards are scored as a quantitative measure that is weighted for geographic impact. This score is the sum of the weighted scores. E.g., someone with 3 departmental awards (3 pts), and 1 national award (10 pts) would receive a score of 13.

II. ASSESSMENT OF LEARNERS

Item 13. Total Learner Assessment Score (QUANT):  This item takes into account TWO attributes of the faculty member’s assessment activities: the number of evaluation activities and the role he/she played in them.
a.  Number of evaluation activities: This number counts the activities listed in the Learner Assessment Grid.
b.  Role in evaluation: Each evaluation role is assigned a different weight: development of a new assessment tool or method gets a weight of 3, analysis or synthesis of evaluation data gets a weight of 2, and implementation of a previously established assessment strategy is assigned a weight of 1.  These roles are indicated in the Learner Assessment Grid, Column 4. (If the faculty member plays more than one role in an eval activity, sum the role scores. E.g., if someone develops an OSCE, implements it, and synthesizes data from it, he/she would earn 6 points (1 x 3 x 2 x 1).
Each individual assessment activity is given a score that reflects the weight of the role. The Total Evaluation Score is the sum of these scores for all evaluation activities listed in the grid.

Item 14: Learner assessment strategies (QUAL):  This item qualitatively rates the suitability and variety of strategies that the faculty member uses in his/her assessment of learners. Assessments should be appropriate for the teaching context (e.g., multiple choice exams for knowledge evaluation, and direct observation for most procedural skills, 360° evaluations for attitude and interpersonal skills assessments). The most authentic evaluation methods are competency-based, and measure the learner outcomes desired, e.g., through direct observation, chart reviews, or practice profiles. Application of these more resource intensive evaluation methods is not feasible in all settings, however.  

Item 15: Balance of methods that include upper level “Miller’s Triangle” strategies (QUAL):  This is an assessment of the strength of the assessment methods using the Miller’s triangle hierarchy: first level = “knows,” next “knows how,” next “shows” and highest= “does.” A score of Intermediate (2) would be given if the faculty member uses the highest possible level of assessment in AT LEAST ONE teaching context.

III: CURRICULUM DEVELOPMENT

Items 17 and 18 can be rated using data from the Curriculum Description grid. Items 19-22 can be rated using data from the GNOME Analysis of Curriculum grid.

Item 16: Curricular Impact Index (QUANT):  This item is a quantitative measure of the impact of all curricula developed by the educator. The rating takes into account the number of learners taught using the curriculum, whether the curriculum is implemented, and the geographic impact of the curriculum. For each curriculum that is described in the EP, the evaluator determines an individual curricular impact score, calculated as learner number score x implementation score x geographic impact score.

• Learner numbers are grouped and weighted as shown on the analysis form: e.g., a curriculum used by 100 learners receives a learner score of 2.
• Implementation is scored simply as: curriculum not yet implemented=1 and curriculum implemented= 2.
• Geographic impact scores are weighted as indicated: e.g., a curriculum that has a national impact will receive 10 times the score of one with departmental impact only.

Each individual curriculum receives its own impact score (learner score x implementation score x geographic impact). The Curriculum Impact Index is the sum of the impact scores for all individual curricula. E.g., suppose that an educator has developed 2 curricula: 1) for 25 learners, already implemented at the regional level = impact score of 10; 2) for 200 learners, not implemented, for institutional use only = impact score of 6. The Curriculum Impact Index will therefore be 16.

Item 17: Curriculum Role Score (QUANT): This item is a quantitative score for the educator’s degree of responsibility for developing a curriculum (contributor = 1, leader = 5).  For each curriculum described in this section of the EP, a degree of responsibility score is assigned. The Curriculum Role Index is the total of these weighted scores for all curricula. E.g., 1 curriculum that one leads plus 3 to which one contributes = score of 8. 

NOTE: For items 18-21, evaluate one curriculum. If more than one is described, evaluate the one designated by the author as the best example. If the author fails to do so, choose the best in your own judgment to rate for these items.

Item 18: Quality of Goals/Objectives (QUAL):  A rating of Intermediate (2) would be appropriate for a curriculum that is based on a set of goals commensurate with the scope of the curriculum (e.g. duration and importance), and a reasonable set of objectives that can be evaluated in the time frame of the learning experience.

Item 19: Quality of Needs Assessment (QUAL):  A rating of Intermediate (2) would be appropriate for a curriculum that is created in response to data collected from prospective learners or a group of recent past learners, and where new data on learner needs is collected over time, so the curriculum can be refined.

Item 20: Quality of Methods (QUAL):  A rating of Intermediate (2) would be appropriate for a curriculum that develops methods that address the goals of the curriculum, are well suited to the level of learners, account for a variety of learning styles, and make good use of the learning setting.

Item 21: Quality of Evaluation (QUAL):  A rating of Intermediate (2) would be appropriate for a curriculum that focuses the evaluation of learners on planned learning objectives, and uses data from learner evaluations for curricular improvement. It should also gather input from both learners and faculty to facilitate continuous program refinement.

IV: MENTORING/ADVISING 

The Mentoring and Advising Grid in the EP template requests data for all items in this section, but younger faculty may have little or no data to provide. If the faculty member’s mentees have only recently completed their training, items 24 and 25 should be marked NA (not available). 

Item 22: Mentee Number (QUANT):  To rate this item, we do not attempt to distinguish between mentees and advisees, because there is no clear consensus on definitions of these categories.

Item 23:  Mentee Productivity Score (QUANT):  This item evaluates evidence provided on the EP about mentees’ publications and grants, and assigns weights from Minimal=1 to Outstanding=7. Score this item as NA if no data is provided about mentee outcomes. An outstanding score (7)  would be appropriate for a faculty member whose mentees have numerous educational publications in high impact journals, have won prestigious education awards, and/or have obtained large educational grants, especially for national programs.

Item 24: Mentee Professional Advancement Score (QUANT):  This item evaluates evidence provided on the EP about mentees’ promotions, leadership roles, and teaching awards, and assigns weights from Minimal=1 to Outstanding=7. Score this item as NA if no data is provided about mentee outcomes. An outstanding score (7) would be appropriate for a faculty member whose mentees have advanced to high rank at their home institution, lead demonstrably successful educational programs, and/or become well respected national educational leaders.

Item 25: Quality of Mentoring (QUAL):  This item is a combined qualitative assessment of a) the educator’s mentoring philosophy, as indicated by the careful reflection on experience as a mentor, and b) the impact of the faculty member on his/her mentee/advisees, as documented in mentee letters.
                                                       
V: EDUCATIONAL LEADERSHIP/ADMINISTRATION 

Item 26: Program Leadership Index (QUANT):  This item is a quantitative indicator of the faculty member’s leadership roles in educational programs. The geographic impact of the program and the length of time served in the leadership role are both weighted, and the weights are summed to yield the Program Leadership Index.  E.g., if someone has led 3 departmental programs for 5 yr (6 pts) and 1 national program for 1 yr (4 pts), the Program Leadership Index would be 10.

Item 27: Committee Leadership Index (QUANT):  For this item, the educator receives a score for each educational committee led that is weighted for geographic impact and the length of time served in the leadership role. E.g., if someone leads 1 departmental committee for 5 yr, 1 institutional committee for 3 yr, and a regional group for 2 yr, the score would be 12.

Item 28: Total Committee Membership Score (QUANT): Committee membership is rated by number of committees x length of service of each committee. Ignore committees led by the educator, as these are credited in Item 28.

Item 29: Quality of Leadership (QUAL): For this item, credit is given for leadership roles when there is external evidence of quality, such as testimonials provided in letters from program stakeholders or the written results of positive program reviews by accreditation agencies. If these data are lacking, record NA (not available).

Item 30: Reviewing or Moderating Index (QUANT):  Reviewing activities at the national level only are given weighted scores depending of the type of review, not the number of journals for which one reviews  (i.e., do not double the score for someone who reviews for 2 journals). No points are earned by local or regional reviews. Moderating a session at a national meeting earns 2 pts/moderating activity. The Total Reviewing/Moderating Score is a total of all the weighted scores for reviewing, plus total of all the weighted scores for moderating.  E.g., someone who reviews articles for nationally distributed journals (5 pts) and reviews abstracts for the PAS meeting (2 pts), and moderates sessions at the PAS meeting for 3 years (6pts) would receive a total score of 13.

EVIDENCE OF EDUCATIONAL SCHOLARSHIP

This section differs from those that precede because it requires review of all parts of the Educator Portfolio taken together, not just a single section. Evidence of Educational Scholarship is divided into two parts.

• Section A, Scholarly Approach to Education, is mostly qualitative.
• Section B, Products of Educational Scholarship is entirely quantitative, and yields two index scores, a Scholarly Productivity Index and an Educational Grants Index.

VI. Scholarly Approach to Education

To complete this section most efficiently, the evaluator may want to take notes or mark EP passages prospectively, while reviewing other parts of the EP. Most relevant data will be found in Educ Philosophy, Five-Year Goals, and Narrative responses in Sections 1, 2, 3, 4. Give special consideration to the educator’s special educational projects and efforts, e.g., a new curriculum developed, an evaluation tool created and tested, a teaching program launched, or a conference series reorganized and improved.  At the instruction “List focal areas of scholarship considered for this section,” write down the focal areas of effort which you have identified in the EP and have chosen to evaluate qualitatively here.

Item 31: Application of an accepted model or a structured approach (QUAL):  This general item earns a score of Intermediate (2) if the faculty member approaches educational tasks systematically or in a structured fashion, potentially using an accepted model from the literature or developing his/her own model.

Item 32: Response to evaluations  (QUAL): This item is a qualitative rating of the evidence (mainly from narrative responses in the EP) that the faculty member has reflected upon evaluations of his/her teaching, as well as outcomes of learner assessments,  and used this information to improve his/her teaching.

Item 33: Adherence of Glassick’s criteria of excellence: (Composite-QUAL):  This set of 6 items asks the evaluator to review the focal educational efforts of the faculty member according the Glassick’s 6 criteria for a scholarly approach/scholarship: clear goals, adequate preparation, appropriate methods, significant results, effective communication and reflective critique. The data to evaluate could come from any part of the EP, but should reflect the faculty member’s focus of special educational effort. The evaluator is asked to create a composite rating of the author’s adherence to the six criteria in his/her scholarly work.

Item 34: Total Professional Development Score: (QUANT):  This item provides a score weighted for the educator’s degree of commitment (effort, time) to personal educational professional development. Participating in a degree granting program (EdD, MSEd, MEd, PhD in Educ) earns 20 pts. Attending a major professional development program in education (e.g., Harvard Macy program for physician educators, Medical Education Fellowship Program, Michigan State University Primary Care Faculty Development Fellowship Program, APA Educational Scholars Program, etc) earns 10 pts. Attending a whole conference devoted to education earns 5 pts/yr. Attending educational workshops at a national conference earns 1 pt/yr. Note: attending professional development courses in leadership or procedure skills or going to specialty-specific meetings does not qualify for educational professional development points. The Total Professional Development Score sums all the individual weighted scores earned. E.g., someone who over time has earned an MS in Education (20 pts), attended the APA Educational Scholars Program (10 pts), and taken educational workshops at three PAS meetings (3 pt) would receive a Total Professional Development Score of 33.

VII. Products of Educational Scholarship

The Worksheet at end of Instructions will help with this tabulation.

Item 35:  Scholarly Productivity Index (QUANT): This index takes into account type of scholarly product, the impact factor of the journals, authorship role for publications, and geographic dissemination of the work. The Scholarly Productivity Index is a compilation of five subtotals:

• Peer reviewed publications (print or electronic): Peer-reviewed print or electronic publications are assigned an impact score and an author score, weighted as shown. This subtotal is the sum of the scores (impact x authorship) for each publication. E.g., one first author article (3) in a high impact journal (10) would receive a score of 30; one third-author paper (1) in a medium impact journal (2) would receive a score of 2.
• Peer reviewed/invited presentations: Peer reviewed or invited presentations are weighted for the mode of presentation and the geographic impact. This subtotal is the sum for all presentations of their mode scores x geographic impact scores. E.g., a platform talk at a national meeting would score 20, while a poster at a departmental forum would score 1.
• Non-peer-reviewed publication/presentations: This item includes the total number of publications or presentations (not peer reviewed) where the faculty member was the first, second or last author/presenter. (No points are given for other authorship positions.)
• Books: Being the editor of book receives a weight of 5; chapter author receives a weight score of 1. Subtotal is the sum for all books of the role scores. E.g. 2 chapter authorships and one book editorship would yield 7 pts.
• Educational Product Dissemination Score: This score sums all educational products with their dissemination weights. A product that is disseminated nationally or internationally gets a weighting of 10, while a product with only institutional dissemination receives a weighting of 3. The Educational Product Dissemination Score is calculated by adding the dissemination scores for all products.

The Scholarly Productivity Index is a simple sum of the five scores above.

Item 36: Educational Grants Index (QUANT): This index score is the product of two subtotals for educational grants on which the author is the PI or Co-PI (only).

• Grants: Number and geographic impact level: Subtotal 1 gives a score based on the number of grants received and their geographic impact scores. E.g., if one receives two departmental grants (2pt) and one regional grant (3pts), the index score will be 5.
• Grants: Dollar amount: Subtotal 2 is a measure of the dollar amount of the grant: the number of grants, weighted by their dollar category. E.g., one grant of $200,000, one or 12,000, and two of 5,000 would yield a subtotal score of 10.

The Grants Index is the product of subtotal 1 and subtotal 2. (E.g., Index = 50 for the examples above)