Inter-Rater Reliability Calculator†
This program will estimate the reliability of a set of ratings or other scores based on a formula presented by Robert Ebel.
Ebel RL. Estimation of the reliability of ratings. Psychometrika 1951:16;407-424.
The formula approximates an intraclass correlation and is very flexible in that it requires no assumptions about the number of judges rating each person or object rated nor which judges rate each person or object. All that is required are an alphanumeric ID indicating the person or object rated and a numeric rating.
Organizing the ratings for analysis is easy and can be done on your personal computer. Once you have created the data file with the ratings, Use the browse button below to select it. If you would like to estimate the reliability of averaging several ratings of the same person or object, select the number of raters from the drop-down menu and press submit to upload your data file and calculate the reliability.
Medical Education Online Editor@Med-Ed-Online.org