Inter-Rater Reliability Calculator†

This program will estimate the reliability of a set of ratings or other scores based on a formula presented by Robert Ebel.

Ebel RL. Estimation of the reliability of ratings. Psychometrika 1951:16;407-424.

The formula approximates an intraclass correlation and is very flexible in that it requires no assumptions about the number of judges rating each person or object rated nor which judges rate each person or object. All that is required are an alphanumeric ID indicating the person or object rated and a numeric rating.

Organizing the ratings for analysis is easy and can be done on your personal computer. Once you have created the data file with the ratings, Use the browse button below to select it. If you would like to estimate the reliability of averaging several ratings of the same person or object, select the number of raters from the drop-down menu and press submit to upload your data file and calculate the reliability.

Specify the text file with the ratings.

Enter the number of raters if you would like to calculate what the reliability would be if a specified number of ratings were averaged for each person/object being rated.

†Note: This program is free software distributed under the GNU General Public License. A copy of the source code is available in the Medical Education Online Software Resource Section