Data Organization The ratings that will be analyzed by this program can be entered into a file on your personal computer. This file will be automatically uploaded and used to estimate the reliability of the ratings when you press the "Submit" button on the form. Enter the ratings into a text file such that each rating is on a single line preceded by an unique identifier for the person/object being rated. The identifier, which can be either numeric or alphanumeric, should be separated from the rating, which must be numeric by comma (,). An example is given below: Joe, 5 In the example above, there are three ratings for Joe and two each for Tom and Sally. For the purposes of estimating the reliability via this approach, it is immaterial whether the same or different raters rated each of the three individuals and there is no need to specify what rater provided each rating, only who (or what) was rated. You can create this file in a number of ways. One is to enter it directly using Notepad. You could also use Word or another word processing program and save the file as a "text" file. Another way to create the data file is by using Excel. Use one column for the identifier and one for the rating, Once the file is created, save it as a comma delimited CSV file by specifying "CSV (comma delimited)" in the drop-down menu below where you enter the file name when you save the Excel workbook. NOTE: If non-numeric data are entered as ratings, the results will be unpredictable. The reliability of ratings in theory ranges between zero and one. The method used to estimate the reliability in this program can potentially result in estimates of the reliability that are negative. This generally means the actual reliability is near zero and the negative reliability estimate generated by the program is due to sampling error. If the reliability generated by the program is large and negative or you have good reason to believe the reliability of the ratings should be fairly high, check your data. As noted above non-numeric characters inadvertently entered as ratings will result in unpredictable results. |