A multigroup comparison of IRR using a modified Fleiss’ Kappa statistic

Inter rater reliability (IRR) among more than two raters may be calculated using a modified Fleiss’ Kappa. I suggest a modified version of the procedures for modified Fleiss’ Kappa enabling the surveyor to compare different groups of respondents across a range of evaluators.

Scripts: IRR for nomial data IRR for ordinal data using a linear weight matrix