measures
Module containing Inter Rater Agreement Measures.
human_protocol_sdk.agreement.measures.agreement(annotations, measure='krippendorffs_alpha', labels=None, bootstrap_method=None, bootstrap_kwargs=None, measure_kwargs=None)
Calculates agreement across the given data using the given method.
Parameters:
annotations (
Sequence
) – Annotation data. Must be a N x M Matrix, where N is the number of annotated items and M is the number of annotators. Missing values must be indicated by nan.measure – Specifies the method to use. Must be one of ‘cohens_kappa’, ‘percentage’, ‘fleiss_kappa’, ‘sigma’ or ‘krippendorffs_alpha’.
labels (
Optional
[Sequence
]) – List of labels to use for the annotation. If set to None, labels are inferred from the data. If provided, values not in the labels are set to nan.bootstrap_method (
Optional
[str
]) – Name of the bootstrap method to use for calculating confidence intervals. If set to None, no confidence intervals are calculated. If provided, must be one of ‘percentile’ or ‘bca’.bootstrap_kwargs (
Optional
[dict
]) – Dictionary of keyword arguments to be passed to the bootstrap function.measure_kwargs (
Optional
[dict
]) – Dictionary of keyword arguments to be passed to the measure function.
Return type:
dict
Returns: A dictionary containing the keys “results” and “config”. Results contains the scores, while config contains parameters that produced the results.
Example:
human_protocol_sdk.agreement.measures.cohens_kappa(annotations)
Returns Cohen’s Kappa for the provided annotations.
Parameters: annotations (
ndarray
) – Annotation data. Must be a N x M Matrix, where N is the number of annotated items and M is the number of annotators. Missing values must be indicated by nan.Return type:
float
Returns: Value between -1.0 and 1.0, indicating the degree of agreement between both raters.
Example:
human_protocol_sdk.agreement.measures.fleiss_kappa(annotations)
Returns Fleisss’ Kappa for the provided annotations.
Parameters: annotations (
ndarray
) – Annotation data. Must be a N x M Matrix, where N is the number of items and M is the number of annotators.Return type:
float
Returns: Value between -1.0 and 1.0, indicating the degree of agreement between all raters.
Example:
human_protocol_sdk.agreement.measures.krippendorffs_alpha(annotations, distance_function)
Calculates Krippendorff’s Alpha for the given annotations (item-value pairs), using the given distance function.
Parameters:
annotations (
ndarray
) – Annotation data. Must be a N x M Matrix, where N is the number of annotated items and M is the number of annotators. Missing values must be indicated by nan.distance_function (
Union
[Callable
,str
]) – Function to calculate distance between two values. Calling distance_fn(annotations[i, j], annotations[p, q]) must return a number. Can also be one of ‘nominal’, ‘ordinal’, ‘interval’ or ‘ratio’ for default functions pertaining to the level of measurement of the data.
Return type:
float
Returns: Value between -1.0 and 1.0, indicating the degree of agreement.
Example:
human_protocol_sdk.agreement.measures.percentage(annotations)
Returns the overall agreement percentage observed across the data.
Parameters: annotations (
ndarray
) – Annotation data. Must be a N x M Matrix, where N is the number of annotated items and M is the number of annotators. Missing values must be indicated by nan.Return type:
float
Returns: Value between 0.0 and 1.0, indicating the percentage of agreement.
Example:
human_protocol_sdk.agreement.measures.sigma(annotations, distance_function, p=0.05)
Calculates the Sigma Agreement Measure for the given annotations (item-value pairs), using the given distance function. For details, see https://dl.acm.org/doi/fullHtml/10.1145/3485447.3512242.
Parameters:
annotations (
ndarray
) – Annotation data. Must be a N x M Matrix, where N is the number of annotated items and M is the number of annotators. Missing values must be indicated by nan.distance_function (
Union
[Callable
,str
]) – Function to calculate distance between two values. Calling distance_fn(annotations[i, j], annotations[p, q]) must return a number. Can also be one of ‘nominal’, ‘ordinal’, ‘interval’ or ‘ratio’ for default functions pertaining to the level of measurement of the data.p – Probability threshold between 0.0 and 1.0 determining statistical significant difference. The lower, the stricter.
Return type:
float
Returns: Value between 0.0 and 1.0, indicating the degree of agreement.
Example:
Last updated