fairscoring.metrics.calibration

fairscoring.metrics.calibration#

A module to define the calibration bias metric.

Classes#

CalibrationMetric

This metric measures the differences between two calibration curves [BeDB24].

Module Contents#

class fairscoring.metrics.calibration.CalibrationMetric(weighting='quantiles', n_bins=50, name='Calibration', score_transform=None)#

Bases: fairscoring.metrics.base.TwoGroupMetric

This metric measures the differences between two calibration curves [BeDB24].

Calibration is a way to measure sufficiency bias for continuous scores. The weighting parameter specifies how differences over the total score range are weighted.

This metric returns a TwoGroupBiasResult object, which allows to split the bias in positive and negative parts.

Parameters:

  • weighting (("quantiles", "scores")) – Integral over quantiles / scores

  • n_bins (int) – Number of bins that are used to compute the calibration curves

  • name (str, default="Calibration") – Name of metric.

  • score_transform ({"rescale","quantile",None}) –

    A transformation of the scores prior to the bias computation. There are two supported methods:

    • rescaling (to the interval [0,1]. In this case, the bias() method can take min and max scores.

    • quantile transformation. This leads to standardized bias measures.

bias(scores, target, attribute, groups, favorable_target, *, min_score=None, max_score=None, n_permute=None, seed=None, prefer_high_scores=True)#

Bias computation

Parameters:

  • scores (ArrayLike) – A list of scores

  • target (ArrayLike) – The binary target values. Must have the same length as scores.

  • attribute (ndarray) – The protected attribute. Must have the same length as scores.

  • groups (list) – A list of groups. Each group is given by a value of the protected attribute. A value of None is used to define a group with all elements that are not in another group.

  • favorable_target (str or int) – The favorable outcome

  • min_score (float) – The minimal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling.

  • max_score (float) – The maximal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling

  • n_permute (int, optional) – Number of iterations for the permutation test. Permutation tests are only performed if this value is >0.

  • prefer_high_scores (bool, optional) – Specify whether high scores or low scores are favorable.

  • seed (int, optional) – Random seed for the permutation test. Only required if the result need to be 100% reproducible.

Returns:

bias – The computed bias (including intermediate results)

Return type:

BiasResult

__call__(scores, target, attribute, groups, favorable_target, *, min_score=None, max_score=None, prefer_high_scores=True)#

Bias computation.

This method allows to use the bias metric as a function.

Parameters:

  • scores (ArrayLike) – A list of scores

  • target (ArrayLike) – The binary target values. Must have the same length as scores.

  • attribute (ndarray) – The protected attribute. Must have the same length as scores.

  • groups (list) – A list of groups. Each group is given by a value of the protected attribute. A value of None is used to define a group with all elements that are not in another group.

  • favorable_target (str or int) – The favorable outcome

  • min_score (float) – The minimal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling.

  • max_score (float) – The maximal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling

  • prefer_high_scores (bool, optional) – Specify whether high scores or low scores are favorable.

Returns:

bias – The computed bias.

Return type:

float

Notes

This method offers fewer parameters than bias(), because not all will affect the pure bias value.

Contents