fairscoring.metrics.base

fairscoring.metrics.base#

Base class for bias metrics

Classes#

BaseBiasMetric

Base bias metric. This class covers the basic bias computation workflow.

BiasResult

A base class to store bias results.

TwoGroupMetric

A mixin for metrics that only support two groups

TwoGroupBiasResult

A base class to store bias results.

Module Contents#

class fairscoring.metrics.base.BaseBiasMetric(name, score_transform=None)#

Bases: abc.ABC

Base bias metric. This class covers the basic bias computation workflow.

Parameters:

  • name (str) – Name of the metric.

  • score_transform ({"rescale","quantile",None}) –

    A transformation of the scores prior to the bias computation. There are two supported methods:

    • rescaling (to the interval [0,1]. In this case, the bias() method can take min and max scores.

    • quantile transformation. This leads to standardized bias measures.

bias(scores, target, attribute, groups, favorable_target, *, min_score=None, max_score=None, n_permute=None, seed=None, prefer_high_scores=True)#

Bias computation

Parameters:

  • scores (ArrayLike) – A list of scores

  • target (ArrayLike) – The binary target values. Must have the same length as scores.

  • attribute (ndarray) – The protected attribute. Must have the same length as scores.

  • groups (list) – A list of groups. Each group is given by a value of the protected attribute. A value of None is used to define a group with all elements that are not in another group.

  • favorable_target (str or int) – The favorable outcome

  • min_score (float) – The minimal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling.

  • max_score (float) – The maximal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling

  • n_permute (int, optional) – Number of iterations for the permutation test. Permutation tests are only performed if this value is >0.

  • prefer_high_scores (bool, optional) – Specify whether high scores or low scores are favorable.

  • seed (int, optional) – Random seed for the permutation test. Only required if the result need to be 100% reproducible.

Returns:

bias – The computed bias (including intermediate results)

Return type:

BiasResult

__call__(scores, target, attribute, groups, favorable_target, *, min_score=None, max_score=None, prefer_high_scores=True)#

Bias computation.

This method allows to use the bias metric as a function.

Parameters:

  • scores (ArrayLike) – A list of scores

  • target (ArrayLike) – The binary target values. Must have the same length as scores.

  • attribute (ndarray) – The protected attribute. Must have the same length as scores.

  • groups (list) – A list of groups. Each group is given by a value of the protected attribute. A value of None is used to define a group with all elements that are not in another group.

  • favorable_target (str or int) – The favorable outcome

  • min_score (float) – The minimal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling.

  • max_score (float) – The maximal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling

  • prefer_high_scores (bool, optional) – Specify whether high scores or low scores are favorable.

Returns:

bias – The computed bias.

Return type:

float

Notes

This method offers fewer parameters than bias(), because not all will affect the pure bias value.

class fairscoring.metrics.base.BiasResult(bias)#

A base class to store bias results.

Child classes of this class are used to store metric-specific intermediate results that allow for further bias analyses, such as metric-specific plots.

Parameters:

bias (float) – The bias value

Variables:

  • bias (float) – The bias value

  • p_value (float, optional) – The p_value of statistical tests.

class fairscoring.metrics.base.TwoGroupMetric(name, score_transform=None)#

Bases: BaseBiasMetric

A mixin for metrics that only support two groups

Parameters:

  • name (str)

  • score_transform (Optional[Literal['rescale', 'quantile']])

bias(scores, target, attribute, groups, favorable_target, *, min_score=None, max_score=None, n_permute=None, seed=None, prefer_high_scores=True)#

Bias computation

Parameters:

  • scores (ArrayLike) – A list of scores

  • target (ArrayLike) – The binary target values. Must have the same length as scores.

  • attribute (ndarray) – The protected attribute. Must have the same length as scores.

  • groups (list) – A list of groups. Each group is given by a value of the protected attribute. A value of None is used to define a group with all elements that are not in another group.

  • favorable_target (str or int) – The favorable outcome

  • min_score (float) – The minimal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling.

  • max_score (float) – The maximal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling

  • n_permute (int, optional) – Number of iterations for the permutation test. Permutation tests are only performed if this value is >0.

  • prefer_high_scores (bool, optional) – Specify whether high scores or low scores are favorable.

  • seed (int, optional) – Random seed for the permutation test. Only required if the result need to be 100% reproducible.

Returns:

bias – The computed bias (including intermediate results)

Return type:

BiasResult

__call__(scores, target, attribute, groups, favorable_target, *, min_score=None, max_score=None, prefer_high_scores=True)#

Bias computation.

This method allows to use the bias metric as a function.

Parameters:

  • scores (ArrayLike) – A list of scores

  • target (ArrayLike) – The binary target values. Must have the same length as scores.

  • attribute (ndarray) – The protected attribute. Must have the same length as scores.

  • groups (list) – A list of groups. Each group is given by a value of the protected attribute. A value of None is used to define a group with all elements that are not in another group.

  • favorable_target (str or int) – The favorable outcome

  • min_score (float) – The minimal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling.

  • max_score (float) – The maximal score. This might influence the bias computation, e.g. by defining the integral bounds. This is also used for rescaling

  • prefer_high_scores (bool, optional) – Specify whether high scores or low scores are favorable.

Returns:

bias – The computed bias.

Return type:

float

Notes

This method offers fewer parameters than bias(), because not all will affect the pure bias value.

class fairscoring.metrics.base.TwoGroupBiasResult(bias, pos, neg)#

Bases: BiasResult

A base class to store bias results.

Child classes of this class are used to store metric-specific intermediate results that allow for further bias analyses, such as metric-specific plots.

Parameters:

  • bias (float) – The bias value

  • pos (float) – The positive component of the bias

  • neg (float) – The negative component of the bias

Variables:

  • bias (float) – The bias value

  • pos (float) – The positive component of the bias

  • neg (float) – The negative component of the bias

property pos_component#
Proportion of the positive component in the total bias
Return type:

Proportion of the positive component in the total bias

property neg_component#
Proportion of the negative component in the total bias
Return type:

Proportion of the negative component in the total bias

Contents