aif360.metrics.MDSSClassificationMetric

class aif360.metrics.MDSSClassificationMetric(dataset: aif360.datasets.binary_label_dataset.BinaryLabelDataset, classified_dataset: aif360.datasets.binary_label_dataset.BinaryLabelDataset, scoring: Union[str, aif360.detectors.mdss.ScoringFunctions.ScoringFunction.ScoringFunction] = 'Bernoulli', unprivileged_groups: dict = None, privileged_groups: dict = None, **kwargs)

Bias subset scanning is proposed as a technique to identify bias in predictive models using subset scanning [1].

This class is a wrapper for the bias scan scoring and scanning methods that uses the ClassificationMetric abstraction.

References

[1]	Zhang, Z. and Neill, D. B., “Identifying significant predictive bias in classifiers,” arXiv preprint, 2016.

Parameters:

• dataset (BinaryLabelDataset) – Dataset containing ground-truth labels.
• classified_dataset (BinaryLabelDataset) – Dataset containing predictions.
• scoring (str or ScoringFunction) – One of ‘Bernoulli’ (parametric), or ‘BerkJones’ (non-parametric) or subclass of aif360.metrics.mdss.ScoringFunctions.ScoringFunction. Defaults to Bernoulli.
• privileged_groups (list(dict)) – Privileged groups. Format is a list of dicts where the keys are protected_attribute_names and the values are values in protected_attributes. Each dict element describes a single group. See examples for more details.
• unprivileged_groups (list(dict)) – Unprivileged groups in the same format as privileged_groups.

Methods

accuracy	\(ACC = (TP + TN)/(P + N)\).
average_abs_odds_difference	Average of absolute difference in FPR and TPR for unprivileged and privileged groups:
average_odds_difference	Average of difference in FPR and TPR for unprivileged and privileged groups:
base_rate	Compute the base rate, \(Pr(Y = 1) = P/(P+N)\), optionally conditioned on protected attributes.
between_all_groups_coefficient_of_variation	The between-group coefficient of variation is the square root of two times the between_all_groups_generalized_entropy_index() with \(\alpha = 2\).
between_all_groups_generalized_entropy_index	Between-group generalized entropy index that uses all combinations of groups based on self.dataset.protected_attributes.
between_all_groups_theil_index	The between-group Theil index is the between_all_groups_generalized_entropy_index() with \(\alpha = 1\).
between_group_coefficient_of_variation	The between-group coefficient of variation is the square root of two times the between_group_generalized_entropy_index() with \(\alpha = 2\).
between_group_generalized_entropy_index	Between-group generalized entropy index that uses self.privileged_groups and self.unprivileged_groups as the only two groups.
between_group_theil_index	The between-group Theil index is the between_group_generalized_entropy_index() with \(\alpha = 1\).
bias_scan	DEPRECATED: Change to new interface - aif360.detectors.mdss_detector.bias_scan by version 0.5.0.
binary_confusion_matrix	Compute the number of true/false positives/negatives, optionally conditioned on protected attributes.
coefficient_of_variation	The coefficient of variation is the square root of two times the generalized_entropy_index() with \(\alpha = 2\).
consistency	Individual fairness metric from [1]_ that measures how similar the labels are for similar instances.
difference	Compute difference of the metric for unprivileged and privileged groups.
differential_fairness_bias_amplification	Bias amplification is the difference in smoothed EDF between the classifier and the original dataset.
disparate_impact
equal_opportunity_difference	Alias of true_positive_rate_difference().
error_rate	\(ERR = (FP + FN)/(P + N)\)
error_rate_difference	Difference in error rates for unprivileged and privileged groups, \(ERR_{D = \text{unprivileged}} - ERR_{D = \text{privileged}}\).
error_rate_ratio	Ratio of error rates for unprivileged and privileged groups, \(\frac{ERR_{D = \text{unprivileged}}}{ERR_{D = \text{privileged}}}\).
false_discovery_rate	\(FDR = FP/(TP + FP)\)
false_discovery_rate_difference	\(FDR_{D = \text{unprivileged}} - FDR_{D = \text{privileged}}\)
false_discovery_rate_ratio	\(\frac{FDR_{D = \text{unprivileged}}}{FDR_{D = \text{privileged}}}\)
false_negative_rate	\(FNR = FN/P\)
false_negative_rate_difference	\(FNR_{D = \text{unprivileged}} - FNR_{D = \text{privileged}}\)
false_negative_rate_ratio	\(\frac{FNR_{D = \text{unprivileged}}}{FNR_{D = \text{privileged}}}\)
false_omission_rate	\(FOR = FN/(TN + FN)\)
false_omission_rate_difference	\(FOR_{D = \text{unprivileged}} - FOR_{D = \text{privileged}}\)
false_omission_rate_ratio	\(\frac{FOR_{D = \text{unprivileged}}}{FOR_{D = \text{privileged}}}\)
false_positive_rate	\(FPR = FP/N\)
false_positive_rate_difference	\(FPR_{D = \text{unprivileged}} - FPR_{D = \text{privileged}}\)
false_positive_rate_ratio	\(\frac{FPR_{D = \text{unprivileged}}}{FPR_{D = \text{privileged}}}\)
generalized_binary_confusion_matrix	Compute the number of generalized true/false positives/negatives, optionally conditioned on protected attributes.
generalized_entropy_index	Generalized entropy index is proposed as a unified individual and group fairness measure in [3]_.
generalized_false_negative_rate	\(GFNR = GFN/P\)
generalized_false_positive_rate	\(GFPR = GFP/N\)
generalized_true_negative_rate	\(GTNR = GTN/N\)
generalized_true_positive_rate	Return the ratio of generalized true positives to positive examples in the dataset, \(GTPR = GTP/P\), optionally conditioned on protected attributes.
mean_difference	Alias of statistical_parity_difference().
negative_predictive_value	\(NPV = TN/(TN + FN)\)
num_false_negatives	\(FN = \sum_{i=1}^n \mathbb{1}[y_i = \text{favorable}]\mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
num_false_positives	\(FP = \sum_{i=1}^n \mathbb{1}[y_i = \text{unfavorable}]\mathbb{1}[\hat{y}_i = \text{favorable}]\)
num_generalized_false_negatives	Return the generalized number of false negatives, \(GFN\), the weighted sum of 1 - predicted scores where true labels are ‘favorable’, optionally conditioned on protected attributes.
num_generalized_false_positives	Return the generalized number of false positives, \(GFP\), the weighted sum of predicted scores where true labels are ‘unfavorable’, optionally conditioned on protected attributes.
num_generalized_true_negatives	Return the generalized number of true negatives, \(GTN\), the weighted sum of 1 - predicted scores where true labels are ‘unfavorable’, optionally conditioned on protected attributes.
num_generalized_true_positives	Return the generalized number of true positives, \(GTP\), the weighted sum of predicted scores where true labels are ‘favorable’, optionally conditioned on protected attributes.
num_instances	Compute the number of instances, \(n\), in the dataset conditioned on protected attributes if necessary.
num_negatives	Compute the number of negatives, \(N = \sum_{i=1}^n \mathbb{1}[y_i = 0]\), optionally conditioned on protected attributes.
num_positives	Compute the number of positives, \(P = \sum_{i=1}^n \mathbb{1}[y_i = 1]\), optionally conditioned on protected attributes.
num_pred_negatives	\(\sum_{i=1}^n \mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
num_pred_positives	\(\sum_{i=1}^n \mathbb{1}[\hat{y}_i = \text{favorable}]\)
num_true_negatives	\(TN = \sum_{i=1}^n \mathbb{1}[y_i = \text{unfavorable}]\mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
num_true_positives	Return the number of instances in the dataset where both the predicted and true labels are ‘favorable’, \(TP = \sum_{i=1}^n \mathbb{1}[y_i = \text{favorable}]\mathbb{1}[\hat{y}_i = \text{favorable}]\), optionally conditioned on protected attributes.
performance_measures	Compute various performance measures on the dataset, optionally conditioned on protected attributes.
positive_predictive_value	\(PPV = TP/(TP + FP)\)
power	Alias of num_true_positives().
precision	Alias of positive_predictive_value().
ratio	Compute ratio of the metric for unprivileged and privileged groups.
recall	Alias of true_positive_rate().
rich_subgroup	Audit dataset with respect to rich subgroups defined by linear thresholds of sensitive attributes
score_groups	Compute the bias score for a prespecified group of records.
selection_rate	\(Pr(\hat{Y} = \text{favorable})\)
sensitivity	Alias of true_positive_rate().
smoothed_empirical_differential_fairness	Smoothed EDF from [#foulds18]_.
specificity	Alias of true_negative_rate().
statistical_parity_difference
theil_index	The Theil index is the generalized_entropy_index() with \(\alpha = 1\).
true_negative_rate	\(TNR = TN/N\)
true_positive_rate	Return the ratio of true positives to positive examples in the dataset, \(TPR = TP/P\), optionally conditioned on protected attributes.
true_positive_rate_difference	\(TPR_{D = \text{unprivileged}} - TPR_{D = \text{privileged}}\)

__init__(dataset: aif360.datasets.binary_label_dataset.BinaryLabelDataset, classified_dataset: aif360.datasets.binary_label_dataset.BinaryLabelDataset, scoring: Union[str, aif360.detectors.mdss.ScoringFunctions.ScoringFunction.ScoringFunction] = 'Bernoulli', unprivileged_groups: dict = None, privileged_groups: dict = None, **kwargs)

Parameters:

• dataset (BinaryLabelDataset) – Dataset containing ground-truth labels.
• classified_dataset (BinaryLabelDataset) – Dataset containing predictions.
• scoring (str or ScoringFunction) – One of ‘Bernoulli’ (parametric), or ‘BerkJones’ (non-parametric) or subclass of aif360.metrics.mdss.ScoringFunctions.ScoringFunction. Defaults to Bernoulli.
• privileged_groups (list(dict)) – Privileged groups. Format is a list of dicts where the keys are protected_attribute_names and the values are values in protected_attributes. Each dict element describes a single group. See examples for more details.
• unprivileged_groups (list(dict)) – Unprivileged groups in the same format as privileged_groups.

bias_scan(privileged=True, num_iters=10, penalty=1e-17)

DEPRECATED: Change to new interface - aif360.detectors.mdss_detector.bias_scan by version 0.5.0.

scan to find the highest scoring subset of records

Parameters:privileged – flag for group to scan for - privileged group (True) or unprivileged group (False).

This abstract the need to explicitly specify the direction of bias to scan for which depends on what the favourable label is. :param num_iters: number of iterations (random restarts) :param penalty: penalty term. Should be positive. The penalty term as with any regularization parameter may need to be tuned for ones use case. The higher the penalty, the less complex (number of features and feature values) the highest scoring subset that gets returned is.

Returns:the highest scoring subset and the score

score_groups(privileged=True, penalty=1e-17)

Compute the bias score for a prespecified group of records.

Parameters:privileged (bool) – Flag for which direction to scan: privileged (True) implies negative (observed worse than predicted outcomes) while unprivileged (False) implies positive (observed better than predicted outcomes). Returns:float –

Bias score for the given group.

The higher the score, the evidence for bias.