Technology, finance & law
A rejected credit application is not proof of discrimination. But nor is it enough to claim that a model is a trade secret and therefore cannot be explained. What matters is the role the automated output actually played, the data that were used and whether a person can obtain a meaningful review.
1. What can be substantiated
Article 22 of the GDPR governs decisions based solely on automated processing that produce legal or similarly significant effects. Where an applicable exception is used, it also establishes safeguards, including the right to obtain human intervention and to contest the decision.[1]
In the SCHUFA case, the Court of Justice of the European Union ruled that creating a probability score may itself constitute automated decision-making if a third party attributes a determining role to it when entering into or performing a contract.[2]
The Artificial Intelligence Act classifies certain systems used to assess the creditworthiness of natural persons as high-risk. This entails obligations concerning risk management, data, documentation, logs, information and human oversight; it does not, however, ban all scoring.[3]
2. How to interpret the claims in context
Discrimination can arise even without using a protected characteristic if another variable acts as its proxy. This is a known risk mechanism, not proof that it occurred in a particular model. Establishing that requires an audit of the data, error rates and impacts on different groups.
A meaningful explanation need not disclose source code or the entire model. It must, however, enable a person to understand the main logic and decisive circumstances well enough to correct inaccurate data or contest the outcome.
Formal human approval does not automatically amount to genuine oversight. If the employee lacks the information, time or authority to change the verdict, it may be no more than a rubber stamp on the automated output.
- European law does not leave automated lending processes unregulated.
- What matters is the score's actual influence, not merely the name of an internal process.
- High-risk classification entails obligations, not automatic illegality.
- The variables and training data used by a particular bank's model.
- Differences in error rates between groups in the absence of audit findings.
- Whether human review at a particular institution can actually change a decision.
3. Five questions to ask
- Was the decision determined solely or effectively by the score?
- Which data can the applicant review and correct?
- What does human review involve, and who has the authority to change the verdict?
- How does the institution test for disparate impacts on different groups?
- Is the explanation sufficient to contest the decision effectively?
4. Conclusion
Algorithmic scoring is neither automatically fair nor automatically prohibited. It becomes trustworthy only through reviewable data, genuine human intervention and the opportunity to correct an error that denied someone access to credit.
— Jiný Kontext