A.I. and tackling the risk of “digital redlining”

This is the web version of Eye on A.I., Fortune’s weekly newsletter on artificial intelligence and machine learning. To get it delivered weekly to your in-box, sign up here.Last week, a Dutch court ordered the government in the Netherlands to stop using a machine-learning algorithm for detecting welfare fraud, citing human rights violations.The system, called System Risk Indicator (SyRI) in English, was being used by four Dutch cities to spot individuals whose benefits applications should receive extra scrutiny. It gathered information from 17 different government data sources, including tax records, vehicle registrations and land registries.

But the cities using SyRI did not run every application through the system—they only deployed it in poor neighborhoods where many residents are immigrants, often from Muslim countries.The court ruled that SyRI violated the “right to private life” enshrined in European human rights law. The application of SyRI, it said, could lead to discrimination against individuals based on their socio-economic status, ethnicity or religion. It also said SyRI did not seem consistent with the requirements of Europe’s stringent data privacy law, GDPR.

Although the judgment only came from a district court and is subject to possible appeal, the decision is likely to set an important precedent within European Union—and it ought to reverberate elsewhere too, as societies around the world come to grips with how to apply fairness in a world of A.I.-driven risk models.Nowhere is this more relevant than in the insurance sector, which is turning to machine-learning algorithms more and more in order to improve underwriting. Last week, I had a fascinating conversation with Daniel Schreiber, the co-founder and CEO of the New York-based insurance startup Lemonade. He shares concerns that the increased use of machine-learning algorithms, if mishandled, could lead to “digital redlining,” as some consumer and privacy right advocates fear.But done right—and with the right measure of fairness—he thinks machine learning has the potential to increase access to financial services and decrease cost.To ensure that an A.I.-led underwriting process is fair, Schreiber promotes the use of a “uniform loss ratio.” If a company is engaging in fair underwriting practices, its loss ratio, or the amount it pays out in claims divided by the amount it collects in premiums, should be constant across race, gender, sexual orientation, religion and ethnicity.

He admits that this means it is entirely possible that some categories of people—Schreiber, who is Jewish, uses the example of Jews—could be charged more on average for property insurance, because, for instance, their religious practice involves lighting candles in the home for certain holidays, and lighting candles might be correlated with a higher risk of house fire.

But, he says, no individual should be charged more because he or she is Jewish. It might turn out that a particular customer isn’t religious and doesn’t light candles. That’s why it is important not to ask people about their religious affiliation—that would be discriminatory. The key is for the insurance company to gather data that actually equates to risk: Do you light candles in your home?In order for it to work properly, insurance companies will need to gather more data about customers, not less. Right now, Schreiber admits, the regulatory winds seem to be blowing in the opposite direction (especially in Europe, as the SyRI case shows). Most insurance regulators don’t understand machine learning. “That creates a fear of the unknown,” he says. What’s more, scandals such as Cambridge Analytica make people reluctant to share more data. But Schreiber says customers might be willing to share more information if the insurers were transparent about why they needed to collect this data, how it was being used, and that it might result in customers paying a lower premium.I wasn’t entirely convinced by Schreiber’s argument. If insurers become that much better at pricing risk, won’t........

© Fortune