A practical approach to ethical integration of AI features
The challenge
I imagine many of you have read about the challenges AI poses to society. Still I want to talk about them. In my experience, many people both over – and underestimate AI and its potential impacts on society. Most people don’t realise how absolutely non-intelligent the large language models really are. „AI not intelligent like humans“, says UC’s Anthony Chemero, University of Cincinnati (see Koenig 2023). „No, Today’s AI Isn’t Sentient. Here’s How We Know“ (Li/Etchemendy 2024). „LLMs have special intelligence, not general, and that’s plenty.“ (Krishnan 2024) Access to this and all links mentioned in this article: 29.07.2024 But those who do, often go the other way, under estimating the impact AI still can and already does have on the world.
Let me first introduce the perspective from which I write. Nextcloud (https://nextcloud.com/about), the company I co-founded, is an organisation with a mission. We started our project because we saw the dangers of big tech firms dominating the market. We do not want to live in a world where 5 or 6 companies own all our data. Google, Facebook, Amazon, Microsoft – you know them. With their control over our digital lives come huge risks to our society.
Freedom and Free Software
In the open source world, where most of the people in our company come from, there is a saying: „In a world where speech depends on software, free speech depends on free software.“ (Kirschner 2010) The term Free Software here is kind of a synonym of open source with a stronger emphasis on the societal and ethical values of open source. Why „Free Software“ is better than „Open Source“; www.gnu.org/philosophy/free-software-for-freedom.en.html
Today, our speech is mediated, if not outright controlled, by Big Tech platforms. For most non-technical people, this is completely normalised. They use Twitter, Facebook, YouTube or Instagram. They see posts promoted on their ‚feed‘, watch the video or image, and like, share and subscribe. Just like you press the pedal on a car to accelerate, you feel like you are in control. But, and this is becoming true for cars as well, nothing is further from the truth.
If you’ve heard the likes of Daniel Dennet and Sam Harris (2012) www.youtube.com/watch?v=pCofmZlC72g; Also read up on the „hard problem“ by David Chalmers; www.youtube.com/watch?v=C5DfnIjZPGw talk about free will, you might be familiar with a little thought experiment. It goes like this: observe your own mind. Now, think of a city in the world. Any city. Pick one, maybe write it down. Which city did you pick? Maybe it is Berlin, New York or Amsterdam. Did you pick Buenos Aires? No? But you DO know the city, do you not? Why did you not pick it? Maybe you were thinking of European cities. Or big ones. Or small ones. Remember, the goal was: observe your mind. What happened is that some names came up in your mind. You then picked one. But Buenos Aires did not come up. So the question now is: where you free to choose Buenos Aires?
One can argue about this all day. But it is certain that you do not pick the posts that show on your Facebook feed or the videos on your TikTok stream. Algorithms, or AI, are used for that. And the thoughts they then put in your head – be it positive, or negative – these weren’t your choice either. And these algorithms have supposedly only one goal: engagement.
Do they?
Fighting Big Tech
I am skeptical, and many with me. Big companies spend millions pushing legislators to make decisions that benefit them. TikTok has used its app to mobilise its millions of users in the USA to protest a TikTok ban Congress was deliberating. Is it hard to imagine they would make some slight tweaks to the algorithm to influence users to its advantage? Remember, just like with the cities that come up in your mind, you don’t choose what you see. And you don’t know how the app does it, either, just like we don’t really understand how our mind works …
Yet, the impact these algorithms have on our society are enormous. It is widely accepted that the genocide in Myanmar was mediated, if not triggered, by Facebook (see Amnesty International 2022). Just think about that for a second. Tens of thousands are dead because an American big tech company decided that moderation in a specific language was too expensive. And what about mis-information about Covid (see Gisondi et al. 2022) and other vaccines? It is clear that mis-information kills – at least 2800 people in Canada (see CCA 2023), just in case of Covid.
We can not, and should not, live in a world where big tech has this much power. So, we founded Nextcloud, to build an alternative to some of this technology, an alternative that is under control of the people.
We can not, and should not, live in a world where big tech has this much power. So, we founded Nextcloud, to build an alternative to some of this technology, an alternative that is under control of the people. Open Source, that is, its users have full access to its source code, not only to study, but also to modify and distribute it further. That means that if there are any algorithms that control what you see, you can study, change, improve and share them. Call it Free Will for the Digital World, if you wish.
Now Nextcloud does of course not tackle everything the big tech firms do. Our focus is on ‚collaboration‘. That is, we started with sharing files, then editing them, then communication and then planning. So, with Nextcloud, you can access and share documents, edit them together with others, have a chat or video call, send emails, and plan using a calendar and project management tools. While we have some social media and other components, they are not core to Nextcloud.
And Nextcloud is not hosting your data. We simply make a piece of software you can run on a computer – a server – by yourself, or shared with others. Your server then offers you a browser interface plus mobile apps similar to Google Workspace or Microsoft 365. But, unlike with those solutions, your data stays on your server and under your control.
So now you know Nextcloud, what it is, and why we created it. Let’s talk about AI.
Then AI came
A disclaimer – I’m not a deeply technical person – my knowledge stops just after differentiating neural networks from stochastic technologies like Markov models. So I know there are different things we might call AI – and, for marketing reasons, I probably do – that most developers would disagree with. Of course, a philosopher would observe the disagreement between marketing and engineering and consider them both wrong.
But, as marketing person, I’ll say – Nextcloud has been including AI features for a while. Even if these weren’t based on terabytes of training data. For example, in 2019, we introduced a feature named ’suspicious login detection‘ (see Wurst 2019). It trains a simple neural network on login data of users, then gives a warning when an unexpected login happens. For example, if you typically work 9–5 from the office in Paris, you’ll get warned about a 2AM login from Brazil.
Shortly after we introduced our smart inbox, technically very similar. In 2020, the Recognise feature caused some technical challenges: it uses a big (up to a gigabyte) neural network to recognise objects in photos and tag them. We fixed the technical issue – Nextcloud wasn’t really used to applications that big, so we had to build a way to download the neural model separately. We did not think much about these technologies from a moral or ethical point of view, other than making sure they would run locally and not leak data of our users.
Then ChatGPT was released. We realised how impactful this was, and started to build integration. But we also immediately understood there was a problem: Nextcloud is all about privacy and control. We could of course not put in a feature that just sends all user data to a big tech firm.
This wasn’t the only issue. In the months after ChatGPT became available, a ton of discussions started. There were debates around consciousness, copyright, its impact on jobs, bias, environmental concerns and more.
Now we are a tech firm, but one with a mission. So, first and foremost, we needed to protect the data of our users. Any solution we would deliver had to do that. But you can’t deny that there are MANY more challenges around AI than just where the data is. So we started a debate: how can Nextcloud integrate AI features in an ethical, responsible way?
AI in an ethical way
To answer this, the first question you should ask is: Is AI a good idea in the first place? There are many perspectives. I’m simply going to share ours, at Nextcloud.
At a high level, we asked ourselves about the utility of AI, and in particular the LLM models and image generators. Can they help our users; do they provide a benefit to their day to day work? Going back to the start of this essay, I posed that many people fail to see the profound impact AI can have on the world. Granted, research shows a nuanced picture when it comes to jobs that would be replaced by AI See „New Research May Calm Some of the AI Job-loss Clamor – For Now“; https://ide.mit.edu/insights/neil-thompson-research-may-calm-some-of-the-ai-job-loss-clamor and certainly some tasks will be more impacted than ever. Without a doubt, however, AI can help users in many ways, and one needs logically only one strong use case to say yes to the question about utility.
And we have strong use cases. AI could help with accessibility, using speech-to-text and text-to-speech for users who otherwise would have trouble interacting with computers. Can you justify not providing this to users? Translation is also a good use case. And in the large language model space, AI is particularly good when summarising, with hallucination rates of the better models well under 5 % (https://github.com/vectara/hallucination-leaderboard). That means that it can help users deal with an overload of information, something so common in the modern workplace one can find a medical review article covering 87 studies on mitigations (see Arnold/Goldschmitt/Rigotti 2023).
But a 5 % hallucination rate is still worthy of pause. We expect mistakes from other humans, but not so much from computers. That brings us to the second part of the question: if it can provide value, what are the downsides? Do these raise to the point where they erode any value?
Now the issues with AI are large, and they impact much of society. The ecological foot print, the risks to jobs, consciousness, copyright, bias and more. But open source, or rather, the Free Software movement we are a part of, has a culture of giving the user choice. So we decided that, rather than making the choice for our users, we should put them in the driver seat.
This decision formed the basis of our approach to ethical AI.
Ethical AI
Fundamentally, that approach is all about transparency. After conversations, both internally and externally, we decided we would devise a rating mechanism. This would give users an idea of the consequences of the AI solution they chose.
In an ideal world, this rating mechanism would cover all consequences. But many of the issues with AI are remote, complicated and cover large, societal impacts (see Hendrycks/Mazeika/Woodsid 2023). They are far removed from a simple „should I install this LLM to summarise my email threads?“ kind of decision. We are not in a position to say „this AI image recognition model might overthrow the world“. So we looked at what we COULD say about the various solutions we implemented.
We narrowed the information we have down to these three factors:
- Is the model freely available? This would allow one to run the model locally, avoiding sending data to a third party service.
- Is the training data freely available? This would allow one to inspect the data set, looking for bias, copyright violations and other issues.
- Is the code for training and inferencing freely available? This would allow one to add to and improve on the training, for example to fix bias, reduce its ecological footprint, or remove copyrighted materials.
Our rating is a simple, four-shade traffic light. Red when all factors are absent, green when all are present, and with orange and yellow indicating 2 and 1 missing factors.
So now, we provide transparency with this rating – and we engineered our product to offer choice. For nearly all functionality, users can pick from a number of solutions. For example, for translation, Deepl is a choice. It is red – as a remote service, we have very little insight in its data set, lose control over the data we sent to its service and can not re-train or optimise its operations. So, we provide another choice with a translation model built by us using free training data by the University of Helsinki (https://github.com/Helsinki-NLP/Opus-MT?tab=readme-ov-file). This can be run locally, so no data leaks.
The weasels
Now, this rating is not perfect. Many risks of AI are simply not covered by it. For example, it would be good to know which model is more or less biased, or uses less energy, or hallucinates less, or has less copyright issues. Unfortunately these are all factors which are very hard, if not impossible, to cover in simple yes/no questions. Nor are there any objective measurements of these – yet.
So, for now, the rating with 3 factors will have to do. Users have to exercise caution, and look beyond the rating.
However, our responsibility does not end just because we gave users information. Our AI team is led by Daphne Muller, who besides being a capable manager, is also a university researcher in the areas of privacy and AI (www.daphnemuller.nl). While the engineers on the team largely consist of highly technically oriented developers and PhD’s in the area of AI, she encourages a skeptical view toward AI integration in Nextcloud. After all, as technical people, we have our biases. In particular, many in the tech world are ‘techno-optimists’ (https://a16z.com/the-techno-optimist-manifesto) – people who believe technology will fix all/most societal issues. This blinds us sometimes to the risks and dangers. And AI discussions are particularly vulnerable to this, as AI is such a nebulous term.
She introduced the team to a rule for when new ideas for AI features are shared, the Weasel rule. It goes as follows.
Whenever you want to propose a feature for which AI could be used, replace the term ‚AI‘ in your description with ‚trained weasel‘. And you have to note that the trained weasel is racist and sexist. Then ask yourself: „Does it still sound like a good idea?“
To illustrate how this works, let’s look at a few examples.
- Would you use a trained weasel to turn your rough drawing into a prettier picture? Well, I suppose it won’t do too much harm. After all, I could improve my drawing skills, too … But keep in mind, the weasel is racist and sexist, so some drawings might go better than others! Caution is still advised.
- Would you use a trained weasel to drive several tons of steel at highway speed? No, that does not sound very safe.
- Would you use a trained weasel to inform hiring and firing decisions? No, the weasels are racist and sexist, that sounds like a terrible idea!
Looking at AI technologies in this way helps make more sensible decisions – even if it doesn’t catch all problems.
Impact on society
So this is our approach to AI at Nextcloud. We offer users a choice, providing some transparency to help them choose. We do our best to make sure open source, locally hosted options are available for AI. And we are careful in how and where we implement it, trying to avoid unforeseen consequences.
Say we live in a perfect world, and every company followed a similar approach. Would we fare well? I’m sorry – maybe you expected me to say yes to that – but I can’t. I don’t believe it would.
First is the obviously lacking elements in this rating I already pointed out. We, at Nextcloud, but also in society at large, are starting to use these technologies at scale at an incredible speed. Without having an idea to measure how biased they are. Without knowing how to deal with copyright issues or environmental impact. And worse, we don’t know what we don’t know!
We have to address these and other points – and sadly we’re doing that from a position of weakness, rather than strength.
Say we live in a perfect world, and every company followed a similar approach. Would we fare well? I’m sorry – maybe you expected me to say yes to that – but I can’t. I don’t believe it would.
We are weak institutionally, as our politics are extremely divided and our bureaucracies are struggling. The market is so dominated by a few huge firms with revenues that rival the GDP of countries that democratic oversight is straining. And our institutions are weakened even more by AI and how it’s used for active attacks by internal and external forces.
But we also suffer from the fact that we’ve failed to address other technological issues – from this dominance of big tech companies to privacy, the spreading of disinformation and challenges with social media. In the legal area, we already have issues with copyright, patents and other intellectual property laws that struggle to function well in the digital realm.
AI worsens all of these problems. Its huge costs and need for data make big tech even stronger. It helps spread disinformation faster than ever. It strains the reason and logic behind existing copyright laws. Its massive energy costs are a growing threat to our environment. And we don’t even know yet what it might do for IT security.
So there are many fires burning, with AI pouring gasoline on all of them, and the fire fighters are mostly out of commission or turn out to be pyromaniacs who love to let it all burn.
Good news
The good news is that there are attempts to address many of these fires. Especially in the EU, but also in the US, lawmakers realise we failed to legislate on tech over the last 30 years. After the GDPR showed that legislation, even if imperfect, can be effective, the taste for more action grew. We now have the DMA and DSA that recently went into effect.
The Digital Markets Act (https://digital-markets-act.ec.europa.eu/index_en) addresses so called gatekeepers, companies like Apple and Google with their app stores, booking.com, Uber, Amazon and Microsoft which have created ecosystems in which other vendors actually do most of the work. These companies start to impose a ‚tax‘ on these services, keep out competitors and innovators – harming consumers in the long run.
AI supercharges the power of these gatekeepers. Take Amazon, for example. They are well known for using sales data to launch new products, out-competing the other vendors on the platform. All these gatekeepers have loads of data they alone have access to, allowing them to maximise earnings and crush competition. The law limits the ability of these platforms to abuse their power in various ways.
The Digital Services Act (https://digital-strategy.ec.europa.eu/en/policies/digital-services-act-package) is primarily designed to protect consumers from mis-information, privacy violations, and so on. Here, AI risks around automated content, mis-information and so on is at least partially mitigated by the law.
Especially in the EU, but also in the US, lawmakers realise we failed to legislate on tech over the last 30 years.
So with these legal measures, the EU is taking moving to protect consumers and the wider market. The oversight will also help curb some of the risks around AI by putting guard rails around the abusive behaviour of the big tech firms.
But there is more. The EU has also moved forward with a law specifically dedicated to AI, the AI Act (https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai).
I will not fully explain the AI Act – I do recommend you read up on it (see also the article by Zimmermann in this issue). It is a comprehensive and fairly cleverly set up legal framework to judge and curb the risks of AI applications. Legislators have taken lessons from the GDPR and the unintended costs it imposed in some cases. Like with the DMA and DSA, which only apply to large, market-dominant organisations, the AI Act separates various use cases and scales limitations with the potential risk. One would almost think they heard Daphne talk about her weasels …
Concluding
I think many of you will understand that the legislation the EU has set up, as well as similar rules coming in other jurisdictions, are barely scratching the surface of the potential challenges with AI. After all, we don’t even know what bears we will find on the road in the coming years.
However, it is a start. These new laws will give our institutions tools to start reigning in the big tech firms, a prerequisite to managing the risks of AI.
Hopefully, other businesses will follow the example of Nextcloud in how they handle AI – with care. In the open source world, efforts are underway to define an ethical approach to AI, very much aligned with our way of thinking (https://opensource.org/deepdive).
The wider tech community, including the well-known ISO standards organisation, is also thinking about some guidance (www.iso.org/artificial-intelligence/responsible-ai-ethics). I do not believe these efforts are enough without oversight and pressure from the government – some of these are likely cynical attempts to avoid legislation. Or, perhaps worse, attempts at regulatory capture (https://en.wikipedia.org/wiki/Regulatory_capture), whereby the large companies use their lobbyists to push for legislation that increases the barrier to entry for smaller competitors.
We are not in the clear yet. But a lot of research into the risks of AI is taking place, and the legislators are taking action. I, personally, am optimistic. AI has great promise – we just have to make sure we stay in control, both of the technology itself but also of those who wield it.
To the author
jos.poortvliet@nextcloud.com