SAP today published Assessing the Security of Large Language Models - Exploring vulnerabilities, threats, and potential ...This brochure by SAP’s Security Research team has been circulating internally but we thought it valuable to share publicly. It's a great nuanced and reasoned overview of the key security risks and benefits of Large Language Models (LLMs) or Generative Artificial Intelligence (AI), and a good companion to the SAP AI Ethics Handbook. I strongly encourage you to read it.
One of the points the paper repeatedly makes is the potential for LLMs to hallucinate and give incorrect results. This possibility is why it recommends that developers and security experts need to review the output when using LLMs to help and accelerate their work to make sure it's correct and reliable. This point is all the more urgent, considering that Stanford University and University of California Berkeley researchers found that model behavior can change substantially in a short amount of time. The researchers stress the need for continuous monitoring of Generative AI models.
Hallucinations are a feature of the architecture of LLMs. Output can be unpredictable and the same question asked twice can yield different results. Even if all the good advice in the white paper is followed, we still need to be careful with AI output. Despite good intentions, we're going to see situations where users will trust AI models more than justified. I call this Generative AI’s Sat nav Problem.
Satellite Navigation and Human Judgment
Satellite or GPS navigation (sat nav) for cars was introduced in 2000 and mapping apps became common in phones from 2007. Sat nav saved us from route planning and paper maps. It also made navigating unfamiliar roads much easier and less stressful. However, a segment of drivers clearly trusted the instructions too much and found themselves in ditches, fields, lakes, or ponds. There are millions of search results for “driving into a lake following GPS navigation” alone.
Maps and apps over the years have improved and are much more up-to-date with better quality guidance. They can include traffic flow, work areas, and even speed traps. But I'm surely not the only one who finds sat nav instructions sometimes still confusing. However, we've learned that sometimes the guidance is wrong and we keep our eyes onto the road and stay aware of our surroundings. We know we can ignore an instruction for road safety reasons, for instance. If we take a wrong turn or miss an exit, it will reroute anyway. We've come to treat it with the right amount of trust.
Large Language Models and Human Judgment
LLMs have a similar problem. I love Kelly Shortridge’s quote that “computer systems are inherently sociotechnical. Humans design, build, and operate them”. I would add that humans also use them. LLMs have the potential to hallucinate, but they also tend to sound authoritative. We may be aware that we can trust LLMs only up to a point, but considering how convincing the answer is likely to be, many of us will be fooled. Even when the LLMs are trained and monitored well, we haven’t yet trained ourselves to use them appropriately, as, for instance two lawyers found out the hard way.
We must remain aware that we’re talking to a machine without a conceptual understanding of the world, and not to a human. If we’re not careful, we can fool ourselves it loves us and want us to leave our spouse. Even LLM software engineers, apparently, are prone to anthropologizing the AI models they develop. It's important we learn to treat Generative AI as we've learned to treat sat nav: as useful guidance that we must be ready to ignore depending on circumstance. We must feel uninhibited to correct it and ask it to try again. You can’t hurt its feelings. It doesn’t have any.
Drive Carefully
SAP is committed to relevant, reliable, and responsible business AI solutions. Despite of our effort and intentions, you still have a responsibility to treat output with the trust it deserves and be ready to discard its advice. It can give great suggestions and propose a scenic route or shortcut you didn’t know was there. But you must decide based on all information available to you and feel free to override its guidance.
Treat LLMs as sat nav. Don’t drive into a lake.
- SAP Managed Tags:
