Governance Myths AI Has Busted

There's plenty of hype about AI, both that it is somehow the savior of all mankind and that it is equally our near immediate demise. There's concern that it has been created very unethically, and joy over its ability to do incredible things, such as diagnose medical issues far sooner than otherwise possible. But let's move past the fears and the promise, and let's focus a bit more on the concrete and measurable. And there have been profound impacts that all organizations should be aware of as they adjust their governance programs to match.

Administrative Controls Are No Longer Acceptable

Much like honey badger, AI don't give a _____ about your administrative controls.

So what do I really mean by that? Let's define administrative controls. In general they're analogous to laws: laws don't somehow prevent you from breaking them, it is up to the authorities to somehow notice you've violated them and then somehow hold you accountable. The control in this case is something like a police officer clocking your speed, or an investigator reviewing your financial records, or a witness sharing their information with the authorities. That can be a fine, loss of a license (like a driver's license, etc.) or even incarceration. In an organization these are usually written policies such as the anti-harassment policy, the acceptable use policy, and the travel policy. Enforcement in organizations generally involves some sort of reprimand, requirement to pay back misused funds, or even termination.

That type of control has multiple weaknesses when we try to apply them to AI.

If AI is "Responsible"

If we take the tack that AI agents are somehow responsible for their own actions, what possible punishments actually work? A stern talking to from management isn't likely to have any impact. At this point AI doesn't get paid - yes, you could try to charge the AI company you bought the agent from, but I promise you those companies won't be putting that in the contract - so you're not likely to be able to get it to pay the company back for something. Finally, "firing" an AI isn't a punishment either, the AI doesn't care, it doesn't "need" to earn to support itself or a family.

For that matter, how do you get your AI to "attest and agree to" following your administrative controls? Are you going to get all your AI agents to digitally sign copies of your policy documents? Does that somehow mean they're going to follow them?

If People Are "Responsible"

If you attempt to hold people responsible for their use of AI you still have a couple of problems;

• How have you trained your people to use the AI correctly? If you're just giving them access and assuming they can be responsible for it you're essentially sending untrained people out to drive forklifts in the office: some of them will be really safe, some of them will be intimidated and won't us it, others will make basic mistakes, and still others will try to do jousting competitions with each other. The responsibility to ensure they're well trained to properly use such powerful tools is yours, not theirs.
• Did you somehow build good guardrails to ensure the AI works properly? Have you deployed good technical controls to prevent accidental or incidental misuse?
• Do you have reliable logging to ensure that the person you're holding accountable for the AI's behavior is, indeed, the one who got it to do something against the rules?
• Have you been able to identify the intentions of the person who caused the AI to do something wrong?

The long and the short of this is that if you don't have some form of a technical control - separate from and outside of the ability of the AI to modify - then you really don't have effective controls in this new era.

The Cost of Data Storage Is No Longer Just the Storage Cost

Data hoarding used to be expensive. Just thinking about the storage - how much room did it take to store all the books in your home library? Now how much room does your e-reader take up? You needed real shelf space for those books. The homes of the wealthy often used to have whole rooms dedicated to storing books. But the ability to look up things on Wikipedia, check out e-books from your local library, and even storing your favorite music on a hard drive instead of records, 8-tracks, tapes, or even CD's meant that even the smallest New York City studio apartment could have all sorts of cultural amenities within it's tiny confines.

The same is true of every company in the world. If you're old enough to remember, think back to your childhood visits to the doctor's office. Right behind the reception desk was probably a whole room full of filing racks and cabinets, with file folders for each and every patient. When you finally got back in and saw the nurse or doctor, they carried your personal file in with them, and, once done with your visit, that file - no possibly slightly thicker because they added pages during your visit - went back into the files. Today those files (and the real estate they took up) are gone, there's a computer terminal in every exam room.

The cost of storing data on reams and reams of paper was massive in comparison to the cost of storing it on a local hard drive - which again dwarfs the cost of storing it in cloud storage. We're paying fractions of pennies on the dollar - especially when we add in the cost of leasing physical space to store all those cabinets worth of files. And so we all - individuals and companies alike - became data hoarders.

Now bring AI into the mix. AI has some funny peculiarities these days, such as what seems to be a serious difficulty with identifying the chronological order of documents. When you let it loose on your (or your organization's) treasure trove of hoarded data it doesn't know which data is current and authoritative, just which data seems to show up most often. The result: you don't get the answer you're expecting as quickly as you would have with a manual search by your own people. Worse, once you've incorrectly trained an AI on data it shouldn't use, you basically have to kill it off and start over.

The cost of all that old hoarded data becomes lost productivity, misinterpreted data, and time spent fighting with the tool that was supposed to fix your data issues with a magic stroke. Suddenly that out of date data is extremely costly.

So this becomes a great opportunity to trim the data fat. Time to put some actual effort into that data lifecycle management program that only really exists in theory in your organization.

Software No Longer Just "Does What It Is Supposed To Do."

Other than "bugs" in software, the general rule is that software does what it is supposed to do. Calculators calculate. Web browsers browse the web. Word processors process, well, words. Given the same input they always provide the same output.

But AI breaks that trust. For a given input to AI you can get different outputs. Worse, as AI tries to do "almost anything" for you, it tends to do it differently than you expect it to. The chart below is a fantastic illustration of this:

You see a basic illustration of how 3 different systems handle the request to "pick a number between 1 and 10".

• A software Random Number Generator (one that works reasonably well anyway) will come very close to an even distribution of picking each number the same number of times for any given sample of asks. This is what we expect software to do when it generates a random number.
• A human will have an affinity for the number 7, followed by a lesser affinity for the number 5 and 8, and then 6, then 3 and 4, 2 and 9, and finally 1 or 10. There are some psychological reasons for this that I would need a very different degree to go into in a meaningful way, so let's just let it be. (different studies show different results, but generally 7 is the most common answer, and 1, 2, 9, and 10 are the least common)
• An LLM will take a look at whatever training data it has for picking such a number and will almost always pick the number 7, assuming the training data it was given was the human generated data where 7 is the "most popular" answer in that training data.

But when you asked the LLM to give you a number between 1 and 10 you were most likely expecting it to act like a random number generator. You expected it to act like "traditional" software and give you the answer you'd expect from that.

Of course you're probably not going to use an LLM as a random number generator terribly often, but you are likely to ask it to do things that you "expect" standard software to do in a predictable way, but the LLM will do it in another way.

I don't have great advice here - other than to consider ways to ensure your LLM tools are either acting the way you expect them to, or are perhaps calling out their own assumptions and actions to the users to limit misaligned output. If you come up with something great you'll share it, right?

Copyright? What Copyright?

As a last governance concern let's talk copyright. Most jurisdictions do not identify AI generated content as being copyrightable. So that new company logo you have, you might be able to trademark it, but you might not be able to get a copyright for it. Or that particularly insightful article you just published? That might not be your property. That code you just wrote, might not be yours to license properly. And then there's the music you just published...you get the idea.

I keep saying "might," because the law seems pretty cloudy here. Jurisdictions seem to recognize that at some point if the AI is being used as a tool wielded by someone who is creating new content with the aid of AI, then the content can be granted a copyright. But as of yet no branch of government has provided usable guidance about where the line of demarcation is between what can and cannot be granted a copyright.

Then there's the question of if your AI content has infringed on another entity's copyright. Given the less than well defined state of the legality of the training data used by many of the LLMs out there, using AI generated content could put you in an actionable position. (read: somebody might be able to sue the crap dollars out of you) You may want to consult with your legal department before signing a corporate contract with an LLM vendor to be sure to understand how they will indemnify you if you're accused of using someone's copyright materials through your interactions with the LLM.

Is This My Chicken Little Moment?

Doe these issues mean you can't use LLMs in your organization? Of course not, and let's not pretend that a blog post written by some random cybersecurity thinker is going to suddenly get your company to rip out all your AI projects and return to the before times. No, that's not what I'm trying to say.

Like so many of the great advancements in our professional lives, AI is causing us to change how we think about things that we may have considered "settled" issues in the past.

• Computers got us to rethink paper record keeping, and had profound governance impacts
• The web got us to rethink how we interact with the mail, shopping, and many other things, and had profound governance impacts
• Cell phones and mobile computers got us to rethink the physical work environment, and had profound governance impacts

If we aren't addressing the governance changes that AI is demanding then we're going to find our organizations bleeding money, shedding jobs, and losing investors because we haven't addressed these concerns.