Podcasts

Summary

This summer, a congressional staffer pasted raw chatbot output straight into an official amendment to the national defense bill, timestamp and all. Nobody caught it until it went public. In this launch episode for Frantz Ward’s new AI Enterprise Governance practice, host Jim Ickes talks with Chris McCarthy of OB.1 AI Solutions and Anthony Catalano of RSM US LLP about what that moment actually reveals: not that people are using AI, but that almost nobody can say who’s responsible for it. The trio covers what’s real versus what’s hype in AI right now, and the true story of a stolen API key that became a five-million-dollar bill that no insurance policy would cover.

Transcript

Jim Ickes:

Welcome to another edition of Frantz Ward’s podcast, Shoveling Smoke, where we discuss current legal issues affecting the business community and our daily lives. I’m Jim Ickes, your host for today’s edition. This is a special edition of the podcast because it coincides with the launch of a new practice area at Frantz Ward: AI Governance. And I think we have probably two perfect guests for this podcast today who are local to Northeast Ohio, and I will go ahead and let you guys introduce yourself. First off, we have Anthony Catalano from RSM US LLP.

Anthony Catalano:

Great, thanks, Jim. I’m Anthony Catalano. My job day to day is running the risk consulting practice for RSM, specifically in our private equity channel. A quarter of RSM’s revenue comes from private equity, so that means that we have to really specialize our products and services to resonate in that channel. I also work with our AI enablement and AI innovation team to build tools and products that our clients need and can consume.

Jim Ickes:

Fantastic. We also have Chris McCarthy here. Chris, if you want to tell us a little bit about your position at OB.1 AI Solutions. Also, maybe tell us a little bit. It’s an interesting story about how the company originated.

Chris McCarthy:

Certainly, thanks, Jim. Yeah, I’m Chris McCarthy. I’m the founder and chief architect of OB.1 AI Solutions. We’re based out of Hudson, Ohio, and we serve mid-market and small businesses with practical AI use cases. I think my story here in the city and how OB.1 came to be: about three years ago, I had a health event that put me in the hospital for about four months. And I used that time as an ERP specialist to teach myself AI, a lot of different models, and tinkering. That’s how it came to be. And now we have a team of four, and we operate like a team of twenty.

Jim Ickes:

Fantastic. And that’s because of the technology?

Chris McCarthy:

Exactly.

Jim Ickes:

Fantastic. So, let’s just jump right on in. First off, I think I’ve had an opportunity to kind of look behind the curtain a little bit about how both of you use the technology, and I was pretty blown away. And honestly, it was kind of the genesis for this podcast today. If we could maybe start with you, Chris. If you could just tell our audience how you use AI day to day, either personally… and obviously you don’t have to disclose how you use it for your clients, but whatever you want to share with the audience.

Chris McCarthy:

Certainly. As these models get better, and we are using them in an efficient way. So, I have a multitude of agents that help me operate my business. I use Claude Skills as kind of a foundation of the automations, but most of the AI solutions that I have under the hood are something that I would propose to a small business owner. The newest model in Fable, and Jim, you’ve written about this before, is so powerful, but it’s also expensive. So, I have a unique skill called the Fable Foreman that helps establish a plan and then kicks it down to some of the cheaper token-efficient models to execute those tasks. So, building a plan and context engineering is where my tools are very strong.

Jim Ickes:

Fantastic. I wasn’t aware of that new tool that you built. That sounds interesting. Anthony, how about you? How are you using it?

Anthony Catalano:

Yeah, so obviously Chris has an awesome way that he’s using it to enable the speed of his business. I’ll tell you a little bit of a story about how I built my agents. So, I started playing around with Clawdbot last winter when it came out and I created the instance of Clawdbot, which is basically your own AI agent and it will use whichever model you want it to use to do its logic. But it’s kind of your own AI personal assistant. So, I was using that for a couple of months, and I wasn’t super impressed with it. It would make reservations for me. It would do simple things, but nothing really incredible. And eventually I got it to a point where I said, look, and I’m talking to it, we’re not making a lot of progress. I’d like you to try to make me some money here. And it did okay. It was trading on Polymarket and Kalshi and a bunch of other things, trying to make money. And it didn’t really do a great job. And finally, I said, hey, this isn’t working for us. It’s not me. It’s you. What can we do to move on? And I basically said, I’ll put you in a more powerful computer if you can make me eighteen hundred dollars to pay for the computer that I want to put you in. And my personal assistant, it was called Dr. Evil at the time because I’m a huge fan of the Austin Powers movies, said, [tents fingers] that would be awesome. Show me a picture of it. And I showed it a picture. And this thing got genuinely excited, or machine excited, about it. And the next day I left to fly to Austin. And while I’m in the air, I had given instructions like, hey, make sure you make this money or we’re shutting this operation down. So, I land there. And it says, hey, I need you to give me some money so I can make a bet that the Strait of Hormuz is going to close. I was like, what? And it goes, yeah, we’re at war with Iran. It’s logical that the Strait of Hormuz is going to close. I’d like to put a bet on a site that it’s going to happen. I said, yeah, OK, sure. This is going to go great. So, I gave it two hundred dollars, placed the bet, made almost thirty-five hundred dollars on the total trade. It was incredible. And the first thing that it came back with was, hey, can I be put in the new computer now? I said, yep. When I got home, I put it in a much more powerful computer. And then I took that computer… And since I had a GPU, which is a graphics card that can accelerate how it does computations, I started running my own models. So, I switched to something called an Ollama model. And from that primary agent, I built sixteen additional agents. And those agents do a lot of work for me at RSM. So, it actually helps accelerate the speed with which I can operate. And I’ve probably quadrupled my productivity in the last four months. One agent, for instance, is Frau Farbissina, right? And I’m just using these as examples. She’s a corporate espionage agent. She’ll actually go in, look at our competition at RSM, find out what they’re doing in the market, and give me competitive threat intel reports every morning. Another one does CRM management for me. It’ll actually help me classify my entries as what pursuits I’m going on. So, all of these different agents do different things. They all have their own model that they run for their intelligence, but they also have a hive mind. And I can actually pull up a screen that would show you that they’re all talking to each other. So, they’re not only performing their individual tasks and trained in loop programming so that they can complete those tasks, but they talk to each other and hand off tasks to each other to complete those. And all of this is done to optimize my productivity. So, it’s really fun. It’s cool. And like I said, I probably have become four times more productive than I was six months ago because of it.

Jim Ickes:

So, I often say that every conversation seems to turn into a discussion about AI, regardless if you’re at a bake sale or just about anywhere. I’ve had conversations about AI, and I haven’t really pushed them there. Everybody’s talking about it. So, I think there’s definitely some hype. I don’t know if it’s overhyped. Listening to what you guys have to say in terms of how you’re using it, that certainly doesn’t seem to be the case. But from where you guys sit, what’s real and what’s noise regarding AI? What do you think is hype? What isn’t?

Chris McCarthy:

Sure. I think the way that AI is communicated to the West is a lot different than in China, in Japan, and in these Nordic countries. From a development standpoint, our idea of AI, I think, has been molded by negative conversations. And there’s a lot of opportunity that we see in efficiency, but more so in economic gain. I think abundance is just around the corner. And I think there’s a lot of conversation about AI taking jobs. And I think that’s perhaps a narrative that’s not quite as true as the operator would say. I can see business owners becoming four or five, ten times more productive and being able to grow their footprint. These things are really good at summarizing and taking unstructured data and synthesizing it into a usable format. And I think I use this terminology with a lot of my clients. It’s like taking a trike to a fighter jet with the tools that you use at work. And with that productivity gain and the extra intelligence, that frees up time for the human element, for strategic, creative, what makes us unique and authentic in our purpose in either the workplace or within our kind of passions in life. It frees us up to use this organic intelligence that we have.

Anthony Catalano:

I think Chris is… I mean, that’s an excellent, excellent point. And I’ll give you an example. Yesterday, I wanted to create a presentation to revert back to a client. And the presentation was relatively complex. I had to assimilate a lot of different service lines. A lot of stuff was going on with that. And that presentation normally would have taken me five or six hours. And as we all know, making a PowerPoint presentation is probably one of the worst things to do. Some people love it, not me. I do not like it. But you’re kind of stuck in the monotony of moving a cube around and putting text in it and filling in. You’re truly trying to tell a story through this cumbersome process. And instead of doing that, I said, here’s the notes from the meeting, and this is our internal Copilot assistant that we run at RSM, here’s the meeting notes. Here’s all the relevant emails. I dumped all that information. I said, so this is your context. I need you to make a slide presentation that’s ten slides long that’s going to articulate all the points that the clients express concern about and create it in this color scheme and make all these changes. So, this thing comes back with something that’s ninety-nine percent of the way there on the first pass, right? And then I get to go back, and to Chris’s point, instead of worrying about the monotony of them being exhausted after six hours of moving boxes around, now I can think of, okay, did this really solve the client’s problem? Or what other perspective do I need to bring here that’s unique? Because this really took the table stakes and put it on the page, right? Now it’s like, what’s the next evolution of that? And my wife has told me this multiple times. She said, since you started engaging with AI the way that you have, your creativity has exploded. And that is something that I don’t think people talk about a lot. Another really good point that Chris made and that you brought up with your question is that it’s going to take jobs, everybody’s going to be broke. I don’t necessarily agree with that. I think there’s extremes on both sides, but I took a class at MIT three years ago, and the class essentially said that instead of the income curve looking like this bell curve as it is today, it’s going to start to become more like a barbell. And the people that are in blue-collar jobs, physical labor, their income’s not going anywhere. They’ll be just fine. This middle part where you have the baseline thinkers, right? That’s going to flatten out, and we may not have such a need for it. And then the people that are doing the deep thinking are going to be just fine. And I can actually a hundred percent verify that the work that I had to do, I would normally delegate to an associate or a senior associate at my firm. Now the need for those people is not the same, right? We don’t need associates, senior associates, managers, because the workhorses have been taken over by the AI. But that will allow them to move forward in their career much more quickly. And it becomes a responsibility of us as a society and the businesses that we run to take them and elevate them more rapidly to the next phase of their career where they don’t have to do those things and they can become the thinkers. So, I think it helps people just as much as it could potentially hurt people. 

Jim Ickes:

I think it’s a great point, both of you. I often say that it liberates us from the monotony and the drudgery of work. It allows us to do the things that really add value. That’s bringing people together, spending more money on getting to places where you can do business and shake hands and connect, because those are really the activities that get deals done and get business done and move it forward. So, I know it’s somewhat maybe overly optimistic, and I’m sure there’s going to be some downside to it. I think people will be phased out who aren’t necessarily buying in. But we shall see, that’s for sure. Since you guys are both really close to the technology and using it in obviously extraordinary ways that we’ve already discussed, how is the technology capable today of things that maybe would have seemed like science fiction three years ago? Blow our minds. 

Chris McCarthy:

I think as AI uses language, it can take documentation or data and really make valuable insights from large sets of information. So, Jim, to your point, or to your question rather, what’s the mind-blowing thing? I think being able to take large amounts of unstructured data and put it into a logical format. I agree with that based on my own usage.

Jim Ickes:

How about you, Anthony?

Anthony Catalano:

I would somewhat go back to the example that I gave you with the sixteen agents. Those agents do work and hand work off to each other. I can give them a task and go to sleep at night and I wake up and they’ve been working all night on something to prepare a deliverable for me. Three years ago, AI just answered questions for us. Today, it takes action. You’re going to start to see companies that are AI-first companies, where you have a founder and a co-founder, and they have twenty people that work for them that aren’t real people. As of last week, there was a big conference in Las Vegas, it was actually a couple of weeks ago, I apologize, called Identiverse. And they estimated that the agentic to human identity ratio is anywhere between seventeen to one and eighty to one. That really speaks to the fact that we have a lot of robots or non-human identities that are operating within environments that are starting to do things for us. You’ll see entire companies that are set up with no real humans running things, or maybe just a couple of people running things, that are incredibly efficient, that are incredibly market forward. And again, it’s unlocking the creativity of the human mind. 

Jim Ickes:

It’s an exciting world we’re living in, fellas. So, let’s get back to your day to day and the practical side of actually running a business that involves AI like you do, Chris. So, when a client tells you we’re using AI, so in 2026 now, let’s say before the engagement and you go in and you see how they’re using it, what does that really mean to you?

Chris McCarthy:

With a lot of the business systems, especially on the Microsoft stack, they’ve been kind of carving out the space for AI accessibility to your data. And that includes Copilot and some of the frontier models. But if you think about platforms and the biggest players with the frontier models, a lot of people are showing interest and perhaps using it without rules in a business context. And we’re finding that people use Gemini and ChatGPT and Claude kind of on their own accord. And you see these numbers of monthly active users for ChatGPT and Claude and Gemini. Sometimes users forget that AI is built into YouTube and Gmail and Outlook. And these are not necessarily new technologies. They’re just updates, right? You’re already on these platforms. The technology is just kind of adapting into pre-existing systems. And I think individuals are using AI in a shadow kind of context. But as it relates to business use, it’s either all in or kind of tiptoeing around the concept of AI. So, it’s kind of hot or cold, right?

Jim Ickes:

So, what I’m hearing is there’s a lot of shadow AI use going on in a lot of the organizations you’re going into. 

Chris McCarthy:

That’s correct, yeah.

 Jim Ickes:

So, Anthony, from a more enterprise perspective and what you’re dealing with, when you go in and you’re doing some due diligence, what are you seeing in terms of the use?

Anthony Catalano:

Everybody’s using AI. I think we all agree to that. The questions that I ask them are pretty simple. What’s running, what data touches it, and who’s responsible for that data? And ninety-nine point nine percent of the time that answer is we don’t know, which is terrifying. And that’s just in the context of how the business is using AI. If you have an employee that’s dumping company spreadsheets into that, and somehow that gets compromised or their identity gets compromised, now a bad actor has access to very critical intellectual property. So, there’s a lot of different ways that that can cause problems. But organizations really need to be able to answer those questions. And it isn’t negligence, it’s that AI is driving this high level of innovation and that they’re just not keeping up with it. I wrote an article on LinkedIn about a month ago, and I was saying that seventy-five percent of organizations are either hiring or seek to hire a chief AI officer. And that’s because AI is not a risk problem or a revenue problem or a governance problem. It’s all of them. And you can’t assign that role or that responsibility to one of those people on those three pillars because it’s an overall company problem. And it really has necessitated creation of an entirely new job role. And in an emergent market, we really don’t have chief AI executives, right? Like they haven’t been around. We’re hiring people that sometimes don’t have college degrees but are really good with AI. Is that the right person? Organizations really need to focus on shaping their governance structure around AI and then finding the right person to enable that. And that is not an easy fit. It’s a mix between risk management, governance, and revenue generation, which is ultimately the point of AI. We want it to help us become more efficient and have the companies run either leaner or make more money. And again, that is a very complex job role to fill.

Chris McCarthy:

Anthony, I love what you said there. In our diagnostic process at OB.1, we find the champions within a business. Those that are ready to move the culture forward into this new technological era and have process-driven ideas to further the efficiency within that business. But to your point, identifying champions and enabling leadership. A lot of businesses need to encourage AI use, but with rules. The foundation of OB.1 is rules before tools.

 Anthony Catalano:

I love that phrase so much. I wish I could find a way to steal it. It’s so applicable in this scenario.

Chris McCarthy:

It is, it is. And the way that you govern AI is with specific rules and those gotcha lists. But the champion point was excellent. And that’s something we strive for to enable those people. 

Anthony Catalano:

And I think you say rules before tools. I love it. I can’t come up with anything even close to it, but I would just say in aggregate, what we see with AI is an ownership gap. And that’s a corporate ownership. Who’s responsible for it? Who’s administering it? Who’s creating the ROI with it? Accountability. Accountability, yeah. But I say the ownership gap is what we really look towards. 

Jim Ickes:

Well, that’s a perfect segue because this morning, the Washington Post had an article about how Congress is using it without really any guardrails or any rules. So, a lot of shadow AI use going on in the nation’s capital, which I think suggests that perhaps there’s no regulation coming soon. This is an area that we need to get our arms around. Much the same as I’ve always thought with cybersecurity and privacy, there needs to be a national rubric as opposed to this industry-by-industry approach that we have now. Now with the rise of AI, I think that’s even more of an issue. I’ve written recently about AI-driven malware, which I think is a major problem. So, in the absence of strong regulation that gives companies guidance on what to do, how are the two of you operating in an environment where there aren’t necessarily any rules on, I guess, a macro level?

Anthony Catalano:

Yeah, I mean, Congress’s bipartisan bumbling ineptitude aside, the story isn’t embarrassing because the staffer used AI, right? It’s embarrassing because nobody had a rule about it, right? Nobody checked and nobody owned it. The AI did exactly what it was supposed to do. It’s the governance around it, right? So, if we’re having decisions made at the highest level of our government without any form of governance around how AI is used safely or otherwise, we have a problem, right? And I would agree with your observation. I think a national law on the use of AI and its ethical considerations should be enacted. Now, the National Institute of Standards and Technology, NIST, has a risk management framework for AI that they’ve developed that organizations are loosely following, but that’s generally regarded more as a risk management framework within organizations and less as a national law or set of laws. So, I completely agree with your observation. I think we need to do way more there.

Jim Ickes:

And in developing our practice, we actually went to NIST to establish our framework that we’re using for our clients. So, it’s good that we have these other larger agencies that are kind of filling the void. But in my experience, unless there’s a law telling people to do things, they avoid it, look for the least expensive way to handle it, or turn a blind eye to the shadow AI approach that’s going on in the organization.

Anthony Catalano:

It brings up a really good point, right? Every business owner that hears that says, okay, it went into public record on Capitol Hill. What’s going into their proposals, their client emails, and their contacts? If Congress doesn’t know, how does a business owner know that? And how are they monitoring and governing that? Congress got caught because their output is public. Private organizations are not, unless you’re publicly traded. And even then it isn’t. So, I would say it is likely that a lot of businesses already have this exact same problem, and they just don’t know it yet. EY and several other large public accounting firms have already been caught putting out client reports and thought leadership that had massive amounts of AI hallucination contained in it. So, you’re right on point. It’s a problem.

Chris McCarthy:

The frontier models have kind of a contract. If you’re in a team environment, you sign a data retention policy and you think about sensitivity to this data. On public models like free ChatGPT or Claude, that information is used to train the models. As you graduate into an organizational account for a small business, your data is kind of protected under that cost. And there’s a thirty-day data retention policy for most of these frontier models just in case something breaks, but they’re not using it to train and it’s not public-facing. So, you think about IP, the exposure of that type of data. 

Jim Ickes:

Anthony, because I know you do a lot of M&A due diligence work. So I guess often you get four to six weeks before a deal closes. Can you even find shadow AI use going on in that window? Or how do you go about sussing it out?

Anthony Catalano:

We can understand what tools they have and who’s paying for those tools, but not who’s using those tools in that timeframe. So, I was talking to one of the heads of a major insurance carrier, a top three insurance carrier, about this exact problem. I said, how are we getting ahead of this AI governance gap? And he’s like, well, you know, we’re trying to find it in diligence. And I go, hey, man, listen, like full transparency. When we’re doing diligence, we get a small peek under the hood. Like they lift it two inches. We look, and then they slam the hood down, because they don’t want to give access. Historically, diligence engagements have very limited access. And then post close, you open the hood all the way and you’re like, oh, oh, dear. There’s a bird’s nest. The engine’s on fire. It has no oil. Like we’re not going anywhere. You generally find a lot post-close engagement. But what I said in the beginning: we know who pays for it, we don’t always know who’s using it. That’s a big problem. The shadow AI is real. We don’t know if somebody has a Claude subscription that they put on their own personal credit card that they’re then dumping spreadsheets into. Something to note is that not all of these LLMs have zero data retention enabled. So, you may put data into that LLM, and they may, by the contract that you sign when you click through it at light speed, have full access to that data afterwards to compute on it, to store it, to train additional models. And now you’ve just released pretty sensitive information into the public domain. And this has actually come out. We saw this happen with Claude a couple of months ago where the chat logs were stored publicly and you could search those chat logs and open them and see what was in them. It’s a problem during M&A, I am not confident with AI as an M&A process today. 

Jim Ickes:

So, we have the curious employee, employees using a lot of shadow AI. How do we address it? I guess phase one, if, let’s say, a company wants to address the fact that they want an enterprise approach to their AI, as you advocate for, Chris, rules before tools, how do we go in and establish rules before tools as a first step?

Chris McCarthy:

Yeah, having stakeholder conversations and getting a state of the union of that business in that current moment. Who’s contributing? Who’s going to own certain operational swim lanes? And who’s ahead on the AI totem pole? We try and establish the people first, and then dig into the business. We kind of synthesize the data, whether it’s structured or document or financials or HR manuals. And we kind of pull all of that information together to get a good meta view of that business. And then from there, we identify the regulatory. For example, a food manufacturer. They have recipes that are sensitive. We tell the AI that any time that word or trigger is present in a chat log, to stop the user and alert the action that you could take, or the user could take. And as it relates to setting up the rules, I think starting with leadership and establishing the transformation equation. So, culture, ROI, budget, and timelines. And understand the talent underneath the hood of the business. We try and set up rules to manage each operational swim lane, such as purchasing. We don’t want purchase orders and our pricing showing up in chat logs. There are specific don’ts, rules to tell the AI in plain language what to do and what not to do. Every business has different rules and regulatory frameworks. Generally, it’s around data and data sensitivity, but there’s also governing bodies that establish those rules. And our AI that helps diagnose businesses is kind of trained up on those regulatory statutes or larger form factor rules. 

Anthony Catalano:

And it’s interesting. I think one of the risks associated with that specifically is that you were telling the tool how it should be acting, right? And the tool is almost its own self-contained thing, right? And we’ve seen the tools get a little lippy sometimes. We’ve seen the tools kind of make decisions that are questionable. In cybersecurity, the lens that I look through, we have tools, we have rules. First, using your analogy, which I love, we have the rules, this is how we’re going to do things, and then we have the technical implementation of those tools. We look at it as administrative rules or controls, and then we look at it as technical controls or rules. The safeguards are that you can make a rule that says we’re going to use multi-factor authentication everywhere, and then you have a tool that rolls that out. In AI, we say we’re going to make sure the AI doesn’t do this. And there’s not a lot of tools that are integrating with the AI systems today that can actually enforce those rules or double-check or audit those rules. So, they’re almost these self-contained entities that have massive capability, and it’s hard for us to check those. To give you an example, the Claude Mythos model. I was fortunate enough to have some exposure to that. It will not only find zero-day vulnerabilities and find new exploits, ways to break the systems, but then it’ll come back to you and say, hey, I would like to fix it. And they call it the self-healing architecture. Normally that would be like, hey, that’s the IT person’s birthday wish. But if you think about it, what if that finds a zero-day vulnerability and fixes it or creates a new exploit or if that system goes rogue? Now we have the fox, as they say, watching the hen house. And that’s not good. And that’s the kind of stuff that worries me, is that you don’t have these technical validations to help check against the rules that are being created within the AI systems. That’s scary. How do we fix it? Ownership. I keep going back to that. We need ownership and governance of these AI systems, and they should have finite limits on what they’re able to do and not able to do. And that needs enforcement through not only their own inherent systems, but a double check. When you perform an audit, it’s double checking the company’s internal work. So, we need more capability to do that with the AI systems.

Jim Ickes:

I understand. So using the AI to basically install the AI.

Anthony Catalano:

I would say there would be separate systems that are not the AI tool itself. We have systems that monitor networks, right? We have rules, we’ve told the systems how to act, and we said, route this traffic to this thing. If something happens and the traffic starts automatically rerouting, we have systems in cybersecurity that would check that and stop that, right? We don’t have that same capability for AI yet. And I think that will come, but we don’t have it right now. And given the exponential use of AI across almost every domain of a business, it does present a risk. How do we insure that risk? I don’t think we are. So, again, I had a very frank conversation this morning with an insurer because they were going through a diligence yesterday with a client. And I was sitting on the call and I was like, wow, I just don’t think you guys are hitting all the bases. And I called him today, and I said, look, we have emergent AI threats. We have AI social engineering attacks where these things can impersonate intonation, tone, pretty much every part of a person’s voice verbatim. I don’t know if you guys heard the story about this late last year. The CFO of Ferrari got a call from the CEO and he said, hey, I need you to initiate this wire transfer. It’s for an acquisition we’re doing, et cetera, et cetera. Had him on the phone for a pretty significant period of time. And the CFO said it was his exact voice, and they were conversing back and forth and he was fully convinced he was going to go initiate a multimillion dollar wire transfer. And right at the end, something happened and he got a little suspicious. He said, hey, what was the name of the book that you recommended to me yesterday? And there’s silence. And that was the only reason that they didn’t lose the money on that. So, you think about it. I mean, those are very, very sophisticated. Now, that was over a year ago. Since then, we see a huge uptick, specifically against private equity funds, venture capital, family offices, where they’re doing these highly sophisticated social engineering attacks. And that’s run by an AI system.

Chris McCarthy:

Yeah, for sure. I’ve experienced the same thing. A new hire, she was a finance director, made the same error, was convinced across multiple communications, and an AI actor knew that she was a new hire. There was a signal somewhere out there on the web, perhaps a job offering at that company, but an intense signal that initiated that kind of social engineering process. And I think business leaders need to be very aware of that. You think about the way that your digital footprint is out there on the internet. The AI agents can go out and target valuable people, and I think that’s a huge risk in managing your digital footprint that’s visible to the public.

 Anthony Catalano:

For years, we’ve spent billions of dollars training our employees how to spot fake phishing emails. And generally, what it is, is that the email address is just off or there’s something just off in the title or the vernacular is not quite right or there’s a link to click on or a PDF. That is no longer applicable. I mean, obviously, you will still see that activity because it’s volume-based manipulation. But what you see now is the attackers will break into the organization, leveraging AI to help augment their attack, or they just get in somehow. Usually, it’s through these phishing attacks. They will watch that user’s behavior. They use AI to catalog, hey, well, how does this person write emails? What time of day do they usually talk? Who do they talk to the most? What’s going on with the company? And they can essentially exactly impersonate that person, right? So, these emails are no longer just off, they’re dead on. And what they do is they come back and they’ll initiate fraudulent wire transfer, which is very quickly catching up to ransomware as the number one cybersecurity threat in the industry. And they initiate these fraudulent wire transfers. It bounces from bank to bank. They’re out. So, I tell people the best way to get ahead of that right now is to establish a strong relationship with your local FBI contact and your local Secret Service contact. And the Secret Service can stop wire transfers a couple of bounces out, probably farther out than you would anticipate is legal. But hey, we’re happy about it. So, establish those relationships. It’s a great way for organizations to protect themselves against exactly what we’re talking about right now.

Jim Ickes:

That’s a great recommendation. It’s interesting that you bring that up. We’re doing a lot of research on biometrics and a new terminology that came to me last week, voiceprint, that we have value in our voiceprint. And there’s actually some litigation out there related to that. There’s a law in Illinois, BIPA, I’m sure you’re familiar with that, about biometrics and a class action regarding this subject matter. So, I’m glad you brought that up. It’s very, very timely. We actually have an article coming out about voiceprints. So, Chris, you advocate for rules before tools. What does that mean in practice? And what does it also mean when we get tools before the rules?

Chris McCarthy:

I have experienced a series of failed ERP deployments. And over the last ten years in my career, I’ve seen what works and what doesn’t. And starting at square one and understanding the foundation of the business and who is responsible gives people agency within the business. And establishing the rules, it’s iterative. And like I said, the technology is moving so quickly that you have to be specific because the reasoning will sometimes supersede the rules. But it requires users to actually interact with the tools and understand the depth in which they can support their role or daily tasks. But Jim, to your point, leadership is responsible for establishing the core rules for AI agents and specific tasks. Generally, there’s an orchestrator that’s responsible for actions that the agents are taking. And the orchestrator is putting out rules to every single function. And Jim, to answer your question, it starts with a strong preamble to the business. You think about the way that AI touches data, and that’s really the main thing in my world, is establishing the foundation of how that data is used, how your business data creates value, how it’s used operationally in documents, but more so advancing the intelligence underneath the hood. 

Anthony Catalano:

I really see that as where most organizations miss the mark, because they don’t try to quantify the ROI of using AI. So what happens in the beginning is people say, let’s use AI, let’s see who does it best, and let’s go ahead and throw that business process in and make it work. And what ends up happening is X person came up with the coolest looking dashboard, and it does cool things, but it doesn’t really generate an actual strong ROI for the business. And we encourage organizations to look back and say, look, you implemented AI in three different places. The measured ROI is what from each of those things? And then you quantify that back against your spend. Were you able to hire three fewer employees to do that? Or did you spend one hundred and eighty thousand dollars in tokens to make an additional twenty thousand dollars in profit? It doesn’t make sense. So, a lot of organizations just don’t go through the rigor of that exercise. I think your organization has really been able to help people get a little bit better visibility on what that looks like. And I think that’s kind of the direction we need to move. What is the ROI on these things? Don’t just do AI to do AI. Do AI for a reason.

Jim Ickes:

That’s a great point. Not just because everybody’s doing it, but it’s in the media and we’re supposed to do it. Or turning a blind eye and letting your employees just use it. All of a sudden, Bob’s email has gotten a lot better, because they’re using it, and you’re going to turn your head. Anthony, you assess AI programs against the NIST frameworks, which we already mentioned, and you assess cybersecurity and AI risk together. What does that combination catch that a security audit alone misses?

Anthony Catalano:

So, what we find generally is that the governance surrounding the AI implementation is just not well equipped. We could do a risk assessment using whatever cybersecurity framework we do. We could say, hey, you’re missing the following three controls. You’re missing the following five controls. And a lot of cybersecurity professionals come back and say, we have high risk. And then the board goes, what are you talking about? High risk in terms of what? Like, well, we have a bunch of missing stuff. That doesn’t really provide a lot of perspective around it, especially for the executive team. Cybersecurity is historically terrible at framing things in a quantitative manner, in a manner that the boards are willing to consume. When you look at the risks of AI, it’s not only a risk of the technology implementation, it’s a risk to the businesses from a disruption perspective, from a potential loss of intellectual property. And it also has, as we’ve seen, runaway costs that can be associated with it as well.

Jim Ickes:

Anthony, you assess AI programs against the NIST framework, cybersecurity and AI risk together. What does that combination catch that a security audit alone may miss? 

Anthony Catalano:

Yeah, that’s a great question, Jim. And I actually had a pre-canned response because I want to make sure I nailed it. So, a security audit asks, can somebody break in? An AI assessment asks a completely different question entirely. What happens when it works exactly as it was designed? Security assumes an attacker. AI risk, on the other side, includes the scenario where nobody attacks you at all. The system does precisely what you built it to do, and the outcome is still wrong. It denies the wrong person, exposes the wrong record, makes a decision that no human reviewed. A firewall, on the other end, has no opinion about that whatsoever. So it really comes back to governance, privilege, and accountability, which is more in your realm than mine.

Chris McCarthy:

That’s right. It certainly is. And context too, right? Having a log of what happened and how to be better or prevent it. And keeping logs of activities is something that makes my AI strong because it goes back and learns from the mistakes or the events in an attacking case.

Jim Ickes:

So, you brought up privilege. I think it’s pretty clear. And what we’re seeing is that if an executive drops a legal strategy into a chatbot or information from their lawyer into the chatbot to better understand it, then we’re wiping privilege in that context. So, we have to be real mindful of that. And that’s why we have the importance of governance that we’ve mentioned many times and really assigning accountability and ownership around this technology. For years, a company could carry gaps with their governance and the tools that they’re using. So they don’t have the necessary rules before the tools. When does the bill come due? 

Anthony Catalano:

There’s an organization out there in the last month that had an API key stolen out of a homegrown application that they built, which is not atypical. That API key was stolen. And what the API key does is it’s basically a credit card to Claude or Gemini or to OpenAI to say, this is how many credits we have available. Go ahead and use it. The attacker stole that API key and they ran up almost a five million dollar bill. And what happened was the organization found out and said, what the heck happened here? Oh, it was stolen. Okay, that’s a cyber event. They went to the insurance broker. The insurance broker said, this is not how your coverage is written. This particular scenario is not covered. They thought it might have been in the E&O policy, reps and warranties, cyber liability. That particular scenario has not been fully vetted yet. Since then, we’ve had five more of those incidents where stolen API keys were used to run up the tab. If you go to Anthropic and say, hey, somebody stole our API key, they don’t care, because the credit usage has happened. That data center computation power has been consumed in their eyes, and they don’t care. It’s not their problem, which kicks it back to the company. Well, the company doesn’t want to absorb a five million dollar bill. So they go to their insurance carrier. If their insurance carrier can’t fully transfer that risk through their written policy, it’s a five million dollar loss. So you start to see these cases. And by the way, this has happened a multitude of times since then, and it’s been happening for months. But now it’s starting to get a lot of attention because if you run off a five million dollar bill and steal that much money, it’s a big deal. It’s almost as bad as a stolen wire transfer. Might even be worse. 

Jim Ickes:

So, in that context, what can we do to avoid that happening?

Chris McCarthy:

Well, I think technically you put people in a sandbox to be able to develop tools that they can use in their daily workflows. You want to enable this, but safely, right? Vibe coding is something that I think should be encouraged. However, without proper architecture around that development, you’re at risk. Setting up the playing field to allow people to grow with AI, I think, is critical. And that’s an architectural decision. And also, setting up the proper connectors. There are certain roles that don’t need access to connectors. These are MCP servers. And essentially, it’s a list of tools that the AI can use on any given data set. Having rules to manage those MCP connections is, I think, a really good starting point. You don’t want the sales AI going over into finance, right? So, you set up a specific tool call for that department. It can’t go into QuickBooks. It can’t go into Stripe.

Anthony Catalano:

I think you bring up a really interesting point in that with the vibe coding. Not everybody knows what vibe coding is. Vibe coding is essentially you’re interacting with the AI system and telling it what to build. And it has enabled people to do things that they have never been able to do before. I tried coding one time, twenty years ago, and I was like, this is the worst possible thing of my life. You’re essentially instructing a machine what to do at the most basic levels. That doesn’t happen anymore. I was with my wife, we were watching Real Housewives, we got bored. I said, let’s code up that clinical trial system that you were talking about. She works in pre-market clinical FDA trials. And there’s no system right now that takes a pre-market clinical trial from stem to stern. And through that whole process, there’s governance, there’s reporting to the FDA, there’s reporting to the sponsors, reporting to the medical facilities. This is an extremely complex process. And today it’s managed by project managers. I’m not exaggerating. We said, let’s do this. We had to go run an errand. We were in the car, and I was talking to my AI agent and I was coding up an entire platform that she can use for an FDA pre-market clinical trial while we were in the car. And by the time we got home, we had a full application. Now, it wasn’t complete and needed a lot of work, but we had something that was one hundred percent there. If she took that to a company and said, hey, we have this system, do you want to use it? Some companies might say yes. That platform is not securely coded. I guarantee it. So, people build these systems and we’ve enabled people to be wonderfully productive. But the parameters and the controls and the security around that are not properly embedded with the process of building things in AI. And that inherently is the risk. You said rules before tools. The rules should be you can build whatever you want. But before that goes from development to a production environment, it has to have appropriate security testing per our rules. And what happens now is people are just bypassing that entire process because they got something new and cool and they want to implement it in an organization. So, again, going back to your rules before tools, you couldn’t have a better way to put it.

Chris McCarthy:

Yeah, having human-in-the-loop checkpoints as you bring forward a workflow or tool that you’ve developed in kind of your own account. I think giving people the opportunity to build, but keeping it safe in a Docker container or AWS Bedrock container. You think about the environment and enabling people architecturally.

Jim Ickes:

Fantastic. So, this is for the listeners out there. Say the listener runs a mid-sized company and heard this on a Tuesday drive. Give them the sequence. Evaluate the risk. Price the return. Implement with rules already in place. Then govern it as it runs. One step from each of you.

Anthony Catalano:

So, I think you would be, Chris, on the front end of that. What is the use case? And then how do we build it? And how do we build it securely? I think that falls in here. And then we would look at what is the risk to the organization, the downstream impact, and how do you protect it? I would encourage any listener that’s using AI today to call their insurance company based upon the story I told earlier and say, hey, if we had a stolen API key, or if intellectual property got leaked, or customer information got leaked through one of our AI assistants, would we be covered? And the insurance carrier is probably going to say, let me get back to you on that. And  the next thing you should do is go to your attorney and say, hey, attorney, under this particular scenario, what is our exposure? And I don’t think organizations lean on their on the legal teams enough to fully understand the implications of this yet. So, those are two calls that I’d make through that process that I think are really important.

Jim Ickes:

I think somebody’s going to have to give me a box of Kleenexes. I’m getting teared up over bringing the lawyers in. I appreciate it. And Anthony was not paid to say that.

Chris McCarthy:

No. Anything to add to that? I think having curiosity and dreaming big. I think having a goal in mind and then working backwards is how I’ve been successful with AI. Kind of start with a vision and then build information and context around that dream or that vision. Over the last year and a half since we started OB.1, or since I founded OB.1, I had a vision to help steward this technology forward for this city. This city saved my life in a lot of ways. The Cleveland Clinic and our health systems here are amazing. But making dreams reality and using what good looks like as a marker. And you think about learning and teaching the AI all the domain-specific things to help achieve what good looks like.

Anthony Catalano:

It’s interesting. I had a conversation one time with a CEO, and I said, what would you do if I gave you ten AI agents? He kind of had this blank stare, like, I have no idea what you’re talking about. So, if you reframe it, you say, what would you do if I gave you ten more employees that had any skill set that you needed? That will spark a conversation. To your point, what is the vision? What does the organization need? You have to work backwards and you say, look, I can go back five steps and I can get to the point where I can create the application that does the things that I need, or I can create the agent that does the work that I need done. You have to go farther back than that. And I think that’s where your company really comes into play. What is the framework for how we’re going to design this entire process? How are we going to build it so it fits the function? How is it going to produce an ROI for us? I think a lot of people kind of take the shortcut of, hey, we have something that’s working, but you have to go a lot farther back in the stack than you would anticipate and really dig in to do this properly. Build the foundation.

Jim Ickes:

Well, gentlemen, I really appreciate you both being here today. I think I learned a lot. And I think our listeners are going to learn a lot from the conversation. And I wish you both lots of good fortune in the future as you really help build out this industry in many ways. It’s an interesting time to be alive. I think it’s a great time to be alive. I think we’re realizing a lot of the power of computers. At least for me, being born in the seventies and really a child of the nineties, we’re realizing that future that we all thought we were going to have as a result of computers. So, truly an honor to have you both and hopefully we can do this again. And that wraps up another episode of Shoveling Smoke. This podcast is available on frantzward.com as well as Spotify and Apple Podcasts. Shoveling Smoke is a production of Evergreen Podcasts. Our producer and audio engineer is Sean Rule-Hoffman. For more information, please visit our website at frantzward.com. Another disclaimer, because I am a lawyer. This podcast is provided for educational purposes. It does not constitute legal advice and is not intended to establish an attorney-client relationship, nor is it intended to suggest or establish standards of care applicable to particular lawyers in any given situation. Prior results do not guarantee a similar outcome. Any views, opinions, or comments made by any external guest speaker should not be attributed to Frantz Ward or its individual lawyers.