Security teams are overusing general purpose AI models for decisions they were never designed to make. This conversation explains why predictive security requires purpose built models, continuous retraining, and disciplined data science.
In this Brand Highlight, we talk with Michael Roytman, CTO of Empirical Security, about a problem many security teams quietly struggle with: using general purpose AI tools for decisions that demand precision, forecasting, and accountability.
Michael explains why large language models are often misapplied in security programs. LLMs excel at summarization, classification, and pattern extraction, but they are not designed to predict future outcomes like exploitation likelihood or operational risk. Treating them as universal problem solvers creates confidence gaps, not clarity.
At Empirical, the focus is on preventative security through purpose built modeling. That means probabilistic forecasting, enterprise specific risk models, and continuous retraining using real telemetry from security operations. Instead of relying on a single model or generic scoring system, Empirical applies ensembles of models tuned to specific tasks, from vulnerability exploitation probability to identifying malicious code patterns.
Michael also highlights why retraining matters as much as training. Threat conditions, environments, and attacker behavior change constantly. Models that are not continuously updated lose relevance quickly. Building that feedback loop across hundreds of customers is as much an engineering and operations challenge as it is a data science one.
The conversation reinforces a simple but often ignored idea: better security outcomes come from using the right tools for the right questions, not from chasing whatever AI technique happens to be popular. This episode offers a grounded perspective for leaders trying to separate signal from noise in AI driven security decision making.
Note: This story contains promotional content. Learn more.
GUEST
Michael Roytman, CTO of Empirical Security | On LinkedIn: https://www.linkedin.com/in/michael-roytman/
RESOURCES
Learn more about Empirical Security: https://www.empiricalsecurity.com/
LinkedIn Post: https://www.linkedin.com/posts/bellis_a-lot-of-people-are-talking-about-generative-activity-7394418706388402178-uZjB/
Are you interested in telling your story?
▶︎ Full Length Brand Story: https://www.studioc60.com/content-creation#full
▶︎ Brand Spotlight Story: https://www.studioc60.com/content-creation#spotlight
▶︎ Brand Highlight Story: https://www.studioc60.com/content-creation#highlight
Keywords: sean martin, michael roytman, ed beis, empirical security, cybersecurity, ai, machinelearning, vulnerability, risk, forecasting, brand story, brand marketing, marketing podcast, brand story podcast, brand spotlight
[00:00:00]
[00:00:24] Sean Martin: And hello everybody, this is Sean Martin, and you're very welcome to a quick brand highlight here on ITSP magazine, brought to you in partnership with, uh, studio C 60. And I'm thrilled to have Michael Roytman on from empirical security. Michael, how are you?
[00:00:42] Michael Roytman: Good. Good. How are you?
[00:00:45] Sean Martin: Doing great. Thanks. And, uh, so we, we have a mutual friend, obviously. Your, your, uh, coworker, ed Beis. Uh, I know I've known for a long time and I stay tuned to a lot of things on, uh, LinkedIn, uh, good or bad. And, uh, [00:01:00] saw a post from Ed that. Uh, piqued my interest. And, and, uh, so this, this is about, uh, that post talking about the, the broader world of AI and LLMs and, and security weaknesses, uh, within all that space.
And I wanted to learn more about what you guys are doing at Empirical and maybe a touch on that blog a little bit as well. Um, so first, a few words from you. Uh, who you are, your role at the company, and, uh, maybe the elevator pitch for empirical. And then we'll get into, uh, into the blog and what you guys are up to.
[00:01:32] Michael Roytman: Yeah, yeah. Uh, my name is Michael Bin. I'm the CTO at Empirical Security. Funny you mention Ed. He's my co-founder, CEO here, but I worked for him at Kenna Security for, uh, 10 years and then Cisco at three years. We essentially created the risk based vulnerability management. Market and product, uh, empirical is us going both broader and deeper with that data-driven Moneyball approach to vulnerability management and expanding it to [00:02:00] all preventative security.
So we build and maintain models ranging from open models like EPSS. We provide that to first.org. About a hundred vendors use that today. Um, we have a global model for vulnerability management. Issues, probabilities, forecast for vulnerability, exploitation, and then we build local models for our customers that help deal in a data-driven matter with application security, cloud security, vulnerability management, building probability ratings, risk ratings, security models that are specific to an enterprise.
That's kind of our new. Approach to making security a little more data driven. We deal entirety with the preventative side, but unlike, uh, the CE vendors out there, we actually grab data from telemetry sources like Sim CrowdStrike, Palo Alto, and use that to retrade the model as well. Um, so we're really excited for where this takes security.
We think that this is, uh, very broad preventative approach and we use. ML AI as tools [00:03:00] in a, in a huge bucket of tools. When I say a model, I mean actually an ensemble of models ranging from classifying GitHub repositories as malicious code to finding probabilities of exploitation, to looking at code and doing summarization to great features for ML models.
We, we kind of spend the gamut of data science.
[00:03:18] Sean Martin: So in in Ed's post, it's uh, he mentioned that it's the right tool for the right job. And so you might, which basically says to me purpose-built specific things, doing specific activities and analysis that they may then pull together to collect a bigger picture, uh, to help drive some better decisions. Um.
Talk to me about what, what you see a lot of companies doing with respect to, I dunno, grab a model, use the public model and, and get good enough, but may have hallucinations or, I don't know, some other challenges there. So what, what are you seeing and how does, how does what you offer kind of help close the
[00:03:57] Michael Roytman: Yeah, hall hallucinations is a very, uh, [00:04:00] 20, 23, 24 concern. It's gotten. So we ran a study, I actually published it in Forbes about six months ago, comparing just our free model EPSS to, which is a predictive deterministic model for predicting likelihood of exploitation in the next 30 days. Comparing that model to three L, lms, A, Google, LLM, and open I-L-L-L-M and philanthropic l LM at the same task.
So when you pick tasks for LLMs that are specific to. Supporting security decisions, issuing predictions about the future, they're terrible at it. They can't really predict the future. They can predict the next character in a sequence. They can auto complete a a text, or they can do a task in that sense, if it's a sequence of tasks.
But to predict. The weather tomorrow, they're just not built for that. There are a suite of models that are really good at predictions. XG Boost for predicting that probability, if you have a sequence of exploitation events is excellent. And [00:05:00] that's what we see continuously outperform all alums. So using the right tool for the job is not just important.
It actually guarantees efficiency and security if you're using. An LLM to predict the probability of a vulnerability being exploited. That's not gonna have a great result if you're using an LLM to summarize code and extract a feature or tell me, is this one of these four categories of potential exploitation types?
It's great at that. It's designed to summarize something into a smaller distillation by using the transformer architecture. So at Empirical, I used to be the Chief Data scientist at Kenna Security. Our third co-founder, Jay Jacobs, uh, was the chief data scientist at Enti Institute and one of the founders there, he was a data scientist at Verizon, DBAR, I think the first there.
Um, and so, you know, two thirds of the company as a data science background we're very particular about using the right model and the right tool for the task. Rather than picking something up off the shelf. And I think part of with that territory [00:06:00] comes the challenge of having to train models from scratch, which I think today a lot of security companies just aren't willing to do.
That's not their forte, that's not their wheelhouse. Um, we think of ourselves as a data science company that's operating on security data rather than a pure security.
[00:06:14] Sean Martin: Yeah, and uh, if anybody knows training it once is not enough.
[00:06:20] Michael Roytman: No, the re the retraining is actually the technical challenge for sure, especially if you think about hundreds of models and hundreds of customers, and then you're retraining each of those when the environment changes or the tooling changes. Um, but that is actually more of a terraform DevOps ML operations challenge than a data science challenge.
Ironically.
[00:06:42] Sean Martin: Yeah. Yeah. Well, no, no lack of challenges and, uh, yeah, it's a, it's a good post. I would encourage everybody to read it and, um. Certainly connect, uh, with Michael, if you're listening or watching, uh, this brand highlight here and I'll, I'll include links to the post, links to the blog, [00:07:00] uh, links to connect with Michael and, and Ed.
And, uh, yeah, why not Jay as well. What, what the heck? And, uh. Throw 'em in there. Michael. Thanks for, uh, thanks for taking time. Share this little, uh, highlight with us and everybody listening, watching. Stay tuned itsp magazine.com for more stories and, uh, if you have a story you wanna share, studio C 60 is a place to connect with us for that.
Thanks again, Michael.
[00:07:24] Michael Roytman: Thanks Sean. Appreciate you.
​