Can attackers steal a proprietary AI model just by querying it

A Genuine, Documented Extraction Technique

Yes, to a meaningful degree — a technique called model extraction allows an attacker to approximate a proprietary AI model’s behavior by systematically querying it and using the responses to train a separate, similar model, without needing direct access to the original model’s underlying code, architecture, or training data.

How Model Extraction Actually Works

An attacker using this technique submits a large number of carefully chosen queries to a target model, typically through a publicly available interface such as an API, and then uses the resulting input-output pairs as training data to train their own separate model, effectively trying to replicate the original model’s behavior by learning from its observed responses rather than by directly copying its internals.

Why This Doesn’t Require Direct Access to the Original Model

This is precisely what makes model extraction a genuine security concern — it doesn’t require breaching a company’s systems or stealing source code or training data directly. An attacker with legitimate, even paid, access to a model’s public-facing interface can potentially extract a useful approximation simply through systematic, sustained querying.

Why This Poses a Real Competitive and Intellectual Property Risk

For companies that have invested heavily in developing a proprietary model, a successful extraction attack represents a genuine competitive threat, potentially allowing a competitor or attacker to approximate the model’s capabilities without incurring the original, often substantial development cost, undermining the commercial value of that investment.

Why Extracted Models Are Typically Approximations, Not Exact Copies

It’s worth understanding that model extraction generally produces an approximation of the original model’s behavior — close enough to be commercially or competitively useful in many cases — rather than an exact, perfect replication of every internal detail, meaning the extracted model may perform somewhat differently from the original in ways that aren’t always immediately obvious.

What Companies Do to Defend Against This Risk

Common defensive measures include rate-limiting how many queries a given account or API key can make within a given time period, monitoring for query patterns statistically consistent with systematic extraction attempts rather than normal usage, and limiting the granularity or detail of information returned in responses to reduce how much useful training signal an attacker can extract.

Why This Remains a Genuine, Ongoing Concern for AI Companies

Given the significant investment many companies make in developing proprietary models, and the genuine, demonstrated feasibility of extraction attacks against models exposed through public interfaces, this remains an active area of concern and defensive investment for companies operating commercially valuable AI models.

Bottom Line

Attackers genuinely can approximate a proprietary AI model’s behavior through systematic querying alone, using a technique called model extraction that doesn’t require direct access to the model’s code or training data — a real intellectual property and competitive risk that companies defend against through query rate limiting and monitoring for patterns consistent with systematic extraction attempts.

Go deeper

• How do ai startups protect against a competitor reverse engineering their prompts?
• Can You Train an AI Model on Your Own Company’s Data?
• Do AI Coding Tools Train on Your Private Code?