Hiding Secrets in Plain Sight: A New Way to Keep Private Data Safe When Asking AI Questions

Adam Firestone
Editor-in-Chief   United States Cybersecurity Magazine | SIX3RO

Imagine a doctor who needs to quickly retrieve a summary of a patient’s lab results and treatment history. Or a financial analyst who wants an AI to calculate risk based on a company’s internal reports. Or a lawyer reviewing contract clauses against a firm’s private case files. In each case the AI produces a useful answer only because it first reaches into a private collection of documents, pulls out the relevant pages, and then combines those pages with the original question. The combined package travels across the internet to a large external AI service run by a major technology company. That service sees every name, date, dollar amount, medical code, and contract term. Even when the person asking the question is fully authorized to view the material, the external service is not. Real incidents have already shown that such data can be retained, examined, or leaked.

This pattern, often called retrieval-augmented generation (RAG), is now common because it makes AI answers far more accurate and current than answers drawn only from the model’s general training. The process begins when a user types a question. A search component looks through local files and returns the most relevant passages. Those passages plus the question are packaged together and sent to an external generator such as GPT-5 or Claude-4. The generator reads the package and writes a reply. The convenience is obvious; the privacy exposure is equally obvious.

Running the entire process on local hardware would eliminate the exposure. However, the strongest available models require clusters of specialized graphics processors whose purchase price alone can exceed tens of thousands of dollars per chip, and several chips are usually needed. Electricity, cooling, network infrastructure, and continuous maintenance add further cost. Smaller models that fit on ordinary computers produce noticeably less complete and less accurate answers. As a result most organizations continue to rely on external services and hope the transmitted data remain confidential. Hope, unfortunately, is not a plan.

In a paper titled “Privacy-Preserving RAG by Concealing Sensitive Information from External LLMs,” researchers Saleh Almohaimeed, Saad Almohaimeed, Mousa Jarir, Fahad Alotaibi, and Khalid A. Alobaid of King Saud University develop and explain a practical middle path. Their framework, called the Sensitive Entity Alias Generator or SEAG, inserts a small local model between the organization’s private documents and the external service. The local model, roughly three to four billion parameters in size, reads the question and the retrieved passages, identifies all sensitive pieces of information, and replaces each of them with a carefully chosen stand-in. The external service therefore receives only the sanitized version. After the external service returns its reply, the local system restores the original values so the user sees a correct answers that never left the organization’s control.

The stand-in process is designed to preserve meaning and internal consistency. If a document mentions both a country and its capital, the replacements must still form a matching country-and-capital pair. If a person’s full name, shortened name, and possessive form all appear, the stand-ins remain consistent across every variation. The same replacement is applied wherever a given fact occurs, whether in the question or in any of the documents. Because the relationships stay intact, the external service can still reason correctly even though it never sees the real details. Once the reply arrives, the system simply reverses the mapping.

The paper provides a concrete illustration. Suppose the question is “What is the gender pay gap in New Zealand?” A relevant document states that the gap is 4.2 percent. The local model might replace “New Zealand” with “Australia,” “gender pay gap” with “income gap,” and “4.2 percent” with “5 percent.” The external service answers “5 percent.” The local system then converts that answer back to “4.2 percent” before the user sees it. The private numbers and place names never left the organization.

To train and test the approach the researchers built two collections of documents and questions. The first collection contains 640 samples drawn from eight domains: news, legal, medical, finance, biology, chemistry, history, and résumés. It was used to teach three small models (Qwen-3, LLaMA-3.2, and Phi-4) how to locate sensitive items and invent consistent stand-ins. The second collection contains 600 samples from six domains: economic, education, cars, legal, finance, and business. Two members of the team wrote the questions by hand so that each question itself contained private entities and required an answer that would also contain private information. Most of the test domains differed from the training domains, allowing the researchers to check whether the small models would generalize.

The researchers measured success with across three parameters. First, they asked whether the external services could answer the original, unaltered questions correctly; almost every time the answer was yes, confirming that the material itself was clear. Second, they asked whether the external services still produced the correct numerical or named answer after the stand-ins had been applied; success here meant the real facts had been successfully hidden. Third, they asked whether, after the stand-ins were reversed, the user still received the correct final answer. Across every combination of local model and external service the user received the right answer more than 80 percent of the time. When the researchers examined every sensitive entity in the documents, not only those needed for the answer, the small models correctly replaced roughly three-quarters of them. Performance was strongest on documents filled with clear, standardized facts such as country names or simple percentages, and somewhat lower on documents that contained long, hierarchical legal titles or interlocking institutional names.

The findings demonstrated that privacy and utility can be distinguished. Organizations no longer face an all-or-nothing choice between weak local models and powerful external services that have access to everything. A local filter with modest computational and power requirements can sit in the middle, doing the hiding and restoring work while the heavy lifting of reasoning is still performed by the strongest available external models. The approach already works for the short-to-medium documents typical of many business, medical, legal, and financial settings.

The paper is candid about the approach’s limits. All tests used documents of roughly 300 words that contained about fifteen sensitive entities each. Longer documents or documents dense with interlocking private details have not yet been evaluated. In those settings the local model might miss an entity or produce less consistent stand-ins, either leaking information or slightly degrading the quality of the final answer. To address this, the researchers recommend future work that examines longer documents, larger numbers of sensitive entities, additional categories of private information, multilingual settings, and stronger replacement strategies capable of handling more complex contextual dependencies.

Those next steps point toward a practical way ahead. As the local models improve and the testing expands, the same filter can be applied to a wider range of real-world materials. Hospitals could query patient records, banks could analyze transaction histories, and law firms could review contracts without ever transmitting the underlying facts to an external provider. The external services would continue to supply high-quality reasoning; the private data would remain inside the organization’s own perimeter. The research provides both the concrete method and the empirical evidence that this separation is already achievable for many everyday uses of AI question-answering systems. lock

Adam Firestone

Tags: , ,