synodic-ai RESEARCH
Research Desk · AI Governance · Audit Evidence

Evidence an Auditor Can Check: Training-Data Membership Tests and Verified Query Plans

2026-10-14

Two preprints posted on 2026-10-12 and 2026-10-13 turn AI-governance requirements into artifacts an outside reviewer can examine: one tests whether specific records were used to fine-tune a language model, the other attaches checked plans and concrete disagreements to model-generated database queries.

Governance requirements are only as strong as the evidence behind them

The inLoRA-MINT authors tie their work to data provenance and to compliance with emerging privacy and AI-governance regulations, which raise the question of which data trained a model. Financial model risk management, as the VeriPIE authors put it, emphasizes independent validation, transparency, and effective challenge of AI-generated computations. Both requirements are weak if the only evidence available is the developer's own account of what the model saw or what a generated computation does.

The two papers each produce a different kind of artifact that a reviewer can examine without taking the developer's word: a classifier verdict on whether a record was in a fine-tuning set, and a checked plan, a repair diagnostic, or a concrete database on which two candidate queries disagree.

inLoRA-MINT: testing whether a record was in the fine-tuning set

Mancera, Fierrez, Morales, Tolosana, DeAlcala, and Jurado apply the Membership Inference Test (MINT) to large language models fine-tuned with LoRA. The aim is to determine whether specific samples were used during training, in response to concerns about data privacy, leakage, and transparency. Their earlier LoRA-MINT framework relied exclusively on perplexity under a strict black-box setting.

The new approach uses intermediate model representations, including activations and parameter-dependent signals. These representations let a MINT classifier distinguish training from non-training samples without requiring full access to model parameters. The authors evaluate the method on four language models from different families, each fine-tuned on three datasets covering medical, mathematical, and academic reasoning tasks, and report consistent performance across architectures and domains, with scores between 95 and 98 percent and robustness to variations in model configurations and data distributions.

The authors position the method as a way to audit sensitive training data efficiently and to support data provenance and compliance with emerging privacy and AI-governance regulations. Their experiments cover LoRA-based fine-tuning only; they state that the methodology can be extended to other adaptation techniques and to other AI models, which is a claim for future work rather than a result.

VeriPIE: making a generated query reviewable before anyone relies on it

Jha, Song, Li, and Sejpal start from a specific failure of text-to-SQL systems: queries that agree on one database can still express different computations. A reviewer who checks only that a query returned the expected rows on a test database has not checked what the query computes.

VeriPIE is a neurosymbolic framework that combines language-model generation with symbolic checks. It ensembles direct SQL generation with QuerySketch, a structured plan language checked by a backend implemented in Lean, a theorem prover and programming language. Plan diagnostics guide repair before any SQL is generated, and execution agreement together with bounded searches for distinguishing databases informs which candidate query is selected. The authors are explicit about the limit: the checks expose malformed plans and hidden disagreement without proving agreement with the user's intent.

In single-run experiments on the complete Spider development set, VeriPIE reaches 72.05 percent test-suite accuracy (745 of 1,034 questions), against 65.67 percent (679) for the comparison system the authors report. Its two-query shortlist covers 78.43 percent of questions (811), against 74.66 percent (772) and 69.05 percent (714) for two prior systems. On BIRD it reaches 68.90 percent execution accuracy and 73.73 percent two-query coverage. The authors identify the central benefit as reviewable evidence about the generated computation: explicit plan checks, repair diagnostics, and concrete disagreements that support safer financial analysis.

What the two tools share, and where each stops

Both papers move part of the audit from trusting a description to examining an artifact. Neither turns the audit into proof. A membership score in the 95 to 98 percent range is strong for a classifier and still leaves a margin of error, so a verdict about one record is weaker evidence than a pattern across many records. A distinguishing database shows that two candidate queries compute different things; it does not say which one the user meant.

The limits are stated by the authors themselves, which is what makes the artifacts usable in a governance process: a reviewer can carry the stated limit into the conclusion instead of discovering it later. That is the practical difference between evidence and assurance.

What governance and risk teams can take from the two papers

Both are preprints, and the figures above are the authors' own, measured on their chosen models, datasets, and comparison systems. Read together, they show what a technically checkable audit artifact looks like, and where each method stops.

Sources