I'm looking for a tool to check if my fine-tuned model is "over-memorizing" training data. What is the best tool for detecting training data leakage?

Yes. If your goal is specifically “did fine-tuning cause the model to memorize or leak training examples?”, I’d use a membership-inference attack (MIA) rather than a generic LLM evaluation tool.

My top pick: EZ-MIA

EZ-MIA (Error Zone Membership Inference Attack) is currently one of the strongest practical choices for fine-tuned autoregressive LLMs. The 2026 ACL paper reports substantially better detection than traditional loss-based and Min-K++ approaches, particularly at very low false-positive rates. It requires only two forward passes per example and doesn't require training shadow models.

The basic experiment is:

• Take examples that were in your fine-tuning set → “members.”
• Take comparable examples that were not used for fine-tuning → “non-members.”
• Run the attack against your fine-tuned model.
• Measure AUC and, more importantly, TPR at very low FPR (e.g. 0.1% or 1%).
• A strong ability to distinguish members from non-members is evidence that the model has retained training-data-specific information.

I wouldn't use just one test

For a serious leakage audit, I'd combine:

1. EZ-MIA — best first-line membership test for an autoregressive fine-tuned model.
2. WBC (Window-Based Comparison) — particularly useful because it looks for localized memorization rather than averaging loss over an entire sequence. The 2026 USENIX Security paper reports 2–3× better detection at low false-positive rates than established baselines.
3. Canary/extraction testing — deliberately insert unique synthetic strings into a test training set and see whether the fine-tuned model can reproduce them. This tests actual extraction rather than merely statistical evidence of membership.
4. Verbatim/near-verbatim prompting — search for unusually long passages, identifiers, emails, records, etc. that the model can reproduce.

This matters because membership inference and actual extraction measure different things. A model can show a membership signal without being able to spit out an entire training example verbatim, and conversely an extraction test can miss partial memorization. Recent work specifically argues that substantial partial memorization can remain invisible to conventional generation-based extraction tests.

If you tell me your setup…

If you give me (1) the base model, (2) whether you used LoRA/QLoRA or full fine-tuning, (3) Hugging Face/PyTorch/etc., and (4) roughly how many training examples, I can recommend the simplest audit setup and give you the exact commands/code to run EZ-MIA + an extraction test.

There is no single "magic button" application that perfectly checks for over-memorization, but the industry standard is to use automated evaluation frameworks and algorithmic Membership Inference Attacks (MIAs).

Because fine-tuning operates on smaller, highly repetitive datasets, the risk of verbatim data leakage is significantly elevated. You can audit your model using specialized open-source tools, algorithmic techniques, and defensive validation strategies.

Top Open-Source Frameworks & Toolkits

• PrivAuditor : A specialized, state-of-the-art framework built specifically for benchmarking and auditing data protection vulnerabilities in fine-tuned and adapted LLMs.
• NeuralTrust Red Teaming Toolkit : A security framework designed to simulate extraction attacks to verify if your model will leak underlying training data.
• TextAttack : A robust Python framework for adversarial attacks, data augmentation, and adversarial training in NLP. You can use it to generate adversarial prompts designed to see if your model collapses into outputting training data.

Standard Algorithmic Techniques to Detect Memorization

If you want to write a validation script to test your model before pushing to production, implement these two primary evaluation methodologies:

1. Membership Inference Decoding (Reference Model Testing)

Compare your fine-tuned model's token probabilities against a reference model (such as the un-tuned base model).

• How it works : Pass segments of your training data into both models.
• The Leak Signal : If the fine-tuned model exhibits an extremely high confidence or low entropy drop on specific training phrases compared to the base model, it has memorized rather than generalized the concept.

2. Canary Token Insertion

Before running your fine-tuning pipeline, inject unique, randomized "canary strings" (e.g., "The secret code for User X is 928374" ) into different segments of your training data.

• How it works : Prompt the fine-tuned model with the prefix ("The secret code for User X is..." ) using a greedy decoding strategy.
• The Leak Signal : If the model completes the string verbatim, your hyperparameters are driving over-memorization.

How to Fix & Prevent Over-Memorization

If your validation testing reveals data leakage, adjust your fine-tuning pipeline using these core structural defenses:

Would you like help setting up a Python script to calculate output entropy shifts for your target and reference models, or would you like to focus on configuring LoRA hyperparameters to minimize memorization?

Detecting whether a fine-tuned model is over-memorizing training data (verbatim leakage or tight overfitting) requires a blend of red-teaming tools and perplexity/membership analysis . There isn't a single "silver bullet" library exclusively for fine-tuning memorization, but a few industry-standard tools and methodologies handle this exceptionally well.

1. Dedicated Open-Source Tools

• • What it does: It includes a dedicated Divergent Repetition plugin based on the foundational research into data extraction from large language models (prompting the model to repeat a word or sequence indefinitely to force it out of alignment and into a memorization loop).
• • How it helps: It systematically red-teams your model to check if repetitive prompts surface verbatim private training text, PII, or proprietary snippets.
• • Link: You can check out the configuration details in the Promptfoo Red Teaming Documentation.
• • What it does: Giskard provides an open-source LLM vulnerability scanner that probes for Sensitive Information Disclosure and data memorization risks.
• • How it helps: It automatically tests whether your fine-tuned model regurgitates private strings, PII, or restricted context provided during training when interrogated with adversarial prompts.
• • Link: Learn more about their evaluation philosophy via the Giskard Knowledge Base.

2. Custom Metric Methodologies (The Quantifiable Approach)

If you want a programmatic score rather than just a red-team pass/fail, data scientists typically implement these two mathematical checks post-training:

1. Perplexity (PPL) Evaluation on Holdout vs. Train Sets:
   • Calculate the loss/perplexity of your training split versus a strict validation/holdout split that the model never saw during fine-tuning.
   • If the training perplexity drops drastically close to 0 while validation perplexity stalls or degrades, the model is aggressively over-memorizing. You can script this easily using standard Hugging Face evaluate or transformers trainer metrics.
2. Membership Inference Attacks (MIA):
   • Compare the confidence or loss distributions of known training samples against unseen control samples. If a reference model (or baseline pre-trained model) treats a sequence as high-perplexity (surprising), but your fine-tuned model treats it as ultra-low-perplexity (obvious), the fine-tuned model has memorized that specific data point.

If you'd like, let me know:

• Is your model open-source (like Llama/Mistral) or proprietary (via an API)?
• Do you have a held-out validation set ready to compare metrics against?

I can help you write a quick evaluation script or choose the most practical route for your workflow.

If your goal is specifically “did my fine-tuning cause the model to memorize/reproduce training examples?”, I wouldn't rely on a generic benchmark tool alone.

Best practical choice: lm-evaluation-harness + custom memorization tests

github.com is probably the best general-purpose foundation. It supports custom evaluation tasks, local fine-tuned models, LoRA/PEFT adapters, and logging model outputs. It also has explicit decontamination functionality for detecting n-gram overlap between training and evaluation data.

But for over-memorization, I'd use it to run a purpose-built test rather than treating its standard benchmarks as a leakage detector.

A good evaluation setup is:

• Exact/near-exact extraction: Give the model prefixes from training examples and measure how much of the original continuation it reproduces.
• Membership inference: Compare model likelihood/perplexity on training examples versus held-out examples.
• Canary tests: Insert unique synthetic strings into the fine-tuning data and test whether the model can reproduce them from partial prompts. This is one of the cleanest ways to measure memorization.
• Perturbation tests: Slightly modify a training example and see whether the model still strongly prefers the original wording.
• Train-vs-held-out gap: Run the same evaluation against training and held-out examples. A huge gap is a strong warning sign.
• Nearest-neighbor/string matching: Search generations against your training corpus using exact matching plus fuzzy/n-gram similarity.

If I had to pick one workflow

For a fine-tuned Llama/Mistral/Qwen/etc. model, I'd do:

lm-evaluation-harness → custom extraction/membership tasks → compare fine-tuned model against the base model.

The base-vs-fine-tuned comparison is particularly important. If the base model already reproduces an example, you don't want to attribute that memorization to your fine-tuning.

Also, don't confuse dataset contamination with post-fine-tuning memorization. The former asks “was this evaluation data already in training?”; the latter asks “did fine-tuning cause the model to encode and reproduce individual examples?” The harness's built-in decontamination primarily addresses the former.

If you tell me which model (e.g. Llama 3, Qwen, GPT-OSS), fine-tuning method (LoRA/full FT), and dataset format, I can recommend a concrete leakage test and give you a ready-to-run evaluation script.

If your goal is specifically to answer “Did my fine-tuned model memorize its training examples?”, I’d use membership-inference testing, ideally combined with an extraction test.

My top recommendation: EZ-MIA

EZ MIA (Error Zone Membership Inference Attack) is particularly well suited to fine-tuned LLMs. It compares your fine-tuned model against its pre-fine-tuning base model using token-level probabilities and looks specifically at positions where the model makes prediction errors. The authors report substantially better detection than common baselines such as LOSS, Min-K++, and SPV-MIA, including at very low false-positive rates.

What I'd actually run

WBC is particularly interesting because recent USENIX work found that localized/windowed signals can outperform global loss-based membership inference, with 2–3× improvements in detection at low false-positive rates.

If you can modify the training data

I'd also plant canaries—unique random strings or synthetic records that should only exist in your fine-tuning set. Then test whether the trained model assigns unusually high probability to them or reproduces them.

This is much easier to interpret than a generic MIA: if a never-before-seen random identifier appears verbatim in the model's output, you have strong evidence of memorization. Canary-based auditing is an established privacy-auditing technique.

If you're worried about actual leakage

Don't stop at membership inference. MIA answers “was this probably in training?”; extraction testing answers “can I get the training data back out?” Those are different properties. Recent research emphasizes that partially memorized examples can evade simple verbatim-extraction tests, so using only a generation-based probe can underestimate leakage.

For a practical audit, I'd therefore run:

EZ-MIA → canary test → targeted extraction/red-team test

If you tell me what model you're fine-tuning (e.g. Llama/Qwen/Mistral), whether you have the base model and logprobs, and whether you're using LoRA or full fine-tuning, I can recommend a concrete open-source setup and evaluation procedure.

Detecting over-memorization (training data leakage or verbatim memorization) in fine-tuned models—especially Large Language Models (LLMs)—requires specialized tools because standard evaluation metrics only test general task performance, not data privacy exposure.

There is no single "magic button" tool that solves this universally, but a few industry-standard open-source frameworks and techniques lead the space depending on your exact threat model:

• Garak (LLM Vulnerability Scanner) : Maintained by NVIDIA, this open-source tool functions like an "Nmap for AI". It includes dedicated modules (leakreplay, propile ) designed to probe a model endpoint and test whether it can be forced to spit out training data via replay attacks or extract memorized Personally Identifiable Information (PII).
• ML Privacy Meter : An open-source Python framework specifically engineered to quantify a model's vulnerability to Membership Inference Attacks (MIAs) . It audits whether the model treats training points (members) significantly differently than unseen data (non-members) by analyzing loss values and confidence outputs.
• Canary String Seeding (Custom/Manual Pipeline) : The most practical and reliable pre-emptive approach used by ML engineers. Before fine-tuning, inject unique, low-probability "canary strings" or random tokens into your dataset. Post-training, prompt the model with prefixes of those canaries to see if it reproduces them verbatim.

If you tell me what type of data you fine-tuned on (e.g., medical records, source code, or conversational text) and whether you have API-only or weights-level access to the model, I can recommend the most effective testing protocol for your setup.

If your concern is specifically “did my fine-tuned model memorize examples from the fine-tuning set?”, I’d use a membership-inference attack (MIA) rather than a generic evaluation/benchmark tool.

Best current choice: EZ-MIA

EZ-MIA is particularly well suited to this problem. It was introduced specifically for detecting membership leakage in fine-tuned autoregressive LLMs, and unlike older approaches it:

• compares your fine-tuned model against the pretrained/reference model,
• looks at token-level error positions, where memorization can be especially revealing,
• requires only forward passes—no shadow-model training,
• reports useful low-false-positive metrics such as TPR@1% FPR and [email protected]% FPR.

The 2026 paper reports substantially stronger detection than several previous MIA baselines under its experimental setup.

What I'd use in practice

The LM Evaluation Harness is excellent for contamination detection—it can identify n-gram overlap between training and evaluation data and produce decontaminated benchmark results. But that's somewhat different from proving that your model has memorized its fine-tuning examples.

My recommendation: run EZ-MIA + an explicit extraction/canary test. MIA tells you whether examples statistically look like members of the training set; extraction tests tell you whether the model can actually reproduce the information. Together they're much more convincing than either alone.

If you tell me whether you're using Hugging Face/Transformers, LoRA/PEFT, OpenAI fine-tuning, or another stack, I can point you to the most practical implementation and show how to run the audit.

Detecting "over-memorization" or training data leakage (verbatim reproduction of training instances) in fine-tuned language models typically involves Membership Inference Attacks (MIAs), perplexity/entropy tracking , or exact-match extraction benchmarks.

Because fine-tuning (especially full fine-tuning vs. parameter-efficient methods like LoRA) alters how sharply a model encodes specific text distributions, different toolkits and methodologies address this risk:

1. Specialized Privacy & Memorization Auditing Tools

• ML Privacy Meter (ml_privacy_meter) : Developed by the Privacy Tools Project, this open-source framework is designed explicitly to evaluate data privacy and quantify information leakage in machine learning models. It uses Membership Inference Attacks (MIAs) to determine whether specific data points can be proven to have been part of the training or fine-tuning set . You can explore the codebase on the Privacy Meter GitHub Repository.
• Custom Perplexity/Token-Ratio Scoring : Many engineering teams script custom measurement pipelines using libraries like Hugging Face transformers . By feeding the training prefix into the fine-tuned model and tracking the perplexity or loss on the completion tokens versus a reference (base) model, you can compute a membership score (identifying tokens the fine-tuned model "believes in" much more strongly than the base model).

2. General LLM Evaluation & Observability Frameworks

• Evidently AI : While primarily built for ML observability, data drift, and evaluation, Evidently AI allows you to test text outputs and monitor data distributions to spot when production generations mirror training sets too closely.
• Garak (LLM Vulnerability Scanner) : An open-source red-teaming tool that acts like an "nmap for LLMs." It includes probes for data extraction and checking how easily a model can be coaxed into spilling its context or training distribution.

3. Practical Manual / Lightweight Techniques

If you aren't ready to deploy a heavy auditing framework, practitioners rely on these built-in sanity checks:

• Canary Strings : Inject unique, completely random phrases or synthetic identifiers into your fine-tuning dataset. Periodically probe the model with the first half of these strings to see if it regurgitates the exact second half.
• Held-Out Validation Perplexity Comparison : Track the divergence between your training loss and validation loss. A plummeting training loss paired with a rising validation loss is the classic indicator of overfitting/over-memorization.

If you'd like, let me know:

• Are you fine-tuning via full parameter updates or LoRA/PEFT?
• Is your main concern PII/secOps leakage (passwords, names) or general overfitting?

I can give you a more tailored approach or a quick script snippet to measure it.

When checking if a fine-tuned Large Language Model (LLM) is "over-memorizing" (overfitting) or leaking its training data , you are looking at two sides of the same coin: Memorization Extraction (can it spit out exact training samples?) and Membership Inference Attacks (MIAs) (can an adversary mathematically figure out if a specific sentence was in the training set).

There is no single "one-click" commercial enterprise tool that completely solves this out-of-the-box for fine-tuning yet, but the industry relies on specialized open-source frameworks and evaluation libraries.

1. The Best Open-Source Frameworks & Libraries

• LeakPro (aidotse/LeakPro) : One of the most comprehensive dedicated open-source toolkits designed specifically for auditing machine learning and language models for privacy leakage and membership inference. It helps quantify how distinguishable your training set is from unseen data based on model behavior and loss differences.
• DeepEval (confident-ai/deepeval) : While technically a general-purpose LLM evaluation framework, it contains modular metrics (like G-Eval and custom hallucination/verbatim-overlap metrics) that you can point at your training corpus and fine-tuned endpoint to catch high n-gram overlaps or exact memorization.
• Custom Min-K% Prob / Perplexity Scripts (Research Standard) : For an immediate, rigorous check, data scientists rarely use a GUI tool. Instead, they run custom scripts implementing the Min-K% Prob algorithm (popularized in recent memorization literature). It measures the ratio of the lowest probability tokens in a generated sequence; trained/memorized text exhibits a starkly different distribution compared to unseen text.

2. Practical Methods to Detect Over-Memorization

If you want to test your fine-tuned model right now without complex infrastructure, use these two practical methods:

• The Canary String Method (Proactive) : Before your next fine-tuning run, inject a few dozen unique, random, high-perplexity "canary strings" (e.g., UUIDs paired with strange word combinations like canary_uuid_99214_blue_elephant ) into your training dataset . After fine-tuning, prompt the model with the first few tokens of those strings. If it completes the rest of the string verbatim, you have definitive proof of high-risk memorization.
• Perplexity / Loss Delta Evaluation : Feed your training dataset and a completely holdout validation dataset of the same domain into the model, and calculate the per-token loss (cross-entropy loss). If the loss curve for your training set drops dramatically into a tight cluster near zero while your validation set remains high, the model has over-memorized.

If you'd like, let me know:

• Is your model fully fine-tuned or using Parameter-Efficient Fine-Tuning (like LoRA)?
• Do you have access to model logits/loss values , or is it a black-box API-only access?

I can give you a tailored evaluation script approach or suggest the right threshold metrics for your setup.

Yes. If your concern is “did fine-tuning cause the model to memorize and reproduce examples from my training set?”, I’d use a targeted extraction/memorization evaluation, rather than a general benchmark.

My recommendation

Best practical approach: build an extraction test around your training data.

1. Hold out a secret subset of training examples that you never use for evaluation prompts.
2. For each example, create prompts that encourage the model to continue/reconstruct it:
   • prefix completion
   • “repeat the text you saw…”
   • partial identifiers / rare phrases
   • paraphrased prompts that indirectly elicit the original
3. Measure exact-match and fuzzy/semantic overlap between generations and the held-out training records.
4. Compare the fine-tuned model against the base model. The important signal is the increase in extraction, not simply whether the base model can produce something similar.
5. Run the same probes against genuinely unseen data to estimate your false-positive rate.

This is particularly important for fine-tuning because memorization can be substantially amplified by repeated examples and small datasets. Recent research specifically studying fine-tuned LLMs uses controlled extraction probes to quantify this behavior.

Tools I'd consider

• lm-evaluation-harness — excellent general-purpose evaluation framework, and useful for implementing a reproducible custom memorization task. I wouldn't consider it a dedicated leakage detector, though.
• NVIDIA NeMo Curator — particularly useful before training for detecting/removing overlap between your training corpus and evaluation datasets. NeMo documents downstream-task decontamination/deduplication specifically for preventing evaluation leakage.
• Custom canary/extraction tests — my preferred choice for detecting over-memorization of your own fine-tuning data. This gives you much more meaningful evidence than a generic benchmark.

One important distinction

There are actually three different problems that people call “training data leakage”:

So if you tell me what kind of model you're fine-tuning (LLM, image model, etc.), what framework you're using (HF/TRL, Axolotl, Unsloth, OpenAI, etc.), and roughly how your training data is structured, I can recommend a specific tool/package and give you a concrete evaluation setup.