Introduction

Maintaining the reliability of machine learning models in production is a constant struggle for data scientists. Once a model is deployed, it often suffers from silent failure—performance degradation that goes unnoticed because ground truth labels are delayed or unavailable. NannyML, an open-source Python library with over 2,100 GitHub stars, solves this by allowing teams to estimate model performance without needing immediate access to targets. By bridging the gap between data drift and actual performance drops, NannyML ensures that ML models continue to deliver business value without the need for constant manual oversight.

What Is NannyML?

NannyML is an open-source Python library that estimates post-deployment model performance and detects data drift for tabular data. It is designed for data scientists and ML engineers who need to monitor their models in production without waiting for ground truth labels to arrive. The project is maintained by NannyML NV and is distributed under the Apache License 2.0.

The library is completely model-agnostic, meaning it can be integrated with any machine learning framework (such as scikit-learn, XGBoost, or PyTorch) as long as the data is tabular. It supports both binary and multiclass classification as well as regression tasks, providing a unified interface for monitoring the health of deployed models.

Why NannyML Matters

In most real-world ML applications, there is a significant time lag between making a prediction and knowing the actual outcome. For example, in credit scoring, it may take months or years to determine if a loan was actually repaid. This “delayed target” problem makes traditional performance monitoring impossible in real-time.

NannyML fills this gap by using novel algorithms to estimate performance metrics (like ROC AUC or RMSE) based on the distribution of model predictions and the input features. This allows data scientists to detect performance drops before they impact the business, rather than discovering the failure weeks later when labels finally arrive.

With a growing community of “Nansters” and strong traction on GitHub, NannyML has become a critical tool for reducing alert fatigue. Instead of reacting to every single data drift alert—which may not actually affect model accuracy—NannyML intelligently links drift to estimated performance, ensuring teams only intervene when the model is truly failing.

Key Features

• Confidence-Based Performance Estimation (CBPE): This core algorithm estimates the performance of a classification model by analyzing the relationship between the model’s confidence (predicted probabilities) and its actual error rate during the training phase, then applying that relationship to production data.
• Direct Loss Estimation (DLE): Specifically designed for regression tasks, DLE allows users to estimate the loss (e.g., RMSE) of a model in production without needing the actual target values.
• Multivariate Drift Detection: Unlike simple univariate tests, NannyML uses PCA-based data reconstruction to detect subtle changes in the overall structure of the data that could lead to model failure.
• Univariate Drift Detection: The library provides a comprehensive suite of statistical tests (including Jensen-Shannon Distance and L-Infinity Distance) to identify which specific features are drifting.
• Intelligent Alerting: By correlating drift alerts with estimated performance drops, NannyML helps teams avoid “alert fatigue” by filtering out drift that does not impact the model’s accuracy.
• Model-Agnostic Integration: Because it operates on the inputs and outputs of the model rather than the model’s internal weights, it works seamlessly with any tabular ML model.
• Interactive Visualizations: NannyML generates detailed plots that show performance and drift over time, making it easy to communicate model health to stakeholders.
• Data Quality Checks: The library includes built-in tools to check for missing values and unseen categorical levels in production data, ensuring the pipeline remains stable.

How NannyML Compares

When comparing NannyML to other tools like Evidently AI or WhyLogs, the primary differentiator is the performance estimation capability. While most monitoring tools can tell you that your data has drifted (which is a proxy for failure), NannyML can actually estimate how much the performance has dropped without needing the ground truth labels.

Evidently AI is an excellent tool for generating comprehensive visual reports and test suites for ML models, making it a strong choice for evaluation phases. WhyLogs is focused on the efficient logging and profiling of data distributions, which is highly scalable for massive datasets. NannyML, however, is the right choice when the cost of waiting for labels is high and you need a quantitative estimate of model accuracy in real-time.

Getting Started: Installation

NannyML can be installed via standard Python package managers. It is recommended to use a virtual environment to avoid dependency conflicts.

Using pip

pip install nannyml

Using conda

conda install -c conda-forge nannyml

Using Docker

For users who prefer containerized environments, NannyML provides a CLI that can be run via Docker:

docker -v /local/config/dir/:/config/ run nannyml/nannyml nml run

How to Use NannyML

The basic workflow in NannyML involves two primary datasets: a reference dataset (usually your test set from the training phase) and an analysis dataset (the production data you want to monitor). NannyML uses the reference set to learn the relationship between confidence and error, and then applies this to the analysis set.

To get started, you load your data into pandas DataFrames. You then initialize an estimator (like CBPE for classification) by specifying the column names for your predictions, predicted probabilities, and the actual targets in the reference set.

Once the estimator is fitted to the reference data, you can call the estimate method on your production data. This produces a result object that can be plotted directly to visualize the estimated performance over time.

Code Examples

The following example demonstrates how to use the Confidence-Based Performance Estimation (CBPE) algorithm for a binary classification task.

import nannyml as nml
import pandas as pd

# Load real-world data
reference_df, analysis_df, _ = nml.load_us_census_ma_employment_data()

# Initialize the CBPE estimator
estimator = nml.CBPE(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    metrics=['roc_auc'],
)

# Fit the estimator to the reference data
estimator = estimator.fit(reference_df)

# Estimate performance on the production (analysis) data
estimated_performance = estimator.estimate(analysis_df)

# Plot the results
figure = estimated_performance.plot()
figure.show()

This code snippet initializes the CBPE estimator, fits it to a known-performance dataset, and then estimates the ROC AUC for production data where the actual labels are missing.

Real-World Use Cases

NannyML shines in scenarios where the feedback loop for ground truth labels is slow or non-existent in the real-time window.

• Credit Scoring: A bank uses a model to predict loan defaults. It may take years for a loan to be repaid or default. NannyML allows the bank to estimate if the model’s accuracy is dropping due to economic shifts without waiting for the loans to actually default.
• Insurance Underwriting: In insurance, the actual outcome of a claim is often delayed. NannyML can monitor the risk assessment models to ensure they are not underestimating risk due to changes in applicant behavior.
• Churn Prediction: For subscription services, the ground truth for churn is often only known after a billing cycle ends. NannyML can provide an early warning system to detect performance degradation before the monthly churn report is generated.
• Fraud Detection: While some fraud is detected immediately, some sophisticated fraud patterns take weeks to be uncovered. NannyML can monitor the fraud model’s confidence distributions to detect silent failures.

Contributing to NannyML

NannyML is a community-driven project and welcomes contributions from the data science community. You can contribute by reporting bugs, proposing new features, or submitting pull requests to the codebase.

The project follows a standard GitHub flow for contributions. If you are encountering an issue, it is recommended to open a GitHub Issue first to discuss the proposed change. For new contributors, look for issues labeled “good first issue” to get started. The project also maintains a Code of Conduct to ensure a welcoming environment for all contributors.

Community and Support

NannyML provides several official channels for support and collaboration. The primary hub for community interaction is the Community Slack, where users can ask questions and get direct help from the core maintainers.

In addition to the Slack channel, the project maintains comprehensive documentation on Read the Docs, and GitHub Discussions is used for tracking technical queries and feature requests. The project’s activity level is high, with frequent updates and a recent commit history that shows active maintenance.

Conclusion

NannyML is an essential tool for any data science team that manages models in production. By solving the delayed target problem, it provides a quantitative way to monitor model performance without waiting for ground truth labels. This transforms model monitoring from a reactive process into a proactive one.

If your team is struggling with alert fatigue caused by data drift alerts that don’t actually impact accuracy, or if you have models where labels are delayed by weeks or months, NannyML is the right choice. We recommend starting with the quickstart guide, starring the repository to stay updated on new releases, and joining the Nanster community on Slack.

Resources

• NannyML GitHub Repository
• Official NannyML Documentation
• NannyML Official Website