Before You Clean Your Data, Ask This One Question: Is It Actually Dirty?

Every data course teaches the same reflex. Load the file, run df.isnull().sum(), see the missing values, remove them. Clean data, tick, move on to the interesting part.

It is a good reflex for a tutorial dataset, where the nulls were inserted deliberately so you would practise removing them. It is a bad reflex for a real one, where the nulls got there by a process, and the process is often the most informative thing in the file.

Consider a table of customer accounts where 18% of rows have no value in last_login_date. The tutorial answer is to drop them or fill them. Both destroy the finding, which is that those are the customers who signed up and never came back. That is not dirt. That is your churn signal, sitting in plain sight, wearing a null.

Why this matters

Cleaning is where most of the damage in a data project happens, precisely because it does not feel like a decision. Modelling feels like a decision. Choosing features feels like a decision. Dropping 4,000 rows at the top of a notebook feels like housekeeping.

But dropna() is a claim. It says: the customers in these rows are not systematically different from the ones I am keeping. If that claim is false, and it usually is, you have not cleaned your dataset. You have introduced a bias, silently, in the first ten lines, and everything downstream inherits it.

The same is true in reverse. fillna(0) says: the true value here is zero. For a purchase amount that may be right. For a blood pressure reading it is not just wrong, it is dangerous. For an income field it invents a population of destitute customers who do not exist.

The reason this is the third lesson in this series is that it is the same discipline as the first two, applied to a different surface. In Part 1 the question was what a row means. Here it is what an empty cell means. Both are questions about the process that produced the data, and neither can be answered by looking harder at the data alone.

The professional mindset: nulls are evidence

Reframe it and the whole task changes.

A missing value is not the absence of information. It is information about the process, encoded as an absence. Something happened, or failed to happen, and that fact was recorded by not recording anything.

So the first move is never to remove it. The first move is to ask: what would have to be true in the real world for this cell to be empty?

Usually there are only a few candidate answers, and they are easy to tell apart once you look. That is the whole method.

The five questions, in order

Run these before any cleaning code.

1. What mechanism produced this gap?

There are three, and they demand completely different responses.

Collection failure. A sensor dropped out. An API timed out. A form field broke on mobile for two weeks in March. This is genuine dirt, and it is the only category where removal or imputation is straightforwardly appropriate.

Structural impossibility. The field cannot apply to this row. cancellation_date is empty for customers who have not cancelled. spouse_income is empty for single applicants. Nothing is missing here at all. The null is the correct and complete answer, and imputing it is fabricating a fact.

Business event. The gap exists because of something the subject did or did not do. No last_login_date because they never logged in. No first_purchase_date because they never bought. This is a signal, frequently a strong one, and dropping it throws away the most predictive column in your table.

2. Is the missingness related to the thing I am trying to predict?

This is the question that decides whether removal is safe. If rows with a missing value are more likely to be churners, or fraudsters, or non-responders, then dropping them removes exactly the cases you care about most. You can test this directly, in one line, and you should.

3. Is the gap concentrated somewhere?

Missingness clustered in one region, one product line, one time window, or one collection channel is nearly always a systems story, not a random one. A field that is 40% empty for mobile signups and 2% empty for desktop tells you a form is broken, and that is a bug report, not an imputation problem.

4. Does the gap have a start date?

Plot missingness over time. If a field is complete until March and 60% empty afterwards, something changed: a migration, a redesign, a vendor switch, a new consent rule. This single plot has saved more projects than any imputation technique.

5. What does the person who owns this system say?

The answer to all of the above usually exists, in someone's head, and takes one conversation to obtain. Data Scientists reliably spend three days inferring what an operations lead would have told them in ten minutes.

A real-world example: the missing income field

A lender asks you to build a credit-risk model. The application table has annual_income missing for 23% of applicants. The obvious move is to impute the median and continue.

Run the five questions instead.

Question 1, the mechanism. You look at the application flow and find that income is optional for applicants who connect their bank account, because the system derives affordability directly from transactions. So the field is not missing at random at all. It is missing precisely for people who chose the bank-connection route.

Question 2, is it related to the target? You check. Applicants missing income default at a noticeably lower rate. That makes sense in hindsight: people willing to share bank access tend to have less to hide and steadier finances.

Question 3, is it concentrated? Yes, heavily among younger applicants and app-based signups, because the bank-connection flow launched on mobile first.

Question 4, does it have a start date? Yes. Near zero missingness before the feature launched, then a step change.

Now look at what each naive option would have done.

dropna() removes 23% of applicants, and specifically the lowest-risk, youngest, most digitally engaged segment. Your model is now trained on an older, higher-risk population and will systematically overprice exactly the customers the business most wants to win.

fillna(median) is worse in a subtle way. It assigns an average income to a group whose actual risk profile is distinctly better than average, blurring a real and useful distinction into noise.

The correct treatment falls straight out of the diagnosis: the missingness itself is the feature. Add a boolean flag for the bank-connected route, keep income missing for those rows, and use a model that handles nulls natively, or fit the two populations separately. The gap was never dirt. It was a behavioural signal that the cleaning step would have deleted.

That is the whole lesson in one example. Nobody could have reached it from df.isnull().sum().

Diagnosing before cleaning, in code

Here is the diagnostic pass I run on any table with meaningful missingness. It is short and it replaces guesswork with evidence.

import pandas as pd
 
df = pd.read_csv("applications.csv")
 
# 1. Where are the gaps, and how big?
missing = (
    df.isna().mean()
    .loc[lambda s: s > 0]
    .sort_values(ascending=False)
    .mul(100).round(1)
)
print(missing.to_string())

# 2. Is missingness related to the target? This is the decisive check.
target = "defaulted"
 
for col in missing.index:
    flag = df[col].isna()
    rates = df.groupby(flag)[target].mean()
    print(f"{col:<24} missing={rates.get(True, float('nan')):.3f}  "
          f"present={rates.get(False, float('nan')):.3f}")

If those two rates differ materially, the missingness carries information and you must not drop those rows.

# 3. Is it concentrated in a segment, or did it start on a date?
print(df.groupby("signup_channel")["annual_income"].apply(lambda s: s.isna().mean()))
 
by_month = (
    df.assign(month=df["applied_at"].dt.to_period("M"))
      .groupby("month")["annual_income"]
      .apply(lambda s: s.isna().mean())
)
print(by_month)

A step change in that last series is a systems event. Go and find out what shipped that month.

# 4. Treatment, once the mechanism is known and only then.
 
# Business event: the gap is the signal. Keep it and flag it.
df["income_undisclosed"] = df["annual_income"].isna().astype(int)
 
# Structural: the null is correct. Encode the meaning, do not impute.
df["has_cancelled"] = df["cancellation_date"].notna().astype(int)
 
# Collection failure, and verified unrelated to the target: now
# imputation is defensible. Impute within a sensible group, not globally,
# and always leave a trace that you did it.
df["was_imputed"] = df["credit_score"].isna().astype(int)
df["credit_score"] = df.groupby("product_type")["credit_score"].transform(
    lambda s: s.fillna(s.median())
)

Note the shape of that last block. Every imputation leaves a flag behind. If a downstream result turns out to depend on imputed rows, you can find out in one line instead of re-deriving the pipeline six months later.

The AI perspective

What AI can do. Write every diagnostic above, faster and more completely than you would bother to. Explain the difference between missing completely at random, missing at random, and missing not at random, with examples, until it actually lands. Suggest imputation strategies appropriate to a described mechanism. Implement iterative imputation, group-wise medians, or a missing-indicator pattern correctly. Spot that you dropped rows before splitting train and test. All of this is real help, and you should take it.

What AI cannot do. Tell you that income is optional for bank-connected applicants. That fact is not in the file. It is in a product decision made eighteen months ago by a team you have not met. It cannot tell you that the March step change was a CRM migration, that a clinic stopped ordering a test because of a supply shortage, or that a field went empty because a regional manager instructed staff to stop filling it in. Ask an AI assistant to clean a dataset and it will produce competent, confident, plausible code, applied to a mechanism it has no way of knowing.

What you must decide. Which of the three mechanisms you are looking at. Whether the missingness is itself a feature. Whether removal biases the population you are reasoning about. What a filled value asserts about the world, and whether you are willing to assert it. Every one of these is a claim someone will have to defend, and it will be you.

The pattern is identical to Part 2: AI is excellent at the transformation, blind to the provenance. Cleaning is a provenance problem wearing a transformation costume.

Common mistakes

Calling dropna() on the full dataframe. One column with 30% missingness can delete a third of your rows, including rows that were complete everywhere you actually cared about.

Filling with zero because zero is a number. Zero income, zero blood pressure and zero purchase amount are not the same kind of claim, and only one of them is ever harmless.

Imputing before splitting train and test. The imputed values carry information from the whole dataset, including the test set. Classic leakage, and your evaluation will flatter you.

Treating structural nulls as missing. Imputing a cancellation date for customers who have not cancelled invents events that never happened, and any model trained on it learns nonsense.

Cleaning without leaving a trace. If nobody can tell which values were real and which were manufactured, nobody can audit the result, including you.

Assuming missingness is random because it is inconvenient not to. This is the assumption that quietly does the most damage, because it is never stated and therefore never checked.

Letting the tutorial reflex run before the thinking. The reflex is fast, feels productive, and is applied before you know anything. That combination is what makes it expensive.

Best practices

1. Diagnose before you touch anything. Mechanism, relationship to the target, concentration, start date. Four checks, ten minutes.
2. Classify every gap into collection failure, structural, or business event. Only the first is dirt.
3. Test whether missingness predicts your target. If it does, it is a feature, and dropping it is throwing away signal.
4. Prefer flags over fills. A missing-indicator column preserves information that imputation destroys.
5. Impute within groups, never globally, and only after splitting. Global medians erase the structure you are trying to model.
6. Leave a trace on every imputed value. A was_imputed column costs nothing and makes the whole pipeline auditable.
7. Ask the system owner before inventing an explanation. One conversation beats three days of inference, reliably.
8. Write down the mechanism you concluded and why. Future you will not remember, and the next analyst will otherwise repeat the whole investigation.

Key takeaways

• Missing values are produced by a mechanism. Find the mechanism before choosing a treatment.
• Three mechanisms, three responses: collection failure is dirt, structural nulls are correct answers, business events are signals.
• dropna() and fillna() are assertions about reality, not housekeeping. Make them deliberately.
• If missingness correlates with your target, it is one of your best features, and removing it is the most expensive mistake in the notebook.
• Concentration by segment and step changes by date are the two checks that most often reveal the real story.
• AI writes the cleaning code well and cannot know why the data is missing. That gap is exactly where your value sits.
• Do not clean data. Understand data, then clean with purpose.

Continue the series

This is Part 3 of the Data Thinking Series.

Previously: AI Won't Replace Data Scientists. It Will Replace Lazy Ones. on the division of labour between fast execution and the judgement that gives it a point.

Start from the beginning: The First Question Every Data Scientist Asks: What Does One Row Represent?, which is where this way of working starts.

More lessons follow in this series, each one about the reasoning that happens before the code. The philosophy does not change: learn the fundamentals, and use AI to accelerate them, not to replace them.

Your turn. Open your current dataset, take the column with the most missing values, and answer one question about it: what would have to be true in the world for that cell to be empty? Whatever you find, it will be more useful than the median.