A Portfolio Project Without Tests Is Just a Screenshot

Most data portfolios are designed to be looked at. The strongest ones are designed to be inspected.

A polished dashboard can show that you know how to arrange charts. A high model score can show that one run produced a good number. Neither proves that the analysis is reproducible, that the data was valid, or that you noticed when your assumptions broke.

That distinction matters because real data work is rarely graded on appearance alone. It is reviewed, rerun, extended, and used to make decisions.

Your portfolio should make that process visible.

The screenshot problem

Imagine two candidates submit customer-churn projects.

Candidate A provides:

• a dashboard screenshot;
• a notebook with 47 cells;
• an accuracy score of 91%; and
• a paragraph recommending a retention campaign.

Candidate B provides:

• the business question and metric definition;
• a small data dictionary;
• a reproducible pipeline;
• tests for duplicate customers, missing labels, and impossible dates;
• a documented train-test strategy;
• saved experiment parameters and metrics; and
• a short decision memo explaining what the model can and cannot support.

Candidate A may have built a better-looking artifact. Candidate B has supplied stronger evidence of professional judgment.

The difference is not more code. It is an audit trail.

What an audit trail actually contains

A useful portfolio project should let a reviewer move through five questions:

1. Claim: What decision or question does the project address?
2. Input: What data was used, at what grain, over what period?
3. Transformation: How did raw inputs become the analytical dataset?
4. Check: What could fail, and how would the project detect it?
5. Decision: What conclusion is supported, and what remains uncertain?

That sequence works for an SQL analysis, a dashboard, an experiment, or a machine-learning project. The tools change. The logic does not.

A sensible repository might look like this:

customer-churn-portfolio/
├── README.md
├── data/
│   ├── README.md
│   └── sample_customers.csv
├── notebooks/
│   └── exploration.ipynb
├── src/
│   ├── prepare.py
│   ├── train.py
│   └── evaluate.py
├── tests/
│   ├── test_data.py
│   └── test_features.py
├── reports/
│   ├── decision-memo.md
│   └── figures/
├── requirements.txt
└── .github/workflows/test.yml

The notebook is still useful for exploration. It is simply no longer the entire project.

Start with a falsifiable claim

“Analyze churn” is not a project question. It is a topic.

A stronger framing is:

Can account activity available by the end of a customer’s second month identify a group with at least twice the baseline 90-day churn rate, without using information created after that cutoff?

Now the project has:

• an observation window;
• a prediction horizon;
• a comparison baseline;
• a target population; and
• an obvious leakage test.

The claim can be wrong. That is a feature.

A reviewer can now evaluate whether your SQL, split strategy, and metrics answer the stated question. Without that framing, even technically correct work can drift into a result that looks interesting but supports no decision.

Make the data contract visible

Before modeling or charting, state what one row represents and what must be true.

For a customer-level training table, the contract might be:

Rule	Why it matters
one row per customer	duplicates would overweight some customers
customer_id is never null	records must be traceable
signup_date precedes cutoff_date	prevents impossible timelines
churn label is defined after the cutoff	separates features from outcomes
categorical values come from a known set	catches schema drift and typos

These checks do not require a complex platform. A few assertions in Python or SQL are enough for a small project. If you use dbt, its documented generic tests include unique, not_null, accepted_values, and relationships. The point is not to mention dbt on your résumé. The point is to encode the assumptions your result depends on.

Move repeatable logic out of the notebook

Notebooks are excellent scratchpads. They are poor substitutes for a stable execution path.

If the final result depends on running cells in a particular hidden order, a reviewer cannot tell whether the outputs came from the code currently on the page. Move the steps that produce the final dataset, model, metrics, and figures into scripts or functions.

Then give the reviewer one clear command:

python -m src.train

or one documented sequence:

python -m src.prepare
python -m src.train
python -m src.evaluate

For machine-learning work, a scikit-learn Pipeline can keep preprocessing and estimation in one cross-validatable object. The official documentation specifically recommends pipelines as a way to reduce leakage by ensuring transformations are learned from the appropriate training subset.

The professional signal is not the library name. It is that the same transformation path is used during training and evaluation.

Let a machine challenge the project

A project becomes more credible when its checks run without you.

GitHub Actions can execute a workflow whenever code is pushed or a pull request is opened. For a portfolio repository, the workflow can be tiny:

name: test
on: [push, pull_request]
 
jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest -q

This does not prove that the analysis is correct. It proves something narrower and still useful: a clean environment can install the dependencies and execute the tests automatically.

That green check is stronger than “works on my laptop,” because the claim is observable.

Track the experiment, not only the winner

A common portfolio pattern is to report the best model and hide the search that produced it.

That makes the final score difficult to interpret. Was the test set consulted repeatedly? How many alternatives were tried? Which threshold produced the reported precision and recall? Did preprocessing change between runs?

A simple experiment table can answer most of this:

Run	Features	Model	Validation F1	Test F1	Note
001	behavior only	logistic regression	0.61	—	baseline
002	behavior + plan	random forest	0.66	—	more variance
003	behavior + plan	gradient boosting	0.68	0.65	final locked test

For larger projects, MLflow Tracking can record parameters, code versions, metrics, datasets, and artifacts as runs. For a small portfolio project, a CSV or Markdown table may be enough.

Use the lightest tool that preserves the evidence.

Write the decision memo

The most valuable file may be a one-page memo, not the notebook.

A good memo answers:

• What decision is being considered?
• What did the analysis find?
• How large is the effect or error?
• What population and time period does it cover?
• What action would you recommend?
• What could make the conclusion fail?
• What should be measured after deployment?

For the churn example, “the model achieved 0.65 F1” is not a decision. A decision might be:

Pilot outreach to the top-risk decile for four weeks. Compare incremental retention and contact cost against a randomized holdout. Do not roll out to all flagged customers until uplift—not prediction accuracy—is measured.

That sentence shows you understand the boundary between prediction and intervention.

The minimum viable portfolio audit

Before sharing a project, try this checklist:

• The README states a specific question and unit of analysis.
• The data source, date range, grain, and limitations are documented.
• A clean environment can reproduce the main output.
• Critical data assumptions are executable tests.
• Preprocessing is part of the repeatable workflow.
• Model selection and the final test evaluation are separated.
• Parameters, metrics, and artifacts are traceable to a run.
• The conclusion is written as a decision with uncertainty.
• Sensitive or proprietary data is not committed.
• A reviewer can identify what would invalidate the result.

You do not need to satisfy every box for every project. A descriptive SQL analysis does not need MLflow. A dashboard may not need a model pipeline. But every project needs a clear path from evidence to conclusion.

What would make me wrong

This advice would be too heavy if the target role genuinely evaluates only visual design, exploratory communication, or a tightly scoped tool demonstration. A short Tableau or Power BI exercise may reasonably prioritize interaction design over repository engineering.

I would also change the recommendation if a hiring process explicitly prohibits code sharing, provides a fixed take-home environment, or evaluates candidates through live work instead of portfolios.

But for roles where the output may inform a business decision, enter a production workflow, or be extended by another analyst, reproducibility and testing are not decoration. They are part of the skill being evaluated.

The point is trust

Hiring teams cannot inspect every line of every project. They look for signals.

A dashboard is a signal that you can present. A notebook is a signal that you can explore. A tested, rerunnable project with a decision memo is a signal that you can be trusted with work after the demo ends.

Do not add infrastructure to look sophisticated. Add evidence that makes your reasoning easier to verify.

The best portfolio project is not the one with the most tools. It is the one that leaves the fewest important questions hidden.

Sources

• GitHub Docs — Quickstart for GitHub Actions
• scikit-learn — Common pitfalls and recommended practices
• scikit-learn — Pipeline documentation
• dbt Developer Hub — Data tests
• MLflow — Tracking experiments and runs