Learn civic data science by answering one real question — is the city building the homes it promised? — from the city's own published data, one checkable step at a time.
This is a hands-on course for a curious beginner or journalist — you don't need to know how to code to start. Every notebook runs free in Google Colab (nothing to install) or locally in Jupyter, and pulls in Berkeley's open data automatically. Start at the top of the on-ramp and work down; each step shows its work. ← Back to the main project page
Nine short notebooks. The first one shows you the data itself — three pictures before any analysis, the way statisticians are taught to start. The rest build the vocabulary from zero: what data, a notebook, a function, a table, a chart, the tools, an AI agent, and the rules an agent follows. Click any one to open it in Colab.
| Module | Notebook | What you'll learn | Run |
|---|---|---|---|
| JN00 | Look at the data first | Plot your data before you analyze it — see the whole dataset three ways, and ask an AI to look too, before any analysis. Start here. | Colab |
| JN0a | Why municipal data | Why "published" isn't the same as "checkable" — and why you can close that gap now. | Colab |
| JN0b | What a computational notebook is | A document that runs — so every number carries its proof, and re-running is re-checking. | Colab |
| JN0c | What a function is | One named rule, asked the exact same way thirty thousand times. | Colab |
| JN0d | What a pandas DataFrame is | A table you query in code instead of scrolling by eye. | Colab |
| JN0e | One set of numbers, many pictures | A chart is an argument — the same data told several honest ways, including a 3D map. | Colab |
| JN0f | The tools we use | pandas, matplotlib, SQLite/SQL, and MapLibre — every one open-source and free. | Colab |
| JN0g | Building with agents | Direct an AI agent in plain English — and verify the answer before you trust it. | Colab |
| JN0h | The agent instruction file | The standing rules an agent reads first, so it follows your project — not generic defaults. | Colab |
Eight notebooks that rebuild Berkeley's housing record from raw permits all the way to a scored Annual Progress Report — checked against the city's own submission to the state, and finished with the re-key and the audit.
| Module | Notebook | What you'll learn | Run |
|---|---|---|---|
| JN1 | Getting the data in the door | Turn the city's messy yearly spreadsheets into one clean, verified table of permits. | Colab |
| JN2 | The address key | Match the same place across sources with a relation, not fragile string-equality. | Colab |
| JN3 | Build the spine + units | Group permits into buildings and count new homes with the corrected unit signal. | Colab |
| JN4 | Dated milestones + completion stage | Derive each building's stage from dated events — never from a status string. | Colab |
| JN5 | Year & cycle tagging | One completion date answers three different questions — learn not to conflate them. | Colab |
| JN6a | Capstone — interrogate the oracle + generate your APR | Verify the city's official report against ground truth, then render your own HCD-form APR. | Colab |
| JN6b | Capstone — join, decompose, score | Score your APR against the city's, account for every unit of difference, and rediscover the limitation you planted earlier. | Colab |
| JN7 | The final act — the re-key and the audit | Fix the limitation you flagged: re-key buildings to permit families, discover "one building" was a class, write your first adjudication ledger, and run the audit's own error-detectors on the city's filing. | Colab |
| JN8 | The coda — the record watches itself | The answer key moves: cities amend filings. Pin a dated snapshot, re-pull the live state dataset through its API, and diff — turn "we checked" into "we are checking." (While we built it, the state respelled a column: the lesson demonstrated itself.) | Colab |
Prefer to work locally?
git clone https://github.com/blockXblock/berkeley-housing-analysis — the repo
already contains everything the notebooks download in Colab (the raw spreadsheets, the shared
modules, the pinned state filing), so the first cell detects the repo and skips all downloads.
Details in the curriculum README.