Products
Sudoku OCR
A photo of a grid in, the solution drawn back on it out
- Role
- Design and development
- Context
- Deployed personal project
- Stack
- Python, OpenCV, TensorFlow / Keras, Streamlit, pytest
Hosted on Streamlit Community Cloud: the first load can take up to a minute
Overview
Computer vision app that finds a sudoku grid in a photo, straightens it with OpenCV, reads the digits with a Keras CNN trained on my own dataset (Tesseract as fallback), solves the grid by backtracking and draws the solution back onto the image. Deployed Streamlit interface to review and fix the reading cell by cell, flag conflicts and download the result, with end-to-end tests on real grids.
Highlights
- 01Grid detection and perspective correction with OpenCV
- 02Keras CNN trained on a custom dataset, Tesseract as fallback
- 03Backtracking solver, solution drawn back onto the photo
- 04Cell-by-cell correction and conflict detection in Streamlit
- 05End-to-end tests on real grids
Architecture
Under the hood
A Python app that finds a sudoku grid in a photo or screenshot, reads its digits and solves it. OpenCV detects and straightens the grid, a small Keras neural network reads each cell, a backtracking solver completes the grid, and the solution is drawn back onto the original image. Two entry points share the same pipeline: a command line, and a Streamlit interface where the user corrects the reading before solving.
- accuracy on 624 test digits
- 98.1%accuracy on 624 test digits
- precision on accepted digits (confidence ≥ 0.6)
- 99.84%precision on accepted digits (confidence ≥ 0.6)
- CNN parameters, 3 convolutions
- 94,505CNN parameters, 3 convolutions
- labelled printed digits for training
- 4,157labelled printed digits for training
- steps from image to read grid
- 10steps from image to read grid
- tests, including end-to-end sample grids
- 89tests, including end-to-end sample grids
Who talks to whom
Entry points
CLI · sudoku-ocr
--image --out --config --backend --weights
Interface · sudoku-ocr-ui
Streamlit: import, correction, solution, download
Scripts
CNN training, validation, OCR experiments
read_grid() cached · render_solution()
Pipeline
config.py
DEFAULTS ← YAML ← options, unknown keys rejected
pipeline.read_grid()
Image → GridReading: read grid, cells, inverse homography
render_solution()
Solution → original image
Specialised modules
Modules
detect · geometry
Grid contour, homography to 450 × 450
cells
Lines, 81 cells, blank check, digit extraction
ocr/cnn · ocr/tesseract
CNNOCR and its .meta.json contract · Tesseract fallback
solver · overlay
MRV backtracking, conflicts · drawing mapped back by Minv
Reads the versioned model and configuration
Repository files
models/sudoku_cnn.keras
Versioned model + .meta.json (classes, scale, polarity)
configs/default.yaml
OCR backend, confidence threshold, rendering
data/assets
4,157 labelled digits for training
data/samples
6 sample grids, light and dark
Design principles
01
Two doors, one pipeline
A single read_grid function does detection, slicing and OCR. Both the CLI and the interface call it and get a GridReading.
02
Always dark ink
A dark-mode grid is read on the inverted image. The candidate whose straightened grid has a light background wins.
03
The model carries its contract
sudoku_cnn.meta.json describes classes, pixel scale and polarity. Network input is prepared from this file, not from constants.
04
A strict configuration
Any key missing from the defaults is rejected: a typo in a YAML raises an error instead of being ignored.
05
The human corrects, the machine solves
The interface never fixes anything silently. It shows conflicts and the number of solutions, and only solves a grid with a unique solution.
06
No server, no database
No persistent state. The interface caches the model and keeps the last 16 readings.
Pipeline · from image to read grid
- 01imread · decode_imageImagePhoto or screenshot, as BGR
- 02bitwise_notTwo candidatesThe image and its inverse, for dark mode
- 03find_sudoku_quadDetectionCLAHE, adaptive threshold, 20 largest contours scored by grid_score
- 04four_point_transformStraightening450 × 450 square, inverse homography kept
- 05grid_scoreChoiceLight background first, then best score
- 06detect_grid_linesLinesProjections, 10 + 10 lines, regular-grid fallback
- 07split_cells_by_lines81 cellsSliced with a 1/40-cell margin
- 08is_blank_cellBlank cell?Centre contrast < 40: no OCR
- 09extract_digitExtractionMost central component, scaled to 28 × 28
- 10predict_digitCNN9 classes, confidence < 0.6 → cell left blank
↳ The CLI repairs on its own
- Removes conflicting digits
- Fills single-candidate cells
- Solves, and rereads blank cells with Tesseract on failure
↳ The interface asks
- Editable table: the user corrects the reading
- Conflicts in red, solutions counted up to 2
- Solves only with a unique solution, then draws on the original
A blank cell the user fills in beats a wrong digit that makes the grid unsolvable: that is the whole point of the confidence threshold. When extraction fails on a non-blank cell, three binarisations let the CNN vote.
Neural network · 94,505 parameters
| Layer | Output | Parameters |
|---|---|---|
| Input | 28 × 28 × 1 | 0 |
| Conv 3×3 · BN · ReLU | 28 × 28 × 32 | 288 + 128 |
| MaxPool 2×2 | 14 × 14 × 32 | 0 |
| Conv 3×3 · BN · ReLU | 14 × 14 × 64 | 18 432 + 256 |
| MaxPool 2×2 | 7 × 7 × 64 | 0 |
| Conv 3×3 · BN · ReLU | 7 × 7 × 128 | 73 728 + 512 |
| GlobalAveragePooling · Dropout 0,3 | 128 | 0 |
| Dense · softmax | 9 (digits 1 to 9) | 1 161 |
No “blank” class: blank cells are filtered out before the network. BatchNorm momentum is 0.9, because with the Keras default the running statistics did not converge and the model collapsed at inference.
Training · test on 624 images
| Metric | Before | Current |
|---|---|---|
| Accuracy | 96,3 % | 98,1 % |
| Coverage (confidence ≥ 0.6) | 91,3 % | 97,9 % |
| Precision of accepted digits | — | 99,84 % |
| Wrong digits accepted | 4 | 1 |
4,157 printed digits, 15% held out for testing. Same preprocessing as the app: every image goes through the interface's extraction. Adam, batches of 64, early stopping, about 2 minutes on CPU.
Solving
| Function | Role |
|---|---|
| solve | Backtracking that starts from the most constrained cell (MRV) and gives up as soon as a cell has no candidate |
| count_solutions | Counts solutions without changing the grid, stopping at 2 |
| find_conflicts | Cells repeated in their row, column or box |
Tests · 89 in total
| File | Covers |
|---|---|
| test_end_to_end | The 6 sample grids, light and dark, with the real model |
| test_ui | Interface helpers and the Streamlit page via AppTest |
| test_config | Merging, unknown keys, invalid values |
| test_solver | Solving, conflicts, uniqueness |
| test_cells · test_detect | Blank cells, grey backgrounds, framed grid |
| test_cnn_ocr | Model contract, polarity, confidence threshold |
Known limits
| Topic | Finding |
|---|---|
| 9 × 9 grids only | Slicing and solver assume 3 × 3 boxes; a 6 × 6 mini sudoku is misread. |
| 81 network calls | One cell at a time, about 28 ms each: 2 to 3 s per grid. |
| No “not a digit” class | A smudge that passes the filters is classed 1 to 9; only the confidence threshold stops it. |
| Cloud memory | About 756 MB once TensorFlow is loaded, against a limit of about 1 GB. |
| No real photos in tests | Samples are screenshots: perspective, blur and uneven lighting are not covered. |
CI/CD and environments
dev
Day-to-day work, local tests
main
Fast-forwarded from dev
Streamlit Cloud
Redeploys on every push to main
No continuous integration: the 89 tests run locally with make test (about 90 s), make test-fast skips those that load the model. The model is versioned in the repository, and OpenCV is the headless build because the server has no libGL.
Full stack
| Tech | Version | Role |
|---|---|---|
| Python | 3.10–3.13 | Language, capped by TensorFlow |
| opencv-python-headless · numpy | — | Thresholds, contours, homographies, drawing |
| TensorFlow (Keras) | ≥ 2.12 | CNN loading and inference |
| pyyaml | — | Configuration loading |
| Tech | Version | Role |
|---|---|---|
| Streamlit (+ pandas) | ≥ 1.40 | Web interface, editable table |
| pytesseract | — | Alternative OCR backend, CLI fallback |
| scikit-learn | — | Stratified split for training |
| pytest | — | Tests, e2e marker |
| Tech | Version | Role |
|---|---|---|
| uv · Makefile | — | Python 3.12 environment, test, train, run, ui commands |
| setuptools · pyproject.toml | — | sudoku-ocr and sudoku-ocr-ui commands |
| Streamlit Community Cloud | — | Hosting for sudoku-ocr.streamlit.app |
Read from the code, dev branch, v0.1.0, 20 October 2026
Screens
Hover or use the keyboard to enlarge a screen
Next project QRTO
Quarto →