The Screenshot Test That Hangs on Real Devices But Not in the Simulator
How a test harness that fakes the viewport size can pass cleanly in a headless simulator and hang forever on a real phone — plus a batching pattern that reports 'all tests passed' while writing zero files.
A green test suite is not proof that anything happened
Here is the part that should worry any operator who trusts a CI dashboard: a test run can print All tests passed! and then produce nothing. No screenshots, no artifacts, no errors — just a green check that means nothing.
We hit exactly this on a Flutter ledger app inside our finance suite. The tool involved is a device walk: an integration test that visits every route in the app, screenshots it, and ships the images so we can eye-check the UI across real hardware. It’s cheap insurance against layout regressions. And it had been sitting on our known-issues list, hanging after the very first route, with no exception and nothing in the logs to explain why.
That silence is the whole story. This trap is good at hiding.
The shim that’s correct in a simulator and fatal on a phone
The test harness had a helper, _pumpAt, that overrode two values on the way in: tester.view.devicePixelRatio and physicalSize. In plain terms, it lied to the app about the size of the screen so we could exercise multiple viewport sizes from one test.
In the headless flutter_tester environment — the fake device the tests normally run against — this is completely correct. There is no real display surface to contradict the lie, so the renderer just believes it and moves on.
On a real Android device, the lie meets physics. The renderer composited a frame at the simulated size (780x1688) instead of the phone’s actual surface size (1080x2400). The frame came out letterboxed, and the platform’s ImageReader — the Android component waiting for a frame that matches the real surface — never received a matching image. It kept waiting.
So the same code that is right in simulation is a silent, permanent wedge on hardware. That’s the transferable point: a shim that fakes input in your test harness — viewport, clock, network, time zone — can be flawless in a headless environment and a hang in a real one. The simulator has no ground truth to disagree with. The device does.
Why it hung forever instead of failing
A wrong frame size shouldn’t lock up a whole run. What turned a mismatch into an infinite hang was the code on the other side.
The capture path, FlutterDeviceScreenshot.takeScreenshot, retried against the Choreographer — Android’s frame-timing clock — with no timeout and no error path. When the matching image never arrived, the retry loop just kept going. The awaited screenshot call never returned. No exception was thrown, so nothing bubbled up to the test runner, so nothing got logged.
That combination is the real landmine: a size mismatch you could have shrugged off, sitting behind a retry loop that has no bound. Every polling or retry mechanism needs a timeout and an error path. Without one, any upstream condition it can’t satisfy becomes a hang with no evidence attached — the worst possible failure to debug, because there’s nothing to grab onto.
Proving it, the boring way
We didn’t want to guess. We ran a live A/B/A test against the device. Forcing the phone’s real display size with adb shell wm size unstuck the wedged run immediately. Resetting it back re-hung the run on the spot. Same code, one variable, flipped twice — that’s causation, not correlation.
The second trap, hiding behind the first
Fixing the hang exposed a separate problem that had been masked the whole time. The walk accumulates every screenshot in memory and ships them all in one batch at the very end of the run. Once the walk actually completed, it tried to send 402 device-resolution PNGs in a single payload — and that killed the VM service connection outright.
The result: the run printed All tests passed! and then died before writing a single image. Zero artifacts, green status.
This is worth saying on its own, because it applies far beyond screenshots: “the suite passed” is not evidence that its outputs exist. Any strategy that accumulates results and ships them in one final payload can succeed on every assertion and still deliver nothing — the failure lands after the last check, in the delivery step nobody asserts on. If your pipeline produces artifacts, test that the artifacts arrived, not just that the code ran.
Two bugs, one lesson each. Bound your retry loops. And check the files, not the checkmark.