HollowByte: How 11 Bytes Turned OpenSSL Into a Memory Leak With No CVE

An unpatched OpenSSL server will happily reserve 131 KB of memory for a TLS message that never shows up — and if it’s running on glibc, that memory doesn’t come back. That’s the crux of HollowByte, a denial-of-service flaw that shipped with a fix in June, no CVE, no advisory, and no changelog entry. If your patch pipeline runs on CVE identifiers and vendor advisories, you already have a blind spot.

Why an 11-Byte Handshake Can Freeze a Server

According to Okta’s Red Team, which reported and named the bug, every TLS handshake message carries a 4-byte header where three bytes declare the body length. Older OpenSSL versions grew the receive buffer to that declared size the moment the header landed — before the body arrived and before the handshake’s own sanity checks ran. For an inbound ClientHello the ceiling is 131 KB per connection, and the worker thread then blocks waiting on a body that never comes. No authentication. No key exchange. Just trust in the attacker’s stated length.

On its own that’s a connection-exhaustion attack, and the industry has been dealing with Slowloris-class abuse for years. What makes HollowByte different is what happens after the attacker walks away. If you’re running NGINX in front of an API — a common configuration for anything from fintech to a hospital’s patient portal — you’d normally rely on per-IP connection limits to shrug this off. HollowByte punches through that assumption.

Our take: this is the kind of bug that will sit unnoticed in production for years precisely because it looks like normal memory pressure on a graph.

The glibc Twist That Makes the Memory Stick

When the attacker drops the connection, OpenSSL frees the buffer, but glibc holds small and medium chunks for reuse rather than returning them to the kernel. Because HollowByte varies the claimed size on every connection, Okta reports the allocator can’t reuse what it freed — the heap fragments, resident set size climbs, and stays climbed. In Okta’s NGINX testing, a 1 GB server was OOM-killed with 547 MB frozen in fragments, and a 16 GB server had 25% of system memory locked without ever hitting connection limits.

Why this matters: it turns a transient DoS into semi-permanent damage. Standard connection-limiting defenses, per Okta’s Red Team, won’t stop it. If you operate long-lived TLS terminators — think an end-to-end supply chain platform with persistent B2B partner connections — a low-and-slow attacker can quietly poison your memory footprint over hours without ever tripping rate limits.

Our take: expect defenders to start monitoring RSS-to-active-connection ratios as a first-order metric, not just connection counts.

Why OpenSSL Refused to Assign a CVE

Here’s where the story gets uncomfortable. The pull request from Matt Caswell, who wrote the patch, states the security team chose to “handle this as a ‘bug or hardening’ only fix.” OpenSSL’s security policy defines four tiers — Critical down to Low — and “bug or hardening” isn’t among them. Even a Low issue earns a CVE, changelog note, and vulnerabilities-page entry. HollowByte has none. The Hacker News found no mention in the release notes or the 4.0.1 changelog’s 23 entries.

Compare that to CVE-2025-66199, a Low-rated TLS 1.3 certificate-compression bug from January that also grew a heap buffer from a peer-declared length — worth around 22 MiB per connection but requiring four preconditions to line up. HollowByte requires none of them, and got no number. The same June 9 release also handed a Moderate CVE (CVE-2026-34183) to a QUIC PATH_CHALLENGE memory-exhaustion issue. Same class of bug, different treatment.

If you’re a downstream consumer this is where the real pain lives. Red Hat’s documented default is to backport patches rather than bump versions, so a fixed package still reports its original version. What normally resolves that gap is the CVE-keyed OVAL feed. No CVE, no key. Our take: OpenSSL just handed every vulnerability scanner vendor a homework assignment, and most of them will fail it.

What Engineering Teams Should Actually Do

The fixed releases are OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21, all dated June 9. If you build OpenSSL yourself, upgrade and restart every process that loaded the old library — LD_PRELOAD tricks and long-lived daemons are exactly how unpatched code lingers. If you consume distro packages, the reliable path is asking your maintainer whether they rebased on the June 9 release or cherry-picked pull requests 30792 (master and 4.0), 30793 (3.6, 3.5, 3.4), or 30794 (3.0).

One more caveat: the fix covers TLS only. Caswell wrote that DTLS was left alone because doing it properly would be far more invasive. The Hacker News confirmed the DTLS handshake file is byte-identical across the 3.6.2-to-3.6.3 fix, and in 4.0.1 that path still sizes its buffer from the peer-declared length. If you run DTLS — WebRTC media servers, VPN concentrators, IoT telemetry — you’re still exposed and OpenSSL hasn’t committed to fixing it.

Practical scenario: if you run a healthcare application with compliance obligations, your auditor is going to ask for a CVE reference. There isn’t one. You’ll need to point at the pull request numbers and the Okta writeup as evidence you patched a real issue.

Prediction: within the next six months, at least one major scanner vendor will add a non-CVE “HollowByte” check keyed on OpenSSL version strings, and OpenSSL will quietly assign a CVE after enough pressure — the same pattern that’s played out with every silently-patched flaw since Heartbleed.

FAQ

Q: What is the HollowByte vulnerability? A: HollowByte is a denial-of-service flaw in OpenSSL where an 11-byte TLS handshake header causes the server to allocate up to 131 KB of buffer for a message body that never arrives. On glibc systems, the freed memory fragments and doesn’t return to the kernel, causing sustained memory pressure. Okta’s Red Team reported and named it.

Q: Which OpenSSL versions are affected? A: Every release before OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21 — all dated June 9 — contains the flaw. Because OpenSSL declined to issue a CVE or advisory, standard vulnerability scanners will not flag affected packages, and downstream distributions that backport patches may still report vulnerable version strings.

Q: Does patching TLS also fix DTLS? A: No. Per Matt Caswell’s pull request notes, the DTLS handshake path was intentionally left unpatched because a proper fix would be far more invasive. The Hacker News confirmed the DTLS handshake code is byte-identical between vulnerable and fixed releases, so WebRTC, VPN, and IoT deployments using DTLS remain exposed.

Key Takeaways

• Stop treating CVE feeds as your source of truth for OpenSSL patching — HollowByte proves upstream can silently ship security-relevant fixes with no identifier to match against.
• Add heap-fragmentation and RSS-per-connection monitoring to your TLS terminators; connection-limit dashboards will not surface this class of attack.
• If you run DTLS anywhere in your stack, plan compensating controls now because OpenSSL has not committed to fixing the equivalent path.
• Ask your Linux distribution maintainers directly whether they picked up pull requests 30792, 30793, or 30794 — do not trust package version strings alone.
• Expect more “silent security fix” disclosures ahead, and build a process to track upstream commits, not just advisories.