A password hash is a pool outage in disguise
Holding a pooled database connection across scrypt let ten sign-ins park every client and stall unrelated routes.
A password hash becomes a pool outage when code keeps a database connection across CPU work. Muniment Cloud did, and unrelated routes answered with intermittent 503 pages.
Connection ownership turned a tenth of a second into pressure across the service.
Ten hashes parked ten connections
Muniment Cloud verifies passwords with scrypt on its sign-in path. One verification burns about a tenth of a second of pure CPU.
The code held one pooled database connection throughout that call.
With ten connections, a burst of ten sign-ins parked the whole pool for that tenth of a second. Every other request that needed a connection waited behind CPU work that never touched the database.
Muniment Cloud traced two separate production defects in one week to this pressure.
Releasing the connection fixes the shape
The correction follows three steps:
- Authenticate, then release the database connection.
- Run the hash while holding zero connections.
- Reconnect for the session write.
Muniment Cloud applied this shape first to login. A search found the same hold on four more paths: password change, operator password reset, user creation, and invite redemption.
Each path now follows the same three steps. Tests assert that the process holds zero clients while the hash runs.
Outside work can exhaust a borrowed pool
Audit CPU-bound and network-bound calls for held resources alongside duration.
A hash, an SMTP send, and an upstream HTTP call can each exhaust a pool when run on a borrowed connection. The resource must end before outside work starts.
The evidence stops before throughput
Muniment Cloud did not benchmark throughput before and after this change. This record cannot assign a throughput gain.
More than one cause contributed to the 503 rate, so these two defects do not explain the full rate. Our evidence consists of traced defects and connection-ownership tests, not a load test.
This evidence comes from our Muniment Cloud note pinned to commit d7bb80d41f9727fbe7d0958d.
A tenth of a second was enough because the hash kept the wrong resource.