The strategy decision in the making of cshttp.server

Exploring the different strategy that makes cshttp.server

Introduction: What is cshttp.server and Who is It For?

cshttp.server is a lightweight, zero-dependency, self-contained HTTP/1.1 and HTTPS web server written in pure C# 7.3. It compiles directly into any .NET Framework 4.8 or .NET 8 / .NET 10+ Console Application, Windows Service, or Linux systemd daemon—requiring no IIS, no Kestrel, no ASP.NET Core hosting pipeline, and zero external NuGet packages.

Built on top of two companion layers:

• cshttp (Tier 1): A resumable, RFC 9112 byte-level HTTP/1.1 wire parser and zero-copy chunked decoder with built-in request-smuggling and Slowloris guards.
• cshttp.web (Tier 2): A familiar, System.Web-compatible content toolkit (HttpContext.Current, HttpRequest, HttpResponse, lazy QueryString / Form / Files / Cookies, StaticFiles, and HostingEnvironment.MapPath).

Who Benefits from cshttp.server?

1. Developers Who Want Self-Contained C# Web Apps Without Framework Bloat:

   If you want to ship a single .exe web application, internal admin dashboard, local desktop companion server, or microservice without pulling in dozens of Microsoft.AspNetCore.* assemblies, dependency injection containers, or middleware pipelines.

2. Developers Who Prefer Synchronous, Step-Through-Debuggable Request Handlers:

   In cshttp.server, your application code is a clean, 100% synchronous Action<HttpContext> handler with a straightforward switch (path) router. You can query SQLite/MySQL, read files, and write responses using HttpContext.Current without viral async/await infecting every method signature in your codebase.

3. Teams Maintaining Both .NET Framework 4.8 and Modern .NET 8/10+ Projects:

   Because the entire stack is written in pure C# 7.3 against standard BCL sockets, the exact same source code compiles and runs identically on legacy Windows .NET Framework 4.8 machines and modern Linux .NET 10 servers.

Exploring the 3 Architectural Strategies Before Building cshttp.server

When designing a standalone HTTP/1.1 web server in pure C#, the most critical architectural decision happens at the boundary between the TCP socket listener and the application request handler:

• How should the server accept incoming TCP connections?
• What happens to threads while HTTP/1.1 Keep-Alive sockets sit idle between browser requests?
• How should active requests be dispatched so that synchronous application handlers (Action<HttpContext>) run at peak speed without starving network I/O?

Before finalizing the architecture of cshttp.server 1.0, we implemented and stress-tested three distinct TCP connection strategies against the exact same HTTP parser (cshttp) and handler pipeline (cshttp.web).

Strategy 1: One Dedicated OS Thread per Connection (cshttp.server.alpha.1.0)

Description

In the classic thread-per-connection model (cshttp.server.alpha.1.0), the server runs a blocking listener.Accept() loop on a dedicated thread and spawns a brand-new OS thread (new Thread(conn.Run).Start()) for every accepted TCP socket. Inside conn.Run(), all socket reads, HTTP parsing, handler execution, and response flushes are 100% synchronous blocking calls.

Why did alpha.1.0 use new Thread() instead of ThreadPool.QueueUserWorkItem()? Because modern web browsers open 6 concurrent keep-alive TCP sockets per origin and hold them open for 60 seconds even when the user is just reading a page. Since conn.Run() blocks synchronously inside _stream.Read() waiting for the next keep-alive request, queuing conn.Run() onto the .NET ThreadPool would mean just 2 open browser tabs (12 idle connections) would consume all 8 MinThreads on an 8-thread CPU—stalling the ThreadPool hill-climbing algorithm. Spawning new Thread(conn.Run) bypassed ThreadPool starvation, but at the cost of ~1 MB of OS stack space per open socket and a hard MaxConnections = 1,000 ceiling.

Sample Code Block Demo for the Concept

// Strategy 1 Concept: 1 Dedicated OS Thread per TCP Connection
void AcceptLoop(Socket listener)
{
    while (!_stopping)
    {
        Socket client = listener.Accept();
        if (_activeConnections >= _maxConnections)
        {
            client.Close(); // Hard wall: refuses connections once MaxConnections (1000) is reached
            continue;
        }

        var conn = new AlphaConnection(this, client);
        // Spawns a dedicated 1 MB OS thread for the entire lifetime of the TCP socket
        new Thread(conn.Run) { IsBackground = true }.Start();
    }
}

void Run() // Runs on the dedicated per-connection OS thread
{
    using (_stream = new NetworkStream(_socket, ownsSocket: true))
    {
        while (!_stopping)
        {
            // BLOCKS this OS thread for up to 60s while waiting for the next Keep-Alive request!
            _socket.ReceiveTimeout = (int)_keepAliveTimeout.TotalMilliseconds;
            int n = _stream.Read(_buf, _end, _buf.Length - _end);
            if (n <= 0) break;
            _end += n;

            // Synchronously parse head, read body, run handler, and flush response
            if (!ProcessRequestSynchronously()) break;
        }
    }
}

Strategy 2: Asynchronous TcpListener + Task.Run + async/await Stream I/O

Description

The second strategy follows the standard modern C# async pattern (tested in two variants: 2a One-Shot where the socket closes after one request, and 2b Keep-Alive where the loop awaits subsequent requests).

An async accept loop calls await _listener.AcceptTcpClientAsync() and dispatches each connection to the .NET ThreadPool via Task.Run(async () => ...). Inside the connection loop, socket reads (await stream.ReadAsync(...)) and response writes (await stream.WriteAsync(...)) are asynchronous, yielding the thread back to the pool while waiting for network bytes, while the HTTP parser and application Action<HttpContext> handler execute synchronously in between.

While Strategy 2b solves the idle-thread problem (idle connections hold 0 threads while paused at await stream.ReadAsync), every active request pays the overhead of allocating async state machines, Task continuations, and thread-pool context switches on every read and write—which is particularly expensive on .NET Framework 4.8.

Sample Code Block Demo for the Concept

// Strategy 2 Concept: AcceptTcpClientAsync + Task.Run + Async Stream I/O
async Task AcceptLoopAsync(CancellationToken ct)
{
    while (!ct.IsCancellationRequested)
    {
        TcpClient client = await _listener.AcceptTcpClientAsync().ConfigureAwait(false);

        // Dispatch connection loop onto the .NET ThreadPool via async Task state machine
        _ = Task.Run(async () =>
        {
            using (client)
            using (NetworkStream stream = client.GetStream())
            {
                byte[] buf = new byte[16 * 1024];
                int start = 0, end = 0;

                while (!ct.IsCancellationRequested)
                {
                    // 1. Async wait for request header bytes (0 threads held while idle)
                    HttpRequestHead head;
                    while (!TryParseHead(buf, ref start, end, out head))
                    {
                        int n = await stream.ReadAsync(buf, end, buf.Length - end, ct).ConfigureAwait(false);
                        if (n <= 0) return;
                        end += n;
                    }

                    // 2. Async read of request body (if Content-Length > 0)
                    byte[] body = await ReadBodyAsync(stream, head, buf, ref start, ref end, ct).ConfigureAwait(false);

                    // 3. Synchronous application handler execution
                    var sink = new MemorySink();
                    var ctx  = new HttpContext(new HttpRequest(head, body), new HttpResponse(sink, head));
                    HttpContext.Current = ctx;
                    try { _handler(ctx); }
                    finally { HttpContext.Current = null; }

                    bool keepAlive = ctx.Response.Complete();

                    // 4. Async flush of response bytes
                    await stream.WriteAsync(sink.Buffer, 0, sink.Length, ct).ConfigureAwait(false);

                    if (!_allowKeepAlive || !keepAlive || !head.KeepAlive) break;
                }
            }
        });
    }
}

Strategy 3: Hybrid Zero-Thread IOCP Idle State + Synchronous Worker Burst Loop (cshttp.server 1.0)

Description

Strategy 3 combines the best of Strategy 1 (zero-allocation, 100% synchronous execution during active requests) and Strategy 2 (zero threads held while waiting for idle keep-alive sockets), while avoiding a critical IOCP thread starvation trap we discovered during testing.

If a server uses _stream.BeginRead(...) (OS I/O Completion Ports) to wait for the first byte of a request and then runs the synchronous application handler directly inside the EndRead callback, any blocking database query (e.g., a 15 ms SQL query) will block the .NET ThreadPool's IOCP completion threads, stalling network I/O completions across all other sockets!

To solve this cleanly, Strategy 3 divides the connection lifecycle into three strict phases:

1. Zero-Thread IOCP Idle Wait (ArmIdleRead): Whenever a socket is newly accepted or finishes a request with no unread bytes remaining (_start == _end), it registers _stream.BeginRead(_buf, ...) with the OS and returns immediately. 10,000 idle keep-alive sockets consume 0 OS threads. A single lightweight 200 ms background timer sweeps expired idle deadlines.
2. Nanosecond IOCP-to-Worker Hand-Off (CompleteAsyncFirstRead): The instant the first packet arrives on an OS IOCP thread, EndRead records the byte count and calls ThreadPool.QueueUserWorkItem(OnRunActiveWorkerCallback, this), freeing the IOCP thread in nanoseconds so synchronous handlers never block network completions.
3. Synchronous Worker Burst Loop (RunActiveWorker): On the ThreadPool Worker thread, header parsing, body reading, HttpContext.Current assignment, the synchronous Action<HttpContext> handler, and BufferedStream flushing run 100% synchronously with zero Task allocations. Before releasing the worker thread, it checks if (_socket.Available > 0). If the client (or an Nginx upstream keep-alive pool) has already sent the next request into the kernel socket buffer, the same hot worker thread processes it immediately in a tight synchronous loop with zero context switches!

┌────────────────────────────────────────────────────────────────────────────┐
│               Strategy 3: cshttp.server 1.0 Connection Lifecycle           │
└────────────────────────────────────────────────────────────────────────────┘

  [Socket Accepted / Keep-Alive Idle]
                  │
                  ▼
  ┌───────────────────────────────────────────────┐
  │ 1. Zero-Thread IOCP Idle State (ArmIdleRead)  │
  │    • _stream.BeginRead(_buf, ...)             │
  │    • 0 OS Threads held while waiting          │
  │    • Swept by lightweight 200ms deadline timer│
  └───────────────────────────────────────────────┘
                  │
                  │  First packet arrives (OS IOCP Interrupt)
                  ▼
  ┌───────────────────────────────────────────────┐
  │ 2. IOCP Completion (CompleteAsyncFirstRead)   │
  │    • _stream.EndRead(ar)                      │
  │    • QueueUserWorkItem(RunActiveWorker)       │
  │    • IOCP thread freed in nanoseconds!        │
  └───────────────────────────────────────────────┘
                  │
                  ▼
  ┌───────────────────────────────────────────────┐
  │ 3. Synchronous Worker Loop (RunActiveWorker)  │◄──────────────┐
  │    • Sync ParseHead + Body Read               │               │
  │    • HttpContext.Current = context            │               │
  │    • Sync Handler(context) (DB / File / HTML) │   _socket.Available > 0
  │    • Sync BufferedStream Flush                │   (Zero-switch keep-alive
  └───────────────────────────────────────────────┘    or pipelined burst!)
                  │                                               │
                  ├───────────────────────────────────────────────┘
                  │  _socket.Available == 0 (Socket is now idle)
                  ▼
        [Back to Step 1: 0 Threads]

Sample Code Block Demo for the Concept

// Strategy 3 Concept: Zero-Thread IOCP Idle Wait + IOCP-to-Worker Hand-Off + Sync Burst Loop
void ArmIdleRead()
{
    _start = _end = 0;
    _idleDeadlineTicks = Now() + ToTicks(_served == 0 ? _o.RequestHeadersTimeout : _o.KeepAliveTimeout);
    _idle = true;

    // 1. Register OS IOCP async read for the first packet; holds 0 threads while idle!
    IAsyncResult ar = _stream.BeginRead(_buf, 0, _buf.Length, OnFirstReadCallback, this);
    if (ar.CompletedSynchronously)
    {
        CompleteFirstRead(ar);
    }
}

static void OnFirstReadComplete(IAsyncResult ar)
{
    if (!ar.CompletedSynchronously)
    {
        ((Connection)ar.AsyncState).CompleteFirstRead(ar);
    }
}

void CompleteFirstRead(IAsyncResult ar)
{
    _idle = false;
    int n = _stream.EndRead(ar);
    if (n <= 0) { Close(); return; }
    _end = n;

    // 2. Immediately hand off from the OS IOCP thread to a ThreadPool Worker thread
    //    so synchronous DB/file handlers NEVER block IOCP network completions!
    ThreadPool.QueueUserWorkItem(OnRunActiveWorkerCallback, this);
}

void RunActiveWorker() // Runs on a ThreadPool Worker thread
{
    while (!_server.IsStopping)
    {
        // 3. 100% synchronous ParseHead -> ReadBody -> Handler(ctx) -> Flush (zero Task allocations)
        if (!ProcessRequestSynchronously()) { Close(); return; }
        _served++;

        // If pipelined bytes are already in _buf, loop immediately on this hot worker thread
        if (_start < _end) continue;

        // Fast check: if the next Keep-Alive request is already in the OS socket buffer,
        // read and serve it synchronously right here without an IOCP round-trip!
        if (_socket.Available > 0)
        {
            _start = _end = 0;
            int n = _stream.Read(_buf, 0, _buf.Length);
            if (n > 0) { _end = n; continue; }
            Close();
            return;
        }

        // Socket is genuinely idle: release the worker thread and re-arm zero-thread IOCP wait
        break;
    }

    ArmIdleRead();
}

Benchmark Reports

Reproducible Benchmark Source Code:
All numbers and findings in this report were produced by the benchmark suite in benchmark-test (see repository subfolder for cshttp.Benchmark.csproj, Program.cs, Adapters.cs, and AlphaServerEngine.cs).

1. The Benchmark Hardware & Why Constrained Cores Tell the Truth

All stress tests in this report were executed on the following machine:

• CPU: Intel® Core™ i7-4770S @ 3.10 GHz (4 Physical Cores / 8 Logical Threads)
• Memory: 24.0 GB DDR3 RAM
• OS: Windows (x64)
• Runtimes Tested: .NET 10.0.12 (Release) and .NET Framework 4.8.9294.0 (Release)

Why a 4-Core / 8-Thread CPU is an Ideal Stress-Test Candidate

When benchmarking web server architectures on a 64-core server, hardware abundance frequently masks fundamental software design flaws. On a 4-core / 8-thread CPU, physical execution cores are finite and unforgiving. Any architectural mistake immediately surfaces in the metrics:

1. OS Thread Context-Switching Overhead: If an architecture spawns 1,500 OS threads across 8 logical processors, Windows kernel scheduler thrashing and CPU cache eviction show up immediately in p99 tail latency.
2. IOCP vs. Worker Thread Starvation: If synchronous blocking code accidentally executes on an I/O Completion Port (IOCP) callback thread with only 8 logical cores, network completion stalls across all other sockets within milliseconds.
3. Ephemeral Port & ThreadPool Injection Delays: If an architecture tears down TCP sockets per request or relies on naive .NET ThreadPool hill-climbing under blocking database loads, throughput collapses immediately.

2. Deep-Dive Architectural Observations During Testing

2.1 The "Behind Nginx" vs. "Direct Browser Exposure" Paradox

• Behind Nginx Reverse Proxy (keepalive 32): When Nginx sits in front of the C# server, Nginx terminates thousands of public client connections and multiplexes active HTTP requests over a small pool of 32 persistent loopback sockets (127.0.0.1). In this topology, Strategy 1 (cshttp.server.alpha.1.0) only ever creates 32 OS threads, pays zero thread-creation overhead per request, and performs admirably.
• Direct Exposure (--public or LAN): When browsers connect directly without Nginx, 167 users with an open browser tab (167 × 6 = 1,002 sockets) completely exhaust MaxConnections = 1000 in Strategy 1 (alpha.1.0). Even when the server is doing 0% CPU work, the 168th user is immediately refused!

2.2 The IOCP Starvation Trap Caught in Early Prototyping

While prototyping Strategy 3 (cshttp.server 1.0), our first iteration replaced the blocking keep-alive wait with _stream.BeginRead(...) (OS IOCP) and executed the synchronous request handler directly inside the EndRead completion callback.

Our stress test immediately caught a critical flaw:

• When 128 concurrent clients hit an endpoint performing a 15 ms synchronous blocking database query, running that synchronous handler directly on the thread that invoked OnFirstReadComplete blocked the .NET ThreadPool's I/O Completion Port (IOCP) threads!
• With IOCP threads blocked inside SQL queries, the OS could no longer dispatch socket read completions for other connections, causing p99 latency to spike to 3,389 ms and throughput to drop by 60%.
• Adding the strict ThreadPool.QueueUserWorkItem(OnRunActiveWorkerCallback, this) hand-off inside CompleteAsyncFirstRead eliminated the IOCP stall completely.

3. Official Stress-Test Benchmark Results

The benchmark suite tests all strategies across three distinct production workloads, on both .NET 10.0 and .NET Framework 4.8.

Workload 1: Nginx Upstream Keep-Alive Pool (32 Persistent Connections, Pure Throughput)

Simulates an Nginx reverse proxy multiplexing high-speed HTTP/1.1 traffic over 32 persistent loopback connections (127.0.0.1). Measures raw parser + dispatcher + socket I/O ceiling.

.NET 10.0 (Release)

.NET Framework 4.8 (Release)

Analysis of Workload 1:

• One-Shot Connections (2a) Collapse Under High RPS: Closing the TCP socket after each request (client.Dispose()) drops throughput from 71k–81k RPS down to 13,020 RPS on .NET 10, and causes 1,257 ephemeral port exhaustion errors (2,210 RPS) on .NET Framework 4.8. Persistent HTTP/1.1 Keep-Alive is mandatory even over 127.0.0.1 loopback.
• On .NET Framework 4.8, Strategy 3 (cshttp.server 1.0) is the fastest engine (51,701 RPS), beating async/await (43,307 RPS) by +19% because .NET 4.8 lacks modern ValueTask / pooled async state machines, making synchronous _socket.Available > 0 burst loops significantly cheaper.

Workload 2: Blocking Database Burst (128 Concurrent Clients, 15ms Sync SQL Wait per Request)

Simulates 128 concurrent connections simultaneously executing a synchronous 15 ms blocking database query (Thread.Sleep(15) inside the handler). Theoretical mathematical maximum for 128 clients with a 15 ms wait is $128 \times (1000 / 15) \approx 8,533\text{ RPS}$ across infinite cores, or ~4,200–4,400 RPS with context-switching and scheduling on 4 cores.

.NET 10.0 (Release)

.NET Framework 4.8 (Release)

Analysis of Workload 2:

• By pre-warming ThreadPool.SetMinThreads in HttpServer.Start() and immediately handing off from the IOCP completion callback to a ThreadPool worker thread, Strategy 3 (cshttp.server 1.0) achieves the highest throughput on both .NET 10 (4,423 RPS) and .NET 4.8 (4,324 RPS) when handlers block synchronously on database or file I/O.

Workload 3: The "C2000" Direct Browser Flood (1,500 Idle Keep-Alive Sockets + 50 Active Workers)

Simulates 250 users with an open browser tab (250 × 6 = 1,500 concurrent idle keep-alive sockets) while 50 active clients continuously hammer the server.

.NET 10.0 (Release)

.NET Framework 4.8 (Release)

Analysis of Workload 3 (The Decisive Test):

1. Strategy 1a (cshttp.server.alpha.1.0, MaxConn=1000) Hits Total Denial of Service (0 RPS): Once 1,000 idle keep-alive sockets connect, every single slot in alpha.1.0 is occupied by an idle thread blocked in _stream.Read(). When the 50 active clients attempt to connect, 100% of their connections are immediately rejected.
2. Strategy 1b (Raising MaxConn=2000 in alpha.1.0) Causes Severe OS Thread Thrashing: Forcing alpha.1.0 to accept all 1,500 idle sockets spawns +1,500 OS threads. On .NET Framework 4.8, scheduling across 1,550 threads causes p99 tail latency to explode to 738.43 ms (nearly three-quarters of a second for a 2-byte "OK" response!) and collapses throughput to 7,444 RPS.
3. Strategy 3 (cshttp.server 1.0) Dominates Both Strategy 1 and Strategy 2:
   • Holding 1,500 idle keep-alive connections in cshttp.server 1.0 adds only +1 OS thread.
   • While those 1,500 idle sockets sit open, the 50 active clients achieve 62,031 RPS (6.32 ms p99) on .NET 10 and 49,966 RPS (7.19 ms p99) on .NET Framework 4.8.
   • That is 4.8× to 6.7× faster than Strategy 1 (alpha.1.0), and 1.5× to 2.0× faster than Strategy 2 (async/await Paper Model at 31,758 RPS), because active keep-alive connections in cshttp.server 1.0 execute straight through on hot worker threads without allocating Task state machines on every read and write.

Conclusion: Why Strategy 3 Was Chosen for cshttp.server

The empirical stress-test data made the architectural decision clear: Strategy 3 (Hybrid Zero-Thread IOCP Idle State + Synchronous Worker Burst Loop) is the clear winner and forms the backbone of cshttp.server 1.0.

It proves that a C# web server does not need async/await infecting application handlers—nor does it need external NuGet packages like System.IO.Pipelines or System.Threading.Channels—to scale to tens of thousands of concurrent connections and deliver over 62,000 requests per second on a modest 4-core CPU:

1. When connections are idle (waiting for the first byte or between Keep-Alive requests): Strategy 3 delegates waiting to the OS kernel via asynchronous IOCP (_stream.BeginRead), holding zero OS threads across thousands of open browser sockets.
2. When data arrives: Strategy 3 immediately hands off from the IOCP completion callback to a pre-warmed ThreadPool worker thread in nanoseconds—preventing synchronous database or file I/O from ever starving network completions.
3. While a connection is actively bursting requests: Strategy 3 executes parsing, handler logic (Action<HttpContext>), response flushing, and follow-up _socket.Available > 0 keep-alive requests in a tight, 100% synchronous loop with zero Task allocations and zero context switches.

The result is a 3-file standalone server engine (HttpServer.cs, Connection.cs, ServerOptions.cs) that gives C# developers the simplicity of synchronous HttpContext.Current programming with the high-concurrency scalability of an asynchronous event-driven server—running identically on .NET Framework 4.8 and .NET 8/10+.