TLS record sizing.
Records, buffering, transport.

Static 4 KiB records — 18 October 2026 recheck

Official nginx 1.31.7 with BoringSSL 594a44291086dcc7cef06cabec4c0b47afd674c0 completed 18/18 H1/H2 transport samples across three shuffled repeats of C/D/F in the isolated 20 ms RTT, 20 Mbit/s, loss-free path. Payload hashes, actual TLS record boundaries and successful nginx SSL_write() sizes were verified.

H2 median time to the first decrypted response bytes was 28.497 ms (C), 23.061 ms (D), and 22.930 ms (F), each from only n=3 samples. This corroborates the earlier delivery direction and the mechanism, but is not an equivalence test, tail-latency claim or production tuning decision.

nginx's new RFC 9218 HTTP/2 scheduling was not benchmarked in combination with TLS record sizing. Static 4 KiB fragmentation is still an experimental direction for a tested workload; adaptive tuning, adverse-loss performance and producer-driven streaming remain outside this refresh. See docs/UPSTREAM_REFRESH_2026-10-18.md.

Hypothesis and scope

After the TLS handshake, BoringSSL's per-connection SSL_set_max_send_fragment() can produce small response records for a bounded number of successful application bytes, then restore normal record sizing. The experiment observes actual ciphertext records, not nginx buffer sizes, SSL_write() sizes, receive chunks or transport packets as substitutes.

Historical TinX adaptive record sizing inspired the question. Its implementation and numeric policy were not copied. This patch has no idle reset, third stage, configuration directive, protocol-specific HTTP/2 changes, or production patch entry. It does not change nxTLS configuration, artifact or API decisions.

Exact inputs:

• nginx 1.31.6;
• BoringSSL b75f405cde1cc3c9fb811be155eecabe38f379bb;
• Debian Clang 21.1.8, LLD, libc++/libc++abi;
• the same unmodified BoringSSL static archives for both nginx binaries.

Pinned source inspection

In third_party/boringssl/src:

• include/openssl/ssl.h:5413 declares int SSL_set_max_send_fragment(SSL *ssl, size_t max_send_fragment) and documents one on success, zero on error. It affects handshake and application records, so this spike changes it only after handshake success.
• ssl/ssl_lib.cc:1763 clamps the value to 512..16384, assigns the existing connection's field and returns 1 unconditionally. There is no handshake state restriction. ssl/s3_pkt.cc:66 reads the current field on each application-record iteration. Repeated changes between successful writes are supported by this implementation; this is not an upstream API stability claim.
• include/openssl/ssl3.h:78 defines SSL3_RT_MAX_PLAIN_LENGTH as 16384. ssl/internal.h:4140 defaults the context's limit to that value, and the connection constructor in ssl/ssl_lib.cc copies it.
• ssl/tls_record.cc:272 adds the TLS 1.3 inner content-type byte and seals records through the AEAD context. The runner measures ciphertext overhead from actual client plaintext rather than assuming a fixed numeric overhead.
• ssl/tls13_server.cc:139 skips session tickets when SSL_OP_NO_TICKET is set.

In third_party/nginx/src:

• src/event/ngx_event_openssl.c:2201: ngx_ssl_handshake() calls SSL_do_handshake(); the n == 1 branch is the successful completion path.
• src/event/ngx_event_openssl.c:3207: ngx_ssl_write() calls SSL_write() and already handles positive partial results and SSL_ERROR_WANT_READ/WRITE through NGX_AGAIN. The patch leaves those branches intact.
• ngx_ssl_send_chain() in that file buffers output and advances by the actual successful byte count; its existing n < size branch retains unsent bytes.
• src/event/ngx_event_openssl.h:117: ngx_ssl_connection_s is the existing per-connection state. The patch adds only one size_t remaining-byte counter.
• src/http/modules/ngx_http_ssl_module.c:702 defaults ssl_buffer_size to NGX_SSL_BUFSIZE (16384). This is an output buffer, not a TLS record policy.

Before creating the patch, an unmodified staged debug build served the test payload and logged SSL_write() requests and successful returns of [16384 x 8, 239]. A single normal write exceeds the early budget. Therefore the patch caps each early call at the remaining budget, leaves that budget unchanged on failed/retry returns, and decrements it only for a positive return.

Test-only constants:

• early fragment: 1024 bytes;
• early application-byte budget: 8192 bytes;
• restored fragment: SSL3_RT_MAX_PLAIN_LENGTH, currently 16384 bytes.

Reproduce

Materialize the pinned sources separately with tools/vendor, then run from the repository root:

tools/spike-tls-record-sizing

The script prints the exact configure/build commands and keeps all generated files under build/tls-record-sizing/1.31.6/<BoringSSL-commit>/. It builds BoringSSL through upstream CMake with Clang, libc++ and LLD, then stages two nginx source copies and uses upstream configure/make. Only the second copy gets nginx-1.31.6.patch. Both use identical configure options, including --with-debug for write/transition evidence; only generated paths differ. The expanded builds also enable --with-http_v2_module identically in both variants. Neither materialized upstream source tree is written to. No fetch or upstream update occurs in the script, and the experiment is not integrated into CMake.

The invoked runner command is:

python3 tests/tls_record_sizing.py \
  --upstream build/tls-record-sizing/1.31.6/b75f405cde1cc3c9fb811be155eecabe38f379bb/upstream/objs/nginx \
  --patched build/tls-record-sizing/1.31.6/b75f405cde1cc3c9fb811be155eecabe38f379bb/patched/objs/nginx \
  --repeat 3 \
  --output build/tls-record-sizing/1.31.6/b75f405cde1cc3c9fb811be155eecabe38f379bb/results.json

A/B observation method

The Python-standard-library-only runner gives both binaries the same test configuration and deterministic payload: bytes(range(256)) * 512 (131072 bytes). It explicitly uses ssl_buffer_size 16k, disables session tickets, session caching, gzip and sendfile, and uses only local Unix-domain sockets. Committed baseline certificate/key fixtures are public, insecure test-only material; certificate verification is disabled for this transport experiment, which establishes no certificate/authentication policy.

A transparent relay forwards bytes in both directions and reconstructs server records using their five-byte TLS headers and length fields. It never decrypts. The observation mark is after the client's TLS 1.3 handshake completes and before it sends the HTTP request. The client separately feeds one complete record at a time through ssl.MemoryBIO, matches the relay's capture, and measures the corresponding plaintext size. It identifies the encrypted close_notify separately; outer type 23 alone is not treated as evidence of HTTP data. Unexpected other post-handshake control records fail the experiment.

Configuration validation, startup, relay accept/connect, handshake, HTTP transaction, nginx shutdown and relay shutdown all have bounded timeouts. Receive/retry loops use monotonic total deadlines. Each case owns nginx, the relay and a private temporary runtime directory and cleans them up on exit.

Measured results

All three A/B runs passed configuration validation, TLS TLSv1.3 with TLS_AES_256_GCM_SHA384, HTTP 200, exact byte/length checks and SHA-256. Both variants returned the same body:

131072 bytes
59f410ae5e17962412e2aed4f815918f634932f2abf084f00bb638c4db017850

The record pattern was identical in runs 1, 2 and 3. Ciphertext lengths below exclude the five-byte TLS header. The final 19-byte record was identified by the client as close_notify, not response data.

The measured response-record overhead was 17 bytes for both variants. The eight small patched records each carried 1024 plaintext bytes. Their sum and nginx's transition log both show restoration after exactly 8192 successful application bytes. That budget includes HTTP headers: these runs had 239 header bytes and 7953 body bytes before restoration.

The first record after restoration carried 8192 plaintext bytes, the remainder of nginx's first 16 KiB buffer. Subsequent full buffers produced 16384-byte plaintext records. A maximum fragment size does not require every later write or record to be full-sized.

All six nginx processes exited with status 0 and were reaped. Relay threads stopped, sockets closed and temporary runtime trees were removed. SHA-256 manifests covering both materialized source trees were unchanged after builds and runs. Neither nginx binary has a dynamic libssl.so/libcrypto.so dependency.

Initial Unix-control limitations

• This measures record behavior only, not latency, TTFB, page-load time or browser rendering; it demonstrates no performance improvement.
• One static response on a fresh HTTP/1.1 TLS 1.3 connection is exercised. Keep-alive, resumption, TLS early data, other protocols/ciphers and large-scale load are not covered.
• These local runs did not force SSL_write() retry/backpressure failures. The patch preserves the inspected retry/partial-write paths, but this result is not a stress-test claim.
• Debug builds and test-only values are deliberate experimental inputs, not production defaults. Future upstream changes require renewed inspection.
• The initial sandbox run was denied nginx's Unix-socket SO_REUSEADDR operation. The successful measurements ran on the canonical host.

No accepted architecture, production sizing policy or configuration interface is established by these observations.

Expanded TCP question and matrix

The primary comparison is D versus E, separating nginx buffering, TLS fragment policy and the TCP write queue. The original patch is unchanged. It is still outside the normal CMake build and packaging, and BoringSSL is unmodified. No historical numeric policy, idle reset, MSS policy or product directive was added. The HTTP/2 responses use stock response HPACK, independently of the frozen response-compression experiments.

A/B use the inspected host's default CC and notsent values. Endpoint fq is held constant in all cases to isolate the CC/notsent/buffer differences. Thus A/B do not reproduce the host's physical-interface qdisc. F versus B isolates low notsent with CUBIC; G versus D isolates low notsent with BBR; F versus D and B versus G compare CC at fixed buffering/notsent.

Each A–E cell exercises deterministic HTTP/1.1 bodies of 1024, 32768, 131072 and 1048576 bytes, plus a 32768-byte HTTP/2 response. A–G exercise H2 competition: a 1 MiB response is active, then a 4096-byte response is requested once at least 32768 bulk body bytes have been decrypted. Record granularity can overshoot that trigger; the actual bulk count is retained. D/E additionally exercise H1 and H2 with a 128 KiB first response, 250 ms idle, then a 4096-byte response on the same connection. The one-shot policy is observed without reset.

There are 50 repetitions per timing cell, and 10 idle repetitions per case/protocol/scenario: 4920 TLS connections and 6090 verified responses in the complete scheduled matrix. All connections start fresh without session resumption. Client H2 SETTINGS and their acknowledgments settle before timing; those server application bytes still consume the adaptive connection budget. H2 stream windows are 4 MiB and the connection window is enlarged by 8 MiB so HTTP flow-control credit does not intentionally limit the 1 MiB workload. No H2 prioritization changes or priority experiments are included.

TCP topology and kernel evidence

The topology is client -- veth -- router -- veth -- server, with three unnamed network namespaces held by processes inside a disposable user/network namespace. Addresses are documentation-only 198.51.100.0/24 and 192.0.2.0/24; there is no route to the host or external networks. Namespace UID/GID 0 maps to the invoking host user. The test-only user root root nginx setting avoids its unmapped default nobody UID; it grants no host root identity. Neither named namespace mounts nor host interfaces/routes are used.

Both endpoint veth interfaces have a root fq with pacing enabled. Each router egress interface has a separate root netem, applying half the RTT, loss and the selected rate, with limit 10000 packets. There are no nested fq/netem qdiscs and no traffic-classification filters. Delay/loss occurs after the sending endpoint, away from its TCP Small Queues accounting. This preserves the endpoint's pacing queue and avoids netem replacing fq's skb control data. TSO, GSO and GRO are disabled and queried on all four veth interfaces through Linux ethtool ioctls. This makes losses act on ordinary MTU-sized packets rather than large offloaded skbs; it deliberately differs from production offloads.

The netem manual warns about TSQ interactions with sender-side netem and timer-driven rate bursts. A routed impairment stage separates it from the sender; netem rate emulation still has timer/burst limitations and is not a physical-link emulator. Effective qdiscs, counters, filters, links, routes, namespace TCP settings and offload queries are captured, rather than assuming successful tc commands prove pacing. Each trial has before/after qdisc snapshots, and seeds are captured. The two directions use successive seeds. Case order is randomized within repetition blocks. Different segmentation consumes different random draws; loss positions are not paired identically between candidates.

The running kernel is Debian 6.6.15-amd64, not the newer installed kernels. The inspected host initially exposed reno/cubic, default cubic, default qdisc fq_codel, and tcp_notsent_lowat=4294967295. Its physical interface uses mq with fq_codel children; loopback uses noqueue. Kernel configuration has CONFIG_TCP_CONG_BBR=m, CONFIG_NET_SCH_FQ=m, CONFIG_NET_SCH_NETEM=m, CONFIG_NET_NS=y, and CONFIG_USER_NS=y. BBR was installed but unregistered. BBR was explicitly loaded for the experiment; namespace TCP values change only inside the server namespace. The harness refuses to run without registered BBR and never runs modprobe or writes initial-namespace sysctls itself.

Authoritative Linux v6.6.15 sources were fetched and read, with URL/hash provenance retained alongside the results:

• include/net/tcp.h resolves a zero per-socket notsent setting through sock_net().
• net/ipv4/tcp_ipv4.c initializes the namespace threshold to UINT_MAX and tests unsent bytes for send/wakeup eligibility, using half the threshold for write-space wakeups.
• net/ipv4/tcp_bbr.c describes fq pacing and the internal TCP pacing fallback. This kernel contains upstream BBR v1; these results do not establish behavior for newer BBR variants.
• net/sched/sch_fq.c honors socket pacing and packet departure timestamps.

The current kernel sysctl documentation also describes TCP_NOTSENT_LOWAT as send/poll eligibility, rather than a hard limit on queue size or data already in flight. The runner records actual accepted-socket CC, MSS, pacing rate, cwnd, retransmissions and notsent state using ss -tinmH; it does not infer those values from namespace defaults. tcp_no_metrics_save=1 is set on the isolated endpoints to avoid cached TCP metrics carrying between otherwise fresh connections. Other TCP settings are captured and retained. The client remains on CUBIC for every server candidate.

Timing and record methodology

tests/tls_record_transport.py connects directly to the server TCP socket. There is no forwarding relay in the measured path. It reconstructs each record from its five-byte TLS header and declared length, irrespective of TCP recv or packet boundaries, and supplies one complete record at a time to ssl.MemoryBIO. The timestamp follows successful decryption using monotonic_ns(); incomplete records cannot contribute useful plaintext. This retains actual record-level evidence without a relay changing TCP queues or congestion control.

Timing begins immediately before TLS-writing the HTTP request, after TCP/TLS setup. H1 records its first decrypted response byte, then separates HTTP headers from body milestones. H2 independently follows decrypted frame headers/payloads across record boundaries to timestamp the first response-frame payload byte for each stream; hyper-h2 separately validates framing, headers and stream endings. The useful-data milestones are body byte counts 1, 1024, 4096 and 8192 when the response is that large. Header/control bytes do not advance body milestones. For H2, these are body bytes delivered through hyper-h2's DataReceived events, which wait for a complete DATA frame. Stock nginx defaults to an 8192-byte H2 chunk (src/http/v2/ngx_http_v2_module.c). Smaller TLS records can expose raw frame payload before that client event; the independent first_decrypted timestamp observes response-frame payload across partial records. H2 body milestones therefore describe this client's frame-delivery semantics, not an incremental DATA-chunk API or browser-independent body-availability claim. Completion is the final body byte; HTTP/2 additionally requires END_STREAM. There is no handshake-inclusive, browser-rendering or page-load claim.

Every body must exactly equal the deterministic fixture and match its expected Content-Length and SHA-256. Every post-handshake record is checked against its case's size bound; its ciphertext length minus decrypted length must be the observed 17 bytes. Record wire overhead includes the extra five-byte TLS header, 22 bytes per record. Positive nginx debug SSL_write() returns must sum to all observed decrypted bytes, including H2 control data. Only successful writes consume E's 8192-byte budget, with exactly one transition when exhausted. The early regime can end partway through an nginx buffer; restoration does not require the next record to be a full 16 KiB. Retried/negative write returns and transition logs are retained. No TCP packet is treated as a TLS record.

For H2 competition, a server socket snapshot is taken immediately before issuing the later request. Its subprocess cost delays that dispatch; timing for the later response starts afterward. The client records bulk bytes at dispatch and before the later body appears. The derived summary also reports dispatch and later-body completion from the original bulk request, so a candidate cannot appear better merely by making the later request ready later. Equal stock H2 scheduling is retained, so this observes head-of-line queues without adopting a prioritization policy.

Per-nginx-process user/system CPU and maximum RSS are collected on reap. Those CPU values include startup, the nsenter exec and debug logging, so they are diagnostic lifetime costs, not production TLS CPU benchmarks. Exact record counts/wire overhead are stronger evidence of sealing cost than tiny lifetime CPU differences. There is no release-load or multi-client CPU result.

Reproduce the TCP campaign

Use the unchanged pins and the two staged binaries from the build command above. Installed dependencies are Python, hyper-h2, iproute2, util-linux and the kernel's BBR/fq/netem/veth support. No package or toolchain installation is part of this experiment. Unprivileged user/network namespaces must be enabled.

tools/spike-tls-record-sizing
sudo -n modprobe tcp_bbr

The module command is needed only when BBR is not already registered. Module registration is kernel-global and must be an explicit, recorded host action; congestion-control defaults are never changed on the host. fq/netem/veth may also load globally when isolated devices/qdiscs are created.

The three commands actually used for the timing campaign are below. They ran concurrently in independent topologies, with each topology doing one trial at a time. Output directories must not exist so old samples cannot mix with a run. Running them sequentially is also supported, but should be recorded as a separate environment. No generated diagnostics belong in Git.

tools/spike-tls-record-transport --repeat 50 --idle-repeat 10 --seed 20261001 --scenarios clean --output build/tls-record-sizing/tcp-20261001-clean
tools/spike-tls-record-transport --repeat 50 --idle-repeat 10 --seed 20271001 --scenarios wan --output build/tls-record-sizing/tcp-20261001-wan
tools/spike-tls-record-transport --repeat 50 --idle-repeat 10 --seed 20281001 --scenarios adverse --output build/tls-record-sizing/tcp-20261001-adverse

Each run writes environment.json, matrix.json, topology.json, raw samples.jsonl.gz, derived summary.json, and completion.json. Failed runs leave failure.json and per-trial diagnostic logs/configurations under the ignored build directory. Trials validate nginx configuration, own all processes, have operation deadlines and remove temporary runtimes. Namespace holders are reaped at completion; destroying those namespaces removes their sysctls, interfaces and qdiscs.

Raw campaign output and host-specific diagnostics are retained separately from Git. The public repository keeps the reproducible harness, methodology and measured summary rather than committing generated result captures.

Campaign provenance records that the 4,920-connection timing run used tests/tls_record_transport.py with SHA-256 6ed288e6dd8d9ba6120c4fcfcc2ef7612ec4f3ccb9fc4cf658f02622cc42babe.

After the campaign completed, the harness entry guard was tightened so an internal-worker invocation from the host is rejected before topology creation. The post-run harness has SHA-256 ca079ce27e4a603839abf9db67b855b59d34d74c458fe82315bc0f9f0e2e7d1a. This post-run hardening did not change the measurement logic or the recorded campaign results.

Summaries use median and nearest-rank p90 only for cells with at least 50 samples (five observations in the upper decile). Sample maxima are observed maxima, not population tail estimates. No p99 is reported. Ten-sample idle observations report medians/ranges only, with no tail percentile. Independent random loss, shared-host scheduling and netem rate timers limit tiny difference claims.

Expanded evidence limits and remaining questions

This is one Linux kernel, one cipher (TLS_AES_256_GCM_SHA384), deterministic static bodies, fresh connections and one active client per topology. The three scenario topologies share an eight-CPU host. Randomized repetition blocks reduce ordering bias; they do not eliminate host scheduling noise or constitute an independent replication on another machine. Netem has symmetric constant delay, independent random loss and timer-based rate emulation; it does not model jitter, bursty loss, wireless behavior, competing flows or physical queues. No result should silently become a default for the eventual Debian release kernel.

TLS record timings are application-observed timestamps, including Python TLS client work and complete-record availability. They exclude connection setup, certificate authentication and browser rendering. The test-only public key and disabled certificate verification have no product authentication role. Shared host CPU and debug logging limit small timing and CPU conclusions. The inspected OWASP principles inform the experiment's restricted privileges, isolated topology and explicit cleanup; the harness makes no production security-conformance claim.

The H2 competition results describe stock scheduling with a receive-triggered later request and generous HTTP flow-control windows. They do not prove a prioritization policy or identical later-response readiness times: record boundaries can overshoot the trigger. Unsent socket bytes are one component of head-of-line delay; already transmitted packets, router queues and nginx/H2 output queues also remain ordered. TCP receive-window autotuning is retained, and socket snapshots expose occasions when the receive window limits sending.

The idle observation covers only 250 ms and a 4096-byte second body. It can verify that the one-shot TLS limit remains restored, but cannot rule out a separate benefit for larger responses after longer idle. No reset variant is justified merely by the historical implementation, and none is added here. The fixed-MTU campaign does not establish an MSS-derived fragment policy.

Production review would still need to settle the target kernel/BBR variant, representative workloads and acceptable latency/throughput tradeoffs. If low notsent is selected separately, its per-socket or operational integration boundary also needs review; the namespace sysctl in this harness is not a product default. Record sizing, numeric transport defaults and integration architecture remain unaccepted. The measured recommendation below is experiment evidence only.

Measured TCP results — 2026-10-01

All 4920/4920 connections and 6090/6090 responses passed. The independent raw-file verifier rechecked deterministic payload hashes, record bounds/overhead, SSL_write totals/transitions, namespace TCP settings and effective router shaping. Every observed MSS was 1448. Both endpoint fq queues had zero drops in every trial; artificial loss was confined to the routed impairment stage.

Each non-idle row below has n=50. Values are median / p90 milliseconds; minima and observed maxima, all response sizes, SSL_write retry counts and all record traces remain in the machine-readable results. No p99 is estimated.

Primary D/E comparison: HTTP/1.1, 128 KiB

E advances median first-body delivery by 1.24, 2.46 and 4.99 ms on the clean, WAN and adverse paths, but delays the median 8 KiB milestone by 17.19, 44.13 and 88.08 ms. On the adverse path its first-byte p90 improves from 211.27 to 103.48 ms: that positive result must not be hidden. The 8 KiB p90 is almost tied there (229.78 versus 228.24 ms), while its median is much worse. There is no uniform latency win across byte milestones.

The record traces explain a concrete boundary: the 8192-byte budget includes headers, leaving 7948 body bytes in the early H1 regime. The 8 KiB body milestone therefore waits for the first restored record, typically 8192 plaintext bytes. The consistent timing cliff is compatible with this record transition and TCP flight/pacing constraints; exact packet-level causality was not separately isolated. The experiment does not assume a buffer is a record.

Buffer/transport controls and complete-transfer cost

The following first-body medians for 128 KiB show why static 4k is already a strong control, and why the TCP setup changes the 16k result. A/C and B/D change both CC and notsent; their individual effects are isolated only in the H2 competition controls below.

For 1 MiB H1, complete-body time is:

E has no reproducible material 1 MiB completion advantage over D across paths. For adverse-path 128 KiB, E improves the median versus D (497.45 versus 537.22 ms), but its p90 is worse (614.99 versus 604.80 ms). Stock 16k C is better on both (478.96 / 601.26 ms). The bulk benefit is available with stock buffering and does not establish a benefit from adaptive early records.

For 1 MiB H1, record counts are invariant: C=65, D=257, E=73. TLS wire overhead is respectively 1430, 5654 and 1606 bytes. E saves 4048 bytes versus D (about 0.39% of the body), but adds 176 bytes versus C. The initial 8192 application bytes cost E eight records rather than D two: 132 extra wire bytes before the bulk-record savings.

These are diagnostic process-lifetime medians including startup and debug logging, not a release CPU/capacity result. Record count demonstrates overhead tradeoffs; adopting policy on the CPU figures alone would overstate the evidence.

HTTP/2: 32 KiB single response

E improves first decrypted payload timing but makes complete DATA-frame body delivery later on every path. On the adverse path, body-delivery p90 is 213.19 ms for E versus 122.45 ms for D. This is a real result for the hyper-h2 client, subject to the frame-buffering limitation described above.

HTTP/2 competing responses: queues matter

The later 4096-byte body completes this long after its own request:

B/F hold CUBIC and 4k buffering constant: low notsent changes completion medians from 133.42→66.00, 215.22→108.67, and 771.63→360.67 ms. G/D hold BBR and 4k constant: the same change gives 309.04→62.26, 432.14→131.49, and 1016.98→240.56 ms. Low notsent is materially relevant under multiplexing. BBR alone is not a uniform improvement: G has worse later-response latency than B, while D/F rankings depend on the path.

Median unsent socket bytes at the later-request observation are:

E often dispatches at 40960 bulk body bytes versus 32768 for D because of record/frame granularity. Its attractive clean-path later-request completion (57.72 versus 62.26 ms) is not an overall pipeline win. Completion measured from the original bulk request is:

That metric includes discovery/dispatch delay and is worse for E on every path, including p90. Current one-shot sizing has already restored normal records before the later response; a later-response difference cannot be attributed to a renewed small-record regime.

Idle observation

All 120 idle trials passed. After the first 128 KiB response and 250 ms idle, E emits a normal-regime 4337-byte H1 plaintext record for the second 4096-byte body (including headers). H2 emits 4222 or 4223 bytes. D emits 4096 plus 241 bytes for H1, and 4096 plus 126/127 for H2. No return to a 1024-byte regime occurs, and there is exactly one successful-budget transition per E connection. Ten samples per idle cell establish this observation, not a tail-latency conclusion. No idle-reset variant was created.

Recommendation for review

Static 4k is sufficient for the tested latency goal; do not adopt the current 1024/8192/16384 adaptive policy. It has a genuine first-byte benefit, including a favorable H1 loss-path first-byte p90, but that gain is not a consistent useful-body or H2 pipeline improvement. The 8 KiB transition cliff, H2 body-delivery tails and later-response chain cost outweigh the modest early median gain in this matrix. Stock 16k also captures the observed bulk benefit.

Keep stock buffering and controlled TCP queue behavior as the smallest review starting point. This recommendation accepts neither a production 4k default nor BBR/notsent numeric defaults. No production directive, build integration, BoringSSL change, idle reset or MSS-derived policy was made. The experiment patch remains as reproducible evidence. The mixed first-byte/throughput results do not trigger an MSS-derived follow-up; a future adaptive experiment would first need a workload objective and a budget transition that avoids this cliff.

Follow-up — nginx buffering versus static TLS fragmentation, 2026-10-02

This follow-up separates a 4 KiB nginx output buffer from a static 4096-byte BoringSSL fragment limit. It does not reopen the adaptive experiment or change D-015. The existing 1024/8192/16384 policy and its patch are unchanged.

Environment and isolation

The rebooted canonical host ran Linux 7.2.8+deb14-amd64, Debian kernel 7.2.8-1 (2026-09-26), x86-64, Debian Testing forky/sid. Initial and final host values were CUBIC, available reno/cubic/bbr, notsent=4294967295, default_qdisc=fq_codel, and tcp_ecn=2. BBR was already loaded and was retained. Host eth0 used mq/fq_codel and had TSO/GSO/GRO enabled, queried through read-only ethtool ioctls before and after the campaign.

Toolchain/client versions: Clang/LLD 21.1.8 (Debian 21.1.8-13), libc++/libc++abi 21.1.8, CMake 4.3.4, Ninja 1.13.2, Python 3.14.7, client OpenSSL 3.6.4, hyper-h2 4.3.0, iproute2 7.2.0, and util-linux 2.42.3. The nginx/BoringSSL pins and identical debug/H2 configure flags are those recorded above; both new binaries use the same unmodified static BoringSSL.

The three-namespace client/router/server topology, MTU 1500, endpoint fq, and separate router netem are unchanged. The clean/WAN/adverse paths retain 20/50/100 ms RTT, 0/0.1/1% loss per direction, and 20/10/5 Mbit/s. All four actual veth interfaces were queried with TSO=GSO=GRO=0, independently of host eth0. Every measured connection reported MSS 1448. Effective qdiscs, filters, routes and socket/namespace TCP state accompany every trial; all endpoint fq drop deltas were zero. The server alone selects BBR/notsent=16384; the client retains CUBIC. Both endpoints use tcp_no_metrics_save=1 and tcp_ecn=2, with other settings captured rather than silently normalized.

Authoritative Linux v7.2.8 TCP namespace source, notsent lookup, BBR implementation, fq, netem and sysctl documentation were fetched and inspected; URL/SHA-256 provenance is saved. The notsent fallback still resolves through the socket's namespace. The BBR implementation retains the original STARTUP/DRAIN/PROBE_BW/PROBE_RTT design and fq/internal-pacing support. ECN=2 permits incoming ECN but does not request it on outgoing connections. This host and fresh endpoints have tcp_limit_output_bytes=4194304, versus 1048576 in the prior campaign. These kernel/default differences prevent a causal kernel claim from the two campaigns alone.

Only veth/sch_fq/sch_netem module registration was transiently host-global; all three were unloaded after namespace cleanup. BBR's pre-existing registration, host TCP/ECN/qdisc defaults and eth0 offloads were verified unchanged. The worker continues to require a fresh network namespace with only loopback and a one-user unprivileged UID mapping. A deliberate initial-namespace invocation was rejected before creating topology or output. This follows the isolation/least-privilege principles in the OWASP Developer Guide.

Four cases and mechanism proof

F/G below are follow-up names, distinct from the prior campaign's transport controls. Raw samples/matrices identify experiment=static4k; the summarizer rejects combining experiments with overlapping case names.

nginx-1.31.6-static4k.patch only sets the existing per-connection fragment limit in nginx's successful SSL_do_handshake() branch and logs that setting. It adds no write-path change, state machine, production directive or BoringSSL edit. Pinned ssl/s3_pkt.cc iterates min(max_send_fragment, remaining_input) without returning per record unless partial-write mode is enabled; nginx does not enable that mode. A 16 KiB request can therefore seal four 4 KiB records before returning 16 KiB successfully, with ordinary retries if its BIO blocks.

Before timing interpretation, all 12 mechanism/smoke connections / 16 responses passed, including both protocols and competition. For the 128 KiB H1 response, all subsequent campaign samples reproduce these successful-write/record patterns:

Ciphertext sizes exclude the five-byte record header. Actual seal overhead is 17 bytes, hence 22 wire bytes per record: C uses 9 records / 198 overhead bytes; D/F/G each use 33 / 726. The F trace proves that one large SSL_write generates multiple small TLS records. Negative returns repeat the same request boundary; H1 median retry counts are six for D/F/G on every path, versus seven for C. Positive returns sum to all decrypted bytes for every connection. Request-size, return-size and record-size frequency distributions are preserved, including nonpositive returns; none is treated as payload failure.

H2 also has eight full 16 KiB successful writes for C/F versus 32 full 4 KiB writes for D/G, with 16 KiB records only for C. Small prelude/frame/tail writes are retained separately (18/40, occasionally combined 58; 108; 157/158 bytes). They must not be mistaken for a second fragment regime. No adaptive transition occurs, and F/G log exactly one fragment-setting success per connection.

For provenance, the follow-up rechecked the prior 128 KiB H1 early-budget arithmetic: its raw headers are 244 bytes, leaving 7948 body bytes before the 8192 application-byte transition. Its 4096-byte idle response has 241 header bytes. This arithmetic correction leaves the measured timing cliff and prior recommendation intact; the adaptive experiment was not rerun or retuned for it.

Timing and sample counts

The primary workload is the existing 131072-byte deterministic static file on fresh TLS 1.3 H1/H2 connections. No broad size matrix, resumption or early data was added. The existing H2 competition workload remains 1 MiB bulk plus a later 4096-byte response after at least 32768 bulk bytes. Case order is randomized per repetition and path; paths run concurrently in independent topologies.

The initial campaign used 20 samples per case/protocol/workload/path. Because D/F early timings were close, another 30 samples extended every D/F cell to 50. G was also extended to 50 on the adverse path to investigate loss-path completion differences; C and clean/WAN G remain at 20. The combined campaign comprises 1350 connections / 1800 responses, all successful. Smoke samples are excluded from its timing distributions. Medians and nearest-rank p90 below use n=50; n=20 controls have medians only. No p99.

Client monotonic timestamps mark request start, completion of local request send, receipt of a complete ciphertext record, and decryption/body delivery. Local send completion establishes kernel acceptance, not server arrival. Debug log line order establishes request parsing before the response's first SSL_write attempt, and the attempt/return sizes. H2's prelude writes precede its request; only writes after the request line are response diagnostics. The debug timestamps have one-second resolution, so they cannot resolve sub-millisecond differences in request availability or first-write time. There is no separate server-emission capture: complete-record receipt is a client observation, not an emission time. No intrusive tracing was added to manufacture those timestamps.

The direct TCP/MemoryBIO record reconstruction and body-integrity methodology above is retained. H1 first decrypted and first body byte coincide here. H2 first decrypted response-frame payload and hyper-h2 DATA delivery are separate: for this stock 8192-byte H2 chunk, its first-body, 4 KiB and 8 KiB body events coincide after a complete DATA frame. These are not interchangeable metrics.

D versus F: H1, 128 KiB

All rows n=50; median / p90 milliseconds.

D versus F: H2, 128 KiB

All rows n=50; median / p90 milliseconds. The body column is the coincident first-body / 4 KiB / 8 KiB frame-delivery event for this client.

Across all six primary protocol/path comparisons, D/F median first-decrypted, 4 KiB and 8 KiB differences are less than 0.1 ms. Clean/WAN single-response completion medians also differ by less than 0.1 ms. These observations reproduce the early-delivery effect with F's larger nginx buffer.

Tail equality is not established. For example, adverse H1 4 KiB p90 is 233.21 ms for D versus 115.40 for F, while H2 completion p90 is worse for F there (562.87 versus 532.63). Seven D versus five F adverse H1 trials exceed 180 ms at the 4 KiB milestone; crossing the nearest-rank boundary strongly affects n=50 p90. F also has fewer affected WAN H2 trials and a better completion p90. These real sample results are retained, rather than asserting universal equality or choosing a winner from inconsistent loss-path tails.

H2 later response and queue observations

Later-response completion from its own request, median / p90 ms for D/F. G has n=20 on clean/WAN (median only), n=50 on adverse.

All D/F/G competition dispatches occur at 32768 received bulk body bytes; no adaptive-budget overshoot is involved. Median unsent socket bytes at dispatch are D/F/G 28960/26064/26064 on clean and 21720/21720/21720 on both lossy paths. Completion from the original bulk request additionally includes discovery and socket-snapshot/dispatch delay:

The clean/WAN later-response medians are close. On adverse, F's own-request completion is slower than D (341.92 versus 318.33 ms), and its chain completion is also slower. Static fragmentation is not a uniform multiplexing improvement.

G control investigation and limits

G reproduces D's 4 KiB write/record mechanism and matches its early single-response medians: the largest G/D difference is below 0.1 ms across the six primary cells. Its adverse completion medians differ materially, however: H1 D/F/G are 488.58/497.47/460.54 ms; H2 498.45/484.81/461.89 ms.

The G adverse cells were extended to 50 before interpretation. Inspection finds no extra fragment transition, different TCP setting, offload, endpoint drop or unexplained record-size regime. In pinned BoringSSL's application writer, min(max_send_fragment, 4096) selects 4096 for both D and G; other fragment-limit uses concern handshake output, with this setting applied after handshake and tickets disabled. Observed queue/retry behavior is also similar.

The loss realization remains an important confound. G's adverse single H2 trials have 65 total router drops versus D 85 / F 87, with 14 zero-drop trials versus 6 / 8. G's initial/extension later-response medians move from 266.64 to 344.02 ms, while D moves 301.76 to 318.33. The faster G ranking in the initial phase is not stable. For the post-hoc zero-router-drop primary trials, D/F/G completion medians differ by less than 0.25 ms (H1 around 453 ms; H2 around 432 ms). This is a diagnostic, not a controlled loss-free comparison; router counts span handshake/prelude/transfer/teardown, and zero-drop subsets are small. Loss timing, not just count, affects recovery, and seeds are not paired packet-loss positions. These checks explain why completion differences cannot be attributed confidently to a G interaction. A complete-transfer tail claim would need a separate controlled loss-location/recovery experiment; none is silently added here.

C/D kernel revalidation

128 KiB H1 medians in ms; prior C/D n=50, current C n=20 / D n=50.

The earlier-delivery direction for stock 4k remains qualitatively stable on all paths. The previous adverse 128 KiB C bulk advantage does not persist in this sample (current C 562.11 versus D 488.58 ms); completion is loss-sensitive and the control has only 20 samples. This is revalidation of observed direction, not proof that the kernel caused any timing or ranking change.

Measured recommendation and remaining review

Outcome A for the tested early-delivery question: static 4096-byte BoringSSL fragmentation with nginx's 16 KiB buffer reproduces the median early-delivery behavior of stock 4 KiB buffering. The data do not show a material additional early benefit from nginx's smaller buffer for an already available static file. Loss-path tails/full completion are mixed, so this is neither a universal latency equivalence claim nor evidence that F is a better production choice.

Architecture review still must compare stock zero-patch configuration with nxTLS-owned static TLS policy, integration/maintenance cost and configuration semantics. Streaming/proxy producer timing, browser/incremental H2 delivery, long-lived/resumed connections, multi-client load and production CPU costs are unmeasured. Debug/startup CPU is diagnostic only. No release default, directive, normal-build integration, adaptive retuning, 0-RTT or HTTP/3 decision is made. D-015 remains authoritative and unchanged.

Reproduction, raw evidence and validation

All new raw artifacts remain under the Git-ignored directory experiments/tls-record-sizing/results/2026-10-02-static4k/: compressed per-trial records/timestamps/log ordering, configurations, socket/qdisc snapshots, completion markers, independent analysis.json, mechanism proof, loss diagnostics, source snapshots/hashes, Linux-source provenance, regression evidence and host restoration. No raw results are added to Git. Timing code was frozen before smoke and remained identical through both measurement phases. The subsequently expanded offline summarizer is separately saved/hashed; there was no post-run timing-harness hardening. SHA256SUMS covers retained artifacts.

The exact build/campaign commands executed from the repository root were:

tools/spike-tls-record-sizing static4k
tools/spike-tls-record-transport static4k --repeat 1 --idle-repeat 0 --scenarios clean --compete --seed 20261002 --output experiments/tls-record-sizing/results/2026-10-02-static4k/smoke
tools/spike-tls-record-transport static4k --repeat 20 --idle-repeat 0 --scenarios clean --compete --seed 20261002 --output experiments/tls-record-sizing/results/2026-10-02-static4k/initial-clean
tools/spike-tls-record-transport static4k --repeat 20 --idle-repeat 0 --scenarios wan --compete --seed 20271002 --output experiments/tls-record-sizing/results/2026-10-02-static4k/initial-wan
tools/spike-tls-record-transport static4k --repeat 20 --idle-repeat 0 --scenarios adverse --compete --seed 20281002 --output experiments/tls-record-sizing/results/2026-10-02-static4k/initial-adverse
tools/spike-tls-record-transport static4k --repeat 30 --idle-repeat 0 --scenarios clean --cases D F --compete --seed 20361002 --output experiments/tls-record-sizing/results/2026-10-02-static4k/extend-clean
tools/spike-tls-record-transport static4k --repeat 30 --idle-repeat 0 --scenarios wan --cases D F --compete --seed 20371002 --output experiments/tls-record-sizing/results/2026-10-02-static4k/extend-wan
tools/spike-tls-record-transport static4k --repeat 30 --idle-repeat 0 --scenarios adverse --cases D F G --compete --seed 20381002 --output experiments/tls-record-sizing/results/2026-10-02-static4k/extend-adverse
tools/summarize-tls-record-transport experiments/tls-record-sizing/results/2026-10-02-static4k/initial-clean experiments/tls-record-sizing/results/2026-10-02-static4k/initial-wan experiments/tls-record-sizing/results/2026-10-02-static4k/initial-adverse experiments/tls-record-sizing/results/2026-10-02-static4k/extend-clean experiments/tls-record-sizing/results/2026-10-02-static4k/extend-wan experiments/tls-record-sizing/results/2026-10-02-static4k/extend-adverse --output experiments/tls-record-sizing/results/2026-10-02-static4k/analysis.json

Both build modes completed successfully. The independent verifier rechecked all 1350 raw trials / 1800 responses. Smoke separately verified 12 / 16. The original adaptive Unix regression remained stable across all three A/B runs; CTest passed 1/1 baseline.runtime. Both static-fragment experiment binaries have no dynamic libssl/libcrypto dependency. All 9181 vendored input files matched their pre-task SHA-256 manifest. Exact shell/build output and invocation metadata remain in the ignored artifacts.

Additional successful validation commands:

tools/spike-tls-record-sizing
ctest --test-dir build/baseline --output-on-failure
python3 -m py_compile tests/tls_record_transport.py tools/summarize-tls-record-transport
bash -n tools/spike-tls-record-sizing tools/spike-tls-record-transport
sudo -n modprobe -r sch_netem sch_fq veth
sha256sum --check SHA256SUMS  # from the raw-artifact directory
git diff --check
git status --short
git diff --stat

The deliberately rejected worker command and its expected status are saved in guard-check.json; it is a safety test, not a failed transport sample. All namespace holders and nginx processes were reaped.