experiments/hpack-response-encoding/README.md. The README remains the canonical report text; this page changes presentation, not evidence.Upstream refresh — 18 October 2026
The official nginx 1.31.7 release and pinned BoringSSL 594a44291086dcc7cef06cabec4c0b47afd674c0 were used for a fresh stock-versus-experiment HTTP/2 matrix. The reviewed 1.31.7 HPACK patch changed its hunk context, not its substantive added/removed encoder code; it applied without fuzz or offsets.
| Revalidation | Stock nginx | Experimental HPACK |
|---|---|---|
| Runtime coverage | 11 cases · 220 responses | 11 cases · 220 responses |
| Stable repeated header block | 204 B throughout | 204 B first, then 107 B |
| Response integrity and HPACK decoding | PASS | PASS |
| Fallbacks and state checks | Stock control | PASS · zero/limited capacity, eviction, changing values and generated fields |
nginx 1.31.7 also includes RFC 9218 HTTP/2 prioritization, but this matrix did not test its stream scheduler or any combined prioritization/TLS-record policy. The result retains the experimental response-reuse direction without selecting its production integration.
Canonical refresh evidence: docs/UPSTREAM_REFRESH_2026-10-18.md. The original October 1 report and its nginx 1.31.6 identifiers below remain historical, unchanged in scope.
Inputs and provenance
Implementation facts come from the materialized nginx 1.31.6 source pinned
in third_party.lock, inspected at nxTLS commit 96f5ad0. Its archive SHA-256 is
974ed5298a5e398e008704ed5db284e655fc270c596493dbccada452448fc9f1.
The runtime observation uses pinned BoringSSL
b75f405cde1cc3c9fb811be155eecabe38f379bb.
Historical evidence comes from the immutable private nsb2_tinx.zip archive,
read only. Archive SHA-256:
1ad24371f7bfabb8149f296f8e2fad7e83a66f776f6e6d76cf489a2ca2c2fde6.
Historical paths below identify archive members, not dependencies of the public
build. The ZIP contains no Git/Mercurial history or standalone patch files;
the surviving source and Debian changelog do not identify every intermediate
revision. No archive source code is copied into this repository.
External sources consulted:
- Vlad Krasnov's nginx-devel patch, 23 June 2017,
changeset
895cea03ac21fb18d2c2ba32389cd67dc74ddbd0. This is the dated reference for original behavior. - Cloudflare's nginx 1.13.1 HPACK patch, a corroborating copy on a mutable branch, not proof of TinX's exact import.
- July 2017 nginx-devel review, which raised the inline encoder's additional per-connection memory cost and configurability. It does not establish why all later nginx versions lack it.
- Upstream zero-table changeset
12cadc4669a7, 30 August 2017, preserved in the freenginx history mirror. Its stated reasons were handling smaller peer table limits correctly and releasing client decoder memory when nginx does not reuse dynamic entries.
Original patch and surviving TinX mechanisms
The original patch added an optional response encoder enabled by
--with-http_v2_hpack_enc / NGX_HTTP_V2_HPACK_ENC. It introduced connection
state, static/dynamic name lookup, full name/value lookup, insertion, eviction,
indexed reuse and peer-sized resize signaling. It routed ordinary headers and
trailers through ngx_http_v2_write_header. Huffman string encoding and common
static status representations already existed; they were not the new dynamic
compression mechanism. With the encoder switch disabled, the patched helper
fell back to incremental-indexing literals with new names, without maintaining
an encoder table. See the original patch.
The following mechanisms survive in TinX and match the corresponding algorithms
and constants in the Cloudflare patch.
Here H/ means archive member prefix tinx/src/http2/.
| Mechanism | Surviving TinX implementation and source location |
|---|---|
| Separate response state | H/http2.h:113-147,164-196: TNX_HTTP2_HPACK_ENC_ENTRY, TNX_HTTP2_HPACK_NAME_ENTRY, TNX_HTTP2_HPACK_ENC, and connection member hpack_enc; distinct from request decoder hpack. |
| Default and maximum | H/http2.h:52-58: default 4096 bytes, maximum 16384; 128 hash slots, at most 100 entries, 10 cached names. Storage is 16384 + 512 bytes, separate from metadata. H/http2.cc:3148-3162 zero-initializes state and sets the default. |
| Entry limit | H/http2_table.cc:478-488, tnx_http2_table_encode_strings: entry cost is name length + value length + 32; no insertion above 512 bytes or the peer-derived cap. The 512 limit includes that 32-byte cost. |
| Static-name lookup | H/http2_table.cc:333-471, hpack_get_static_index: a length-filtered dictionary of 28 response header names with static indices and case-folded comparisons. This is name lookup, not arbitrary static full-field matching. |
| Dynamic-name lookup | H/http2_table.cc:313-328, hpack_get_dynamic_index: a 10-name cache, hash/byte comparison and an eviction-age check; returns top - index + 61. |
| Full name/value lookup | H/http2_table.cc:490-529: seeded 64-bit MurmurHash2, at most eight candidate hash slots, lengths and byte comparisons; a full hit returns a dynamic index. Hash helper: tinx/src/core/tnx_hash.c:756, declaration tinx/src/core/tnx_hash.h:108. |
| Insertion and eviction | H/http2_table.cc:534-607: evict oldest inserted entries until byte/entry limits permit insertion, maintain linked order and cyclic storage, update name cache. A lookup does not promote a full entry. Unavailable hash slots cause a literal fallback. |
| Wire representations | H/http2_table.cc:623-684, tnx_http2_write_header: full hits emit indexed fields (0x80, seven-bit prefix); insertions emit literals with incremental indexing (0x40, six-bit name index); failed insertions emit literals without indexing, using an available name index or a new name. |
| Huffman/raw strings | H/http2_filter.cc:99-118, tnx_http2_string_encode; H/huff_encode.cc:10-61 and H/huff_encode.h: use Huffman only when shorter, otherwise raw; names are lowercased for transmission. Full indexed hits transmit neither name nor value strings. |
| Peer settings and resize | H/http2.cc:2520-2528: cap SETTINGS_HEADER_TABLE_SIZE at 16384 and set indicate_resize. H/http2_filter.cc:1640-1646: signal the selected size before ordinary response fields and call tnx_http2_table_resize (H/http2_table.cc:611-620) to evict excess entries. Zero prevents insertion. |
| Ordinary response routing | H/http2_filter.cc:1650-1742 routes generated fields and general response headers through the encoder. Known statuses still use static indexed representations. H/http2.h:339-354 supplies string/header helper macros. |
These are source observations, not a claim that the archived encoder was fully correct in every setting sequence or response path.
Table limits are compile-time constants and a peer-derived connection cap,
not an operator-configurable directive in H/http2_module.cc:283-369.
With no peer table-size setting, the encoder starts with its 4096-byte default.
Identifiable later changes and refactoring
Archive tinx/debian/changelog records:
| Version/date | Entry/location | What can be established |
|---|---|---|
| 2.0.0-40, 2017-07-03 | Line 423: "Import HPACK patch from cloudflare" | Explicit import provenance; no exact patch revision recorded. |
| 2.0.0-76 / -77, 2018-01-31 | Lines 173 / 167: trailer-frame fix / revert of -76 | A fix was attempted and reverted; its precise diff is unavailable. |
| 2.0.0-79, 2018-01-31 | Line 155: settings-parser fix | Settings handling changed later; the changelog does not specify the fix. |
| 2.0.0-82, 2018-02-13 | Line 135: HTTP/2 push added | Later response-path integration, not evidence about the original patch. |
| 2.0.0-85, 2018-03-19 | Lines 109-110: upstream trailers support and static name/value accessors | Accessors survive at H/http2_table.cc:79-84; they are distinct from encoder name lookup at line 333. |
| 2.0.0-87, 2018-05-02 | Line 89: BUGFIX: revert "HTTP/2: signal 0-byte HPACK's dynamic table size". |
Explicit deliberate revert; see below for its surviving implementation and limits of causal evidence. |
Observable differences from the June patch:
- TinX's encoder state and call paths are unconditional; the original compile
switch is absent.
H/CMakeLists.txtbuilds its HTTP/2 objects;tinx/src/CMakeLists.txtandtinx/CMakeLists.txtinclude/link them. - TinX trailers use new-name literals without indexing in
tnx_http2_create_trailers_frame,H/http2_filter.cc:286-384, rather than the dynamic helper used by the original patch's trailer change. The snapshot establishes this difference; the fix/revert entries do not establish which subsequent edit produced it. - TinX adds representation-specific encoder debug messages
(
H/http2_table.cc:627-677) absent from the original helper. - C-to-C++ types/casts,
ngx_*totnx_*renaming, reorganized files/functions, andserver: TinXare refactoring/product differences. They do not establish a new compression algorithm.H/internal.hretains request-table interfaces.
Meaning of the zero-table revert
The August upstream change initially set table_update = 1 on connection
initialization and emitted a zero-size update before response fields. That was
appropriate for its encoder with no dynamic reuse, according to the
changeset's explanation.
TinX still initializes that flag (H/http2.cc:3161), but the ordinary response
zero-update emission is commented out (H/http2_filter.cc:1632-1638), as is
the PUSH_PROMISE emission (1094-1100). An old push allocation-size allowance
also survives (1081); it does not emit a byte. The ordinary header allocation
was adjusted (1459-1461). The live indicate_resize path instead advertises
the peer-derived encoder size and resizes the actual encoder table.
Thus the changelog and snapshot establish a deliberate removal of unconditional
zero signaling alongside a real response dynamic encoder. Author recollection
(2026) identifies the triggering bug as the zero-size update emitted after an
explicit peer SETTINGS_HEADER_TABLE_SIZE: TinX intended to retain and use its
response-side dynamic encoder, while that update disabled the peer's response
dynamic table. The archive independently establishes the relevant mechanics,
but the original error-log symptom, exact intervening diff and historical
debugging sequence have not survived. This is not evidence of page-load or
"stuttering" improvements.
Exact pinned nginx 1.31.6 behavior
Here N/ means third_party/nginx/src/, and V/ means
N/src/http/v2/. Line numbers refer to the pinned materialized source.
Request decoding
nginx maintains a request decoder dynamic table: ngx_http_v2_hpack_t and
connection member hpack in V/ngx_http_v2.h:106-149.
ngx_http_v2_state_header_block (V/ngx_http_v2.c:1380-1463) parses indexed
fields, incremental-indexing literals and table-size updates.
ngx_http_v2_state_process_header:1761-1767 inserts incoming indexed literals.
V/ngx_http_v2_table.c implements indexed lookup (104-184), insertion
(188-298), oldest-entry eviction/accounting (303-331) and resize
(336-362), with a 4096-byte maximum (13). These operations decode client
request headers; they do not compress server responses.
Response encoding
There is no response dynamic encoder table and no dynamic indexed reuse.
The connection structure (V/ngx_http_v2.h:125-177) has decoder state and
table_update, but no encoder entries/name cache. The response paths below
write fields directly. V/ngx_http_v2_encode.c:17-68 only serializes strings
and integers. Inspection/search of HTTP/2 sources and N/auto/{options,modules}
found no alternate response table implementation or HPACK encoder build switch.
Module configuration (V/ngx_http_v2_module.c:74-182 and
V/ngx_http_v2_module.h:17-26) also supplies no response encoder table option.
Incremental-indexing literals nevertheless exist.
V/ngx_http_v2.h:371-372 defines indexed(i) as 128 + i and
inc_indexed(i) as 64 + i. The latter is a literal with incremental indexing,
not an indexed full field. Generated response fields use it in
ngx_http_v2_header_filter, V/ngx_http_v2_filter_module.c:
| Representation | Fields / static indices | Pinned source lines |
|---|---|---|
| Indexed static full field | :status 200/204/206/304/400/404/500, indices 8-14 |
Filter 166-210; header definitions ngx_http_v2.h:394-400 |
| Incremental literal, static name | Other statuses (name index 8), server (54), date (33), content-type (31), content-length (28), last-modified (44), location (46), vary (59) | Filter 440,462,493,499,541,551,572,582; header definitions 394-409 |
| Literal without indexing, new name | General response header list, including custom headers; no general static-name search | Filter 587-623, specifically 616-622 |
| Literal without indexing, new name | Trailers | ngx_http_v2_create_trailers_frame, filter 949-1058, specifically 1049-1055 |
There is no later dynamic full-field or dynamic-name reference to these fields. If the peer retains a nonzero decoder table, incremental literals can populate it, but nginx does not exploit those entries. With size zero they cannot remain in that table. This is compatible with sending incremental literals and having no local encoder table.
Huffman behavior is selective, not universal.
V/ngx_http_v2_encode.c:23-39 tries ngx_http_huff_encode, else writes raw.
N/src/http/ngx_http_huff_encode.c:192-253 rejects encoding that is not shorter.
Generated server version/build strings use this helper/cache (filter 464-481);
server: nginx with tokens off and vary: Accept-Encoding use pre-encoded
Huffman constants (filter 123-127,484,583). Date, content-type, last-modified,
location and general literal names/values use the helper (494,530,564,573,
618-622); normal HTTP dates take the Huffman path. Generated content-length
digits (543-545) and uncommon/103 status digits (440-442,751-753) are raw.
Static indexed fields transmit no value string. Explicit header-list overrides
follow the general literal path rather than these generated-field shortcuts.
Exact zero-update condition: receiving a peer
SETTINGS_HEADER_TABLE_SIZE sets h2c->table_update = 1, regardless of its
value (V/ngx_http_v2.c:2261-2264); nginx does not retain the advertised value
for a response encoder. The next ordinary response header filter checks the
flag, emits 0x20 (Dynamic Table Size Update, value zero), then clears it
(filter 425-429). The 103 early-hints filter does the same (740-744).
Trailers do not check this flag. Another such setting can schedule another
zero update; it is not an unconditional per-response prefix.
With no HEADER_TABLE_SIZE setting, the zero-initialized connection
(V/ngx_http_v2.c:233-247) leaves table_update clear: no zero-size update is
sent on this path, but there is still no dynamic response reuse. The peer's
default table capacity is not an nginx encoder table. Do not generalize the
2017 initialization behavior to this pin.
The absence is intentional in the historical upstream explanation, and directly
observable in the pinned implementation; it is not a dormant table waiting for
a configure switch. N/auto/options:248,484 provides HTTP/2 module enablement;
N/auto/modules:232-236,441-455 selects the filter and HTTP/2 sources, with no
separate dynamic encoder. This establishes facts about this pin, not an upstream
promise never to implement dynamic encoding.
Feature matrix
Response encoding only. Evidence locations are detailed in the sections above; Cloudflare means the June patch with its encoder switch enabled.
| Feature | Cloudflare 2017 | Surviving TinX | nginx 1.31.6 |
|---|---|---|---|
| Static indexed status fields | Present | Present | Present, indices 8-14 |
| Literal values | Present | Present | Present |
| Huffman strings when shorter | Existing helper retained | Present | Present; generated length/status digits raw |
| Response dynamic table | Optional encoder state | Unconditional encoder state | Absent; request decoder table is separate |
| Incremental indexing | Successful insertions | Successful insertions | Generated fields, despite no local table |
| Full-header dynamic indexed reuse | Present | Present | Absent |
| Dynamic header-name reuse | Name cache | Name cache | Absent |
| General static-name lookup | 28-name dictionary | 28-name dictionary | Only hardcoded generated-field indices |
| Peer HEADER_TABLE_SIZE | Selected cap, maximum 16384 | Same selected cap | Selects zero after any explicit setting |
| Encoder eviction | Oldest entries, byte/count limits | Same mechanism | No encoder entries to evict |
| Table-size update | Selected peer-bounded size | Live resize; unconditional zero disabled | Zero when pending setting flag is consumed |
| Trailer dynamic helper | Used | Bypassed: literals without indexing | Literals without indexing |
Runtime observation
Installed tools: nghttp and h2load, nghttp2 1.70.0; no packages or
dependencies were added. Only nghttp was needed. The canonical CMake baseline
does not enable HTTP/2, so an unmodified staged copy was built with HTTP/2
and debug logging for this observation. Root CMake and vendored sources were
not changed. Existing static BoringSSL artifacts from the committed TLS-record
experiment were reused at the same pin.
From the repository root, staging/configure/build commands were:
mkdir -p build/hpack-response-encoding/source
cp -a third_party/nginx/src/. build/hpack-response-encoding/source/
cd build/hpack-response-encoding/source
./configure \
--builddir=/home/daniel/code/nxtls/build/hpack-response-encoding/objs \
--with-cc=clang --with-debug --with-http_ssl_module --with-http_v2_module \
--with-cc-opt=-I/home/daniel/code/nxtls/third_party/boringssl/src/include \
'--with-ld-opt=-L/home/daniel/code/nxtls/build/tls-record-sizing/1.31.6/b75f405cde1cc3c9fb811be155eecabe38f379bb/boringssl -fuse-ld=lld -Wl,--no-as-needed -lc++ -lc++abi' \
> /home/daniel/code/nxtls/build/hpack-response-encoding/configure.log 2>&1
make -s -f /home/daniel/code/nxtls/build/hpack-response-encoding/objs/Makefile -j8 \
> /home/daniel/code/nxtls/build/hpack-response-encoding/build.log 2>&1
build/hpack-response-encoding/objs/nginx -V reported nginx/1.31.6, Clang
21.1.8, BoringSSL at build/runtime, and the configure arguments above.
readelf -d build/hpack-response-encoding/objs/nginx showed libc++, libc++abi,
PCRE2, zlib, libgcc_s and libc dependencies; no dynamic libssl/libcrypto.
An ad hoc Python-standard-library process wrapper in ignored build output
reserved an ephemeral loopback port, created private temporary runtime state,
copied the public/insecure TLS fixtures from tests/fixtures, validated and
started nginx, ran the existing client, and stopped/reaped nginx in finally.
It implemented no HTTP/2 or HPACK protocol code. Invocation:
python3 build/hpack-response-encoding/observe.py \
> build/hpack-response-encoding/observation.log 2>&1
The sandbox's first attempt was denied at socket creation; the same command passed with permission to run outside the socket-restricted sandbox. The wrapper is an ignored work product, not a new committed runner or dependency.
Runtime configuration used daemon off, master_process off, debug error
logging, 128 worker connections, server_tokens off, http2 on,
listen 127.0.0.1:55929 ssl, TLSv1.3 only, session tickets off, etag off,
and a private static root. /payload.txt contained the 24-byte string
nxTLS HPACK observation\n. Repeated fields on every response were:
x-experiment-alpha: repeat-alpha-0123456789
x-experiment-beta: repeat-beta-0123456789
x-experiment-gamma: repeat-gamma-0123456789
The wrapper executed these exact server/client commands:
/home/daniel/code/nxtls/build/hpack-response-encoding/objs/nginx -p /tmp/nxtls-hpack-2tblhm_x/ -c nginx.conf -t
/home/daniel/code/nxtls/build/hpack-response-encoding/objs/nginx -p /tmp/nxtls-hpack-2tblhm_x/ -c nginx.conf
/usr/bin/nghttp -nv -y -t 5 -m 20 --hexdump --no-push https://127.0.0.1:55929/payload.txt
/usr/bin/nghttp -nv -y -t 5 -m 20 --hexdump --no-push -c 4096 https://127.0.0.1:55929/payload.txt
/usr/bin/nghttp -nv -y -t 5 -m 20 --hexdump --no-push -c 1024 https://127.0.0.1:55929/payload.txt
/usr/bin/nghttp -nv -y -t 5 -m 20 --hexdump --no-push -c 0 https://127.0.0.1:55929/payload.txt
Each client invocation used one TLS HTTP/2 connection, with 20 concurrent
GET streams (IDs 1,3,5,...,39), not 20 separate connections. All received 200,
all three repeated fields and a complete 24-byte DATA frame. ALPN negotiated
h2; nginx debug logs confirmed TLSv1.3. All four clients exited zero.
Configuration validation passed; nginx exited zero after SIGQUIT and the
temporary runtime directory was removed. Client operation timeout was five
seconds, with a 15-second process bound and five-second lifecycle bounds.
Client SETTINGS in all runs: MAX_CONCURRENT_STREAMS=100,
INITIAL_WINDOW_SIZE=65535, NO_RFC7540_PRIORITIES=1, ENABLE_PUSH=0.
The default run omitted HEADER_TABLE_SIZE; the other runs explicitly sent the
values shown below. The -c option controls the client's response decoder,
not its request encoder.
| Client table setting | HPACK bytes, response 1 | HPACK bytes, each response 2-20 | Zero-update debug count |
|---|---|---|---|
| Omitted (default 4096) | 181 | 181 | 0 |
| Explicit 4096 | 182 | 181 | 1 |
| Explicit 1024 | 182 | 181 | 1 |
| Explicit 0 | 182 | 181 | 1 |
This reports every response size: streams 3 through 39 in each row all had the
same listed length. Every received HEADERS had flags 0x04 (END_HEADERS),
without PADDED/PRIORITY, and there were no CONTINUATION frames. Thus its
payload length is the entire compressed HPACK block, excluding the nine-byte
frame header. These are not TLS-record or TCP-packet measurements.
Decrypted client hexdumps show first-block prefixes 88 76 ... when the setting
is omitted and 20 88 76 ... when explicit. 0x20 is the zero table update;
0x88 is static indexed status 200. nginx debug logs record http2 setting 1:4096 / 1:1024 / 1:0, followed once per explicit-setting connection by
http2 table size update: 0; they also record the repeated custom fields and
HEADERS lengths 182/181 (or 181 throughout). There is no material shrinkage
from later dynamic reuse; the one-byte difference is only the initial update.
For contrast, client request HEADERS shrank from 56 to 18 bytes, consistent with
the separate request decoder accepting client dynamic indexing.
Ignored local work products live in build/hpack-response-encoding/:
configure.log, build.log, runtime-config.txt, runtime-commands.json,
observation.log, nghttp-{default,4096,1024,zero}.log, nginx-debug.log,
process.log, observe.py, staged sources and build objects, and before/after
SHA-256 manifests. Manifests matched for all 521 nginx files and 8660
BoringSSL files; the historical archive hash also remained unchanged.
Missing capability, limitations and next-experiment gate
Missing from this nginx pin: a response encoder table with insertion/eviction, full name/value dynamic indexed reuse, dynamic name reuse, and general static name lookup. Literal emission and Huffman encoding are already present. The historical dynamic mechanisms could reduce repeated response-header block sizes; this comparison does not measure a patched encoder or predict savings.
Limits: one synthetic static endpoint on the canonical Debian host, four initial setting variants, no historical TinX execution, no mid-connection resize/eviction/trailer/push stress testing, and no CPU/memory/latency/browser benchmark. The archive lacks intermediate diffs and the original error-log symptom for the zero-update bug. Legacy code and its best-effort caches have not been audited for current correctness or security. Historical recollection about stuttering or page load remains separate from the established HPACK bug recollection and from measured performance evidence.
Recommendation C: a focused implementation A/B spike is justified for the next experiment only. A concrete response dynamic-table capability is absent, and existing tooling can measure header blocks on repeated streams. Compare unmodified nginx with one isolated experimental encoder change; measure block bytes and memory/CPU trade-offs, and verify table-limit/resize/eviction behavior, changing values, oversized fields and response-path synchronization. Do not assume the old implementation can be ported unchanged. This recommendation does not select a production encoder, upstream patch, integration boundary or policy.
No production decision is made. CMake, upstream sources, accepted decisions and public history remain unchanged.
Implementation spike — 2026-10-01
Experimental mechanism only; not a production fix or adoption. Tested at
nxTLS HEAD 8d9074b61df1ad7a88fc033049a2333fe0f3cc50, with the four-file
uncommitted experiment change described here. The preceding archaeology and
unmodified runtime evidence retain their original scope and provenance.
nginx/BoringSSL pins are unchanged. No historical implementation was ported.
Patch and synchronization model
nginx-1.31.6.patch changes only four files in a staged nginx source tree:
src/http/v2/ngx_http_v2.{h,c}, ngx_http_v2_encode.c, and
ngx_http_v2_filter_module.c. It adds a separate outbound table, connection
capacity/pending-minimum state, encoding helpers, narrow response-filter hooks
and connection cleanup. The inbound hpack decoder is untouched.
Only ordinary response-list names beginning with x-nxtls-hpack- (case
insensitive before wire lowercasing) enter the dynamic reuse helper. A linear
newest-first scan compares exact name/value bytes; full hits emit indexed fields,
name-only hits use a dynamic name index in an incremental literal. New entries
use lowercased names and exact uncompressed values. The first dynamic index is
62, after the 61-entry static table. Hits do not promote entries. FIFO eviction
uses uncompressed name length + value length + 32.
Stock generated incremental literals also occupy the peer's table. Ignoring them would make eligible indexes incorrect even on response 2. The patch records their exact entries/accounting in wire order, without dynamically reusing them or changing their existing bytes: uncommon status, server, date, content-type, content-length, last-modified, location and gzip vary. This bookkeeping does not create an indexing policy for these headers; nginx already emits incremental literals for them. Stock oversized generated literals clear the mirrored table, as they clear the decoder table. Table capacity zero retains no entries.
Each ordinary block is encoded against a heap-owned working copy of the last queued table. Entry allocation/copy completes before eviction. Failed eligible insertion emits the original new-name, non-indexing literal, leaving the working table unchanged. Capacity-zero/oversized eligible entries use the same fallback. Excluded headers go directly through the stock literal path. Failure to create the working table, copy it, or record a generated entry aborts the block before queueing; request-pool cleanup frees the working copy. There is no knowingly unrecorded new experimental incremental literal.
After complete HEADERS/CONTINUATION frame construction, nginx's existing blocked frame queue preserves compression order. Queueing commits the working copy and clears pending updates. Stock stream cleanup retains blocked HEADERS even when a stream is canceled; the send path reverses the queue before transmission. The unchanged frame-construction/queue/send routines were inspected for this ordering. No connection-table mutation occurs merely because a block encode attempt began. Allocation/frame-failure paths received source review, not runtime fault injection.
The 103 early-hints path needs the same pending-update/transaction bookkeeping
and a record of its existing incremental :status literal to keep the shared
outbound context coherent. Its header list remains stock non-indexing encoding;
it does not gain dynamic reuse. Static generated status representations, all
other generated wire shortcuts, trailers, request decoding and protocol/TLS
paths remain unchanged. This pin has no response PUSH_PROMISE generation path
to extend. No root CMake, accepted decision, architecture or configuration policy
file changes.
Table-size semantics and ownership
Before implementation, semantics were checked against RFC 7541 sections 2.3, 4 and 6, including FIFO eviction, indexed names referring to entries evicted by insertion, and the smallest-then-final pending size rule; and RFC 9113 sections 4.3.1 and 6.5, including the 4096 default and SETTINGS ordering/acknowledgment.
Selected capacity starts at 4096 and becomes min(peer_setting, 4096).
Shrinking evicts committed local entries immediately. The next ordinary or
103 block starts with the smallest changed selected capacity since the previous
queued block, then the final capacity if different. Identical selected sizes
schedule nothing. Six reserved buffer bytes cover two five-bit integers up to
4096; actual updates use one to three bytes each. Explicit 4096 and 8192 both
leave the selected 4096 unchanged and emit no update. Explicit zero clears the
table, emits zero and prevents eligible insertions/hits.
Each connection adds only one table pointer and two size_t fields (24 bytes on
this target), reusing the existing pending flag. Table and entry storage is lazy
heap allocation, not several kilobytes embedded in every connection or churn
storage retained by the connection pool. Each node owns contiguous name/value
bytes; eviction actually frees the node. Commit frees the old snapshot. The
existing HTTP/2 connection-pool cleanup frees all retained entries/table on
close, and request-pool cleanup releases uncommitted working copies.
The 4096 cap bounds HPACK-accounted bytes, not exact heap consumption: node metadata, the table object, one in-flight snapshot and insertion allocation add bounded overhead. No memory/CPU benchmark or allocation-efficiency claim is made.
Reproduce the A/B build
With upstreams already materialized, no network or new dependencies are needed:
tools/spike-hpack-response-encoding
The tool requires the exact documented nginx version/archive hash and BoringSSL
commit in third_party.lock, checks source markers and nginx's version header,
and writes SHA-256 manifests before any build. It builds pinned static BoringSSL
with Clang/libc++/LLD and BUILD_SHARED_LIBS=OFF, deletes only its dedicated
generated nginx stage/object directories, copies pristine inputs, and patches
only the second copy. patch --batch --forward --fuzz=0 must succeed without
offset/fuzz/reversal diagnostics. Both nginx builds use identical flags except
their output directories:
--with-cc=clang --with-debug --with-http_ssl_module --with-http_v2_module
--with-cc-opt=-I<vendored BoringSSL>/include
--with-ld-opt=-L<static BoringSSL build> -fuse-ld=lld -Wl,--no-as-needed -lc++ -lc++abi
Logs/artifacts live under build/hpack-response-encoding/ab/: exact versions,
configure/build/patch logs, dependency reports, before/after source manifests,
results.json, and per-run configs/commands/client/server logs under results/.
The EXIT trap compares the original-source manifests even if a build/test fails.
Canonical CMake is unaffected. The tool runs this committed runner command:
python3 tests/hpack_response_encoding.py \
--upstream build/hpack-response-encoding/ab/upstream/objs/nginx \
--patched build/hpack-response-encoding/ab/patched/objs/nginx \
--repeat 3 --output build/hpack-response-encoding/ab/results.json
The runner uses Python stdlib and installed nghttp 1.70.0; it implements no
HTTP/2 or HPACK stack. Each case owns private temporary state/public TLS fixtures,
validates/starts one nginx, and runs nghttp -v -y -t 5 -m 20 --no-push plus the
case's -c arguments on one TLS connection. It associates nghttp's actual printed
body bytes with each DATA stream diagnostic, checks exact lengths/END_STREAM,
status/fields, and client/server ALPN/TLS evidence. Lifecycle deadlines are five
seconds and the client subprocess bound is 15 seconds; finally stops/reaps
nginx and preserves logs before removing temporary state. Certificate checking
is disabled solely for the existing public transport fixtures.
Endpoint choice and measured results
/reuse returns the identical 23-byte body nxTLS HPACK spike body\n and the
three requested x-nxtls-hpack-{alpha,beta,gamma} name/value pairs. Both binaries
also send Set-Cookie: spike=public-test-only; Path=/, x-unrelated: stock-path,
and Content-Type: text/plain through the stock ordinary non-indexing path.
For this endpoint only, default_type "" plus explicit add_header Content-Type
avoids one generated incremental insertion per response. Generated server/date/
content-length remain active and mirrored. /stock-generated separately tests
normal generated text/plain encoding; it exposes duplicate generated-entry
churn rather than hiding it. No product configuration defaults are chosen.
All three full A/B repetitions passed: 66 TLS connections, 1320 responses.
Every response was 200 with exact body/expected fields, ALPN h2, TLSv1.3 and
no decoder/protocol errors. All 66 nginx processes exited 0, were reaped, and
temporary runtime trees were removed. Each case's byte sequence and debug counts
were identical in repetitions 1, 2 and 3. Each block was one unpadded HEADERS with
END_HEADERS and no PRIORITY; no CONTINUATION occurred. The runner explicitly
rejects CONTINUATION observations rather than treating fragment sizes as block
sizes. Bytes below exclude frame headers and are not TLS/TCP measurements.
Notation: B x N is N consecutive blocks of B bytes, in response order; every
row specifies all 20 responses in each of the three repetitions.
| Case / peer setting | Upstream block bytes, responses 1..20 | Patched block bytes, responses 1..20 | Patched size updates |
|---|---|---|---|
/reuse, omitted |
204 x 20 |
204, 107 x 19 |
none |
/reuse, 4096 |
205, 204 x 19 |
204, 107 x 19 |
none |
/reuse, 1024 |
205, 204 x 19 |
207, 107 x 5, 204, 107 x 5, 204, 107 x 5, 204, 107 |
1024 |
/reuse, 0 |
205, 204 x 19 |
205, 204 x 19 |
0 |
/reuse, 1024 then 4096 |
205, 204 x 19 |
210, 107 x 19 |
1024 then 4096 |
/reuse, 0 then 4096 |
205, 204 x 19 |
208, 107 x 19 |
0 then 4096 |
/reuse, 8192 |
205, 204 x 19 |
204, 107 x 19 |
none; cap 4096 |
/reuse, 64 |
205, 204 x 19 |
206, 204 x 19 |
64 |
/churn, 1024 |
386, 385 x 8, 387 x 11 |
388, 344 x 8, 346 x 11 |
1024 |
/oversized, 1024 |
1274, 1273 x 19 |
1276, 1176 x 5, 1273, 1176 x 5, 1273, 1176 x 5, 1273, 1176 |
1024 |
/stock-generated, 4096 |
195, 194 x 19 |
194, 97 x 15, 99, 100 x 2, 194 |
none |
The control selects zero once after any explicit setting, even 4096, and shows no dynamic reuse shrinkage. The one-byte initial difference is its stock zero update. The patched default/4096 later blocks are 97 bytes smaller (47.5%) than the first block. Explicit 4096 totals 4081 control bytes versus 2237 patched bytes for all 20 blocks. These are header-byte observations only.
The historical 181-byte blocks are not a target: this fixture changes header names, sends the cookie/unrelated fields, uses an explicit Content-Type path, and uses a return body instead of a static file's last-modified/accept-ranges.
Reuse, eviction, fallback and review evidence
Counts below are per patched connection, identical in all three runs. Insertions count eligible entries only; evictions include mirrored generated entries and exclude freeing old snapshots/connection destruction.
| Case | Eligible insertions | Full dynamic hits | Dynamic-name hits | Evictions | Eligible fallbacks |
|---|---|---|---|---|---|
| default / 4096 / 1024→4096 / 0→4096 / 8192 | 3 | 57 | 0 | 0 | 0 |
| 1024 | 12 | 48 | 0 | 54 | 0 |
| 0 | 0 | 0 | 0 | 0 | 60 |
| 64 | 0 | 0 | 0 | 39 | 60 |
| churn | 60 | 0 | 57 | 112 | 0 |
| oversized | 12 | 48 | 0 | 54 | 20 |
| stock-generated | 6 | 54 | 0 | 10 | 0 |
The three eligible entries total 220 HPACK bytes. At 1024, repeated stock
generated entries evict them before responses 7, 13 and 19, which reinsert them.
FIFO correctness therefore yields periodic refresh, even when the repeated
fields themselves fit. /stock-generated evicts/refills on response 20 at
4096; response 17..19 also exercise indexes crossing the seven-bit prefix limit.
/churn uses the existing $connection_requests variable in three 100-character
padded eligible values. All counters 1..20 and exact resulting fields are
checked. It forces genuine eviction and dynamic-name reuse. /oversized adds
a 1200-byte eligible value: it always falls back without clearing eligible
entries, so the other three still reuse/refill correctly. Capacity 64 cannot
hold any eligible field; zero disables the entire table.
Final source review covered index/address calculation, name indexes captured before insertion/eviction, resize-minimum/final signaling, update-byte allowance, stock generated accounting, allocation-before-eviction, commit ordering, cleanup and error exits. Set-Cookie and unrelated fields retain exactly the stock new-name literal path, with no lookup/insertion; the runtime checks their decoded values on every stream. No extra experiment log contains header values or names: only capacities, sizes, indexes, eligibility and return codes. Stock nginx debug field logging remains present and is exercised only with public synthetic data.
Review also consulted current OWASP C toolchain hardening and logging guidance: allocation/error ownership and avoiding sensitive values in new diagnostics were applied to this native-server experiment. This is not an OWASP compliance claim. Production work would require an explicit sensitive/volatile indexing policy, Never-Indexed review and HPACK compression-context security review.
Validation and development failures
Executed successfully from the repository root:
tools/spike-hpack-response-encoding
bash -n tools/spike-hpack-response-encoding
python3 -m py_compile tests/hpack_response_encoding.py
cmp build/hpack-response-encoding/ab/sources-before.json build/hpack-response-encoding/ab/sources-after.json
readelf -d build/hpack-response-encoding/ab/upstream/objs/nginx
readelf -d build/hpack-response-encoding/ab/patched/objs/nginx
git diff --check
git status --short
git diff --stat
The full tool includes three complete runtime A/B repetitions. Source manifests
matched for all 521 nginx and 8660 BoringSSL files. The historical archive
still hashes to the recorded 1ad24371…c2fde6 digest. Both binaries' NEEDED lists
are libc++, libc++abi, PCRE2, zlib, libgcc_s and libc; neither links dynamic
libssl/libcrypto. Clang 21.1.8 builds succeeded with nginx's -Wall -Wextra -Werror. Inspection with rg -n 'warning:' of all three build logs found no
compiler warnings (rg exit 1 means no matches); configure's intentional feature
probes are not treated as compiler diagnostics from the build.
Failures/findings during development:
- The first full-tool invocation built both binaries, then the sandbox rejected Python socket creation with EPERM. Manifest verification still passed. The final full tool ran outside that socket restriction and exited 0.
- The initial normal generated-content-type fixture decoded successfully but failed an overly strong expectation of 57 full hits: stock duplicate generated insertions caused response-20 refresh. The final primary fixture separates its explicit fixed Content-Type; the additional normal-shortcut case records that eviction behavior rather than claiming universal per-response shrinkage.
- An initial oversized-case assertion demanded 10% whole-block shrinkage. The deliberately non-indexed 1200-byte field dominated that block, so the assertion was corrected to check exact fallback, valid decoding and reuse of the other fields. There were no compile failures or observed HPACK decoder errors.
Limitations and decision gate
This is a synthetic ordinary-response mechanism test on the canonical host, with 20 requests per connection. It is not a CPU, RSS, throughput, latency, browser-rendering or stuttering measurement. Snapshot copying/linear lookup are deliberately simple, not a proposed production encoder implementation.
Repeated settings within the initial SETTINGS payload are exercised by nghttp. Post-population settings changes across later frames, forced allocation failures, frame-creation failures, backpressure/cancellation races, early hints, trailers, CONTINUATION and long-lived load were not runtime-tested. Shrink/clear logic for a populated table was source-reviewed, not independently fault-injected. Trailers remain entirely stock; a settings reduction arriving immediately before an outstanding trailer block would require a separate synchronization design because this spike emits pending updates only in ordinary/103 blocks. The patch therefore does not establish correctness for every outbound HTTP/2 path.
Gate C: the mechanism works and results justify a separate production-design discussion. Repeated eligible fields materially shrink on one connection, nonzero/zero capacities and pending minimum/final updates behave as observed, and eviction/name reuse complete without decoder errors. The generated-entry churn and cross-path synchronization questions are concrete design inputs. Even C is not adoption, a production indexing policy, or approval to integrate this experiment.