Protocol Studio: custom protocol health checks, routing, field rules and live tracing

Protocol Studio teaches your load balancer the protocol your servers speak. You describe its messages once, as a pattern, and the load balancer can then:

• check health by sending your own request and reading the reply field by field,
• weigh servers by a number they report (players online, queue length), so busier servers get fewer new connections,
• route TCP and UDP connections by fields of their first message (a region, a protocol version, a message type),
• count fields live on every packet: how often each value is seen, histograms, rates, sums and averages,
• act on every message with field rules in the kernel: drop forbidden messages, rate-limit and ban by a field, keep sessions on their server, split a canary share, mirror to a test pool, rewrite a field, and measure request to response latency by an id, with alerts on any of these numbers,
• trace and debug live traffic: decoded packets, breakpoints, and holding one connection while everything else keeps flowing.

Every packet is read in the kernel at native speed: the load balancer compiles your pattern to eBPF, which the Linux kernel verifies and JIT-compiles before it runs. What cannot be compiled that way keeps working on the pattern interpreter, automatically, and the editor tells you which parts run where. Messages split over several packets are reassembled before they are decoded.

It is part of every Nextify load balancer, at no extra cost. Game servers, IoT devices, custom RPC, anything with bytes on the wire works. The HTTP, TCP and send/expect health checks described in Getting started with load balancers stay as they are; Protocol Studio adds to them.

Before you start

• A load balancer with a TCP or UDP listener, or any pool whose servers you want to check with your own protocol.
• The layout of your protocol's messages: which bytes come first, how long each field is, and whether numbers are big-endian (network order) or little-endian.
• A sample message helps: a hex dump, a capture saved as a file, or a server you can probe.

1. Write a pattern

Open the load balancer in the portal and choose the Protocol Studio tab. Its Patterns list holds the patterns of this load balancer. Choose New pattern and start from a template:

A pattern is written in PatternLanguage, the pattern language of the ImHex hex editor. It looks like C: structs of typed fields, placed at an offset of the data. For example, a game server's status reply:

#pragma endian big

enum State : u8 { Ready = 0, Draining = 1, Full = 2 };

struct Status {
    u8    magic;        // always 0xFE
    State state;
    u16   players;
    u16   max_players;
    u8    region;
};

Status msg @ 0x00;

The editor helps as you type:

• Highlighting and completion of keywords, built-in types (u8 to u128, s8 to s128, float, char, bool), be / le, attributes and the std:: library.
• Checked as you type: half a second after you stop, the pattern is checked. Errors and warnings are underlined in the editor and listed under Problems with their line and column; select one to jump to it. A warning also tells you when a field can be used for health checks but not for routing, and why.
• Hover a field to see its offset, size and value from the last run.
• Format tidies the indentation.

Patterns are part of the load balancer's configuration: up to 20 per load balancer, 64 KiB each. Like every edit they are staged until you choose Review changes and Apply. A pattern with an error cannot be applied; the review names the pattern and the line and column at fault.

2. Run it on a sample

Run (or Ctrl+Enter, Cmd+Enter on a Mac) runs the pattern on the sample in the panel below the editor. Choose where the sample comes from:

• Hex: paste bytes as hex (fe 00 00 25 00 40 02; spaces and line breaks are ignored).
• File: upload a capture or a saved reply, up to 64 KiB.
• Snippet: a sample you saved earlier. Snippets (patterns and samples) are kept in your account, up to 200, and shared by all your load balancers.
• Live probe: the load balancer itself sends your steps (see health checks below) to one server of a pool and shows its reply. Nothing else is sent, and the server must be in that pool.

The results:

• Fields: the parsed tree, with each field's type, offset, size and value.
• Hex view: the sample's bytes. Select a field to see its bytes highlighted; select a byte to see the field it belongs to.
• Condition: the result of a health check condition on this sample (true, false, or the error).
• Routing: for each field, whether a routing rule can use it, and if not, the reason and the line that prevents it; and the native rules a rule on it compiles to (HAProxy rules for TCP listeners, nftables payload matches or the inline program for UDP).
• Inline (eBPF): the compile report (see below).

A sample that ends before the message does is reported as incomplete, with how many more bytes it needs: on live traffic the load balancer waits for the rest of the flow (see Message framing).

Step through replays the pattern on the sample one step at a time: Step, Continue, line breakpoints (select the line number in the editor's margin), and the variables at each step. It runs on the copy of the sample in the editor, never on traffic.

Save as snippet keeps the pattern or the sample in your account. Download .hexpat + sample saves both as files that open in ImHex, if you prefer to work there.

Inline: every packet in the kernel

The Inline (eBPF) tab of the results, and the chip next to the pattern's name, show how the pattern runs on the load balancer. For each listener and direction that uses it (a routing rule, a field metric), the pattern is compiled to an eBPF program, which the kernel verifies (it always ends and only reads the packet) and JIT-compiles to machine code. The report gives:

• the number of instructions and the kernel memory the program uses,
• the fields it reads inline, on every packet,
• each field that stays on the interpreter, with the line that keeps it there (for example, a string whose size is computed at run time). These lines are also marked in the editor, in blue (information), never as errors: the pattern still works.

The compiler aims at the whole language the kernel can run safely: structs and nested structs, enums and bitfields, conditions, loops and arrays the packet holds, length-prefixed and variable-offset data, pointers inside the message, functions, floats and the pure std:: functions. The Inline (eBPF) report is the authority for your pattern: it lists every field that runs inline and, for any that does not, the line and the reason (those keep running on the interpreter). A pattern that is not used by a rule or a metric yet can be compiled for UDP or TCP and either direction from the same tab.

Message framing

A message may be longer than one packet: a large UDP message split over datagrams, or a TCP message split over segments. The load balancer keeps the bytes of a flow's unfinished message and decodes it when it is complete. Under the editor, Message framing sets where a message ends:

Largest message (#pragma nx_message_max_bytes, 64 KiB by default) and Wait for the rest (#pragma nx_message_timeout_ms, 5 seconds by default) bound an unfinished message; past either it counts as a parse error. The settings are kept in the pattern as #pragma lines, so they travel with the .hexpat file; ImHex ignores them. The form also shows how the engine reads them.

Sizing

Sizing, under the framing, holds the inline program's settings: the bytes of a message it decodes, the flows with a message in progress and the bytes kept for each, the capture buffer, and the pattern steps per message. These are settings, not limits: each shows the node memory it takes, and the meter shows what the whole configuration needs against what the load balancer's size has for it. A configuration that would not fit is refused with the setting to lower; otherwise there is no cap.

3. Health checks on your protocol

On the Backends tab, open a pool and set its Health check to Custom protocol (pattern):

1. Pattern: one of the load balancer's patterns.
2. Steps: 1 to 8. Each step sends bytes, as hex or as text (it may send nothing and only read), and reads the server's reply: up to 64 KiB, until the server pauses for 50 ms or closes the connection. The pattern reads the reply to the last step.
3. Condition: an expression on the pattern's fields that must be true for the server to be healthy, for example msg.magic == 0xFE && msg.state == 0. A reply that the pattern cannot read, a timeout or a closed connection count as a failed check.
4. Weight from field (optional): a numeric field sets the server's weight. Its value between min and max maps linearly to a weight from 1 to 256. With inverse, the minimum gives the highest weight, so a server with more players gets fewer new connections; with direct, the opposite.
5. Fields to show (up to 8): their values appear next to each server on the Backends tab and in the health report.

Interval, timeout, rise and fall work as for every check. Pools whose servers speak TCP or HTTP are checked over TCP; UDP pools over UDP. On a pool served by HAProxy, the pattern check replaces HAProxy's own check for its servers.

Use Run with a Live probe first: it sends the same steps to a server and shows whether the condition holds, before anything is applied.

4. Route on fields

A rule on a TCP or UDP listener can send connections to a pool by the fields of their first message. On the Routing tab, add a rule to the listener and, under Pattern fields, choose the pattern and one or more conditions:

Example: If msg.region is 2 → pool eu.

Only routable fields can be chosen; the others are listed with the reason and the line:

• UDP: any field the pattern reads inline, wherever it is: after a length, a loop, a condition or a variable-length array. Floating-point fields can be compared too (is, is not, less, greater). Byte strings compare with is / is not.
• TCP: a field at a fixed offset or after a length prefix; numbers up to 8 bytes, byte strings up to 64 bytes.

How it runs: rules are compiled when you apply, never interpreted per packet. Each rule shows the engine it runs on:

• TCP (HAProxy): the listener waits for the first bytes of each connection (up to the listener's HTTP request timeout, at most 5 seconds), matches them and picks the pool.
• UDP (nftables or Inline eBPF): nftables handles fields at fixed offsets and Inline (eBPF) the others, so the listener stays in the Linux kernel. The first message of a flow decides its pool, and the rest of the flow stays there.
• Wait for the whole message (UDP): the load balancer holds a flow's datagrams until its first message is complete (see Message framing), then routes the flow on it.

If the badge says Not running, the rule needs the inline program and the load balancer cannot run it at the moment; until it can, those flows go to the listener's default pool.

The Test a connection box on the Routing tab runs the pattern on a sample and shows which rule and pool it would reach, condition by condition. Review changes shows the generated rules.

5. Field metrics

In the Protocol Studio tab, Field metrics counts the fields of your traffic, live. Add metric and choose:

• Pattern and Field: any field of the pattern's layout.
• Count:
  • Count by value: how often each value is seen (message types, regions, versions), as a rate chart of the top values and bars with each value's total and share. Distinct values (4,096 by default) is a setting with the node memory it takes; values beyond it count as other values.
  • Histogram: how values spread over the bucket bounds you give (8, 16, 32, 48): each value counts in the first bucket it is below, larger ones in the last.
  • Sum and average: the sum per second, the average per message and the total (players online, bytes requested).
• Listener (TCP or UDP) and Direction (client to server, or server to client).

Choose 15 min, 1 hour or 24 hours; the charts refresh every few seconds. Each metric is marked Inline (eBPF) when its field is read in the kernel on every packet, or Sampled when the field stays on the interpreter and is counted on samples. Messages the pattern cannot read are counted as parse errors, and messages where the field is not present (another branch of a condition) as not placed. Metrics are part of the configuration, staged until you apply.

6. Field rules

In the Protocol Studio tab, Field rules acts on your protocol's messages in the kernel, on every complete message (reassembled across packets). Field rules are for TCP and UDP listeners: HTTP, HTTPS and TLS listeners terminate in HAProxy, where the Routing and Security tabs have their own rules, rate limits and bans.

Add rule and choose:

• Pattern, Listener and Direction (client to server, or server to client).
• Acts on: messages where a condition holds, written like a breakpoint's (msg.op == 9 && msg.len > 1200; empty means every message), or invalid messages (strict mode, below).
• Action:
  • Drop: a protocol firewall. On UDP the datagram is dropped. On TCP the connection is reset, because dropping one segment would stall the stream: the client gets a reset at once, the server too.
  • Count: only counted; useful with a ban or an alert.
  • Rate limit: a token bucket per key: one or more fields (a player id, an API key, a session id) and, optionally, the client address. Rate per second (fractions allowed) and burst; over the limit a message is dropped or only counted. Buckets kept (65,536 by default) is a setting with the node memory it takes; the least recently seen keys make room. The busiest keys are listed with what passed and what was limited.
  • Route to a pool (UDP): pin keys to a pool, for example msg.player_id == 42 || msg.player_id == 77.
  • Canary split: send a share of keys (10 %) of a field to another pool. The same key always lands on the same side.
  • Sticky by field: consistent hashing of a field carried in the payload (a session or connection id) onto the servers of a pool, so a client keeps its server when its address or port changes (mobile roaming, NAT rebinding). When a server goes down only its own keys move.
  • Mirror: copy matching client messages to a server of a test pool, at most a rate you set (1,000 per second by default). The mirror's replies are read and discarded, so a new server version sees real traffic without answering it.
  • Rewrite a field: set a field to a new value of the same length (TCP sequence numbers forbid length changes), with the checksum fixed in the kernel: stamp a region, mask an identifier. Numbers are checked against the field's size; text and hex values must be the field's length.
• Ban sources that keep hitting this rule (client to server rules): after N violations within a window (drops, invalid messages, rate-limited messages, or every hit of a count rule) the source is banned in the kernel and on the host like every other ban, for the time you choose or the Security tab's ban time, with the reason pattern:<rule id>. Trusted sources (Security tab) are never rate-limited or banned.

Strict mode is a switch per TCP or UDP listener: it adds a rule that drops client messages the pattern cannot parse, that grow past the framing's maximum size, or that time out half-received.

Rules run in order; the first that routes a message wins, before the Routing tab's rules on pattern fields. On TCP, a connection's server is chosen before its first byte, so canary splits and sticky rules render into HAProxy (the listener then runs on HAProxy): they need no condition and a field at a fixed offset or with a length prefix. Route to a pool on TCP is refused: use a rule on pattern fields on the Routing tab.

Every rule shows where it runs (Inline (eBPF), HAProxy) and live counters per label (matched, dropped, over the limit, routed, mirrored, rewritten, sources reported for a ban, and so on) over 15 minutes, 1 hour or 24 hours. A rule the kernel cannot run (its condition or key reads a field that stays on the interpreter, for example after a computed-size array) is refused when you save, with the line and the reason; field rules have no interpreter fallback, so nothing is skipped silently. Rules are part of the configuration: staged, reviewed and versioned like every change, and switched off without being deleted.

Latency by request id

In Field metrics, Add metric with Count: Request / response latency pairs each client request with the server's response by an id field (msg.seq; the response's id can be another field), optionally grouped by a request field such as the message type. The load balancer measures the time between the request arriving and the response leaving towards the client, in the kernel, the same way on every engine. You see p50, p95 and p99 over time per group, the histogram over the buckets (microseconds), the average, responses per second, and requests that got no answer within the response timeout (30 seconds by default).

7. Alerts

In the Protocol Studio tab, Alerts puts thresholds on any field rule counter or field metric:

• Watch a field rule (a counter such as dropped or over the limit) or a field metric (a value, a bucket, or a latency group).
• Statistic: the rate per second, the share (for example the share of rate-limited messages), or for latency and histograms the average, p50, p90, p95 or p99.
• Fires when the statistic is above or below a threshold over a window (15 seconds to 1 hour; 60 seconds by default).

Alerts are evaluated every 20 seconds. Each alert shows OK, Firing or No data with its value now; every change is kept in the load balancer's health history (30 days). While an alert fires, a notification shows at the top of every portal page and on the load balancer list, linking to the alert.

8. Live trace and debugger

In the Protocol Studio tab, open Live trace and choose Start trace. The load balancer must be active. A session lasts at most 15 minutes, a load balancer has at most 2 at a time, and each start is recorded in your account's audit log. Stop ends it at once.

• Packets: a live list of the load balancer's traffic: time, direction, client and server, listener, pool, the rule that matched, the size, and the fields decoded with the pattern you choose. Filters narrow it by address or network, port, listener, direction and fields; Pause freezes the list while traffic keeps flowing.
• Breakpoints (up to 5): a condition on the connection (client or destination address or network, ports, listener, pool, server, direction) and, optionally, on fields (msg.type == 3), with a hit count and stop after N hits. A field condition on inline fields is evaluated in the kernel on every message (the breakpoint says Inline (eBPF)): a hold then holds the very packet that matches, and a capture copies exactly the matching packets. Each breakpoint does one of three things:
  • Capture: a copy of the matching packet and the next packets of that flow, in both directions, is kept in the inspector. The flow keeps going.
  • Hold: only the matching flow waits; every other connection, health check and ban carries on. Then Continue (release everything), Step (release one packet and hold the next), Drop (this packet) or Close (end the connection). At most 3 flows are held at once, and a hold continues by itself after 60 seconds. The client's own timeouts may fire sooner.
  • Log: prints a line into the trace instead of stopping, for example {client} {msg.seq} {msg.len}.
• Inspector: for a captured or held packet, the fields tree and the hex view linked to it, how it was captured (in the kernel, or a sampled copy), how many packets its message was assembled from, watch expressions of your own, and a timeline of the flow's packets. Save as sample keeps a packet as a snippet you can run patterns on later.

What a trace sees and keeps:

• Only packets that match your filters and breakpoints are copied, the first 512 bytes of each. Where the pattern compiles inline, the kernel decodes every message and copies exactly the matching ones through a fixed-size buffer; otherwise copies are sampled at a capped rate (50 flows per second) and decoded beside the traffic. Either way, when the trace falls behind, copies are dropped, never traffic, and no packet is ever delayed by decoding.
• Nothing is stored. Packets are streamed to your browser only; a packet is kept only if you save it as a sample, in your account, and you can delete it. No other account ever sees your traffic.
• An HTTPS listener shows the decrypted requests, because the load balancer terminates TLS. A TCP listener that passes TLS through shows only the handshake and the server name.

9. The API

Everything above is also in the API (see the API documentation).

# Check a pattern (the same check as the editor), then run it on a sample
curl -s -X POST https://api.nextifynow.com/v1/load-balancers/patterns/check \
  -H "Authorization: Bearer $NEXTIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"source": "struct S { u8 magic; u8 state; }; S msg @ 0x00;"}'
curl -s -X POST https://api.nextifynow.com/v1/load-balancers/patterns/run \
  -H "Authorization: Bearer $NEXTIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"source": "struct S { u8 magic; u8 state; }; S msg @ 0x00;", "data_hex": "fe00", "condition": "msg.magic == 0xFE"}'

# The inline (eBPF) compile report for a transport and direction
curl -s -X POST https://api.nextifynow.com/v1/load-balancers/patterns/compile \
  -H "Authorization: Bearer $NEXTIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"source": "struct S { u8 magic; u8 state; }; S msg @ 0x00;", "transport": "udp", "direction": "c2s"}'

# Live field metrics: rates per value or bucket, and totals (range 15m, 1h or 24h)
curl -s "https://api.nextifynow.com/v1/load-balancers/<id>/pattern-metrics?range=1h" -H "Authorization: Bearer $NEXTIFY_TOKEN"

# Alerts with their state and history; the account's firing alerts
curl -s "https://api.nextifynow.com/v1/load-balancers/<id>/alerts" -H "Authorization: Bearer $NEXTIFY_TOKEN"
curl -s "https://api.nextifynow.com/v1/lb-alerts" -H "Authorization: Bearer $NEXTIFY_TOKEN"

# Starter templates and the language's keywords, types and functions
curl -s https://api.nextifynow.com/v1/load-balancers/patterns/templates -H "Authorization: Bearer $NEXTIFY_TOKEN"

# Fetch a real reply from one server of a pool
curl -s -X POST https://api.nextifynow.com/v1/load-balancers/<id>/patterns/probe \
  -H "Authorization: Bearer $NEXTIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"pool_id": "p1", "server_key": "<instance id>:27015", "steps": [{"send": "fe01", "send_type": "hex"}]}'

Snippets: GET and POST /v1/pattern-snippets, GET, PUT and DELETE /v1/pattern-snippets/{id} (name, kind pattern or sample, source or data_hex).

In the configuration document (GET|PUT /v1/load-balancers/{id}/config), patterns, checks and rules look like this:

{
  "patterns": [{ "id": "pt1", "name": "game status", "source": "#pragma endian big\nstruct Status { ... };\nStatus msg @ 0x00;",
    "metrics": [{ "id": "m1", "name": "Players by region", "listener": "l1", "direction": "s2c", "field": "msg.region",
                  "kind": "count_by_value", "buckets": null, "max_values": 4096 },
                { "id": "lat1", "name": "Round trip", "listener": "l1", "kind": "latency", "field": "msg.seq", "response_field": null,
                  "by": "msg.op", "buckets": [1000, 5000, 25000, 100000], "max_values": 4096, "timeout_ms": 30000 }],
    "field_rules": [{ "id": "fr1", "name": "Forbidden opcode", "enabled": true, "listener": "l1", "direction": "c2s", "on": "message",
                      "condition": "msg.op == 9", "action": { "type": "drop" },
                      "ban": { "after": 20, "window_s": 60, "seconds": null } },
                    { "id": "fr2", "name": "Per player", "enabled": true, "listener": "l1", "direction": "c2s", "on": "message", "condition": null,
                      "action": { "type": "rate_limit", "rate_limit": { "key": ["msg.player_id"], "per_client": true, "rate_per_s": 20,
                                  "burst": 40, "over": "drop", "entries": null } }, "ban": null }],
    "limits": null }],
  "alerts": [{ "id": "a1", "name": "Forbidden opcodes", "enabled": true,
               "source": { "pattern_id": "pt1", "kind": "field_rule", "id": "fr1", "label": "dropped", "by": null },
               "stat": "rate", "op": "gt", "threshold": 5, "window_s": 60 }],
  "pools": [{ "id": "p1", "...": "...",
    "health_check": { "type": "pattern", "interval_ms": 5000, "timeout_ms": 2000, "rise": 2, "fall": 3,
      "pattern": { "pattern_id": "pt1",
                   "steps": [{ "send": "fe01", "send_type": "hex", "read_max_bytes": 65536 }],
                   "condition": "msg.magic == 0xFE && msg.state == 0",
                   "weight_from": { "field": "msg.players", "mode": "inverse", "min": 0, "max": 64 },
                   "show_fields": ["msg.players", "msg.max_players"] } } }],
  "rules": [{ "id": "r1", "listener": "l1",
    "match": { "pattern": { "pattern_id": "pt1", "conditions": [{ "field": "msg.region", "op": "eq", "value": 2 }], "wait_for_message": false } },
    "action": { "type": "forward", "pool": "p2" } }]
}

The API fills in each pattern's compiled layout (compiled: its fields, offsets and whether each is routable, its framing, and bpf: the inline compile per listener and direction) when it checks the configuration. A pattern with an error answers 422 at patterns[N].source with the line and column. The dry run (POST /v1/load-balancers/{id}/config/validate) also answers notes (what runs inline and what stays on the interpreter), routes_engine (the engine of each pattern rule) and memory (the node memory the configuration needs). GET /v1/load-balancers/{id}/health reports the fields to show per server (servers[].fields) and, under bpf, the inline programs and the engine each route runs on; GET .../pattern-metrics also answers each field rule's counters (kind rule, with the busiest keys of a rate limit) and each latency metric (kind latency).

Trace sessions: POST /v1/load-balancers/{id}/trace answers 201 with a one-time ticket (valid 30 seconds) and the WebSocket path, GET /v1/load-balancers/{id}/trace/ws?ticket=… is the stream, DELETE /v1/load-balancers/{id}/trace/{session_id} ends a session, and GET /v1/load-balancers/{id}/trace lists recent sessions.

About the engine

Patterns are read by PatternLanguage, the pattern language of the ImHex hex editor, by WerWolv and its contributors (LGPL-2.1). On a load balancer it runs in its own sandboxed helper process, with limits on CPU time, memory and output, and no access to files, the network or the environment. Patterns you write in ImHex work here (functions that read files or the environment are switched off), and the community collection ImHex-Patterns is a good starting point for formats that already have one. These are independent open-source projects; Nextify is not affiliated with them.