--- license: apache-2.0 tags: - security - intrusion-detection - waf - network - numpy - beam library_name: none --- # DibaGuard Open **A CPU-light attack-detection and protection layer for network flows, HTTP, JSON-RPC and databases, with calibrated false-alarm rates, explainable verdicts and a policy-driven enforcer. This is the public, licence-clean build. Code is Apache-2.0; the model files are CC BY-SA 4.0 (one training source requires ShareAlike). Built for the [Build with Beam](https://huggingface.co/reflection/Beam-1.0-501B-A23B/discussions/5) challenge; the companion Space is [`DibaAi/beam-dibaguard`](https://huggingface.co/spaces/DibaAi/beam-dibaguard).** DibaGuard sits where a WAF, an IDS and a host firewall would sit: inline in front of an application, as an external decision service behind the load balancer you already run (nginx, Envoy, HAProxy, Traefik, Caddy), or beside a host reading flow and log exports. Every alarm comes with a controlled false-alarm rate (conformal calibration on the site's own traffic, default budget 10^-3 per source-minute), a named attack family, the evidence that produced it, and an action chosen by the operator's policy (report only, or enforce). Inference is numpy-only in Python, with a dependency-free Go fast path proven bit-exact against the Python one on parity fixtures. ## What is in this repository, and what is not | path | content | licence | |---|---|---| | `models/seq_v1/nf_entity_mono.npz` | tier-1 monotone tree prior (24 behaviour counters, CatBoost oblivious trees evaluated in numpy) | CC BY-SA 4.0 | | `models/seq_v1/seqnet.npz`, `seq_v1_info.json` | tier-2 GRU sequence network (d = 64, 95,293 parameters) | CC BY-SA 4.0 | | `models/entity_v1/nf_entity_detector.npz`, `lu_entity_detector.npz` | fallback entity detectors (NetFlow view; LUFlow view) | CC BY-SA 4.0 | | `models/dbnet.npz` | DbNet, the database query network (own grammar-generated corpus, no third-party data) | CC BY-SA 4.0 | | `models/**/*.bin`, `*.bin.json` | the same weights as flat little-endian arrays for the Go runtime | CC BY-SA 4.0 | | `runtime/python/dibaguard/` | the engine, shield, HTTP edge, database tier, LLM bridge (MCP + REST) and operator console | Apache-2.0 | | `runtime/go/` | the Go fast path (`dibaguard-authz`, `dibaguard-proxy`) and its parity fixtures | Apache-2.0 | | `beam/beam_client.py` | the stdlib-only Beam client the Space uses (OpenAI-compatible endpoint; key from the environment) | Apache-2.0 | | `docs/`, `reports/` | technical documentation, independent validation report, licence audit, benchmark | Apache-2.0 | **Not in this repository: the HTTP request model (`http_v2/http_tier1_v2.npz`).** The one in the private build is trained on data that is not cleared for redistribution (CSIC 2010, a labelled access log whose raw file the owner withholds, several payload lists of unknown licence; `docs/DATA_LICENSES.md` rows 12-19), and the licence-clean retrain collapses on applications it never saw (`docs/CLEAN_MODELS.md` section 0.4), so it is not shipped either. The engine supports this configuration natively (`Guard.load` with no `http_v2` file; `guard.http is None`): the HTTP edge of this build runs the **request signatures A01-A22** (JNDI, CMS exploit paths, SQL / NoSQL / command injection, traversal, XSS, SSTI / XXE / SSRF, sensitive-file probes, declared AI crawlers), the **SQL-context analysis of every parameter (A22)** with **DbNet as a second opinion**, and the **behavioural session tier** (every request is an event of its client's stream, so enumeration, brute force, scraping and scanners are caught as behaviour). What the HTTP model added on top in the private build is a per-request anomaly score for payloads the signatures do not name; this build does not have that. ## The cascade Every tier looks at a different scale, every score becomes a p-value against the site's own reference, and a tier alarms when its p-value is under its share of the false-alarm budget (`docs/TECHNICAL.md`, sections 2-7): * **Tier 0, entity state**: 36 exponentially decayed behaviour counters per source over three horizons (10 s, 2 min, 30 min): no-reply, SYN-only, reset, new-port, new-host, repeat and short-flow shares, gaps, and three destination-side counters. Identity-free and scale-free, which is why the models transfer between networks. * **Tier 1, monotone tree prior**: oblivious trees over 24 attack-direction counters with *increasing* constraints (more port walking or more failed connects can never look more benign), calibrated per service x source age (Mondrian conformal). * **Tier 1', site-only anomaly**: no labels; Fisher's combination of upper-tail p-values of the shape features against the site's own traffic. * **Tier 2, GRU SeqNet**: a causal GRU with one hidden state per source, FiLM-conditioned on a site context vector; runs on warm sources only. * **Tier 4, per-source evidence**: fast and slow CUSUM over a source's stream, so weak signals accumulate and a first-contact source is never blocked on one event. * **Tier 5, destination flood** and **tier 6, campaign tracker**: floods of new sources on one destination; distributed low-and-slow campaigns found through a fingerprint shared across thousands of sources. * **HTTP edge**: one canonical request text on every path (Python and Go produce byte-identical text), signatures A01-A22, SQL-context analysis of parameters, DbNet second opinion, payload provenance between the database tier and the HTTP edge, session tier. * **Database tier**: six engines' query / audit logs; DbNet + signatures + behaviour + provenance; an application server is never blocked at the network level, the web source behind the payload is acted on instead. * **Shield**: policy (report | enforce), enforcer (nft / iptables / netsh / dry-run) with TTLs, posture audit, rules in any language, NAT / CGNAT / IPv6 identity, operator feedback that an attacker cannot poison, review queue, report. ## Open build versus private build The network models here are the **licence-clean set** of `docs/CLEAN_MODELS.md`, trained only on sources the audit marks GREEN (SIMARGL 2021 / 2022, APPRAISE, ACI-IoT-2023, CSE-CIC-IDS2018 from the publisher's AWS bucket, LUFlow from its author). Head-to-head with the private ("shipped") models on the same rows, conformal threshold at alpha = 10^-3, tier-1 prior, per flow: | scored network (neither model trained on it) | shipped AUROC | open (clean) AUROC | |---|---|---| | SIMARGL2022 | 0.968 | **0.999** | | APPRAISE | 0.614 | **0.985** | | ACI-IoT-2023 (author release) | 0.622 | 0.644 | | CSE-CIC-IDS2018, AWS, Tue 20-02 | 0.949 | **1.000** | Per-source evidence on SIMARGL2021 (an unseen network for the clean run), judged per (source, minute): clean FPR 0.00081 with 8 / 8 attacking source-minutes caught, shipped FPR 0.0029 with 16 / 17. The GRU's leave-out AUROC on SIMARGL2021 is 0.991 (shipped 0.822); the LUFlow detector improves from 0.984 to 0.9998 after a loader clock fix. **Where the open models are weaker**: on NF-UNSW-NB15-v3 (AUROC 0.958 -> 0.066) and NF-ToN-IoT-v3 (0.955 -> 0.879), which are the private model's own training networks (UNSW Canberra, academic-only terms) and have no openly licensed twin. Expect weaker tier-1 scores on IoT / UNSW-style traffic; the site-only anomaly tier and the evidence tiers do not depend on training data. The open build has **no HTTP request model** (above). ## Results Reproducible labs of the main repository (`scripts/benchmark_all.py`, three seeds, `reports/benchmark.md`). These numbers were measured on the **private build with its full model set**, including the HTTP model that is not in this repository; the two SQL-injection lines and the libinjection comparison are signature + SQL-context + DbNet results that involve no HTTP model. The open build has not yet been re-run through the labs (see *Verified at build time* for what was checked on this exact folder). | lab (3 seeds) | item | mean | min-max | |---|---|---|---| | evolving attacker, 12 waves | Log4Shell / SSH brute force / NTP amplification / credential stuffing / peer flood / rapid reset / WordPress exploit / AI scraper, final recall | 1.000 each | 1.0-1.0 | | evolving attacker, 12 waves | API enumeration, final recall | 0.979 | 0.973-0.983 | | evolving attacker, 12 waves | benign sources blocked (flows / HTTP) | 0 / 0 | - | | evolving attacker, 12 waves | benign alarm rate per source-minute (flows / HTTP) | 0.0011 / 0.0010 | budget 0.001 | | six-wave playbook lab | SSH brute force / HTTP/2 rapid reset / credential stuffing / slow scan / chain peer flood, recall | 0.945 / 0.980 / 0.984 / 0.935 / 0.875 | see report | | databases, 6 engines, 12 scenarios | detected / correct response | 1.000 / 1.000 (all twelve) | 1.0-1.0 | | databases | benign alarms per million queries / app servers enforced | 0 / 0 | - | | low-and-slow campaigns C1-C5 (150-3,000 sources) | member recall | 1.000 / 0.959 / 0.953 / 1.000 / 1.000 | detection delay 4.7 / 8.3 / 15.3 / 0.0 / 0.0 min | | unfamiliar networks: hosting | flood on one tenant / slow distributed scan / SSH guessing | 0.991 / 0.918 / 1.000 | benign alarms 0.00096 per source-minute, 0 enforced | | unfamiliar networks: office | SMB lateral movement / Kerberoasting / RDP spraying / LDAP enumeration | 1.000 each | benign alarms 0.00014 per source-minute, 0 enforced | | unfamiliar networks: OT | PLC subnet scan / Modbus writes / telnet-ssh guessing / MQTT flood | 1.000 / 1.000 / 1.000 / 0.839 | benign alarms 0.000012 per source-minute, 0 enforced | | platforms (WordPress, Joomla, Drupal, JSON API, generic) | 17 attack scenarios detected | 0.91-1.00 (13 of 17 at 1.000) | 0 benign sources blocked | | chain RPC | eth_getLogs flood / debug_traceTransaction flood / mempool spam | 1.000 / 1.000 / 0.967 | - | Real data (same report): SQL injection in the 2025 ModSecurity blocked-request archive with a visible payload **1.000** [0.9977, 1.0] of 1,671 (libinjection on the same requests: 0.9856); all CRS-labelled SQLi rows 0.531 of 3,225; SR-BH 2020 WordPress SQLi 0.608 of 9,886; false alarms on 80,000 benign requests of a production shop log 0.00013 [0.00007, 0.00023]; DbNet + signatures on unseen query templates TPR 0.983 at FPR 0.0; DbNet alone on unseen injection techniques TPR 0.743. ## Independent validation A team that did not build DibaGuard treated every claim as unverified and measured it (`docs/VALIDATION_REPORT.md`). They reported **22 findings** (bypasses of the canonicaliser, silent caps, identity tricks, reference poisoning, state attacks, harm to legitimate users, a global tier that flooded unfamiliar networks with alarms). **All 22 were fixed and each reproduction is now a permanent regression test** (section 8 of the report). After the fixes: canonicalisation fuzz of 3,000 cases, Python edge and Go path each block 99.3 % with 0 % diverging verdicts (72.5 % / 76.9 % and 32.8 % divergence before); 80,234 benign requests of a real shop log, 0 blocked on every path; harm lab (26 simulated hours, 3 seeds), users blocked per 1,000 per day 146 -> **0.0**, partner API clients and the search crawler no longer challenged; OWASP CRS regression tests (4,138 attack / 1,054 negative), hard tiers 14.7 % of attack tests and 4.3 % of negatives (a lower bound: many CRS tests are fragments a context engine is not meant to flag). ## Charts Rendered by `scripts/beam_charts.py` from the result files (`charts/`). ![Rules in 8 languages: behaviour agreement of the installed rule with the intended one](charts/beam_rules_by_language.png) ![Rule interpretation by reasoning effort](charts/beam_rules_by_effort.png) ![The analyst: 8 attacks and 5 legitimate decoys](charts/beam_analyst.png) ![Detection in the labs, 3 seeds](charts/labs_recall.png) ![Tier-1 prior AUROC per network: open vs private models](charts/clean_vs_shipped.png) ![Legitimate users harmed per 1,000 per day, before and after the independent validation](charts/harm_before_after.png) ## How Beam is used Detection never depends on a language model: every verdict above is deterministic and explainable. Beam (`reflection/Beam-1.0-501B-A23B`, through an OpenAI-compatible endpoint: Reflection's API or Hugging Face Inference Providers, `beam/beam_client.py`) is used where language is the interface: * **Teach with Beam.** The operator writes a rule in any language ("block anyone brute forcing ssh over 30 attempts a minute", "challenge every user-agent containing bot", or the same in Persian, Turkish, Spanish). Beam is asked for one JSON object against the fixed rule schema of `dibaguard/shield/rules.py` (action, scope, conditions filled from the engine's own counters: requests per minute, new ports per minute, no-reply share, repeat share). The precise rule and Beam's reasoning are shown back to the operator, who confirms before it is installed (`Layer.add_rules(texts, interpret_only=True)` then `add_rules(texts)`); from then on it is enforced from the first second with no learning window. A built-in bilingual parser handles the common forms without a model. * **Analyst.** For an incident, Beam receives the engine's evidence (`dibaguard.report.REPORT_SCHEMA`: families, per-source evidence, timeline, what the layer did and why, database remediation) and writes the explanation an on-call engineer needs, with a proposed action. The action sits **behind an approval gate**: nothing Beam proposes is applied until the operator approves it in the console, and Beam never sees raw credentials or the console token. The Space ([`DibaAi/beam-dibaguard`](https://huggingface.co/spaces/DibaAi/beam-dibaguard)) demonstrates both on the labs' synthetic traffic. ## Quick start ### Python (numpy only) ```bash pip install numpy export PYTHONPATH=$PWD/runtime/python # or: pip install -e runtime/python after adding a pyproject of your own ``` ```python import numpy as np from dibaguard.engine import Guard from dibaguard.shield.layer import Layer guard = Guard.load("models") # tier-1 prior, GRU SeqNet; guard.http is None in this build L = Layer.load("models", policy={"mode": "report"}, # report: verdicts and would-be actions, nothing enforced inputs={"http": True, "flows": True}, backend="dryrun") # DbNet is picked up from models/dbnet.npz # 1. calibrate on this site's own benign traffic (no attack labels needed) L.calibrate_flows(benign_flows) # dict of columns, see dibaguard.engine.FLOW_FIELDS L.calibrate_http(["get /shop?item=12 | h:user-agent=mozilla/5.0", ...]) # canonical texts, dibaguard.http.parse L.calibrate_db(benign_db_events) # {t, engine, db, client, query, rows, status}, adapters in shield/database.py # 2. score, in time order out = L.flows(flows) # per-flow alarm / family / score + actions per source h = L.http(texts, client=clients, host=hosts, status=statuses, t=times) # playbook ids, families, actions d = L.db(events) # verdicts with remediation; app servers are never blocked print(L.report()) # incident report (JSON); dibaguard.report.render_en for text ``` The sidecar serves the same engine over MCP and REST, with the operator console: ```bash python -m dibaguard.llm http --models models --port 8787 --policy policy.json --backend dryrun # REST + console python -m dibaguard.llm mcp --models models # MCP over stdio ``` Start in **report mode** with the dry-run backend; switch the policy to `enforce` only after reviewing the report (`docs/INTEGRATIONS.md` section 10). ### Go fast path ```bash cd runtime/go go build -o dibaguard-authz ./cmd/dibaguard-authz # stdlib only, Go 1.22 ./dibaguard-authz -listen 127.0.0.1:9300 -models ../../models -sidecar http://127.0.0.1:8787 -mode report ``` `dibaguard-authz` answers nginx `auth_request`, Envoy `ext_authz`, Traefik `ForwardAuth`, Caddy `forward_auth` and HAProxy (`docs/INTEGRATIONS.md`). Without an HTTP model file it logs `model unavailable` once and runs on signatures and the sidecar's state, which is the intended configuration of this build. `dibaguard-proxy` (the inline reverse proxy) requires `models/http_v2/http_tier1_v2.bin` at start and therefore does not run against this model folder; use `dibaguard-authz` behind your load balancer, or the Python layer inline. ## Data and licences * **Code**: Apache-2.0 (`LICENSE`). * **Model files under `models/`**: CC BY-SA 4.0 (`LICENSE-WEIGHTS`), because LUFlow (Mills et al., IEEE TNSM 2021), one training source, is CC BY-SA 4.0 and whether trained weights are Adapted Material is legally unsettled; this release applies ShareAlike rather than relying on the answer. * **Training data of the network models** (`NOTICE`): RoEduNet-SIMARGL2021 and SIMARGL2022 (CC BY 4.0, Mihailescu et al., Sensors 2021); APPRAISE H2020 real labelled NetFlow (CC BY 4.0, ITTI); ACI-IoT-2023 (CC BY 4.0, Nack et al., MILCOM 2024, doi:10.21227/qacj-3x32); CSE-CIC-IDS2018 from the publisher's AWS bucket (Sharafaldin et al., ICISSP 2018; https://registry.opendata.aws/cse-cic-ids2018); LUFlow 2020 / 2021 (CC BY-SA 4.0, Mills et al., IEEE TNSM 2021, doi:10.1109/TNSM.2021.3091517). DbNet: own grammar-generated corpus. The same block is stored in each model's metadata. * **Audit**: `docs/DATA_LICENSES.md` classifies every dataset the project ever touched (GREEN / AMBER / RED, with the owner's terms quoted); `docs/CLEAN_MODELS.md` records how the clean set was built, the licence gate in the training jobs, and every head-to-head number. Evaluation-only material (OWASP CRS tests, SR-BH 2020, the ModSecurity 2025 archive) was never trained on and nothing from it is bundled. ## Limitations * **No HTTP request model** in this build (above). Payloads the signatures and the SQL-context analysis do not name are caught only when their client's behaviour is; the benign alarm rate of the HTTP edge is correspondingly lower, and so is its recall on obfuscated XSS / command-injection variants. * **Benign alarm rate on public flow captures is above the budget** when the scored period differs strongly from the calibration window (`docs/VALIDATION_REPORT.md` section 8.5, measured with the private models after the fixes): CSE-CIC-IDS2018 (AWS, DDoS day) 6.77 % of benign source-minutes alarmed (3.04 % without the two global tiers; 99.7 % of attack source-minutes caught); SIMARGL 2021 (a 2-million-flow scan) 24.07 % (1.03 % without the global tiers; 100 % caught); LUFlow 2021 temporal hold-out 1.12 % (10.3 % of reputation-labelled attack sources caught). The budget is 0.1 %; the labs meet it, these captures do not yet. A longer calibration window (a day rather than hours) is the first lever. * **IoT / UNSW-style networks**: the open tier-1 prior is weak there (section above). * Payload inspection needs clear text (TLS-terminating proxy or logs); encrypted traffic is covered for behaviour only. * A slow *distributed* scan among internet background radiation is found by the campaign tier in 6-17 minutes. * Python state costs about 3-3.6 KB per tracked source (design budget 300 B): size `max_entities` to RAM or put the Go tiers in front; re-calibration still runs on the hot path. * Prose that has the exact shape of an injection (`admin'#'`) is still blocked (2.09 % of technical docstring sentences with an apostrophe; 0 of 16 ordinary review sentences). * Several uploader identities behind the CC BY sources (SIMARGL, LUFlow, ACI-IoT Kaggle accounts) are inferred from display names, not verified; the CIC redistribution wording should be confirmed by e-mail. Not legal advice. ## Verified at build time On this exact folder, in a fresh interpreter with `PYTHONPATH=runtime/python`: `Guard.load("models")` loaded the tier-1 prior and the GRU with `guard.http is None`; `Layer.load` picked up DbNet; after calibrating on 60,000 synthetic benign flows and 300 benign requests, a batch of 31,300 flows (synthetic SSH brute force from one source, a 400-probe slow scan from another, 30,000 benign flows from 3,000 clients): the brute-force source was reported as `bruteforce` and the scanner as `portscan`, both with would-be action `block` (report mode, nothing enforced); 57 of the 3,000 benign clients had 67 alarmed flows of their 30,000 (0.22 %; 19 block-safe, families ['attack_generic', 'dos'], would-be actions ['monitor']) during the attack minutes, on the test-suite's synthetic benign generator; HTTP: a JNDI request matched ['A08'] (family `rce`), a UNION SQL injection in a query parameter matched ['A22'] (family `sqli`), a benign shop request raised nothing; database tier: a UNION injection through the application's own connection alarmed as ['injection'], a benign query did not, and `DROP DATABASE` from an unknown public client alarmed as ['destructive', 'new_client']; the rule parser and both report renderers ran from the transformed sources. `python -m pytest -q tests/test_parity.py` on the main tree (Python <-> Go parity fixtures, unchanged): not run (--skip-parity). ## Model files | file | bytes | |---|---| | `models/dbnet.bin` | 3,160,300 | | `models/dbnet.bin.json` | 680 | | `models/dbnet.npz` | 2,403,349 | | `models/dbnet_eval.json` | 6,016 | | `models/entity_v1/lu_entity_detector.bin` | 188,976 | | `models/entity_v1/lu_entity_detector.bin.json` | 1,916 | | `models/entity_v1/lu_entity_detector.npz` | 152,441 | | `models/entity_v1/nf_entity_detector.bin` | 274,400 | | `models/entity_v1/nf_entity_detector.bin.json` | 2,820 | | `models/entity_v1/nf_entity_detector.npz` | 221,205 | | `models/seq_v1/nf_entity_mono.bin` | 82,358 | | `models/seq_v1/nf_entity_mono.bin.json` | 3,116 | | `models/seq_v1/nf_entity_mono.npz` | 17,794 | | `models/seq_v1/seq_v1_info.json` | 560 | | `models/seq_v1/seqnet.bin` | 381,172 | | `models/seq_v1/seqnet.bin.json` | 5,512 | | `models/seq_v1/seqnet.npz` | 390,463 | ## Documentation `docs/TECHNICAL.md` (design, models, calibration, HTTP edge, database tier, shield, history, results, limitations), `docs/VALIDATION_REPORT.md` (independent validation and the remediation), `docs/CLEAN_MODELS.md` (licence-clean models, head-to-head), `docs/DATA_LICENSES.md` (licence audit), `docs/INTEGRATIONS.md` (nginx, Envoy, HAProxy, Traefik, Caddy, Kubernetes), `docs/CONSOLE.md` (operator console), `docs/HA.md` (state, restarts, replicas), `docs/PERFORMANCE.md` (throughput, latency, memory), `docs/DESIGN.md` (the original design note), `reports/benchmark.md`. The copies in this repository had their Persian sections removed and carry a note about what applies to the open build. ## Citation ```bibtex @software{dibaguard_open_2026, title = {DibaGuard Open: a CPU-light, calibrated attack-detection and protection layer}, author = {Dibachain (DibaAi)}, year = {2026}, url = {https://huggingface.co/DibaAi/DibaGuard-Open}, license = {Apache-2.0 (code), CC-BY-SA-4.0 (models)} } ``` Built 2026-10-31 by `scripts/make_open_release.py` of the main repository.