Argent Sentinel

Argent Sentinel is a self-hosted security event collector, correlation engine, CrowdSec decision bridge, and guarded abuse-reporting system.

Version 0.5.5.1 combines authenticated event transport, central policy, a read-only dashboard, and per-site traffic analytics:

WordPress / Nginx / OpenSSH
          | local immutable spool or journald cursor
          v
argent-sentinel-agent
          | HTTPS + per-node mTLS events and protection inventory
          v
sentinel.example.org
          | Nginx verified-client proxy
          v
argent-sentinel-api -> central collector -> CrowdSec / abuse reporting

A combined host may run both the agent and server packages. Remote nodes run the agent package and submit immutable event batches to the central API.

Packages

• argent-sentinel-common: collector, agent, API engines, shared command-line programs, configuration examples, and documentation.
• argent-sentinel-agent: node timer, WordPress/Nginx staging helpers, SSH journald collection, and enrollment helpers.
• argent-sentinel-server: ingestion API, collector and watchdog timers, Nginx log rotation, PKI helpers, config migration, and reporting cutover tools.
• argent-sentinel: combined agent/server metapackage.

Installing packages never enables provider abuse reporting automatically. Validate redirected test mode before production delivery.

Configuration files

File	Purpose	Important options
/etc/argent-sentinel/collector.json	Central ingestion, correlation, enforcement, and reporting policy	incoming_globs, abuse_context, protection_inventory, policy, sshd_policy, web_policy, trusted_cidrs, enforcement_protection, crowdsec, enrichment, abuse_reporting
/etc/argent-sentinel/agent.json	Node transport, dynamic local-address protection, and SSH collection	enabled, node, central_url, certificate paths, local WordPress/Nginx globs, local_address_protection, sshd
/etc/argent-sentinel/server-api.json	Unix-socket ingestion API	socket_path, socket_group, nodes_dir, receipt_db, event_drop_root
/etc/argent-sentinel/nodes.d/NAME.json	Per-node enrollment and service authorization	node_id, enabled, allowed services, allowed WordPress site_ids
/etc/logrotate.d/argent-sentinel-nginx	Hourly rotation and staging of the filtered Nginx JSONL log	retention, compression, stage helper
/etc/nginx/conf.d/argent-sentinel-log-format.conf	Dedicated structured Nginx format and conditional logging maps	JSON fields and suspicious-request conditions
/etc/argent-sentinel/agent-privacy.key	HMAC secret used to pseudonymize SSH account names	root-readable, at least 32 bytes
/etc/argent-sentinel/pki/	Sentinel CA, node certificate, and private key material	protect private keys as root-only
/etc/argent-sentinel/watchdog.json	Global watchdog scheduling, retention, and notification policy	administrative and emergency recipient groups
/etc/argent-sentinel/watchdog.d/*.json	Local watchdog enable/disable and threshold overrides	merge by watchdog id

Package examples are installed under /usr/share/argent-sentinel/.

Collector input paths

collector.json must include both local and API-delivered event batches:

"incoming_globs": [
  "/var/lib/argent-sentinel/drop/wordpress/*/incoming/*.json",
  "/var/lib/argent-sentinel/drop/remote/*/events/incoming/*.json"
]

abuse_context.incoming_globs should include local and remote Nginx inputs:

[
  "/var/lib/argent-sentinel/drop/nginx/*/incoming/*.jsonl",
  "/var/lib/argent-sentinel/drop/nginx/*/incoming/*.json",
  "/var/lib/argent-sentinel/drop/remote/*/abuse-context/incoming/*.jsonl",
  "/var/lib/argent-sentinel/drop/remote/*/abuse-context/incoming/*.json"
]

The v0.4.7 server package appends missing required paths to preserved configurations without replacing custom settings.

Modular watchdogs

Version 0.5.5.0 adds a common one-minute scheduler with independently timed, process-isolated modules. Package definitions live in /usr/lib/argent-sentinel/watchdog.d/; operator overrides live in /etc/argent-sentinel/watchdog.d/. Packaged modules are disabled until locally enabled. The first upgrade migrates the recognized Unbound automatic-recovery watchdog and provides an observe-only PHP-FPM health module. Version 0.5.5.1 discovers the active versioned PHP-FPM systemd service and derives its matching binary, process name, and log path unless the operator supplies explicit overrides. Incremental log analysis is separated by selected target and positive master PID: a target or master change rebases the cursor at the current log end while service state, target-scoped zombies, queues, and application probes remain active on that check. Event-mechanism verification runs only when an operator explicitly requires a mechanism; an omitted, auto, or any expectation skips the sandbox-incompatible PHP-FPM configuration command.

Configure administrative and emergency recipients separately in /etc/argent-sentinel/watchdog.json. Emergency recipients may include email-to-SMS gateway addresses and receive concise critical messages only.

sudo argent-sentinel-watchdog --config /etc/argent-sentinel/watchdog.json validate-config
sudo argent-sentinel-watchdog --config /etc/argent-sentinel/watchdog.json status --json

The read-only dashboard publishes a Watchdogs page from sanitized status files. See docs/watchdogs.md for module behavior and override examples.

New combined server installation

The example assumes Debian or Ubuntu, Nginx, DNS for the Sentinel API hostname, and locally built .deb files.

1. Install all packages together:

   sudo apt install      ./argent-sentinel-common_0.4.7-1_all.deb      ./argent-sentinel-agent_0.4.7-1_all.deb      ./argent-sentinel-server_0.4.7-1_all.deb      ./argent-sentinel_0.4.7-1_all.deb
   

2. Review the generated configuration:

   sudoedit /etc/argent-sentinel/collector.json
   sudoedit /etc/argent-sentinel/agent.json
   sudoedit /etc/argent-sentinel/server-api.json
   

3. Set stable node.id and node.fqdn values.

4. Configure the API virtual host from:

   /usr/share/argent-sentinel/nginx-sentinel.conf.example
   

   Replace hostname and certificate paths, then:

   sudo nginx -t
   sudo systemctl reload nginx
   

5. Initialize the client-certificate CA and enroll nodes. Put node authorizations in /etc/argent-sentinel/nodes.d/. See docs/remote-transport.md.

6. Validate configuration:

   sudo argent-sentinel      --config /etc/argent-sentinel/collector.json validate-config
   
   sudo argent-sentinel-agent      --config /etc/argent-sentinel/agent.json validate-config
   
   sudo argent-sentinel-api      --config /etc/argent-sentinel/server-api.json validate-config
   

7. Enable the runtime:

   sudo systemctl enable --now      argent-sentinel-agent.timer      argent-sentinel-collector.timer      argent-sentinel-nginx-logrotate.timer      argent-sentinel-api.service
   

8. Verify schedules:

   systemctl list-timers --all |
     grep -E 'argent-sentinel-(agent|collector|nginx-logrotate)'
   

New agent-only node

Install argent-sentinel-common and argent-sentinel-agent, enroll the node certificate, authorize the node on the central server, and configure:

{
  "enabled": true,
  "node": {
    "id": "remote-node",
    "fqdn": "remote-node.example.org"
  },
  "central_url": "https://sentinel.example.org/",
  "cert_file": "/etc/argent-sentinel/pki/node.crt",
  "key_file": "/etc/argent-sentinel/pki/node.key",
  "sshd": {
    "enabled": true,
    "unit": "ssh.service",
    "initial_lookback_minutes": 60,
    "destination_ip": "PUBLIC_TARGET_IP",
    "destination_port": 22
  }
}

The lookback is used only when no journald cursor exists.

Adding a WordPress site

Use one stable lowercase site ID per installation. The PHP-FPM account must be able to create files under:

/var/lib/argent-sentinel/drop/wordpress/SITE_ID/incoming

The packaged onboarding helper creates the directory, grants the PHP-FPM user membership in the sentinel group, configures plugin options with WP-CLI, and runs a test export:

sudo argent-sentinel-onboard-wordpress   --wordpress-path /var/www/example-site   --site-id example-site   --node-id "$(hostname -s)"   --php-user www-data   --plugin-zip /root/argent-sentinel-wordpress.zip

When the plugin is installed already, omit --plugin-zip.

The lower-level directory helper is:

sudo argent-sentinel-create-wordpress-drop SITE_ID PHP_FPM_USER

It creates the directory with setgid mode 2770 and group sentinel. Restart the relevant PHP-FPM service after group membership changes.

Manual WP-CLI setup and checks

sudo -u PHP_FPM_USER -- wp --path=/var/www/example-site   argent-sentinel setup   --site-id=example-site   --source-host="$(hostname -s)"   --drop-directory=/var/lib/argent-sentinel/drop/wordpress/example-site/incoming   --format=json

sudo -u PHP_FPM_USER -- wp --path=/var/www/example-site   argent-sentinel status --format=json

sudo -u PHP_FPM_USER -- wp --path=/var/www/example-site   argent-sentinel export --format=json

Verify the filesystem independently:

namei -l /var/lib/argent-sentinel/drop/wordpress/example-site/incoming

sudo -u PHP_FPM_USER --   test -w /var/lib/argent-sentinel/drop/wordpress/example-site/incoming

For exact WordPress-to-Nginx request correlation, add this to every monitored PHP-FPM location:

fastcgi_param ARGENT_SENTINEL_REQUEST_ID $request_id;

WordPress plugin limitation in v0.4.7

The plugin admin diagnostics/setup page is not yet reliable for showing nonce validity or provisioning/writability failures. Until the plugin follow-up release, treat WP-CLI status, the onboarding helper, and direct filesystem tests as authoritative. The plugin should not be expected to create a missing system drop directory by itself.

Nginx web-probe collection

The active structured log is normally:

/var/log/nginx/argent-sentinel-abuse-context.jsonl

The package-owned timer rotates it hourly. Rotated files are staged by /usr/sbin/argent-sentinel-stage-abuse-context and imported on the next collector cycle.

Complete per-site traffic logs and filtered Sentinel JSONL

Use two Nginx access logs for monitored public sites:

access_log /var/log/nginx/wolfandraven.blog.access.log
           argent_site_access;

access_log /var/log/nginx/argent-sentinel-abuse-context.jsonl
           argent_sentinel_json
           buffer=64k
           flush=5s
           if=$argent_sentinel_loggable;

The per-site argent_site_access log is complete traffic accounting for AWStats, bandwidth/referrer analysis, and site-specific troubleshooting. The shared argent_sentinel_json file remains a filtered security/review feed for cross-site Sentinel correlation. Do not replace the complete per-site log with the filtered JSONL, and do not send all ordinary traffic into the Sentinel JSONL merely for AWStats.

Install the packaged per-site format:

sudo argent-sentinel-install-site-log-format

Then change each monitored virtual host from the older abuse_context format to argent_site_access, retain its site-specific filename, and keep the conditional argent_sentinel_json line. Validate every Nginx change:

sudo nginx -t
sudo systemctl reload nginx

Legacy per-site rotations in standard combined format remain usable because the AWStats manager normalizes them. Hostless records from a shared /var/log/nginx/access.log are ambiguous and are skipped rather than assigned to the wrong virtual host.

AWStats site inventory and reports

Create a clean proposed site inventory from configured Nginx server names and the extended host fields in current logs:

sudo argent-sentinel-awstats discover \
  --write-proposed /tmp/traffic-sites.proposed.json

sudo jq . /tmp/traffic-sites.proposed.json
sudo install -o root -g root -m 0640 \
  /tmp/traffic-sites.proposed.json \
  /etc/argent-sentinel/traffic-sites.json

Bare domains and matching www names are consolidated into one report with a host alias. Each generated site entry records only the Nginx files in which that virtual host was observed.

Inspect the resolved per-site log assignment before enabling the timer:

sudo argent-sentinel-awstats inspect
sudo argent-sentinel-awstats render
sudo argent-sentinel-awstats update

For parser troubleshooting, write the normalized combined stream to a file and inspect the completed output:

sudo argent-sentinel-awstats stream \
  --site wolfandraven.blog \
  > /tmp/wolfandraven.blog.combined.log

head /tmp/wolfandraven.blog.combined.log

Do not normally pipe the live stream directly into head. When head exits, the closed output pipe can produce an expected SIGPIPE (logresolvemerge status -13) even though the complete scheduled AWStats update is healthy.

Enable scheduled static report generation only after inspect shows the expected files:

sudo systemctl enable --now argent-sentinel-awstats.timer

Sites with no unambiguous matching log are reported as skipped; they no longer cause the entire AWStats service to fail.

The generated site configuration explicitly enables the AWStats report sections used by awstats_buildstaticpages.pl. This ensures that links from the summary page have matching static files, including urldetail and allrobots. Regenerate and verify after an upgrade:

sudo systemctl start argent-sentinel-awstats.service

find /var/lib/argent-sentinel/dashboard/awstats/wolfandraven.blog \
  -maxdepth 1 -type f \
  -name 'awstats.wolfandraven.blog.*.html' \
  -printf '%f\n' | sort

The listing should include at least:

awstats.wolfandraven.blog.urldetail.html
awstats.wolfandraven.blog.allrobots.html

Dashboard commands

Generate a fresh sanitized snapshot and restart the read-only service:

sudo systemctl start argent-sentinel-dashboard-snapshot.service
sudo systemctl restart argent-sentinel-dashboard.service

sudo systemctl status \
  argent-sentinel-dashboard-snapshot.service \
  argent-sentinel-dashboard.service \
  --no-pager -l

The dashboard service receives /etc/argent-sentinel/dashboard.json through a systemd read-only credential. It does not need traversal permission on the root-only /etc/argent-sentinel directory.

The publication tree uses a narrow filesystem boundary:

/var/lib/argent-sentinel
    root:sentinel 0750
    ACL group:www-data:--x

/var/lib/argent-sentinel/dashboard
/var/lib/argent-sentinel/dashboard/awstats
    root:www-data 0750

snapshot.json and generated AWStats files
    root:www-data 0640

The execute-only ACL lets the presentation group traverse the private state root without listing or reading the other Sentinel state directories. Do not add Nginx broadly to the sentinel group. Verify every ancestor directory:

namei -l /var/lib/argent-sentinel/dashboard/snapshot.json
getfacl -p /var/lib/argent-sentinel
sudo -u www-data test -r \
  /var/lib/argent-sentinel/dashboard/snapshot.json

Test the dashboard directly through its Unix socket:

curl --unix-socket \
  /run/argent-sentinel-dashboard/dashboard.sock \
  http://localhost/healthz

In the combined Nginx virtual host, /healthz checks the ingestion API and /dashboard-healthz checks the dashboard worker. Server-level client verification must remain optional; /v1/ingest separately requires $ssl_client_verify = SUCCESS.

The dashboard locations use satisfy all, so an allowed client without valid credentials receives HTTP 401 while a disallowed address receives HTTP 403. IPv6 ULA space (fc00::/7) does not include a residential globally routed prefix. Add the actual LAN IPv6 prefix, normally a /64, when browsers connect over global IPv6.

Developer and deployment host conventions are recorded in AGENTS.md. In particular, browser downloads on fafnir are under ~/Downloads/ (plural), not inside the ChatGPT /mnt/data sandbox.

Slow-burn WordPress credential policy

The short-window WordPress rules remain the primary burst detector. A separate site-scoped policy detects confirmed failures deliberately spread across a day:

"persistent_wordpress_policy": {
  "enabled": true,
  "window_seconds": 86400,
  "failure_threshold": 6,
  "distinct_accounts": 2,
  "single_account_threshold": 12,
  "incident_merge_seconds": 86400,
  "abuse_reporting_enabled": false
}

The rules are wordpress-persistent-credential-spray and wordpress-persistent-single-account-bruteforce. They use only WordPress login_failed / denied events, exclude evidence already assigned to the stronger 15-minute WordPress rules, and never combine evidence from separate sites. CrowdSec decisions follow the normal policy. Provider abuse reports are created as suppressed with the detail Persistent WordPress policy reporting disabled pending production review until the operator enables reporting for newly created persistent incidents.

The collector evaluates the current rolling window on every run, including after an upgrade, so recent stored evidence can qualify without waiting for a new event batch. Existing Nginx login rate limiting remains an independent resource-protection layer.

Abuse-report connection formatting is service aware: OpenSSH events use scheme=ssh in both human-readable connection details and sanitized normalized evidence, while web events retain http or https.

SSH collection

The agent reads journalctl -u ssh.service, pseudonymizes account names with HMAC-SHA256, and submits ssh_auth_failed events. It never sends usernames. SSH thresholds are controlled by sshd_policy in collector.json.

Redirected test reporting

Before production, use test mode:

"abuse_reporting": {
  "enabled": true,
  "test_mode": true,
  "recipient_override": "operator@example.org",
  "max_reports_per_run": 2,
  "max_reports_per_recipient_per_day": 10,
  "recipient_cooldown_minutes": 15
}

Provider abuse contacts and administrative Bcc recipients are not used in test mode. Production also requires report_not_before_utc; see docs/production-reporting.md.

Operational checks

argent-sentinel --version

systemctl status   argent-sentinel-agent.timer   argent-sentinel-collector.timer   argent-sentinel-nginx-logrotate.timer   argent-sentinel-api.service   --no-pager

Database inventory:

sqlite3 -header -column   /var/lib/argent-sentinel/collector/state.sqlite3 '
SELECT service, event_type, COUNT(*) AS events
FROM events
GROUP BY service, event_type;

SELECT rule_id, report_status, COUNT(*) AS incidents
FROM incidents
GROUP BY rule_id, report_status;

SELECT attempted_at, recipient, status, test_mode, detail
FROM report_attempts
ORDER BY attempt_id DESC
LIMIT 20;
'

pending_reports: 0 means no report is currently awaiting work. It does not mean there are no events or incidents; eligible reports are normally processed during the same collector run.

See docs/ for packaging, remote enrollment, SSH privacy, web-probe policy, XARF reporting, production reporting, and legacy migration.

Architecture and roadmap

See ARCHITECTURE.md, TODO.md, and docs/fail2ban-review-policy.md.

Operator dashboard

Version 0.5.1.1 provides a read-only dashboard intended for sentinel.argentwolf.org, a root-generated sanitized snapshot, static per-site AWStats reports, and an operator-controlled Nginx crawler policy. See ARCHITECTURE.md and docs/dashboard.md.

PHP-FPM open_basedir onboarding

argent-sentinel-onboard-wordpress inspects PHP-FPM pools matching the supplied PHP user. When an active pool-level open_basedir excludes the protected event drop, the helper prompts to append it. Automation can choose --open-basedir-mode append, warn, or ignore. Automatic edits receive a timestamped backup and PHP-FPM configuration validation. Use --restart-php-fpm to restart a modified pool immediately.

Automatic PHP-FPM restart after onboarding

argent-sentinel-onboard-wordpress restarts every matching PHP-FPM service once, at the end of successful onboarding. This activates the PHP user's new sentinel supplementary group and any accepted pool-level open_basedir change before the operator returns to the WordPress web UI. Use --no-restart-php-fpm only when a controlled restart will immediately follow.

Unrestricted PHP-FPM pools

A matching per-site PHP-FPM pool with no active pool-level open_basedir is reported as a security-hardening failure. Interactive onboarding requires confirmation; non-interactive onboarding stops before plugin configuration. Use --no-open-basedir-action continue only after reviewing inherited/global restrictions or deliberately accepting the broader filesystem reach. open_basedir remains defense in depth rather than a complete security boundary.

Hourly CIDR report batching

When report_batching.enabled is true, the minute collector continues to import and correlate events and submit immediate per-source enforcement decisions, but leaves provider reports queued. The packaged argent-sentinel-report-batch.timer sends CIDR-grouped summaries at five minutes past each hour.

Each group is separated by CIDR, activity family, and recipient set. A multi-incident message carries one XARF JSON attachment per incident. Recipient limits count distinct outbound Message-ID values rather than incident rows.

The default ban-only policy includes Meta AS32934 and 2a03:2880::/32. meta-externalagent is retained as a supplemental token, but User-Agent-only suppression is disabled by default.

See docs/hourly-report-batching.md.

WordPress collector inventory

argent-sentinel-wordpress-sites \
  --expect 'example.org=example-org' \
  --format table

seen means the database contains an imported batch or WordPress event. provisioned-no-import means the protected drop exists but no batch has yet arrived.

Bounded operational report groups

Version 0.5.1.1 separates provider ownership scope from report aggregation scope. Broad registered allocations remain available for recipient selection, while hourly messages are bounded to /24 for IPv4 and /48 for IPv6 by default. The Reports dashboard shows both prefixes, current queues, the latest hourly run, recent message IDs, and ban-only suppressions.

Audited dashboard reviews

Version 0.5.2.0 publishes one open review item per incident rather than counting individual report attempts. Authenticated dashboard actions are written to a restricted spool and applied by a root-owned systemd path processor under the collector lock. See docs/dashboard-review-workflow.md.

Review refinements in 0.5.2.1

No-contact incidents close automatically only after the local CrowdSec IP ban is verified. Suppressed WordPress credential-spray incidents expose audited approval, suppression, duplicate/subsumed, contact-refresh, and note actions. See docs/review-workflow-0.5.2.1.md.

Enforcement-protected CIDRs in 0.5.3.1

enforcement_protection.protected_cidrs identifies addresses and prefixes that Argent Sentinel must never submit to CrowdSec as an IP or range decision. Unlike trusted_cidrs, these networks remain eligible for telemetry and review. The dashboard marks overlapping proposals as protected, suppresses block actions, and permits an audited acknowledgment or note. The root-owned review processor independently refuses enforcement even if a stale or forged request reaches the spool.

Version 0.5.4.0 extends this foundation with authenticated per-node dynamic inventories. Each agent recalculates qualifying public IPv6 addresses on every run and sends a changed inventory immediately, with periodic heartbeats. The collector publishes the merged effective set and the root review processor fails closed if that publication is unavailable or stale.

Dynamic local-address modes

The argent-sentinel-agent package prompts on first installation, or on an upgrade where local_address_protection is absent. It displays discovered public IPv6 addresses and a recommendation:

• host: protect each current address as /128; recommended for VPS, cloud, virtualized, or uncertain systems.
• lan-prefix: protect each current connected public IPv6 prefix; select only after confirming ownership or control of the complete prefix.
• manual: protect only explicitly entered CIDRs, bounded to /24 or narrower for IPv4 and /48 or narrower for IPv6.
• off: disable dynamic inventory protection while leaving static collector protections intact.

An unattended installation never broadens to a prefix. It records unconfirmed host mode and begins protecting current /128 addresses as soon as the agent is enabled and enrolled. Review or change the choice later with:

sudo dpkg-reconfigure argent-sentinel-agent

Inspect discovery and the current effective inventory with:

sudo argent-sentinel-local-protection discover
sudo argent-sentinel-agent \
  --config /etc/argent-sentinel/agent.json \
  protection-inventory

The address inventory is an enforcement safety boundary, not a source-trust allowlist. Dynamic addresses remain visible in telemetry and incident correlation.

Audited CIDR review in 0.5.3.0

The Networks page preserves each RDAP registered allocation as the ownership scope while displaying a separate most-specific enforcement proposal. Argent Sentinel first selects the strongest /24 IPv4 or /48 IPv6 evidence group, then narrows the proposal to the smallest common prefix containing those hostile addresses. The snapshot shows proposal counts, active days, address-space coverage, derivation basis, revision, and CrowdSec decision state.

Authenticated operators may queue 180-day or 365-day range blocks, keep a case under observation, reject the current recommendation, add a note, or remove an existing range block. The dashboard writes only an immutable request file. The root-owned review processor verifies the current proposal revision, containment inside the registered case, configured /24 and /48 safety bounds, trusted network overlap, exact duration, and a nonempty block justification before using cscli decisions add --range. It never requests allowlist bypass.

Automatic CIDR blocking remains disabled. Count/coverage thresholds and VPN endpoint classification are deferred until manual audited range decisions have enough production history. A future percentage threshold must use an explicit bounded-scope denominator rather than the minimal proposal's inherently biased coverage. See docs/cidr-review-0.5.3.0.md.