> For the complete documentation index, see [llms.txt](https://docs.soniclabs.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-3-get-alerted.md).

# Tier 3: Get Alerted

Find out within minutes when someone tries to abuse the node.

**Goal:** if someone tries to abuse the node, you know about it within minutes, not weeks.

[Tier 2](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-2-lock-the-node-in-a-box.md) blocks attempts silently. This tier makes them loud. With [Tier 2](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-2-lock-the-node-in-a-box.md) in place, any alert from this tier, apart from the expected events listed in 3.1, is a strong sign that someone has broken into the node process. Treat it as an incident.

{% hint style="info" %}
**Validators:** this tier is required, because the signing key is on the server.
{% endhint %}

## 3.1 Log every program start by the node user

Install the Linux audit system:

```bash
sudo apt install auditd        # Debian/Ubuntu (already installed on RHEL)
sudo systemctl enable --now auditd
```

Create the rule file (replace `sonic` if your user is named differently):

```bash
SONIC_UID=$(id -u sonic)
sudo tee /etc/audit/rules.d/sonic.rules > /dev/null <<EOF
# Any attempt by the node user to start a program (successful or blocked)
-a always,exit -F arch=b64 -S execve,execveat -F uid=$SONIC_UID -k sonic_exec
# The same through the 32-bit interface, which attackers use to slip past 64-bit-only rules
-a always,exit -F arch=b32 -S execve,execveat -F uid=$SONIC_UID -k sonic_exec
# Any change to the node binary or its config
-w /usr/local/bin/sonicd -p wa -k sonic_tamper
-w /etc/sonic -p wa -k sonic_tamper
EOF
sudo augenrules --load
```

Adjust the paths in the last two lines to your binary and config folder.

**Validators:** also watch the keystore, which shouldn't change once the signing key is in place. Add this line to `/etc/audit/rules.d/sonic.rules` (with your data folder) and run `sudo augenrules --load` again:

```
-w /var/lib/sonic/keystore -p wa -k sonic_tamper
```

The `/etc/sonic` line already covers a password file there (Validator Node step 9). If you use an encrypted credential instead, add `-w /etc/credstore.encrypted -p wa -k sonic_tamper` too.

To see events:

```bash
sudo ausearch -k sonic_exec -i --start today
sudo ausearch -k sonic_tamper -i --start today
```

**Expected events** (not alarms, but check that each one matches something you did):

* **One** `sonic_exec` event each time the service starts: that's the node itself starting.
* `sonic_exec` events when an admin runs a tool as the node user, e.g. `sudo -u sonic sonictool ...` for database maintenance. Write these down when you do them, so whoever reads the alert knows.
* `sonic_tamper` events when you upgrade the node or change its config.

**Anything else is an alarm**: in particular any event showing `/bin/sh`, `bash`, `curl`, `wget`, `python` or `res=no`/`EACCES` (a blocked attempt).

## 3.2 Log blocked attempts to reach your private network

The block from [Tier 2B](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-2-lock-the-node-in-a-box.md) drops traffic silently. This adds a matching firewall rule that also **writes a log line** for every blocked attempt.

Create the rules file:

```bash
sudo nano /etc/sonic-guard.nft
```

```
# Recreate our table cleanly every time this file is loaded
table inet sonic_guard {}
delete table inet sonic_guard

table inet sonic_guard {
  set internal4 { type ipv4_addr; flags interval;
    elements = { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, 169.254.0.0/16 } }
  set internal6 { type ipv6_addr; flags interval;
    elements = { fc00::/7, fe80::/10 } }

  # EDIT: the same private IPs and subnets you allowed in Tier 2: IPv4 in allowed4,
  # IPv6 in allowed6. A set with no entries has no "elements" part, like allowed6 here.
  set allowed4 { type ipv4_addr; flags interval; elements = { 10.0.1.20 } }
  set allowed6 { type ipv6_addr; flags interval; }

  chain out {
    type filter hook output priority 0; policy accept;
    meta skuid "sonic" ip  daddr @allowed4 accept
    meta skuid "sonic" ip6 daddr @allowed6 accept
    # Log at most 10 lines a minute, so an attacker can't flood your logs...
    meta skuid "sonic" ip  daddr @internal4 limit rate 10/minute log prefix "SONIC-PIVOT "
    meta skuid "sonic" ip6 daddr @internal6 limit rate 10/minute log prefix "SONIC-PIVOT "
    # ...but always drop. Keep the drop on its own line: if it shared a line with
    # the rate limit, traffic above the limit would be let through.
    meta skuid "sonic" ip  daddr @internal4 drop
    meta skuid "sonic" ip6 daddr @internal6 drop
  }
}
```

Load it now and on every boot, using a small service so it doesn't interfere with `ufw`/`firewalld`:

```bash
sudo tee /etc/systemd/system/sonic-guard.service > /dev/null <<'EOF'
[Unit]
Description=Log and block Sonic node connections to internal networks
After=network-pre.target nftables.service
Before=sonic.service
# If the nftables service is restarted, reload these rules too
PartOf=nftables.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/nft -f /etc/sonic-guard.nft
ExecStop=/usr/sbin/nft delete table inet sonic_guard

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now sonic-guard
sudo nft list table inet sonic_guard      # check it's loaded
```

{% hint style="info" %}
On Debian, the default `/etc/nftables.conf` starts with `flush ruleset`, which deletes **all** firewall rules, including these. After any `systemctl reload nftables` or `nft -f /etc/nftables.conf`, run `sudo systemctl restart sonic-guard`.&#x20;

The alert script in [Appendix C](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/appendix-c-slack-alert-script.md) also warns you if the rules have gone missing.
{% endhint %}

To see blocked attempts:

```bash
sudo journalctl -k -g SONIC-PIVOT --since today
```

## 3.3 Send alerts somewhere a human will see them

Logs on the server aren't enough. An attacker with enough access can delete them, and nobody reads them anyway. Pick one:

* **If you already have central logging** (Loki, Elastic, Datadog, Splunk, CloudWatch, ...): forward the journal and `/var/log/audit/audit.log`, and create alerts for:
  * audit key `sonic_exec` (other than the normal one-per-start event)
  * audit key `sonic_tamper`
  * kernel log lines containing `SONIC-PIVOT`
  * the node service stopping unexpectedly
* **If you don't**, start simple: [**Appendix C**](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/appendix-c-slack-alert-script.md) has a script that posts new events to a Slack channel every 5 minutes. Then plan proper central logging.

## 3.4 Know what to do when an alert fires

Write this down now, and agree on who does what:

1. **Isolate.** Cut the server off from your internal network (cloud security group / switch port), but don't power it off.
2. **Preserve evidence.** Take a disk snapshot (and a memory snapshot if your platform supports it). Copy logs off the machine.
3. **Find the entry point.** Check your RPC proxy logs ([Tier 4](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-4-control-access.md)) for the requests just before the alert: source IP, client identity, method, request body.
4. **Rotate everything** the server could have touched: API keys, tokens, any credentials on or used by that machine.
5. **Rebuild, don't clean.** Reinstall the server from a known-good image and re-sync or restore the data folder from a trusted copy.
6. **Tell the Sonic team** at <security@soniclabs.com>, so other operators can be warned.

**Validators: assume the signing key is stolen.** Someone with a copy can sign in your validator's name, and if they sign while your node does, that's double-signing. Change the steps above like this:

* **Stop the validator** right after step 1, and make sure it can't start again: `sudo systemctl disable --now sonic`.
* **Tell the Sonic team at once**, not at step 6, and follow their advice on the signing key.
* **Never run the old and the new server at the same time.** In step 5, start the rebuilt validator only once the old one is stopped for good, and don't start it with a key that may be stolen until you've talked to the Sonic team.

**The node may warn you first.** A validator can stop itself when it receives a recent event signed with its key that it didn't create. Look for it in the log:

```bash
sudo journalctl -u sonic -g 'duplicating instance|previous error'
```

`duplicating instance` means another node is signing with your key: a forgotten second copy of your own validator (stop it), or a stolen key (an incident). After that, the node refuses to start with `Node isn't allowed to start due to a previous error`, until someone deletes the `errlock` file the message names. Don't delete it until you know which of the two it was.

## ✅ Tier 3 checklist

* [ ] `auditd` running with the `sonic_exec` and `sonic_tamper` rules (validators: keystore too)
* [ ] `sonic-guard` loaded and enabled
* [ ] Alerts reach a person, not just a log file (alert script or central logging, tested once)
* [ ] Expected events (restarts, maintenance, upgrades) documented so they aren't ignored or mistaken
* [ ] Incident steps written down, with an owner
* [ ] Validators: incident steps cover the signing key (stop first, tell the Sonic team at once, never two servers with the same key)
