> For the complete documentation index, see [llms.txt](https://docs.soniclabs.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/appendix-c-slack-alert-script.md).

# Appendix C: Slack Alert Script

Post Tier 3 security events to a Slack channel.

A simple way to get the [Tier 3](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-3-get-alerted.md) alerts to a person if you don't have central logging yet. Every 5 minutes, it posts any new security events to a Slack channel. It also warns you if the `sonic-guard` rules ([Tier 3.2](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-3-get-alerted.md)) have gone missing or the node has stopped.

**1. Create a Slack webhook.** Follow Slack's guide at <https://api.slack.com/messaging/webhooks>: create an app, turn on "Incoming Webhooks", and add a webhook for the channel that should get the alerts. You get a URL starting with `https://hooks.slack.com/services/`. Treat it like a password: anyone who has it can post to that channel.

**2. Install the script:**

````bash
sudo apt install jq      # used to build the message safely (dnf install jq on RHEL)

sudo tee /usr/local/sbin/sonic-alert-check > /dev/null <<'EOF'
#!/bin/bash
# Posts new Sonic security events to a Slack channel.
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin   # cron's default PATH misses ausearch and nft
WEBHOOK="https://hooks.slack.com/services/REPLACE/ME"   # EDIT: your Slack webhook URL
STATE=/var/lib/sonic-alert-check
mkdir -p "$STATE"

# Each command remembers where it stopped last time, so you only get new events.
EVENTS=$(
  ausearch --checkpoint "$STATE/exec.ckpt"   -k sonic_exec   -i 2>/dev/null
  ausearch --checkpoint "$STATE/tamper.ckpt" -k sonic_tamper -i 2>/dev/null
  journalctl --cursor-file="$STATE/journal.cursor" -k -g SONIC-PIVOT -q --no-pager 2>/dev/null
  nft list table inet sonic_guard > /dev/null 2>&1 || echo "WARNING: sonic-guard firewall rules are not loaded"
  systemctl is-active --quiet sonic || echo "WARNING: the sonic service is not running"
)

if [ -n "$EVENTS" ]; then
  # Slack needs &, < and > escaped. The events go in a code block so they stay readable.
  jq -n --arg host "$(hostname)" --arg events "$(printf '%s' "$EVENTS" | head -c 3500)" '
    ($events | gsub("&"; "&amp;") | gsub("<"; "&lt;") | gsub(">"; "&gt;")) as $e
    | {text: "*Sonic security alert on \($host)*\n```\n\($e)\n```"}' |
    curl -fsS -m 10 -H 'content-type: application/json' -d @- "$WEBHOOK" > /dev/null
fi
EOF
sudo chmod 700 /usr/local/sbin/sonic-alert-check
````

**3. Test it once by hand, then run it every 5 minutes:**

```bash
sudo /usr/local/sbin/sonic-alert-check
echo '*/5 * * * * root /usr/local/sbin/sonic-alert-check' | sudo tee /etc/cron.d/sonic-alert-check
```

* The first run posts everything already in the logs (at least the node's own start event). After that, only new events.
* Messages are cut at about 3,500 characters, so always look at the full logs on the server too.
* While the node is stopped, for example during maintenance, the script posts a warning every 5 minutes.
* On a validator, an unexpected stop can mean the node saw another node signing with its key. Check the log as described in [Tier 3.4](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-3-get-alerted.md).
* Other chat tools with incoming webhooks (Discord, Mattermost, ...) work too; only the message format in the `jq` line changes.
