> For the complete documentation index, see [llms.txt](https://docs.soniclabs.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-2-lock-the-node-in-a-box.md).

# Tier 2: Lock the Node in a Box

Keep a compromised node from reaching your private network or starting other programs.

**Goal:** even if an attacker exploits a bug and takes control of the node process, they **can't connect to anything on your private network**, and it's **very hard for them to start any other program**.

Both halves of this tier matter:

* **2A** stops an attacker from starting a shell or downloading and running their own tools, which is usually the first thing they try.
* **2B** stops a compromised node from being used to attack your other servers.

What this tier can't do: an attacker who controls the node process can still use the node itself. They can read its data folder, reach the internet, and do anything their own code inside the node can do. That's why [Tier 1.7](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-1-essentials.md) (no secrets on the machine) and [Tier 3](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-3-get-alerted.md) (alerts) still matter.

**Validators:** that includes the signing key. The node holds it unlocked in memory to sign, so an attacker in control of the node can use or copy it. This tier keeps them from going further, and [Tier 3](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-3-get-alerted.md) tells you it happened.

We'll use systemd's built-in sandboxing. You don't edit your existing service file. You add a small extra file next to it (systemd calls this a "drop-in"). That makes the change easy to review and easy to undo.

**Before you start this tier:**

* The node must run as its own non-root user ([Tier 1.6](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-1-essentials.md)). Most of this tier can be undone from inside a node that runs as `root`.
* `ExecStart=` must run `sonicd` directly, with no wrapper script and no `ExecStartPre=` lines ([Before you start](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/before-you-start.md), step 2).

## 2.0 Preparation: list the files the node needs to run

The node binary depends on a few system libraries. We'll allow exactly those files to be executed and nothing else. Run this (adjust `BIN` if your binary is elsewhere):

```bash
BIN=/usr/local/bin/sonicd
echo "ExecPaths=$( { echo "$BIN"; ldd "$BIN" 2>/dev/null | grep -o '/[^ ]*'; } \
  | xargs -n1 readlink -f | sort -u | tr '\n' ' ')"
```

It prints one line similar to:

```
ExecPaths=/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /usr/lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libm.so.6 /usr/local/bin/sonicd
```

Keep that line. You'll paste it below. If the output contains only the binary path, your build is statically linked, and that's fine.

{% hint style="info" %}
**Re-run this after every Sonic upgrade or OS upgrade**, in case the list of libraries changes. If the node fails to start after an upgrade with "Permission denied", this is the first thing to check.
{% endhint %}

## 2A + 2B: create the hardening file

```bash
sudo mkdir -p /etc/systemd/system/sonic.service.d
sudo nano /etc/systemd/system/sonic.service.d/10-hardening.conf
```

Paste the following, then **edit the five places marked `EDIT`**:

```ini
[Service]

########################################################################
# 2A. The node may not start any other program
########################################################################

# Nothing on the system may be executed by the node...
NoExecPaths=/
# ...except the node itself and the libraries it needs.
# EDIT 1: replace this line with the one printed in step 2.0
ExecPaths=/usr/local/bin/sonicd

# Block the usual tricks for running a program without a file on disk.
MemoryDenyWriteExecute=yes
SystemCallFilter=~memfd_create
SystemCallErrorNumber=EPERM

# Use Go's own DNS lookup instead of system plugins (which would be blocked above).
Environment=GODEBUG=netdns=go

# The node can never gain more privileges than it starts with.
NoNewPrivileges=yes
RestrictSUIDSGID=yes
# The node gets no special privileges at all.
# EDIT 2: only if the node listens on a port below 1024 (e.g. 80), use instead:
#   CapabilityBoundingSet=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=

# The rest of the system is read-only for the node...
ProtectSystem=strict
# ...except its data folder.
# EDIT 3: your --datadir folder
ReadWritePaths=/var/lib/sonic

# Hide home folders, give the node its own empty /tmp and a minimal /dev.
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes

# The node can't see other processes or change kernel settings.
ProtectProc=invisible
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
RestrictNamespaces=yes
LockPersonality=yes
RestrictRealtime=yes

# Only normal network sockets: no raw packet sniffing.
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK

########################################################################
# 2B. The node may not talk to your private network
########################################################################

# Block all private / internal address ranges (both directions).
IPAddressDeny=10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 169.254.0.0/16 fc00::/7 fe80::/10

# Allow this machine itself (for a local proxy, local DNS resolver, etc.).
IPAddressAllow=localhost

# EDIT 4: allow ONLY the specific private IPs that really must talk to the node,
# one per line, for example:
#   - your load balancer or RPC proxy (if it's on another machine)
#   - RPC clients on your private network
#   - your monitoring server (if it scrapes metrics over the private network)
#   - your DNS server, if it has a private IP (see "Checking DNS" below)
#   - private peers (e.g. your own other Sonic nodes on private IPs)
#IPAddressAllow=10.0.1.20
#IPAddressAllow=10.0.1.21

# The node can only open listening ports it's supposed to.
# This stops an attacker from opening a new port to connect in through.
# EDIT 5: one line per port from "Before you start" step 4.
SocketBindDeny=any
SocketBindAllow=tcp:5050
SocketBindAllow=udp:5050
SocketBindAllow=tcp:18545
SocketBindAllow=tcp:18546
SocketBindAllow=tcp:6060
```

{% hint style="info" %}
systemd only accepts comments on their own line, starting with `#`. Never put a comment at the end of a setting line.
{% endhint %}

**Notes on the edits:**

* **If your data folder is under `/home`**, change `ProtectHome=yes` to `ProtectHome=read-only`. The `ReadWritePaths=` line still lets the node write to its own folder.
* **If the node writes logs to a file** (not just the journal), add that folder to `ReadWritePaths=` (space-separated).
* **The block in 2B also applies to incoming connections.** Any private-network machine that needs to reach the node (load balancer, monitoring, internal apps, VPN users) must be listed in `IPAddressAllow=`. Public-internet peers and clients aren't affected.
* **Public RPC behind a cloud load balancer:** many cloud load balancers (e.g. AWS ALB/NLB, or a Hetzner load balancer on a private network) connect from private IPs that change over time. Allow the load balancer's whole subnet, e.g. `IPAddressAllow=10.0.1.0/24`, not single IPs. Your cloud console shows the subnet.
* **`100.64.0.0/10`** is used by some VPNs (e.g. Tailscale). If you reach the node over such a VPN, allow the specific addresses you use.
* **If step 3 showed `-BPF_FRAMEWORK`**, keep the `SocketBind…` lines anyway (they're harmless), but know that they aren't enforced on your system.
* **Validators, EDIT 5:** keep only the two `5050` lines (plus `6060` if you use profiling or metrics). Then the node can't open an RPC port, even if a flag turns RPC on by mistake.
* **Validators, password file:** it must not be under `/home` or `/root`, which `ProtectHome=yes` hides from the node. The options in [Validator Node](https://docs.soniclabs.com/pages/ycklmVREhe4bDmkq6Vvn#id-9.-run-your-validator-node) step 9 all keep it elsewhere.

**Checking DNS.** The node needs DNS to find bootnodes and peers by name. Find your DNS server:

```bash
grep nameserver /etc/resolv.conf
```

* `127.0.0.53` or `127.0.0.1`: local resolver, already allowed by `IPAddressAllow=localhost`.
* A public IP (e.g. `1.1.1.1`): not affected.
* A private IP (e.g. `10.0.0.2`) or `169.254.169.253` (AWS): add it with `IPAddressAllow=`.
* `169.254.169.254` (GCP): **don't** add it. On GCP the DNS server is also the metadata service, and allowing it would let the node reach your cloud credentials again. Switch the server to a local resolver such as systemd-resolved (the default on Ubuntu), so `/etc/resolv.conf` shows `127.0.0.53`.

## Apply it

```bash
sudo systemd-analyze verify sonic.service     # reports typos, if any
sudo systemctl daemon-reload
sudo systemctl restart sonic
sudo systemctl status sonic
sudo journalctl -u sonic -f       # watch for errors; Ctrl+C to stop
```

**Validators:** every restart takes the validator offline until the node is back. With a named pipe (Validator Node step 9), that's until you send the password again with `sudo sonic-unlock`.

## Test that it works

**a) The node is healthy.** It's syncing and answering requests:

```bash
curl -s -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"eth_blockNumber","params":[]}' \
  http://127.0.0.1:18545        # use your RPC port
```

Run it twice a few seconds apart. The block number should increase. Also check that the peer count recovers (`net_peerCount`).

**Validators** have no RPC, so check the log instead:

```bash
sudo journalctl -u sonic --since "2 min ago" -g 'New block|New event emitted'
```

Once the node has caught up, `New block` lines show an `age` of a few seconds, and `New event emitted` lines show that the validator is signing again.

**b) The node can't start other programs.** This tries to start a shell *inside the node's sandbox*:

```bash
PID=$(systemctl show -p MainPID --value sonic)
sudo nsenter --target "$PID" --mount /bin/sh -c 'echo NOT PROTECTED'
```

✅ Expected: `nsenter: failed to execute /bin/sh: Permission denied`

❌ If you see `NOT PROTECTED`, the setting didn't apply. Check `systemctl cat sonic` shows your file, and your systemd version.

An attacker's next move is to save their own program somewhere the node can write, and run it from there. Check those places too:

```bash
DATADIR=/var/lib/sonic      # EDIT: your data folder
NODETMP=$(sudo sh -c 'ls -d /tmp/systemd-private-*-sonic.service-*/tmp' | head -1)
sudo cp /bin/true "$DATADIR/.exec-test"
sudo cp /bin/true "$NODETMP/.exec-test"
sudo nsenter --target "$PID" --mount "$DATADIR/.exec-test" && echo "data folder: NOT PROTECTED"
sudo nsenter --target "$PID" --mount /tmp/.exec-test && echo "/tmp: NOT PROTECTED"
sudo rm -f "$DATADIR/.exec-test" "$NODETMP/.exec-test"
```

✅ Expected: two `Permission denied` messages, and no `NOT PROTECTED`.

**c) The network block works.** Pick a private IP and port on your network that you know is reachable from this server (e.g. another server's SSH port), and try it once normally and once with the same block applied:

```bash
TARGET=10.0.1.5; PORT=22      # EDIT: a reachable internal machine

timeout 5 bash -c "echo > /dev/tcp/$TARGET/$PORT" && echo "normal: REACHABLE"

sudo systemd-run --quiet --wait --pipe \
  -p IPAddressDeny="10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 169.254.0.0/16" \
  -p IPAddressAllow=localhost \
  /usr/bin/timeout 5 /bin/bash -c "echo > /dev/tcp/$TARGET/$PORT && echo 'sandboxed: REACHABLE (BAD)' || echo 'sandboxed: BLOCKED (GOOD)'"
```

❌ If you see `REACHABLE (BAD)`, your system can't enforce 2B. See "Older systems" below.

Also confirm the live service picked up the settings:

```bash
systemctl show sonic -p IPAddressDeny -p IPAddressAllow -p NoExecPaths -p ExecPaths -p CapabilityBoundingSet
```

**d) The port restriction works (optional).** This needs `python3`, which Ubuntu installs by default:

```bash
sudo systemd-run --quiet --wait --pipe -p SocketBindDeny=any /usr/bin/python3 -c '
import socket
try:
    socket.socket().bind(("127.0.0.1", 9999)); print("port block: NOT ENFORCED")
except PermissionError:
    print("port block: ENFORCED (GOOD)")'
```

`NOT ENFORCED` is expected if step 3 of [Before you start](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/before-you-start.md) showed `-BPF_FRAMEWORK`.

**e) Get an overall score (optional).**

```bash
systemd-analyze security sonic
```

Lower is better. Don't chase a perfect score; the settings above cover what matters most.

## If something breaks: roll back

```bash
sudo mv /etc/systemd/system/sonic.service.d/10-hardening.conf /root/10-hardening.conf.disabled
sudo systemctl daemon-reload
sudo systemctl restart sonic
```

Common causes:

| Symptom                                                                            | Likely cause                                     | Fix                                                                                                         |
| ---------------------------------------------------------------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| Fails to start, "Permission denied" / status `203/EXEC`                            | Library missing from `ExecPaths=`                | Re-run step 2.0 and update the line                                                                         |
| Fails to start, status `203/EXEC`, and `ExecStart=` is a script or `bash -c`       | Scripts can't run in the sandbox                 | Put the `sonicd` command line directly in `ExecStart=`                                                      |
| Starts, but no peers / can't resolve names                                         | DNS server on a private IP is blocked            | Add it to `IPAddressAllow=`                                                                                 |
| Fails with "Address already in use" / bind error                                   | A port is missing from `SocketBindAllow=`        | Compare with `ss -tulpn` output from before                                                                 |
| Fails with "permission denied" on a port below 1024 (e.g. 80)                      | `CapabilityBoundingSet=` is empty                | See EDIT 2, or move RPC to a high port                                                                      |
| Internal app / monitoring / load balancer can't connect anymore                    | Its IP is blocked by 2B                          | Add its IP (or subnet) to `IPAddressAllow=`                                                                 |
| Fails with "Read-only file system"                                                 | Node writes somewhere outside its data folder    | Add that folder to `ReadWritePaths=`                                                                        |
| Crashes with a memory error or "operation not permitted", and works after rollback | `MemoryDenyWriteExecute=` or `SystemCallFilter=` | Remove those lines one at a time to find which, and report it to the Sonic team at <security@soniclabs.com> |

## Older systems

On the outdated systems in [Before you start](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/before-you-start.md) step 3, systemd can't block program starts (2A), and the network block (2B) may not be enforced either (test (c) tells you). The protections in this tier depend on a recent OS, so **don't run Sonic nodes on these systems**. Move to the recommended Ubuntu 24.04 LTS.

Until you've upgraded, Tiers 1, 3 and 4 still apply. The `sonic-guard` rules from [Tier 3.2](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-3-get-alerted.md) also block the node's outgoing connections to your private network, so set them up first.

## 2C. Optional extras (test before relying on them)

These allow only the kernel functions that normal services need. Add them, restart, and watch the node for a day:

```ini
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@mount @debug @module @reboot @swap @obsolete @cpu-emulation @raw-io memfd_create
```

**Replace** the `SystemCallFilter=~memfd_create` line from 2A with these three lines, and keep `SystemCallErrorNumber=EPERM`. The order matters: the `@system-service` line must come before any line starting with `~`, or systemd treats the whole list differently and blocks much less.

**Remove privileged-port access.** If the node listens on a port below 1024 (EDIT 2 and [Tier 1.6](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-1-essentials.md)), move RPC to a high port, for example behind a proxy ([Tier 4](https://docs.soniclabs.com/sonic/node-deployment/node-hardening/tier-4-control-access.md)). Then remove `AmbientCapabilities=CAP_NET_BIND_SERVICE` and set `CapabilityBoundingSet=` back to empty, so the node has no special privileges at all.

## ✅ Tier 2 checklist

* [ ] Node runs as a non-root user and `ExecStart=` runs `sonicd` directly
* [ ] Hardening file in place and node healthy for 24h
* [ ] Program-start tests (b) give "Permission denied" for the shell, the data folder and `/tmp`
* [ ] Network test (c) gives "BLOCKED"
* [ ] Port restriction enforced, or you know it isn't (`-BPF_FRAMEWORK`)
* [ ] `IPAddressAllow=` lists only the specific machines (or load balancer subnets) that need access
* [ ] Validators: `SocketBindAllow=` only for 5050 (plus 6060 if used); password file outside `/home` and `/root`
* [ ] Re-run step 2.0 added to your upgrade procedure
