neru has modest hardware specs but I'm not planning to use her as a router. She's a small, low-power appliance that can run containers and takes a USB stick for extra storage. neru is a hexS 2025 model: * Homepage: https://mikrotik.com/product/hex_s_2025 * Product code: E60iUGS <> = Features and uses = * 5x 1G RJ45 ports * 1x 2.5G SFP port * PoE powered * arm CPU, meaning we can run containers = Hardware = * CPU: EN7562CT * Switch chip: EN7523 (https://help.mikrotik.com/docs/spaces/ROS/pages/15302988/Switch+Chip+Features) A surprisingly decent switch chip for such a cheap device, it does hardware STP and VLAN filtering sensibly. = Config = Notes from when I set her up as a PXE server, running TFTP (native) and HTTP (nginx in a container), serving files from the USB stick. Served up here: http://neru.thighhighs.top:8080/ == Config dump == Taken from a `/export` on the CLI. It might look a bit unusual because there's no default bridge, we're only using ether1 as the WAN port, and NATing traffic to/from the container NIC running on there. {{{ # This is for the container NICs (veth) to attach to, and it works just like a normal LAN segment. # The router is the gateway and has the .1 address on the bridge, and the clients on the segment # get .2, .3, .4 etc as usual. You masquerade-NAT traffic from the containers to the outside # world, and DNAT any inbound connections that need to go to a container listening port. # # Mikrotik/ROS doesn't have host-mode networking for containers, but you can emulate it by giving # the veth interface an address on the "public" network, basically a sibling to the router's # outbound interface. Join both to a bridge (be careful with the STP settings), and now you have # a container directly on the network, no NAT needed. The only caveat is that you need an IP for # each container, rather than them all sharing the host's IP address. # # In this case, I'm happy enough with NAT and it works just fine. /interface bridge add name=containers priority=0xd000 protocol-mode=mstp vlan-filtering=yes # Virtual NIC that the container will own. We give it an IP address that we'll NAT from/to /interface veth add address=172.25.0.2/24 container-mac-address=2E:9E:C1:16:16:92 dhcp=no gateway=172.25.0.1 gateway6="" mac-address=2E:9E:C1:16:16:91 name=nginx # Add the container veth to the bridge /interface bridge port add bridge=containers interface=nginx # These VLANs live directly on ether1 because it's all that matters, we're not using the default bridge. /interface vlan add interface=ether1 name=VLAN10_GENERAL vlan-id=10 add interface=ether1 name=VLAN11_MGMT vlan-id=11 # We don't really need interface lists, but it's kinda handy in case we need them later. # For example we might have a bunch of containers and want them to talk to each other, # but not necessarily with the outside world. /interface list add comment=defconf name=WAN add comment=defconf name=LAN /interface list member add interface=ether1 list=WAN add interface=VLAN11_MGMT list=LAN add interface=VLAN10_GENERAL list=LAN /user group add name=mikrodash policy=read,test,api,!local,!telnet,!ssh,!ftp,!reboot,!write,!policy,!winbox,!password,!web,!sniff,!sensitive,!romon,!rest-api /container add domain-name=thighhighs.top hostname=neru interface=nginx layer-dir="" logging=yes memory-high=128.0MiB mount=\ /usb1p1/srv:/usr/share/nginx/html:ro,/usb1p1/nginx-conf.d:/etc/nginx/conf.d:ro name=nginx remote-image=arm32v5/nginx:trixie root-dir=\ /usb1p1/container-roots/nginx start-on-boot=yes /container config set memory-high=256.0MiB registry-url=https://registry-1.docker.io tmpdir=/usb1p1/tmp /disk add mount-point-template=usb1p1 parent=usb1 partition-number=1 partition-offset=16384 partition-size=61524131840 type=partition /disk settings set auto-media-interface=*8 /ip neighbor discovery-settings set discover-interface-list=all lldp-mac-phy-config=yes lldp-max-frame-size=yes lldp-vlan-info=yes # Note that the VLAN 10 interface gets its address via DHCP, only the mgmt address is static. # We need to make sure our NAT rules handle this correctly, don't hard-code addresses. /ip address add address=192.168.11.9/24 interface=VLAN11_MGMT network=192.168.11.0 add address=172.25.0.1/24 interface=containers network=172.25.0.0 /ip dhcp-client add interface=VLAN10_GENERAL name=client1 # Can be queried as a nameserver, probably to serve the containers (not that they would need to, I think) /ip dns set allow-remote-requests=yes # This is really standard stuff, with tight rules to only accept exactly what we want. # The INPUT chain is boring, the FORWARD chain is where the fun happens (just a little). # # There's an implicit Accept at the end of chain. That means that if a connection arrives and: # - it's not destined for the local host (this router's mgmt or general IP address) # - that means it would need to be forwarded # - but it's been dstnat'ed already # then it's okay to forward it along. Otherwise it's getting dropped. That means we won't # route traffic between the mgmt and general networks, but we will forward traffic through # to the container subnet. # # Note that if you went with host-style networking, and the veth has an IP address on the # public network, I'm not sure if that counts as hitting the Input or Forward chain. /ip firewall filter add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp add action=accept chain=input comment="Mikrotik Winbox" dst-port=8291 in-interface=VLAN11_MGMT protocol=tcp add action=accept chain=input comment=SSH dst-port=22 in-interface-list=LAN protocol=tcp add action=accept chain=input comment="TFTP server" dst-port=69 in-interface-list=LAN log-prefix="udp/69 TFTP" protocol=udp add action=accept chain=input comment="SNMP server" dst-port=161 in-interface-list=LAN protocol=udp add action=accept chain=input comment="tcp/8728 Mikrotik api" dst-port=8728 in-interface=VLAN11_MGMT protocol=tcp add action=accept chain=input comment="tcp/8729 Mikrotik api-ssl" dst-port=8729 in-interface=VLAN11_MGMT protocol=tcp add action=accept chain=input comment="udp/5678 Neighbour Discovery" dst-address=255.255.255.255 dst-port=5678 log-prefix="udp/5678 Neighbour Discovery" protocol=udp add action=drop chain=input comment="Drop everything else" add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid add action=accept chain=forward comment="Allow containers outbound DNS (udp)" connection-state=new dst-port=53 in-interface=containers out-interface=VLAN10_GENERAL protocol=udp add action=accept chain=forward comment="Allow containers outbound DNS (tcp)" connection-state=new dst-port=53 in-interface=containers out-interface=VLAN10_GENERAL protocol=tcp add action=accept chain=forward comment="Allow containers outbound HTTP" connection-state=new dst-port=80 in-interface=containers out-interface=VLAN10_GENERAL protocol=tcp add action=accept chain=forward comment="Allow containers outbound HTTPS" connection-state=new dst-port=443 in-interface=containers out-interface=VLAN10_GENERAL protocol=tcp add action=drop chain=forward comment="Drop anything not DSTNATed, nginx implicit accepted" connection-nat-state=!dstnat connection-state=new log=yes log-prefix="dropped forward" # Heaps simple: masquerade (SNAT) traffic from containers to the outside network # (which is kinda incorrectly called "LAN" here), and DNAT inbound connections if # they match a listening container. /ip firewall nat add action=masquerade chain=srcnat in-interface=containers out-interface=VLAN10_GENERAL add action=dst-nat chain=dstnat dst-port=8080 in-interface-list=LAN protocol=tcp to-addresses=172.25.0.2 to-ports=80 # The TFTP server is a bit fiddly on Mikrotik, and people report not being able # to get it work, despite being absolutely trivial to configure. # # This config maps all filenames directly through to `/usb1p1/srv/` on the USB stick. I've made it # writeable because that's useful sometimes, and I trust the clients enough on my network. # # Note that the TFTP server can only handle files up to 64MiB in size, it'll error out if # the transfer gets that far. That's why we need to run an HTTP server in the first place, # because initrd files are a couple hundred MiB. Also it's waaaaaay faster, you'll # appreciate the time savings. /ip tftp add read-only=no real-filename=usb1p1/srv/ req-filename=.* /ip tftp settings set max-block-size=8192 # This is mostly stock standard defaults, I don't really run IPv6 on my network. /ipv6 address add address=fdfd:2501:2762:11::7 interface=VLAN11_MGMT /ipv6 firewall address-list add address=::/128 comment="defconf: unspecified address" list=bad_ipv6 add address=::1/128 comment="defconf: lo" list=bad_ipv6 add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6 add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6 add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6 add address=100::/64 comment="defconf: discard only " list=bad_ipv6 add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6 add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6 add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6 # I should clean these up but I'm lazy. /ipv6 firewall filter add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6 add action=accept chain=input comment="defconf: accept UDP traceroute" dst-port=33434-33534 protocol=udp add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10 add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN add action=fasttrack-connection chain=forward comment="defconf: fasttrack6" connection-state=established,related add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6 add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6 add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6 add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6 add action=accept chain=forward comment="defconf: accept HIP" protocol=139 add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN # Basic system stuff /snmp set contact=Furinkan enabled=yes location="Conferta Ave" /system clock set time-zone-name=Australia/Sydney /system identity set name=neru /system ntp client set enabled=yes /system ntp client servers add address=pool.ntp.org add address=au.pool.ntp.org # This is nice, means it doesn't beep after booting up every time. It's fine, but I want to do # frequent RouterOS updates without annoying the family, and the beep is LOUD. /system routerboard settings set silent-boot=yes /tool mac-server set allowed-interface-list=LAN /tool mac-server mac-winbox set allowed-interface-list=LAN }}} = Original setup exploration = I have aris, she's an original hex S with microSD card slot and USB. She has an mmips CPU, which means no container support :( But I have neru, she's a hex S 2025 model, no microSD but with USB. I'd prefer microSD because I have spares, but beggars can't be choosers. I really don't want stuff sticking out, so it'll have to be a micro-profile USB stick. I've already messed with this before, I've enabled container mode (https://help.mikrotik.com/docs/spaces/ROS/pages/93749258/Device-mode#Devicemode-Enablingdevice-modefeature) and installed the container package. pop in the usb stick (Sandisk Ultra Fit 64gb USB 3.2 gen1 5Gbps), it already has a partition and FS. Set the mountpoint and format it. {{{ /disk/set usb1-part1 mount-point-template=usb1p1 /disk/settings/set auto-media-sharing=no auto-smb-sharing=no /disk/format usb1-part1 file-system=ext4 label=neru-data }}} Follow the network setup outline, I've picked an IP range that shouldn't conflict with any of my stuff. The suggested docker network settings are fine in most cases though: https://help.mikrotik.com/docs/spaces/ROS/pages/84901929/Container#Container-RunningPi-hole {{{ /interface/bridge/add name=containers priority=0xD000 protocol-mode=mstp vlan-filtering=yes /ip/address/add address=172.25.0.1/24 interface=containers /interface/veth/add name=netbooting address=172.25.0.2/24 gateway=172.25.0.1 /interface/bridge/port/add bridge=containers interface=netbooting /ip/firewall/nat/add chain=srcnat action=masquerade src-address=172.25.0.0/24 }}} Setup container stuff {{{ /file/add type=directory name=usb1p1/containers /file/add type=directory name=usb1p1/srv /file/add type=directory name=usb1p1/tmp /container/config/set tmpdir=usb1p1/tmp /container/config/set registry-url=https://registry-1.docker.io /container/config/set memory-high=256MiB /container/add auto-restart-interval=1m domain-name=thighhighs.top hostname=neru interface=netbooting logging=yes memory-high=128.0MiB mount=/usb1p1/srv:/var/www:ro name=caddy remote-image=library/caddy root-dir=/usb1p1/container-roots/caddy start-on-boot=yes workdir=/srv }}} The container dies immediately after start, log shows "exited with signal 4", which is a SIGILL. I'm almost certain this is an architecture limitation. Sure enough... {{{ > For devices with EN7562CT CPU like the hEX Refresh, only arm32v5 container images are supported }}} I bet it's more than just that. I need a caddy image that supports v5 specifically. It's probably pulled a v7 or v8 image. Okay caddy is too much fucking work. I'd have to build it myself on a different machine, and the Dockerfile for it only uses alpine, which isn't supported on arm32v5. I'll just use nginx, I only need static fileserving anyway so it's all moot. https://hub.docker.com/r/arm32v5/nginx {{{ /container/add name=nginx remote-image=arm32v5/nginx:trixie check-certificate=yes interface=netbooting mount=/usb1p1/srv:/usr/share/nginx/html:ro hostname=neru domain-name=thighhighs.top memory-high=128.0MiB root-dir=/usb1p1/container-roots/nginx start-on-boot=yes logging=yes /container/start nginx /ip/firewall/nat/add chain=dstnat dst-address=192.168.1.25 dst-port=8080 protocol=tcp action=dst-nat to-addresses=172.25.0.2 to-ports=80 }}} Configure it: {{{ root@abc3def4c047:/etc/nginx/conf.d# cat default.conf server { listen 80; server_name localhost; #access_log /var/log/nginx/host.access.log main; location / { root /usr/share/nginx/html; index index.html index.htm; } #error_page 404 /404.html; # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } # proxy the PHP scripts to Apache listening on 127.0.0.1:80 # #location ~ \.php$ { # proxy_pass http://127.0.0.1; #} # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 # #location ~ \.php$ { # root html; # fastcgi_pass 127.0.0.1:9000; # fastcgi_index index.php; # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name; # include fastcgi_params; #} # deny access to .htaccess files, if Apache's document root # concurs with nginx's one # #location ~ /\.ht { # deny all; #} } }}}