4d ago

Virtualizing my router - any experience to share? Pros/cons?

I'm thinking about moving my router to be a VM on a server in my homelab. Anyone have any experience to share about this? Any downsides I haven't thought of?

Backstory: My current pfSense router box can't keep up with my new fibre speeds because PPPOE is single threaded on FreeBSD, so as a test, I installed OpenWRT in a VM on a server I have and using VLANs, got it to act as a router for my network. I was able to validate it can keep up with the fibre speeds, so all good there. While shopping for a new routerboard, I was thinking about minimizing power and heat, and it made me realize that maybe I should just keep the router virtualized permanently. The physical server is already on a big UPS, so I could keep it running in a power outage.

I only have 1 gbps fibre and a single GbE port on the server, but I could buff the LAN ports if needed.

Any downsides to keeping your router as a VM over having dedicated hardware for it?

38 comments

Wrapping up this thread, I really appreciate all the opinions and experiences everyone shared! Gave me lots of new perspectives to think about.

I considered something like this at one point, but I ended up installing OpenWRT on my existing router instead because what I ultimately wanted was more flexibility, and was concerned about a single point of failure. Now, I have the ability to do things like always run certain devices through a VPN, block specific devices from the internet with a firewall, as well as DNS for self-hosted stuff.
- Yeah, this might be the way to go. OpenWRT supports hardware NAT with many of these ARM-based routers like many of the MediaTek-based ones, which gives them super high throughput at very low CPU usage. The efficiency blows x86 out of the water. The ability to migrate your OpenWRT config to new hardware (real or virtual) in the future means you kinda get the best of both worlds....

I would advise against it. Separation of concerns isn't important until it is. If your host server is unavailable for any reason, now EVERYTHING is unavailable. Having your server go down is bad. Being unable to browse the internet when your host is down and you're trying to figure out why is worse.
There are also risks involved in running your firewall on the same host as all your other VM's without adding a lot of complex network configurations.
- I appreciate the advice. I have like 3 spare routers I can swap in if the server fails, plus I have internet on my phone lol. It's a home environment, not mission critical. I'm glad you mentioned this though, as it made me realize I should have one of these routers configured and ready-to-go as a backup.
  My logic is partly that I think a VM on an x86 server could potentially be more reliable than some random SBC like a Banana Pi because it'll be running a mainline kernel with common peripherals, plus I can have RAID and ECC, etc (better hardware). I just don't fully buy the "separation of concerns" argument because you can always use that against VMs, and the argument for VMs is cost effectiveness via better utilization of hardware. At home, it can also mean spending money on better hardware instead of redundant hardware (why do I need another Linux box?).
  There are also risks involved in running your firewall on the same host as all your other VM’s
  I don't follow. It's isolated via a dedicated bridge adapter on the host, which is not shared with other VMs. Further, WAN traffic is also isolated by a VLAN, which only the router VM is configured for.

my isp also does pppoe, i have a virtual pfsense, 1gbps up/down, it's never been an issue for me. ive had this setup for maybe three years.
- Do a speed test and run htop... you'll see CPU usage only on one core spiking. Not a big deal if your CPU can handle it, but the AMD GX-412TC in the APU2 I was using is too slow.
  
  I have 2 vcpu (host) for the pfsense vm, xeon e5-2667 3.2ghz, i see both cpu hit about 80% max during speed tests.
  pfsense ce 2.7.2-RELEASE

I did it for a few years, it looks interesting on paper, but in practice, it's a nightmare.
At home, you'll be getting real sick of asking for change windows to reboot your hypervisor.
At work, you will rue the day you convinced mgmt to let it happen, only to now have hypervisor weirdness to troubleshoot on top of chasing down bgp and TCP header issues. If it's a dedicated router, you can at least narrow the scope of possible problems.
- Gotta disagree, for home use at least. I have found it to be the opposite of a nightmare.
  Moving my home routing and firewall to a VM saved me hours, and hours, and hours of time in the long run. I have a pretty complex home network and firewall setup with multiple public IPs, multiple outbound gateways, and multiple inbound and outbound VPN setups for various purposes. I'm also one of those loons that does outbound firewall with deny by default on my network, except the isolated guest VLAN. With a complex setup like that, being in a VM means it's so easy to tweak stuff safely and roll back if you mess something up or it just doesn't work the way you expected. Turns what would be a long outage rebuilding from scratch into a 30 second outage while you roll back the VM. And being able to snapshot your setup for backup is incredibly useful when your software doesn't behave properly (looking at you, PFsense).
  All that said, I run redundant, synced hypervisors which takes care of a lot of the risk. A person who is not well versed in hypervisor management might not be a good fit for this setup, but if you have any kind of experience with VM management (or want to), I think it's the way to go.
  
  For sure, if your thing is leaning into network configs, nothing wrong with it, especially if you have proper failover set up.
  I think virtualized routing looks fun to the learning homelabber, and it is, but it does come with some caveats.
- For home use, if used in an HA setup, the change window issue should disappear. Do you see any other issues that might crop up?
  
  HA... Do you mean failover? It would need some consideration, either a second wan link or accepting that a few TCP sessions might reset after the cutover, even with state sync. But it's definitely doable.
  I'm currently in a state of ramping down my hardware from a 1u dual Xeon to a more appropriate solution on less power-hungry gear, so I'm not as interested in setting up failover if it means adding to my power consumption simply for the uptime. After 25 years in IT, its become clear to me that the solutions we put in place at work come with some downsides like power consumption, noise, complexity and cost that aren't offset by any meaningful advantage.
  All that said, i did run that setup for a few years and it does perform very well. The one advantage of having a router virtualized was being able to revert to a snapshot if an upgrade failed, which is a good case for virtualizing a router on its own.

Ran it for 1.5 years and it stepped away from it. Besides the fact that as soon as your host goes down or you do maintenance on your host, the network becomes kind of useless (ESP if you have multiple segmentated nets). The other thing to keep in mind is to pass through physical nics. Using just the vnics will potentially lead to security risks. That's the reason I went back to physical fws.
- The other thing to keep in mind is to pass through physical nics. Using just the vnics will potentially lead to security risks. That’s the reason I went back to physical fws.
  I could throw an extra NIC in the server and pass it through, but what are the security risks of using the virtualized NICs? I'm just using virtio to share a dedicated bridge adapter with the router VM.

I've been doing it for probably 8 years now without any major issues related to being a VM. In fact, that made recovery extremely easy the two times my PFsense VM shot itself in the head. Just load the backup of the VM taken the day before and off to the races. After switching to OPNsense a couple years ago I haven't had a single issue.
These days I run two identically spec'd hypervisors that constantly sync all my VMs to each other over 10GB NICs, so even a hardware failure won't take out my routing. That is something to consider if you don't have redundant hypervisors. Not really any different than if your physical router died, just something to plan for.

I run OPNsense on a 2 node proxmox server and have for a few years now. I have HA set up and have had it fail over gracefully when I've been away and not even noticed it having failed over for more than a week. If I want to upgrade it, I snapshot it, and if I upgrade the host I live migrate it, and I've done this all remotely more than a few times with no issues.
It takes some planning and I'd say you'd want a cluster (at least a pair of nodes) where you can do HA. But I wouldn't do it any other way at this point. If you have only one port, you can VLAN it for using on both LAN and WAN.
- That is pretty sweet. I have a second server I could use for an HA configuration of the router VM. I've been meaning to play around with live migrations (KVM) so this could be a cool use case for testing.
  
  It works well. I have my docker hosts on HA as well because they're almost as important as the router.
  If you just use 2 nodes, you will need a q-device to make quorum if you have one of the nodes down. I have the tiebreaker running on my Proxmox Backup Server shitbox I3.
  Proxmox is basically just debian with KVM and a better virt-manager. And it deals with ZFS natively so you can build zpools, which is pretty much necessary if you want snapshotting and replication, which are necessary for HA.

I run opnsense as a VM and have done for maybe 5 years now, moved across 3 different sets of hardware.
I DO have a hardware router under the ONT for if / when I feck up proxmox.
Snapshots are great when you start to play with the firewall settings or upgrades

I used the same approach at the family business for years without any major problems. Go for it.

I personally wouldn't do this. You want your network to be dedicated hardware

It works great as long as you have a method to access the server directly when the router machine is down. A laptop set to a static IP on the same subnet will let you access the host when you b0rk something. Keep a backup config on that machine It's pretty great though. Just remember pfsense won't support more than 7 external interfaces when you start getting crazy with vlans
- Even if the virtualized router is down, I'll still have access to the physical server over the network until the DHCP lease expires. The switch does the work of delivering my packets on the LAN, not the router.
  Thanks for the tip about the pfSense limit. After running pfSense for like 8 years, my opinion is that is flush with features but overall, it's trash. Nobody, not even Netgate, understands how to configure limiters, queues, and QoS properly. The official documentation and all the guides on the internet are all contradictory and wrong. I did loads of testing and it worked somewhat, but never as well as it should have on paper (ie. I got ping spikes if I ran a bandwidth test simultaneously, which shouldn't happen.) I don't necessarily think OpenWRT is any better, but I know the Linux kernel has multithreaded PPPOE and I expect some modern basics like SQM to work properly in it.
  
  Even if the virtualized router is down, I'll still have access to the physical server over the network until the DHCP lease expires. The switch does the work of delivering my packets on the LAN, not the router.
  Yes, of course it depends on your network topology. If you have a link in the same subnet you're good (and can configure a static IP if need be). But if you're using vlans you can get in a pickle if the router is down. In my setup everything on the user side is segregated so if the router goes down I have to take a dedicated management laptop and plug into the host management network directly on the management switch where i keep a port empty. This maintains segregation and in practices means I take my ancient Acer Aspire One used for nothing else into the server room that looks strangely like a laundry room and plug it in.

Wow planning to do the same thing. Amazing find of a post.
And we have opinions for and against! Wow 🍿
Good luck everyone

38 comments