Selfhosted @lemmy.world lemmyvore @feddit.nl 8 mo. ago

How do you guys use Tailscale (or other VPN) with containers

I wanted to run my VPN/Tailscale setup past you, see if anybody has suggestions on how I could do things better.

Setup: home LAN (10.0.0.0/24), router+DNS on 10.0.0.1, server running docker containers on 10.0.0.2.
LAN DNS points *.local.dom.tld to the server, public DNS points *.dom.tld to my dynamic public IP.
Containers run in bridge mode with host, expose ports on host IPs via "ports:" mapping.
NPM with LE certs also in container, exposes 10.0.0.2:443, forwards to various other services.

Goals for Tailscale:

Accessing HTTP services via NPM from my phone when away from home.
Exposing select UDP and TCP non-HTTP services such as syncthing (:22000) or deluge RCP admin (:58846) to other tailnet devices or to phone on the go.

Goals in general:

Some containers need to expose ports on the LAN.
Some containers need to expose ports via Tailscale.
Some containers need to broadcast on the LAN (DLNA stuff) – but I don't want them broadcasting to Tailscale.
Generally speaking I'd like to explicitly control what's exposed from each container on either LAN or Tailscale.
I'd like to avoid hacking images with Dockerfile. I can make my own images to do stuff, just don't want to keep up with hacking other images.

How I progresed with Tailscale:

First tried running it directly on the host. Good: tailnet IP (let's call it 100.64.0.2) available on the host's default network stack. Containers can use "ports:" to map to 100.64.0.2 (tailscale) and/or 10.0.0.2 (LAN). Bad: tailscale would mess with /etc/resolv.conf on host. Also bad: tailscale0 on host picked up stuff that binds to 0.0.0.0.
Moved tailscale to a container running on the host network stack (network_mode: host). Made it leave /etc/resolv.conf alone. tailscale0 on host stack still picks up everything on 0.0.0.0.

This is kinda where I'm stuck. I can make the tailscale container bridged which would put the tailscale0 interface inside the container. It wouldn't pick up 0.0.0.0 from host but how would I publish ports to it?

The tailscale recommended way of doing it is by putting other containers in the tailscale's container network stack (network_mode: container:tailscale). This would prevent said containers from using "ports:" to map to host anymore. Also, everything they publish locally would end up on tailscale0 whether I like it or not.
Tailscale has an env var TS_DEST_IP that can mirror another IP. I could allocate an IP on host eth0 like 10.1.1.1, mirror that from the tailscale container, and target it from other containers explicitly with "ports:" when I want to publish a port to tailscale. Downside: 10.1.1.1 would be in the host's network stack so still picks up 0.0.0.0.
I could bridge the tailscale container with other containers on a private subnet, say 192.168.1.0/24 and use tailscale serve to forward specific ports to other containers over that subnet. Unfortunately serve is fairly limited; it can't do UDP and technically it refuses to forward TCP either to non-localhost (but you can dump the serve config to JSON, and hack that config, and use it with TS_SERVE_CONFIG= 🤮).
I could bridge tailscale with other containers and create a special container with a fixed IP on that subnet, mirror the IP from tailscale, and use iptables on that container to forward specific ports to other containers. This would actually solve everything I want except...
If I ever want to use another VPN which doesn't have the mirror feature. I don't know how I'd deal with that.

7 comments

This doesn't exactly match your goals, but you may be able to adapt it or take pieces from it.

I have containers running on two subnets:

LAN + Tailscale

LAN only

Subnet 1 has a DNS server, which resolves all of my services to IPs on either subnet.

I have Tailscale set up on a machine as a subnet router (directing to Subnet 1).

Result:

When local, I can access all services on the LAN with local DNS entries, both Subnet 1 and 2.

When remote via Tailscale, I can access all services on Subnet 1 with the same local DNS entries. I cannot access services on Subnet 2.

This is nice because my apps don't care which network I'm on, they just use the same URL to connect. And the sensitive stuff (usually management tools) are not accessible remotely.

It's also ridiculously simple: Only one Tailscale service is running at home.

This does not solve your issue of broadcasting vs not broadcasting, though. There's probably other things missing as well. But maybe it's a start?
I too am playing with Tailscale at the moment. What's working for me (although I believe you're well past this) is just running an exit node into my network. I have Adguard and NPM forwarding and reverse proxying and as long as I don't use .local it seems that the nameserver works on Tailscale too, although I do get some errors in my testing at work yesterday.

I wonder if you could set up an Openwrt container or VM and add Tailscale to it. That way you could port forward all the ports you want to Tailscale
- I tried on OpenWRT but the tailscaled binary is huge and barely fits my router's resources. I can sort of squeeze it in if I compact the daemon binary and run the client binary temporarily only when I change config and then delete it (the daemon doesn't need it to run).
  
  A tiny VM might be an interesting idea though. Would also be able to simulate some scenarios in case I ever need to get a VPS.
Sharing the network space with another container is the way to go IMHO. I use podman and just run the main application in one container, and then another VPN-enabling container in the same pod, which is essentially what you're achieving with with the network_mode: container:foo directive.

Ideally, exposing ports on the host node is not part of your design, so don't have any --port directives at all. Your host should allow routing to the hosted containers and, thus, their exposed ports. If you run your workloads in a dedicated network, like 10.0.1.0/24, then those addresses assigned to your containers need to be addressable. Then you just reach all of their exposed ports directly. Ultimately, you then want to control port exposure through services like firewalld, but that can usually be delayed. Just remember that port forwarding is not a security mechanism, it's a convenience mechanism.

If you want DLNA, forget about running that workload in a "proper" container. For DLNA, you need the ability to open random UDP ports for communication with consuming devices on the LAN. This will always require host networking.

Your DLNA-enabled workloads, like Plex, or Jellyfin, need a host networking container. Your services that require internet privacy, like qBittorrent, need their own, dedicated pod, on a dedicated network, with another container that controls their networking plane to redirect communication to the VPN. Ideally, all your manual configuration then ends up with a directive in the Wireguard config like:

PostUp = ip route add 192.168.1.0/24 via 192.168.19.1 dev eth0

Wireguard will likely, by default, route all traffic through the wg0 device. You just then tell it that the LAN CIDR is reachable through eth0 directly. This enables your communication path to the VPN-secured container after the VPN is up.
https://tailscale.com/blog/docker-tailscale-guide

I used the above link to start on my project. I’m trying to add Tailscale service to my existing docker-compose files to forward all traffic on the primary container to an exit node. It works, but I’m not able to find a way to access the web apps on those containers that are forwarding their traffic. Looks like you are well beyond this guide.

Best of luck.
- That guide did help me find out about TS_ env vars, which I don't think are well documented elsewhere. From what I understand they're container specific? I think they're set up by containerboot, which is what the tailscale container image uses to boot.
  
  TS_DEST_IP in particular is a game-changer. Docker needs more options for forwarding ports and interfaces. 🙁 "ports:" only forwards to the host and that's about it.
  
  Best of luck.
  
  You too, what you're trying to do looks like a challenge as well.
I just posted how I handled this after spending almost a day or two trying to work it out. I'm on phone right now so I will post it when I get on my pc or you can check it from my history

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters	More Letters
DNS	Domain Name Service/System
IP	Internet Protocol
Plex	Brand of media server package
UDP	User Datagram Protocol, for real-time communications
VPN	Virtual Private Network
VPS	Virtual Private Server (opposed to shared hosting)

6 acronyms in this thread; the most compressed thread commented on today has 14 acronyms.

[Thread #726 for this sub, first seen 30th Apr 2024, 14:05] [FAQ] [Full list] [Contact] [Source code]

You've viewed 7 comments.