Looking for advice on prioritizing traffic in my setup

My current setup is in two rooms. These rooms are connected via a TL-SG108E switch in one room (Room A) and a SG300-10 switch in the other (Room B). I have two WANs, one in each room. The primary WAN (fiber) is in Room A. The pfSense is in Room B. I have two VLANs set up, one for WAN and one for LAN. There are LAN devices in both rooms. The traffic is tagged so that the WAN traffic goes only to the pfSense and any LAN traffic is then separated as well, but using the same connection between the switches. All connections are 1 Gbps except for a few individual devices that don't support it that are irrelevant to this issue.

I have an OpenVPN interface set up in pfSense over WAN B. A device in Room A uses this connection exclusively via pfSense rules that force traffic from that device IP to use the VPN gateway and blocks when the gateway is down. Even though I've limited the bandwidth to 10 Mbps and connections to 1000, and WAN A is 1 Gbps and WAN B is 450 Mbps, whenever the VPN connection is being used it negatively affects my normal WAN A traffic.

I don't know for sure but I'm guessing this is some kind of issue with the connection between the two switches. Are there any settings or configuration changes I can make to alleviate the strain or prioritize my normal LAN traffic to make the VPN secondary? Would another physical setup work better? I was considering potentially moving the pfSense box to Room A, but my primary device is in Room B as well. If there's any more information that might be useful please ask.