Input Drops Through KVM Switch

For a long time now, I've had audio drops and sometimes even keyboard input drops through my KVM switches.
When a drop occurs, all audio across all applications drop at once. If there's a video playing then it'll pause briefly. In a game, I just miss a second of audio with no video stuttering.
When I connect my USB speakers directly to my motherboard then the issue does not occur; only through the KVM switch. I was using this decent switch I got off Amazon but just replaced it with one from Level 1 Techs and the issue remained. I've tried different ports on the switches and my board as well.
The issue also only occurs on my Linux computer. When I switch over to my work laptop running Windows I don't have the issue.
I'm not seeing anything in journalctl that seems related after swapping KVM switches. With the old switch I was seeing logs like below but no longer see that with the new switch.

kernel: retire_capture_urb: 1 callbacks suppressed

My PC
CPU: AMD Ryzen 9 7950X
Motherboard: Asus PRIME X670E-PRO WIFI
Memory: G.Skill Ripjaws S5 64 GB (4 x 16 GB) DDR5-5600 CL28
GPU: Gigabyte GAMING OC Radeon RX 6900 XT 16 GB
Kernel: 6.8.1-arch1-1
DS: Wayland 1.22
DE: Gnome 45.0.1
AS: Pipewire
SM: Wireplumber