The closest we have right now is probably Resonite. It's the successor to NeosVR; the original dev team managed to claw the rights to the code away from the cryptobro who owned Neos and started their own with the code. It's not open source, but you're able to code and model in-game.
We do need some kinda fediverse VR platform though. The biggest issue is that it'd probably be a lot more demanding than something like Matrix or Lemmy. However, I think it'd fix a lot of issues that platforms like VRC have (one of the biggest user complaints about VRC specifically is that they technically don't allow anything NSFW and have been dragging their heels when it comes to adding an adult-only instance option).
That's unfortunate. I was hoping for an actual Foss VR Chat like game...
Edit: then again, makes sense.
Most VR games require a non FOSS component to them (example: Oculus client, Steam VR) so the people who are really into FOSS are probably not too interested in using some non FOSS blob to even create such an experience. Quite unfortunate.
a while back a metaverse foundation of sorts was formed with the linux foundation and others part taking in it if not mistaken (its 4am) but nothing has come out yet.