Plato did talk about other universes, but it was more like planes of existence. Sort of like the concepts of heaven and hell in a way. A key element, to me, of the multiverse is that it's variant timelines, not just another realm.
There was a novel in the early 30s called "Sideways in Time" that talked about not just going forward and backward in time, but also sideways.
You could make an argument that the early 20s HG Wells story "Men Like Gods" did it earlier. It has a branch in time with the two paths existing simultaneously.
I still think, when people today talk about the multiverse, they usually mean not just parallel universes, but parallel timelines. Like in the MCU where there are all sorts of different versions of earth, but most are pretty recognizable. I realize the definition doesn't depend on that, but it seems to be the common usage.