In Hinduism, the right-facing symbol (clockwise) (卐) is called swastika, symbolizing surya ('sun'), prosperity and good luck, while the left-facing symbol (counter-clockwise) (卍) is called sauvastika, symbolising night or tantric aspects of Kali.
I thought the big difference came down to the direction it was facing, but turns out both directions are the Hindu way with different meaning. And depending which side of the cake or the slice you look at it will be both ways. The other distinction between the two is that the Nazi one is usually tilted to 45 degrees.