Engineering has its share of math, it can get fairly complex (in the case of electrical engineering, it's literally complex), but being engineering it's often based in practical things. But, pure physics has weird-ass math invented just to deal with the messed up calculations required by quantum mechanics.
If you hate weird-ass math, you'll hate pure physics as lot more than any engineering discipline.
Engineering has the kind of math that can be plugged into spreadsheets and CFD simulations. It's the kind of math that might be really complicated, but you can get answers out of it, and those answers can be compared to reality. Physics has the "symbol manipulation" kind of math where you don't even deal with numbers, other than the occasional 2 or 3 when when something is squared or cubed.