As it is with most things, our first interaction with math in large ways influence how we see the subject. Most people’s first interaction with mathematics is in school, where our experience heavily depends on the teacher.
Other factors obviously play a role. If you have a disposition for math it’ll feel easier and more rewarding, making you enjoy the subject more. I also think math is a subject where the really beautiful parts only become pronounced later on, when you already understand the basics.
But even so, most people I have met who actually hate math have also had bad math teachers. I have never encountered a person that hated mathematics without associating it with someone or something else. On the other hand, I don’t necessarily think this is entirely a failure on the part of the teacher. It’s extremely difficult to teach 30-something kids that would rather be anywhere else, keep up with your other duties, get by in an underfunded & mismanaged industry while also trying to make the subject interesting to a bunch of iPad-having children.
All in all, I think this is a failure of properly structuring and funding an education system where teachers are taken care of, children get the chance to grow up while learning and becoming ready for society, and proper resources are provided.