Are there Asian hairstyles that are unseen in canon but would make sense for Fire Nation male characters?
If they're supposed to be law-abiding citizens, then no. The Fire Nation's trademark topknot comes from the Qin Dynasty (221–206 BCE), a very brutal and militaristic period in Chinese history. There are quite a few aspects of the Qin Dynasty that are present in the Fire Nation, for example...
During the Qin Dynasty (221–206 BCE), Shi Huangti banned religion and burned philosophical and religious works. Legalism became the official philosophy of the Qin government and the people were subject to harsh penalties for breaking even minor laws. Shi Huangti outlawed any books, which did not deal with his family line, his dynasty, or Legalism... Confucian scholars hid books as best as they could and people would worship their gods in secret but were no longer allowed to carry amulets or wear religious charms.
In this sort of climate, men are viewed as tools of the state, so self-expression is not encouraged. That said, even under those conditions, men found ways to jazz up their topknot by adding little braid patterns and decorations in the back. They also had unique facial hair styles, although that might have been due to each soldier having a different capacity to grow a 'stache and beard.
However, in both the context of the Qin Dynasty and the Fire Nation, a man without a topknot would indicate he's some sort of criminal or shameful person. In which case, I suppose any hairstyle is acceptable for a law-breaking character.
This is why I'm endlessly fascinated by Lt Jee's lack of top knot
Sir what did you doooo
And did a thirteen year old Zuko with an eye patch read it in your file, and proceed to look at you judgementally for the next three years
