I think that what many people fail to understand about The Wicker Man (the original, not the remake) is that the horror aspect comes not so much from the cult-y pagan stuff itself (of course depending on who you ask), but from the fact that the islanders are collectively gaslighting Howie to lead him to his death. And that they did all this out of desperation, from the failing crops, and it's heavily implied that the sacrifice will not "appease" the gods unless the islanders adapt new agricultural practices, which they definitely won't due to their reliance on the gods, resulting in the eventual destruction of the community. If anything, the horror comes from the islanders taking a life in a meaningless sacrifice.