I apologize if this comes off as rude at all (I genuinely don't mean it that way; I'm just trying to organize my thoughts), but for me, the more I learn about magic from experienced practitioners like yourself, the less invested I am in pursuing it.
To put it in phrasing I've heard used elsewhere on Tumblr, if sufficiently advanced magic is indistinguishable from chores, then I'm not sure that's a burden I'm willing to take on. I struggle enough trying to find the motivation to do my mundane chores; having an extra set of spiritual ones on top of that workload just doesn't sound... well, for lack of better phrasing, "worth it" to me.
Granted, I also think a big part of my feelings on this topic come from my current circumstances. Without getting too far into it, I've become kinda spiritually stagnant as of late, and whatever energies are around me right now don't particularly care for that. (I don't really look (or ask) for signs anymore, but the few I have seen all indicate a need for change/growth in my practice.)
But from my perspective, stepping away from witchcraft feels like finally finding solid ground again after years of trudging through a flooded swamp. Getting any more invested in a magical practice just feels... wrong.
Is it such a bad thing to want to "refuse the call," as it were, and choose to disengage with practicing magic altogether rather than pursue it?
The most important part first: no, it's not bad to refuse the call to magic. That's not bad at all. This old post may be of use to you. If witchcraft is something that you can sow into your life like an annual wildflower, and it's not currently/no longer serving you, then why stick with it? What would be the point of that?
(Of this, another post may be helpful for you: growth in witchcraft as shedding of snake-skin.)
"Sufficiently advanced witchcraft is chores" is a bite of a sound bite to be sure, and so I have made this longer post for you stepping it out.
Besides that long post, my other answer to your question is:
I don't think that investing in a magical path is supposed to increase your net difficulty/labor.
This was something that really got me early on. I'd hear practitioners saying how much work and struggle it all is. To which I'd say, sure - but then you have MAGICAL POWERS. You do all this so you can have magical powers which presumably you can use to make your life better.
So can't you then just use your magical powers to untangle, de-escalate, resolve, soothe, improve, and elevate your life in other ways, thus ultimately making your life much better than it was before you got into magic?
Magic is not a zero sum game, for me. I'm not in this for veneration or worship or earth spirituality. I'm in it because I want the utility. I'm here because I want the joy of deep connection with my friends when I'm tranced out and giving readings that zap their socks off. I'm here because I want to see what's behind people's eyes. I want to see how the conversation can go and know the exact words to make it go that way. I want victory, I want success. I want catastrophe and grace, not measured and mixed by a higher power and bestowed upon me but meted by me and sent outwards, downwards, forwards.
I can have all of this. But I myself can only have it one way.
I can only have it if I treat my practice as something so mundane and common that it's likened unto chores, that I put altar tending on my chore chart next to cleaning the bathroom, Saturdays are laundry and warding and wealth. Laundry is an endless chore only if you want endless clean clothes; warding is an endless chore only if you want endless protection.
It's no choice for me any more. I half run on magic now. And so I have to turn the spigot on and water myself first, or else I can never get the mundane chores finished anyway. I am not unique in this, although this is not the path of every Witch.
So if you are saying: this doesn't seem right for me right now, or maybe never again, and stepping away brings relief - then there is no argument from me, I would encourage you to find your solid ground.
But if you are saying, I see all these practitioners who seem to like the chore label, and isn't that overwhelming? isn't it a drag - then, no: for me it is energizing, liberating, fulfilling, actualizing.
And if your life is a big swamp, and you choose to sort it out with magic (there are other ways), then at first it may feel very laborious with very little reward, like the first year of building a new garden. But the returns begin to come, as long as you tend the garden. That's the trade, if you're in it for the harvest.