what would a ttrpg that prioritizes roleplay and actually functions as such look like? i've played a few that claim to be "rp forward" and every time the mechanics meant to facilitate roleplay ended up impeding it - and meanwhile i've had perfectly rewarding rp experiences in crunchier systems with no mechanical social encounter support at all. is there really a way to build rp into a system that works, or is it just a unicorn idea?
"Proiritising roleplaying" doesn't mean anything – it's a piece of vacuous marketing text targeted at people who've constructed their identity politics upon arguing about the correct way to pretend to be an elf.
The basic problem is that the term "roleplaying" is, itself, not well defined; in practice, it means whatever the person trying to sell you something wants it to mean. Here, for example, by invoking the presence or absence of "mechanical social encounter support" as the distinguishing feature of self-styled "RP forward" systems, you seem to be implicitly defining "roleplaying" to mean "set-piece encounters in which a player character attempts to persuade an NPC to do something for them without resorting to violence". Is this justified? Is playing out the process of hitting each other with sticks not "roleplaying"? Why not?
What most people mean when they toss the term "roleplaying" around in the context of tabletop games is something in the vicinity of "roleplaying is when we do things I'm interested in doing, and not-roleplaying is when we do things I'm not interested in doing". As all game rules are unavoidably opinionated about what player characters ought to spend their time doing – indeed, arguably this is the only thing that rules can meaningfully express opinions about! – the question of "does this system 'prioritise roleplaying'?" is typically reducible to "does this system agree with me about what kind of game I'm playing?". Games are then sorted into "priorities roleplaying" and "does not prioritise roleplaying" based on which side of the answer to that question they fall on for the person doing the sorting.
This is the ultimate root of a lot of this "the best sessions I ever had never touched the rules at all" stuff. For a variety of reasons, many people have genuinely never experienced playing a tabletop RPG whose rules agree with them about what sort of experience of play they ought to be having, and in some cases they can't even imagine what that would look like. If you and the system you're using disagree so badly about what kind of game you're playing that "engaging with the rules" and "engaging with my desired experience of play" are mutually exclusive activities, it's not surprising that ignoring the rules entirely would be your best play.
In this light, your question of "what would a system that really prioritises roleplaying look like?" translates to "what would a system that actually agrees with me about what kind of game I'm playing look like?", and that's not a question I can answer unless you're willing and able to get a lot more rigorous about what you mean when you say "roleplaying".
While GNS Theory is arguably outdated and messy, I think its "stances" idea may help elucidate things. Edwards describes a few different stances; I'll try and connect a couple of them to this discussion.
Actor Stance is when a player is making decisions as their character. They are acting only on the character's knowledge and motivations. I think this is what a lot of people (from a DnD5e or similarly trad background) are referring to when they say "roleplay" while meaning "outside combat".
Author Stance is when a player is making decisions as the player, then (optionally) justifying those decisions retroactively to fit their character. They are making the most optimal decisions they can within the game system to meet the combat's win condition, and "acting" is secondary.
So when you have a game like DnD5e, where there is a complicated system full of optimal system decisions to make, it behooves a player to drop out of Actor Stance and into Author Stance. Or at the very least, heavily prioritize the latter over the former. When combat starts, "roleplay" generally stops, because players switch Stances to meet the demands of the strategy board game part of the system.
Meanwhile, the vast majority of stuff outside of combat in 5e has very little optimal decision-making to do -- just pick a Skill and roll a die. This low complexity means the player more easily stays in Actor Stance. I think that if 5e approached social situations with same kind of complex, optimization-encouraging methods that it does with combat, you'd see a lot of people complaining that there's no "roleplay" in these situations, because thinking as the character (Actor Stance) becomes secondary to thinking as the player (Author Stance), strategizing out-of-character to ensure better outcomes.
The thing is, players of 5e are doing this all the time in non-combat situations already -- my Intimidation Skill is really high so I'll try that approach -- it's just that the decision-making there is quick and simple enough not to disrupt a player's Actor Stance.
In games that do non-combat complexity well, you as a player will be dipping briefly out-of-character to think about optimal strategies through the lens of the player, just like you would with 5e Skill checks. The trick to keeping the "roleplay" feeling is to be well-versed enough in the system that you can quickly and confidently switch lenses without disrupting that Actor Stance feeling.
So, system mastery is important to make complex social rules "still feel like roleplay" (but a completely different kind of system mastery from what you might expect, if you're only well-versed in trad games), and a game also needs to be designed so that its social scenarios are not slow, technical slogs on the level of 5e combat.
One recent example of a game lending to this really well is the Temeraire RPG, currently in development but available in quickstart. (In a nutshell, Temeraire is a fantasy series set in the early 1800s; it's the Napoleonic Wars, with air forces, with dragons instead of planes.) The quickstart presents three conflict types: Ball, Draconic Feast, and Aerial Battle. The aerial battle system is very clunky and I hope they rework it, but the ball and feast conflicts are excellent. In a ball, human characters choose dance partners (or to not dance), pick types of action from a short list, and pursue goals like embarrassing rivals, securing information, and extracting promises from others (giving them a Duty or Desire, one of the primary mechanics of the game). In a draconic feast, dragons boast and flex to climb a social hierarchy (or challenge the hierarchy's valuation system entirely) for control over the spoils of the feast. There are mechanical steps throughout balls and feasts, but each of those steps is short and flavorful, and always cascades directly into dialogue and "roleplay". Running these a few times, I found that Actor Stances were only really disrupted when we had rules questions, because we were pretty fresh to the system and learning as we played.
