So like, "associative mating" is just another word for "trophy or reward"; you are in some way better than other people and therefore deserve to get the girl for being attractive. If you let yourself go after your high school jock years, well, maybe you don't deserve that hot wife anymore.
There are some components of physical attractiveness that don't require effort but a lot of them do.
Look, in media fantasies of romance have two psychological components, which are "That person loves me because of all the effort I put in" and "That person loves me even though I don't deserve it" and you'll get both of those in various mixes catering to both genders.
How many romance books aimed at women involve a protagonist who thinks of herself as mousy and unremarkable but is swept off her feet by a handsome, charismatic rich guy with a troubled past? Because as an outsider it seems like a lot of them.
Any romantic fantasy that leans heavily into "They love me even though I don't deserve it" is based on the expectation that the audience is projecting themselves into the protagonist, and viewing the love interest as a fantasy figure, and so of course if you instead project yourself onto the fantasy love interest you're going to go "Why does (s)he like that schlub?"
Well, because the fantasy is "I might end up in a world of romance and excitement even though I'm a schlub."
Meanwhile, maybe this changed but like... Nerds used to really love Conan and shit like that. The enormous body-builder who gets women because he's impossibly tough and charismatic is a constant figure in science fiction and fantasy, I would say just as much as the nerd who gets the girl through cunning and brains over brawn. These are just two sides of the "they love me because I deserve it" coin.
Like, personally I think the real difference is merely just general quality, if you are a better writer the protagonist is likeable enough that people of both sexes will root for them.
For example, like, Pierce Brosnan and Daniel Craig were both pretty sexy as James Bond, but I've seen plenty of knock-off spy movies from the 60s where men who were more or less as fit and handsome as Sean Connery go through the same motions of effortlessly dispatching bad guys with judo skill and bedding sexy foreign spies and just come off as the least sexy men imaginable.