Since the "fanfiction vs literature" discourse is making the rounds again, here we go:
All literature is intertextual--that is, it refers back to other literature ("all writing is rewriting," etc). This is a basic feature of language itself, if you want to get all postmodernist-theory about it.
Some works of literature, though, are highly intertextual to a specific pre-existing work, that the audience is expected to recognize and be familiar with. Kristina Busse calls this "intense intertextuality." It has also been called "recursive literature," the term I prefer.
Famous examples include Rosencrantz and Guildenstern are Dead, Ulysses, March, Une Tempête, Wicked, Wide Sargasso Sea, etc etc etc.
"Using recognized pre-existing stories/characters to tell a new story"--the entire basis of the category "recursive literature"--is as old as literature itself. In the premodern West, reworking a known story, and succeeding, was often considered to be the pre-eminent form of artistic achievement. Milton and Dante were absolutely operating under that assumption.
There are a number of artistic moves, such as allusion and irony, that take on particular weight within recursive lit. The fact that the audience is expected to know the source text enables the writer to play with that knowledge in highly specific ways: this is a central aesthetic signifier of recursive literature as a category.
The cultural shift toward prizing "originality" in fiction ramps up during the late 18th-early 19th century Romantic movement; not at all coincidentally, this is also when the concept of modern copyright starts developing.
Recursive lit thus, in this period, gets tangled up with both artistic and legal/financial discourses of originality and intellectual property, and we have inherited this. While the basic idea of recursive lit remains the same as it ever was, these new financial/legal dimensions affect how recursive lit can be distributed. (The best book on this history is David Brewer's The Afterlife of Character.)
Ever since, if you want to commercially publish recursive lit, you can only do so if the text you are referencing is out of copyright (Homer, Shakespeare, Jane Austen, the Brontes, etc...).
As technology improves and democratizes, informal, unofficial, non-commercial distribution networks are created among various communities--like fandoms. Now, recursive narratives for in-copyright texts can circulate freely, outside of the oversight/control of commercial publishing.
The basic artistic move of fanfiction--using pre-existing texts (especially characters) as a springboard for a new story--is exactly the same as every other kind of recursive lit. The only real quantifiable difference is how it is distributed.
This, then is the best definition of fanfic: Fanfiction is a sub-type of recursive literature that, whether out of preference or necessity, circulates informally and non-commercially.
Because fanfiction has a much lower barrier to entry than commercial publishing, and is not bound by the demands of what publishers think they can sell, fanfiction displays enormous variability in form, content, and quality (lol). While there are fanfics that could be published as-is, there are far more that are completely entrenched within specific community discourses; we've all read fanfic that only makes sense to the 5 weirdos with that particular headcanon. This huge variability is a direct result of the unofficial distribution method.
When it comes to published recursive literature, whether premodern (Inferno) or modern (Wicked), I don't like using the term "fanfiction," because it's inaccurate! Neither of those works was distributed through informal non-commercial channels! Fanfiction is, again, a SUB-TYPE of the larger artistic category "recursive lit."
In conclusion: The Other Bennet Sister is recursive literature. Because it was recursive to an out-of-copyright text, of novel length, and written capably enough to suggest commercial appeal, Janice Hadlow was able to formally publish it for money. Therefore, even though it is doing the same artistic thing fanfic does (because fanfic is a type of recursive lit), it is not fanfiction. Now, if she had slapped it up on AO3, without changing a word? THEN it would be fanfiction.