How old do you think Selwyn is? Based on the other parent/child ages in asoiaf I would suggest he is in his early to midfourties. Which somehow leads me always to how expanded the Tarth Family really is, since Stannis' talked about "The Tarths" in acok, has he brothers or sisters, so that Brienne could have cousins go inherit, so that Selwyn didn't bother, after her failed engagements.
We actually know Selwyn's exact age. He's 54:
A flush crept up her [Brienne's] neck. "My father's only four-and-fifty. Not too old to wed again and get a son by his new wife." (AFFC, Brienne VIII).
A Feast for Crows is set in 300 AC, which gives Selwyn a birth date of 245 AC or 246 AC, depending on what time of year he was born.
That being said, I totally agree with you in thinking he has relatives/siblings. There is at least one other Tarth we know of in text: Ser Endrew Tarth, a man of the Night's Watch. In fic, Endrew is often portrayed as Selwyn's brother, and I think this makes sense. There are some hints that he might be a similar age to Selwyn. By the events of A Game of Thrones, he is clearly some way up the ranks in the Night's Watch, as he is made Ser Alliser Thorne's replacement as Master-at-Arms at Castle Black. This suggests two things:
He's not a total newbie. He's clearly been at the Wall for some time, which could make him older.
In AGOT, Jon describes Alliser Thorne as being "fifty", which would give him a birth date of 247/248 AC. It might make sense if Endrew, his successor, is a similar age.
There is also something interesting about Endrew: he is a southern knight, and they are really quite rare at the Wall. Why would Endrew, who presumably grew up on Tarth at the southern end of Westeros, have ended up at the Wall?
The backstory of Ser Alliser Thorne might give us another clue. During Robert's Rebellion, Alliser was a Targaryen loyalist, but after Robert's victory he was forced to choose between taking the Black or death. Given there are possible hints that House Tarth are Targaryen loyalists, the same thing could also be true of Endrew. If Endrew is Selwyn's brother and he was forced to join the Watch after Robert's Rebellion, that would mean that he left Tarth c. 283 AC - ie. when Brienne was three years old. Brienne not knowing her family outside her dad therefore might not be due to Selwyn's laziness, but circumstance.
Sure, we have no direct evidence Endrew is Selwyn's brother, but it is a headcanon that I think is not totally crazy. Unfortunately for Brienne, Endrew was killed at the Fight at the Bridge of Skulls against the Wildlings in A Storm of Swords, so there is no way that Brienne will ever get to know this branch of her family.
Whether there are more Tarths remains to be seen, but I would be very interested to find out!
