Could you please talk more about how/why agreement systems develop in language? I always feel that agreement systems are poorly justified in my conlanging
Agreement systems are retained in language because the redundancy strengthens the signal. It's better to think about linguistics systems in terms of "why didn't speakers get rid of this" as opposed to "why did this come about in the first place". Sometimes things happen randomly. They're retained because they're useful.
But like, consider gender in English. We used to have it. Gender was mostly defined by the endings of words. We lost all the endings. We lost some major forms of agreement (consider that French still has different articles). At a certain point, it was impossible to tell if a noun was m/f/n, so of course English lost its gender system. It was no longer useful. In fact, going even further, it was the opposite of useful, because it was totally unpredictable and didn't buy you anything.
As an example of the latter, there's this sign system called Signed Exact English (SEE). It's often (not always, but often) pedaled as a replacement for ASL, because it will "help" Deaf signers learn English. One of the features it retains is the distinction between "a" and "an". English speakers know how to do this instinctively: You use "a" before a noun phrase (not a noun, but a noun phrase) that begins with a consonant sound, and you use "an" before a noun phrase that begins with a vowel sound (so "umbrella" gets "an", but "union" gets "a"). In SEE, there's a separate sign for "a" and "an", and then ASL signs are used for English words like "man" and "old". So then you have to sign:
But, of course, the difference is based on the sound of the English pronunciation of the word the sign stands for, so it is quite literally impossible to predict for a Deaf signer. It has to be memorized. Which is an extraordinary task. Basically, all nouns, adjectives, and adverbs (consider "a really old man") have to be dumped in either the A class or the AN class with absolutely no way to predict which will be in which.
This is a great example of a feature that would quickly die in a natural language.
So looking at gender, the question is how useful is it? If it's (a) predictable, and (b) spread across multiple areas of the language, then it's more useful, and more likely to be retained. If you look at Spanish, agreement is present in pronouns, adjectives, demonstratives, and articles. The gender of a noun highly predictable (not 100%, but highly predictable). That's a stable gender system. French is similar, but the gender is less predictable for nouns. If one was going to lose gender first, you'd predict French. Even so, it's still predictable enough that more will probably have to happen for French to actually lose it.
As for where it comes from, if you want to read a detailed account of the development of Indo-European gender, this is an intro. Most of the time it's the incorporation of pronouns or small, generic nouns that become commonly associated with particular classes and are used as modifiers. We've got a pretty good example of the development of noun class in Sarkezhe, season 4 of LangTime Studio. If you want to see it done from beginning to end, check that out.