I have a fantasy merchant republic with a stratified society. To simplify - elites and serfs. I had it planned out with histories, religions, etc, none consciously modeled on a particular culture. Then I realized that this might LOOK like an inverse Indian caste system w/ darker>med. There aren't clear similarities beyond a gradated stratification - no mirrored correspondence. Could this still be problematic? I could flip it, but I was trying to have more powerful non-white/light characters.
First of all, what you're worrying about is not looking like the caste system, but looking like the varna system. This is a common mistake in Western circles and it's nearly futile trying to correct it, but I'll discuss some of this later on.
However, what you describe doesn't strike me as particularly reminiscent of the varna system. I’d expect to see some fourfold division, and your "elites" would in turn be divided into the classic "three estates" of medieval France, actually: clergy, nobility, and freemen/urban bourgeoisie, in addition to the "serf" class.
Unless that's your class division system, what you describe doesn't seem much like varna, but more like a plain-old feudal system. To avoid seeming like you're modeling a system off of varna, avoid these points, and avoid using the word "caste." Those are the easy points, but I feel that's actually missing the heart of the issue.
By putting darker-skinned people at the top and making lighter-skinned people oppressed, you do seem to be invoking some colorism, albeit of an inverted sort to the kind usually seen. There was a YA SF novel called Revealing Eden that used a similar conceit in its imagined social structure and it received all sorts of backlash for coming off as racist, intentionally or otherwise.
Finally, "caste" is not the same as social class, but is actually a lineage or community group who've been in the same line of work (or similar lines of work) over the generations. A given caste usually falls within a certain varna, though may cross varna and in different regions the associated varna may be different (for example, members of the Kayasth caste may be found in any varna depending on where they're from). The caste "system" is less a system and more a knotted web of hangups (sometimes severe or even deadly) that certain caste groups have about associating with other caste groups based on what work they do, respectively.
In earlier times, both caste and varna were very fluid and were less about how you were born than about what work you did, but as work became more hereditary, the concepts of varna and caste started to crystallize (about the 1st century AD), and were quite entrenched in certain circles by around the 10th century. There's been a huge amount of religious, philosophical, and legal debate about the concepts since the early Vedic period.
Overall, I would advise some reimagining of your society's social structure, but not for the reasons you might be worried about. Why must there be a stratification based on skin color? Colorism exists in India, for sure, but even in the most restrictive imaginings of the varna system, it's not like all the fairer skinned people automatically become Brahmins.
