So the cynical, out-of-universe, fuck-RIB answer is that Brittany writes in crayon because TPTB at Glee want to depict her as being childish.
Note that in S1 and S2, while Brittany is occasionally shown to doodle in crayon, when she writes, she is most often shown using her beloved (and age-appropriate) Koosh Ball gel pens.
The one instance in which she writes using crayons on a major assignment is on her “heart attacks” report from episode 2x03, and in that instance, her doing so can be explained by the fact that her report includes artwork (and using crayons to draw pictures makes sense). She is otherwise shown using typical writing utensils for her schoolwork.
This choice in writing tools tracks with her general S1/S2 depiction: i.e., as a ditz but at least an average teenage ditz who could plausibly exist (and function) in a mainstream high school classroom.
Only in S3 and especially S4, when the writers lean hard into the “Brittany is infantile” party line, does the crayon writing become a standard, repeated joke. The downgrade from Koosh pens to crayons tracks with the general degradation of Brittany’s character during this period.
As for within the universe of the show, one very simple and viable explanation for Brittany writing in crayon is personal preference: i.e., crayon is a brighter and more colorful medium than pencil or (standard) pen and is therefore an attractive medium for someone as creative and whimsical as she.
However, a more complicated (and psychoanalytical) explanation also potentially exists, and it ties into the argument I make in this post: namely, that over the course of her high school education, Brittany’s teachers, counselors, and coaches alternately ignore and disparage her when she struggles in her classes, and the fact that they do so prompts her to “act out academically.”
Though it would be in Brittany’s best interest to ask for help directly, she doesn’t, largely because she has been made to feel unwelcome to do so. Rather than approach Will, Sue, Mrs. Hagberg, Emma, etc., all of whom have, in their own ways, made it clear that they consider Brittany stupid and/or obnoxious, and therefore unworthy of being taken seriously, Brittany puts up a façade, pretending that she’s too aloof to care about her schoolwork. She lives down to their low expectations for her because doing so allows her at least a modicum of control in a situation where she otherwise has none.
If we accept this model, then we can perhaps view writing her assignments in crayon as a symptom of Brittany’s scholastic desperation.
Essentially, Brittany feels like she’s going to fail her assignments no matter what she does or how hard she tries not to, because her teachers have proven to her, class after class, year after year, that they are unwilling to deal with her and/or give her a fair shake. She therefore decides (either consciously or subconsciously) that if she’s going to fail regardless, she is at least going to choose the mechanism by which she does so—and somehow it hurts less to fail because she writes her assignments in crayon than it does to fail because her teachers think that her ideas are stupid/wrong.
Remember that math test Sue whips out in episode 4x02? The one that Brittany draws “Happyville: the Town Where Math Was Never Invented” on?
Notice that Brittany doesn’t even bother to write down answers on that test.
Maybe she doesn’t write down answers because she’s using the crayon drawings as a defense mechanism.
If her teacher is just going to pick her apart anyway, why bother to put in any sort of real effort? The faculty all view her as an imbecile, so she’s going to give them what they expect. She’s not going to make herself vulnerable by expending her actual brainpower to come up with an answer that they’re just going to (arbitrarily) strike out because she’s her.
I would wager that as a yet undiscovered math genius, Brittany probably finds it difficult to “show her work” in the way that high school teachers typically require. She probably just “knows what she knows” intuitively/instinctively, and since she can’t write down the answer without showing her work, lest she be accused of cheating on the exam, she opts not to write anything at all. Things are just easier that way.
As I talk about in this Brittanalysis, Brittany may also be “crying for help” with the crayon drawing, hoping against hope that maybe someday, some teacher (possessed of both a brain and a heart) will realize that any twelfth grader who’s drawing stick figures in crayon on their math midterm probably needs help and actually pull her aside to ask her, in a genuine way, how they can be of assistance.
She’s too scared to ask for what she needs directly, so she hints at it—with Crayola cartoons—instead.
Unfortunately for Brittany, the only person who does seem to notice and/or care about the crayon drawings is Sue, who only cares insofar as this increasingly erratic behavior is having a bad influence on the younger Cheerios, and so summarily kicks Brittany off the squad.
Admittedly, a hole in this theory is that Brittany does seem to continue using crayon to write long after she has left both WMHS and MIT, even into S6, where we see her working on the Euler Brick problem using a rather impressive Crayola collection.
However, we can perhaps explain this usage by means of our simpler explanation from above: i.e., by this point in her life, now that Brittany is no longer beholden to academic rules and is just doing math for fun because she’s good at it, she uses the crayons because they’re colorful and pretty and because they make her calculations pop on the page.
Also, she may use crayons because she is potentially numerically synesthestic, as a few math geniuses, including famous nonnormative thinker Daniel Tammet, are. Note that in her Euler equations, she alternates colors, perhaps indicating that different calculations are chromatically different in her mind. (“The square root of four is rainbows,” anyone?)