I highly doubt I’m the first person to think this up, but, what if Doofenschmirtz has prosopagnosia, a.k.a ‘face-blindness’?
Per Wikipedia, “People with face blindness may struggle to notice differences in faces of strangers. Others may even have a hard time recognizing familiar faces. It’s estimated to affect about 2 percent of the general population.“
Doof is smart enough to work around this easily. People with this condition, (particularly with severe cases,) memorize the most unique part of an individual’s face and use that to identify them. Doof is especially good at this, to the point where his condition is seen as just another one of his eccentricities. Let’s say he’s middling on the scale, where strangers can be hard to remember, acquaintances less so, friends are easy, and family is a no-brainer unless he’s particularly stressed and scatterbrained at that moment.
But, the thing is, that’s with humans. Now, with most ‘standard’ mammals--think cat-like, dog-like, bear-like--he can pick a facial feature to remember. Being raised by ocelots helped a lot with this. However, with faces of creatures like snakes, anteaters, platypi, etc., he has a harder time differentiating between them unless one has a very distinct mark.
When he runs into animal agents like this, his focus tends to land on the fedora. It’s obviously the most prominent feature, and each agent has very unique scuffs and scrapes on theirs, like a fingerprint.
And that’s why he doesn’t recognize Perry without the fedora.
...Though, the fact that he can’t put two and two together regarding the disappearance of Agent Perry and sudden emergence of Random Normal Teal Platypus is aaaaaall Doof.