animals have emotional intelligence, problem solving and communicative abilities that easily rival humans. despite general human ignorance toward it, certain bird species communicate in a way that is very close to language as we define it and chimpanzees can consider how other chimpanzees will act based on their environment (e.g observing violence may make them prone to it).
animals are able to be deeply compassionate in a way humans are often unable to be due to the sheer complexity of our brains and all of the conditioning it undergoes over the process of its life. for instance, i have borderline personality disorder, and so i find it hard to maintain normal perceptions of others, often shifting between very positive and very negative perceptions of anyone i care a lot about. animals don't usually have to deal with that because their brains aren't as complex and they don't live in the same social settings as us, and so can be more compassionate in certain settings even though they are primarily driven by instinct
in buddhism, the intelligence of animals, even if necessarily lower, isn't downplayed. buddhist stories are rife with narratives where animals and animal-like deities (nāgas, etc) are capable of near-human complex thought and behaviour, often intentionally practicing the dharma. it's debatable whether you can say the buddha taught this (i would say no), but it is indicative of the early buddhist perception of animals are near-human or possessing the potential to be near-human.
but this is not important to attaining nirvana. nirvana is supremely difficult even for humans - certain buddhist sects record a story wherein the buddha was hesitant to teach in the first place, because he was skeptical of whether it would even be worth it. most people are extremely ignorant ('with dust in their eyes'), and violent ignorance in the way of religious fanaticism likely caused the deaths of one of his chief disciples, Moggallāna, who is said to have suffered a violent end at the hands of non-buddhist religious extremists. ignorance also led to the only major schism (in its strict buddhist legal sense) buddhism has ever had, when the buddha's cousin devadatta attempted to defame the buddha and start his own community because the buddha did not make certain austerities (e.g wearing robes of rags) mandatory. devadatta was undoubtedly ignorant for this - he wasn't enlightened and buddhism wouldn't be around today if it didn't have the flexibility the buddha gave it. even the buddha's cousin, someone who was a genuinely good meditator and completely capable of being a monk, was able to be so ignorant as to try to split the buddha's community against him and, if the stories are true, attempt to kill him in the process.
to attain enlightenment you need to be able to critically, consistently and constantly reflect on your own actions, develop insight into what leads to suffering, and refine your mind such that you completely sidestep instinct. animals can critically reflect and avoid some of their instincts (think a dog who has been trained out of a behaviour) but can only do this to limited degrees. the average animal pretty much cannot do this at all.
can a cat understand that suffering is an issue, develop spiritual urgency, and make a conscious resolution to attain enlightenment? can they meditate, reflect on their behaviour and figure out the origin of suffering? if humans can almost never do this, the chances an animal could is so impossibly tiny it's not even worth mentioning - or, as the buddha said, it just isn't impossible.
i also dispute your claim that he was unable to recognise the behaviours animals have display with one eachother. ascetics at the time, including buddhist ones, literally lived in the forest (or, later on, forest monasteries). they lived alongside lions, jackals, elephants, snakes, birds, deer and a plethora of other wild animals because they lived right next to them.
most early buddhist literature contain frequent references to the animal behaviours they observed -- a great declaration is called a 'lion's roar', a feeble one is a 'jackal's yelp', in the event you're unable to make good friends one is urged to 'wander alone like a rhino (or rhinoceros' horn)' or 'wander alone like a tusker (male elephant)'. the ideal sleeping position is the 'lion's position' and so on.
consider this section of verses which is found in some early buddhist sources:
“Strewn with garlands of the musk-rose tree,
these regions are so delightful, so lovely,
echoing with the trumpeting of elephants:
these rocky crags delight me!
Glistening, they look like blue stormclouds,
with waters cool and streams so clear,
and covered all in ladybugs:
these rocky crags delight me!
Like the peak of a blue stormcloud,
or like a fine bungalow, lovely,
echoing with the trumpeting of elephants:
these rocky crags delight me!
The rain comes down on the lovely flats,
in the mountains frequented by seers.
Echoing with the cries of peacocks,
these rocky crags delight me!
(...)
Covered with flowers of flax,
like the welkin covered with stormclouds,
full of flocks of many different birds,
these rocky crags delight me!
Empty of householders,
frequented by herds of deer,
full of flocks of many different birds,
these rocky crags delight me!
The water’s clear and the rocks are broad,
monkeys and deer are all around;
festooned with dewy moss,
these rocky crags delight me!”
(Mahākassapattheragāthā, trans. Bhikkhu Sujato, but some of these verses are also found elsewhere - clearly they were common verses that early Buddhists thought were representative of their experiences.)
they spent a great deal of their time (and for less social monks like the monk to whom the above quoted text is attributed, Mahākassapa the 'disciple foremost in renunciation', essentially all of it) in the wilderness right next to these animals and certainly witnessed and took note of their behaviours. they will have seen the compassionate social dynamics elephants have with one another, lionesses caring for their cubs and birds spending time with their chicks. but in practice these behaviours don't make it any easier for them to attain enlightenment.
also, your belief that early buddhists can't have considered humans animals seems to come from your own background and what people think today, and i don't think it's true for late vedic indians. early buddhists came from a predominantly indian animist environment and likely would have been brought up considering themselves pretty much the same as the animals that surrounded them, which is probably another reason why early buddhist stories personify animals so much (local non-buddhist or even pre-buddhist stories, which were often adopted into the religious canon, were from that same animist environment).
brahmins, who made up a fairly substantial portion of the buddha's followers, will have had a different view as they were raised in a very different culture but the buddha was not one of them and they will have grown up knowing others held these animist beliefs
the only thing i can think of is that for practical purposes one can be born in a 'human womb' (manussayoni) and an 'animal womb' (tiracchānayoni, lit. 'horizontal' i.e 'quadrapedal' but this is only its literal meaning) alongside being born in the realm of gods (devaloka), domain of the forefathers (pettivisaya) or realm of ghosts (petaloka), and a realm with nothing good in it (nirayaloka) or hell (narakaloka). keep in mind for those last two that the dialect the early buddhist texts was composed in was very phonetically liberal and similar words could easily be confused (hence petti/peta and niraya/naraka). animals are further subdivided into how they're born, e.g womb-born and egg-born.
