Traceroute aberrant behaviour?
A colleague of mine recently advised a systems admin he was working with that network traffic seemed to be bouncing back and forth between the same switches off the back of a trace route he’d just taken, so I thought I’d do a quick post to clarify what he and I’m sure others were seeing / have seen.
To explain we’ll take a look at traffic from a client in the 10.1.1.0/24 network to a server in the 10.1.6.0/24 network that traverses several layer 3 devices with redundant paths. The topology we’ll be examining is shown above.
Looking at the output below, one could almost be forgiven for thinking that traffic is actually bouncing between R1 and R2:
DSW1#trace 10.1.6.1 Type escape sequence to abort. Tracing the route to 10.1.6.1 1. 10.1.11.1 16 msec 10.1.12.1 8 msec 10.1.11.1 12 msec 2. 10.1.23.1 20 msec 10.1.13.3 32 msec 10.1.23.1 40 msec 3. 10.1.45.1 64 msec 56 msec 40 msec 4. 10.1.6.1 56 msec * 64 msec
If you take a closer look at the output above the clue is in the numbers, the traceroute utility actually tells you how many hops you’re dealing with, but, if that’s the case then why are we seeing multiple IP addresses per hop? A look at the route to Server1 on DSW one should cast a little light on the issue.
DSW1#sh ip ro 10.1.6.0 Routing entry for 10.1.6.0/24 Known via eigrp 100, distance 90, metric 38400, type internal Redistributing via eigrp 100 Last update from 10.1.12.1 on FastEthernet1/2, 00:07:34 ago Routing Descriptor Blocks: * 10.1.11.1, from 10.1.11.1, 00:07:34 ago, via FastEthernet1/1 Route metric is 38400, traffic share count is 1 Total delay is 500 microseconds, minimum bandwidth is 100000 Kbit Reliability 255/255, minimum MTU 1500 bytes Loading 1/255, Hops 4 10.1.12.1, from 10.1.12.1, 00:07:34 ago, via FastEthernet1/2 Route metric is 38400, traffic share count is 1 Total delay is 500 microseconds, minimum bandwidth is 100000 Kbit Reliability 255/255, minimum MTU 1500 bytes Loading 1/255, Hops 4
As you can see, DSW1 has 2 equal cost paths to get to the network Server1 is on.
How / why does traceroute return multiple IPs when there are multiple paths?
The trace route utility sends out 3 UDP packets at a time with incrementing TTLs These UDP probes are process switched packets sourced from router, not actual network traffic. Due to the fact there are multiple paths the first probe is sent via the first path, the second probe via the second path and the third probe via the first path again. For more information you can check out the ever helpful traceroute command on Cisco devices article.
Ultimately, it doesn’t usually matter that there are multiple IPs in the same hop in a traceroute you’ve run from a layer 3 device or which path actual network traffic takes (Unless there’s a firewall in the way, which brings its own packet out-of-state headaches or you have some specific traffic engineering requirements) as long as said traffic reaches to and from the distribution switches connected to our end networks.
