As mentioned in my last post, I had some trouble with resets in my PBX due to “some kind of network event”. It was very enlightening when we discovered the root cause, so I wanted to share my experience here in hopes that it saves you from similar sleepless nights.
The symptom was quite simple – the PBX was losing connectivity with its host processor for some reason. Avaya called it a “network event” but couldn’t be more specific. I was fortunate enough to work on the same team as the LAN/WAN techs. If we had been in different departments, this would have been even more of a nightmare. So the Avaya PBX loses heartbeats – after 15 of them it caused a reset. Heartbeats are about one second apart, so a failure of 15 heartbeats is a very long time. Surely a WAN outage of 15 seconds would be noticed by other systems, right? So when Avaya says there was a network event, my response was “uh, you gotta give me more than that. Nothing else on the network noticed”.
I’ll spare you the details here – they’re in my previous post anyway. The issue is the PBX is marking all IP traffic with “Expedited Forwarding” (EF), or Diffserv 46, or High Priority, or QOS. There are plenty of synonyms but it just means all IP packets are tagged with high priority and should be preferred over other packets in the LAN and WAN. Our MPLS carrier honors this tag through their network. Perfect, right?
So when you purchase QOS in the MPLS network, you are given a certain amount of bandwidth allowed. Obviously you’re not allowed to mark all traffic as high priority. MPLS is a shared cloud and you’ll pay a premium for expedited delivery of QOS packets. This bandwidth allotment is called the Committed Access Rate or CAR. What happens when you exceed this CAR? Well, as a telephone guy, I would assume the packets are delivered as “best effort” after that. But I was very wrong. Packets over the CAR are discarded by the MPLS carrier. Think about it – if you have high-priority packages to deliver overnight and the you’re only allowed to send ten per day. The eleventh package isn’t held for best effort. It’s thrown in the dumpster. Sorry. You exceeded your ten packages today, I’m throwing this one out.
The fix is simple and elegant. And probably crystal clear to you network routing engineers out there. It’s up to your edge router to strip the QOS tag from any packets that exceed the CAR. It sounds dangerous to me – what if my edge router and the carrier’s MPLS router disagree slightly on the current amount of traffic at this particular second? Especially if we’re using different brands of router? However, Mbps seems to be an agreed-upon measurement across all vendors, so having my edge router strip the QOS tag from these packets simply allows them to be delivered at best-effort across MPLS.
What I discovered, is if this setting is not correct, it only affects high-priority traffic. In my experience, most companies use a different network for video and everything else on the network such as Internet, email, chat, file servers, replication, database synchronization, etc. are not marked with QOS. So guess what? It looks like your phone system is hosed! And nothing else on the network is complaining! This gets back to a trend I’ve been noticing: Being a phone guy doesn’t have much to do with telephones anymore. It’s almost always the network. And to be a really good phone tech, you have to know networking well. I hope this helps.