Monthly Archives: May 2016

Avaya Call Detail Recording to Kiwi syslog server

In a previous post about busy/release of CDR, I asked if anyone was curious about capturing CDR from Avaya Communication Manager using a Kiwi Syslog server. At least one of you was, so here it is.

There are two parts of course. One part is programming the phone system to send CDR. And the other part is programming Kiwi to capture CDR. These can be done in either order, but if you program the phone system first, the CDR buffer will fill as is tries to establish the connection with Syslog.

So let’s start with Kiwi. Your version may differ. In my case, the network team already had a rather powerful installation of Kiwi on some great hardware. All I had to do was configure it to capture Avaya CDR. So when you launch the Kiwi dashboard, click File->Setup and in the “Server Setup” window, scroll to the bottom for “Inputs”.

kiwi-config-0

In the TCP section, tell Kiwi to listen for TCP messages on whatever port you wish. I’m using 5013 – probably because I saw it in an Avaya doc somewhere. But you can use any port.

Then scroll up to the “Rules” section and configure a new rule to match for messages from your Avaya Communication Manager.

kiwi-config-1

In my case, I named my rule “Call Records from LA PBX”. I set a filter by IP address. If you also send any other information to Kiwi from the CMs, then you’ll want to filter by the Priority field. You want the Local7 field to be Debug. I don’t recall why this works. I think I was looking at the data one day to try to separate the CDR from the other syslog messages from Avaya and this seemed the easiest way to do it.

kiwi-config-3

Once you have those two filters – IP address and Local7.Debug, you need to create an action. My action above is a little misleading. All you really need is one that logs to a file, then stops processing. If you’d like, you can also display it to one of the virtual displays in Kiwi. I log to a text file on the local file system:

kiwi-config-2

Now, if we program our PBX correctly, we should get CDR to these text files!

Step 2 – Program the Avaya Communication Manager

Programming the Communication Manager is three steps: Add the node-name, set the ip-services, and change the system-parameters cdr.

To set the node name, ‘change node-name ip’ and create an entry for the IP address of your Kiwi server. This is basically the DNS lookup for the CM. This sets the name and IP address. Any name will do. Enter the IP address of your Kiwi server.

change node-names ip                                            Page   1 of   2
                                  IP NODE NAMES
    Name              IP Address
SL_LSP              10.10.60.10
CHA_LSP             10.10.1.10
Chicago             10.10.148.2
Chicago_LSP         10.30.10.10
DO_LSP              10.20.16.10
Smoot               10.87.176.11
Gateway006          10.46.12.1
LA                  10.10.19.6
LA_ESS              10.44.192.10
LE_ESS              10.58.200.10
LA_SM               10.59.29.16
( 16 of 44   administered node-names were displayed )
Use 'list node-names' command to see all the administered node-names
Use 'change node-names ip xxx' to change a node-name 'xxx' or add a node-name

Next, set the ip-services. I have squeezed all three screens here. The important thing here is page three. We need to set the “Reliable Protocol” to n. I think Avaya defaults this to y and it works great for their pre-packaged CDR partners. For home-grown syslog, we set this to n. It may not hold up in court if there’s ever a dispute.

change ip-services                                              Page   1 of   3

                                   IP SERVICES
 Service     Enabled     Local        Local       Remote      Remote
  Type                   Node         Port        Node        Port
CDR1                 procr            0       la_syslog       5013
CDR2                 procr            0       prognosis       50000

change ip-services                                              Page   2 of   3

                                   IP SERVICES
 Service     Enabled     Local        Local       Remote      Remote
  Type                   Node         Port        Node        Port


change ip-services                                              Page   3 of   3

                              SESSION LAYER TIMERS
  Service     Reliable  Packet Resp   Session Connect  SPDU  Connectivity
   Type       Protocol     Timer       Message Cntr    Cntr     Timer

 CDR1            n         10                3          3         10
 CDR2            n         10                3          3         10


And last step – set the system-parameters for CDR. Here’s the thing. You have lots of options here. You can define fields, separators, etc. But I discovered that there is a line length limit that is not obvious. Also, the caller-id name is surprisingly absent. Rather than customize the CDR, I have gone back to “unformatted” and let further processing figure it out.

change system-parameters cdr                                    Page   1 of   1
                            CDR SYSTEM PARAMETERS

 Node Number (Local PBX ID): 2                     CDR Date Format: month/day
      Primary Output Format: unformatted   Primary Output Endpoint: CDR1
    Secondary Output Format: unformatted Secondary Output Endpoint: CDR2
           Use ISDN Layouts? n                   Enable CDR Storage on Disk? n
       Use Enhanced Formats? n      Condition Code 'T' For Redirected Calls? n
      Use Legacy CDR Formats? y                 Remove # From Called Number? n
Modified Circuit ID Display? n                             Intra-switch CDR? y
                  Record Outgoing Calls Only? n     Outg Trk Call Splitting? y
  Suppress CDR for Ineffective Call Attempts? n       Outg Attd Call Record? y
      Disconnect Information in Place of FRL? y      Interworking Feat-flag? n
 Force Entry of Acct Code for Calls Marked on Toll Analysis Form? n
                                    Calls to Hunt Group - Record: member-ext
Record Called Vector Directory Number Instead of Group or Member? n

     Inc Trk Call Splitting? n
  Record Non-Call-Assoc TSC? n           Call Record Handling Option: warning
      Record Call-Assoc TSC? n   Digits to Record for Outgoing Calls: dialed
   Privacy - Digits to Hide: 0               CDR Account Code Length: 15
Remove '+' from SIP Numbers? y


I have a perl script that parses unformatted cdr. Anyone want to see it?

Anyway, when you have this set up, you can status CDR to make sure the link is up:

status cdr-link
                                CDR LINK STATUS
                   Primary                      Secondary

       Link State: up                           up

      Date & Time: 2016/04/17 08:43:29          2016/05/09 10:12:20
  Forward Seq. No: 0                            0
 Backward Seq. No: 0                            0
CDR Buffer % Full:   0.00                         0.00
      Reason Code: OK                           OK





The good news here is the link state(s) are “up”  and the CDR buffer % full is 0. If you have trouble, you can “busy cdr primary” and “release cdr primary” to close and open the socket. You should see a blip in the Kiwi server when this happens.

Make a call and watch Kiwi! You may need to check the log files rather than rely on the Kiwi display. The extra CR/LF will sometimes make Kiwi look funky but the text file is usually just fine.

Let me know how it goes for you!

Roger

 

Avaya phone reboots, and then doesn’t have dialtone

I have had this problem in three offices starting in 2013. And it just happened to me today in an overseas office. This problem can remain hidden in your telephone system for years and not be a problem because it’s so rare for a phone to get assigned a new IP address via DHCP.

Basically, a phone reboots for some reason. When it comes back up, there’s no dialtone. Or, if you have more than one gateway,  you might get intermittent dialtone.

In my network, these items were also true:

  1. The gateway and the phones are on the same subnet
  2. The phone was assigned a different IP address when it rebooted (this may not be obvious since you don’t always know the phone’s previous IP address)

If you have more than one gateway, the problem might seen intermittent. you can “list trace station” and watch the phone pull dialtone:

12:19:49   G711A ss:off ps:20
           rgn:1 [10.9.10.90]:39798
           rgn:1 [10.9.10.3]:2094

That IP address on the third line – that’s the gateway assigned to serve dialtone for this call. In my case, I have a gateway in the local office, and a gateway in the datacenter. Whenever the user said “hey, there’s no dialtone!”, it had drawn a resource from the local gateway in the same subnet. The phone worked in all other ways – there was just no media.

There are MILLIONS of Avaya forum postings regarding no dialtone, one-way audio, etc. In my case here, the problem was caused by the arp cache in the G450 gateway. That’s pretty obscure. The IP address changed, but the MAC address in the arp cache did not change. So the gateway is trying to feed dialtone to the wrong MAC address.

The fix was simple: log into the G450 and issue these commands:

  • no ip arp inspection‘ (this disables arp caching)
  • clear arp-cache‘ (this clears the arp-cache, which sounds scary but will not affect service)
  • copy run start‘ (this will write the settings to flash memory to survive a reboot)

That fixed the issue of no dialtone. It did not fix the issue of phones rebooting and getting different IP addresses though. I will write a different post for that, since it was a completely unrelated issue.

I hope this helps! Please let me know if this works for you.

Roger