Avaya Call Detail Recording to Kiwi syslog server

In a previous post about busy/release of CDR, I asked if anyone was curious about capturing CDR from Avaya Communication Manager using a Kiwi Syslog server. At least one of you was, so here it is.

There are two parts of course. One part is programming the phone system to send CDR. And the other part is programming Kiwi to capture CDR. These can be done in either order, but if you program the phone system first, the CDR buffer will fill as is tries to establish the connection with Syslog.

So let’s start with Kiwi. Your version may differ. In my case, the network team already had a rather powerful installation of Kiwi on some great hardware. All I had to do was configure it to capture Avaya CDR. So when you launch the Kiwi dashboard, click File->Setup and in the “Server Setup” window, scroll to the bottom for “Inputs”.

kiwi-config-0

In the TCP section, tell Kiwi to listen for TCP messages on whatever port you wish. I’m using 5013 – probably because I saw it in an Avaya doc somewhere. But you can use any port.

Then scroll up to the “Rules” section and configure a new rule to match for messages from your Avaya Communication Manager.

kiwi-config-1

In my case, I named my rule “Call Records from LA PBX”. I set a filter by IP address. If you also send any other information to Kiwi from the CMs, then you’ll want to filter by the Priority field. You want the Local7 field to be Debug. I don’t recall why this works. I think I was looking at the data one day to try to separate the CDR from the other syslog messages from Avaya and this seemed the easiest way to do it.

kiwi-config-3

Once you have those two filters – IP address and Local7.Debug, you need to create an action. My action above is a little misleading. All you really need is one that logs to a file, then stops processing. If you’d like, you can also display it to one of the virtual displays in Kiwi. I log to a text file on the local file system:

kiwi-config-2

Now, if we program our PBX correctly, we should get CDR to these text files!

Step 2 – Program the Avaya Communication Manager

Programming the Communication Manager is three steps: Add the node-name, set the ip-services, and change the system-parameters cdr.

To set the node name, ‘change node-name ip’ and create an entry for the IP address of your Kiwi server. This is basically the DNS lookup for the CM. This sets the name and IP address. Any name will do. Enter the IP address of your Kiwi server.

change node-names ip                                            Page   1 of   2
                                  IP NODE NAMES
    Name              IP Address
SL_LSP              10.10.60.10
CHA_LSP             10.10.1.10
Chicago             10.10.148.2
Chicago_LSP         10.30.10.10
DO_LSP              10.20.16.10
Smoot               10.87.176.11
Gateway006          10.46.12.1
LA                  10.10.19.6
LA_ESS              10.44.192.10
LE_ESS              10.58.200.10
LA_SM               10.59.29.16
( 16 of 44   administered node-names were displayed )
Use 'list node-names' command to see all the administered node-names
Use 'change node-names ip xxx' to change a node-name 'xxx' or add a node-name

Next, set the ip-services. I have squeezed all three screens here. The important thing here is page three. We need to set the “Reliable Protocol” to n. I think Avaya defaults this to y and it works great for their pre-packaged CDR partners. For home-grown syslog, we set this to n. It may not hold up in court if there’s ever a dispute.

change ip-services                                              Page   1 of   3

                                   IP SERVICES
 Service     Enabled     Local        Local       Remote      Remote
  Type                   Node         Port        Node        Port
CDR1                 procr            0       la_syslog       5013
CDR2                 procr            0       prognosis       50000

change ip-services                                              Page   2 of   3

                                   IP SERVICES
 Service     Enabled     Local        Local       Remote      Remote
  Type                   Node         Port        Node        Port


change ip-services                                              Page   3 of   3

                              SESSION LAYER TIMERS
  Service     Reliable  Packet Resp   Session Connect  SPDU  Connectivity
   Type       Protocol     Timer       Message Cntr    Cntr     Timer

 CDR1            n         10                3          3         10
 CDR2            n         10                3          3         10


And last step – set the system-parameters for CDR. Here’s the thing. You have lots of options here. You can define fields, separators, etc. But I discovered that there is a line length limit that is not obvious. Also, the caller-id name is surprisingly absent. Rather than customize the CDR, I have gone back to “unformatted” and let further processing figure it out.

change system-parameters cdr                                    Page   1 of   1
                            CDR SYSTEM PARAMETERS

 Node Number (Local PBX ID): 2                     CDR Date Format: month/day
      Primary Output Format: unformatted   Primary Output Endpoint: CDR1
    Secondary Output Format: unformatted Secondary Output Endpoint: CDR2
           Use ISDN Layouts? n                   Enable CDR Storage on Disk? n
       Use Enhanced Formats? n      Condition Code 'T' For Redirected Calls? n
      Use Legacy CDR Formats? y                 Remove # From Called Number? n
Modified Circuit ID Display? n                             Intra-switch CDR? y
                  Record Outgoing Calls Only? n     Outg Trk Call Splitting? y
  Suppress CDR for Ineffective Call Attempts? n       Outg Attd Call Record? y
      Disconnect Information in Place of FRL? y      Interworking Feat-flag? n
 Force Entry of Acct Code for Calls Marked on Toll Analysis Form? n
                                    Calls to Hunt Group - Record: member-ext
Record Called Vector Directory Number Instead of Group or Member? n

     Inc Trk Call Splitting? n
  Record Non-Call-Assoc TSC? n           Call Record Handling Option: warning
      Record Call-Assoc TSC? n   Digits to Record for Outgoing Calls: dialed
   Privacy - Digits to Hide: 0               CDR Account Code Length: 15
Remove '+' from SIP Numbers? y


I have a perl script that parses unformatted cdr. Anyone want to see it?

Anyway, when you have this set up, you can status CDR to make sure the link is up:

status cdr-link
                                CDR LINK STATUS
                   Primary                      Secondary

       Link State: up                           up

      Date & Time: 2016/04/17 08:43:29          2016/05/09 10:12:20
  Forward Seq. No: 0                            0
 Backward Seq. No: 0                            0
CDR Buffer % Full:   0.00                         0.00
      Reason Code: OK                           OK





The good news here is the link state(s) are “up”  and the CDR buffer % full is 0. If you have trouble, you can “busy cdr primary” and “release cdr primary” to close and open the socket. You should see a blip in the Kiwi server when this happens.

Make a call and watch Kiwi! You may need to check the log files rather than rely on the Kiwi display. The extra CR/LF will sometimes make Kiwi look funky but the text file is usually just fine.

Let me know how it goes for you!

Roger

 

Avaya phone reboots, and then doesn’t have dialtone

I have had this problem in three offices starting in 2013. And it just happened to me today in an overseas office. This problem can remain hidden in your telephone system for years and not be a problem because it’s so rare for a phone to get assigned a new IP address via DHCP.

Basically, a phone reboots for some reason. When it comes back up, there’s no dialtone. Or, if you have more than one gateway,  you might get intermittent dialtone.

In my network, these items were also true:

  1. The gateway and the phones are on the same subnet
  2. The phone was assigned a different IP address when it rebooted (this may not be obvious since you don’t always know the phone’s previous IP address)

If you have more than one gateway, the problem might seen intermittent. you can “list trace station” and watch the phone pull dialtone:

12:19:49   G711A ss:off ps:20
           rgn:1 [10.9.10.90]:39798
           rgn:1 [10.9.10.3]:2094

That IP address on the third line – that’s the gateway assigned to serve dialtone for this call. In my case, I have a gateway in the local office, and a gateway in the datacenter. Whenever the user said “hey, there’s no dialtone!”, it had drawn a resource from the local gateway in the same subnet. The phone worked in all other ways – there was just no media.

There are MILLIONS of Avaya forum postings regarding no dialtone, one-way audio, etc. In my case here, the problem was caused by the arp cache in the G450 gateway. That’s pretty obscure. The IP address changed, but the MAC address in the arp cache did not change. So the gateway is trying to feed dialtone to the wrong MAC address.

The fix was simple: log into the G450 and issue these commands:

  • no ip arp inspection‘ (this disables arp caching)
  • clear arp-cache‘ (this clears the arp-cache, which sounds scary but will not affect service)
  • copy run start‘ (this will write the settings to flash memory to survive a reboot)

That fixed the issue of no dialtone. It did not fix the issue of phones rebooting and getting different IP addresses though. I will write a different post for that, since it was a completely unrelated issue.

I hope this helps! Please let me know if this works for you.

Roger

How to load-test your telephone system, IVR, or DID number port

When you work on telephone systems, at some point you need to make some test calls into your trunks. And sometimes it’s not realistic to use the same phone at your desk, since the call may not truly leave your PBX or your carrier’s network. So you end up using your mobile phone,or if you’re remote, then your home phone. However,

  • What if you need to make dozens of calls?
  • What if you need to load up a trunk group?
  • What if you need to test a number port of hundreds of individual DIDs?
  • What if you need to simulate calls from different areas of the country?
  • What if you need to dial an extremely specific sequence of digits once connected, and you need to do this over-and-over-and-over?

About a year ago, I found a great tool called CallsByCloud.com. This is an automated testing framework – completely self service using your browser. I liked it so much I became a business partner recommend it highly to all of you telecom administrators. I even recorded the training videos for them.

It’s very straight-forward to use. You build a “test scenario” that contain simple “scripts” that are use cases for each type of call. If you’re testing a call center IVR, this could be “make payment” or “renew contract” or “queue for customer service”. And if you’re just testing a trunk group, you can dial into your PBX and “hold” the channel for x seconds. This lets you load the trunk group and test overflow.

cbc-script-commands

Once the tests are complete, you can see the call history, channel usage, a spectrogram of the call, and you can listen to or download the audio.

cbc-history

Once you set up a test, you can schedule it to run at a recurring time and alert you if there are any problems with the call.

cbc-qa-schedule

There are other advanced features such as

  • Caller-id manipulation to simulate geographic routing.
  • Detailed logs of the call processing.
  • Number pooling for testing multiple numbers.
  • The ability to set a “disposition” or “goto” a different spot in the script based upon the timing of noise, silence, or DTMF in the channel.
  • The ability to export call records for the tests.

It’s flexible, cost effective, and has made my job so much easier. I recommend it highly to all of you. If you sign up and use the promotion code ‘roger2015’, you’ll get $10 in credit when you sign up and a 20% discount on the per-minute rate.

Check out the training videos and let me know what you think!

Thanks for watching!

Roger

 

How to redirect Avaya phones to different config files based upon source IP address

Hello everyone. I had a situation a while back where I needed to tell the 9630 phones in a remote office about a new call server. It was after-hours and I could not reach the network administrator responsible for the DHCP option strings for that site. I ended up creating a rewrite rule in Apache to redirect these phones to a different config file where I could overwrite the call server settings. Later I did the same thing for an international site where their date/time format was different from everyone else’s.

You can also use the GROUP setting in the phone to load specific config information, but as one reader discovered, the GROUP settings are not available to SIP phones! If you’re using Apache to host your config files, here is how to re-direct those phones to custom files.

If you are using defaults, your Apache config file is probably at /etc/httpd/conf/httpd.conf, and your document root for your web server is probably /var/www/html/

So open /etc/httpd/conf/httpd.conf in your favorite editor and find the section

<Directory "/var/www/html">
...buncha stuff
</Directory>

Inside the <Directory> </Directory> tags, add these lines:

    RewriteEngine On
    RewriteCond %{REMOTE_ADDR}= 10.101.
    RewriteRule 46xxsettings.txt SPAIN_46xxsettings.txt
    RewriteCond %{REMOTE_ADDR}= 10.139.
    RewriteRule 46xxsettings.txt OHIO_46xxsettings.txt

This will turn on the Apache rewrite engine and if the remote IP address matches the pattern 10.101, then any request for 46xxsettings.txt will be rewritten to SPAIN_46xxsettings.txt. Likewise, anything from 10.139 will get OHIO_46xxsettings.txt

As I look a this, the rewrite condition should probably start with a carrot ^ to try to match the beginning of the line, but it’s working for me so I don’t want to fiddle. But if I have a phone from 10.210.139.5, this rewrite will probably kick in because of that 10.139. within the IP address. Putting a carrot at the beginning should force it to only match addresses that begin with 10.139.

Unfortunately, you’ll need to restart Apache, but the phones won’t care if they miss a PUT.

[root@phonewebserver httpd]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
[root@phonewebserver httpd]#

Mine takes less than a second. I don’t think anything noticed.

So then just create your SPAIN and OHIO settings files and the phone should get those. If it doesn’t seem to work, here are some troubleshooting steps.

Make sure the rewrite module is loading in Apache

Check the httpd.conf file and make sure there is a line to actually load the rewrite module. Mine is on line 190:

LoadModule rewrite_module modules/mod_rewrite.so

Tail the Apache access_log to confirm the IP addresses and GET requests from the phones

In a busy environment, it is not helpful to just tail the access log. But you can tail and grep at the same time! When troubleshooting, I just want to see the GET requests, so enter this command:

tail -f /var/log/httpd/access_log|grep --line-buffered GET

And that will follow the log but only show GET requests. You could also

tail -f /var/log/httpd/access_log|grep --line-buffered 10.139

And this will follow the log but only show activity from the 10.139 subnet

Temporarily enable the rewrite log so you can see what is getting rewritten

Just before the <Directory> tag in Apache, enable the rewrite log with

RewriteLog "/var/log/httpd/rewrite.log"
RewriteLogLevel 4
<Directory "/var/www/html">

You’ll need to restart Apache again. There should be enough information in that rewrite.log to figure out what is happening. It’s a lot of information, so be sure to set the RewriteLogLevel to 0 when you’re done troubleshooting.

I hope this helps! Please let me know how your testing is going. It would also be helpful to know if any of you get your GROUP settings to work over SIP. Thanks all!

Avaya System Manager – unable to login

There are many reasons for the login to fail on System Manager. Version 6.3.x seems better but this thing still feels like it’s built on a house of cards. Tonight I just found another reason and it was new to me so I want to tell you all about it.

Earlier today I tired to login and was confronted with this error

avaya-system-manager-internal-error

Some internal error has occurred in the service. Please contact the support team.

Guess who the support team is? Oh yeah, that’s us!

Well, I didn’t really know what to do and it was a Sunday so I figured I would reboot it. Now my System Manager is a virtual machine and I don’t have access to the VMWare console. But I was able to SSH into System Manager as admin. Then su and enter the root password. From there, I was able to issue a reboot command. It dropped my session, and then it never did come back to ping. So I contacted my VM team and they rebooted it. Once they did that, I was able to ping and SSH to it. But the web page said “This page has a redirect loop”. I knew it could take a long time to come back up, so I gave it a while. But no dice. I contemplated rebooting it again, but rather than that, I went through the “ask Ava” chat pages on support.avaya.com. I was hoping with such a specific error like “redirect loop” I would find something and sure enough, Ava pointed me to SOLN270630, which was exactly my problem. According to this page, the Postgres certificate had expired. Avaya provided the command to confirm it and the command to fix it. All I had to do was run that command and then restart jboss. Then wait. Don’t forget, it can take 15-20 minutes for the restart to finish. Don’t panic. Actually, I think you do have to panic. Three minutes after you start to panic it comes back up. Don’t forget to re-enable Geographic Redundancy!

root >openssl x509 -text -in /var/lib/pgsql/data/server.crt | grep "Not After"
 Not After : Oct 22 15:14:31 2015 GMT
root >/opt/Avaya/Postgres/*/utils/securePostgres.sh
Mon Oct 26 00:13:18 EDT 2015 : Restarting Postgres service...
Stopping postgresql service: [ OK ]
Starting postgresql service: Waiting for Postmaster PID file creation .
Waiting for Postmaster DB connection availability .
Starting postgresql service: [ OK ]
Mon Oct 26 00:13:21 EDT 2015 : Startup of Postgres done
root >service jboss restart
Perform cleanup ....
Stopping System Manager JBoss - PID: 6580
.............................................
Stopped System Manager JBoss
Starting System Manager JBoss
.
Service started successfully - PID: 21296
root >

Avaya CM – will I lose the CDR buffer if I busy/release the cdr link?

I have an Avaya Communication Manager sending CDR to a kiwi syslog server. For some reason, the link is down and the buffer is filling up. What happens if I busy/release the CDR link? Will I lose the buffer? Scary!

Short answer is no. I just tried it. I performed a busy/release on the primary CDR link and the buffer is still at 40%. Unfortunately, the link is still down so I’ll have to check with the syslog team. But I just wanted to let all of you know that it’s safe to restart the link.

status cdr-link
                                CDR LINK STATUS
                   Primary                      Secondary

       Link State: down                         up
Number of Retries: 18
      Date & Time: 2015/09/23 06:35:59          2015/09/12 07:46:56
  Forward Seq. No: 0                            0
 Backward Seq. No: 0                            0
CDR Buffer % Full:  41.21                         0.00
      Reason Code: CDR connection is closed     OK


I ended up disabling and then enabling the TCP listener in Kiwi and the buffer immediately cleared. Is anyone interested in a post on how to capture CDR to a SYSLOG server?

Flowchart of Avaya Communication Manager Routing

Click Here for the flowchart PDFs

One might argue that telephone systems do only one thing. They route calls. In the process they turn lamps on and off, and they make telephones buzz and beep. But the only point is to route voices from point A to point B.

If you’ve been working with Avaya Communication Manager for a while, you have been exposed to AAR and ARS routing. And you hopefully have a fair grasp of how it all fits together. But there are a whole lot of decisions that are made when routing outbound calls. And if you’re new to Avaya, it might seem overwhelming to troubleshoot routing issues. So here are four pages of flowcharts that, based on my experience, are the most common configuration issues you’ll see when managing your telephone system. Click Here for the flowchart PDFs

Page 1 is a flowchart of the digit collection. This is what the CM does to collect the dialed digits, translate them as necessary, and select a network (AAR vs. ARS). Digits can be translated many times and re-pointed to different networks. Once this process is complete, the logic continues to page 2.

avaya_call_routing_flowchart-page_1_of_4

Page 2 is the process of using the final dialed digits (and station’s location) to select a route and a trunk group. There are many location and facilities restriction level checks here. And there is even more digit translation that can take place during this process. Once a trunk is selected and the call delivered, this may not be the end of it. Look Ahead Routing may pull the call back and try again.

avaya_call_routing_flowchart-page_2_of_4

Those two pages compose the primary routing logic that you will use in your typical troubleshooting. However, many decisions are based upon the calling party’s location. It may be obvious to you how the CM know’s the station’s location. But if things get strange, you can see how the CM associates a station to a location on page 3.

avaya_call_routing_flowchart-page_3_of_4

And lastly, there is one major headache associated with the “Prefix Mark” in a route pattern. Prior to 1995, there were many, many dialing rules associated with seven digit dialing, one plus seven digits, ten digits, etc. I remember living in a small town in the California Bay Area, and to dial Concord (over the hill), I had to dial 1+seven digits. Then when the area code 415 was split in to 415/510, we could dial 10 digits between them. Unfortunately, if you dialed 1+10 digits, it would be charged as a long distance call. All this went away in 1995 when the North America Numbering Plan was implemented. It is much easier to route calls now. But there is still the dreaded “Prefix Mark” that may still be in your routes. This causes havoc with the dialed-digits that are sent to your carrier. Page 4 is a summary of how this prefix mark works.

avaya_call_routing_flowchart-page_4_of_4

Please note there are two extremely important troubleshooting tools when dealing with Avaya routing. One, is the ‘list ars route-chosen’ and ‘list aar route chosen’ commands, and the other is the ‘list trace station’ and ‘list trace tac’ commands. If you use Avaya Site Administrator, you will not be able to use the ‘list trace’ command. I encourage you to use telnet or ssh in your day-to-day management of your PBX so you can use these commands when necessary. It may not sound like much, but being able to list trace at a moment’s notice is very helpful.

I will save the “list aar/ars route-chosen” and “list trace station/tac” for another posting. There’s a lot of information available in these commands.

Please let me know if you find these flowcharts helpful. Also, if you find any typos please let me know. Please keep in mind there are many more routing decisions the CM makes that are not in these flowcharts. If your CM uses other routing (for example tenants or time-of-day) and you think I should incorporate them into these flowcharts, please let me know! Happy routing everyone. This is really the core of any PBX, so I personally really enjoy working with routing.

Roger

Click Here for the flowchart PDFs

PPM / Feature Buttons not working on Avaya SIP phones

This one was very frustrating. My SIP phones could log into the SBC-E and got dialtone fine, but wouldn’t pull PPM. The Avaya One-X Communicator for iPhone could do it though. Whaaaaa??

I have my phones configured to pull a config file from a web server – it’s the only manual setting required in the phone. All other settings are in the config file on the web. Well, it turns out that the phone didn’t like the SET CONFIG_SERVER w.x.y.z parameter. Once I removed that, then PPM flowed just fine.

By the way, if you have trouble with feature buttons, the other issue I had was the entity link between Session Manager and the SBC. If your’re using TLS connections between the phone and the SBC, then you also need to use TLS on the entity link between the SBC and the Session Manager, and the Session Manager to the Communication Manager. In other words, all connections from the endpoint to the CM need to be TLS. If any of them go back to unencrypted TCP or UDP, then the PPM breaks. At least it did for me!

Unable to log into the Avaya Utility Server as cust or admin on System Platform

I recently had a strange issue with my utility servers. I have four of them on System Platform on the S8300D servers. But I was unable to log into any of them as cust nor admin. I could log into my standalone utility servers, but not the ones on system platform.

It turns our the ldap config was corrupted on the servers and Avaya needed to fix it. Once complete, the tech sent me a nice summary:

If this were to happen again, you can verify if the account works by attempting login to SSH of Dom0 or Cdom or webconsole of Cdom.  If it works there, then we know that LDAP configuration is corrupted and needs to be corrected.  If it happens again, have a ticket open stating that LDAP is corrupted on VUS and that /etc/ldap.conf and /etc/nsswitch.conf need to be fixed.