How to parse Avaya Communication Manager output into useful csv or database tables

Hello all!

This is a follow-up to my post about extracting data from Avaya Communication Manager. In that post, I showed you how to extract any “list”, “display”, or “status” from Avaya Communication Manager to a text file. First, I would like to show you my bash shell script that extracts this data. And then I’ll show you my perl script that converts the output to a text file that you can open with Excel or import into a database.

What information should you pull from Communication Manager? Well, in my case, I have a text file called ‘commands.txt’ that contains these lines:

display time
list ips
list surv
list med
status cdr
status trunk 1
status trunk 2
status trunk 8
status trunk 11
status trunk 16
status trunk 21
status trunk 79
list station
list reg
status station 2291
status station 2292
status station 2293
status station 2294
list call-forwarding
list off s
display time

So those are the various pieces of information that I care about. Looking at that info, those are my active trunk groups, and I want to know the status of four particular extensions (those happen to be the digital ports of my fax server). Also, I have a port network, so I want the status of IPSIs. Anyway, think about all the stuff you care about when you log into your PBX in the morning. And stick them in that “commands.txt” file. I put a “display time” at the beginning and end so I can look at the file later and tell how long it took to run the file.

So we want to run these commands automatically. So let’s create a file called “sanity.sh” and put this in the file:

file=/home/roger/avaya/sanity/data/sanitycheck_`date +"%Y-%m-%d-%H-%M"`.txt
perl /home/roger/avaya/sanity/av.pl /home/ARES/randerson/avaya/sanity/la.pbx /home/roger/avaya/sanity/commands.txt >$file
perl /home/roger/avaya/sanity/sanity.pl $file

Once you save the file, make it executable with

chmod +x sanity.sh

Let’s go over this script line-by-line, okay?

  1. The first line creates a variable for the file that will contain the data. There’s a little linux trick to embed a timestamp in the filename.
  2. The second line runs the file av.pl that I shared with you in a previous post about this script. It’s magical. I cannot take all the credit, but I have modified it for our purposes.
  3. The third line runs a file that parses the output of all the commands. This is a very useful script and I can take full credit for this one. It generates a list of “key/value” pairs to a text file.

My thought process with this script is to create a unique key for every piece of useful data. For example, I want the name, IP address, firmware version, gatekeeper, and network region for every station. I also want the EC500 mapping. And I would like the port number for analog stations. Oh! I also want a sense of low or high priority (Is it bad if the data changes?) and I need to include a site identifier in case I have the same extension in multiple sites. That’s a lot of information to display. And, if I have 1000 stations, it’s too much to display all this information on separate lines. That’s like 7000 lines of data for my stations. Fine for machines to read, but it seemed like too much for humans. I want to put them into one (and in the case of ec500, two) lines of data. I do this by assigning a “Key name” for my stations like this:

Key Name Key Value
lo.la.station.8348.offpbx EC500=>2135552978, OPS=>8348
lo.la.station.8348.status Anderson, Roger (ip=10.10.60.138;reg=3;ver=3.260A;gk=10.2.86.180)

See how much data I squeezed into that? And note that very few stations will have off-pbx information. Most stations will just have that station.status key. The script called sanity.pl takes the raw output from the Communication Manager and converts it to these key/value pairs above. Here is a link to sanity.pl. You’ll need to rename it to a ‘.pl’ file. This should compile just fine. When you copy it to your linux server, try

perl -c sanity.pl

That -c just means to check the syntax and don’t run it. It hopefully says “syntax okay”. Actually, I’m sorry but you’ll need to edit the file and change line 45 to the path where you want your ‘keyvalues.txt’ file, which will contain all your data. This should use a command line param, but it doesn’t. Sorry.

You might consider looking in more detail at this file. It’s where the magic happens and it’s a good framework for parsing other data. If you’re interested, here is a summary of how it works:

  • Starting in line 56, we check the raw text file line-by-line looking for the header for each ‘list’ or ‘display’ or ‘status’ section. For example, when we see the text ‘REGISTERED IP STATIONS’, then we know we are expecting to see the result of a ‘list reg’ (by setting our variable of type to 7).
  • Then on line 183 we actually process the registration status of stations. If type=7, then we are in the ‘list reg’ mode and if a line contains registration information for a station, then we capture the data.
  • Each section has some customization, but you’ll notice that (almost) each section performs a ‘set_data’ of the keys and values appropriate for that section.
  • At the end, the trunk group and station summary is generated and the keys are reported alphabetically.

Here are some examples of what the script does:

  • The script parses ‘list station’ and assumes all stations are unregistered.
  • The script parses ‘list reg’ and fills in the registration status for all stations (anything not parsed in this section has a status of ‘unregistered’)
  • The script parses ‘status trunk xxx’ and simply counts the trunks that are in service (ignoring the “idle/in use” values. We just count trunk members in service
  • The script parses ‘status cdr’ and simply stores the percent full of the buffers
  • The script parses ‘list off s’ and stores the off pbx mappings for all stations
  • The script parses various critical fields for IPSIs, media gateways, and survivable servers
  • The script parses ‘list call-forwarding’ and generates a list of all stations that are call forwarded.

At this point, if you ran the bash script, you would have an awesome list of keys and values in a text file called keyvalues.txt. You probably don’t need to know anything about Perl to get this working for you. But if you do know a bit of Perl, you’ll be able to do amazing things with it.

There’s more!

This data is useless unless you’re looking at it. In a future post, I will show you how to schedule this script with cron, push the data into a database, read it with a simple ‘web site in a single file’ php script, filter the data (including historical values!), and get alerts when the values change.

And then I have something REALLY amazing to show you!

Thanks for reading, everyone! Feel free to contact me at roger@rogerthephoneguy.com or post a comment here. If you need help getting this code working, or you want to tweak it a bit, let me know!

Roger

Roger in Business Insider about my plans to destroy the inbound Windows Support scam

Not to mix business and pleasure, but my “hobby” made the news this week. I was in the top 10 on reddit/r/technology and Business Insider wrote a nice article. Besides my career as a telecommunications consultant, I also build robots that talk to telemarketers. I told the story in a recent TEDx talk, but just last week I got a popup saying that I had a virus on my computer and I needed to call Microsoft support. So I called them back 600 times on 20 simultaneous channels. After about 300 calls, they shut off the number. One down…

Anyway, the article is linked below. If you’re inclined, you can go to JollyRogerTelephone.com and subscribe to the service. You will love telemarketer calls when you use my bots.

http://www.businessinsider.com/telecom-guy-uses-bots-to-foil-microsoft-support-scammers-2017-2

I know I owe you all some scripts to parse the output from Avaya phone systems. Sorry I’m so distracted by this Jolly Roger thing…

Roger

 

How to automatically extract data from an Avaya Communication Manager

You might call this post “the secret of my success” as a PBX admin for Avaya Communication Manager. Avaya provides some utilities for their techs and business partners, but customers don’t really have a lot of tools to easily extract data. In this post, I’ll show you how to extract ANYTHING from Avaya from a command line so it’s suitable for scripting. In my case, I parse the data further and generate a list of all stations and report of changes to my PBX every 15 minutes. Have you ever wondered who used to have extension 4438? How about that phone for the intern over the summer named “Bodkin Van Horn” that you deleted but now he has been hired and you want to give him the same extension? How about the question “how long has this extension been unplugged”?

Back in mid 00s, I wrote some PHP integrate with CM using telnet. Since I didn’t know much about PHP’s ANSI support, I ended up writing my own ANSI parser of the raw data. Anyway, it was great, but many customers don’t support telnet and want all communication to CM to be through ssh, which is reasonable, right?

Then I found this post from Benjamin Roy where he provided a perl module to do it through ssh. Also, he meticulously reverse-engineered Avaya’s OSSI protocol, which I think is impressive and sounds like fun. But I didn’t use OSSI; I just wanted a way to get data using ssh. So I took Ben’s code and tweaked it a little bit. It’s straight perl code – it’s not a module or anything like that. It does have dependencies on a couple other modules though. You do not need to be a perl developer to use this. It’s really easy. Here is a link to the script. Just save this as “av.pl”. Try to run it with “perl -c av.pl” (the -c means just check to make sure it compiles but do not run it). You might get some errors about missing modules. There are only three, and this is how you install them if you need to:

perl -MCPAN -e "install Expect"
perl -MCPAN -e "install Data::Dumper"
perl -MCPAN -e "install Term::VT102"

I should probably mention here – if you don’t have a Linux server in your environment, you should really get one. Your Virtual Machine team can make one for you – just about any distribution of Lunux will do and they take almost no resources. If you need to justify it, just say something like “utility server for PBX monitoring” or something like that.

Anyway, back to the script. Naturally, it needs to connect to your PBX, so create a little text file like “mypbx.txt” with the contents

10.10.40.89,5022,cmusername,Passw0rd,ssh

Obviously, replace the IP address, username, and password of your PBX.

And then create another file called “commands.txt” with the contents

list trunk
display station 3100

This can contain whatever commands you’d like. list reg, list station, list trunk, list locations, list measurements, stat cdr, etc. It should be any command that can be “paged” through, you know?

And then simply run this line

perl av.pl mypbx.txt commands.txt

The script will connect to the PBX defined in mypbx.txt, run the commands in commands.txt and output the results to STDOUT!

This is the core script to a bunch of stuff I automate in my PBXs. In a later post, I’ll show you how I parse the results with Perl and make use of the data.

I call this “the secret of my success” because I use this data to build a simple html page of all extensions in all of my PBXs every 15 minutes. I keep this page up all the time in a browser tab, which enables me to simply “control-f” for any name in the PBX and I have the extension number and the IP address of the station (or ‘unregistered’ if it’s unplugged). I can see unregistered ‘guest’ numbers, see the station’s history, etc. There’s a lot more I do with this information and I look forward to showing you.

Please let me know what you think! And if there’s anything I can do to help you get it working.

Lastly, if any of you need to do any telephone system testing (capacity, QA, DID number ports, call-flow, queuing, etc), please check out my post about CallsByCloud. If you sign up and use the promotion code ‘roger2015’, you’ll get $10 in credit and a 20% discount on the per-minute rate.

Also, if you hate unsolicited outbound telemarketing, consider subscribing to the Jolly Roger Telephone Company. Your telemarketers will entertain and delight you. Sample recordings here.

Thanks all!

Roger

 

Roger speaks at TEDx Naperville about disrupting the telemarketing industry!

Way back in July of 2014, I wrote about how I stopped telemarketing at my house. Little did I know that my efforts to combat and disrupt unsolicited telemarketers would eventually lead to me giving a TEDx talk about it!

Some of you may know that, in an effort to stop telemarketing, I created a robot to talk to them. It’s an interesting story and I posted some audio at the beginning of 2016. My story was picked up by Gizmodo on February 1, 2016 and things got really crazy from there.

Anyway, Fast forward to November 4, 2016 and I find myself on stage at TEDxNaperville in Illinois speaking in front of about 500 people at a beautiful venue. I’ve never spoken in front of a crowd like that, and it was amazing.

Don’t get me wrong – as a phone guy, I love call centers. But as a phone guy I also love the telephone network. The Public Switched Telephone Network has been around for about 140 years and it is crushed under the burden of unsolicited telemarketing. Blocking doesn’t work. Their predictive dialers simply call the next sucker. I think the only solution (short of abandoning the entire network) is the Jolly Roger Telephone Company. Here is me story about how I’m disrupting the scammers, spammers, and crooks, which make up most of the unsolicited telemarketing industry.

By the way, this goes for cold callers too. If your business is struggling to handle cold callers (ask your receptionists, Investor Relations team, or any admins for your CxOs) then I also have a solution for you. It’s called a Biz-Bot and it will handle all your pesky cold callers for all your employees.

But anyway, here is the talk!

How to parse Avaya CDR into MySQL database with Perl

This is a follow-up to my earlier post about setting up CDR processing with Kiwi Syslog Server. In that post, I discuss how to capture the data to a text file. But of course, if we are capturing CDR, we probably want to analyze it, right?

So this post is dedicated to capturing the data to a MySQL database. It is not for the faint-of-heart though. It requires a little knowledge of MySQL and willingness to read a tiny bit of perl. I am happy to help you set this up, though. Either via this blog as time allows, or I would love to work for you as an hourly contractor – and I love all things telecom!

If you don’t have a Linux server in your network that you can use for your own purposes, I highly recommend one. Any Linux distribution will do, and you probably have a VMWare infrastructure to support it. If you need to justify it, I can help – perhaps even this blog entry will help. Tell your boss you need to perform text analysis of your PBX data. This blog entry assumes you have Linux, but I suppose it will work on Windows with a LAMP stack (but good luck with local firewall rules in Windows).

So first. you must create a MySQL database to store the call records. Let me know if you need help doing this. Since I have to start somewhere with this post, I will assume you have access to a MySQL database. You just need to create a table to store the data. Here is the statement to do that:

drop table if exists cdr_syslog;
create table cdr_syslog (
 id integer auto_increment primary key,
 method varchar(8),
 reported_on datetime,
 deduced_starttime datetime,
 source_ip varchar(32),
 from_number varchar(24),
 to_number varchar(24),
 call_type varchar(12),
 call_time varchar(16),
 direction varchar(3),
 duration_ss integer,
 call_hash varchar(64),
 pbx_time_of_day_hh varchar(2),
 pbx_time_of_day_mm varchar(2),
 pbx_duration_h varchar(1),
 pbx_duration_mm varchar(2),
 pbx_duration_6s varchar(1),
 pbx_condition_code varchar(1),
 pbx_access_code_dialed varchar(4),
 pbx_access_code_used varchar(4),
 pbx_dialed_number varchar(23),
 pbx_calling_number varchar(14),
 pbx_account_code varchar(15),
 pbx_authorization_code varchar(7),
 pbx_frl varchar(1),
 pbx_incoming_ckt varchar(3),
 pbx_outgoing_ckt varchar(3),
 pbx_feature_flag varchar(1),
 pbx_attendant_console varchar(4),
 pbx_incoming_tac varchar(4),
 pbx_node_number varchar(2),
 pbx_ins varchar(5),
 pbx_ixc varchar(3),
 pbx_bcc varchar(1),
 pbx_ma_uui varchar(1),
 pbx_resource_flag varchar(1),
 pbx_packet_count varchar(5),
 source_line varchar(128),
 created_on datetime
);

alter table cdr_syslog add index i_call_hash(call_hash);
alter table cdr_syslog add index i_deduced_starttime(deduced_starttime);
alter table cdr_syslog add index i_from_number(from_number);
alter table cdr_syslog add index i_to_number(to_number);

Those statements create the table and a few indexes to speed things up. The indexes for call_hash and deduced_starttime are used when inserting records. The indexes for from_number and to_number are added since these are probably the most common fields to query. Typically, I will add an index for any field that appears in my ‘where’ clause, but I’m no DBA so there are probably better ways to tune it. However, this works fine for me and should for you as well.

I am using a Kiwi Syslog server to capture the CDR. But the cool thing about Kiwi is that you can “forward” the message along to another server. So I configured mine to send it to my Linux server. This is the screenshot for that forward command within Kiwi:

kiwi-send-logs-out-via-syslog

This simply sends the Syslog message to my server on UDP port 514. This is the typical Syslog port. Then I have a Perl script running on my server. That Perl script has some dependencies

use strict;
use DBI;
use Net::Syslogd;
use Digest::MD5 qw(md5_hex);
use POSIX qw(setsid);

You don’t need to be familiar with Perl to run this script, but you will need to make sure these packages are installed. For example, if I have a script that runs this line:

use Roger::Wilco;

When I try to run the script, I will get this error:

Can't locate Roger/Wilco.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at syslog.pl line 2.
BEGIN failed--compilation aborted at syslog.pl line 2.

So if you try to run my CDR script and you get any kind of “Can’t Locate” error, then you need to install that package. To to this, type this at a command line:

Perl -MCPAN -e shell

And that will get you to the CPAN command line where you can install packages.

# perl -MCPAN -e shell
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.9402)
Enter 'h' for help.

cpan[1]>

And from here, you can type

install Net::Syslogd

Or whatever other packages you need to install. Note that case matters, and that’s a double colon between the words. When it asks you a yes/no question, say yes. If all goes well, then it should install fine. CPAN installs all dependencies. If you get errors, the most common is you don’t have a c++ compiler on your machine. I never did figure out how to fix that in Windows. In Linux, there are lots of tutorials. With that in mind, here is the Perl script that will catch the CDR forwarded from Kiwi. Be sure to read my comments at the end.

#!/usr/bin/perl

### Read unformatted CDR data from an Avaya server
### This script uses Net::Syslogd to listen to standard syslog messages
### The Avaya phone systems send to TCP ports. Unfortunately, this script is to listen from 
### A 'proxy' instance of CDR. for example, a Kiwi Syslog Server that forwards syslog messages
### out to another server.
### see my blog http://rogerthephoneguy.com/?p=404 for details


### Usage!
### set the $dir folder to where you store text files from your syslog server
### and run this script with 'perl cdrd.pl file' to tell the script to process files
### or run as 'perl cdrd.pl' to just watch the processing fly by
### or run as 'perl cdrd.pl -d' as a daemon process
### to stop the daemon, just create a file matching $killfile
### Or do a 'ps -ef |grep cdrd.pl to find the PID
### and kill -9 that pid
### not the best way to manage daemons, but we are telephone engineers here, not linux admins, right?

use strict;
use DBI;
use Net::Syslogd;
use Digest::MD5 qw(md5_hex);
use POSIX qw(setsid);

my $db;
my $dir = "/home/roger/avaya/cdr/data/";
my $killfile = "/home/roger/avaya/cdr/cdrd.end";


sub db_open {
        my $dsn = "DBI:mysql:dbname_ie_avaya:localhost";
        my $user = "dbuser";
        my $pass = "dbpassword";
        while (!($db)) {
                $db = DBI->connect($dsn, $user, $pass);
        }
}

sub db_close {
        if($db) {
                $db->disconnect();
        }
        undef($db);
}

sub sql { #check
        my ($sql) = (@_);
        my $sth = $db->prepare($sql);
        if(!($sth->execute())) {
                print "WARNING - unable to execute $sql\n";
        }
        $sth->finish();
}

sub generic_create {
        my ($table, $fields, $values) = (@_);
        my $sql = "insert into $table ($fields) values ($values)";
        my $sth = $db->prepare($sql);
        if(!($sth->execute())) {
                print "WARNING - unable to insert with $sql\n";
        }
        $sth->finish();
}

sub generic_read {
        my ($select, $table, $where) = (@_);
        my @datarow;
        my $sql = "select $select from $table where $where limit 1";
        my $sth = $db->prepare($sql);
        if($sth->execute()) {
                @datarow = $sth->fetchrow_array();
        } else {
                print "unable to execute $sql\n";
        }
        $sth->finish();
        return $datarow[0];
}

sub one {
        my ($sql) = (@_);
        my @datarow;
        my $sth = $db->prepare($sql);
        if($sth->execute()) {
                @datarow = $sth->fetchrow_array();
        } else {
                print "unable to execute $sql\n";
        }
        $sth->finish();
        return $datarow[0];
}

sub process_cdr_line {
    my $cdr = shift;
    chomp($cdr);
    my ($logtime,$facsev,$remoteaddr,$message) = split(/\t/,$cdr);
    my $md5message = md5_hex($message);
    #print "012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789\n$message";
    #here are the start and length of each data point within the message of an "unformatted" cdr line
    #Nobody knows what some of these are. It's a mystery today
    #your condition codes may be different, so you should watch your data closely for several days
    my @parses = (
        0,2,     #time of day hours
        2,2,     #time of day minutes
        4,1,     #duration hours
        5,2,     #duration minutes
        7,1,     #duration tenths
        8,1,     #condition code
        9,4,     #access code dialed
        13,4,    #access code used 
        17,15,     #dialed number
        32,10,     #calling number
        42,15,     #account code
        57,7,     #auth code
        66,1,     #frl
        67,3,     #incoming ckt
        70,3,     #outgoung ckt
        73,1,     #feature flag
        70,4,     #attendant console
        76,4,     #incoming tac
        82,2,     #node number
        84,4,     #INS
        88,3,     #IXC
        92,1,     #BCC
        93,1,     #MAUUI
        94,1,    #Resource flag
        95,4    #packet count
    );
    my $values = '';
    my $insert;
    my $i;
    my $calltime;
    my $duration;
    my $direction = 'unk';
    my $dialed;
    my $calling;
    my $node = '01';
    my $condition;
    for($i=0; $i<25; $i++) { my $start = $parses[$i*2]; my $length = $parses[$i*2+1]; my $part = substr($message,$start,$length); #print "this $i from $start with length $length is '$part'\n"; $part =~ s/^ *//g; $part =~ s/ *$//g; if($i == 0) { #time of day hh $calltime = "$part:"; } elsif($i == 1) { #time of day mm $calltime .= $part; } elsif($i == 2) { #duration h $duration = $part * 60 * 60; } elsif($i == 3) { #duration mm $duration += ($part * 60); } elsif($i == 4) { #duration 1/10 min (6 sec) $duration += ($part * 6); } elsif($i == 5) { #condition code $condition = $part; if($part == 9) { $direction = "in"; } elsif($part == 7) { $direction = "out"; } } elsif($i == 8) { #dialed number $dialed = $part; $dialed =~ s/E$//; if(($dialed ne '6226') && ($dialed =~ /^6[234]\d\d$/)) { $node = '44'; } } elsif($i == 9) { #calling number $calling = $part; $calling =~ s/E$//; if(($dialed ne '6226') && ($dialed =~ /^6[234]\d\d$/)) { $node = '44'; } } elsif($i == 17) { #incoming tac if($part =~ /^#0\d\d/) { $node = '44'; } } elsif($i == 18) { #node number $part = $node; } $values .= "'$part',"; } if($direction eq "in") { my $temp = $dialed; $dialed=$calling; $calling=$temp; } if(($dialed =~ /^\d*$/) && ($calling =~ /^\d*$/)) { #dialed_name and calling_name are filled from a 'display_name' table that is not part of this demo. #for more information, contact me through RogerThePhoneGuy.com and we can chat about how this table is filled my $dialed_name=''; #generic_read("name", "avaya_display_name","station='$dialed'"); my $calling_name=''; #generic_read("name", "avaya_display_name","station='$calling'"); $dialed_name =~ s/'/\\'/g; $calling_name =~ s/'/\\'/g; #print "logtime:$logtime\ncondition $condition\ndirection $direction\ndialed $dialed ($dialed_name)\ncalling $calling ($calling_name)\nduration $duration\n"; $insert = "insert into cdr_syslog values (null,'syslog','$logtime',null,'$remoteaddr','$calling','$calling_name','$dialed','$dialed_name','type','$calltime','$direction',$duration,'$md5message',"; $insert .= "$values '$message',now())"; my $check = one("select count(*) from cdr_syslog where call_hash = '$md5message'"); #print "hash check = $check\n"; if($check == 0) { #We insert the md5 of this line, to prevent double-logging. This allows us to re-process the cdr as much as necessary w/o dups in the database #print "$insert\n"; sql($insert); sql("update cdr_syslog set deduced_starttime = DATE_SUB(reported_on,INTERVAL duration_ss SECOND) where call_hash='$md5message'"); } else { #print "duplicate\n"; } } else { $insert = "insert into pbx_syslog values (null,'$message')"; #sql($insert); } } sub daemonize { #check open STDIN, '/dev/null' or die "Can't read /dev/null: $!"; open STDOUT, '>>/dev/null' or die "Can't write to log: $!";
    open STDERR, '>>/dev/null' or die "Can't write to log: $!";
    defined(my $pid = fork)   or die "Can't fork: $!";
    exit if $pid;
    setsid                    or die "Can't start a new session: $!";
    umask 0;
}

if($ARGV[0] eq "-d") {
    daemonize;
}

db_open();

if($ARGV[0] eq "file") {
    opendir(DH, $dir);
    my @files = readdir(DH);
    closedir(DH);
    foreach my $file (@files) {
        open CDR, "$dir$file";
        my $i = 0;
        while (my $line = ) {
            $i++;
            my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday) = localtime(time);$mon++;$year+=1900;
            $line =~ s/[^!-~\s]//g; #strips unprintable
            process_cdr_line($line);
            #die if($i>2);
        }
        print "$dir$file - $i lines\n";
        close CDR;
        unlink "$dir$file";
    }

} else {
    my $syslogd = Net::Syslogd->new()
      or die "Error creating Syslogd listener: ", Net::Syslogd->error;

    while (!(-e $killfile)) {
        my $message = $syslogd->get_message();
        if (!defined($message)) {
            printf "$0: %s\n", Net::Syslogd->error;
            exit 1;
        } elsif ($message == 0) {
            next
        }

        if (!defined($message->process_message())) {
            printf "$0: %s\n", Net::Syslogd->error
        } else {
            my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday) = localtime(time);$mon++;$year+=1900;
            my $logtime = sprintf("%04d-%02d-%02d %02d:%02d:%02d",$year,$mon,$mday,$hour,$min,$sec);
            my $remoteaddr = $message->remoteaddr;
            my $remoteport = $message->remoteport;
            my $facility = $message->facility;
            my $severity = $message->severity;
            my $time = $message->time;
            my $hostname = $message->hostname;
            my $msg = $message->message;
            my $line = "$logtime\t$facility.$severity\t$remoteaddr\t$msg\n";
            print $line;
            #now $line looks like any line from the CDR files
            #so this next part works via syslog or files
            $line =~ s/[^!-~\s]//g; #strips unprintable
            process_cdr_line($line);
        }
    }
    unlink $killfile;
}

db_close();

There’s a lot in that script. Things you must know:

  1. You’ll need to replace the database name, username, and password in the “db_open” subroutine.
  2. The comments at the top explain how to run it to process files, or to capture syslog live, or even how to run as a daemon.
  3. Each CDR line is hashed and included in the record, so don’t worry about processing the CDR files multiple times. It will NOT insert duplicate records.
  4. When processing files, the script will delete the file when done. So always copy the CDR files into a separate folder before running this script.
  5. I kludged this together from a working Perl script. I removed some identifying information, moved some variables around, and added a bunch of comments. If you get any errors, let me know – it’ll probably be really easy to fix.
  6. For any hobbyist Perl developers out there, this script includes some handy database functions that insert records, return rows, and return one record.
  7. This script is for information purposes only. Naturally, I don’t provide any warranties, guarantees, or assertions that it will work with your systems. Run it at your own risk.
  8. If you’d like me to help set this up, I’d love to work for you. I love telecom and I’m always available for hire as a consultant. I would LOVE to be your telephone guy!

If you’re planning on trying this, then I’m really flattered. Let me know how it goes. Mine has been running for about two years and I cannot tell you how handy it is to have that much CDR data in a MySQL table. I have a PHP script that I use to analyze the data. Do you want to see that?

Thanks all,

Roger

 

Avaya may be filing Chapter 11?

http://www.nojitter.com/post/240172156/avaya-ready-to-sell-off-contact-center-business

So Avaya may be selling off their contact center business and filing for Chapter 11 protection?

What does this mean to Avaya customers? Those of you who had Nortel and later became “Avaya Blue” customers went through this. Siemens ICN kind-of went through this as they changed to “Unify” and were bought a couple times.

Cisco has a huge advantage in enterprise telephony. They enjoy a great relationship with the network managers and can price their telephony solutions well under market since much of the infrastructure is already in place.

Anyway, some of you readers out there are Avaya customers, and some of you have gone through this before with Nortel. What do you think this means?

I loved the Nortel Option 11.

 

Avaya Call Detail Recording to Kiwi syslog server

In a previous post about busy/release of CDR, I asked if anyone was curious about capturing CDR from Avaya Communication Manager using a Kiwi Syslog server. At least one of you was, so here it is.

There are two parts of course. One part is programming the phone system to send CDR. And the other part is programming Kiwi to capture CDR. These can be done in either order, but if you program the phone system first, the CDR buffer will fill as is tries to establish the connection with Syslog.

So let’s start with Kiwi. Your version may differ. In my case, the network team already had a rather powerful installation of Kiwi on some great hardware. All I had to do was configure it to capture Avaya CDR. So when you launch the Kiwi dashboard, click File->Setup and in the “Server Setup” window, scroll to the bottom for “Inputs”.

kiwi-config-0

In the TCP section, tell Kiwi to listen for TCP messages on whatever port you wish. I’m using 5013 – probably because I saw it in an Avaya doc somewhere. But you can use any port.

Then scroll up to the “Rules” section and configure a new rule to match for messages from your Avaya Communication Manager.

kiwi-config-1

In my case, I named my rule “Call Records from LA PBX”. I set a filter by IP address. If you also send any other information to Kiwi from the CMs, then you’ll want to filter by the Priority field. You want the Local7 field to be Debug. I don’t recall why this works. I think I was looking at the data one day to try to separate the CDR from the other syslog messages from Avaya and this seemed the easiest way to do it.

kiwi-config-3

Once you have those two filters – IP address and Local7.Debug, you need to create an action. My action above is a little misleading. All you really need is one that logs to a file, then stops processing. If you’d like, you can also display it to one of the virtual displays in Kiwi. I log to a text file on the local file system:

kiwi-config-2

Now, if we program our PBX correctly, we should get CDR to these text files!

Step 2 – Program the Avaya Communication Manager

Programming the Communication Manager is three steps: Add the node-name, set the ip-services, and change the system-parameters cdr.

To set the node name, ‘change node-name ip’ and create an entry for the IP address of your Kiwi server. This is basically the DNS lookup for the CM. This sets the name and IP address. Any name will do. Enter the IP address of your Kiwi server.

change node-names ip                                            Page   1 of   2
                                  IP NODE NAMES
    Name              IP Address
SL_LSP              10.10.60.10
CHA_LSP             10.10.1.10
Chicago             10.10.148.2
Chicago_LSP         10.30.10.10
DO_LSP              10.20.16.10
Smoot               10.87.176.11
Gateway006          10.46.12.1
LA                  10.10.19.6
LA_ESS              10.44.192.10
LE_ESS              10.58.200.10
LA_SM               10.59.29.16
( 16 of 44   administered node-names were displayed )
Use 'list node-names' command to see all the administered node-names
Use 'change node-names ip xxx' to change a node-name 'xxx' or add a node-name

Next, set the ip-services. I have squeezed all three screens here. The important thing here is page three. We need to set the “Reliable Protocol” to n. I think Avaya defaults this to y and it works great for their pre-packaged CDR partners. For home-grown syslog, we set this to n. It may not hold up in court if there’s ever a dispute.

change ip-services                                              Page   1 of   3

                                   IP SERVICES
 Service     Enabled     Local        Local       Remote      Remote
  Type                   Node         Port        Node        Port
CDR1                 procr            0       la_syslog       5013
CDR2                 procr            0       prognosis       50000

change ip-services                                              Page   2 of   3

                                   IP SERVICES
 Service     Enabled     Local        Local       Remote      Remote
  Type                   Node         Port        Node        Port


change ip-services                                              Page   3 of   3

                              SESSION LAYER TIMERS
  Service     Reliable  Packet Resp   Session Connect  SPDU  Connectivity
   Type       Protocol     Timer       Message Cntr    Cntr     Timer

 CDR1            n         10                3          3         10
 CDR2            n         10                3          3         10


And last step – set the system-parameters for CDR. Here’s the thing. You have lots of options here. You can define fields, separators, etc. But I discovered that there is a line length limit that is not obvious. Also, the caller-id name is surprisingly absent. Rather than customize the CDR, I have gone back to “unformatted” and let further processing figure it out.

change system-parameters cdr                                    Page   1 of   1
                            CDR SYSTEM PARAMETERS

 Node Number (Local PBX ID): 2                     CDR Date Format: month/day
      Primary Output Format: unformatted   Primary Output Endpoint: CDR1
    Secondary Output Format: unformatted Secondary Output Endpoint: CDR2
           Use ISDN Layouts? n                   Enable CDR Storage on Disk? n
       Use Enhanced Formats? n      Condition Code 'T' For Redirected Calls? n
      Use Legacy CDR Formats? y                 Remove # From Called Number? n
Modified Circuit ID Display? n                             Intra-switch CDR? y
                  Record Outgoing Calls Only? n     Outg Trk Call Splitting? y
  Suppress CDR for Ineffective Call Attempts? n       Outg Attd Call Record? y
      Disconnect Information in Place of FRL? y      Interworking Feat-flag? n
 Force Entry of Acct Code for Calls Marked on Toll Analysis Form? n
                                    Calls to Hunt Group - Record: member-ext
Record Called Vector Directory Number Instead of Group or Member? n

     Inc Trk Call Splitting? n
  Record Non-Call-Assoc TSC? n           Call Record Handling Option: warning
      Record Call-Assoc TSC? n   Digits to Record for Outgoing Calls: dialed
   Privacy - Digits to Hide: 0               CDR Account Code Length: 15
Remove '+' from SIP Numbers? y


I have a perl script that parses unformatted cdr. Anyone want to see it?

Anyway, when you have this set up, you can status CDR to make sure the link is up:

status cdr-link
                                CDR LINK STATUS
                   Primary                      Secondary

       Link State: up                           up

      Date & Time: 2016/04/17 08:43:29          2016/05/09 10:12:20
  Forward Seq. No: 0                            0
 Backward Seq. No: 0                            0
CDR Buffer % Full:   0.00                         0.00
      Reason Code: OK                           OK





The good news here is the link state(s) are “up”  and the CDR buffer % full is 0. If you have trouble, you can “busy cdr primary” and “release cdr primary” to close and open the socket. You should see a blip in the Kiwi server when this happens.

Make a call and watch Kiwi! You may need to check the log files rather than rely on the Kiwi display. The extra CR/LF will sometimes make Kiwi look funky but the text file is usually just fine.

Let me know how it goes for you!

Roger

 

Avaya phone reboots, and then doesn’t have dialtone

I have had this problem in three offices starting in 2013. And it just happened to me today in an overseas office. This problem can remain hidden in your telephone system for years and not be a problem because it’s so rare for a phone to get assigned a new IP address via DHCP.

Basically, a phone reboots for some reason. When it comes back up, there’s no dialtone. Or, if you have more than one gateway,  you might get intermittent dialtone.

In my network, these items were also true:

  1. The gateway and the phones are on the same subnet
  2. The phone was assigned a different IP address when it rebooted (this may not be obvious since you don’t always know the phone’s previous IP address)

If you have more than one gateway, the problem might seen intermittent. you can “list trace station” and watch the phone pull dialtone:

12:19:49   G711A ss:off ps:20
           rgn:1 [10.9.10.90]:39798
           rgn:1 [10.9.10.3]:2094

That IP address on the third line – that’s the gateway assigned to serve dialtone for this call. In my case, I have a gateway in the local office, and a gateway in the datacenter. Whenever the user said “hey, there’s no dialtone!”, it had drawn a resource from the local gateway in the same subnet. The phone worked in all other ways – there was just no media.

There are MILLIONS of Avaya forum postings regarding no dialtone, one-way audio, etc. In my case here, the problem was caused by the arp cache in the G450 gateway. That’s pretty obscure. The IP address changed, but the MAC address in the arp cache did not change. So the gateway is trying to feed dialtone to the wrong MAC address.

The fix was simple: log into the G450 and issue these commands:

  • no ip arp inspection‘ (this disables arp caching)
  • clear arp-cache‘ (this clears the arp-cache, which sounds scary but will not affect service)
  • copy run start‘ (this will write the settings to flash memory to survive a reboot)

That fixed the issue of no dialtone. It did not fix the issue of phones rebooting and getting different IP addresses though. I will write a different post for that, since it was a completely unrelated issue.

I hope this helps! Please let me know if this works for you.

Roger

How to load-test your telephone system, IVR, or DID number port

When you work on telephone systems, at some point you need to make some test calls into your trunks. And sometimes it’s not realistic to use the same phone at your desk, since the call may not truly leave your PBX or your carrier’s network. So you end up using your mobile phone,or if you’re remote, then your home phone. However,

  • What if you need to make dozens of calls?
  • What if you need to load up a trunk group?
  • What if you need to test a number port of hundreds of individual DIDs?
  • What if you need to simulate calls from different areas of the country?
  • What if you need to dial an extremely specific sequence of digits once connected, and you need to do this over-and-over-and-over?

About a year ago, I found a great tool called CallsByCloud.com. This is an automated testing framework – completely self service using your browser. I liked it so much I became a business partner recommend it highly to all of you telecom administrators. I even recorded the training videos for them.

It’s very straight-forward to use. You build a “test scenario” that contain simple “scripts” that are use cases for each type of call. If you’re testing a call center IVR, this could be “make payment” or “renew contract” or “queue for customer service”. And if you’re just testing a trunk group, you can dial into your PBX and “hold” the channel for x seconds. This lets you load the trunk group and test overflow.

cbc-script-commands

Once the tests are complete, you can see the call history, channel usage, a spectrogram of the call, and you can listen to or download the audio.

cbc-history

Once you set up a test, you can schedule it to run at a recurring time and alert you if there are any problems with the call.

cbc-qa-schedule

There are other advanced features such as

  • Caller-id manipulation to simulate geographic routing.
  • Detailed logs of the call processing.
  • Number pooling for testing multiple numbers.
  • The ability to set a “disposition” or “goto” a different spot in the script based upon the timing of noise, silence, or DTMF in the channel.
  • The ability to export call records for the tests.

It’s flexible, cost effective, and has made my job so much easier. I recommend it highly to all of you. If you sign up and use the promotion code ‘roger2015’, you’ll get $10 in credit when you sign up and a 20% discount on the per-minute rate.

Check out the training videos and let me know what you think!

Thanks for watching!

Roger

 

How to redirect Avaya phones to different config files based upon source IP address

Hello everyone. I had a situation a while back where I needed to tell the 9630 phones in a remote office about a new call server. It was after-hours and I could not reach the network administrator responsible for the DHCP option strings for that site. I ended up creating a rewrite rule in Apache to redirect these phones to a different config file where I could overwrite the call server settings. Later I did the same thing for an international site where their date/time format was different from everyone else’s.

You can also use the GROUP setting in the phone to load specific config information, but as one reader discovered, the GROUP settings are not available to SIP phones! If you’re using Apache to host your config files, here is how to re-direct those phones to custom files.

If you are using defaults, your Apache config file is probably at /etc/httpd/conf/httpd.conf, and your document root for your web server is probably /var/www/html/

So open /etc/httpd/conf/httpd.conf in your favorite editor and find the section

<Directory "/var/www/html">
...buncha stuff
</Directory>

Inside the <Directory> </Directory> tags, add these lines:

    RewriteEngine On
    RewriteCond %{REMOTE_ADDR}= 10.101.
    RewriteRule 46xxsettings.txt SPAIN_46xxsettings.txt
    RewriteCond %{REMOTE_ADDR}= 10.139.
    RewriteRule 46xxsettings.txt OHIO_46xxsettings.txt

This will turn on the Apache rewrite engine and if the remote IP address matches the pattern 10.101, then any request for 46xxsettings.txt will be rewritten to SPAIN_46xxsettings.txt. Likewise, anything from 10.139 will get OHIO_46xxsettings.txt

As I look a this, the rewrite condition should probably start with a carrot ^ to try to match the beginning of the line, but it’s working for me so I don’t want to fiddle. But if I have a phone from 10.210.139.5, this rewrite will probably kick in because of that 10.139. within the IP address. Putting a carrot at the beginning should force it to only match addresses that begin with 10.139.

Unfortunately, you’ll need to restart Apache, but the phones won’t care if they miss a PUT.

[root@phonewebserver httpd]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
[root@phonewebserver httpd]#

Mine takes less than a second. I don’t think anything noticed.

So then just create your SPAIN and OHIO settings files and the phone should get those. If it doesn’t seem to work, here are some troubleshooting steps.

Make sure the rewrite module is loading in Apache

Check the httpd.conf file and make sure there is a line to actually load the rewrite module. Mine is on line 190:

LoadModule rewrite_module modules/mod_rewrite.so

Tail the Apache access_log to confirm the IP addresses and GET requests from the phones

In a busy environment, it is not helpful to just tail the access log. But you can tail and grep at the same time! When troubleshooting, I just want to see the GET requests, so enter this command:

tail -f /var/log/httpd/access_log|grep --line-buffered GET

And that will follow the log but only show GET requests. You could also

tail -f /var/log/httpd/access_log|grep --line-buffered 10.139

And this will follow the log but only show activity from the 10.139 subnet

Temporarily enable the rewrite log so you can see what is getting rewritten

Just before the <Directory> tag in Apache, enable the rewrite log with

RewriteLog "/var/log/httpd/rewrite.log"
RewriteLogLevel 4
<Directory "/var/www/html">

You’ll need to restart Apache again. There should be enough information in that rewrite.log to figure out what is happening. It’s a lot of information, so be sure to set the RewriteLogLevel to 0 when you’re done troubleshooting.

I hope this helps! Please let me know how your testing is going. It would also be helpful to know if any of you get your GROUP settings to work over SIP. Thanks all!