Domain Controller Performance Monitoring

These are some counters to check in PerfMon against your Active Directory Domain Controllers. These counters should be checked in addition to the usual CPU, memory, networking, etc. counters.


NTDS Counters

Directory Services

DS Threads In Use: This counter shows the number of threads in use by Active Directory, with a lack of activity typically pointing to network problems that are preventing client requests from succeeding.

Kerberos Authentications/Sec: This counter shows the number of Kerberos authentications on the server per second. A lack of activity can indicate network problems that are preventing authentication requests from succeeding.

NTLM Authentications: This counter shows the number of NTLM authentications per second handled by the domain controller (from Windows 98 and Windows NT clients). A lack of activity points to network problems.

LDAP

LDAP Bind Time: This counter shows the time required for completion of the last LDAP binding, with a higher value pointing to either hardware or network performance problems.

LDAP Client Sessions: This counter shows the number of connected LDAP client sessions, with a lack of activity pointing to network problems.

LDAP Searches/Sec: This counter shows the number of LDAP searches per second performed by clients in the directory. A lack of activity points to network problems.

LDAP Successful Binds/Sec: This counter shows the number of successful LDAP binds per second, with a lack of activity pointing to network problems.

Replication

DRA Inbound Bytes Total/Sec: This counter shows total bytes received through replication per second. Lack of activity indicates that the network is slowing down replication.

DRA Inbound Object Updates Remaining in Packet: This counter shows the number of object updates received for replication that have not yet been applied to the local server. The value should be low, with a higher value indicating that the hardware is incapable of adequately servicing replication (warranting a server upgrade).

DRA Outbound Bytes Total/Sec: This counter shows the total bytes sent per second. Lack of activity indicates that the hardware or network is slowing down replication.

DRA Pending Replication Synchronizations: This counter indicates the replication backlog on the server. This value should be low, with a higher value indicating that the hardware is not adequately servicing replication.


Database Counters

Cache % Hit: This counter shows the percentage of database page requests handled by the cache, thereby not causing a file I/O. A lack of activity can indicate that the server has insufficient physical memory.

Cache Page Fault Stalls/Sec: This counter shows the number of page faults per second that go unserviced due to lack of available pages in the database cache. A value other than zero indicates insufficient physical memory in the server.

Cache Page Faults/Sec: This counter shows the number of page requests per second that cause the database cache to allocate new pages from the cache. This value should be low, with a higher value indicating insufficient physical memory in the server.

File Operations Pending: This counter shows the number of file operations for the database file(s) currently pending by the operating system. The value should be low, with a higher value indicating insufficient physical memory and/or inadequate CPU availability or performance.

File Operations/Sec: This counter shows the number of file operations per second generated by the database cache manager against the database files. The value should be low, with a higher value indicating inadequate physical memory in the server.

Log Record Stalls/Sec:This counter shows the number of log records per second that could not be added to the log buffers because the buffers were full. The value should be zero or close to zero, with a higher value indicating inadequate physical memory in the server.

Log Threads Waiting: This counter indicates the number of threads waiting on pending log writes. The value should be low, with a higher value indicating insufficient physical memory, poor disk performance, or poor disk structuring.

Table Open Cache Hits/Sec: This counter shows the number of directory database tables open per second from the cache. A high value indicates better caching, with a lower value typically indicating inadequate physical memory in the server.

Cacti Backup

Here is a Cacti backup script that I found the book “Cacti 0.8 Beginner’s Guide.” It’s really handy if you’ve gone through all the trouble to install and configure the PA.

It seems to work, but I haven’t done a restore yet.

#!/bin/bash#Set Variables
DATE=`date +%Y%m%d`
FILENAME=”cacti_database_$DATE.sql”;
BACKUPDIR=”/backup/”;
TGZFILENAME=”cacti_files_$DATE.tgz”;
DBUSER=”<dbuser>”;
DBPWD=”<dbpassword>”;
DBNAME=”cacti”;
GZIP=”/bin/gzip -f”;

cd /

#Remove files older than 3 days
find /backup/cacti_*gz -mtime +3 -exec rm {} \;

#Backup Cacti Database
mysqldump –user=$DBUSER –password=$DBPWD –add-drop-table –databases $DBNAME > $BACKUPDIR$FILENAME
$GZIP $BACKUPDIR$FILENAME

#Backup Important Files
tar -csvpf $BACKUPDIR$TGZFILENAME ./etc/cron.d/cacti ./etc/php5/apache2/php.ini ./etc/apache2/sites-available/default ./usr/share/cacti

Additional files or directories can be added to the backup by entering them as arguments to the final line on the script separated by a space.

Schedule the script to run daily at 2:00am by creating “/etc/cron.d/cactibackup” with the following lines:

# Cacti backup Schedule
0 2 * * * root /bin/bash /backup/backupcacti.sh > /dev/null 2>&1

Displaying Logs in Splunk (Updated)

HOWTO: Simple Splunk install and config

INSTALLING SPLUNK

1) Navigate to the /opt directory (the default for most Splunk installs… don’t ask me why)
2) Grab Splunk from the offical repositories with the wget command
3) Unpack the downloaded .tgz file using tar
4) Run the script to install/start Splunk

cd /opt

sudo wget ‘http://www.splunk.com/index.php/download_track?file=3.4.8/linux/splunk-3.4.8-54309-Linux-i686.tgz&ac=&wget=true&name=wget&typed=releases’
sudo tar xvfz splunk-3.4.6-51113-Linux-i686.tgz
sudo splunk/bin/splunk start

Here is the link for 4.2.2

wget -O splunk-4.2.2-101277-linux-2.6-amd64.deb ‘http://www.splunk.com/index.php/download_track?file=4.2.2/splunk/linux/splunk-4.2.2-101277-linux-2.6-amd64.deb&ac=&wget=true&name=wget&typed=releases’

Accept the E.U.L.A. and your install is complete. The pretty web UI is now waiting for you at http://your.server.ip.address:8000 Simple, no?

UPGRADING SPLUNK
Stop the old version, download the new version and extract it in the same folder. Start Splunk back up and it will recognize the upgrade.

cd /opt
sudo splunk/bin/splunk stop
sudo wget ‘new-splunk-version-link-goes-here’
sudo tar xvfz new-splunk-downloaded-version.tgz
sudo splunk/bin/splunk start

CONFIGURING SPLUNK
This step will vary, depending on your needs. I still recommend a few settings for everyone:

Listen for logs on port 514:
Most devices and many apps (including syslog) use port 514 for sending log info. You’ll want Splunk to be listening.
navigate to your Splunk web UI (http://your.server.ip.address:8000)
click “Admin”
click “Data Inputs”
click “Network Ports”
“New Input” button.
choose “UDP” and the port number will automagically change to 514.
click the “Submit” button to save the configuration change

Start upon bootup:
Pretty self-explanatory. When the machine boots up, so does Splunk.

Code:

sudo /opt/splunk/bin/splunk enable boot-startOnly allow certain IP addresses to access the Web UI:
Since the free version of Splunk doesn’t secure the web UI, I lock down access to all that sensitive information through iptables. Obviously, you’ll want to replace “ip.address1.to.allow” with your address or a range you want to allow access from (i.e. 10.10.10.35 or 10.10.10.0/24).

Code:

sudo iptables -A INPUT -s ip.address1.to.allow -p tcp –dport 8000 -j ACCEPT
sudo iptables -A INPUT -s ip.address2.to.allow -p tcp –dport 8000 -j ACCEPT
sudo iptables -A INPUT -p tcp –dport 8000 -j DROP

SEND MAC/LINUX LOGS TO SPLUNK:

This is a two step process where you add your Slunk server to the list of known hosts on the client machine and then tell the syslog process to forward logs to Splunk.

Add the following line to /etc/hosts (NOTE: Use tabs, spaces won’t work.)

Code:

ip.address.of.splunkserver splunkserver

Where splunkserver is the name of your Splunk server. Now, add the following lines to /etc/syslog.conf:

Code:

# additional config for sending logs to splunk
*.info

@splunkseverWhere *.info is the level of detail you desire to be sent.

SEND WINDOWS LOGS TO SPLUNK

Download and Install Snare here: http://www.intersectalliance.com/dow…-MultiArch.exe

Open the Snare interface to configure its log management:
Click on “Network Configuration”
Set the “Destination Snare Server Address” to Splunk’s IP
Change “Destination Port” to 514
Click the checkbox to “Enable SYSLOG header”
Select your desired “Syslog Priority” level from the drop down menu.
Click the “Change Configuration” button

You might need to add an exception for Snare in the Windows Firewall. (tested in XP)
Navigate to the Windows Firwall settings (Start > Control Panel > Windows Firewall)
Click on the Exceptions Tab
Click the “Add Program” button
Browse to C:\Program Files\Snare\SnareCore (or wherever you installed Snare)

Cammer Perl Script from the Creator of MRTG

I haven’t had a chance to try this script out yet. From what I’ve read it’s pretty handy.

This is a site that looks like it was taken from a SNMP book that I’ve read.
cammer.pl

cammer – list switch ports with associated IP-addresses

SYNOPSIS

cammer [–arp=comunity@router] [–verbose] community@switch

DESCRIPTION

Cammer is a script which polls a switch is able to query a switch for its
ethernet to port asignements. If a source of ip2mac mappings is provided as
well, it will resolve the ethernet addresses too.

Cammer will use the machines local arp cache as well as the the /var/lib/dhcp3/dhcpd.leases
file to resolve etherent addresses
If you have a router, the router might know a lot about mac2ip mapping. Use

–arp=comunity@router

to query the router for its arp table.

We found that running something like

nmap –host-timeout 2s -sP 192.168.0.0/24

(as user!) helps to populate the local arp cache.

COPYRIGHT

Copyright (c) 2000 ETH Zurich, All rights reserved.
Copyright (c) 2008 OETIKER+PARTNER AG

LICENSE

This script is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

AUTHOR

Tobi Oetiker E <tobi@oetiker.ch>

A Tek-Tips user, PalmTest, had this to say in a post:

Hello Leo74,

You only have to combine the ARP table from your router with the “show mac-address-table” output from your switch.
And fortunately there is a perl script which will do this exact thing for you.
Find a MRTG mirror and find the /contrib/ folder.
http://public.planetmirror.com/pub/mrtg/contrib/
Now get your hands on the file “cammer”.
Read the cammer.readme.txt and off you go.

I run this perl script every hour and use the output to fill a database. Now I can keep track of MAC addresses which pop up in my network (wanted and unwanted).

I have a dedicated “Network monitor” PC installed for this.
The Monitor-PC does two things.
1: MRTG, to get a graphical overview of my most important switches and connections.
(http://www.openinnovations.com/mrtgbundle.html)
2: Cammer, to get a hold of all the changes in my network.
Cammer runs every 15 minutes.

When a problem occurs (whatever the reason), I can check the global utilization of my important connections to see if Production is in danger, and when necessary, I can trace any MAC address back to it’s switchport.
I can even trace MAC addresses which do not use TCP/IP and are not validated (Laptops!!) by the Helpdesk.
It surprises people that I know where a malicious machine is without them loggin in to my Domain.
The Network Monitoring PC is setup in such a way that there is no need to be logged in, so it runs stand-alone.

Here is the script. Like I said, I haven’t tried this yet. So, it is what it is. Don’t cry to me if it doesn’t work.

#! /usr/bin/perl
# -*- mode: Perl -*-
##################################################################
# Cammer 2.0
##################################################################
# Created by Tobi Oetiker <tobi@oetiker.ch>
##################################################################
# $Id: cammer 253 2008-07-08 19:22:10Z oetiker $

require 5.005;
use strict;
my $DEBUG = 0;

use FindBin;
use lib “${FindBin::Bin}”;
use lib “${FindBin::Bin}/../lib/mrtg2”;

use SNMP_Session “0.78”;
use BER “0.77”;
use SNMP_util “0.77”;
use Getopt::Long;
use Pod::Usage;
use Socket;

my %OID = (‘vlanIndex’ => [1,3,6,1,4,1,9,5,1,9,2,1,1],
‘vmVlan’ => [1,3,6,1,4,1,9,9,68,1,2,2,1,2],
‘dot1dTpFdbPort’ => [1,3,6,1,2,1,17,4,3,1,2],
‘dot1dBasePortIfIndex’ => [1,3,6,1,2,1,17,1,4,1,2],
‘sysObjectID’ => [1,3,6,1,2,1,1,2,0],
‘CiscolocIfDescr’ => [1,3,6,1,4,1,9,2,2,1,1,28],
‘ifAlias’ => [1,3,6,1,2,1,31,1,1,1,18],
‘ifName’ => [1,3,6,1,2,1,31,1,1,1,1],
‘ipNetToMediaPhysAddress’ => [1,3,6,1,2,1,4,22,1,2],
);

# Add the Cisco model number as displayed by the sysDescr.0 OID:
# $ snmpget -v 1 -c public <IP|hostname> sysDescr.0
#
# or by using your favorite SNMP MIB browser
my $CiscoCatIOS = “2900|3500|2950|2960|3550|4000 L3”;

sub main {
my %opt;
options(\%opt);
$DEBUG=1 if $opt{verbose};
# which vlans do exist on the device
my @vlans;
my $vlani;
my %vlan;
my $sws = SNMPv2c_Session->open ($opt{sw},$opt{swco},161)
|| die “Opening SNMP_Session\n”;

warn “* Query sysDescr to identify Switch\n”;
my $sysdesc = (snmpget($opt{swco}.’@’.$opt{sw},’sysDescr’))[0];
my $mode = ‘vlan’;

if ($sysdesc =~ /$CiscoCatIOS/){
warn “* Query VLAN list with vmVlan (Cisco Style)\n”;
$sws->map_table_4 ( [$OID{‘vmVlan’}],
sub { my($x,$value) = pretty(@_);
$vlan{$x} = $value; # catalyst 2900
print “if: $x, vlan: $value\n” if $DEBUG;
if (not scalar grep {$_ eq $value} @vlans) {
push @vlans, $value;
print “vlan: $value\n” if $DEBUG;
}
}
,100);
} else {
warn “* Query VLAN list with vlanIndex\n”;
$sws->map_table_4 ([$OID{‘vlanIndex’}],
sub {
my($x,$value) = pretty(@_);
push @vlans, $value;
print “vlan: $value\n” if $DEBUG;
}
,100 );
}

if (scalar @vlans == 0){
warn “* No VLANs were found, switching to normal mode\n”;
$mode = ‘normal’;
push @vlans, ”;
}

# which ifNames
my %name;
warn “* Gather Interface Name Table with ifName\n”;
$sws->map_table_4 ([$OID{‘ifName’}],
sub { my($if,$name) = pretty(@_);
print “if: $if, name: $name\n” if $DEBUG;
$name{$if}=$name;
}
,100);
$sws->close();

my %ip;
my %dhcp_host;

if (open my $arp, “arp -a|”){
warn “* Calling arp command to gather local arp table\n”;
while (<$arp>){
chomp;
#gumpu.oetiker.ch (192.168.0.157) at 00:0C:29:11:3F:92 [ether] on lan
/\((\S+?)\)\sat\s(00:\S+)/ or next;
my $ip = $1;
my $mac = lc($2);
push @{$ip{$mac}}, $ip;
print “ip: $ip, mac: $mac\n” if $DEBUG;
}
close $arp;
}
if (open my $dhcp, “/var/lib/dhcp3/dhcpd.leases”){
my $ip;
my $mac;
my $name;
while (<$dhcp>){
/lease\s(\S+)\s\{/ and do {
$ip = $1;
$mac = undef;
$name = undef;
next;
};
/client-hostname\s”(\S+?)”;/ and $name = $1;
/hardware\sethernet\s(\S+?);/ and $mac = lc($1);
/^\}/ and do {
push @{$ip{$mac}}, $ip if $ip;
$dhcp_host{$mac} = $name if $name;
}
}
}

warn “* Calling arp command to gather local arp table\n”;
open my $arp, “arp -a|” or die $!;
while (<$arp>){
chomp;
#gumpu.oetiker.ch (192.168.0.157) at 00:0C:29:11:3F:92 [ether] on lan
/\((\S+?)\)\sat\s(00:\S+)/ or next;
my $ip = $1;
my $mac = lc($2);
push @{$ip{$mac}}, $ip;
print “ip: $ip, mac: $mac\n” if $DEBUG;
}

if ($opt{ro}){
# get mac to ip from router
warn “* Gather Arp Table from $opt{roco}\@$opt{ro} with ipNetToMediaPhysAddress\n”;
my $ros = SNMPv2c_Session->open ($opt{ro},$opt{roco},161)
|| die “Opening SNMP_Session\n”;

$ros->map_table_4 ([$OID{‘ipNetToMediaPhysAddress’}],
sub {
my($ip,$mac) = pretty(@_);
$mac = unpack ‘H*’, pack ‘a*’,$mac;
$mac =~ s/../$&:/g;
$mac =~ s/.$//;
$ip =~ s/^.+?\.//;
push @{$ip{$mac}}, $ip;
print “ip: $ip, mac: $mac\n” if $DEBUG;
}
,100);
$ros->close();
}
# walk CAM table for each VLAN
my %if;
my %port;
foreach my $vlan (@vlans){
# catalist 2900 does not use com@vlan hack
$vlan = ‘@’.$vlan if $vlan;
warn “* Connecting to $opt{swco}$vlan\@$opt{sw}\n”;
my $sws = SNMPv2c_Session->open ($opt{sw},$opt{swco}.$vlan,161)
|| die “Opening SNMP_Session\n”;
warn “* Reading mac2port assignement from dot1dTpFdbPort table\n”;
$sws->map_table_4 ([$OID{‘dot1dTpFdbPort’}],
sub {
my($mac,$port) = pretty(@_);
next if $port == 0;
$mac = sprintf “%02x:%02x:%02x:%02x:%02x:%02x”, (split /\./, $mac);
print “mac: $mac,port: $port\n” if $DEBUG;
$port{$vlan}{$mac}=$port;
}
,100);
warn “* Reading port2if assignement from dot1dBasePortIfIndex table\n”;
$sws->map_table_4 ( [$OID{‘dot1dBasePortIfIndex’}],
sub { my($port,$if) = pretty(@_);
next if $port == 0;
print “port: $port, if: $if\n” if $DEBUG;
$if{$vlan}{$port} = $if;
}
,100);
$sws->close();
}
my %output;
foreach my $vlan (@vlans){
foreach my $mac (keys %{$port{$vlan}}){
my @ip = $ip{$mac} ? @{$ip{$mac}} : ();
my @host;
foreach my $ip (@ip) {
my $host = gethostbyaddr(pack(‘C4’,split(/\./,$ip)),AF_INET);
push @host, ($host or $ip);
}
my $name = $name{$if{$vlan}{$port{$vlan}{$mac}}};
my $truevlan = $vlan eq ‘none’ ? $vlan{$if{$vlan}{$port{$vlan}{$mac}}} : $vlan;
my $quest = scalar @ip > 1 ? “(Multi If Host)”:””;
push @{$output{$name}}, sprintf “%4s %-17s %-15s %s %s”,$truevlan,$mac,$ip[0],$host[0],$dhcp_host{$mac}?”($dhcp_host{$mac})”:”;
}
}
foreach my $name (sort { ($a =~ /(\d+)/)[0] <=> ($b =~ /(\d+)/)[0]} keys %output){
my $tag = ‘>’;
foreach my $line (@{$output{$name}}) {
printf “$tag %-4s %s\n”, $name , $line;
$tag = ‘ ‘;
}
}
}

main;
exit 0;

sub options {
my $opt = shift;
GetOptions( $opt,
‘arp=s’,
‘verbose’,
‘help|?’,
‘man’) or pod2usage(2);
pod2usage(-exitstatus => 0, -verbose => 2) if $opt->{man};
pod2usage(-verbose => 1) if $opt->{help} or scalar @ARGV != 1;

$opt->{sw} = shift @ARGV;

$opt->{sw} =~ /^(.+)@(.+?)$/;
$opt->{sw} = $2;
$opt->{swco} = $1;

if ($opt->{arp} and $opt->{arp} =~ /^(.+)@(.+?)$/){
$opt->{ro} = $2;
$opt->{roco} = $1;
}
}

sub pretty(@){
my $index = shift;
my @ret = ($index);
foreach my $x (@_){
push @ret, pretty_print($x);
};
return @ret;
}

Bulk Add Devices to Cacti

Working on a script to bulk add devices to Cacti. I’ve done on in a bash script, but that will only work on a windows box. I wrote this perl script so that I should be able to run it on any Linux or Windows box as long as perl is installed.

#!/usr/bin/perl -w

$LOGFILE = “/home/hades/perl/devices”;
open(LOGFILE) or die(“Could not open log file.”);
foreach $line () {

($description, $ip, $template, $notes, $avail, $version, $community) = split(‘,’,$line);

system(“php /usr/share/cacti/cli/add_device.php –description=$description –ip=$ip –template=$template –notes=\”$notes\” –avail=$avail –version=$version –community=$community”);
}

The source for the script is this text file named devices.

Firewall1,172.16.10.1,9,Firewall,snmp,2,public
Router1,10.10.0.1,5,Edge router,snmp,2,public
Switch2,10.10.0.5,5,Core switch,snmp,,2,public
Switch3,10.10.0.6,5,Access switch,snmp,,2,public
Switch4,10.10.0.7,5,Access switch,snmp,2,public
Switch5,10.10.0.8,5,Access switch,snmp,2,public

I have some more tweaking to do, but it’s basically functional.

Install Cacti on Debian Squeeze

Not too hard to get started. Once you have SNMPD installed and working just run:

apt-get install cacti

When that is complete you will have to modify the Apache config.

Add the follwing to “/etc/apache2/sites-available/default” between the VirtualHost tags:

Alias /cacti/ “/usr/share/cacti/site/”
<Directory “/usr/share/cacti/site/”>
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>

Then restart Apache:

invoke-rc.d apache2 restart

Use the url http://hostname/cacti and log in with the username admin and password admin. You will be asked to change the password and then you’re in!

Install SNMP on Debian Squeeze

This seems to have changed some from Debian Lenny. I don’t think I have all the bugs worked out yet, but here it goes.

First install SNMPD.

apt-get install snmpd

The default config file seems to be what was giving be some trouble early on. You can create one from scratch using the following command:

snmpconf -g basic_setup

Just answer the questions and it will create the file for you. If you run the command from the “/etc/snmp” directory and you won’t have to move the file when you’re done.

You will have to install the MIB files yourself since they are no longer included due to the usual licensing issues. You can, however, add them with a couple changes.

First you will have to have “non-free” added to your apt sources and uncomment the one line in “/etc/snmp/snmp.conf”

Then run the the following command:

apt-get install apt-get install snmp-mibs-downloader

That should do it. Try a snmpwalk command against the box and see what you get.

snmpwalk -c community -v 2c 192.168.0.10