Holy goosh!

Feeling bored of the same old search engine? Then check this out. Cool eh!

EOT

more...

less...

Internet access, linux not ok, windows ok, *bsd ok

So what seemed to be a problem? Recently, within our network segment, guarded by pf firewall, somehow linux(redhat,fedora,*buntu) found difficulty in accessing the internet. Either the page is half loaded, or did not come out at all. It is not the browser; same result using wget. Obviously it is not the network; since XP and *BSD can go out. So, I suspect the pf is the mole. But since the pf owner is not around, and IT department could not be much of a help unless the problem affecting their segment, nothing much can be done. Plus I'm half system admin I used to be, too lazy to trace out the root cause now. Btw, here's the definition of problem based on ITIL concept(taken from wikipedia):

A problem is a condition often identified as a result of multiple Incidents that exhibit common symptoms. Problems can also be identified from a single significant Incident, indicative of a single error, for which the cause is unknown, but for which the impact is significant.

A known error is a condition identified by successful diagnosis of the root cause of a problem, and the subsequent development of a Work-around.

EOT

more...

less...

And then nagios asked "Hi servers, how are you today?"

There are quite a numbers of Network or Health Monitoring System. Being an opensource user I have shortlisted several of them; nagios, zabbix, zenoss, opennms. If you really need a quick one, choose zabbix, then you got yourself a car. If you got time, learn nagios. You'll be given an engine, then you have to choose the type of body, rear wheel drive or 4 wheel drive, how many doors do you want and so on, until finally you got yourself a car. OpenNMS has the enterprise look but like zabbix, quite dependant on SNMP but not as easy to install. Zenoss has gained a good position in term of ranking at sourceforge.net and currently is ranked at no 9. I might wanna try to look at this(and openNMS) later on but in the meantime, nagios, i choose you.

This guide is intended for those using Redhat,Fedora,Centos and I have added some steps to fit my needs. This steps works for me but I am not responsible for what you are going to do to your system. Like any other system admins, you should always have and setup a test server before put it into production.

refer http://nagios.sourceforge.net/docs/3_0/quickstart-fedora.html for original quickstart documentation

Introduction

This guide is intended to provide you with simple instructions on how to install Nagios from source (code) on Fedora and have it monitoring your local machine inside of 20 minutes. No advanced installation options are discussed here - just the basics that will work for 95% of users who want to get started.

These instructions were written based on a standard Fedora Core 6 Linux distribution.

What You'll End Up With

If you follow these instructions, here's what you'll end up with:

* Nagios and the plugins will be installed underneath /usr/local/nagios
* Nagios will be configured to monitor a few aspects of your local system (CPU load, disk usage, etc.)
* The Nagios web interface will be accessible at http://localhost/nagios/

Prerequisites

During portions of the installation you'll need to have root access to your machine.

Make sure you've installed the following packages on your Fedora installation before continuing.

* Apache
* GCC compiler
* GD development libraries
* RRDtool

You can use yum to install these packages by running the following commands (as root):

# yum install httpd
# yum install gcc
# yum install glibc glibc-common
# yum install gd gd-devel gd-progs
# yum install rrdtool

1) Create Account Information

Become the root user.

# su -l

Create a new nagios user account and give it a password.

# /usr/sbin/useradd nagios
# passwd nagios

Create a new nagcmd group for allowing external commands to be submitted through the web interface. Add both the nagios user and the apache user to the group.

# /usr/sbin/groupadd nagcmd
# /usr/sbin/usermod -G nagcmd nagios
# /usr/sbin/usermod -G nagcmd apache

2) Download Nagios and the Plugins

Create a directory for storing the downloads.

# mkdir ~/downloads
# cd ~/downloads

Download the source code tarballs of both Nagios and the Nagios plugins (visit http://www.nagios.org/download/ for links to the latest versions). At the time of writing, the latest versions of Nagios and the Nagios plugins were 3.0rc1 and 1.4.11, respectively.

# wget http://osdn.dl.sourceforge.net/sourceforge/nagios/nagios-3.0rc1.tar.gz
# wget http://osdn.dl.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.11.tar.gz

3) Compile and Install Nagios

Extract the Nagios source code tarball.

# cd ~/downloads
# tar xzf nagios-3.0rc1.tar.gz
# cd nagios-3.0rc1

Run the Nagios configure script, passing the name of the group you created earlier like so:

# ./configure --with-command-group=nagcmd

Compile the Nagios source code.

# make all

Install binaries, init script, sample config files and set permissions on the external command directory.

# make install
# make install-init
# make install-config
# make install-commandmode

Don't start Nagios yet - there's still more that needs to be done...

4) Customize Configuration

Sample configuration files have now been installed in the /usr/local/nagios/etc directory. These sample files should work fine for getting started with Nagios. You'll need to make just one change before you proceed...

Edit the /usr/local/nagios/etc/objects/contacts.cfg config file with your favorite editor and change the email address associated with the nagiosadmin contact definition to the address you'd like to use for receiving alerts.

# vi /usr/local/nagios/etc/objects/contacts.cfg

5) Configure the Web Interface

Install the Nagios web config file in the Apache conf.d directory.

# make install-webconf

Create a nagiosadmin account for logging into the Nagios web interface. Remember the password you assign to this account - you'll need it later.

# htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

create a file called .htaccess in /usr/local/nagios/sbin containing:

AuthUserFile /usr/local/nagios/etc/htpasswd.users
AuthName "Welcome To Nagios"
AuthType Basic
Require valid-user

Restart Apache to make the new settings take effect.

service httpd restart

6) Compile and Install the Nagios Plugins

Extract the Nagios plugins source code tarball.

# cd ~/downloads
# tar xzf nagios-plugins-1.4.11.tar.gz
# cd nagios-plugins-1.4.11

Compile and install the plugins.

# ./configure --with-nagios-user=nagios --with-nagios-group=nagios
# make
# make install

7) Start Nagios

Add Nagios to the list of system services and have it automatically start when the system boots.

# chkconfig --add nagios
# chkconfig nagios on

Verify the sample Nagios configuration files.

# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

If there are no errors, start Nagios.

# service nagios start

8) Modify SELinux Settings

Fedora ships with SELinux (Security Enhanced Linux) installed and in Enforcing mode by default. This can result in "Internal Server Error" messages when you attempt to access the Nagios CGIs.

See if SELinux is in Enforcing mode.

getenforce

Put SELinux into Permissive mode.

setenforce 0

To make this change permanent, you'll have to modify the settings in /etc/selinux/config and reboot.

Instead of disabling SELinux or setting it to permissive mode, you can use the following command to run the CGIs under SELinux enforcing/targeted mode:

# chcon -R -t httpd_sys_content_t /usr/local/nagios/sbin/
# chcon -R -t httpd_sys_content_t /usr/local/nagios/share/

For information on running the Nagios CGIs under Enforcing mode with a targeted policy, visit the NagiosCommunity.org wiki at http://www.nagioscommunity.org/wiki.

9) Login to the Web Interface

You should now be able to access the Nagios web interface at the URL below. You'll be prompted for the username (nagiosadmin) and password you specified earlier.

http://localhost/nagios/

Click on the "Service Detail" navbar link to see details of what's being monitored on your local machine. It will take a few minutes for Nagios to check all the services associated with your machine, as the checks are spread out over time.

10) Other Modifications

Make sure your machine's firewall rules are configured to allow access to the web server if you want to access the Nagios interface remotely.

Configuring email notifications is out of the scope of this documentation. While Nagios is currently configured to send you email notifications, your system may not yet have a mail program properly installed or configured. Refer to your system documentation, search the web, or look to the NagiosCommunity.org wiki for specific instructions on configuring your system to send email messages to external addresses. More information on notifications can be found here.

11) You're Done(Well, I'm not)

Congratulations! You sucessfully installed Nagios. Your journey into monitoring is just beginning. You'll no doubt want to monitor more than just your local machine, so check out the following docs...

* Monitoring Windows machines
* Monitoring Linux/Unix machines
* Monitoring Netware servers
* Monitoring routers/switches
* Monitoring publicly available services (HTTP, FTP, SSH, etc.)

12) Beautify it a lil bit

Download nuvola style front-end from nagiosexchange.org.
Browse also for some other interesting logo package available there.
Previously, I used whiteline.zip and extract it into /usr/local/nagios/share/images/logos
# mkdir /usr/local/nuvola
# cd /usr/local/nuvola
# tar -zxvf ~/downloads/nagios-nuvola-1.0.3.tar.gz

Backup(just in case)
# cp /usr/local/nagios/share /usr/local/nagios/share.old
# cp -rf html/ /usr/local/nagios/share
edit /usr/local/nagios/share/config.js to make sure cgi-bin pointing to the right path and some other option

Done.

13) Replace nagios logo with your custom logo in statusmap

make sure you have install gd-progs package
turn your logo into gif or png format(eg mine.gif) and put it in /usr/local/nagios/share/images/logos
# cd /usr/local/nagios/share/images/logos
use pngtogd2 tool or giftogd2 tools to convert it into gd2 format
# ls -al mine.gif
-rw-r--r-- 1 root root 1536 Jan 24 12:24 mine.gif

# giftogd2 mine.gif mine.gd2 1536 1

backup old and replace with new logo
# cp nagios.gd2 nagios.gd2.old
# cp mine.gd2 nagios.gd2

Done.

14) Nagiosgraph

http://www.novell.com/coolsolutions/feature/19843.html provides good details
# cd /usr/local/
# tar -zxvf ~/downloads/nagiosgraph-0.9.0.tgz
read the INSTALL file carefully
you might need to create nagiosgraph.log and /var/spool/perfdata.log
and can manipulate the data like in serviceextinfo.cfg like:

define serviceextinfo {
service_description PING
hostgroup MYHOSTGROUP
notes_url show.cgi?host=$HOSTNAME$&service=$SERVICEDESC$&db=ping,losspct&db=ping,rta
icon_image graph.gif
icon_image_alt View graphs
}
so it gives a clearer picture by separating the data in 2 different graph

any problem refer here:
http://nagiosgraph.wiki.sourceforge.net/errors_and_troubleshooting

15) SMS Notification

download smstools and buy GSM Modem Wavecom 1306b

Now there's your car.

more...

less...

Installing rrdtool using yum

Between apt-get, up2date, yum or smart and a few more, which one is the best?. Actually I dont have favorite package manager. I just use whatever the server already has on board. If on Ubuntu, apt-get is more than enough. Right now I'm working on Centos 5 that need rrdtool package. By default, the yum repository configuration on Centos 5 does not have rrdtool in it. Thanks to dag wieers for his remarkable effort. Although he had some issue with yum developers previously. Nothing much I can help over there, so back to work.

Well, now create a file called dag.repo in /etc/yum.repos.d/ just like below

[root@yoebuntu yum.repos.d]# cat dag.repo
[dag]
name=Dag RPM Repository for Red Hat Enterprise Linux
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag
gpgcheck=1
gpgkey=http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt
enabled=1
[root@yoebuntu yum.repos.d]#

and then
[root@yoebuntu yum.repos.d]# yum install rrdtool

reference

more...

less...

Note to self: Perl Module Installation using CPAN

First approach:
[root@svr rrdtool]# perl -MCPAN -e shell

cpan shell -- CPAN exploration and modules installation (v1.61)
ReadLine support available (try 'install Bundle::CPAN')

cpan> install Time::HiRes
CPAN: Storable loaded ok

Makefile:91: *** missing separator
then set the environment variable LC_ALL to "C" and retry
from scratch (re-run perl "Makefile.PL").
(And consider upgrading your Perl.)
(You got this message because you seem to have
an UTF-8 locale active in your shell environment, this used
to cause broken Makefiles to be created from Makefile.PLs.)
Makefile:91: *** missing separator. Stop.
Deng!


To search for module while in the shell:
cpan> i /HiRes/
Or if you know the exact name of the modules, can install it straight away:
[root@svr rrdtool]# perl -MCPAN -e install 'Time::HiRes'

To check the installed modules:
[root@svr4 nagios]# perl -e 'use Time::HiRes;'
Can't locate Time/HiRes.pm in @INC -- meaning the modules is not(yet) installed

For manual installation:
Go search for the module at here, then:

[root@svr rrdtool]wget http://search.cpan.org/CPAN/authors/id/J/JH/JHI/Time-HiRes-1.9711.tar.gz
[root@svr rrdtool] tar -zxvf Time-HiRes-1.9711.tar.gz
[root@svr rrdtool] cd Time-HiRes-1.9711
[root@svr Time-HiRes-1.9711] perl Makefile.PL
[root@svr Time-HiRes-1.9711] make
[root@svr Time-HiRes-1.9711] make test
[root@svr Time-HiRes-1.9711] make install

Back to the error, google brought me to perl monks website

"RH9 caused more than a few problems for me with a number of modules until I edited my /etc/sysconfig/i18n file. Here's mine:

#LANG="en_US.UTF-8"
LANG="en_US"
SUPPORTED="en_US.UTF-8:en_US:en"
SYSFONT="latarcyrheb-sun16"

It looks like some modules (like CPAN!) don't like that UTF-8. If you don't want to monkey with your system-wide locale setting, just type "export LANG=en_US" and then try your install again."

So i did just that.
Test the module again:
[root@svr4 Time-HiRes-1.9711]# perl -e 'use Time::HiRes;'
[root@svr4 Time-HiRes-1.9711]#

No error means the modules is there.

more...

less...

Large file support for Apache

I'm about to do some Linux installation on several PC. Somehow I just feel to do it via network. Since I've done it using NFS before, this time around http method will be used. The server running on Centos 4.4 already had httpd-2.0.52-28.ent.centos4 installed and already got fedora 6 DVD iso on my external hard drive. So everything's seemed to be in place, right? TTEEETTTT!

I'd uploaded the iso into /var/www/html/fc6 and start up the httpd daemon. Then, fire up firefox and put http://fileserver/fc6/ just to see an empty directory. Moreover, I got the 403 forbidden error when try to access the file directly. Check the error logs and see the the infamous error message
"[Thu Nov 01 14:18:58 2007] [error] [client 192.168.1.207 ] (75)Value too large for defined data type: access to /fc6/FC-6-i386-DVD.iso failed".
Asked google and get a few reference, something was missing in my installed apache so that it could not support any file bigger than 2GB.

OK, no big deal, just use yum to update. But the version suggested by yum is still did not have this feature supported. Next alternative, download a tarball from the the apache itself(version 2.0.61) and run ./configure. Deng!! no gcc installed. Thanks to yum update, the compiler is installed a few minutes later.

./configure, make && make install, went smoothly. But still the get the error messages. I first thought that the later version has automatically set this large file option, guess I was wrong. So google some more to find the required parameters.
Later, I reconfigure my apache using the flags as below:

[root@fileserv httpd-2.0.61]# CFLAGS="-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64" ./configure --prefix=/usr/local/www

The file appeared in the browser and accessible. There goes the half day. Case closed. Got a DBA to meet.

more...

less...

Memory Lane - Tape Backup For Linux (Recovery)

A few days after using flexbackup as a backup tool, then I've reached the next phase in the procedure; Data Recovery. In other words, how'd you want to extract the data from the tape. Being absolutely vain in tape backup, I quickly typed 'ls' to view the content of the tape. Well, I wish it was that easy.

But it was not that difficult either provided you understand some basic things about a tape. Note to self: a tape is nothing like a cd where you can easily mount and unmount. A tape is not a folder where it holds all the files and sub-folder. It is just a media that use sequential access in archiving data instead of random access method used in disk.

So I compare the requirement given to me with what Flexbackup can offers. Flexbackup has an extract feature(--extract) and it also can read(--flist) a file that has list of archives to extract. That will do it. But first I need to get the list. So using tar -itvf and mt command, I end up with this tape-list.sh:

#
#!/bin/bash
# This simple script is to create a list of files from the tape.
# This list will be use to extract - [eg. tape-extract ]
# by yoe Dec,2005

help_usage()
{
echo "Usage: $0 filename "
exit 0
}

if [ $# -ne 1 ]; then
#echo "Usage: $0 [filename] "
help_usage
exit
fi

if [ -f "$1" ]; then
echo file exist!
echo choose another filename
exit 1
fi

tape=/dev/nst0
currentdir=`echo $PWD`
now=`date +'%Y%m%d'`
tempfile="temp.$now"
h=0

/bin/mt -f $tape rewind
/bin/mt -f $tape eod
lastcount=`/bin/mt -f $tape status |grep -i file |awk '{print $2}' |tr -d "number=" |tr -d ","`
echo "There are $lastcount blocks on the tape .."
echo
echo "Preparing to create filelist $2"
echo
/bin/mt -f $tape rewind
echo "Start creating filelist $2"

cd $currentdir
while [ $h -le $lastcount ]
do
tar -itvf $tape | awk '$1 !~ /V/' | awk '{print $6}' | sed -e 's/\.\///g' | grep "." >> $tempfile
h=$((h+1))
done

cat $tempfile | sort | uniq > $2
echo "Done creating filelist $2"
echo

#remove temporary file
#echo "Clearing temp files"
#echo
rm -rf $currentdir/$tempfile
#echo DONE
EOF

eg. #./tape-list list1.txt

so every files on the tape will be listed into a file called list1.txt. So, in list1.txt I just leave which ever file I need to restore and delete the unwanted. Then here's another script to extract the files listed in list1.txt called tape-extract.sh:

#!/bin/bash
# This simple script is for extracting files from backup.
# It requires a list that can be easily created using tape-list script
# by yoe Dec,2005

help_usage()
{
echo "Usage: $0 filename "
exit 0
}

currentdir=`echo $PWD`
logdir="/usr/local/test/log/"
now=`date +'%Y%m%d'`
tape="/dev/nst0"
logfile="tape-extract.log.$now"
flex_config="/etc/flexbackup.nst0"
if [ $# -ne 2 ]; then
help_usage
exit
fi

if [ -f "$2" ]; then

echo "extracting files into $currentdir"
h=0
/bin/mt -f $tape rewind
/bin/mt -f $tape eod
#/bin/mt -f $tape bsf 1
lastcount=`/bin/mt -f $tape status |grep -i file |awk '{print $2}' |tr -d "number=" |tr -d ","`
echo "There are $lastcount blocks on the tape .."
echo
echo Preparing to extract

/bin/mt -f $tape rewind

echo "Start finding and extracting files"
echo

cd $currentdir
while [ $h -le $lastcount ]
do
#/bin/mt -f $tape rewind
#/bin/mt -f $tape fsf $h
/usr/bin/flexbackup -c $flex_config -extract -flist $2 1>> $logdir$logfile 2>> $logdir$logfile
#/usr/bin/flexbackup -extract -flist $1
#/bin/mt -f $tape rewind
#echo h: $h
#echo 2m: $m
h=$((h+1))
done

echo "Done extracting"
echo "Create logfile named $logfile in log directory"
else
echo "file does not exist!"
echo
fi
EOF

eg. #./tape-extract list1.txt

more...

less...

Memory Lane - Tape Backup for Linux

Remembering the first task given to me, to find a backup solution for server logs. Being given a Redhat 9 box (dont ask why), and a tape drive (stop asking!), I have to choose appropriate tools to get the job done. Yes, you can simply do backup using built-in commands such as tar, cpio, dump etc. But I've never done this before and I'm sort of short of time, so I needed a quick (some people called it dirty) way to do this. After numerous searh engine and reviews, I end up with flexbackup tool.


Why flexbackup? Firstly, it is flexible as it sounds. It's like a middle-man software, where you first decide what kind of archive you want to use (afio, dump, tar, cpio, star, pax, zip, lha, ar, shar) the backup device, logfiles etc. and it will take care the rest. In my case I use 'tar' as the archive type.

So I downloaded the flexbackup tarball and installed it on the machine. The tape drive (Dell PV100T) is connected to the server via SCSI interface. So on RH9, you might want to load certain module for the tape drive to be recognized.

[root@bekap]# insmod /lib/modules/2.4.20-8smp/kernel/drivers/scsi/aic7xxx_old.o
[root@bekap]# insmod /lib/modules/2.4.20-8smp/kernel/drivers/scsi/st.o

Use the mt command to check the status of the device. On linux with one tape drive, the drive may be recognized as /dev/st0(or nst0). As far as I remember, st0 and nst0 are reffering to the same device, with different condition. If you run a command using /dev/nst0, the tape will be rewinded first before the running the command. If /dev/st0 is used, the command will be run at the current location on the tape.

[root@bekap]# whatis mt
mt (1) - control magnetic tape drive operation

[root@bekap]# mt -f /dev/nst0 status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x25 (DDS-3).
Soft error count since last status=0
General status bits on (41010000):
BOT ONLINE IM_REP_EN

To automate almost everything(what a sys admin always do), I just need to write a simple and sluggish bash script that contains mt command to operate the tape drive, and flexbackup command to backup files/folders.
Let say I need to backup all files in directory called /var/log/msglog.
Here's what I did:

[root@bekap]# cat /usr/local/test/tape-backup-log
#!/bin/bash
# This simple script is to be run for incremental log backup
# by yoe Dec,2005

if [ $# -ne 1 ]; then
echo "Usage: $0 [full] [incremental]"
exit
fi

tape="/dev/nst0"
rew=`mt -f $tape rewind`
flex_config="/etc/flexbackup.nst0"
#rewind the tape

#backup /var/log/msglog
#echo "backup /var/log/msglog"

if [ "$1" = "full" ];
then
echo "backup full for msglog"
echo
$rew
flexbackup -c $flex_config -dir /var/log/msglog -level full &> /dev/null
$rew
echo "Done . Refer log directory for details."

elif [ "$1" = "incremental" ];
then
echo "backup incremental for msglog"
echo
$rew
flexbackup -c $flex_config -dir /var/log/msglog -level incremental &> /dev/null
$rew
echo "Done . Refer log directory for details."
else


If you want to minimize user intervention, than blow it to the cronjob

[root@bekap]# crontab -l
# after inserting new tape, rewind, erase and full backup every 1st of the month at 10:05 am
5 10 1 1-12 * /bin/mt -f /dev/nst0 rewind && /bin/mt -f /dev/nst0 erase && /usr/local/test/tape-backup-log full

# run incremental backup every thursday 11:30 pm.
30 23 * 1-12 4 /usr/local/test/tape-backup-log incremental

Finally make an appropriate schedule for tape replacement.

dirty enough?

more...

less...