LogViewer
Darrin Husmann and Andy Sackrieter On our work as both consultants and systems administrators, we recognized the potential of log files as a valuable nugget in the enterprise systems rhetoric. System and application log files are where competent systems administrators usually find the true cause and applicable solution protocol to most system and application issues. After exhaustive searches, we realized that an easy-to-use, semi-autonomous log viewing application that offers a client-graphical user interface did not exist. So, while learning the Perl programming language, we wrote the application described in this article. LogViewer presents distributed log files via a centralized desktop application. A LogViewer instance is composed of a minimum of three elements, each containing unique and uniform sub-modules. The first element is the Obelisk, or master server. The purpose of the Obelisk is to act as the networking server and the primary database administrator, to monitor text and log files for exception or alarm conditions, and to keep the status of the instance current. The second element is the auxiliary server, one server per node being monitored. The purpose of an auxiliary server is to keep configuration data applicable to the node current, maintain communication with the Obelisk, and analyze log files for exceptions or alarms, and to tail applicable logs on that node. The third element is the Client/Graphical User Interface, which differentiates LogViewer from the other log-monitoring applications. The client offers three desktops of two log files each (six log files total), a simple text editor, and a menu from which users can dynamically select the log files to monitor. The primary benefits of the Client are:
• Decreased chance of reading error -- During emergencies, the client makes selection and viewing almost foolproof. • Increased security -- An operator does not need a system login to each node that has log files to monitor. Because of the LogViewers' custom RPC communication architecture, a desktop shortcut (on Windows or UNIX) is enough to launch the client. • Decisive access to critical information -- Having six logs in one application, with the ability to easily monitor even more (just one click to view another log file), simplifies gathering information. Additionally, the stop/start buffer option allows the user of the client application to temporarily stop or resume scrolling to assess a situation. • Built-in easy editor -- The built-in text editor and associated bindings allow a user to produce a running log of events, complete with actual log file entries, for more complete reviews of critical events.
The following sub-elements are instrumental in the composition of a robust instance:
• Alarm monitor -- Analyzes incoming text from logs and nodes for exceptions. Upon receiving an exception, it runs an associated system command on either the Obelisk or the Auxiliary server node. • Frequency monitor -- Same as the alarm monitor. • ftp module -- Used to periodically issue configuration files to subscribers, which allows dynamic updates and keeps the entire application in sync. • Status module -- Monitors instance status and attempts to correct exceptions.
In the next sections, we will show sub-modules operation, describe how they work, then use these to build the above elements. Figure 1 shows a screen dump of LogViewer in action. Figure 2 is a screen dump of LogViewer on a UNIX platform, and Figure 3 is an illustration of how all these processes work together as a LogViewer instance. Fundamental Elements Obelisk Server The Obelisk Server is composed of the following sub-modules:
• RPC/communication module • System integrity module • Database module • Alarm module • Configuration governance module (i.e., ftpr and database code) Auxiliary Servers The auxiliary servers are composed of the following sub-modules:
• RPC/communication module • Tailing module Client Servers The client servers are composed of the following sub-modules:
• RPC/communication module • Graphical interface code RPC/communication module The purpose of this is to provide a uniform, robust communication platform for the application. This module uses a modified version of the RPC code in the book Advanced Perl Programming (Sriram Srinivasa, O'Reilly Publishing, 1997). Program Files Configuration of these files will be explained in the next section, Installing LogViewer. The application gets its configuration data (port number, IP address of Obelisk, etc.) from the file ../cfg/setup.ini. Listing 1 shows a typical setup.ini file. This file will be discussed in the Installing LogViewer section. The application knows which logfiles to monitor by reading the file ../cfg/SUBSCRIBER_TABLE. Note the difference between UNIX and Win32 files and that the port is saved for later versions of LogViewer, (i.e., not used).
# Notes: # 1.note that the files for win32 should still be of the form # C!/windows, with the drive letter in caps and a shebang (!) # being subbed for the colon # # Format: ip:local_file:port:master_ip #unix files 24.6.30.3:/var/log/syslog:2112:24.6.30.3 24.6.30.3:/tmp/one:2112:24.6.30.3 24.6.30.3:/tmp/511:2112:24.6.30.3 24.6.30.3:/export/home/greycat/prod/log/gl_syslog_cli:2112:24.6.30.3 #wintel files 24.6.30.9:C!/apps/balsa/prod/log/gl_syslog:2112:24.6.30.3 24.6.30.9:C!/apps/balsa/prod/log/gl_syslog_cli:2112:24.6.30.3 The application knows which files to alarm by reading the file ../cfg/filters.
# # this allows you to take a specified action when a string appears # in the monitored files of a given node. # # Notes: # 1.note that the files for win32 should still be of the form # C!/windows, with the drive letter in caps and a shebang (!) # being subbed for the colon # # Form: ip:string:command:event_name(default is # event0):execute_location(local or on master server) # # use the balsa_filter_bldr command to build additional filters #this will exec rm /tmp/nothing/* (if rm is not in ../cfg/ua_commands) 24.6.30.3:darrin:/usr/bin/rm /tmp/nothing/* :event0:m #explain: if the text madeline occurs in a monitored file on 24.6.30.3, #do the command /usr/bin/echo "gluphus2". The l (or m) at the end #specifies to execute the command on the master (m) or the local server, #here 24.6.30.3. 24.6.30.3:madeline:/usr/bin/echo "gluphus2" :event0:l 24.6.30.9:cheryl:/usr/bin/echo "TEST42k" :event0:m 24.6.30.9:cheryl:echo "TEST#221ab" :event0:l 24.6.30.9:steve:echo "hello":event0:l The application performs frequency analysis by reading the file ../cfg/frequency (Listing 2). Installing LogViewer 1. Install Perl (version 5.004 or later). Perl is available from www.perl.com and www.activestate.com (for Win32) on each node that will be running LogViewer code. 2. Install the following Perl modules on each node that will be running LogViewer code. (Most modules are available from the sites listed in subtask #1, except RPC.pm and MSG.pm, which can be obtained from the O'Reilly ftp site for the book Advanced Perl Programming:
• libnet • RPC.pm • MSG.pm • FreezeThaw.pm • Crypt.pm • Tk.pm
3. Download the Balsa LogViewer code from 4. Review the LogViewer README file and the ../bin/README.addm file. 5. Add a user “balsa” (with password “balsa”) to an ftp server on the Obelisk. Note that the entry ftp_source in ../cfg/setup.ini must match this verbatim. 6. Run the script ../bin/balsa_install.pl (i.e., perl balsa_install.pl). 7. Review the files ../cfg/setup.ini and ../cfg/SUBSCRIBER_TABLE because these often require custom settings. The setup.ini file defaults to itself as master server, port 2111, and its bin directory as containing the desired Perl scripts (balsa_aux.pl, balsa_master.pl, balsa_client.pl). Therefore, you should review this file for proper settings. 8. Contact your network administrator to verify a valid port number is entered in ../cfg/setup.ini. s_port=2111 is the default setting. 9. Start the Obelisk:
perl balsa_master.pl We recommend that you then start the master timer, balsa_cycle_master.pl. Balsa_cycle_master kills the master after master_kill_time seconds to limit stack size. To restart the master application, reference the example in the ../scripts directory. Note that only one master can run at any time -- the system won't allow more than one server, but more than one aux_server or client can run simultaneously. 10. Start each auxiliary server:
perl balsa_aux.pl 11. Start the client:
perl balsa_client.pl -l logon_name Note that the logon_name will allow balsa_api to control this window and that logon_name will also look in the directory ../profiles for itself and customize its GUI appearance accordingly. 12. Start the frequency daemon on the master server:
perl balsa_freq_analysis.pl This application runs only once. To facilitate periodic frequency monitoring, it is best to put this into a while() loop such as:
$leave=1;
while ( $leave ) {
system("perl balsa_frequency.pl -c \
This application will serve as a frequency daemon sufficient for most needs -- if more granular needs occur, adjust the sleep number accordingly. Once the Obelisk is started, it must initialize the environment. This includes such tasks as ftping the configuration files to all subscribing nodes, reading in what files to watch for, what databases to configure, and what alarms to prepare for. Note that the Obelisk will ftp configuration files over to all subscribers. Additionally, ftp is then used periodically to update nodes, ensuring that all nodes have uniform configuration data and that the original configuration data resides in ../cfg, while the transferred data resides in ../cfg2. (This is because the Obelisk can also perform the Auxiliary server duties and will periodically ftp to itself.) To ready the database on the Obelisk, another process is called. This uses standard Perl databases. Notice that this takes into account the log files and nodes to be monitored as per the file ../cfg/SUBSCRIBER_TABLE. The Perl databases reside in the directory ../db. A limitation of this is that many Perl database structures exhibit decreased system performance when size exceeds 1Mb. To accommodate this limitation, you can run a database truncater in the system crontab or scheduler. Review the README and README.addm for use of the truncator application (balsa_dbtrunc.pl). The rest of the operations performed by the Obelisk, alarm and frequency analysis, system integrity, etc., are performed as corollary events to receipt of new data from auxiliary servers and timed operations. The Client The client offers the most options of all the modules. Most of the features of the Client GUI are shown in this section:
• Logon -- Balsa LogViewer does not require you to issue a logon. By default, it assigns you the logon “balsa”. However, a unique logon is beneficial in its ability to use the api (see the Options section). • Log menu selection -- This allows a user to specify which log will be viewed on a desktop. The menu resides on desktop four, beside the text editor. To select a log or desktop, specify the desktop, then double click on the desired log file to view. • Text editor -- Note that when you subscribe to a desktop, it gives historical log data up to db_buffer_time seconds.
Starting and Stopping • Stoplight button -- This stops and resumes scrolling of logfiles. While logfiles are not scrolling, all data is written into a buffer. This buffer is dumped when scrolling resumes. • Bindings -- Note that a double-click on a log entry automatically puts it into the text editor. This can be verified by looking at the text above the log portal on which the log file is running and the verification text that reports successful connection to the logfile. Depending upon user needs, cut and paste, and OLE integration can be added to the Client. Options LogViewer offers many options. The API interface allows another application to launch LogViewer to a specified log set automatically. This is used in alarms and batch job failure. Here is the format for enacting an API session (assuming a client GUI is logged in, with a username “darrin”) where it automatically shows the log /tmp/one on the node 192.168.3.33 on the right side of its first desktop:
perl balsa_api -i "darrin;right: desktop_1;192.168.3.33;/tmp/one" Note the quotes around the submitted argument. It is important to note in Windows systems that the colon in the drive should be substituted with an exclamation point (!). To log on as a given user, use the following command:
perl balsa.pl -l darrin This will also look for a profile in the directory ../profiles/ for darrin and customize the GUI accordingly. For an example of customization, refer to Figure 3. Caveats LogViewer will work on both Win32 and UNIX systems, even allowing an instance to have both Win32 and UNIX elements simultaneously. However, the following issues must be considered when porting the Perl source code back and forth:
• The different file structure between UNIX and Win32 will result in different changes in the configuration setup and files. Additionally, the “!” is substituted for the “:” in the configuration files. • The GUI EOL statements have trouble between platforms. This is just a visual limitation and does not degrade system performance. • This version of LogViewer has limitations on its frequency analysis time-scope and resolution of addresses within the filter viewer.
If you wish to evaluate Balsa LogViewer in a production environment, there are other modules involved -- a Perl cron application, a control console, system log viewers, and configuration of the application. See the README files and the Web site for updates and offerings. Conclusion LogViewer offers a simple solution to remote system monitoring of logfiles via a semi-secure environment. Bypassing command prompts and manual typing, LogViewer decreases the chance of operator error in shell operations and incorrect file monitoring during times of crisis. The API, customization, self-recovery, and versatility make LogViewer a great accessory to existing enterprise system monitoring and batch-file processing applications.
About the AuthorDarrin Husmann is a systems analyst with AT&T wireless and principal architect with Balsa Software. He can be reached at: dthusma@dthusma.balsa-tech.com.
Andy Sackrieter is a systems analyst for AT&T wireless in Minneapolis, MN.
|