Savegroup Summarizer -- A Legato NetWorker Reporting Tool
NetWorker Savegroup Summarizer (NSS) came about when we replaced
our old four-drive, 8mm (5-GB capacity!) tape jukebox with a new
tape jukebox holding two DLT7000 tape drives, driven by a dual-CPU
Sun Ultra 2 server. At the time, we were running Legato's NetWorker
Backup software version 4.2.x to back up about 100 clients of mixed
types, mostly UNIX with some PC and NetWare clients. I wanted to
get a better idea of exactly how much data we were backing up, how
long backups took to complete, and where the performance bottlenecks
were. I also wanted to see how much real improvement we were getting
in our backup performance after the upgrade to the new hardware.
Legato NetWorker is a powerful tool to manage backup needs, but
it has some frustrating limitations when it comes to reporting useful
statistics from a completed backup. The 5.x version was especially
limited in regard to reports you could get from the base product.
See Example 1 for a sample savegroup notification report. At the
time this program was first started, the NetWorker Management Console
product was not available, and the GEMs reporting tool was an expensive
add-on. (See the sidebar for an overview of Legato.)
After looking around and asking on the NetWorker mailing list
(see Resources), I ended up writing my own code as presented here.
The goals of the code were as follows:
1. Email the administrator a concise daily report detailing overall
status of the backups completed each night.
2. Report statistics on how much data was backed up and how long
3. Report on the largest saveset backup(s) in terms of the amount
of data, as well as the saveset backup(s) that took the most time,
since these are not always the same.
4. Keep the report in plain ASCII format, 80 columns maximum width,
so that reports are useful even on dumb terminal displays.
See Example 2 for a sample report generated by NSS.
The initial implementation of NSS just took the raw information
stored in the /nsr/logs/messages log file and parsed out the needed
information. This worked at first, because I could first look for
the matching lines that signified the start of a savegroup report,
and then look for the matching end of the report. Then some post-processing
would be done on the data to make it more presentable.
This process broke, however, when we had multiple savegroups running
at different times and with different clients that overlapped in
their start/end times. Since the messages file was just an ever-growing
file, it was impossible to tell which savegroup a line of output
belonged to without knowing lots of internal details of each savegroup.
Because I didn't want to do inquiries to NetWorker's internal database,
and because all the information I need was in the standard savegroup
notification, I wrote the savegroup notifications to a file and
ran them through NSS by hand. This was automated fairly quickly
with a procmail recipe.
Over time, NSS has been expanded with extra features, such as
a tape report to show how many tapes in each tape pool are full,
partially full, or empty. This is a useful check to ensure that
you have enough tapes for the next night's run. The limitation here
is that you must make a call to the mminfo command, which
may or may not be available on the system running the report. Another
useful feature is the ability to save easily parsed summaries of
the backups for each savegroup recording such details as:
- Start and end time
- Time taken to save
- Number of clients
- Number of failed clients
- Size of the backup in kilobytes
- Backup level
The server and savegroup names are implicit in the default directory
structure and filenames used. See the online usage of NSS for details,
and Example 3 for a sample log file.
The eventual goal is to write a plotting (nss-plot, see the resources)
program to graph the output in a nice overview format. This project,
however, has been going very slowly due to time constraints and
limitations of the various plotting tools.
Breakdown of NSS and How It Works
The first step of the script is the parsing of the various command-line
options. You can change all settings and actions from the command-line
switches, so you can embed the script inside another script if need
The core stage is to process STDIN and to scan the input for the
start and end markers that determine the savegroup report. This
is really the heart of the code, and can be thought of as a simple
finite state machine. Any good book on computer theory and programming
will describe state machines in more detail, but one definition
- An initial state
- A set of possible input
- A set of new states that may result from the input
- A set of possible actions or output events that result from
a new state
This is a very useful way of thinking when you are trying to write
a program (which is just a fancy state machine) to parse a set of
data to turn it into a more useful representation. Humans are very
good at pattern matching, but as seen when trying to write a regular
expression to handle all the myriad possibilities to pull out that
one piece of data, computers are not as good at pattern recognition,
or more accurately, exception handling in pattern matching.
This is where the power of Perl comes in, since we have both a
need for regular expressions to match and find the markers that
determine which state we are in, as well as the regular expressions
that are used to pull out the needed data inside each section. Luckily,
Legato has been quite static with the format of the savegroup reports
over three major versions of the software.
Another useful feature of Perl 5.x is the addition of complex
data structures, so we can do hashes of hashes to store the various
bits of information that are pulled from the savegroup report. This
document cannot really go into this in any great depth, so I would
recommend that you see the Perl documentation, especially the perldsc
and perlreftut man pages, or the O'Reilly & Associate
books Programming Perl (3rd Edition) by Larry Wall, Tom Christiansen,
and Jon Orwant and Advanced Perl Programming by Sriram Srinivasan.
Breaking down the problem into smaller steps is just one technique
that can be used to make the problem more manageable. In general,
when processing a random stream of input, looking for data, you
will have the following states and transitions that are possible
as you scan through the input stream.
- I have not found the data I need.
- I have found the data I need.
- I have found a different set of data I need.
- I am at the end of the data I need.
- No more data to read.
Transitions are how you move from one state to another, and are
really just how you indicate to the computer what to do on the next
step of input.
When parsing input, you can choose the chunk size, such as characters,
words, lines, paragraphs, etc. It can be tricky to determine which
state to be in (i.e., what to do with the input) when states span
multiple chunks. In such situations, it makes sense to break those
spans into multiple states. So, when you find Chunk A, you know
to look for Chunk B, or you fail and reset to the state before you
found Chunk A, or just go back to a default state.
For example, say you are reading input one line at a time, but
the markers you need to match against are split across two lines,
with the markers being "foo" and "bar", and the end marker being
"END", or the "EOF". Some basic Perl code to handle this can be
seen in Figure 3. Note that while we are reading our input, the
variable $state keeps track of the state we are in, which
tells us what we are expecting for input. This lets us skip the
processing of other states that don't apply or that might cause
problems if we can't tell what to do with the input without knowing
where we are in the processing. Also notice how we jump out of the
various "if-then" constructs and back up to the top of the loop
to read another line of input.
Parsing Input in Perl
Using this type of programming, we can process almost any type
of input. When parsing a NetWorker savegroup report, there are four
possible states the input can be in:
1. We haven't found the start of the savegroup report.
2. We've found the header and parsed the info from it.
3. We're done with the header but now we have "Unsuccessful Save
Sets" to read in and process.
4. We've found the "Successful Save Sets" marker and we're processing
In state one, we are looking for the marker that will drop us
into state two, where we will process the actual savegroup. This
lets us skip mail headers or other extraneous information.
In state two, we have matched the regular expression that tells
us we have found the start of the savegroup report. The code currently
supports both Legato NetWorker and "Solstice Backup", which is Sun's
name for their OEM'd version from Legato.
In this state, we pull out information on the number of clients,
the savegroup name, the start (or restart) time, and the end time.
We might also find out the name of the backup server if we're lucky.
This is one of the more frustrating limitations of the savegroup
report format, because it does not explicitly state the name of
the backup server.
So, we need state three to help figure out the name of the backup
server, because it's not explicitly stated in the savegroup report
anywhere. But, because we know that all indexes are saved on the
server, we can look for lines that mention "index save" and pull
the server name information from there. State three then may or
may not be used, depending on the savegroup report and whether there
were any failures.
The main work is handled in the fourth state, which is where we
process the individual client saveset (think filesystems) reports.
Again, we try to determine which host is really the server. Note
that when the server is finished writing all of a client's savesets
to tape, it will write the client index on the server to tape as
well, so we also look for that information. As each client's information
is read in, we find: client name, saveset (directory), level, total
amount of data written, the scale used for this measurement (e.g.,
kilo-, mega-, giga- or terabytes), the time it took to write the
data, and the number of files saved.
There is no finish state. I assume that the input will continue
to match in the fourth state, but this is handled by skipping any
input that doesn't match the NetWorker report format, so processing
continues until the EOF.
One key step during the processing of client saveset reports in
state four is to take the amount of data saved and scale it into
kilobytes, so that it's all consistent. This simplifies other steps
further in the processing and printing of reports. (Note that I
use the standard 1024 bytes in a kilobyte, not the marketing driven
version that uses 1000 bytes in a kilobyte.)
Perl's ability to handle complex data structures is also a key
element here, because it lets us store the data we've parsed in
a hash of hashes of hashes. This could have been done using split()
and join() to build long keys to hold the information, or
using multiple parallel hashes, each for just one piece of info.
Using a complex data structure contains all the data in one structure
for each use. The basic data structure is as follows:
Client --> Saveset Name --> Level
--> Total Data Written
--> Time to Write Data
--> Number of Files Written
Each level of the above structure is a hash, pointing to one or
more sub-hashes as needed. The first level is the name of the client,
and since each client can have multiple Savesets, that leads to
the second level of the hash. At the third level, we could have
gone to a fixed array to hold the information, but continuing the
use of hashes serves two purposes. One, it's self documenting --
no need to remember that index 2 is the number of files written
by the Client in that particular saveset. Two, the sorting and report
generation functions are simpler and most consistent since the entire
data structure is just hashes.
Post-Processing and Reporting
After all the data has been parsed, we process the data to pull
out the start and end times, as well as sum the total amount of
data written to tape across all savesets from all clients. This
is a simple step, since we already have all the sizes in kilobytes,
so we simply sum them up by both level and the overall total.
Once this is done, we must determine which scale to use when showing
the reports. Generally, I like to use the biggest scale possible,
since if we have written gigabytes of data to tape, it's not too
informative to know how many megabytes it is.
After that, we can pick and choose among the various reports and
output them. When printing reports in Perl, most people think of
using formats as a quick and dirty option, but there are some limitations
to this method that I found frustrating -- mostly dealing with empty
line compaction. So, I only use formats for the main summary section,
and instead use printf() for the various extra reports.
These other reports are fairly self-explanatory, but I will look
at how a couple of them work. The print_top_n_size() function
will sort and display the clients that wrote the most data. This
is broken into two steps -- the first of which goes through all
the clients and totals up the size of all savesets written by that
client. The data is then put into a temporary hash. The second step
is to print out the header of the report and loop through the temporary
hash holding the client totals, printing until we either run out
of clients or reach the maximum number of clients to show. Generally,
only the top five or ten clients are interesting in terms of the
amount of data written.
In contrast, the report for the top N hosts by time is broken
down by saveset, since a client with a very small amount of data
could have a problem and take a large amount of time to write that
data. This is interesting data and should be shown for troubleshooting
purposes. The general structure of the report is the same though,
where a first loop through the data builds a temporary hash of the
needed info, then the header is printed and the report is written.
One suggestion would be to include the size of the data written,
as well as the time, but I haven't felt a need for this, and it
has not been requested. There is also an issue with the report width,
since I am trying to keep the entire report less than 80 columns
in width if at all possible.
Setting Up and Using NSS
To run these scripts, you need a reasonably up-to-date version
of Perl (see resources) and the Time::Local and Getopt::Long libraries,
both of which come standard with Perl 5.000 and newer.
Edit the file to make sure that you have the correct path to your
locally installed version of Perl. You can also edit the first few
lines that specify the default directories for where the logs and
the savegroup input should be saved. These can also be specified
on the command line with the -L and -O options, respectively.
You can feed the raw Savegroup summaries directly to NSS via STDIN
to get a nice report, as shown in Example 2. This is very useful
for testing or just running off a quick report to make sure things
are working correctly for your site.
Another technique would be to use procmail to filter incoming
savegroup notification emails and forward the summaries, while saving
the raw savegroup notification to a mail folder. This is slightly
trickier, but eliminates the worry that you'll lose the raw notifications
sent to your sys admins. See Figure 2 for an example procmail recipe;
you will probably have to tweak it to recognize the format of email
sent to you. In this example, the email is sent from SERVER, and
it is saved in a mail folder in the user's Mail/ directory. For
a more complete discussion of procmail and how to write recipes,
see the resources section.
You can call NSS directly from NetWorker so that only summaries
get emailed to the sys admins. Here are instructions for Legato
NetWorker 5.1.1 on Solaris. They might be slightly different depending
on which version of NetWorker and which OS you are running.
1. Start up the /usr/bin/nsr/nwadmin GUI and connect to your NetWorker
2. Click on Customize -> Notifications.
3. A new window will pop up with a list of notifications. Scroll
down and highlight "Savegroup completion".
4. Edit the action to be as follows:
/path/to/nss -o -l -s 10 -t 10 -T -m "email@example.com"
5. Click on the "Apply" button.
The above options deserve some explanation, as shown in Figure
1. Note, however, that you can see the online help with an explanation
of all the arguments by passing the -h flag to nss
when you run it from the command line.
The -o option tells NSS to write the savegroup notification
to the default saveinput directory, as specified in the source code,
or by the -O option. The filename format option, -F,
defaults to "%S/%G-%D" where %S is the backup server name, %G is
the savegroup name, and %D is the date of the savegroup notification.
This lets you log the data from multiple backup servers, each with
multiple savegroups into a central and consistent directory structure.
If necessary, you can recreate reports by running NSS on the file(s)
The -l option is similar, but it turns on the logging of
the summary data to be plotted later in some sort of data analysis
tool, such as plot-nss. It also has a companion option, -L
for which directory to log the data to.
The -s and -t options can be used with or without
optional numbers. They turn on the reporting of the top N (default
5) savesets in terms of size and time, respectively.
The -T option turns on the tape report. Please note that
this option can only show you the current status of tape use at
the time the report is run. So, if you run the report immediately
after the savegroup notification is sent by NetWorker, you will
get a reasonably accurate status of the tapes remaining at that
time. If you run this report later, you may encounter problems,
because it depends on the permissions of the user running the report
in order to use the mminfo command to extract information
from NetWorker. Such access might not be permitted to general users.
The -m option is obvious, because it is the email address
to which to send the Savegroup Summary. If this is left off, the
default of STDOUT is used. Its companion option is -S <string>,
which specifies the subject of the Savegroup Summary being emailed.
It defaults to the following: "Backups %E: %S - %G", where %S and
%G are the same as the -F option, and the %E gives the status
of the backup that completed, either "SUCCEEDED" or "FAILURES".
The idea here is that if you miss reading email for several days,
you can sort your inbox by "Subject:", and all the successful reports
can be disposed of quickly. In this way, you can focus on the failures,
since they are the most important.
Plans for future work include dynamic sizing of column widths
in reports based on client hostname length, plotting/visualization
tool for the data saved with the -l option, handling newer
versions of NetWorker, and adding support for Veritas NetBackup.
NSS is still under development as time and energy permit, but
the basic layout has stabilized over the past year because it does
what I need it to do without muss or fuss.
In this article, I've tried to provide both a useful script and
some pointers on the concepts you can use to write your own application
for parsing arbitrary input and generating useful reports.
NSS Homepage -- http://jfs.ecotarium.org/sources/nss
Procmail -- http://www.procmail.org
Legato -- http://www.legato.org
NetWorker Users Mailing List -- http://listmail.temple.edu/archives/networker.html
John Stoffel attended Worcester Polytechnic Institute where
he earned a degree in computer science and spent way too much time
doing Theatre and Rock'n'Roll lighting on the side. He currently
works as a senior UNIX sys admin for a not-so-large-anymore major
telecommunications company. He is also a Board member of the USENIX
Sage Certification program at http://www.sage-cert.org. He
can be reached at: firstname.lastname@example.org.