///Automate Perl Module Deployment

Automate Perl Module Deployment

Scripts go Beyond CPAN to Ease Network Installations

If you run Perl across many different computers of any sort, you know how frustrating it can be to install Perl extension modules across those machines. The administrative process is even worse if you have a Web server farm and need to keep each machine up to date with a set suite of extension modules for your installation. CPAN helps, but there are issues with CPAN that make it an unwieldy solution for use on a network. This article provides possible solutions before covering the final system. The main goals are a unified installation/module set, a single download, and a guaranteed unified set of version numbers across all the computers in the network.

Perl Module Installation

As soon as you install Perl, you start to realize just how many modules make up the “base” installation of the language. As time goes on, you install more and more modules to extend and improve the functionality of your Perl installation. Installing modules is not a difficult process. Most modules are packaged as a simple compressed tarball. The tarball contains the necessary files — including C/C++ source code to integrate with a third-party library, if necessary.

Most Perl modules use the MakeMaker system — another Perl module installed with the base Perl installation — which provides information about the Perl installation as well as a mechanism for building and installing components in the right place. MakeMaker works by translating a simple configuration file into a standard makefile that can be used with the familiar make command.

Using CPAN

If you don’t know the Comprehensive Perl Archive Network (CPAN) already, refer to the main page that comes with each installation. CPAN can work in two ways — either through an interactive shell or through a series of functions that can be used as part of a wider Perl script-based solution. At its simplest, you can use CPAN to install a module simply by typing perl -MCPAN -e “install modulename”, where modulename is the name of the package, bundle, or full Perl module you want to install. CPAN will do the rest for you.

Automating CPAN

Most people are familiar with CPAN as an interactive shell for building and installing Perl modules. What many don’t realize is that the CPAN module that provides this functionality is actually just a wrapper around an internal application program interface (API). For example, to generate a list of installed modules that are out of date compared with the versions available on CPAN, you can simply call CPAN::Shell->r. Listing 1 provides an example of the command and its output.

Listing 1. Code to generate a list of outdated modules

$ perl -MCPAN -e 'CPAN::Shell->r'

CPAN: Storable loaded ok
Going to read /export/build/solaris/cpan/Metadata
Database was generated on Sun, 25 Jul 2004 02:10:00 GMT

Package namespace installed latest in CPAN file
B::Assembler 0.06 0.07 N/NW/NWCLARK/perl-5.8.5.tar.gz
Cwd 2.19 2.20 K/KW/KWILLIAMS/Cwd-2.20.tar.gz
DBD::mysql 2.9003 2.9004 R/RU/RUDY/DBD-mysql-2.9004.tar.gz
DateTime 0.2101 0.22 D/DR/DROLSKY/DateTime-0.22.tar.gz
DateTime::TimeZone 0.27 0.28 D/DR/DROLSKY/DateTime-TimeZone-0.28.tar.gz
File::Scan 1.23 1.25 H/HD/HDIAS/File-Scan-1.25.tar.gz
File::Spec 0.87 0.88 K/KW/KWILLIAMS/File-Spec-0.88.tar.gz
Image::Magick 5.5.7 6.0 J/JC/JCRISTY/PerlMagick-6.02.tar.gz
Mail::ClamAV 0.08 0.11 S/SA/SABECK/Mail-ClamAV-0.11.tar.gz
Module::CoreList 1.94 1.95 R/RC/RCLAMP/Module-CoreList-1.95.tar.gz
POE 0.2802 0.29 R/RC/RCAPUTO/POE-0.29.tar.gz
XML::RSS::Parser 2.11 2.15 T/TI/TIMA/XML-RSS-Parser-2.15.tar.gz
htmlop v0.2.6 0.26 J/JA/JANL/w3mir-1.0.10.tar.gz
5 installed modules have a version number of 0
890 installed modules have no parseable version number

To install these modules automatically so that you are up to date, embed the call to CPAN::Shell->r in a call to the CPAN::Shell->install function:

$ /usr/local/bin/perl -MCPAN -e ‘CPAN::Shell->install(CPAN::Shell->r)’

This code dumps the information you see in Listing 1, then installs each module to bring it up to date.

Automating CPAN Across a Network

CPAN is an excellent tool, but the system has its limitations. One major drawback is that it’s a single-system solution, which is fine if you want to manage your Perl installations on a single-computer basis. But if you have a set of Perl modules spread across your Web server farm or even across your lab on a range of platforms, trying to keep up to date across all the computers can be a time-consuming process — even if you use the automation techniques I discussed above.

CPAN also consumes a lot of Internet bandwidth in that for each computer on which you use CPAN, CPAN expects to download a copy of the files from one of the central CPAN repositories. CPAN also relies on a configuration that has to be tailored to each computer: You can copy a configuration file, which resides in /perlinstalldirectory/CPAN/Config.pm, only if you can configure your platforms identically.

An obvious route to take when using CPAN is to use the autobundle function. This function generates a list of every installed module in a Perl installation, then dumps the list into a format that you can execute to re-install the selection either on the same computer (after an update to the latest Perl version) or onto another computer. The code in Listing 2 shows you how to create a new bundle.

Listing 2. Code to create a CPAN autobundle

$ perl -MCPAN -e autobundle

...

Package namespace installed latest in CPAN file
AnyDBM_File 1.00 1.00 N/NW/NWCLARK/perl-5.8.5.tar.gz
Apache::ASP 2.57 2.57 C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::ApacheCommon undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::Application undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::CGI undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::CGI::Table undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::CGI::Test undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::Collection undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::CollectionItem undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::Date undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
...


Wrote bundle file
/export/build/solaris/cpan/Bundle/Snapshot_2004_08_08_00.pm

The file that is generated is created within your CPAN installation directory. You can copy it from that location or, if you’re upgrading to a later version of Perl, you can simply run the bundle module to re-install everything:

$ perl -MCPAN -e ‘install Bundle::Snapshot_2004_08_08_00’

Although the bundle function works fine, I have to admit that I’m not a huge fan of the bundling system as a network solution for a few reasons. First, bundling relies on having an identical configuration on each computer, which is fine if everything is identical. But if you have different platforms and environments, you can easily run into problems. Second, because the autobundle feature lists every module installed, it lists multiple packages (individual packages may contain multiple modules). However, the feature can also list references to later versions of Perl (as the source for some module updates).

The final problem, however, is that bundling relies on having a single computer responsible for containing the latest version of all the modules. You can’t use the feature to upgrade a standard suite up to the latest versions — whatever they may be — and this stipulation makes the system complex to use in a network environment.

if you want to build a custom solution, two are available. You can use a central CPAN installation directory, or you can use a central CPAN source directory.

Use a central CPAN installation directory

A relatively straightforward way of configuring CPAN across a network — and one that works well if all your computers are identical — is to create a single CPAN directory that is shared over Network File System (NFS), copy the CPAN/Config.pm file onto each computer, then use the CPAN::Shell techniques discussed earlier to install the modules on each computer. You install the “base” module set on the main computer (usually the one sharing the CPAN directory over NFS). For every other computer, the process of building and installing each required module should be quick, because the source tarballs already exist. Therefore, you don’t need to download the tarball again, saving precious Internet bandwidth and significant time.

The main problem with this technique is that for computers that are in all other ways identical, it’s wasteful on processor time: You’ve already compiled it once, and it’s wasteful to compile it all again. You also run into a dangerous situation in which the same build directory is being used for multiple computers, which really isn’t a good idea because different computers could be compiling the source at the same time, causing a combination of timing errors and, in a cross-platform environment, bad object files when linking.

Use a central CPAN source directory

An adaptation of the previous solution is to use the same source directory and basic CPAN configuration but manually change the configuration file so that the bulk of the configuration information for each host is host-specific. In this way, you solve the build directory problem but still use a local (that is, on your network) cached version of the tarballs. Listing 3 provides such a configuration file.

Listing 3. Example of a CPAN::Config file

# This is CPAN.pm's systemwide configuration file. This file provides

# defaults for users, and the values can be changed in a per-user
# configuration file. The user-config file is being looked for as
# ~/.cpan/CPAN/MyConfig.pm.

$CPAN::Config = {
'build_cache' => q[200],
'build_dir' => q[/export/build/solaris/cpan/build],
'cache_metadata' => q[1],
'cpan_home' => q[/export/build/solaris/cpan],
'dontload_hash' => { },
'ftp' => q[/usr/bin/ftp],
'ftp_proxy' => q[proxy.mcslp.pri:8080],
'getcwd' => q[cwd],
'gpg' => q[],
'gzip' => q[/usr/bin/gzip],
'histfile' => q[/export/build/solaris/cpan/histfile],
'histsize' => q[100],
'http_proxy' => q[proxy.mcslp.pri:8080],
'inactivity_timeout' => q[0],
'index_expire' => q[1],
'inhibit_startup_message' => q[0],
'keep_source_where' => q[/export/build/solaris/cpan/sources],
'lynx' => q[],
'make' => q[/usr/local/bin/make],
'make_arg' => q[-j3],
'make_install_arg' => q[UNINST=1],
'makepl_arg' => q[],
'ncftp' => q[],
'ncftpget' => q[],
'no_proxy' => q[],
'pager' => q[/usr/bin/less],
'prerequisites_policy' => q[follow],
'scan_cache' => q[atstart],
'shell' => q[/usr/local/bin/bash],
'tar' => q[/usr/local/bin/tar],
'term_is_latin' => q[1],
'unzip' => q[/usr/bin/unzip],
'urllist' => [q[ftp://cpan.etla.org/pub/CPAN], q[ftp://cpan.teleglobe.net/pub/CPAN],
q[ftp://ftp.demon.co.uk/pub/CPAN/], q[ftp://ftp.flirble.org/pub/languages/perl/CPAN/],
q[ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl/CPAN/],
q[ftp://ftp.mirror.anlx.net/CPAN/], q[ftp://ftp.plig.org/pub/CPAN/],
q[ftp://usit.shef.ac.uk/pub/packages/CPAN/], q[http://cpan.hambule.co.uk/]],
'wget' => q[],
};
1;
__END__

You’ll need to change all the configuration directory parameters to something local on each computer — for example, change the line ‘build_dir’ => q[/export/build/solaris/cpan/build], to point to a directory on the local machine. Meanwhile, share the CPAN source directory on your main computer, then change the keep_source_where parameter to point to this NFS-mounted directory.

Unfortunately, a few issues remain with this solution. First, you have no way of conveniently recording which modules you want to install on each computer. You also run into the problem of keeping the information in the source directory (and the rest of the CPAN directories) correct and current without updating or downloading a newer version of the tarball and upsetting the status quo among computers. This issue can lead to minor but nearly fatal differences between the installed versions of the modules on each system, which is the key issue that we are trying to resolve.

Use CPAN to Create a Rigid Installation

The problem with both of the previous solutions is that they rely on using CPAN and some manipulation of the configuration file to get CPAN to do the work. CPAN is not an ideal solution within a local area network (LAN), even with the above tricks. Therefore, you need to bypass those system limitations. To do so, I’ll show you how to configure the master distribution, create an installation set for each platform, and then run the installation on each computer.

Configure the master distributionFor this first step, you’re going to use some of the elements we’ve already covered. Begin by configuring one computer to use CPAN as normal and setting up your default set of modules. Next, use NFS to set up a shared directory that you’ll use to distribute the necessary files to the other computers in the network. Now, edit the $destdir variable you see in Listing 4, then run the Listing 4 script. This script does three things:

• Identifies the modules

• Downloads the source files

• Copies the source files to the distribution directory

The first stage determines the list of installed modules by examining the content of the perllocal.pod file. This file is updated each time a module is installed using the MakeMaker system. By using this file rather than the built-in modules in CPAN, you get a list of third-party modules installed with your base Perl installation. The second stage uses CPAN to download the source tarball by whatever methods are configured within CPAN itself before finally copying the tarball or .zip file. As part of that process, you also create a list of the modules that were copied, which you’ll use when you install the files on the other computers.

Listing 4. Code to create a base distribution

use CPAN;

use Config;
use File::Copy;
use File::Spec::Functions;

# Set up the directories to store the module packages
my $basedestdir = catfile('export','cpaninst');
mkdir($basedestdir);
my $destdir = catfile($basedestdir,'srcs');
mkdir($destdir);

# Extract a list of the third party modules installed on this machine
my $files = {};
my $podfile = catfile($Config{'archlibexp'},'perllocal.pod');
open(DATA,$podfile) or die "Can't open module list ($podfile): $!";
while(<DATA>)
{
if (m/.*C<Module> L<(.*)>/)
{
my ($module) = split /\|/,$1,0;
my $mod = expand('Module',$module);
next unless $mod;
$mod->get(); #Make sure to download the version again, in case we no
#longer have it locally
$files->{$mod->{RO}->{CPAN_FILE}} = 1; #Save the location of the file
}
}
close(DATA);

# Now copy each source installer to the source destination directory
# We save a copy of the module file, for reference, as part of the process

my $modulelist = catfile($basedestdir,'modules.lst');
open(DATA,">$modulelist") or die "Can't open the module list file ($modulelist): $!";
foreach my $file (keys %{$files})
{
my $src = catfile($CPAN::Config->{'keep_source_where'},$file);
my ($vol,$path,$filename) = splitpath($file);
my $dest = catfile($destdir,$filename);
copy($src,$dest) or warn "Copy of $src failed: $!\n";
print DATA "$filename\n";
}
close(DATA);

You now have a directory containing all the source installation packages that you can use to build a suite of ready-compiled distributions for each platform to which you need to distribute the corresponding Perl module. How you proceed from there will depend on your environment. If you have a homogeneous environment, you need to perform the next stage just once for your entire network. If, however, you have a heterogeneous network with a variety of different platforms, you’ll need to repeat the next step for each different environment.

Create an installation set for each platform

For each platform, you need to extract the source file, run MakeMaker, then run the installer. To perform this task automatically, use the script in Listing 5. Unfortunately, not all MakeMaker installations are completely automatic, so you’ll need to follow the interactive systems — for example, to set directory locations or default options — each time you build the distribution.

Listing 5. Code to build the modules on each platform

srcdir="/export/cpan"

srcfile="$srcdir/module.lst"
platformdir="/export/cpan/solaris-x86"

for file in `cat $srcfile`
do
cd $platformdir
tar zxf $srcdir/$file
dir=`echo $file|sed -e "s/\.tar\.gz//g"`
cd $dir
perl Makefile.PL
make
done

Run the installation on each computerWhen the script in Listing 5 has run, you’ll have a single directory that contains all the “ready-to-install” modules for a given platform. To install these modules on any other computer using the same platform, simply change into each directory and run make install. Again, you can use a script such as the one in Listing 6 for this process.

Listing 6. Code to install the files on each host

srcdir="/export/cpan"

srcfile="$srcdir/module.lst"
platformdir="/export/cpan/solaris-x86"

for file in `cat $srcfile`
do
cd $platformdir
tar zxf $srcdir/$file
dir=`echo $file|sed -e "s/\.tar\.gz//g"`
cd $dir
make install
done

Conclusions

Despite the functionality provided by Perl, CPAN, and the MakeMaker system, you can’t completely automate the installation and distribution of Perl modules. But you can make the process simpler across a range of computers. Individual modules have their own configurations, and individual platforms have their own nuances making them more complex than they need to be. By using the scripts in this article, you can eliminate much of the hardship while reducing the time overhead that might normally be associated with the process.

Not all issues are covered by this solution. For example, third-party modules often make use of third-party libraries that are not installed using this method. See my other developerWorks articles for help here.

Resources

• ” Automate your build and distribution process” (developerWorks, September 2004) shows you how to simplify the installation of any open-source software project.

• If you’re interested in squeezing more performance from your Perl code, read ” Optimize Perl” (developerWorks, October 2004).

• Perl has long been an important scripting tool for Linux developers, and developerWorks has an extensive collection of Perl articles to pick from.

Comprehensive Perl Archive Network is the home of the CPAN network system.

• Find more resources for Linux developers in the developerWorks Linux zonE.

• Get involved in the developerWorks community by participating in developerWorks blogs.

• Purchase Linux books at discounted prices in the Linux section of the Developer Bookstore.

Order the no-charge SEK for Linux, a two-DVD set containing the latest IBM trial software for Linux from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

• Innovate your next Linux development project with IBM trial software, available for download directly from developerWorks.

2010-05-26T11:27:28+00:00 April 27th, 2005|CGI and Perl|0 Comments

About the Author:

Martin C. Brown is a former IT Director with experience in cross-platform integration. A keen developer, he has produced dynamic sites for blue-chip customers including HP and Oracle and is the Technical Director of Foodware.net. Now a freelance writer and consultant, MC, as he is better known, works closely with Microsoft as an SME, is the LAMP Technologies Editor for LinuxWorld magazine, is a core member of the AnswerSquad.com team, and has written a number of books on topics as diverse as Microsoft Certification, iMacs, and open source programming. Despite his best attempts, he remains a regular and voracious programmer on many platforms and numerous environments. You can contact MC at questions@mcslp.com or through his Web site.

Leave A Comment