Subject: HTML tools

From: [email-masked] (Avishai Yacobi) (by way of [email-masked] (Hans-Martin Adorf))
Date: Wed, 19 Oct 1994 09:53:17 GMT

"ian c rogers" <[email-masked]> writes:

>I'm looking to do keyword searches on information in a small portion of
>my server.  There are many files in this section, in many directories,
>but all beneath a single directory.

>Are there packages available which solve just this function?  

I wrote a cgi-bin script that recursively searches an html tree and
build an html page with the hits where each hit is a URL to the matched data.

You will have to modify the $WWWhome to the root of your server portion
and the $title to something you like.

Here is an example of using this script:

Click on <b><a href="cgi-bin/htmlgrep.pl">Search MCIL WWW pages</a></b>
to invoke a regular expression search on all our local documents.<p>

----------------------------------------------
Avishai Yacobi, Motorola Communications Israel
Phone: (972-3)-565-9549  Fax: (972-3)-565-9507
email: [email-masked]
   or  [email-masked]
   or  [email-masked]

---- 8< ------- cut here -------------------
#!/tools/pd/bin/perl
# Name: htmlgrep.pl
# Author: Avishai Yacoby <[email-masked]>

# Search the entire local WWW tree for regular expression.
# return indexed list where each hit is a URL for the matched data.

# The grep code was enhanced from Piet van Oostrum's <[email-masked]> rgrep
# The simple cgi interface was taken from ??? whois.pl

$WWWhome = "/disk1/ftp/pub/www";
$title = "Search MCIL WWW pages";

$debug = 0;
$regexp = "";

sub send_index {
    print "Content-type: text/html\n\n";
    print "<HEAD>\n<TITLE>", $title, "</TITLE>\n</HEAD>\n";
    print "<BODY>\n<H1>", $title, "</H1>\n";
    print "<b>Type a regular expression for search:</b><p>";
    print "<ISINDEX>";
    print "</BODY>\n";
}

sub grep {
    do { &'send_index; #'
         return;
     } unless defined @ARGV;
    $regexp = $ARGV[$[];

    print "Content-type: text/html\n\n";
    print "<HEAD>\n<TITLE>", $title, "</TITLE>\n</HEAD>\n";
    print "<BODY>\n<H1>", $title, "</H1>\n";
    print "Search pattern: \"$regexp\".<p>";
    print "To view the full text, click on the appropriate match.<p><hr>";

    &do_grep($WWWhome);

    print "</BODY>\n";
}

sub do_grep {
    local ($file) = @_;
    local (*F, $ln, $f, $g, @dirfiles);
    if (-f $file) {
        if (open (F, $file)) {
            if ( $file =~ /.*\.(gz|Z)$/ ) {
                close(F);
                open (F, "zcat < $file|");
            }
            print "Reading $file\n" if $debug;
            &grep_file ($file);
        } else {
            print stderr "Cannot open $file\n";
        }
    }
    elsif (-d $file) {
        print "Entering $file\n" if $debug;
        if (opendir (F, $file)) {
            @dirfiles = readdir (F);
            closedir (F);
            for $f (@dirfiles) {
                next if ($f eq '.' || $f eq '..');
                $g = "$file/$f";
                next if (-l $g);
                if (-f $g && $g =~ /^.*\.html(\.(gz|Z))?$/ || -d _) {
                    &do_grep ($g);
                }
            }
        } else {
            print stderr "Can't open $file\n";
        }
    }
}

sub grep_file {
    local ($file) = @_;
    $file=substr($file, length($WWWhome));
    local ($name, $title, $found) = ("", "",0);
    while (<F>) {
        $ln++;
#        unless (defined $title) {
            if (/<title>(.*)<\/title>/i) {
                $title = $1;
            }
#        }
        if (/<\s*a\s+name\s*=\s*"([^"]+)"\s*>/i) { #"
            $name = "#" . $1;
            print $name if $debug;
        }
        if (/$regexp/oi) {
            unless ($found) {
                $found =1;
                printf("\n<p><U>In document \"%s\" (%s):</U><p>", $title,
$file);
            }
            chop if substr($_, -1, 1) ne "\n";
            s/<[^<>]*>//g;
            s/$regexp/<b>$&<\/b>/oig;
            print "<li>";
            print "line $ln: <a href=\"$file$name\">$_</a>\n";
        }
    }
}

eval '&grep';