Subject: How to implement local searching

From: Bob Jackson <[email-masked]>
Date: Fri, 5 May 95 11:40:56 EDT

AstroWeb has the virtue that user interface is decentralized,
>>>>> except for the search tools <<<<<<<.

I think it would be a good idea to decentralize the search
tools as well.

Do all this and you no longer have to point to anarky.stsci.edu for
searching.

				Bob

.......................................................................

A. To make a local version of the WAIS search tool:

1. Build and install Tcl, Tk, and TclX:

	ftp://ftp.aud.alcatel.com/tcl/ftp.cs.berkeley.edu
		tcl7.3.tar.gz 
		tk3.6.tar.gz 
	ftp://ftp.aud.alcatel.com/tcl/extensions
		tclX7.3b.tar.gz 

2. Build and install freeWAIS:

	ftp://ftp.cnidr.org/pub/NIDR.tools/freewais/

Actually only 'waisindex' and 'waissearch' are needed.

3. Build and install Lynx:

	ftp://ftp2.cc.ukans.edu/pub/lynx/

4. Here is the file 'index-master' used to update the WAIS index.
I run it in a nightly CRON job.

*******************************************************************

#!/usr/local/bin/tcl
#########################################################
# CHANGE the above line to fit your site
#########################################################

#########################################################
# EDIT THESE TO FIT YOUR SITE
#########################################################

# URLs of the data source - The TCL database
set url http://lor.stsci.edu/astroweb/newmaster.tcl

# Working directory where outputs will live
set homedir /anarky/data1/html-data/astroweb
cd $homedir

# Name of file indexed by WAIS
set datafile $homedir/master.wais

# Location of the WAIS Database
#set waisdb $homedir/astroweb
set waisdb /anarky/data1/gopher-data/.waisindex/astroweb

# Location of the binaries
set waisindex /anarky/data1/bin/waisindex
set lynx /anarky/data1/bin/lynx

#########################################################
# PROC's used by script
#########################################################

# I like to display everything

proc writeurl { url fh } {
    global longtitle category owner shorttitle description
    puts $fh "<DT>"
    puts $fh "<A HREF=\"$url\">"
    puts $fh "$longtitle($url)"
    puts $fh "</A>"
    if [info exists category($url)] then {
	puts $fh "<!-- CATEGORY $category($url) -->"
    }
    if [info exists owner($url)] then {
	puts $fh "<!-- OWNER $owner($url) -->"
    }
    if [info exists shorttitle($url)] then {
	set words [string trim $shorttitle($url) () ]
	puts $fh "\($words\)"
    }
    puts $fh "</DT>"
    if [info exists description($url)] then {
	puts $fh "<DD>"
	puts $fh "$description($url)"
	puts $fh "</DD>"
    }
}

#########################################################
# The script
#########################################################

# Grab file
if [catch { set input [exec $lynx -source $url] } msg] then {
    puts stdout "ERROR - failed to grab $url\n$msg"
    exit
}

if [catch { eval $input } msg] then {
    puts stdout "ERROR - evaling input\n$msg"
    exit
}

set fh [open $datafile w]
foreach url [array names longtitle] {
    puts $fh "\n-----------------------"
    writeurl $url $fh
}
close $fh

# Since WAISINDEX always return an error condition
catch {exec $waisindex -d $waisdb -t dash $datafile} msg

puts stdout "WAISINDEX - $msg"

*******************************************************************

5. Here is the file 'search-master' which I put in the cgi-bin 
directory of my HTTPD server.

*******************************************************************
#!/usr/local/bin/tcl

#########################################################
# CHANGE the above line to fit your site
#########################################################

#########################################################
# EDIT THESE TO FIT YOUR SITE
#########################################################

# Location of the waisdatabase
#set waisdb /anarky/data1/html-data/astroweb/astroweb
set waisdb /anarky/data1/gopher-data/.waisindex/astroweb

# Location of the waissearch binary
set waissearch /anarky/data1/bin/waissearch

#########################################################
# The procedures and script
#########################################################

proc unescape {v} {
    # This a kludge
    regsub -all {\%25} $v {XXXPERCENTXXX} v
    set nv $v
    while {[regexp {%[0-9A-F][0-9A-F]} $v blah]} {
	scan $blah "%%%x" cv
	if {[ctype char $cv]=="&"} {
	    regsub -all $blah $v \\& nv
	} else {
	    regsub -all $blah $v [ctype char $cv] nv
	}
	set v $nv
    }
    regsub -all {XXXPERCENTXXX} $v {%} v
    return $v
}

# 
# For those pesky functions which aren't always
# ready to read or write
# Return 1 if unable to read or write
# Return 0 if able to read or write
# 

proc cantread { fh seconds } {
	set selectlist [select $fh { } { } $seconds]
	if { [lindex $selectlist 0] != $fh } then {
		return 1
		} else {
		return 0
		}
	}

proc cantwrite { fh seconds } {
	set selectlist [select { } $fh { } $seconds]
	if { [lindex $selectlist 1] != $fh } then {
		return 1
		} else {
		return 0
		}
	}

puts stdout "Content-type: text/html\n"
puts stdout "<TITLE>AstroWeb Index</TITLE>"
puts stdout "<H1>AstroWeb Index</H1>"

if { [llength $argv] == 0 } then {
	puts stdout "<ISINDEX>"
	puts stdout "Enter search parameters<P>"
	flush stdout
	exit
	} 

set searchstring [unescape $argv]

set fh [open "|$waissearch -d $waisdb -m 200 \"$searchstring\"" w+]

if { [cantread $fh 30] } then {
	puts stdout "ERROR - can't read from WAISSEARCH"
	}

# This read does not get all the information, but enough.
set reply [read $fh [fstat $fh size]]

if [string match "*Search produced no result*" $reply] then {
	puts stdout "<ISINDEX>"
	puts stdout "No items found for: $searchstring"
	flush stdout
	exit
	}

foreach line [split $reply \n] {
	if [string match "*NumberOfRecordsReturned*" $line] {
		set numitems [lindex $line 1]
		break
		}
	}

set itemcount $numitems

if { $numitems == "" || ![ctype digit $numitems]  } then {
	puts stdout "Software error: numitems = $numitems"
	flush stdout
	exit
	}

loop i 1 [incr numitems] {
	puts $fh $i
	}

puts $fh "q"
puts $fh "q"
flush $fh

puts stdout "<ISINDEX>"
puts stdout "Following $itemcount items found for: $searchstring<P>"
puts stdout "<DL>"

set ignorelines 1

while { [eof $fh] == 0 } {
    gets $fh line
# Ignore all lines until a ----- line is received
# Since the first read is not getting all the information
	if { $ignorelines && [string match "-*" $line] } then {
		set ignorelines 0 }

# Stop with these lines 
	if !$ignorelines {
		if [string match "Search for new words*" $line] then {
			break }
# Don't want these lines either, but keep trying
		if { [string match "*View document number*" $line] ||
		     [string match "*Headline:*" $line] ||
		     [string length [string trim $line]] == 0 ||
		     [string match "-*" $line] } then {
			continue } else {
			puts stdout $line
			}
		}
	}

close $fh

puts stdout "</DL>"

flush stdout
exit

*******************************************************************

B. To make a local version of the Harvest search tool:

1. Obtain binaries for Harvest:

	http://harvest.cs.colorado.edu/harvest/gettingsoftware.html

2. Create a broker on your system, by running the command

	CreateBroker

All you have to do is to tell it to collect from the gatherer
at host:

	anarky.stsci.edu

at port:

	8500

and to do an incremental collection with compression.

3. You also need to make some changes to your HTTPD server
as described in the Harvest file INSTRUCTIONS.html.

4. In your toplevel directory served by your HTTPD server, add two
symbolic links like:

  lrwxrwxrwx  1 jackson        29 May  3 11:16 brokers -> /anarky/data1/harvest/brokers
  lrwxrwxrwx  1 jackson        31 May  3 11:16 gatherers -> /anarky/data1/harvest/gatherers

---

From: Jose Daniel Ponz VILSPA/VCS <[email-masked]>
Date: Mon, 22 May 1995 19:01:17 +0300 (CET-DST)

>    I think it would be a good idea to decentralize the search

For your information, the AstroWeb search tool is now local in our server 
at Villafranca.

Regards,
Daniel Ponz