Subject: master list

From: [email-masked] (anton koekemoer)
Date: Tue, 15 Mar 94 21:46:32 EST

Don Wells writes:
> ...
> it would be easy to generate other sort orders. Take a look at the
> files in 
>  
>     file://fits.cv.nrao.edu/www/astroweb/

Unfortunately this URL doesn't work for me; one that does is:

      file://fits.cv.nrao.edu/fits/www/astroweb/

(by looking at the anon ftp tree) - is this the same one? It has all the
relevant files. I've copied the Makefile and been playing with it and the awk
scripts, which are all very nice. (- by the way is it possible please to put
the file /home/fits/dwells/bin/getdate.awk in the astroweb directory as well?)

What's the current procedure for adding new features to (and removing bugs
from) the Makefile and awk scripts - is there an anon ftp 'incoming area' where
the most recent Makefile and awk scripts can be put/got by any of us, or is
there one maintainer (Don?) who takes suggestions and modifies the files, or
incorporates our new files into the system. Similarly for new sites - could we
all add them to yp_protoname.html, or do we just advertise their existence on
this email exploder?

I may well start looking at making an 'MSSSO' style - it could consist of a
list of one-liners, a bit like what I've had till now (listing homepage, ftp,
gopher access after the name of each site), but adding the feature that
clicking on the name of a site brings up its descriptive paragraph. It may
serve as a kind of index. All the raw data for it would come (currently) from
yp_protoname.html, I'll try to integrate it with the current scheme of things.

						cheers,
						 - anton.

---

From: dwells (Don Wells)
Date: Tue, 15 Mar 94 08:35:00 EST

>       file://fits.cv.nrao.edu/fits/www/astroweb/

Sigh... you are correct. The FTP path must include '/fits', whereas
the HTTP path is rooted at /fits/. 

 > .. please.. put
 > the file /home/fits/dwells/bin/getdate.awk in the astroweb directory..

DONE.

 > .. procedure for adding new features to (and removing bugs
 > from) the Makefile and awk scripts..

No procedure -- I merely hope that you will put your code with
improvements in a place where I can see it!  At the moment I see no
reason for formality.  If you are able to execute the code on your
computer, I suggest that you do so. If you make improvements, or if
you invent other programs (perhaps in other languages), tell
'astroweb' about them.

 > .. Similarly for new sites - could we
 > all add them to yp_protoname.html, or do we just advertise their
 > existence on  > this email exploder?

IMHO, if we try to maintain one unique master file at a central site
it will lead to frustration. 

The strategy that I have been advocating, and which was the basis of
the design of the AWK programs, is that there will be multiple master
files, at multiple sites, all in the same format, and with many
identical resource records. Many (all?) of the consortium sites will
maintain copies of the master file, and will read the master files
from the other sites periodically and merge changes with their own
changes. If you maintain your own yp_protoname, and make it available
via anonFTP, I will code a crontab FTP daemon that will fetch your
file every day and will compare it against the previous file to
generate a report for me on changes you have made. You won't need to
tell me by Email --- my daemon will tell me! It will also tell me
about changes made by other consortium members. I can watch such
reports to decide for myself when there are enough interesting changes
to justify taking the time to integrate the changes into my own copy
of the master file. One great advantage of this strategy is that each
of us will be independently reviewing the changes, correcting the
errors, and taking personal responsibility for all of the
information. I.e., this design implies distributed quality control, no
single point of failure.

My suggestion of a daily FTP daemon is not theoretical --- I have had
such a daemon running for six months to fetch "Series-7" earth
rotation reports from the US Naval Observatory. The script differences
successive retrieved files to detect the weekly editions of the report
(which usually are issued on Thursdays). The program is robust against
a variety of failure modes. I know that Bob has some similar
technology, because my httpd log shows that he is fetching my master
file every day. I append my IERS daemon (coded in the Bourne shell
language) for your amusement.

 > .. an 'MSSSO' style - it could consist of a
 > list of one-liners, .. (listing homepage, ftp,
 > gopher access after the name of each site.. adding.. feature..
 > clicking.. brings up... descriptive paragraph.

This is exactly the kind of innovation that I have been anticipating.
Bob's searchable version is another (it is significantly better than a
plain WAIS server). "The purpose of standardization is to aid the
creative craftsman, not to enforce the common mediocrity."

 > .. raw data for it.. > yp_protoname.html..

Right. In the opening round of negotiations we agreed on an exchange
format. I argued that it should be HTML, mainly so that Mosaic can
display the files directly. I asserted that records can be parsed in
HTML about as easily as in any other language. My AWK programs are
parsing the resource records in a variety of ways. Rearranging the
data into an entirely different style will be quite practical.

-Don

-=-=-=-=-=-=-=- iers_s7_update.sh -=-=-=-=-=-=-=-
#!/bin/sh
#
# iers_s7_update.sh -- a shell script to fetch the latest IERS report
#                        D.Wells, NRAO-CV, October1993.
#
# set environment variable EDITOR to your favorite editor, then
# execute "crontab -e" and insert a line similar to this one:
# 19	5	* * 2-6	/fits/dwells/bin/iers_s7_update.sh
# which will cause the crontab daemon to run the script as the owner
# at 05:19am on days 2 (Tuesday) through 6 (Saturday). The current
# crontab can always be reviewed by issuing command "crontab -l". If
# the script fails it will Email an error report to $OWNER.
#
# The following command turn on debugging of shell script.
#set -x
#
NUMSAVE=9
#
FTPHOST=maia.usno.navy.mil
FTPUSER=anonymous
FTPPASS=[email-masked]'
FTPPATH=/ser7/
FTPFILE=ser7
FTPCLASS=.dat
FTPFULL=$FTPPATH$FTPFILE$FTPCLASS
#
DESTPATH=$HOME/ser7/
DESTFILE=$FTPFILE
DESTCLASS=$FTPCLASS
DESTFULL=$DESTPATH$DESTFILE$DESTCLASS
LOGFILE=$DESTPATH$DESTFILE.log
OWNER=dwells
#
echo "`date` $0" 				>>$LOGFILE
rm -f $DESTPATH$DESTFILE.prv
ftp -n $FTPHOST 				>>$LOGFILE 2>&1 <<!
user $FTPUSER $FTPPASS
dir $FTPFULL
get $FTPFULL $DESTFULL.tmp
quit
!
if [ ! -f $DESTFULL.tmp ]
then
#---------------------------------------------------------------
MESSAGE="Shell-script $0\
failed to fetch file ftp://$FTPHOST$FTPFULL"
/usr/ucb/mail -s "$0 ERROR." $OWNER <<!
$MESSAGE
!
echo "$MESSAGE" 				>>$LOGFILE
rm -f $DESTFULL.tmp
else
#---------------------------------------------------------------
# A new copy of the file was fetched:
echo "" >>$DESTFULL.tmp 
diff $DESTFULL.tmp $DESTFULL >$DESTFULL.dif
if [ -s $DESTFULL.dif ]
then
	# shift the names of the saved files:
	mv $DESTFULL $DESTFULL.0
	I=$NUMSAVE
	while true
	do
		IP=`expr $I - 1`
		if [ -f $DESTFULL.$IP ]
		then
			mv -f $DESTFULL.$IP $DESTFULL.$I
		fi
		I=$IP
		if [ "$I" -eq 0 ]
		then
			break
		fi
	done
	mv -f  $DESTFULL.tmp $DESTFULL
	ls -lg $DESTFULL			>>$LOGFILE
else
	echo "	<new file identical to old>"	>>$LOGFILE
fi
rm -f  $DESTFULL.dif $DESTFULL.tmp
#
#     +--------------------------+
#     | do something useful here |
#     +--------------------------+
fi
#---------------------------------------------------------------
# Filter and truncate the logfile for this daemon:
awk '((NF>0)&&($1!~/pattern1/)&&($1!~/pattern2/))' $LOGFILE \
                | tail -100 >${LOGFILE}2
rm -f $LOGFILE
mv ${LOGFILE}2 $LOGFILE