Subject: Re: A proposal for merging Astroweb and VL at CERN

From: Jose Daniel Ponz VILSPA/VCS <[email-masked]>
Date: Wed, 28 Dec 1994 14:41:36 +0200 (CET)

Bob Jackson writes

 > ..the location of the reference database is irrelevant 
 > since its location is completely hidden from the user ..

The function (I foresee) for the reference database is just to serve as focal 
point as follows: 
  Reference database is the source (in HTML-format) of all mirror sites.
  The AstroWeb consortium could provide a common procedure to _all_ centers
  that want to keep a mirror copy of the database. This procedure, to be 
  included in the local crontab, will just periodically download the 
  _reference_  database. 

This is (I understand) the way in which the current VL is maintained at CERN:
Sergio Paoli keeps the master version, but the database used as reference by 
all centers is located at CERN.

If we want to negotiate (1) the merging and (2) that the new -merged- master 
database is at ST/ScI, with remote editing (it is a good technical solution), 
we should also give something back to Sergio, namely, that users _continue_ 
accessing CERN as reference. This will be the main question to negotiate with 
Sergio, once we agree upon it.

-------------------------------

On robots, Bob writes

 > As part of the value-added services provided by AstroWeb, we verify 
 > that each of the URL's is alive by a program...

Users should be informed, when they request to be added into the AstroWeb 
database, about the verification service provided by AstroWeb
(by executing the robot - MOMspider? - periodically to verify the URL's).
In addition, the consortium could also inform them on the usage of the
standard for robot exclusion in the local URL "/robots.txt".  

-------------------------------

On a different subject: I wish you a Happy New Year !!

Daniel

---

From: [email-masked] (Don Wells)
Date: Wed, 28 Dec 94 10:02:03 EST

>   Reference database is the source (in HTML-format) of all mirror sites.
                                          ^^^^^^^^^^ Bob chose to use
TCL as the programming language for the central database.  He,
therefore, developed an alternative format for the data. The
information content is identical, and we can freely translate from one
language to the other. Our documentation about the format does not
make this clear, because Bob did not develop the central database
concept until about June, and the documentation has not been revised.

 >   included in the local crontab, will just periodically download the 

MSSSO and STECF operate with variations of the procedures which I
originally built (in the programming language AWK). Any new consortium
members are welcome to my code, and I am sure that Anton and
Hans-Martin will make the same offer for their versions.  Bob uses
TCL. All of these solutions work, and other languages could be used
too.  There is no need to agree on a single master version of the
software.

 > Sergio Paoli keeps the master version, but the database used as
    reference by > all centers is located at CERN.

Bob and I are recommending that Sergio simply substitute the STScI
database for his master version at La Plata.  He will compute the CERN
pages, and any mirror copies of the VL will work from the CERN
version. CERN can continue to be the master version for astronomy VL.

 > .. users _continue_  > accessing CERN as reference. 

Actually, it appears to me that the astronomy users know that AstroWeb
is the reference. As soon as Sergio looked at our database, even he
acknowledged that it was more complete than the VL pages. Changing the
CERN VL pages to be computed from AstroWeb will mean that the general
public will be seeing the full database.

                -*-

In the end, our real strength is our distributed character, the fact
that we are a _team_.  My real reason for suggesting that Sergio be
invited to join us is that he is likely to add to the strength of the
team. Frankly, in the end, I don't care about the CERN VL, I care
about professional astronomy.  AstroWeb is a team effort, _by_ people
associated with professional astronomy organization, _for_ people
associated with professional astronomy organizations.  We need
diversity in our team in order to assure that we can serve _all_ of
professional astronomy.  Support for the general public and amateurs
is a secondary priority for us.  Sergio has found URLs on his own, and
has developed a category on his own; he should be on our team.

 > standard for robot exclusion in the local URL "/robots.txt".  

Many of the records in our database have OWNER fields.  We should
devise a robot which will verify the addresses periodically, and will
produce a list of "dead" Email addresses.  We should add more OWNER
fields to the database. Eventually we should build a robot which will
send Email to OWNERs periodically. The messages would ask them to
review their record, and it would be reasonable to remind the OWNERs
that their URLs and Email addresses are being checked by daemon
processes. We need some convenient and safe update procedure so that
OWNERs can help us to maintain the data.

-Don