Subject: A proposal for merging Astroweb and VL at CERN

From: Jose Daniel Ponz VILSPA/VCS <[email-masked]>
Date: Tue, 20 Dec 1994 12:03:21 +0200 (CET)

Dear AstroWeb colleages,

After the positive comment from Don on the idea of merging AstroWeb and
the Astronomy Virtual Library at CERN, here you will find my proposal for 
merging. Please, send me your comments so that we can reach an agreement,
and then, invite Sergio Paoli for further discussions.

Take into account that this is only my proposal (as newcomer), based on the 
idea of sharing efforts, to provide a good service to the A&A community...

   Proposal for merging AstroWeb and WWW VL-Astron.&Astrophys into
                 ``The AstroWeb Virtual Library''
   ---------------------------------------------------------------

1. Generate a common database by merging the URLs in AstroWeb and WWW-VL.
   The database format should be the one currently used by AstroWeb (with 
   possibly minor additions, new keywords, etc).

2. The complete (merged) AstroWeb database has three instances:
   - Master database: stored in the AstroWeb Maintenance Center.
   - Reference database: generated daily from the master database, and 
     stored at CERN.
   - Mirror databases: downloaded daily by all the centers participating
     in the AstroWeb consortium (AstroWeb/Level 2 members) from the
     Reference database at CERN.

3. Access to database URLs is done using two levels of pages, very much
   along the structure of the yp_*.html pages at NRAO. These access
   pages are generated (daily) at the AstroWeb centers (Level 2 Members),
   from the Mirror databases, using a set of procedures to be developed 
   (recommendation to use Perl for portability).
   Common look and feel should be defined for the main AstroWeb-VL page
   (it is not the case now).

4. The master database is updated at the Maintenance Center based on 
   the information given in the requests forms, e-mails or discussions
   within the AstroWeb e-mail list. These updates are manually entered
   into the database.

5. The Maintenance Center updates the Master Database and generates the
   Reference Database at CERN. Maintenence Centers are selected for a
   period of 2 (?) months among the AstroWeb Level 1 members (bussiness 
   class in  H.-M.'s e-mail, a few days ago ;-).

6. At the end of the 2 (?) months period, the maintenance duty is assigned 
   to a new Maintenance Center. At that moment, the new database 
   administrator downloads from CERN the Reference Database becoming the 
   new Master Database.
  
7. E-mail discussions and updates (database quality control) are posted
   in the AstroWeb e-mail exploder. The exploder is implemented at, and 
   controlled by NRAO. E-mail is distributed among AstroWeb Level 1 members.

8. Update proposals are done via (HTML-) forms, mantained at ST/ScI.

9. AstroWeb Level 2 membership completely open to centers requesting it. 
   These centers will generate mirror copies from the Reference Database.
   (In fact, if you are running a modern WWW proxy server you
   will not even need to generate explicitly the AstroWeb files, the 
   information could be stored in the cache, updated when required).

10.Requirements for AstroWeb Level 2 members have to be defined.
   
11.AstroWeb Level 1 members are: current AstroWeb members + Sergio Paoli.
   They will receive (develop) a (simple) maintenance SW kit.

12.If the proposal is accepted, I submit a full Level1 membership request ;-)

FUNCTIONAL SUMMARY
------------------
Tentative definition of functions assigned to each center. The procedure to
merge both databases is not included.

   Functions assigned to AstroWeb/Level 1
   -------------------------------------
   NRAO     AstroWeb Exploder
   NRAO     Update/correction forms, incoming records
   ST/ScI   Unreliable/dead URL's global list
   CERN     Reference database

   Functions supported by all AstroWeb/Level 1 
   -------------------------------------------
   - Download Reference database and generate yp*.html files.
   - Shared database maintenance, for a period of 2 months, in rotation.
   - Generation of local WAIS indices.
   - Check unreliable/dead URL's (as seen from that site).
   
   Functions supported by the Maintenance Center on duty
   -----------------------------------------------------
   - Update Master database and transfer it to CERN as Reference database 
   - Generate yp_*.html files 
   - Check syntax, entries and keywords in generated html files
   
ADVANTAGES:
-----------
1. Unified (optimal) access to astronomical resources.
2. Simplification of the procedure for announcing new resources.
   Currently one has to submit (at least) two update requests in the 
   Astro&Astrophys. context.
3. Maintenance workload is shared. 
---------------------------------------------------end of proposal

This is all for the moment.
Best regards,

Daniel

---------------------------------------------------------------------------
J.D.Ponz                                          e-mail: [email-masked]
ESA/Villafranca, 
P.O.Box 50727, 28080 Madrid, Spain.               Phone:  34-1-8131183
---------------------------------------------------------------------------

---

From: Bob Jackson <[email-masked]>
Date: Tue, 20 Dec 94 10:57:34 EST

There many valuable suggestions in your proposal, but I 
should clarify how the "Maintenance Center" currently works.

AstroWeb members modify the Master Database at STScI via
the HTML forms at STScI.  (Don may have semi-automated
the input to the STScI forms from the NRAO forms.)

The Master Database at STScI uses CVS to provide 
prior version recovery and to allow SIMULTANEOUS 
editing WITHOUT LOCKING.  This use of CVS would make
rotating the "Maintenance Center" a little difficult,
but not impossible.

Whenever a change is made to the Master Database, new
versions of

	http://lor.stsci.edu/astroweb/newmaster.tcl
	http://lor.stsci.edu/astroweb/newmaster.html

are generated.

This allows the Master Database to be remotely edited,
and does not impose any serious burden on the site
containing the Master Database.
	
If you want to see how the remote editing feature works,
email me a

	username
	password
	domain

and I will give you the permissions to use the remote
editing tools.

e.g.,

	ponz
	???
	vilspa.esa.es

The software checks that the user's client is running
on a machine within the domain.  My domain is set to
the machine on which I run Mosaic.

			Bob

---

From: [email-masked] (Don Wells)
Date: Tue, 20 Dec 94 11:21:22 EST

Bob Jackson writes:
 > .. Don may have semi-automated > the input.. from the NRAO forms..

Sigh.... not yet! I have been busy with GBT matters for the past three
months. I do still intend to do this enhancement for AstroWeb.  The
strategy that I intend to try is that the output of the NRAO forms
will be an ADD command to the master database. I.e., it will have
username/password fields which one of us will have to fill out in
order to authorize an ADD transaction on the master database.

I wish that the DELETE command on the master database would return an
ADD form. If it did, one could DELETE a record and then edit it in the
window and then do an ADD, which would be effectively a MODIFY
operation.

                -*-
 
No elaborate new procedures or technology are needed for the CERN
pages. Sergio Paoli already computes the CERN pages from a master file
at La Plata. All he needs to do is to fetch the master AstroWeb file
from STScI (or NRAO or..) and compute his pages in whatever style he
considers appropriate. He should add a reference to the AstroWeb
Consortium on his top-level page, and should probably have either a
link or a field for a search utility. His style can be the same as
what he already has, or he could modify it. He could (and probably
should) continue as the person officially responsible for the CERN
pages. He would gain the advantage that (1) he would have more links
in his pages (because the AstroWeb database is larger than his) and
(2) he would have all of us helping him to maintain the database. We
would gain (1) by adding some records that we don't have and (2) by
adding an AstroWeb site at a well-known and prestigious location.
Everybody wins, so we really should be able to agree on this concept.

Regarding categories, Sergio has a list of 'mailing lists' (exploders).
I have long wanted to add this to AstroWeb, but dreaded having to do
the work; Sergio has a good starting set of records.

-Don

---

From: Bob Jackson <[email-masked]>
Date: Wed, 21 Dec 94 14:48:01 EST

w.r.t.

***************************************************************

I don't see the reason for a separate database stored at CERN.
CERN should be just like STScI, NRAO, ... and download the
database from STScI and reformat it.

***************************************************************
w.r.t.

***************************************************************

One of the virtues of the master database is that it allows each
site to pick their look and feel.  In particular there is no
agreement on whether or not to display the Description of each
resource.  A site may also want to have separate listings sorted
different criteria, e.g,. one by Category and one by Domain.

I think it is more important that each site list all the
available resources, which include:

http://anarky.stsci.edu/astroweb/search-master.html
http://anarky.stsci.edu/brokers/AstroWeb/query.html
http://anarky.stsci.edu/cgi-bin/search-master
http://anarky.stsci.edu/astroweb/astrowebdead.html
http://anarky.stsci.edu/astroweb/astrowebflake.html

as well as others at NRAO.

I don't see the real need for a LOOK AND FEEL.
I am more concerned about COMPLETENESS.

***************************************************************
w.r.t.

  

***************************************************************

I think that the existing tools work fairly well.

The forms at STSCI allow the Level 1 people to modify the
database at any time.

OK, I admit that I should implement the "MODIFY Existing Resource"
feature, but the "EDIT Entire Database" is almost enough.

The forms at NRAO allow other people to submit additions
and changes.

Thus I think no real changes are needed here.

***************************************************************
w.r.t.

***************************************************************

I have been thinking about some sort of change announcement
facility or change tracking facility for the master database.

But alas I have done nothing yet.

***************************************************************

The only distinction between Level 1 and Level 2 is that
Level 1 folks can edit the Master Database.  Otherwise
they all must routinely download the HTML or TCL version
of the Master Database and reformat it to their needs.

We already have tools which can be used to do this
which we could give to any level 2 site.

Level 2 people could not edit their local version
to add or change records (not that Level 1 people
should either).  Any changes must be made to the
Master Database and not to a local HTML file.

				Bob