Subject: search and classification

From: [email-masked] (Hans-Martin Adorf)
Date: Wed, 26 Oct 1994 09:47:38 +0100

Dear friends,

I noticed a pecularity of our WAIS index which I should like to
communicate. I was searching the AstroWeb database for ESO and found 143
(!) records, many of which have no relation to ESO other than they have
been submitted by me, and I happen to be sitting at a machine
xxx.hq.eso.org.

While in general I like the idea that the target URLs are being indexed -
that assures completeness of one's search - I am disfavouring the inclusion
of the submitter's host machine. I do not know whether it is possible to
discard that information prior/during indexing, but I hope so.

Also when I inspected my new pages, which have resource records grouped
according to the assigned hidden categories, I noticed quite a number of
misclassifications, which were not present in my old handcrafted page.

I believe that these misclassification will negatively influence the
reception of the AstroWeb database in the community, and I think that it
should be the top-priority item to
(a) revise the classification scheme,
(b) subdivide the database into equal chunks among ourselves and check the
classification (which might include accessing the recorded URL)

Cheers
Hans-Martin

---

From: Bob Jackson <[email-masked]>
Date: Wed, 26 Oct 94 07:26:58 EDT

I suspect that I can eliminate the spurious hits on 'eso' by 
eliminating all html comments in the DESCRIPTION fields.

>>> Does anyone object to eliminating HTML comments from the
>>> descriptions?

			Bob

---

From: [email-masked] (Don Wells)
Date: Thu, 27 Oct 94 14:26:22 EDT

Hans-Martin Adorf writes:
 > .. I was searching the AstroWeb database for ESO and found 143
 > (!) records.. happen to be.. at.. machine > xxx.hq.eso.org.

Sigh... this feature of the WAIS indexing is both a blessing and a curse.
I am reluctant to recommend disabling it, but your particular example
(ESO) definitely demonstrates the curse aspect.

 > .., I noticed quite a number of > misclassifications..

Yes. The quality of classification codes in the AstroWeb database has
declined as the size of the database has grown.

 > reception of the AstroWeb database in the community..

That is a real possiblity.

 > should be.. top-priority item to > (a) revise the classification scheme,

The existing category codes were not planned, but rather are mostly
classifications which various Consortium members had made before
January 94; I invented most of the codes as I merged the five old
databases.  A starting point might be to construct a Web page
containing a paragraph of text for each code to specify the range of
cases that the code will include.  At the present time, the only thing
that we have to document the codes is the menu in the entry form.

 > classification (which might include accessing the recorded URL)

That may be necessary. There are 7 of us, so an equal partition would
be something like 160 records to check. I suggest partitioning the
yp_topzone file.  I am willing to do 160 records, but I would like to
see agreement on classification guidelines and the text describing
each code before I start on my piece of the database.  

I hope that we can also agree on issues like whether there should be
both FTP and HTTP records for the same institution (I say 'no', HTTP
should take precedence).  Likewise, should the WAIS search engine for
a database be listed separately from record for the database itself?
These are style and policy questions, questions which we have mostly
avoided trying to answer because we know that there are a range of
opinions in the Consortium.

                -*-

Improving the classifications is going to be truly hard work in the
AstroWeb project.  Maybe we should recruit some discipline experts to
aid us.  For example, we could advertise for applications to be the
'solar' specialist, or the 'planetary' specialist, two possibilities
which are not currently represented in our group.  Also, I believe
that we could recruit one or more astronomy librarians to join with us
in order to maintain the library-related categories if we just asked
them.

-Don