Subject: HTML checker in AWK (source)

From: [email-masked] (Henry Churchyard) (by way of [email-masked] (Hans-Martin Adorf))
Date: 18 Oct 1994 19:01:03 -0500

Here's an HTML-checking program in awk.  It doesn't pretend to
understand all of SGML, but it's simple and easy to use, can be run
on any platform where awk is available, and gives _lots_ of
information (including about some stylistically bad practices):

#htmlchek.awk: Syntactically checks HTML files for a number of possible errors.
#
# Typical use:
#
#   awk -f htmlchek.awk infile.html > outfile.check
#
#   This program is written in the ``awk'' programming language (on Sun systems
# and some others, non-archaic ``awk'' is called ``nawk''; also, a freely-
# redistributable ``awk'' interpreter called ``gawk'' is available for most
# platforms, and as source from the FSF GNU project).  It checks for quite a
# number of possible defects in the HTML (Hyper-Text Mark-up Language) SGML
# files used on the World-Wide Web.
#
# Definite syntactic errors are signaled one per line, using the string
# "ERROR!".  Stylistically deprecated HTML coding is signaled by "Warning!".
# (However, I don't bother checking for some common tricks such as using
# headings in lists.)  SGML constructs outside the scope of HTML proper are not
# checked for, and can result in error messages.  At the end of the output,
# diagnostics are generated as to the tags used in the file and the options
# used with each tag, along with possible additional global warnings (these
# final diagnostics/warnings can be longer than 80 columns).  Note that
# checking the same HTML file twice with this program can result in a different
# ordering of the final tag diagnostics, due to the indeterminacy of the awk
# "for (x in array) {...}" looping construct.
#
# As with almost any parser or lint-type program that doesn't just give up
# at the first error, the presence of one real error can generate a cascade of
# subsequent bogus errors.  The only remedy is to fix the first error (which is
# the real one), and run the check again.  In particular, if you get a "nesting
# underflow" error (i.e. at some point the number of </...> tags, of all types,
# that have been encountered is more than the number of all pairing non-</...>
# tags so far), then most subsequent error messages are unlikely to be helpful.
#
# Copyright H. Churchyard 1994 -- freely redistributable.  This code is
# functional but not very well commented (and readability has not been improved
# by pushing all lines which would overflow 80 columns flush left) -- sorry!
# If you find an error in this program, e-mail me at [email-masked]
# (but please don't ask me what the errors and warnings mean -- they should be
# fairly self-evident, assuming a familiarity with the basic HTML language
# documentation; one warning that may be obscure, "Jump from H0", means that
# the first heading in the file is not at level <H1>; also, "underflow" means
# that more tokens of </x> than <x> have been encountered).
#
# Version 1.0 10/15/94 -- Tested with gawk 2.15 and nawk on SunOS, gawk 2.14
# and nawk on DEC Alpha OSF/1, and gawk 2.11 on 16-bit MS-DOS.
#
BEGIN{#List of known HTML tagwords, divided into pairing tags (<X>...</X>) and
      #non-pairing tags (those where <X> occurs without a following </X>).
      #Pairing tags are further classified into list tags, and those tags which
      #do not self-nest.  The status of some tags (e.g. <P>) may change with
      #the evolution of the HTML language.  Edit these lists if necessary, but
      #carefully preserve the existing format (``propertyarray["TAG"]=1;'').
#
#Non-pairing:
#
unpair["!--"]=1;unpair["BASE"]=1;unpair["BR"]=1;unpair["DD"]=1;
unpair["DT"]=1;unpair["HR"]=1;unpair["IMG"]=1;unpair["INPUT"]=1;
unpair["ISINDEX"]=1;unpair["LI"]=1;unpair["LINK"]=1;unpair["NEXTID"]=1;
unpair["OPTION"]=1;unpair["P"]=1;unpair["PLAINTEXT"]=1;
#
#Pairing:
#
pair["A"]=1;pair["ADDRESS"]=1;pair["B"]=1;pair["BLOCKQUOTE"]=1;pair["BODY"]=1;
pair["CITE"]=1;pair["CODE"]=1;pair["DFN"]=1;pair["DIR"]=1;pair["DL"]=1;
pair["EM"]=1;pair["FORM"]=1;pair["H1"]=1;pair["H2"]=1;pair["H3"]=1;
pair["H4"]=1;pair["H5"]=1;pair["H6"]=1;pair["HEAD"]=1;pair["HTML"]=1;
pair["I"]=1;pair["KBD"]=1;pair["KEY"]=1;pair["LISTING"]=1;pair["MENU"]=1;
pair["OL"]=1;pair["PRE"]=1;pair["SAMP"]=1;pair["SELECT"]=1;pair["STRONG"]=1;
pair["TEXTAREA"]=1;pair["TITLE"]=1;pair["TT"]=1;pair["U"]=1;pair["UL"]=1;
pair["VAR"]=1;pair["XMP"]=1;
#
#Deprecated:
#
deprecated["PLAINTEXT"]=1;deprecated["XMP"]=1;deprecated["LISTING"]=1;
deprecated["KEY"]=1;deprecated["COMMENT"]=1;
#
#Allowed in <head>...</head> element:
#
inhead["!--"]=1;inhead["BASE"]=1;inhead["ISINDEX"]=1;inhead["LINK"]=1;
inhead["NEXTID"]=1;inhead["TITLE"]=1;inhead["HEAD"]=1;
#
#Lists (all <LI> must be first order daughter of these and vice versa):
#
list["DIR"]=1;list["MENU"]=1;list["OL"]=1;list["UL"]=1;
#
#List that does not involve <LI>:
#
nonlilist["DL"];
#
#Pairing but non-self-nesting:
#
nonnest["A"]=1;nonnest["ADDRESS"]=1;nonnest["B"]=1;nonnest["BLOCKQUOTE"]=1;
nonnest["CITE"]=1;nonnest["CODE"]=1;nonnest["DFN"]=1;nonnest["EM"]=1;
nonnest["FORM"]=1;nonnest["H1"]=1;nonnest["H2"]=1;nonnest["H3"]=1;
nonnest["H4"]=1;nonnest["H5"]=1;nonnest["H6"]=1;nonnest["HTML"]=1;
nonnest["I"]=1;nonnest["KBD"]=1;nonnest["LISTING"]=1;nonnest["PRE"]=1;
nonnest["SAMP"]=1;nonnest["SELECT"]=1;nonnest["STRONG"]=1;nonnest["TEXTAREA"]=1;
nonnest["TITLE"]=1;nonnest["TT"]=1;nonnest["U"]=1;nonnest["VAR"]=1;
nonnest["XMP"]=1;
#
#-=-  -=- Don't edit below this line unless you know what you're doing -=-  -=-
#
#nonnest["BODY"]=1;nonnest["HEAD"]=1; #Separate checks for these
state=0;continuation=0;blank=0;nestvar=0;htmlwarn=0;maxlist=0;headone=0;
headlevel=0;br=0}
#
# Main
#
{lastbeg=0;currsrch=1;if (("HTML" in lev)&&(lev["HTML"]>0)) {htmlwas=1};
 while (match(substr($0,currsrch),/[<>]/)!=0)
      {currsrch=(currsrch+RSTART);#-1
       if (substr($0,(currsrch-1),1)=="<")
         {if (state) {print "Multiple `<' without `>' ERROR! at line",NR};
          lastbeg=currsrch;state=1}
        else {if (substr($0,(currsrch-1),1)==">")
                {if (state==0)
                   {print "`>' without `<' ERROR! at line",NR}
                  else {parsetag(currsrch-1);
                        if ((lasttag=="!--")&&(lastcomt!="--"))
{print "!-- comment not terminated by \042--\042 ERROR! at line",NR};
                        if (inquote)
{print "Malformed tag option ERROR! at line",NR,"on tag",lasttag};
                        state=0;continuation=0;blank=0}}
               else {print "Internal error at line",NR,"ignore"}}};
 if (((state==1)&&(blank==0))||((lastbeg==0)&&(continuation==1)))
   {parsetag(length($0)+1);continuation=1};
if ((!htmlwarn)&&(($1!="")&&($1!="<")&&($1!~"^>"))&&((!("HTML" in
lev))||(lev["HTML"]==0)))
   {if (!htmlwas)
{print "Non-whitespace found outside <HTML>...</HTML> Warning! at line",NR;
       htmlwarn=1}};
 htmlwas=0;
 if ($0~/&/)            # Don't actually check against the list of &xxx; codes.
   {gsub(/&[A-Za-z][a-z]*;/,"");gsub(/&#[0-9][0-9]*;/,"");
    if ($0~/&/)
      {print "Apparent non-complying ampersand code ERROR! at line",NR}}}
#
#
# parsetag() communicates with main() through these global variables:
# - lastbeg (zero if no `<' ocurred on line, otherwise points to character
#   immediately after the last `<' encountered).
# - state (one if unresolved `<', zero otherwise).
# - continuation (one if unresolved `<' from previous line, zero otherwise),
# - blank (one if continuation and no tag-word encountered yet).
# - inquote (one if inside option quotes <tag opt="....">).
#
function parsetag(inp)
{if (!lastbeg) {lastbeg=1};
 #print substr($0,lastbeg,(inp-lastbeg));#XXdebugging#
 numf=split(substr($0,lastbeg,(inp-lastbeg)),arr);
 if (numf==0)
   {if (substr($0,(lastbeg-1))~/<[ \t]*$/)
      {continuation=1;blank=1;state=1;return}
     else {if (substr($0,1,inp)~/^[ \t]*>/)
             {if (continuation&&blank)
                {print "Multiline Blank <> ERROR! at line",NR};
              continuation=0;blank=0;state=0;return}
            else {if (!continuation)
                    {print "Blank <> ERROR! at line",NR;
                     continuation=0;blank=0;state=0;return}}}}
  else {if ((!continuation)||(continuation&&blank))
          {arr[1]=upcase(arr[1]);if (arr[1]~/^!--/) {raw=arr[1];arr[1]="!--"}
                                   else {raw=""};
           lasttag=arr[1];
           if (arr[1]~/^\//)                                       #</TAG> found
             {sub(/^\//,"",arr[1]);--lev[arr[1]];
              if (lev[arr[1]]<0)
                {print "Underflow ERROR! at line",NR, "on tag /" arr[1]};
              if (arr[1] in pair)
                {if (nestvar<=0)
{print "Nesting underflow ERROR! at line",NR,"on tag /" arr[1]}
                   else {if (nestarr[nestvar]!=arr[1])
{print "Nesting ERROR! at line",NR,": /" nestarr[nestvar],"expected, /"
arr[1],"found"}};
                 if ((nestarr[nestvar] in list)&&(!isli[nestvar]))
{print "Empty list (without <LI>) ERROR! at line",NR,"on tag /" arr[1]};
                 --nestvar}
               else {revusarr[arr[1]]=1};
                 if ((arr[1] in list)||(arr[1] in nonlilist))
                    {--listdep}}
            else {if ((arr[1]=="HEAD")||(arr[1]=="BODY"))           #<TAG> found
                    {hbwarn=0;if ((!("HTML" in lev))||(lev["HTML"]==0))
{print "HEAD or BODY outside of <HTML>...</HTML> ERROR! at line",NR};
if ((("HEAD" in lev)&&(lev["HEAD"]>0))||(("BODY" in lev)&&(lev["BODY"]>0)))
{print "HEAD or BODY nested inside HEAD or BODY element ERROR! at line",NR}}
                   else {if ((!hbwarn)&&(lasttag!="!--"))
{if ((!(("BODY" in lev)&&(lev["BODY"]>0)))&&(!(("HEAD" in
lev)&&(lev["HEAD"]>0))))
                                  {if (("HTML" in lev)&&(lev["HTML"]!=0))
{print "Tag outside of HEAD or BODY element Warning! at line",NR,"on
tag",arr[1];
                                     hbwarn=1}}}};
                  ++lev[arr[1]];
if ((arr[1]!="LI")&&(nestarr[nestvar] in list)&&(!isli[nestvar]))
{print "Tag in list occurred before first <LI> ERROR! at line",NR,"on
tag",arr[1]};
                  if (arr[1] in pair)
                    {++nestvar;nestarr[nestvar]=arr[1];isli[nestvar]=0};
                  if ((arr[1] in list)||(arr[1] in nonlilist))
                    {++listdep;if (listdep>maxlist) {maxlist=listdep}};
                  if (arr[1]=="LI")
                    {isli[nestvar]=1;
                     if (!(nestarr[nestvar] in list))
                       {print "<LI> outside of list ERROR! at line",NR}};
                  if (("HEAD" in lev)&&(lev["HEAD"]>0))
                    {if (!(arr[1] in inhead))
{print "Disallowed tag in <head>...</head> ERROR! at line",NR,"on tag",arr[1]}};
                  if (arr[1]~/^H[1-6]$/)
                    {newheadlev=substr(arr[1],2,1);
                     if (newheadlev>(headlevel+1))
{print "Warning! Jump from header level H" headlevel, "to level H"
newheadlev,"at line",NR};
                     headlevel=newheadlev;
                     if (headlevel==1)
                       {if (headone==1)
                          {print "Multiple <H1> headings Warning! at line",NR};
                        headone=1}};
                  if (arr[1]=="HTML") {htmlwarn=0;hbwarn=0;htmlwas=1;headone=0};
                  if ((arr[1] in nonnest)&&(lev[arr[1]]>1))
{print "Self-nesting of unselfnestable tag ERROR! at line",NR,"on tag",arr[1]}};
           blank=0;startf=2;usarr[arr[1]]=1;inquote=0}
         else {startf=1};
        if (lasttag!="!--")          #Remainder of stuff in <...> after tag word
          {for (i=startf;i<=numf;++i)
              {if (!inquote)
#((arr[i]~/^[^=\042]+=\042[^=\042]*\042$/)||(arr[i]~/^[^=\042]+(=(\042)?[^=\042]*)?$/))
{if
((arr[i]~/^[^=\042]*(=\042[^=\042]*\042)?$/)||(arr[i]~/^[^=\042]*=(\042)?[^=\042]*$/))
                   {split(arr[i],arr2,"=");
                    if (arr2[1]=="")
{print "Null tag option ERROR! at line",NR,"on tag",lasttag}
                      else {arr2[1]=upcase(arr2[1]);
                            optarr[lasttag,arr2[1]]=1};
                    if (gsub(/\042/,"",arr[i])==1)
                      {inquote=1}}
                  else
{print "Malformed tag option ERROR! at line",NR,"on tag",lasttag}}
                else {if (arr[i]~/\042/)
                        {inquote=0;
                         if (arr[i]!~/^[^\042]*\042$/)
{print "Malformed tag option ERROR! at line",NR,"on tag",lasttag}}}}}
         else {if ((numf==1)&&(!continuation)) {sub(/!--/,"",raw);arr[1]=raw};
               if (arr[numf]~/--$/)
                 {lastcomt="--"}
                else {lastcomt=""}};
        return}}
#
# Uppercasing routine; in GAWK can replace upcase() with built-in function
# toupper() for a speed boost.
#
BEGIN{
upc["a"]="A";upc["b"]="B";upc["c"]="C";upc["d"]="D";upc["e"]="E";upc["f"]="F";
upc["g"]="G";upc["h"]="H";upc["i"]="I";upc["j"]="J";upc["k"]="K";upc["l"]="L";
upc["m"]="M";upc["n"]="N";upc["o"]="O";upc["p"]="P";upc["q"]="Q";upc["r"]="R";
upc["s"]="S";upc["t"]="T";upc["u"]="U";upc["v"]="V";upc["w"]="W";upc["x"]="X";
upc["y"]="Y";upc["z"]="Z";
}
#
function upcase(upcins,k)
{if (upcins~/[a-z]/)
   {for (k in upc) {if (upcins~k) {gsub(k,upc[k],upcins)}}};
 return upcins}
#
# End-of-file routine.  Information is passed here through arrays:
# - usarr[x]:    The tag <x> was used.
# - revusarr[x]: The reverse tag </x> was used.
# - lev[x]:      Current degree of self-nesting of paired tag <x>...</x>.
# - optarr[x,y]: The option y was used with tag <x>.
#and also through the variables maxlist and continuation.
#
END{if (continuation)
      {print "Was awaiting a `>' ERROR! at END"};
    for (x in usarr)
       {if (x in pair)
          if (lev[x]>0)
{print "Pending unresolved <x> without </x> ERROR! of level",lev[x],"at
END on tag",x};
          if (lev[x]<0)
            {print "Residual underflow of level",lev[x],"at END on tag",x}};
if (!("HTML" in usarr)) {print "<HTML> not used in document Warning! at END"};
if (!("HEAD" in usarr)) {print "<HEAD> not used in document Warning! at END"};
if (!("BODY" in usarr)) {print "<BODY> not used in document Warning! at END"};
    for (x in usarr)
       {options="";head=("^" x SUBSEP);
        for (z in optarr)
           {if (z~head)
              {split(z,optx,SUBSEP);
               options=(options " " optx[2])}};
        if (!br) {print "";br=1};
        printf "%s %s %s","Tag",x,"occurred";
        if (options)
          {printf "%s%s",", with options",options};
        if (!((x in pair)||(x in unpair)))
          {printf "; Warning! tag is unknown"};
        if (x in deprecated)
          {printf "; Warning! tag is obsolescent and deprecated"};
        if ((x in unpair)&&(x in revusarr))
{printf ("; Warning! " x " not a paired tag, yet </" x "> encountered")};
        if ((!(x in pair))&&(x in revusarr)&&(lev[x]!=0))
{printf ("; Balance of <" x "> minus </" x "> nonzero ERROR! of level "
lev[x] " at END")};
        printf "\n"};
    if (maxlist) {print "Maximum depth of list embedding was",maxlist}}
#-=-  -=-  -=-  -=-  -=-  -=-  -=-  -=-  -=-  -=-  -=-  -=-  -=-  -=-  -=-  -=-
##EOF

--                                                  _  ___    _   ____
 (<   __|~`---,  <(     Henry Churchyard             |    |    |     |
  )>  \  _ *  |   >)    University of Texas               |  __|  |  |
 (<    `~ \  /   <(                                       .   .    . .
  )> Austin`-' TX >)    [email-masked]           .     .