We report on VLA HI spectral line observations of five early-type galaxies classified as optically peculiar due to the presence of jets, ripples or other optical ``fine structure''. We detect HI within the primary beam (30' HPBW) in four of the five systems. However, in only one case is this gas associated with the targeted elliptical galaxy. In the other cases the HI is associated with a nearby gas-rich disk or dwarf galaxy. The one HI detection is for NGC 7626, where we tentatively detect an HI cloud lying between 20 and 40 kpc southwest of the galaxy center. Its origin is unclear. Our failure to detect obvious tidal HI features suggests that if these fine-structure ellipticals are remnants of disk galaxy mergers, either the progenitors were gas poor, or they are well evolved and any gaseous tidal features have dispersed and/or been converted into other phases. Our targeted systems all reside in groups or clusters, and it seems likely that tidal HI is shorter lived in these environments than suggested by studies of more isolated merger remnants.