Subject: Re: Full-text searching: don't keep it simple and stupid!

From: Olle Jarnefors <[email-masked]>
Date: Fri, 17 Jun 94 21:04:07 +0200

Alain LaBont<e'> raises the important question about how
searching should be adapted to the needs of the language of the
searched text/data. The following is what is said about this in
the Nordic report on cultural requirements [1]:

   2.4.4  Searching
   ----------------

   Because the Nordic languages are more complicated than
   English, as far as inflection and formation of compound words
   are concerned, more sophisticated search functions are
   desirable. To be most useful, interactive searching for
   strings or words in a text should be available in three
   modes:

   1. Search for exactly given words

   2. Search for all words consisting of a given string and
      possibly an inflectional ending

   3. Search for the given string, irrespective of its
      surroundings.

   Orthogonal to this, searching should be performed on three
   levels, defined by the treatment of individual letters:

   a) Exact search

   b) Search disregarding the case of letters

   c) Search also disregarding the distinction between letter
      variants which in sorting are treated as the same basic
      letter.

   For effective searching in text, it is also important that
   hyphenated words are dehyphenated, since hyphenation is more
   frequent in the Nordic languages than e.g. English, due to
   the many long compound words.

What is said here about Danish, Faroese, Finnish, Icelandic,
Norwegian, and Swedish is also true for German, Dutch and other
languages, but of course such properties as
-- which basic letters are distinguished
-- which other letters are treated as variants of a basic letter
-- which capital letter or capital string is equivalent to a
   small letter
-- which characters are allowed in a word
-- which inflectional endings are possible
vary between languages.

[1] INSTA:
    Nordic Cultural Requirements on Information Technology
    | Technical Report STRI TS3, 1992. Distributed by: 
      Icelandic Council for Standardization, Keldnaholti,
      IS-112 Reykjavik, Iceland. Phone: +354-1-877 000.
      Fax: +354-1-877 409

--
Olle Jarnefors, Royal Institute of Technology, Stockholm <[email-masked]>