Subject: Unicode, pattern matching and repertoires [Was Re: SSC and Hangul]

From: Dominic Dunlop <[email-masked]>
Date: Tue, 12 Feb 91 09:16:25 GMT

[Asmus -- please treat this as a formal comment on Unicode 1.0.  I am
making it as an individual, not as a representative of any corporation
or organization.  Please acknowledge receipt.]

[From "SSC and Hangul" dated Feb 11]
> 
> > Let's consider a simple string search operation.  We want to find all
> > occurrences of /l/su/rog/ You will enter LSuRog as a search string
> > although you need to find lSuRog.
> 
Not sure that I understand the subtleties of this example, but anyway...
The internationalization extensions proposed to the traditional UNIX
regular expression mechanism by the draft POSIX 1003.2 shell and tools
standard may help: they allow the definition of arbitrary ``collating
symbols'' which can be called out in match strings and matched as single
units.  Does this solve the problem?  Does it even help?

> By "loose" matching criteria, I
> guess you want to ignore spacing or non-spacing characteristics of
> characters.

You can't ignore this issue.  The proposed POSIX mechanism can handle
non-spacing diacritics.  Essentially, the combination of a spacing
character and one or more diacritical character is defined to be a
single collating symbol.  The current glaring holes in Unicode in
respect of such things are:

1.  Unicode does not define a repertoire.  Actually, I don't think this
    is a problem: those who define sets of collating symbols for -- say
    -- national or corporate use will, in effect, be defining a private
    repertoire.

2.  Unicode defines no conventions for applying non-spacing diacritics
    to spacing characters: 
    
    -- it does not state which spacing characters are valid targets for
        diacritical characters;

    -- it does not adequately define the ordering of spacing and non-
       spacing characters intended to fill a single display position.
       (While the diacritical characters are defined as following the
       spacing character in the data stream; the ordering of multiple
       diacritical marks ``from the center of the base character
       outward'' (Unicode 1.0 Final Review Document, page 3) seems to
       me to leave scope for multiple interpretations.  While an
       ordering by the encoding of the diacritical mark would displace
       ``visual'' considerations with an arbitrary programming
       convention, it would be unambiguous.);

    -- it does not prohibit multiple applications of the same
       diacritical character to a single spacing character;
       
    -- it does not prohibit the application of diacritical marks to a
       base character where the result would have the same appearence
       as a character to which a single Unicode cell has already been
       allocated.

Unless something is done about the second point, everybody will be able
to define their own repertoires -- which is fine -- but they'll all
disagree on the representation of particular characters -- which will
result in anarchy.  Worse, the developers of word-processors and such
will not know which representation of particular characters they should
use in the files created by their products, and consequently will each
go their own way, creating truly horrendous data interchange problems.
This would be a tremendously retrograde step.

-- 
Dominic Dunlop