<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>Problem with Unicode is multiplied with a fact how most users never really dealt with international fonts. People are usually not so good at solving problems they never really felt. </div><div><br></div><div>My mothers tongue, OTOH, is written in two scripts and Unicode is only way to cover all of it. All Western European languages and most European and near-European languages are covered with single ISO-8859-x 8-bit sets. My language needs two of these - ISO-8859-2 for Latin script, and and ISO-8859-5 for Cyrillic one.</div><div><br></div><div>Also, I don’t think fixed bitwidth of characters is crucial, as we can cover all uses with good abstraction. Maybe like this one.</div><div><br></div><div>(* (C) 2013 Dragiša Durić, <a href="mailto:dragisha@m3w.org">dragisha@m3w.org</a></div><div>*)</div><div>INTERFACE UTF8;</div><div><br></div><div>IMPORT RefSeq;</div><div><br></div><div>(* UTF8.T is a subtype of TEXT, and it is also a UTF8 encoded Unicode string. </div><div>*)</div><div>TYPE</div><div> Char = CARDINAL;</div><div> </div><div> T <: Public;</div><div> Public = TEXT OBJECT </div><div> METHODS</div><div> init(t: TEXT): T;</div><div><br></div><div> isValid(): BOOLEAN; (* so I can do all this without exceptions *)</div><div><br></div><div> length(): CARDINAL; (* in glyphs *)</div><div> byteSize(): CARDINAL; (* in CHARs/bytes *)</div><div> empty(): BOOLEAN; (* shorter than ".length() = 0" *)</div><div><br></div><div> (* hash(): Word.T;</div><div> No need for this neither here nor at all. Text.Hash would be good enough, when I come to this.</div><div> *)</div><div> </div><div> sub(start: CARDINAL; length: CARDINAL := LAST(CARDINAL)): T;</div><div> getText(start: CARDINAL := 0; length: CARDINAL := LAST(CARDINAL)): TEXT;</div><div> getChar(pos: CARDINAL): Char;</div><div> setChars(VAR a: ARRAY OF Char);</div><div><br></div><div> pos(pat: T; start: CARDINAL := 0): INTEGER;</div><div> (* Uses Boyer-Moore [1] for fast search, Observations 1 & 2 are currently implemented.</div><div><br></div><div> [1] Boyer, Robert S.; Moore, J Strother (October 1977). "A Fast String Searching Algorithm.".</div><div> Comm. ACM (New York, NY, USA: Association for Computing Machinery) 20 (10): 762–772.</div><div> *)</div><div> findChar(ch: Char; start: CARDINAL := 0): INTEGER;</div><div> findCharR(ch: Char; start: CARDINAL := LAST(CARDINAL)): INTEGER;</div><div> (* findChar returns first position of ch to the right from start position, or start if ch is there.</div><div> findCharR returns first position of ch to the left from start position, excluding start position.</div><div> *)</div><div> </div><div> iterate(start: CARDINAL := 0; steps: CARDINAL := LAST(CARDINAL)): Iterator;</div><div> END;</div><div> (* All positional/count/length parameters for methods are in Unicode glyphs</div><div> *)</div><div><br></div><div> Iterator <: PublicIterator;</div><div> PublicIterator = OBJECT</div><div> METHODS</div><div> next(VAR char: Char): BOOLEAN;</div><div> (* TODO prev? *)</div><div> END;</div><div><br></div><div>(* Construction</div><div>*)</div><div>PROCEDURE New(t: TEXT): T;</div><div><br></div><div>PROCEDURE Cat(u, t: T): T;</div><div><br></div><div>PROCEDURE FromChars(READONLY chars: ARRAY OF Char): T;</div><div><br></div><div>PROCEDURE FromCHARArray(VAR chars: ARRAY OF CHAR): T;</div><div><br></div><div>(* Validation. Checks both NIL value, and invalid UTF8 string.</div><div>*)</div><div>PROCEDURE IsValid(t: T): BOOLEAN;</div><div><br></div><div>(* Comparation/ordering</div><div>*)</div><div>PROCEDURE Equal(u, t: T): BOOLEAN;</div><div><br></div><div>PROCEDURE Compare(t1, t2: T): [-1..1];</div><div><br></div><div>(* Future UTF8Ops, here for now:</div><div>*)</div><div>PROCEDURE EscapeS(t, s: T; escapeWith: CHAR := '\134'): T;</div><div><br></div><div>PROCEDURE UnEscape(t: T; esc: CHAR := '\134'): T;</div><div><br></div><div>PROCEDURE SplitS(t, s: T; skipSucc: BOOLEAN := TRUE): RefSeq.T;</div><div>(* Treating escaped chars like normal ones. We need to define semantics for special treatment</div><div> Fri Apr 19 12:00 2013: For now, I only implement skipSucc=TRUE case. </div><div>*)</div><div><br></div><div>PROCEDURE RemoveSpaces(t: T): T;</div><div><br></div><div>PROCEDURE Caps(t: T): T; (* This is probably titlecase in Unicode-speak. CHECK. Also, see what happend with Lows() in case we treat titlecase *)</div><div> </div><div>PROCEDURE Lows(t: T): T;</div><div> </div><div>END UTF8.</div><div apple-content-edited="true"><br class="Apple-interchange-newline">
</div>
<br><div><div>On 01 Dec 2013, at 01:16, Hendrik Boom <<a href="mailto:hendrik@topoi.pooq.com">hendrik@topoi.pooq.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-size: 18px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Sat, Nov 30, 2013 at 01:59:47PM -0600, Rodney M. Bates wrote:<br><blockquote type="cite"><br><br>On 11/30/2013 11:29 AM, Hendrik Boom wrote:<br><blockquote type="cite">On Sat, Nov 30, 2013 at 10:52:44AM -0600, Rodney M. Bates wrote:<br><blockquote type="cite">Another devilish detail to be aware of: UTF-16 is _not_ the same as<br>the current Modula-3 16-bit WIDECHAR, even when restricted to values<br><= 16_FFFF. Current Wr/Rd library code just writes/reads<br>exactly 16 bits in two bytes, with whatever code point is in the<br>WIDECHAR variable.<br><br>In contrast, UTF-16 will encode code points greater than<br>UFFFF as a pair of 16-bit code units with surrogate values in them.<br>Then to make this work right, the surrogate values are not<br>allowed in unencoded variables. So attempting to encode a surrogate<br>in UTF-16 is an error, and decoding a surrogate that is not part of a<br>proper first-surrogate/second-surrogate pair is "ill formed" and usually<br>decodes to UFFFD.<br><br>You could get by with treating these as interchangeable only be being<br>careful to ensure there is never either a surrogate code nor a code<br>point > UFFFF, in either input or output.<br><br>Also, current Wr/Rd always write/read only in little-endian byte order,<br>whereas there are both little- and big-endian variants of UTF-16.<br>I have no idea which endianness of UTF-16 is used by various GUI<br>libraries, but it would have to be little for this to work.<br></blockquote><br>It lools as if one might as well use UTF-8 if one is going to consider UTF-16.<br></blockquote><br>Hmm. Actually, *if* one could live with the restrictions on values above,<br>passing the same strings back and forth, with the GUI considering them UTF-16LE<br>and the Modula-3 app code considering them cm3's 16_bit WIDECHAR, would have<br>the advantage that the M3 app code could deal naturally in characters, rather<br>than varying numbers of fragments of characters. UTF-8 would require<br>the latter.<br></blockquote><br>And then we just wait for the potential user who can't, and we'll have<span class="Apple-converted-space"> </span><br>this discussion all over again.<br><br>With the disadvantage that we'll end up having to put still more<span class="Apple-converted-space"> </span><br>mechanisms for handling text everywhere.<br><br>-- hendrik<br><br><br><blockquote type="cite"><br><br><blockquote type="cite"><br>I looked up XIM on Wikipedia (<a href="http://en.wikipedia.org/wiki/X_Input_Method">http://en.wikipedia.org/wiki/X_Input_Method</a>).<br>and it referred to newer systems, SCIM, uim, and IIMF. IIMF ppears to have<br>been superseded by SCIM, I don't know the status of uim, except that<br>it has a uim bridge.<br><br>It does look as if SCIM<br>(<a href="http://en.wikipedia.org/wiki/Smart_Common_Input_Method">http://en.wikipedia.org/wiki/Smart_Common_Input_Method</a>) is intended<br>as a simple way to interface to many other input methods, such as XIM.<br>It may be worth a look.<br><br>--- hendrik</blockquote></blockquote></div></blockquote></div><br></body></html>