AI22-0161-1

!standard B.6(0)                                       26-05-11  AI22-0161-1/02

!class Amendment 26-02-11

!status No Action  11-0-0   26-05-13

!status work item 26-02-11

!status received 26-02-11

!assigned author Tucker Taft

!submitter Tucker Taft

!priority High

!difficulty Hard

!subject Provide Unicode and universal Text interfaces

!summary

This AI proposes an Interfaces.Unicode subsystem and an Ada.Unicode_Text subsystem that together provide easy access to unicode-based encodings of various sorts, and a universal Text type that is intended to be widely usable in place of the String/Wide_String/Wide_Wide_String triplet of types defined in package Standard.

!issue

Many languages are struggling with the transition to a world where Unicode is used more widely. One impetus behind Unicode these days is the ever growing number of emojis and miscellaneous symbols that have been assigned Unicode character positions, often of more than 16 bits, so requiring full 21-bit unicode support.

Ada has Wide_Characters, Wide_Strings, Wide_Wide_Characters, and Wide_Wide_Strings, but these require a lot of advance planning, and a decision between Wide_ and Wide_Wide_, both of which are annoying if the need for more Unicode support comes during maintenance of some existing program. Ada also has the UTF_Encoding package, but this results in "hiding" a UTF-8 string inside a Standard.String, which can be easier to introduce "after the fact," but then muddies the waters in terms of whether any given String is a sequence of Latin-1 characters, or a sequence of multi-byte UTF-8 encodings.

Should we consider defining a more "universal" string type that interfaces well with encodings such as UTF-8 and UTF-16, and is easily convertible to the string types declared in package Standard? (Yes.)

!recommendation

We propose to define an Interfaces.Unicode subsystem, comprising Interfaces.Unicode itself, and a child package Interfaces.Unicode.Conversions, which provide basic definitions of unicode elements plus conversions to/from the existing String types.

In addition, we propose to define an Ada.Unicode_Text subsystem, comprising Ada.Unicode_Text where the type Text is declared, plus child packages Ada.Unicode_Text.Conversions, Ada.Unicode_Text.UTF_Conversions, and Ada.Unicode_Text.Graphemes which provide, respectively, conversions to/from the standard string types, conversions to/from UTF-8 and UTF-16, and iteration through a Text by grapheme cluster (a set of Unicode code points that together form a user-visible "character").

!wording

This is a preliminary wording section, comprising mostly just a possible set of packages making up the Interfaces.Unicode and Ada.Unicode_Text subsystems.

Add the following subclause after B.5:

B.6 Interfacing with Unicode

The language-defined subsystem Interfaces.Unicode comprises two library packages, Interfaces.Unicode, and Interfaces.Unicode.Conversions.

Interfaces.Unicode is a follows:

package Interfaces.Unicode is
   --  Exception which can be raised for certain conversions
   Character_Error : exception;
   --  Define which Wide_Wide_Characters are valid for Unicode
   subtype Code_Point is Wide_Wide_Character
      range Wide_Wide_Character'First .. Wide_Wide_Character'Val (16#10_FFFF#);
   subtype Unicode_Scalar_Value is Wide_Wide_Character
      with Static_Predicate =>
             Unicode_Scalar_Value in Code_Point
                --  incorporate range check into predicate
                --  so Character_Error raised on failure
               and then
             Unicode_Scalar_Value not in
                 Wide_Wide_Character'Val (16#D8000#) ..
                     Wide_Wide_Character 'Val (16#DFFF#),
           Predicate_Failure => raise Character_Error;
   subtype Unicode_String is Wide_Wide_String
      with Dynamic_Predicate =>
            (for all C of Unicode_String => C in Unicode_Scalar_Value),
           Predicate_Failure => raise Character_Error;
end Interfaces.Unicode;

Interfaces.Unicode.Conversions is as follows:

Add the following subclause after A.4.12

A.4.13 Unicode Text

The language-defined subsystem Ada.Unicode_Text comprises four library packages, Ada.Unicode_Text, Ada.Unicode_Text.Conversions, Ada.Unicode_Text.UTF_Conversions, and Ada.Unicode_Text.Graphemes.

The package Ada.Unicode_Text is as follows:

The package Ada.Unicode_Text.Conversions is as follows:

package Ada.Unicode_Text.Conversions is
   --  Conversion To_String might fail because the code-point value >= 2**8.
   --  If a Replacement is not provided, Character_Error will be raised.
   function To_String (T : Text) return String;
   function To_String (T : Text; Replacement : Character) return String;
   function From_String (Str : String) return Text;
   --  Conversion might fail because the code-point value >= 2**16.
   --  If a Replacement is not provided, Character_Error will be raised.
   function To_Wide_String
     (T : Text) return Wide_String;
   function To_Wide_String
     (T : Text; Replacement : Wide_Character) return Wide_String;
   function From_Wide_String (Str : Wide_String) return Text;
   --  Provide the to/from operations for Wide_Wide_String
   --  by renaming Unicode conversions.
   --  Can check before doing the conversion to see if
   --  the Wide_Wide_String satisfies predicate of Unicode_String.
   function To_Wide_Wide_String
     (T : Text) return Wide_Wide_String
      renames To_Unicode;
   function From_Wide_Wide_String
     (Str : Wide_Wide_String) return Text
     renames From_Unicode;
end Ada.Unicode_Text.Conversions;

The package Ada.Unicode_Text.UTF_Conversions is as follows:

with Interfaces.Unicode.Conversions;
use  Interfaces.Unicode.Conversions;
package Ada.Unicode_Text.UTF_Conversions is
   function To_UTF8 (T : Text) return UTF8_Array;
   function From_UTF8 (UTF : UTF8_Array) return Text;

   function To_UTF16 (T : Text) return UTF16_Array;
   function From_UTF16 (UTF : UTF16_Array) return Text;

   --  Provide the UTF32 conversions by renaming Unicode_String conversions.
   function To_UTF32 (T : Text) return Interfaces.Unicode.Unicode_String
      renames Ada.Unicode_Text.To_Unicode;
   function From_UTF32 (UTF : Interfaces.Unicode.Unicode_String)
     return Text
      renames Ada.Unicode_Text.From_Unicode;
   --  Unicode_String is being treated as a synonym for a UTF32 array.
   --  Its Dynamic_Predicate checks that all characters are valid.
end Ada.Unicode_Text.UTF_Conversions;

The package Ada.Unicode_Text.Graphemes is as follows:

!discussion

In Interfaces.Unicode, we have proposed subtypes of Wide_Wide_Character and of Wide_Wide_String with predicates that ensure that each character is a valid Unicode code point. We augment this with a child package for conversions to/from the UTF-8 and UTF-16 encodings.

In Ada.Unicode_Text we have proposed a type Text that can represent an arbitrary string of Unicode "scalar values" representing any legal Unicode string of characters. Within this basic package we provide a simple read-only iterator over Text by code point. We do not provide direct indexing by code point, as the expectation is that Text may be UTF-8 based, so indexing by code point is likely very inefficient. Converting to a Unicode_String is best if indexing by code point is desired.

In a child package Ada.Unicode_Text.Conversions, we provide basic conversion functions to/from the string types defined in package Standard. In a child package Ada.Unicode_Text.Graphemes, we provide an iterator by grapheme cluster, a sequence of code points that together form a user-recognizable "character". We have also provided a child package Ada.Unicode_Text.UTF_Conversions which provides conversions of Text to/from the UTF-8 and UTF-16 encodings.

It would make sense to provide further child packages that provide many of the string manipulation primitives provided by such packages as Ada.Strings.Bounded and Ada.Strings.Fixed. We have not chosen to do that in this AI, as this AI is focused on the most basic capabilities. An essentially infinite set of useful string manipulation primitives should be able to be constructed on top of these two subsystems.

One simple thing that would make a lot of sense is to add a function to Ada.Strings.Text_Buffers which, given a text buffer, returns an Ada.Unicode_Text.Text. This way a text buffer can be used to build up a Unicode string incrementally, and then it can be converted to a Text type for passing around to various operations, presuming we move toward having most interfaces take a Text rather than the three different {Wide_}*String types. Similarly, we should add a Put_Text procedure to Ada.Strings.Text_Buffers, to add a Text to a text buffer.

In earlier discussions we combined the subprograms in Interfaces.Unicode with those of Ada.Unicode_Text, but it seems preferable to keep them separate, as some users are interested in Unicode but might have no interest in a "universal" string of some sort. Similarly, it is possible to be interested in universal strings, without any particular need to use Unicode representations.

We use the name "Unicode_Text" rather than "Texts" for the package name, as it emphasizes the Unicode orientation, and avoids the awkward English term "Texts".

Note that we declare Text to be of an indefinite type, with unknown discriminants. This means it can be implemented without a level of indirection and without finalization, which should make it usable with more restricted run-time environments. If there is a desire to get something closer to the current Unbounded_String, the general support for "unbounded" objects proposed in AI22-0148-1 is designed to provide that for a type like this.

!example

Here are some simple examples using Unicode_Text.Text, imagining we add a 'Text_Image attribute. (Good examples are TBD!)

with Ada.Unicode_Text; use Ada.Unicode_Text;
with Ada.Unicode_Text.Conversions;

use  Ada.Unicode_Text.Conversions;
function Message (Y : String; Z : Wide_String) return Text is
   X : constant Text :=
     "Y = " & From_String (Y) & ", Z = " & From_Wide_String (Z);
begin
   return X & "; Message length = " & Length (X)'Text_Image;
end Message;

These examples might be nicer with interpolated strings (see ARG GitHub issue #37).

!ACATS test

Presumably mostly C tests to test each interface.

!appendix

This is based on ARG GitHub issue #152, which is in turn based on ARG GitHub issue #40.

[a]Why are we making new types here when we already have similar types and operations in Ada.Strings.UTF_Encoding??

[b]UTF_Encoding should be moved to Annex J, I believe.  It uses subtypes of Standard.String and Standard.Wide_String which pretty much defeats the whole effort to properly handle UTF8 and UTF16.

[c]I think this is overkill, I'll explain why in the !appendix of AI22-0161-2.

[d]Your reasoning should be in the !discussion somewhere.

[e]I don't quite understand why we need this to have unknown discriminants? I guess it behaves more like a String that way, but maybe it should be closer to (Un)bounded_Strings?

[f]The intent is to *not* bury a level of indirection and finalization within the Text type, since finalization imposes its own overhead and is restricted in some runtimes.  The AI on Unbounded objects is intended to work in conjunction with this one to allow something approximating an Unbounded_String.

[g]If you have AI22-0148-1's mechanism, then all one needs is a unconstrained array component. There is no important reason to expose that in the private part, and in particular there are no discriminants here at all. Moreover, the reason for using (<>) is to prevent default initialization, and that is unnecessary and inappropriate for these types.

[h]Can we not also have overloads that take String (and (Wide_)Wide_), and not have to therefore do all that weird "& From_String (X) &" ugliness?

[i]The problem with adding overloads to "&" is that it makes string literals ambiguous.  You would need some kind of "preference" rule to prevent the ambiguity, and that just further muddies the waters.

[j]Right. It's the overloads that prevent adding string literals to the existing Ada.Strings.Unbounded. That's a mistake we cannot repeat!