AI22-0161-2

!standard A.4.13(0)                                    26-07-14  AI22-0161-2/03

!standard 4.10(36/5)

!standard 4.10(40/5)

!standard A.4.12(3/5)

!standard A.4.12(11/5)

!standard A.4.12(24/5)

!standard A.4.12(29/5)

!standard A.4.12(33/5)

!standard A.4.12(34/5)

!class Amendment 26-05-04

!status work item 26-05-04

!status received 26-05-04

!assigned author Randy Brukardt

!submitter Randy Brukardt

!priority High

!difficulty Hard

!subject Universal Unicode Text

!summary

This AI proposes an Ada.Strings.Unicode subsystem that provides a universal Text type that is intended to be a replacement for the String/Wide_String/Wide_Wide_String triplet of types defined in package Standard while still providing access to the entire range of Unicode characters. [No one should ever have to do without the pile of poo emoji again.]

!issue

Many languages are struggling with the transition to a world where Unicode is used more widely. One impetus behind Unicode these days is the ever growing number of emojis and miscellaneous symbols that have been assigned Unicode character positions, often of more than 16 bits, so requiring full 21-bit unicode support.

Ada has Wide_Characters, Wide_Strings, Wide_Wide_Characters, and Wide_Wide_Strings, but these require a lot of advance planning, and a decision between Wide_ and Wide_Wide_, both of which are annoying if the need for more Unicode support comes during maintenance of some existing program. Ada also has the UTF_Encoding package, but this results in "hiding" a UTF-8 string inside a Standard.String, which can be easier to introduce "after the fact," but then muddies the waters in terms of whether any given String is a sequence of Latin-1 characters, or a sequence of multi-byte UTF-8 encodings.

The current situation in Ada is a mess; we need to essentially start over to provide proper string operations from the start.

!recommendation

We propose to define an Ada.Strings.Unicode subsystem, to provide a set of “universal” string operations, along with child packages to provide various less used operations. These include conversions to/from the various existing String types, conversions to/from UTF-8 and UTF-16, operations on grapheme clusters (a set of Unicode code points that together form a user-visible "character"), including iteration, and operations for important Unicode operations such as normalization.

!wording

Create a new subclause, A.4.13 package Ada.Strings.Unicode

A Text string can be used to store text of any type, supporting the entire range of Unicode text.

Static Semantics

The text string library packages have the following declarations:

package Ada.Strings.Unicode
   with Preelaborate, Nonblocking, Global => in out synchronized is

   Character_Error : exception;
      -- Raised when converting an invalid characters

   subtype Unicode_Character is Wide_Wide_Character
      with Static_Predicate =>
             Unicode_Character in Wide_Wide_Character'First ..
                   Wide_Wide_Character'Val (16#10_FFFF#)
                and then
             Unicode_Character not in
                Wide_Wide_Character'Val (16#D800#) ..
                Wide_Wide_Character 'Val (16#DFFF#),
           Predicate_Failure => raise Character_Error;
        -- A valid Unicode "scalar value" (character). The D800..DFFF
        -- range are the parts of two-part UTF-16 characters, which are
        -- not allowed for any other purpose (and thus will never appear
        -- as legitimate characters). Technically, those are valid
        -- Unicode "code points" (that’s the difference between code
        -- points and scalar values).

   type Text is tagged private
      with String_Literal => From_Wide_Wide_String,
           Default_Iterator => Iterate_By_Character,
           Iterator_Element => Unicode_Character,
           Constant_Indexing => Element;
     -- Provides an iterator by character, and character/cursor indexing
     -- for reading. But we do not provide any direct writable access,
     -- including indexing. For most purposes, one should use Text
     -- strings (even for individual characters), as the conversion
     -- to/from Unicode_Character is not free. (It's not particularly
     -- expensive, but still it is best to avoid it.)

     -- The names of most operations here are from
     -- Ada.Strings.Unbounded. Using the existing names has smaller
     -- cognitive overhead (and reduces the work creating this proposal.

   type Text_Cursor is private;
      -- Designates a particular character in a Text string.
      -- Typically, this will be implemented as a Natural (or a similar
      -- type) that indexes into an array representing the storage
      -- representation of a string.

   Empty_Cursor : constant Text_Cursor;
      -- Represents no character in the string. This value is compatible
      -- with any Text string object.

   type Text_Cursor_Range is record
      Low, High : Text_Cursor;
   end record;
      -- If Low is not the empty cursor and High is the empty cursor,
      -- then this is interpreted as a null range. If Low is the empty
      -- cursor, there is no range (for instance, as a "Not Found"
      -- return from Index). Operations do not allow Low to be the empty
      -- cursor.
      -- If one of the proposals for ".." syntax sugar is adopted,
      -- it will be applied here.
   No_Range : constant Text_Cursor := (Low | High => Empty_Cursor);
         -- [Editor’s Note: This is not the same as a null range,

         -- which represents a result that contains no characters.]


  function Is_Valid (Item : in Text; Position : in Text_Cursor)
     return Boolean;
     -- Returns True if Position designates the first unit of the
     -- representation of a character in Item or if Position is the
     -- empty cursor, and return False otherwise (including when
     -- Position is beyond the end of the string).

  function Is_Only_Valid (Item : in Text; Position : in Text_Cursor)
     return Boolean;
     -- Returns True if Position designates the first unit of the
     -- representation of a character in Item, and return False
     -- otherwise (including when Position is beyond the end of the
     -- string). (This is the same as Is_Valid, except that the empty
     -- cursor is not allowed.)
     -- (Author's note: I know some people hate this, and would prefer
     -- the "no empty cursor" part is written out in each precondition,
     -- but I don't want to spend a lot of time figuring out complex
     -- preconditions. If we do adopt this, changing the preconditions
     -- is easy enough.)

  function Add_Characters[a][b][c][d] (Item  : in Text;
                           Position : in Text_Cursor;
                           Count : in Integer)
       return Text_Cursor
       with Pre => Is_Only_Valid (Item, Position)

                      or else Program_Error;
      -- Add Count characters relative to Item to the cursor Position.
      -- (Note: Count can be negative.) Constraint_Error is raised if
      -- the specified character does not exist (is either before the
      -- beginning of the string or after the end).

  function Subtract[e][f][g][h]_Characters (Item : in Text;
                                Position : in Text_Cursor;
                                Count : in Integer)
       return Text_Cursor
       with Pre => Is_Only_Valid (Item, Position)
                     or else Program_Error;
      -- Subtract Count characters relative to Item to the cursor
      -- Position. (Note: Count can be negative.) Constraint_Error is
      -- raised if the specified character does not exist (is either
      -- before the beginning of the string or after the end).

   function First (Item : in Text) return Text_Cursor;
       -- Returns a Text_Cursor for the first character in Item;
       -- if Item is empty, returns Empty_Cursor.

   function Last (Item : in Text) return Text_Cursor;
       -- Returns a Text_Cursor for the last character in Item;
       -- if Item is empty, returns Empty_Cursor.

   function Nth (Item : in Text; N : in Natural) return Text_Cursor;
       -- Returns a Text_Cursor for the Nth character in Item;
       -- if N does not represent a character in Item, then raises
       -- Character_Error.

       -- [Editor’s Note: I did not use a precondition to check
       -- the error case, as that would require doing the operation
       -- twice: once to figure the character length of Item for the
       -- precondition and again to determine the actual cursor.]

   function Next (Item : in Text; Position : in Text_Cursor)
      return Text_Cursor is (Add_Characters (Item, Position, 1));

   function Previous (Item : in Text; Position : in Text_Cursor)
      return Text_Cursor is (Subtract_Characters (Item, Position, 1));

   -- Conversions to string types in Standard:

   function To_Wide_Wide_String (Item : in Text)
      return Wide_Wide_String;

   function From_Wide_Wide_String (Item : in Wide_Wide_String)
      return Text
      with Pre => (for all Char of Item => Char in Unicode_Character)
                   or else raise Character_Error;

   function To_Wide_String (Item : in Text) return Wide_String
       with Pre => (for all Char of Item => Char in Unicode_Character
                      or else raise Character_Error;
       -- The result Wide_String uses UTF-16 encoding when necessary.
       -- If Item only contains BMP characters (those in the range of

       -- Wide_Character), then the result contains no surrogate
       -- (encoded) characters.

   function To_Wide_String (Item : in Text;
                            Substitute : in Wide_Character)
       return Wide_String;
       -- If a character in Item is not in the range of
       -- Wide_Character, then use Substitute instead.

   function From_Wide_String (Item : in Wide_String) return Text;
       -- Item is assumed represented as UTF-16. If characters in Item
       -- cannot be represented in Text, then Character_Error is raised.
       -- This can happen if surrogate characters (16#D800# .. 16#DFFF#)
       -- are not paired properly. Surrogate characters standing alone
       -- are not legal BMP characters.

   -- [Editor’s note: We are treating Wide_String as formatted as

   -- UTF-16, on the recommendation of our Unicode liaison. See the

-- -- !discussion for more on this topic.]

   function To_String (Item : in Text) return String
       with Pre => (for all Char of Item =>
                        Char in Wide_Wide_Character'First ..
                           Wide_Wide_Character'Val(16#FF#))
                        or else raise Character_Error;

   function To_String (Item : in Text; Substitute : in Character)
      return String;
       -- If a character in Item is not in the range of Character,
       -- then use Substitute instead.

   function Is_Empty (Source : in Text) return Boolean;
      -- Returns True if Source is empty.

   function Character_Length (Source : in Text) return Natural;
       -- Returns the length, in characters, of Source.

   function Character_Length (Source : in Text;
                              Rng : in Text_Cursor_Range)
                                   return Natural
      with Pre => (Is_Only_Valid (Source, Rng.Low) and then
                   Is_Valid (Source, Rng.High)) or else Program_Error;
       -- Returns the length, in characters, of Rng relative to Source.

   -- Following are operations to provide basic operations similar
   -- to those natively available for type String.

   procedure Append (Source   : in out Text;
                     New_Item : in Text);

   procedure Append (Source   : in out Text;
                     New_Item : in Unicode_Character);

   function "&" (Left, Right : in Text) return Text;

   function "&" (Left : in Text; Right : in Unicode_Character)
      return Text;

   function "&" (Left : in Unicode_Character; Right : in Text)
      return Text;

   function Element[i][j][k] (Source : in Text;
                    Index  : in Positive)
     return Unicode_Character;
     -- Here, "Index" represents a character count. This
     -- operation is relatively expensive.

  function Element (Source   : in Text;
                    Position : in Text_Cursor)
     return Unicode_Character
     with Pre => Is_Only_Valid (Source, Position)
                 or else Program_Error;

   procedure Replace_Element (Source   : in out Text;
                             Position : in Text_Cursor;
                             By       : in Unicode_Character)
     with Pre => Is_Valid (Source, Position) or else Program_Error;[l][m]

   function Slice (Source : in Text;
                   Rng    : in Text_Cursor_Range) return Text
      with Pre => (Is_Only_Valid (Source, Rng.Low) and then
                   Is_Valid (Source, Rng.High)) or else Program_Error;

   procedure Slice
      (Source : in     Text;
       Target :    out Text;
       Rng    : in     Text_Cursor_Range)
      with Pre => (Is_Only_Valid (Source, Rng.Low) and then
                   Is_Valid (Source, Rng.High)) or else Program_Error;

   function "="  (Left, Right : in Text) return Boolean;

   function "<"  (Left, Right : in Text) return Boolean;

   function "<=" (Left, Right : in Text) return Boolean;

   function ">"  (Left, Right : in Text) return Boolean;

   function ">=" (Left, Right : in Text) return Boolean;

   function Index (Source  : in Text;
                  Pattern : in Text;
                  From    : in Text_Cursor := Empty_Cursor;
                  Going   : in Direction := Forward)
     return Text_Cursor_Range
     with Pre => Is_Valid (Source, From) or else Program_Error;
     -- If From = Empty_Cursor, then we start from the front (or back)
     -- of the string.
[n]
  function Replace_Slice (Source   : in Text;
                          Rng      : in Text_Cursor_Range;
                          By       : in Text)
     return Text
[o][p]      with Pre => (Is_Only_Valid (Source, Rng.Low) and then
                  Is_Valid (Source, Rng.High)) or else Program_Error;

  procedure Replace_Slice (Source   : in out Text;
                           Rng      : in Text_Cursor_Range;
                           By       : in Text)
     with Pre => (Is_Only_Valid (Source, Rng.Low) and then
                  Is_Valid (Source, Rng.High)) or else Program_Error;

  package Iterators is
      --  Iterator over Text by character (code point)

     function Has_Element (C : Text_Cursor) return Boolean is
        (C /= Empty_Cursor);

     package Character_Iterators is
       new Ada.Iterator_Interfaces (Text_Cursor, Has_Element);

     type Iterator_By_Character is

        limited new Character_Iterators.Reversible_Iterator
           with private;

      --  Usage:
      --  for TC in U_Str.Iterator_By_Character loop
      --      --  TC is of type Text_Cursor
      --      Do_Something_With (U_Str(TC));
      --  end loop;
      --  Or, using the default iterator for Text strings:
      --  for UC of U_Str loop --  UC is of subtype Unicode_Character
      --      Do_Something_With (UC);
      --  end loop;

   private
      type Iterator_By_Character is
        limited new Characters.Reversible_Iterator with null record;
         --  not specified by the language, will need a reference to the
         -- Text string being iterated.

      function First (Iter : Iterator_By_Character) return Text_Cursor;
         -- Calls First (Iter.Text);
      function Next (Iter : Iterator_By_Character; Pos : Text_Cursor)
        return Text_Cursor; -- Calls Next (Iter.Text, Pos).
      function Last (Iter : Iterator_By_Character) return Text_Cursor;
         -- Calls Last (Iter.Text);
      function Previous (Iter : Iterator_By_Character;
                         Pos  : Text_Cursor)
        return Text_Cursor; -- Calls Previous (Iter.Text, Pos).
   end Iterators;

   function Iterate_by_Character (Item : in Text)
      return Iterators.Iterator_By_Character;

end Ada.Strings.Unicode;

with Ada.Strings.Wide_Wide_Maps;
package Ada.Strings.Unicode.Operations is

   -- These are the operations from Ada.Strings.Unbounded that

   -- are not in the parent package, suitably modified for type Text.

   -- Search subprograms
   function Index (Source  : in Text;
                   Pattern : in Text;
                   From    : in Text_Cursor := Empty_Cursor;
                   Going   : in Direction := Forward;
                   Mapping : in Wide_Wide_Maps.
                                 Wide_Wide_Character_Mapping)
      return Text_Cursor_Range
      with Pre => Is_Valid (Source, From) or else Program_Error;
      -- If From = Empty_Cursor, then we start from the front (or back)
      -- of the string.

   function Index (Source  : in Text;
                   Pattern : in Text;
                   From    : in Text_Cursor := Empty_Cursor;
                   Going   : in Direction := Forward;
                   Mapping : in not null Wide_Wide_Maps.
                               Wide_Wide_Character_Mapping_Function)
      return Text_Cursor_Range
      with Pre => Is_Valid (Source, From) or else Program_Error;
      -- If From = Empty_Cursor, then we start from the front (or back)
      -- of the string.

   function Index (Source : in Text;
                   Set    : in Wide_Wide_Maps.Wide_Wide_Character_Set;
                   From   : in Text_Cursor := Empty_Cursor;
                   Test   : in Membership := Inside;
                   Going  : in Direction := Forward)
      return Text_Cursor_Range
      with Pre => Is_Valid (Source, From) or else Program_Error;
      -- If From = Empty_Cursor, then we start from the front (or back)
      -- of the string.

   function Index_Non_Blank (Source : in Text;
                             From   : in Text_Cursor := Empty_Cursor;
                             Going  : in Direction := Forward)
      return Text_Cursor_Range
      with Pre => Is_Valid (Source, From) or else Program_Error;
      -- If From = Empty_Cursor, then we start from the front (or back)
      -- of the string.

   function Count (Source  : in Text;
                  Pattern : in Text;
                  Mapping : in Wide_Wide_Maps.
                                 Wide_Wide_Character_Mapping[q][r][s]
                                         := Wide_Wide_Maps.Identity)
     return Natural;

  function Count (Source  : in Text;
                  Pattern : in Text;
                  Mapping : in not null
                                Wide_Wide_Maps.
                                   Character_Mapping_Function)
     return Natural;

  function Count (Source  : in Text;
                  Set     : in Wide_Wide_Maps.Wide_Wide_Character_Set)
     return Natural;

  function Find_Token (Source : in Text;
                       Set    : in Wide_Wide_Maps.
                                     Wide_Wide_Character_Set;
                       From   : in Text_Cursor := Empty_Cursor;
                       Test   : in Membership)
     return Text_Cursor_Range
     with Pre => Is_Valid (Source, From) or else Program_Error;
     -- If From = Empty_Cursor, then we start from the front (or back)
     -- of the string.

   -- String translation subprograms

   function Translate (Source  : in Text;
                       Mapping : in Wide_Wide_Maps.
                                      Wide_Wide_Character_Mapping)
      return Text;

   procedure Translate (Source  : in out Text;
                        Mapping : in Wide_Wide_Maps.
                                       Wide_Wide_Character_Mapping);

   function Translate
     (Source  : in Text;
      Mapping : in not null Wide_Wide_Maps.
                              Wide_Wide_Character_Mapping_Function)
      return Text;

   procedure Translate
     (Source  : in out Text;
      Mapping : in not null Wide_Wide_Maps.
                              Wide_Wide_Character_Mapping_Function);

   function Insert (Source   : in Text;
                    Before   : in Text_Cursor;
                    New_Item : in Text)
      return Text
      with Pre => Is_Only_Valid (Source, Before) or else Program_Error;

   procedure Insert (Source   : in out Text;
                     Before   : in Text_Cursor;
                     New_Item : in Text)
      with Pre => Is_Only_Valid (Source, Before) or else Program_Error;

   function Overwrite (Source    : in Text;
                       Position  : in Text_Cursor;
                       New_Item  : in Text)
      return Text
      with Pre => Is_Only_Valid (Source, Position)
                     or else Program_Error;

   procedure Overwrite (Source    : in out Text;
                        Position  : in Text_Cursor;
                        New_Item  : in Text)
      with Pre => Is_Only_Valid (Source, Position)
                     or else Program_Error;

   function Delete (Source  : in Text;
                    Rng     : in Text_Cursor_Range)
      return Text
      with Pre => (Is_Only_Valid (Source, Rng.Low) and then
                   Is_Valid (Source, Rng.High)) or else Program_Error;

   procedure Delete (Source  : in out Text;
                     Rng     : in Text_Cursor_Range)
      with Pre => (Is_Only_Valid (Source, Rng.Low) and then
                   Is_Valid (Source, Rng.High)) or else Program_Error;

   function Trim (Source : in Text;
                  Side   : in Trim_End)
      return Text;

   procedure Trim (Source : in out Text;
                   Side   : in Trim_End);

   function Trim (Source : in Text;
                  Left   : in Maps.Character_Set;
                  Right  : in Maps.Character_Set)
      return Text;

   procedure Trim (Source : in out Text;
                   Left   : in Maps.Character_Set;
                   Right  : in Maps.Character_Set);

   function Head (Source : in Text;
                  Count  : in Natural;
                  Pad    : in Unicode_Character := ' ')
      return Text;

   procedure Head (Source : in out Text;
                   Count  : in Natural;
                   Pad    : in Unicode_Character := ' ');

   function Tail (Source : in Text;
                  Count  : in Natural;
                  Pad    : in Unicode_Character := ' ')
      return Text;

   procedure Tail (Source : in out Text;
                   Count  : in Natural;
                   Pad    : in Unicode_Character := ' ');

   function "*" (Left  : in Natural;
                 Right : in Unicode_Character)
      return Text;

   function "*" (Left  : in Natural;
                 Right : in Text)
      return Text;

end Ada.Strings.Unicode.Operations;

with Ada.Strings.UTF_Encoding;
package Ada.Strings.Unicode.UTF_Conversions is
   function To_UTF8 (T : in Text)
      return Ada.Strings.UTF_Encoding.UTF_8_String;
   function From_UTF8 (UTF : in Ada.Strings.UTF_Encoding.UTF_8_String)
      return Text;

   function To_UTF16 (T : in Text)
      return Ada.Strings.UTF_Encoding.UTF_16_Wide_String;
   function From_UTF16
              (UTF : in Ada.Strings.UTF_Encoding.UTF_16_Wide_String)
      return Text;
end Ada.Strings.Unicode.UTF_Conversions;

with Ada.Iterator_Interfaces;
package Ada.Strings.Unicode.Graphemes is

   type Grapheme_Cursor is private;
   function Has_Element (C : Grapheme_Cursor) return Boolean;

   package Grapheme_Iterators is
     new Ada.Iterator_Interfaces (Grapheme_Cursor, Has_Element);

   type Iterator_By_Grapheme_Cluster is
     limited new Grapheme_Iterators.Forward_Iterator with private;

   function By_Grapheme_Cluster (T : in Text)
     return Iterator_By_Grapheme_Cluster;
   function Grapheme_Cluster (T : in Text; C : Grapheme_Cursor)
     return Text;

   --  Usage:
   --  for C in U_Str.By_Grapheme_Cluster loop
   --      Do_Something_With (Grapheme_Cluster (Str, C));
   --  end loop;

private
   type Grapheme_Cursor is null record;
      --  not specified by the language


   type Iterator_By_Grapheme_Cluster is
     limited new Grapheme_Iterators.Forward_Iterator with null record;
      --  not specified by the language

   function First (Iter : Iterator_By_Grapheme_Cluster)
     return Grapheme_Cursor;
   function Next (Iter : Iterator_By_Grapheme_Cluster;
      Pos : Grapheme_Cursor) return Grapheme_Cursor;

end Ada.Strings.Unicode.Graphemes;

package Ada.Strings.Unicode.Handling
   with Pure is
   -- The individual character functions from
   -- Ada.Wide_Wide_Characters.Handling can be used with
   -- Unicode_Characters, so we didn't repeat them here.

   -- These are as defined in Ada.Wide_Characters.Handling.
   function To_Lower (Item : in Text) return Text;
   function To_Upper (Item : in Text) return Text;

   function To_Basic (Item : in Text) return Text;

   -- We provide some normalization routines for safety; these ensure
   -- that all characters use the same encoding by appropriately folding
   -- combining characters. These correspond to the normalizations
   -- expected by Ada compilers for source code.
   --
   -- Normalizations are important if text is coming from untrusted

   -- sources. Without normalizing the text, a bad actor can use unusual
   -- encodings to hide substrings from searches intended to find
   -- dangerous constructs.

   function Normalize_to_NFC (Item : in Text) return Text;
       -- Normalize Item to the NFC format. This does not lose
       -- information, just eliminates combining characters when an
       -- alternative not using them is possible.

   function Normalize_to_NFKC (Item : in Text) return Text;
       -- Normalize Item to the NFKC format. This does lose some
       -- information, as some characteristics like subscripts and
       -- superscripts are eliminated.

end Ada.Strings.Unicode.Handling;

Bounded Errors

A Text_Cursor is invalid if any of the following are true:

It is a bounded error to pass an invalid Text_Cursor to any operation of the Ada.Strings.Unicode subsystem. If detected, Program_Error is raised. Otherwise, the operation proceeds normally, using some character of the String (not necessarily the correct character). Under no circumstances can using an invalid Text_Cursor result in a Text string with an invalid encoding.

AARM Reason: We allow (but do not require) implementations to detect a cursor that no longer designates the correct character. We believe an implementation that does such detection would be too expensive in time and space for general use, but it could be useful for debugging and the absolute maximum of portability.

Implementation Requirements

The output generated by the Text'Output or Text'Write subprograms shall consist of a call to a write of some integer ‘Write subprogram (the integer should be a 32-bit type) followed by the UTF-8 representation of the string (and nothing else). Text’Input and Text’Read shall read this format.

AARM Reason: This makes the streaming output compatible across targets and implementations. Other data (such as memory management information) should be omitted from streaming of Text values.

Implementation Advice

Type Text should use a UTF-8 representation internally.

AARM Discussion: The design of the Text type was chosen to make operations directly on UTF-8 represented text as efficient as possible.

Add Image for Text strings:

Add after 4.10(36/5):

S'Text_Image

S'Text_Image denotes a function with the following specification:

function S'Text_Image(Arg : S'Base)
  return Ada.Strings.Unicode.Text

S'Text_Image calls S'Put_Image passing Arg (which will typically store a sequence of character values in a text buffer) and then returns the result of retrieving the contents of that buffer with function Get_Text. [Redundant: Any exception propagated by the call of S'Put_Image is propagated.]

Add after 4.10(40/5):

X'Text_Image

 X'Text_Image denotes the result of calling function S'Text_Image with Arg being X, where S is the nominal subtype of X.

Modify A.4.12(3/5):

with Ada.Strings.UTF_Encoding.Wide_Wide_Strings;{
with Ada.Strings.Unicode;}
package Ada.Strings.Text_Buffers
    with Pure is

Add after A.4.12(11/5):

procedure Put_Text (
       Buffer : in out Root_Buffer_Type;
       Item   : in     Unicode.Text) is abstract;

Add after A.4.12(24/5):

function Get (
       Buffer : in out Buffer_Type)
       return Unicode.Text
       with Post'Class =>
          Current_Indent (Buffer) = 0;

Modify A.4.12(29/5):

   -- Get, Wide_Get, Wide_Wide_Get, Get_UTF_8, [and ]Wide_Get_UTF_16
   -- {and Get_Text }are declared here just as in the Unbounded child.

Modify A.4.12(33/5):

A call to Put, Wide_Put, Wide_Wide_Put, Put_UTF_8, [or ]Wide_Put_UTF_16{, or Put_Text} stores a sequence of characters into the text buffer, preceded by Current_Indent(Buffer) spaces (Wide_Wide_Characters with position 32) if there is at least one character in Item and it would have been the first character on the current line.

Modify A.4.12(34/5):

A call to function Get, Wide_Get, Wide_Wide_Get, Get_UTF_8,[ or] Wide_Get_UTF_16{, or Get_Text} returns the same sequence of characters as was present in the calls that stored the characters into the buffer, if representable. For a call to Get, if any character in the sequence is not defined in Character, the result is implementation defined. Similarly, for a call to Wide_Get, if any character in the sequence is not defined in Wide_Character, the result is implementation defined. As part of a call on any of the Get functions, the buffer is reset to an empty state, with no stored characters.

Add Value for Text strings:

Add after 3.5(55/5):

S'Text_Value

S'Text_Value denotes a function with the following specification:

function S'Text_Value(Arg : Text_String)

  return S'Base

This function returns a value given an image of the value as a Text string, ignoring any leading or trailing spaces. The evaluation of a call on S'Text_Value with Arg of type Text is equivalent to a call on S'Wide_Wide_Value with Arg converted to type Wide_Wide_String.

 

All existing operations that take a String should be considered to get additional versions that take a Text instead. (For ambiguity reasons, these will typically need to be in nested packages, otherwise all calls using string literals would be ambiguous.) This is especially true for the I/O packages.

!discussion

The existing facilities for string manipulation in Ada are a mess, so for this design we are working from first principles. The goal is a self-contained string abstraction that does not need to use any operations from any other string type in order to be complete. (Of course, it might be necessary to convert to/from some other representation in order to use legacy code.)

A “string” is a sequence of characters, so operations should primarily work from that abstraction. However, the underlying representation may not have a simple mapping from characters to storage units. For instance, UTF-8 representations use between 1 and 5 octets to represent a single character.

Performance requirements suggest that many operations need to use “cursors” of some sort, which access the appropriate storage units, rather than converting from numbers of characters each time (which is likely expensive).

It should be noted that the UTF-8 and UTF-16 representations were designed so that operations can be done as efficiently as in a simple Latin-1 string, with operations like searching requiring nothing more than matching of octets (encodings differ for the first octet from any other octet of a character, so mismatches aren’t possible). [Truth-in-advertising: This ignores the issues associated with normalization, but in normal use for strings created locally those aren’t important and should not be used to harm performance.] As such, there is almost no reason to convert a UTF-8 (or UTF-16) string to some other representation other than to use existing code that needs some other representation.

We define operations so that a Text string always is a valid string of its type. Thus, operations that might break characters (such as unrestricted slicing) are not supported, and any operation with such a potential is checked.

We design Text strings with the intention that they will be implemented as UTF-8 strings internally. We allow the possibility of using UTF-16 instead, if that makes more sense for a particular target, but we do not expect that possibility to be commonly used. In particular, we define the streaming format in terms of UTF-8, so that streaming of Text strings is compatible between different targets and even different compilers.

We define our Text_Cursor type to designate the first unit of a character in a particular Text string. Any use of a Text_Cursor that does not do so raises Program_Error. (Note that such a check is relatively cheap, as the encodings that can be the first unit of a UTF-8 or UTF-16 character are unique.)

While a Text_Cursor belongs to a particular string, unlike Ada.Containers we do not make any effort to check or require that. Similarly, while a Text_Cursor that no longer designates its original character in a Text string is invalid (and any use is a Bounded Error that could raise Program_Error), we do not check that either. The reason in both of these cases is for space and performance reasons. Since we do not have anything like a tampering check, the underlying string can change at any time. Thus, checking the string does not provide much safety. For checking the movement of octets, one would need a matching serial number scheme. This would at a minimum triple the amount of space needed to store a Text string. As such, we leave the door open for such an implementation (for the ultimate in portability), but do not require or even expect it.

A Text_Cursor is usually just an index into an underlying octet (8-bit byte) array for the associated Text string. We make the type private, however, as it does not make sense to calculate cursors without reference to an underlying string. In particular, it is meaningless to add 1 to a Text_Cursor, as that may or may not be appropriate for the associated Text string – it might just create a cursor that points into the middle of a character (which is invalid). We avoid talking about octets, in order to avoid introducing another term, and to allow implementations to use another representation if they wish.

We also provide a Text_Cursor_Range type, which is just a pair of Text_Cursors. We hope that one of the proposals for “..” is adopted, in order to make using this type more readable. Note that the upper bound points at the FIRST octet of a character; any use of the range for slicing or similar operations needs to adjust that to point to the last octet of the character. This is similar to the check for validity in complexity. We could have defined a separate cursor type for the upper bound (that always points to the last octet of a character), but that would have doubled the needed operations.

We use Text_Cursors or Text_Ranges instead of Positive/Natural in all of the operations provided for an Unbounded_String. These operations are provided as part of the Text package.

A Text string is fundamentally an unbounded string. There is no straightforward relationship between the number of octets and the number of characters of a string, so a fixed length (as is usual in Ada) is pretty much meaningless. It could only be made to work with nul terminated strings, which has never been the Ada way to handle anything – this does not seem to be the time to start. We are assuming that the bounded mechanisms of AI22-0148-1 become available. If they do not, then we would define a matching Bounded_Text package with a capacity discriminant added to it. (Note that there is a relatively simple way to avoid the problems with “:=” for such types; if AI22-0148-1 is ultimately not adopted, we should consider such a light overriding of the assignment statement operation. The author has not proposed that as it is unnecessary in light of AI22-0148-1 and the effort to write it up is substantial.)

We did not provide any additional Unicode operations outside of the universal Text string. The existing facilities in Ada.Strings.UTF_Encoding should be sufficient for those that are not interested in a universal string (and that should be a small cohort, given how hard it is to use Ada’s existing strings).

 

The conversions to the types declared in Standard were originally declared in a nested package. However, that would prevent them from being used in prefix notation, which is very handy for conversion routines (as noted when writing the example).

Having these directly in the main package should not cost any overhead, as the types are predefined in Standard, and any unused code should be eliminated by the usual mechanisms for doing so. OTOH, conversions to the UTF_8 and UTF_16 subtypes in Ada.Strings.UTF_Encoding are handled separately, as those would drag in an additional package.

Similarly, all of the operations that use a character map are placed in an Operations package (along with some rarely used routines defined in the other Strings.[Fixed/Bounded/Unbounded] packages).

The operations in the base package are those that correspond to the operations directly available for type String. In particular, because String is an array, we get comparison, indexing, slicing (both for reading and writing), and iteration. It would be unfortunate if any of those primitive operations were not available directly for the type (and in particular via prefix notation). [Prefix notation is effectively only available for primitive operations for untagged types.]

The additional operations that are defined are those that are defined for the cursor types, and the most basic search routine (Index). We include Index as it is common to iterate to find a single character in a string (such as a delimiter like a ‘,’ or ‘;’). While one could write the same code for Text, it would be a rather inefficient way to find a particular character. Rather, the usual searching routine should be used for any sort of searching, even for a single character (as it does not need to find character boundaries, unlike the iterator). For this reason, we provide the simplest Index routine in the base package.

 

We treat Wide_String as UTF-16 strings for the purposes of conversions here. We do that on the recommendation of our Unicode liaison (Robin Leroy). The effect is to not check what characters are used in the Wide_String, other than that surrogate characters need to be properly paired.

Robin can explain the reasons better than I can:

As you note in the !issue, Wide_ vs. Wide_Wide_ is a choice that is made long in advance (in practice, decades ago), and with which users will be stuck. Since those conversions are, as you note in the !appendix, for interfacing with legacy code, using Wide_String as UTF-16 is a way to make that legacy code compatible with non-BMP text.

Importantly, this is strictly an extension to "Wide_String as BMP characters", as there is no other meaning that can be assigned to a Wide_Wide_String that contains surrogates: it is either garbage or UTF-16—or both, in the case of  Wide_Character'Val(16#D83D#) & Wide_Character'Val(16#DDD1#)  ;-).

The situation is very different with String, as there it is likely to be (and indeed, supposed to be) Latin-1, with the UTF-8 interpretation in Ada.Strings.UTF_Encoding being a dangerous—if necessary—hack.

!example

Here is a simple example using Ada.Strings.Unicode.Text, using the new 'Text_Image attribute:

with Ada.Strings.Unicode; use Ada.Strings.Unicode;
function Message (Y : String; Z : Wide_String) return Text is
   X : constant Text :=
     "Y = " & From_String (Y) & ", Z = " & From_Wide_String (Z);
begin
   return X & "; Message length = " & X.Characters_Length'Text_Image;
end Message;

Here is a routine extracting a value from a line of text read from a report. The item of interest is labeled “Time Δ:” and ends with a semicolon or the end of the line. We use the new Text_Value attribute to extract the value. [Note: We assume here that the range syntax (“..”) is supported as proposed in AI22-0158-1, if not, a record aggregate could be used instead.]

with Ada.Strings.Unicode; use Ada.Strings.Unicode;
function Get_Time_Delta (Line : in Text) return Float is
   Label, Item_End : Text_Cursor_Range;
begin
   Label := Line.Index (Pattern =>  “Time Δ:”);
   if Label.Low = Empty_Cursor then
      raise Program_Error;
   end if; -- No label found.
   Item_End := Line.Index (Pattern => “;”, From => Label.Low);
   if Item_End.Low = Empty_Cursor then
      Item_End.High := Line.Last;
   else -- found the end already, but we want the character before.
      Item_End.High := Line.Previous(Item_End.High);
   end if;
   return Float’Text_Value (
       Line.Slice(Line.Next(Label.High) .. Item_End.High));
end Get_Time_Delta;

!ACATS test

C-Tests are needed for each package and individual operation (much like the existing tests for Unbounded Strings).

!appendix

This is based alternative one (AI22-0161-1), which was based on ARG GitHub issue #152, which is in turn based on ARG GitHub issue #40.

 

From: Randy Brukardt

Sent: Tuesday, May 05, 2026 6:45 PM

I've posted Tucker's Universal string proposal as AI22-0161-1, and my (many times better, IMHO ;-) proposal as AI22-0161-2. As always, comments on either or both of these are welcome.

Seriously, my version tries to make a true universal string that is intended to replace all other string types for most uses (outside of interfacing where a particular format is required). There is much more required to achieve that (A Text_Text_IO for certain, and many other things could be changed as well), but I have provided a start. (Tuck's version doesn't have any manipulation operations nor tries to explain how they would be provided, which is the tough nut for this concept.)

I provided a (temporary) brief overview of the differences at the end of the !discussion section. I don't plan to keep that as a permanent part of the AI (one imagines that we will only advance one of the proposals, or neither, but not both).

 

From: Tucker Taft

Sent: Wednesday, May 06, 2026 1:52 PM

Thanks for doing this.  It is helpful to have a fully worked-out alternative approach.  I was planning on leaving the various operations, such as those in Strings.Unbounded, to a separate AI built on top, but it is interesting to see your approach of integrating them.

I still think we should make the existing UTF_Encoding package obsolescent, given its use of subtypes of String and Wide_String, but perhaps that is not essential.

In any case, hopefully these two AIs will allow us to have a good discussion, and we can see whether one or the other, or some mixture, makes the most sense once we dig into them.

 

From: Randy Brukardt

Sent: Wednesday, May 06, 2026 7:14 PM

>Thanks for doing this.  It is helpful to have a fully worked-out alternative approach.

>I was planning on leaving the various operations, such as those in Strings.Unbounded,

>to a separate AI built on top, but it is interesting to see your approach of

>integrating them.

They have to be efficient in order to have a truly Universal text type, and thus leaving them for later really doesn't provide the needed functionality.

>I still think we should make the existing UTF_Encoding package obsolescent, given

>its use of subtypes of String and Wide_String, but perhaps that is not essential.

The idea here is to provide a Text string that can store any possible Unicode character. As such, it potentially replaces uses of all existing string types. For that to happen, it has to be quite universal (almost every language-defined operation needs to take or produce a Text as an alternative to existing types); it has to be quite efficient; and it has to be as easy to use as the existing types (if not easier).

For me, that's the entire reason for creating this abstraction. We have plenty of bad or limited string abstractions already in Ada, we surely don't need any more of those. I expect that we will keep working until we get there (or at least as close as practical).

For new code, if we have this sort of type, then we don't need any other kind of type to work on text in Ada. And the exact representation used internally to Text should not be relevant (so long as it can represent all Unicode characters in a sequence).

So the cases where a particular representation is needed should be limited:

This is a long winded way of saying that the need for a UTF_8 type and a UTF_16 type should be quite limited. And if so, the importance of having them perfect is also limited. We already have such types (not ideal, but they exist).

Of course, if we don't get Text right, then of course the need goes up. But I would rather do nothing than create yet another bad solution, so we will need to keep working (and writing examples) until we get it right.

>In any case, hopefully these two AIs will allow us to have a good discussion,

>and we can see whether one or the other, or some mixture, makes the

>most sense once we dig into them.

Yup, that was the idea. Mine didn't end up as different as I expected when I started. I used the names from Ada.Strings.Unbounded in general (I did use "Empty_Cursor" rather than "Null_Cursor", hope you're happy about that. ;-), but the layout is fairly similar. My main objection to your version was that you didn't trust in your universal Text string to be truly universal. That's fundamental to me, the reason to do this at all is to replace almost all existing string operations with this new type.

 

 

[a]This "_Character" suffix seems unnecessary, here and in Subtract_Characters, and also the similar prefix in Character_Length. No other unit of Text structure or measure of string length is used here.

[b]Many UTF-8 implementations use "octets" as the unit of measurement; indeed, any bounding mechanism (either the one of AI22-148-1 or an explicit one) would be in terms of octets. I wanted to make the difference clear, even if it doesn't occur in the current interfaces.

[c]I understand your reasons, but maintain my position.

[d]For what its worth, my original intent was to call these "+" and "-", but given that there are three parameters, that doesn't work. In any case, there certainly will be low-level octet operations (to specify bounds at a minimum), and these need to be separated from those. There might be some other way to make that clear, but I don't know what it is.

[e]I find it strange to have both "Add" and "Subtract" when both allow both positive and negative offsets. A single "Move" operation with a signed offset seems simpler. Perhaps the reason for having both Add and Subtract is a different meaning of Empty_Cursor in the two cases?

[f]In my experience, some algorithms naturally need to subtract a value, and some naturally need to add a value. Typically, you are translating some algorithm that adds or subtracts a character position. "Move" to me invokes moving text, not a position. We could make Count Natural if this it truly bothersome, although that can make things more complicated in some obscure cases. Note that the language-defined "+" and "-" allow adding and subtracting negative numbers, it seems to me this is similar.

[g]When I first read the name Add_Characters, I thought it meant to insert characters in the text, and only when I saw that there is no "source" parameter did I start to doubt.

Perhaps "Forward"/"Backward" would be better terms for moving cursors.

My experience of scanning text suggests that one almost always moves the cursor by a constant, positive or negative, so there we agree. But for arithmetic "+"/"-", very often the operand signs are determined dynamically, but the choice of "+" or "-" is determined by the static logic of the algorithm.

[h]I think we agree on how "+" and "-" are used, the question is what to call them. I think it is best to leave this question for the group.

[i]Perhaps force use of Nth and leave this out?

[j]Maybe, but this is a basic operation of type String and it seems weird to not provide it.

[k]Also note that we are providing Element (both of them) as (constant) indexing operations in order to be as close to String as is possible. One should use it sparingly, but it is available.

[l]This strikes me as a slightly weird operation, because it is quite rare that you would operate on arbitrary strings and want to only do one-to-one code point substitutions, and because unless the string is stored in UTF-32, this is just as complex as Replace_Slice.

But perhaps it is worth having it to look like the existing string types.

[m]A typical use case could be remove all diacritical marks from a text - Airplane tickets still do not accept accents!

[n]The reason this is included here is described in the !discussion, below.

[o]String mutation is already pretty specialized and complicated.

[p]This is a basic (and native) operation of type String; it would be bizarre not to provide it for Text. Child packages don't allow prefix notation, so they are sub-optimal for basic operations.

[q]Since a one-to-one mapping is not very different from a one-to-many mapping from an implementation perspective with a non-UTF-32 backing store, it may be useful to have a one-to-many version of Character_Mapping for use in these functions. While there are many transformations that require context-aware mappings, there are still some useful operations that are context-free one-to-many mappings—though they may need to be applied after normalization and the like—: case folding (the recommended mapping for case-insensitive matching), or even confusable skeletons (the thing that tells you that Cyrillic САТ looks like Latin CAT).

[r]_Marked as resolved_

[s]_Re-opened_

Oops, didn't mean to close this one. I personally think that is getting too complex, especially for something that isn't used very often (at least in my experience). But that is for the group to decide.