AI22-0102-1/03

AI22-0102-1

!standard A.4.11(28/3) 24-05-03 AI22-0102-1/03

!standard A.4.11(36/3)

!standard A.4.11(51/3)

!standard A.4.11(56/3)

!standard A.4.11(57/3)

!class Amendment 24-04-17

!status Amendment 1-2022 25-03-21

!status WG9 Approved 24-10-10

!status ARG Approved 14-0-0 24-05-02

!status work item 24-04-17

!status received 24-04-17

!submitter Pascal Pignard

!priority Low

!difficulty Easy

!subject Substitute parameter for UTF Decode functions

!summary

Versions of Decode are added to Ada.Strings.UTF_Encoding.Strings and Ada.Strings.UTF_Encoding.Wide_Strings that have a Substitute parameter; this allows graceful handling of characters outside of the result range.

!issue

The Decode functions in Ada.Strings.UTF_Encoding.Strings and Ada.Strings.UTF_Encoding.Wide_Strings raise Encoding_Error for any characters whose code points are out of range. The similar functions in Ada.Characters.Conversions use a substitute value in this situation.

For many uses, raising an exception is inconvenient and a potential source of a bug (if an unexpected character causes an unexpected exception to be raised). One can avoid this exception by decoding to a Wide_Wide_String and then passing that to Ada.Characters.Conversions.To_String, but it is easy to forget to do this.

Thus, we conclude that the Decode functions in Ada.Strings.UTF_Encoding.Strings and Ada.Strings.UTF_Encoding.Wide_Strings should have a Substitute parameter similarly to the functions in Ada.Characters.Conversions.

!recommendation

(See Summary.)

!wording

Add after A.4.11(28/3):

function Decode (Item : UTF_String;
Input_Scheme : Encoding_Scheme;
Substitute : Character) return String;

function Decode (Item : UTF_8_String;
Substitute : Character) return String;

function Decode (Item : UTF_16_Wide_String;
Substitute : Character) return String;

Add after A.4.11(36/3):

function Decode (Item : UTF_String;
Input_Scheme : Encoding_Scheme;
Substitute : Wide_Character) return Wide_String;

function Decode (Item : UTF_8_String;
Substitute : Wide_Character) return Wide_String;

function Decode (Item : UTF_16_Wide_String;
Substitute : Wide_Character) return Wide_String;

Add after A.4.11(51/3):

For a Decode function with a Substitute parameter yielding a String, if the decoding of a sequence results in a code point whose value exceeds 16#FF#, the resulting character is replaced by the Substitute Character in the result String. Similarly, for a Decode function with a Substitute parameter yielding a Wide_String, if the decoding of a sequence results in a code point whose value exceeds 16#FFFF#, the resulting character is replaced by the Substitute Wide_Character in the result Wide_String.

[Editor's note: This wording is a combination of the wording used in A.4.11(51/3), A.4.11(56/3), and A.3.4(19/2). It's more vague than I would like, but that is consistent with the rest of A.4.11 (which is rather vague itself).]

Modify A.4.11(56/3):

By a Decode function {without a Substitute parameter }yielding a String when the decoding of a sequence results in a code point whose value exceeds 16#FF#.

Modify A.4.11(57/3):

By a Decode function {without a Substitute parameter }yielding a Wide_String when the decoding of a sequence results in a code point whose value exceeds 16#FFFF#.

!discussion

For compatibility reasons, we need these to be new functions rather than overloading the existing functions. An application that expects to get an exception in this situation would instead return an unexpected result if the parameter was simply added to the existing functions with a default of ' ' as in Ada.Characters.Conversions.

Similarly, compatibility concerns make it necessary to not have a default for the Substitute parameter. Otherwise, existing calls would become ambiguous, and there would be no way to call the function that does not have the Substitute parameter.

Note that we do not add a Substitute parameter to the Decode functions that return a Wide_Wide_String; there are no code points that cannot be represented in a Wide_Wide_String, and therefore a substitution is never needed. This is consistent with the definitions in Ada.Characters.Conversions.

!corrigendum A.4.11(28/3)

@dinsa

@xcode{ @b{function} Decode (Item : UTF_16_Wide_String) @b{return} String;}

@dinss

@xcode{ @b{function} Decode (Item : UTF_String;
Input_Scheme : Encoding_Scheme;
Substitute : Character) @b{return} String;}

@xcode{ @b{function} Decode (Item : UTF_8_String;
Substitute : Character) @b{return} String;}

@xcode{ @b{function} Decode (Item : UTF_16_Wide_String;
Substitute : Character) @b{return} String;}

!corrigendum A.4.11(36/3)

@dinsa

@xcode{ @b{function} Decode (Item : UTF_16_Wide_String) @b{return} Wide_String;}

@dinss

@xcode{ @b{function} Decode (Item : UTF_String;
Input_Scheme : Encoding_Scheme;
Substitute : Wide_Character) @b{return} Wide_String;}

@xcode{ @b{function} Decode (Item : UTF_8_String;
Substitute : Wide_Character) @b{return} Wide_String;}

@xcode{ @b{function} Decode (Item : UTF_16_Wide_String;
Substitute : Wide_Character) @b{return} Wide_String;}

!comment The A.4.11(51/3) change was removed by AI22-0143-1.

!corrigendum A.4.11(56/3)

@drepl

@xbullet{By a Decode function yielding a String when the decoding of a sequence results in a code point whose value exceeds 16#FF#.}

@dby

@xbullet{By a Decode function without a Substitute parameter yielding a String when the decoding of a sequence results in a code point whose value exceeds 16#FF#.}

!corrigendum A.4.11(57/3)

@drepl

@xbullet{By a Decode function yielding a Wide_String when the decoding of a sequence results in a code point whose value exceeds 16#FFFF#.}

@dby

@xbullet{By a Decode function without a Substitute parameter yielding a Wide_String when the decoding of a sequence results in a code point whose value exceeds 16#FFFF#.}

!ACATS test

An ACATS C-Test should check that the Substitute parameter exists and works as expected for these Decode functions.

!appendix

See issue #78 on ARG GitHub issue list.