String encoding utility methods and tests for Unicode, UTF-8, -16 and -32.

utils/string_encoding/UnicodeCore.dart

Index: utils/string_encoding/UnicodeCore.dart
diff --git a/utils/string_encoding/UnicodeCore.dart b/utils/string_encoding/UnicodeCore.dart
new file mode 100644
index 0000000000000000000000000000000000000000..62758f70250fc2b12246453586719489de4b528b
--- /dev/null
+++ b/utils/string_encoding/UnicodeCore.dart
@@ -0,0 +1,94 @@
+// Copyright (c) 2011, the Dart project authors.  Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+#library("UnicodeCore");
+
+/*
+ * Test for presence of bug related to the use of UTF-16 code units for

[jat, 2012/01/31 15:19:22]

I'm not sure I would call it a bug -- it simply isn't practical to implement
UCS32 in JS, as you wouldn't be able to make much use of native string support
in JS.

[Dan Rice, 2012/01/31 15:59:55]

Since strings are immutable, you can at least do all the work up front at
construction time, and in the case where there are no surrogate pairs the
constructed instance can rely on natives.  This seems like it might be worth
prototyping and benchmarking.

On 2012/01/31 15:19:22, jat wrote:

[dcarlson, 2012/01/31 22:11:38]

The calculation is once, then cached.

+ * Dart compiled to JS.
+ */
+bool _test16BitCodeUnit = null;
+bool is16BitCodeUnit() {
+  if(_test16BitCodeUnit == null) {
+    _test16BitCodeUnit = (new String.fromCharCodes([0x1D11E])) ==

[jat, 2012/01/31 15:19:22]

I would have expected this to result in a 2-character string in JS (ie, the
surrogate pair corresponding to the non-BMP character).  If it isn't, then this
is definitely a bug (in fromCharCodes and perhaps elsewhere) as non-BMP
characters simply aren't supported at all when compiled to JS -- in JS if I give
it a surrogate pair the non-BMP character is properly displayed.

[Dan Rice, 2012/01/31 15:59:55]

Done.

[dcarlson, 2012/01/31 22:11:38]

:)

+        (new String.fromCharCodes([0xD11E]));
+  }
+  return _test16BitCodeUnit;
+}
+
+
+/**
+ * Invalid codepoints or encodings may be substituted with the value U+fffd.
+ */
+final int REPLACEMENT_CHARACTER_CODEPOINT = 0xfffd;
+
+/**
+ * Encode code points as UTF16 code units.
+ */
+List<int> codepointsToUtf16CodeUnits(List<int> codepoints,
+    [int start = 0, int length = -1]) {

[jat, 2012/01/31 15:19:22]

Seems like null is a better "not-supplied" default than -1 to avoid confusion,
since in many APIs a negative length means counting from the end.

[Dan Rice, 2012/01/31 15:59:55]

Maybe use length = null as the sentinel?  The -1 makes me expect some kind of
general behavior defined for negative values.

[dcarlson, 2012/01/31 22:11:38]

Done.

+  List<int> codeUnitsBuffer = <int>[];
+  int end = length >= 0 ? Math.min(codepoints.length, start + length) :
+      codepoints.length;
+  int i = start;
+  while (i < end) {
+    int value = codepoints[i++];
+    if ((value >= 0 && value <0xd800) || (value >= 0xe000 && value <0x10000)) {

[Dan Rice, 2012/01/31 15:59:55]

space after '<'

[dcarlson, 2012/01/31 22:11:38]

Done.

+      codeUnitsBuffer.add(value);
+    } else if (value >= 0x10000 && value < 0x110000){

[Dan Rice, 2012/01/31 15:59:55]

Space before '{'

[dcarlson, 2012/01/31 22:11:38]

Done.

+      int base = value - 0x10000;
+      codeUnitsBuffer.add(0xd800 + ((base & 0xffc00) >> 10));
+      codeUnitsBuffer.add(0xdc00 + (base & 0x3ff));

[jat, 2012/01/31 15:19:22]

These ranges should be in constants rather than repeating them.

[dcarlson, 2012/01/31 22:11:38]

Done.

+    } else {
+      codeUnitsBuffer.add(REPLACEMENT_CHARACTER_CODEPOINT);
+    }
+  }
+  return codeUnitsBuffer;
+}
+
+/**
+ * Decodes the utf16 codeunits to codepoints.
+ */
+List<int> utf16CodeUnitsToCodepoints(List<int> utf16CodeUnits,
+    [int start = 0, int length = - 1]) {

[Dan Rice, 2012/01/31 15:59:55]

length sentinel

[dcarlson, 2012/01/31 22:11:38]

Done.

+  List<int> codepointBuffer = <int>[];
+
+  int end = length >= 0 ?
+      Math.min(utf16CodeUnits.length, start + length) :
+      utf16CodeUnits.length;
+
+  int i = start;
+  // skip the first entry if it is a BOM.
+  if(end > 0 && utf16CodeUnits[0] == 0xfeff) {

[jat, 2012/01/31 15:19:22]

If you are checking for BOM, do you want to also check for U+FFFE and reverse
the bytes below?

I note you don't generate BOM above.

[Dan Rice, 2012/01/31 15:59:55]

space before '('

[dcarlson, 2012/01/31 22:11:38]

Not here. Only do this when encoding code units to bytes.

[dcarlson, 2012/01/31 22:11:38]

Not when going from codepoint to code unit. The BOM is introduced only when we
go form code unit to encoded bytes (where we would know 1) the byte order and 2)
where we could decide whether we want to include the BOM).

+    i++;
+  }
+  while (i < end) {
+    int value = utf16CodeUnits[i++];
+    if (value >= 0x0) {
+      if (value < 0xd800 || (value >= 0xe000 && value <= 0xffff)) {
+        // transfer directly
+        codepointBuffer.add(value);
+      } else if (value < 0xdc00 && i < end) {
+        // merge surrogate pair
+        value = (value - 0xd800) * 0x400;

[Dan Rice, 2012/01/31 15:59:55]

can use '<< 10' instead of '* 0x400'

[dcarlson, 2012/01/31 22:11:38]

Done.

+        int nextValue = utf16CodeUnits[i++];
+        if (nextValue >= 0xdc00 && nextValue < 0xe000) {
+          value += 0x10000 + (nextValue - 0xdc00);
+          codepointBuffer.add(value);
+        } else {
+          if (nextValue >= 0xd800 && nextValue < 0xdc00) i--;

[Dan Rice, 2012/01/31 15:59:55]

better not to put the 'then' clause on same line

[dcarlson, 2012/01/31 22:11:38]

Done.

+          codepointBuffer.add(REPLACEMENT_CHARACTER_CODEPOINT);
+          continue;
+        }
+      } else {
+        codepointBuffer.add(REPLACEMENT_CHARACTER_CODEPOINT);
+        continue;
+      }
+    } else {
+      codepointBuffer.add(REPLACEMENT_CHARACTER_CODEPOINT);
+      continue;
+    }
+  }
+  return codepointBuffer;
+}