Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(202)

Side by Side Diff: lib/i18n/bidi_utils.dart

Issue 10592011: Add BiDirectional Text formatting utilites to the i18n library. (Closed) Base URL: http://dart.googlecode.com/svn/branches/bleeding_edge/dart/
Patch Set: Created 8 years, 6 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch | Annotate | Revision Log
OLDNEW
(Empty)
1 // Copyright (c) 2012, the Dart project authors. Please see the AUTHORS file
2 // for details. All rights reserved. Use of this source code is governed by a
3 // BSD-style license that can be found in the LICENSE file.
4
5 #library('bidi_utils');
6
7 /**
8 * Bidi stands for Bi-directional text.
9 * According to http://en.wikipedia.org/wiki/Bi-directional_text:
10 * Bi-directional text is text containing text in both text directionalities,
11 * both right-to-left (RTL) and left-to-right (LTR). It generally involves text
12 * containing different types of alphabets, but may also refer to boustrophedon,
13 * which is changing text directionality in each row.
14 *
15 * This file provides some utility classes for determining directionality of
16 * text, switching CSS layout from LTR to RTL, and other normalizing utilities
17 * needed when switching between RTL and LTR formatting.
18 *
19 * It defines the TextDirection class which is used to represent directionality
20 * of text,
21 * In most cases, it is preferable to use bidi_formatter.dart, which provides
22 * bidi functionality in the given directional context, instead of using
23 * bidi_utils.dart directly.
24 */
25
26 /** Class containing constants to represent the directionality of text. */
27 class TextDirection {
28 static final LTR = const TextDirection._('LTR', 'ltr');
29 static final RTL = const TextDirection._('RTL', 'rtl');
30 static final UNKNOWN = const TextDirection._('UNKNOWN', 'ltr');
Alan Knight 2012/06/22 00:16:53 I presume the 'ltr' on UNKNOWN means that we assum
Emily Fortuna 2012/06/25 20:25:35 Done.
31
32 /** A string value representation of the directionality constant. */
33 final String value;
Alan Knight 2012/06/22 00:16:53 Probably more informative to enumerate the values.
Emily Fortuna 2012/06/25 20:25:35 Done.
34 /**
35 * A string indicating the directionality that should be indicated by the text
36 * for explicitly specifything the direction in a span tag.
37 */
Alan Knight 2012/06/22 00:16:53 Same as above. Also specifying is spelled wrong.
Emily Fortuna 2012/06/25 20:25:35 Done.
38 final String spanText;
39
40 const TextDirection._(this.value, this.spanText);
41 }
42
43 class BidiUtils {
44 /** Unicode "Left-To-Right Embedding" (LRE) character. */
45 static final LRE = '\u202A';
46
47 /** Unicode "Right-To-Left Embedding" (RLE) character. */
48 static final RLE = '\u202B';
49
50 /** Unicode "Pop Directional Formatting" (PDF) character. */
51 static final PDF = '\u202C';
52
53 /** Unicode "Left-To-Right Mark" (LRM) character. */
54 static final LRM = '\u200E';
55
56 /** Unicode "Right-To-Left Mark" (RLM) character. */
57 static final RLM = '\u200F';
58
59 /** Constant to define the threshold of RTL directionality. */
60 static num _RTL_DETECTION_THRESHOLD = 0.40;
61
62 /**
63 * Practical patterns to identify strong LTR and RTL characters, respectively.
64 * These patterns are not completely correct according to the Unicode
65 * standard. They are simplified for performance and small code size.
66 */
67 static final String _LTR_CHARS =
68 @'A-Za-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02B8\u0300-\u0590'
69 @'\u0800-\u1FFF\u2C00-\uFB1C\uFDFE-\uFE6F\uFEFD-\uFFFF';
70 static final String _RTL_CHARS = @'\u0591-\u07FF\uFB1D-\uFDFD\uFE70-\uFEFC';
71
72 /**
73 * Returns the input [text] with spaces instead of HTML tags or HTML escapes,
74 * which is helpful for text directionality estimation.
75 * Note: This function should not be used in other contexts.
76 * It does not deal well with many things: comments, script,
77 * elements, style elements, dir attribute,`>' in quoted attribute values,
78 * etc. But it does handle well enough the most common use cases.
79 * Since the worst that can happen as a result of these shortcomings is that
80 * the wrong directionality will be estimated, we have not invested in
81 * improving this. If [isStripNeeded] is true, this function does the
82 * stripping, otherwise the function returns the input unchanged.
83 */
84 static String stripHtmlIfNeeded(String text, bool isStripNeeded) {
85 if (isStripNeeded) {
86 // The regular expression is simplified for an HTML tag (opening or
87 // closing) or an HTML escape. We might want to skip over such expressions
88 // when estimating the text directionality.
89 return text.replaceAll(const RegExp(@'<[^>]*>|&[^;]+;'), ' ');
90 } else {
91 return text;
92 }
93 }
94
95 /**
96 * Determines if the first character in [text] with strong directionality is
97 * LTR. If [isHtml] is true, the text is HTML or HTML-escaped.
98 */
99 static bool startsWithLtr(String text, [isHtml=false]) {
100 return const RegExp('^[^$_RTL_CHARS]*[$_LTR_CHARS]').hasMatch(
Alan Knight 2012/06/22 00:16:53 That comes out really nicely. I like the use of st
Emily Fortuna 2012/06/25 20:25:35 Done.
101 BidiUtils.stripHtmlIfNeeded(text, isHtml));
102 }
103
104 /**
105 * Determines if the first character in [text] with strong directionality is
106 * RTL. If [isHtml] is true, the text is HTML or HTML-escaped.
107 */
Alan Knight 2012/06/22 00:16:53 I worry a bit with these about people calling many
Emily Fortuna 2012/06/25 20:25:35 As mentioned at the top of this file, bidi_formatt
Alan Knight 2012/06/25 23:29:56 OK
108 static bool startsWithRtl(String text, [isHtml=false]) {
109 return const RegExp('^[^$_LTR_CHARS]*[$_RTL_CHARS]').hasMatch(
110 BidiUtils.stripHtmlIfNeeded(text, isHtml));
111 }
112
113 /**
114 * Determines if the exit directionality (ie, the last strongly-directional
115 * character in [text] is LTR. If [isHtml] is true, the text is HTML or
116 * HTML-escaped.
117 */
118 static bool endsWithLtr(String text, [isHtml=false]) {
119 return const RegExp('[$_LTR_CHARS][^$_RTL_CHARS]*\$').hasMatch(
120 BidiUtils.stripHtmlIfNeeded(text, isHtml));
121 }
122
123 /**
124 * Determines if the exit directionality (ie, the last strongly-directional
125 * character in [text] is RTL. If [isHtml] is true, the text is HTML or
126 * HTML-escaped.
127 */
128 static bool endsWithRtl(String text, [isHtml=false]) {
129 return const RegExp('[$_RTL_CHARS][^$_LTR_CHARS]*\$').hasMatch(
130 BidiUtils.stripHtmlIfNeeded(text, isHtml));
131 }
132
133 /**
134 * Determines if the given [text] has any LTR characters in it.
135 * If [isHtml] is true, the text is HTML or HTML-escaped.
136 */
137 static bool hasAnyLtr(String text, [isHtml=false]) {
138 return const RegExp(@'[' '$_LTR_CHARS' @']').hasMatch(
139 BidiUtils.stripHtmlIfNeeded(text, isHtml));
140 }
141
142 /**
143 * Determines if the given [text] has any RTL characters in it.
144 * If [isHtml] is true, the text is HTML or HTML-escaped.
145 */
146 static bool hasAnyRtl(String text, [isHtml=false]) {
147 return const RegExp(@'[' '$_RTL_CHARS' @']').hasMatch(
148 BidiUtils.stripHtmlIfNeeded(text, isHtml));
149 }
150
151 /**
152 * Check if a BCP 47 / III [languageString] indicates an RTL language.
153 *
154 * i.e. either:
155 * - a language code explicitly specifying one of the right-to-left scripts,
156 * e.g. "az-Arab", or
157 * - a language code specifying one of the languages normally written in a
158 * right-to-left script, e.g. "fa" (Farsi), except ones explicitly
159 * specifying Latin or Cyrillic script (which are the usual LTR
160 * alternatives).
161 *
162 * The list of right-to-left scripts appears in the 100-199 range in
163 * http://www.unicode.org/iso15924/iso15924-num.html, of which Arabic and
164 * Hebrew are by far the most widely used. We also recognize Thaana, N'Ko, and
165 * Tifinagh, which also have significant modern usage. The rest (Syriac,
166 * Samaritan, Mandaic, etc.) seem to have extremely limited or no modern usage
167 * and are not recognized.
168 * The languages usually written in a right-to-left script are taken as those
169 * with Suppress-Script: Hebr|Arab|Thaa|Nkoo|Tfng in
170 * http://www.iana.org/assignments/language-subtag-registry,
171 * as well as Sindhi (sd) and Uyghur (ug).
172 * The presence of other subtags of the language code, e.g. regions like EG
173 * (Egypt), is ignored.
174 */
175 static bool isRtlLanguage(String languageString) { //TODO(case insensitive??)
176 return const RegExp(@'^(ar|dv|he|iw|fa|nqo|ps|sd|ug|ur|yi|.*[-_]'
177 @'(Arab|Hebr|Thaa|Nkoo|Tfng))(?!.*[-_](Latn|Cyrl)($|-|_))'
178 @'($|-|_)', ignoreCase : true).hasMatch(languageString);
Alan Knight 2012/06/22 00:16:53 Is a regular expression, especially a 3-line one,
Emily Fortuna 2012/06/25 20:25:35 The language code specification system is fairly c
Alan Knight 2012/06/25 23:29:56 Grumble, grumble, ok.
179 }
180
181 /**
182 * Enforce the [html] snippet in RTL directionality regardless of overall
183 * context. If the html piece was enclosed by a tag, the direction will be
184 * applied to existing tag, otherwise a span tag will be added as wrapper.
185 * For this reason, if html snippet start with with tag, this tag must enclose
186 * the whole piece. If the tag already has a direction specified, this new one
187 * will override existing one in behavior (tested on FF and IE).
188 */
Alan Knight 2012/06/22 00:16:53 How badly do we need these? The other things don't
Emily Fortuna 2012/06/25 20:25:35 Since this is being developed not on a Windows mac
Alan Knight 2012/06/25 23:29:56 Maybe we should change it to something like "shoul
Emily Fortuna 2012/06/26 01:04:29 Done.
189 static String enforceRtlInHtml(String html) {
190 return _enforceInHtmlHelper(html, 'rtl');
191 }
192
193 /**
194 * Enforce RTL on both end of the given [text] using unicode BiDi formatting
195 * characters RLE and PDF.
196 */
197 static String enforceRtlInText(String text) {
198 return '$RLE$text$PDF';
199 }
200
201 /**
202 * Enforce the [html] snippet in LTR directionality regardless of overall
203 * context. If the html piece was enclosed by a tag, the direction will be
204 * applied to existing tag, otherwise a span tag will be added as wrapper.
205 * For this reason, if html snippet start with with tag, this tag must enclose
206 * the whole piece. If the tag already has a direction specified, this new one
207 * will override existing one in behavior (tested on FF and IE).
208 */
209 static String enforceLtrInHtml(String html) {
210 return _enforceInHtmlHelper(html, 'ltr');
211 }
212
213 /**
214 * Enforce LTR on both end of the given [text] using unicode BiDi formatting
215 * characters LRE and PDF.
216 */
217 static String enforceLtrInText(String text) {
218 return '$LRE$text$PDF';
219 }
220
221 /**
222 * Enforce the [html] snippet in the desired [direction] regardless of overall
223 * context. If the html piece was enclosed by a tag, the direction will be
224 * applied to existing tag, otherwise a span tag will be added as wrapper.
225 * For this reason, if html snippet start with with tag, this tag must enclose
226 * the whole piece. If the tag already has a direction specified, this new one
227 * will override existing one in behavior (tested on FF and IE).
228 */
229 static String _enforceInHtmlHelper(String html, String direction) {
230 if (html.startsWith('<')) {
231 StringBuffer buffer = new StringBuffer();
232 var startIndex = 0;
233 Iterator iterator = const RegExp('<\\w+').allMatches(
234 html).iterator();
Alan Knight 2012/06/22 00:16:53 This code seems confusing. Why are we asking for a
Emily Fortuna 2012/06/25 20:25:35 Fixed. That was from a refactor that apparently di
235 if (iterator.hasNext()) {
236 Match match = iterator.next();
237 buffer.add(html.substring(
238 startIndex, match.end())).add(' dir=$direction');
239 startIndex = match.end();
240 }
241 return buffer.add(html.substring(startIndex)).toString();
242 }
243 // '\n' is important for FF so that it won't incorrectly merge span groups.
244 return '\n<span dir=$direction>$html</span>';
245 }
246
247 /**
248 * Apply bracket guard to [str] using html span tag. This is to address the
249 * problem of messy bracket display that frequently happens in RTL layout.
250 * If [isRtlContext] is true, then we explicitly want to wrap in a span of RTL
251 * directionality, regardless of the estimated directionality.
252 */
253 static String guardBracketInHtml(String str, [bool isRtlContext]) {
254 var useRtl = isRtlContext == null ? hasAnyRtl(str) : isRtlContext;
255 RegExp regexp =
256 const RegExp(@'(\(.*?\)+)|(\[.*?\]+)|(\{.*?\}+)|(&lt;.*?(&gt;)+)');
257 if (useRtl) {
Alan Knight 2012/06/22 00:16:53 Is a regex really necessary here? Seems like strin
Emily Fortuna 2012/06/25 20:25:35 Renamed to make clearer?
Alan Knight 2012/06/25 23:29:56 Not a fan of all these regexes, but it'll do.
258 return _guardBracketHelper(str, regexp, '<span dir=rtl>', '</span>');
Alan Knight 2012/06/22 00:16:53 probably worth a statement to set a string to 'rtl
Emily Fortuna 2012/06/25 20:25:35 Done.
259 }
260 return _guardBracketHelper(str, regexp, '<span dir=ltr>', '</span>');
261 }
262
263 /**
264 * Apply bracket guard to [str] using LRM and RLM. This is to address the
265 * problem of messy bracket display that frequently happens in RTL layout.
266 * This version works for both plain text and html, but in some cases is not
267 * as good as guardBracketInHtml.
268 * If [isRtlContext] is true, then we explicitly want to wrap in a span of RTL
269 * directionality, regardless of the estimated directionality.
270 */
271 static String guardBracketInText(String str, [bool isRtlContext]) {
272 var useRtl = isRtlContext == null ? hasAnyRtl(str) : isRtlContext;
273 var mark = useRtl ? RLM : LRM;
274 return _guardBracketHelper(str,
275 const RegExp(@'(\(.*?\)+)|(\[.*?\]+)|(\{.*?\}+)|(<.*?>+)'), mark, mark);
276 }
277
278 /**
279 * (Mostly) reimplements the $& functionality of "replace" in JavaScript.
280 * Given a [str] and the [regexp] to match with, optionally supply a string to
281 * be inserted [before] the match and/or [after]. For example,
282 * `_guardBracketHelper('firetruck', const RegExp('truck'), 'hydrant', '!')`
283 * would return 'firehydrant!'.
284 */
285 // TODO(efortuna): Get rid of this once this is implemented in Dart.
286 // See Issue 2979.
287 static String _guardBracketHelper(String str, RegExp regexp, [String before,
288 String after]) {
289 StringBuffer buffer = new StringBuffer();
290 var startIndex = 0;
291 Iterable matches = regexp.allMatches(str);
292 for (Match match in matches) {
293 buffer.add(str.substring(startIndex, match.start())).add(before);
294 buffer.add(str.substring(match.start(), match.end())).add(after);
295 startIndex = match.end();
296 }
297 return buffer.add(str.substring(startIndex)).toString();
298 }
299
300 /**
301 * Estimates the directionality of [text] using the best known
302 * general-purpose method (using relative word counts). A
303 * TextDirection.UNKNOWN return value indicates completely neutral input.
304 * [isHtml] is true if [text] HTML or HTML-escaped.
305 *
306 * If the number of RTL words is above a certain percentage of the total
307 * number of strongly directional words, returns RTL.
308 * Otherwise, if any words are strongly or weakly LTR, returns LTR.
309 * Otherwise, returns UNKNOWN, which is used to mean `neutral'.
310 * Numbers and URLs are counted as weakly LTR.
311 */
312 static TextDirection estimateDirection(String text, [bool isHtml=false]) {
313 text = BidiUtils.stripHtmlIfNeeded(text, isHtml);
314 var rtlCount = 0;
315 var total = 0;
316 var hasWeaklyLtr = false;
317 // Split a string into 'words' for directionality estimation based on
318 // relative word counts.
319 for (String token in text.split(const RegExp(@'\s+'))) {
320 if (BidiUtils.startsWithRtl(token)) {
321 rtlCount++;
322 total++;
323 } else if (const RegExp(@'^http://').hasMatch(token)) {
324 // Checked if token looks like something that must always be LTR even in
325 // RTL text, such as a URL.
326 hasWeaklyLtr = true;
327 } else if (BidiUtils.hasAnyLtr(token)) {
328 total++;
329 } else if (const RegExp(@'\d').hasMatch(token)) {
330 // Checked if token contains any numerals.
331 hasWeaklyLtr = true;
332 }
333 }
334
335 if (total == 0) {
336 return hasWeaklyLtr ? TextDirection.LTR : TextDirection.UNKNOWN;
337 } else if (rtlCount > BidiUtils._RTL_DETECTION_THRESHOLD * total) {
338 return TextDirection.RTL;
339 } else {
340 return TextDirection.LTR;
341 }
342 }
343
344 /**
345 * Find the first index in [str] of the first closing parenthesis that does
346 * not match an opening parenthesis.
347 */
348 static int _unmatchedParenIndex(String str) {
349 int sum = 0;
350 int index = 0;
351 while (sum >= 0 || index > str.length) {
352 int char = str.charCodeAt(index);
353 if (char == '('.charCodeAt(0)) sum++;
354 else if (char == ')'.charCodeAt(0)) sum--;
355 index++;
356 }
357 return index;
358 }
359
360 /**
361 * Reimplementing the $num capability in the JavaScript regular expression
362 * "replace" method. Returns a Map with keys 'one'-'five' (yes,
363 * currently only implemented for up to five $num elements, since this
364 * function should be a temporary measure.), where dict\['one'\] is the
365 * equivalent of $1 in JavaScript, dict\['two'\] is $2, etc.
366 */
367 static Map _backreferenceHelper(String usr_str, String regexp) {
368 var keys = ['one', 'two', 'three', 'four', 'five'];
369 Iterator iterator = keys.iterator();
370 Map dict = {};
371 int i = regexp.indexOf('(');
372 Match match = new RegExp(regexp).firstMatch(usr_str);
373 if (match == null) return null;
374 String fullMatchedString = usr_str.substring(match.start(), match.end());
375 while (i > -1) {
376 Match beforeOneMatch = new RegExp(
377 regexp.substring(0, i)).firstMatch(fullMatchedString);
378 // Matching parenthesis is a context free language! We can't use
379 // regular expressions to help us here. :-( Awesome! (not really)
380 // Also note this code should not be used un-modified for general purpose
381 // $1, $2 support -- technically to be correct you need to match with
382 // regexp.substring(closingParenIndex+i+1) on the fullMatchedString, and
383 // then subtract the remainder as $1. We're able to take this shortcut
384 // here because we know for the particular regular expressions that this
385 // is used, this doesn't matter.
386 // TODO(efortuna): rewrite all this silliness when we have $1, $2 support
387 // in RegExp. See Issue 2979.
388 int closingParenIndex = _unmatchedParenIndex(regexp.substring(i+1));
389 String partialString = fullMatchedString.substring(beforeOneMatch.end());
390 Match oneMatch = new RegExp(regexp.substring(i,
391 closingParenIndex + i + 1)).firstMatch(partialString);
392 dict[iterator.next()] = partialString.substring(oneMatch.start(),
393 oneMatch.end());
394 i = regexp.indexOf('(', closingParenIndex + i + 1);
395 }
396 return dict;
397 }
398
399 /**
400 * Replace the double and single quote directly after a Hebrew character in
401 * [str] with GERESH and GERSHAYIM. This is most likely the user's intention.
402 */
403 static String normalizeHebrewQuote(String str) {
404 String regex1 = @'([\u0591-\u05f2])"';
405 String regex2 = @"([\u0591-\u05f2])'";
406 Map keys = _backreferenceHelper(str, regex1);
407 Map keys2 = _backreferenceHelper(str, regex2);
408 if (keys != null) {
409 str = str.replaceAll(new RegExp(regex1), '${keys["one"]}\u05f4');
Alan Knight 2012/06/22 00:16:53 Does this actually work? With a hard-coded "one"?
Emily Fortuna 2012/06/25 20:25:35 Removed regular expression version.
410 }
411 if (keys2 != null) {
412 str= str.replaceAll(new RegExp(regex2), '${keys2["one"]}\u05f3');
413 }
414 return str;
415 }
416
417 /**
418 * Swap location parameters and 'left'/'right' in CSS specification in
419 * [cssStr]. The processed string will be suited for RTL layout. Though this
420 * function can cover most cases, there are always exceptions. It is suggested
421 * the developer put those exceptions in separate group of CSS string.
422 */
Alan Knight 2012/06/22 00:16:53 Do we really need to do this? Especially for CSS,
Emily Fortuna 2012/06/25 20:25:35 Done.
423 static String mirrorCSS(String cssStr) {
424 var regex = ':\\s*([.\\d][.\\w]*)\\s+([.\\d][.\\w]*)\\s+([.\\d][.\\w]*)'
425 '\\s+([.\\d][.\\w]*)';
426 var tempStr = '%%%%';
427 Map keys = _backreferenceHelper(cssStr, regex);
428 if (keys != null) {
429 // Reverse dimensions regex.
430 cssStr = cssStr.replaceAll(new RegExp(regex),
431 ':${keys["one"]} ${keys["four"]} ${keys["three"]} ${keys["two"]}');
432 }
433 // Swap left and right.
434 return cssStr.
435 replaceAll(const RegExp('left', ignoreCase:true), tempStr).
436 replaceAll(const RegExp('right', ignoreCase:true), 'left').
437 replaceAll(new RegExp(tempStr), 'right');
438 }
439
440 /**
441 * Check the directionality of [str], return true if the piece of
442 * text should be laid out in RTL direction. If [isHtml] is true, the string
443 * is HTML or HTML-escaped.
444 */
445 static bool detectRtlDirectionality(String str, [bool isHtml]) {
Alan Knight 2012/06/22 00:16:53 Wouldn't this be better called e.g. isRTL?
Emily Fortuna 2012/06/25 20:25:35 isRTL is reserved for knowing for certain that the
446 return estimateDirection(str, isHtml) == TextDirection.RTL;
447 }
448 }
OLDNEW

Powered by Google App Engine
This is Rietveld 408576698