Allow the ZWJ and ZWNJ characters in IdentifierPart as per ES5.

Allow the ZWJ and ZWNJ characters in IdentifierPart as per ES5.

Patch from Mathias Bynens <mathias@qiwi.be>;
BUG=1958
TEST=Try `var a\u200c\u200d;` and confirm it doesn't throw an error.

Committed: https://code.google.com/p/v8/source/detail?r=10800

[mathias, 2012-02-22 09:49:37]

As an `IdentifierStart` character is the most common case for an
`IdentifierPart`, I figured it would be best to perform the U+200C and U+200D
checks right after, but before the more expensive Unicode category lookups.

https://chromiumcodereview.appspot.com/9433031/diff/1/src/char-predicates.h
File src/char-predicates.h (right):

https://chromiumcodereview.appspot.com/9433031/diff/1/src/char-predicates.h#n...
src/char-predicates.h:60: || c == 0x200D
As an `IdentifierStart` character is the most common case for an
`IdentifierPart`, I figured it would be best to perform the U+200C and U+200D
checks right after, but before the more expensive Unicode category lookups.

[Lasse Reichstein Nielsen, 2012-02-22 14:03:28]

https://chromiumcodereview.appspot.com/9433031/diff/1/src/char-predicates.h
File src/char-predicates.h (right):

https://chromiumcodereview.appspot.com/9433031/diff/1/src/char-predicates.h#n...
src/char-predicates.h:60: || c == 0x200D
Good thought.
I'd prefer moving the number (digit, really) test before the ZWNJ/ZWJ test,
since digits are probably more likely to be in actual identifiers. 

Perhaps add a comment after the constants (like "  // U+200C is
ZeroWidthNonJoiner.").

[mathias, 2012-02-22 14:13:34]

> I'd prefer moving the number (digit, really) test before the ZWNJ/ZWJ test,
> since digits are probably more likely to be in actual identifiers. 

I was wondering about that. Done.

> Perhaps add a comment after the constants (like "  // U+200C is
> ZeroWidthNonJoiner.").

Done.

[mathias, 2012-02-22 22:24:53]

Adding ricow@chromium.org (issue owner) as reviewer.

[Lasse Reichstein Nielsen, 2012-02-23 08:16:40]

Committed. Thank you for the patch.