Add WebIDL compliant parser plus tests

Add WebIDL and Pepper compliant lexer/parser tool. 

This CL is the first step towards supporting automatic doc 
generation. This lexer more closely follows the WebIDL 
spec. 

Please review idl_node, idl_parser, idl_node
idl_log.py is unchanged from the original

This is a complete re-write of the original lexer/parser, as well as simplifications to IDLNode. 

All *.in and *.idl files are for test purposes only.

To test: src/tools/idl_parser/run_tests.py

NOTRY=true
R=sehr@chromium.org
BUG=224150
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=193973

[noelallen1, 2013-04-04 20:26:23]

ptal

[sehr, 2013-04-08 16:41:20]

First few comments.  Also please note that copyright dates are frequently using
2012, when they should be 2013.  More as I get further.

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer.py
File tools/idl_parser/idl_lexer.py (right):

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:8: The lexer is uses the PLY to build a tokenizer
which understands WebIDL
s/is uses/uses/

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:11: 2- The grammer for single characters
operators: '?', ':', '{', ...
grammar

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:160: r'(/\*(.|\n)*?\*/)|(//.*(\n[ \t]*//.*)*)'
what is the "?" there for?

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:222: caret = '\t^'.expandtabs(pos)
This is a "cute" way of generating pos * ' '.  Is this the best way to do this?

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:275: # single space.  The new source is then
tokenized and compared against the
s/space/newline/  ?

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:331: args = ['--test', 'test_lexer/values.in',
'test_lexer/keywords.in']
This is sort of ugly, baking in test input paths.  Is this really how they want
tests to be done?

[sehr, 2013-04-08 18:37:06]

Some more nits and a couple of questions.  I have been through the whole CL
once.

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_node.py
File tools/idl_parser/idl_node.py (right):

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_node....
tools/idl_parser/idl_node.py:31: # If the item is not a list
add something like: "return a list containing one item"

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_node....
tools/idl_parser/idl_node.py:35: # Make a copy we can modify
Please make sure that comments end with a period consistently.

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_node....
tools/idl_parser/idl_node.py:116: return self._properties['FILENAME'],
self._properties['LINENO']
use self.GetProperty here.

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_node....
tools/idl_parser/idl_node.py:168: if self._cls in typelist: return True
Does this work?

return self._cls in typelist

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
File tools/idl_parser/idl_parser.py (right):

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:138: # The various productions are anotated so
that the WHOLE number greater than
annotated

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:139: # zero in the comment denote the matching
WebIDL grammer definition.
denotes ... grammar

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:141: # Productions with a factional component in
the comment denote additions to
fractional

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:165: # [0.2] Produce a COMMENT and agregate
sibling comments
aggregate

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:426: val = '-Infinitiy'
Infinity

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:830: # is triggered logging an error, and parsing
recover happens as the
recovery

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:1084: # Renable output
Re-enable

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:1153: args = ['--test', 'test_parser/*.idl']
Again, should we be encoding paths here?

[noelallen1, 2013-04-11 21:52:09]

ptal

Moved tests into their own harness, which now runs on the command-line via -
run_tests.py or automatically via PRESUBMIT whenever a file in this directory
changes.

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer.py
File tools/idl_parser/idl_lexer.py (right):

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:8: The lexer is uses the PLY to build a tokenizer
which understands WebIDL
On 2013/04/08 16:41:20, sehr1 wrote:
> s/is uses/uses/

Done.

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:11: 2- The grammer for single characters
operators: '?', ':', '{', ...
On 2013/04/08 16:41:20, sehr1 wrote:
> grammar

Done.

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:160: r'(/\*(.|\n)*?\*/)|(//.*(\n[ \t]*//.*)*)'
On 2013/04/08 16:41:20, sehr1 wrote:
> what is the "?" there for?
I will add a new test to the lexer to vet comments, and validate.

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:222: caret = '\t^'.expandtabs(pos)
On 2013/04/08 16:41:20, sehr1 wrote:
> This is a "cute" way of generating pos * ' '.  Is this the best way to do
this?

' ' * pos + '^' would also work.

So would using width formats, but I ' '*pos is probably more common.

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:275: # single space.  The new source is then
tokenized and compared against the
On 2013/04/08 16:41:20, sehr1 wrote:
> s/space/newline/  ?

Done.

https://codereview.chromium.org/13498002/diff/5001/tools/idl_parser/idl_lexer...
tools/idl_parser/idl_lexer.py:331: args = ['--test', 'test_lexer/values.in',
'test_lexer/keywords.in']
This is fixed in a different CL
On 2013/04/08 16:41:20, sehr1 wrote:
> This is sort of ugly, baking in test input paths.  Is this really how they
want
> tests to be done?

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
File tools/idl_parser/idl_parser.py (right):

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:138: # The various productions are anotated so
that the WHOLE number greater than
On 2013/04/08 18:37:07, sehr1 wrote:
> annotated

Done.

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:139: # zero in the comment denote the matching
WebIDL grammer definition.
On 2013/04/08 18:37:07, sehr1 wrote:
> denotes ... grammar

Done.

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:141: # Productions with a factional component in
the comment denote additions to
On 2013/04/08 18:37:07, sehr1 wrote:
> fractional

Done.

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:165: # [0.2] Produce a COMMENT and agregate
sibling comments
On 2013/04/08 18:37:07, sehr1 wrote:
> aggregate

Done.

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:426: val = '-Infinitiy'
On 2013/04/08 18:37:07, sehr1 wrote:
> Infinity

Done.

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:830: # is triggered logging an error, and parsing
recover happens as the
On 2013/04/08 18:37:07, sehr1 wrote:
> recovery

Done.

https://codereview.chromium.org/13498002/diff/9001/tools/idl_parser/idl_parse...
tools/idl_parser/idl_parser.py:1084: # Renable output
On 2013/04/08 18:37:07, sehr1 wrote:
> Re-enable

Done.

[sehr, 2013-04-12 15:54:57]

LGTM with a few nits.

https://chromiumcodereview.appspot.com/13498002/diff/29001/tools/idl_parser/i...
File tools/idl_parser/idl_node.py (right):

https://chromiumcodereview.appspot.com/13498002/diff/29001/tools/idl_parser/i...
tools/idl_parser/idl_node.py:119: return self._properties['FILENAME'],
self._properties['LINENO']
sometimes you use _properties, and sometimes GetProperty.  Please be consistent.

https://chromiumcodereview.appspot.com/13498002/diff/29001/tools/idl_parser/i...
File tools/idl_parser/idl_parser_test.py (right):

https://chromiumcodereview.appspot.com/13498002/diff/29001/tools/idl_parser/i...
tools/idl_parser/idl_parser_test.py:45: msg = 'Mismatch property %s: %s vs
%s.\n' % (key, expect, actual)
Mismatched

(to match the below)

https://chromiumcodereview.appspot.com/13498002/diff/29001/tools/idl_parser/i...
tools/idl_parser/idl_parser_test.py:51: msg = 'Misatched tree at line
%d:\n%sVS\n%s' % (lineno, value, quick)
Mismatched

[noelallen1, 2013-04-12 16:52:15]

https://chromiumcodereview.appspot.com/13498002/diff/29001/tools/idl_parser/i...
File tools/idl_parser/idl_node.py (right):

https://chromiumcodereview.appspot.com/13498002/diff/29001/tools/idl_parser/i...
tools/idl_parser/idl_node.py:119: return self._properties['FILENAME'],
self._properties['LINENO']
On 2013/04/12 15:54:57, sehr1 wrote:
> sometimes you use _properties, and sometimes GetProperty.  Please be
consistent.

Done.

https://chromiumcodereview.appspot.com/13498002/diff/29001/tools/idl_parser/i...
File tools/idl_parser/idl_parser_test.py (right):

https://chromiumcodereview.appspot.com/13498002/diff/29001/tools/idl_parser/i...
tools/idl_parser/idl_parser_test.py:45: msg = 'Mismatch property %s: %s vs
%s.\n' % (key, expect, actual)
On 2013/04/12 15:54:57, sehr1 wrote:
> Mismatched
> 
> (to match the below)

Done.

https://chromiumcodereview.appspot.com/13498002/diff/29001/tools/idl_parser/i...
tools/idl_parser/idl_parser_test.py:51: msg = 'Misatched tree at line
%d:\n%sVS\n%s' % (lineno, value, quick)
On 2013/04/12 15:54:57, sehr1 wrote:
> Mismatched

Done.

[commit-bot: I haz the power, 2013-04-12 16:52:40]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/noelallen@chromium.org/13498002/35001

[commit-bot: I haz the power, 2013-04-12 16:52:46]

Presubmit check for 13498002-35001 failed and returned exit status 1.

INFO:root:Found 12 file(s).

testExpectedType (__main__.PepperIDLLexer) ... ok
testRebuildText (__main__.PepperIDLLexer) ... ok
testExpectedType (__main__.WebIDLLexer) ... ok
testRebuildText (__main__.WebIDLLexer) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.049s

OK
Traceback (most recent call last):
  File "./idl_parser_test.py", line 67, in <module>
    unittest.main(verbosity=2)
TypeError: __init__() got an unexpected keyword argument 'verbosity'
Running presubmit commit checks ...
Running /b/commit-queue/workdir/chromium/PRESUBMIT.py
Running /b/commit-queue/workdir/chromium/tools/PRESUBMIT.py
Running /b/commit-queue/workdir/chromium/tools/idl_parser/PRESUBMIT.py
Running ./idl_lexer_test.py
Running ./idl_parser_test.py

** Presubmit Warnings **
Found lines longer than 80 characters (first 5 shown).
  tools/idl_parser/idl_parser.py, line 217, 83 chars \
  tools/idl_parser/idl_parser.py, line 258, 86 chars \
  tools/idl_parser/idl_parser.py, line 268, 83 chars \
  tools/idl_parser/idl_parser.py, line 281, 81 chars \
  tools/idl_parser/idl_parser.py, line 309, 83 chars

License must match:
.*? Copyright (\(c\) )?(2013|2012|2011|2010|2009|2008|2007|2006|2006-2008) The
Chromium Authors\. All rights reserved\.\n.*? Use of this source code is
governed by a BSD-style license that can be\n.*? found in the LICENSE file\.(?:
\*/)?\n
Found a bad license header in these files:
  tools/idl_parser/idl_parser.py

** Presubmit ERRORS **
./idl_parser_test.py failed!
Command ./idl_parser_test.py --verbose returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/idl_parser

Presubmit checks took 1.4s to calculate.

[noelallen1, 2013-04-12 18:48:52]

Committed patchset #11 manually as r193973 (presubmit successful).