Extensions Docs Server: Preserve JSON declaration order in extensions documentation

Extensions Docs Server: Preserve JSON declaration order in extensions documentation

All JSON is now parsed using an OrderedDict, so the order of properties is preserved.
The order was also being messed up within the JSON Schema Compiler, because
lists were being converted to (unordered) dictionaries.

BUG=146649, 151867
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=168702

[cduvall, 2012-10-06 01:01:35]

kalman: code changes
jennb: can you confirm that this works with your Windows checkout?

Thanks!

[not at google - send to devlin, 2012-10-08 03:52:05]

There's a bug filed somewhere about sorting items in dictionaries. Summary: we
shouldn't sort them, they should appear in the order that they are declared in
the JSON.

So I'd try to fix this bug in two ways:

Firstly by using an OrderedDict everywhere. It might be a bit of a pain but
hopefully there's a nice way to swap it in (we did it earlier when reading the
JSON). Also, they only work in python 2.7+ so we'll need to fail gracefully if
it's not available (probably have a little library).

If that's not possible, then write a custom comparison in the test that's
failing (perhaps something which runs over the arrays within the dicts and sorts
them :\).

[cduvall, 2012-10-12 23:48:34]

+aa since kalman is out

[Aaron Boodman, 2012-10-16 06:22:34]

Tests for json_parse and ordered_dict?

http://codereview.chromium.org/11079010/diff/9009/tools/json_schema_compiler/...
File tools/json_schema_compiler/ordered_dict.py (right):

http://codereview.chromium.org/11079010/diff/9009/tools/json_schema_compiler/...
tools/json_schema_compiler/ordered_dict.py:17: class OrderedDict(object):
Makes me feel icky to have the O(n) stuff in here. Consider implementing by
having a nested list as well as a nested dictionary in order to get O(1).

I know it will be rare for us to have enough data to matter, but it will be
embarrassing that one time it is used that way.

http://codereview.chromium.org/11079010/diff/9009/tools/json_schema_compiler/...
tools/json_schema_compiler/ordered_dict.py:68: return any(key == k for k, v in
self._dict)
Does any() stop on the first result?

http://codereview.chromium.org/11079010/diff/9009/tools/json_schema_compiler/...
tools/json_schema_compiler/ordered_dict.py:81: elif isinstance(other,
OrderedDict):
Can you do self._dict == other._dict?

[cduvall, 2012-10-17 00:30:13]

https://codereview.chromium.org/11079010/diff/9009/tools/json_schema_compiler...
File tools/json_schema_compiler/ordered_dict.py (right):

https://codereview.chromium.org/11079010/diff/9009/tools/json_schema_compiler...
tools/json_schema_compiler/ordered_dict.py:17: class OrderedDict(object):
On 2012/10/16 06:22:34, Aaron Boodman wrote:
> Makes me feel icky to have the O(n) stuff in here. Consider implementing by
> having a nested list as well as a nested dictionary in order to get O(1).
> 
> I know it will be rare for us to have enough data to matter, but it will be
> embarrassing that one time it is used that way.

Done.

https://codereview.chromium.org/11079010/diff/9009/tools/json_schema_compiler...
tools/json_schema_compiler/ordered_dict.py:68: return any(key == k for k, v in
self._dict)
On 2012/10/16 06:22:34, Aaron Boodman wrote:
> Does any() stop on the first result?

Using the dict now, so any isn't used, but it does stop when it finds a match.

https://codereview.chromium.org/11079010/diff/9009/tools/json_schema_compiler...
tools/json_schema_compiler/ordered_dict.py:81: elif isinstance(other,
OrderedDict):
On 2012/10/16 06:22:34, Aaron Boodman wrote:
> Can you do self._dict == other._dict? 

Done.

[Aaron Boodman, 2012-10-24 01:52:22]

http://codereview.chromium.org/11079010/diff/12001/tools/json_schema_compiler...
File tools/json_schema_compiler/ordered_dict.py (right):

http://codereview.chromium.org/11079010/diff/12001/tools/json_schema_compiler...
tools/json_schema_compiler/ordered_dict.py:32: self._dict = dict(self._list)
You should make one of these a pointer into the other, so that you can do all
operations quickly.

For example, the list value could be the hash key, and the hash value could be
the actual value. That way you never need to iterate the list looking for
something.

http://codereview.chromium.org/11079010/diff/12001/tools/json_schema_compiler...
tools/json_schema_compiler/ordered_dict.py:47: if key in self:
Won't this always be true due to the above line?

http://codereview.chromium.org/11079010/diff/12001/tools/json_schema_compiler...
tools/json_schema_compiler/ordered_dict.py:67: self._dict.update(other)
Seems this is unnecessary?

[cduvall, 2012-10-26 01:09:57]

I like the list as a pointer to the dict, I should have done that from the
start.

https://codereview.chromium.org/11079010/diff/12001/tools/json_schema_compile...
File tools/json_schema_compiler/ordered_dict.py (right):

https://codereview.chromium.org/11079010/diff/12001/tools/json_schema_compile...
tools/json_schema_compiler/ordered_dict.py:32: self._dict = dict(self._list)
On 2012/10/24 01:52:22, Aaron Boodman wrote:
> You should make one of these a pointer into the other, so that you can do all
> operations quickly.
> 
> For example, the list value could be the hash key, and the hash value could be
> the actual value. That way you never need to iterate the list looking for
> something.

Done.

https://codereview.chromium.org/11079010/diff/12001/tools/json_schema_compile...
tools/json_schema_compiler/ordered_dict.py:47: if key in self:
On 2012/10/24 01:52:22, Aaron Boodman wrote:
> Won't this always be true due to the above line?

Done.

https://codereview.chromium.org/11079010/diff/12001/tools/json_schema_compile...
tools/json_schema_compiler/ordered_dict.py:67: self._dict.update(other)
On 2012/10/24 01:52:22, Aaron Boodman wrote:
> Seems this is unnecessary?

Done.

[not at google - send to devlin, 2012-11-16 18:45:56]

aa has already reviewed ordered_dict.py so I didn't look at it.

https://codereview.chromium.org/11079010/diff/18001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/api_data_source.py (right):

https://codereview.chromium.org/11079010/diff/18001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/api_data_source.py:23: if
item.get('nodoc', False):
might be cleaner exposing like json_parse.IsDict(...)

https://codereview.chromium.org/11079010/diff/18001/tools/json_schema_compile...
File tools/json_schema_compiler/json_parse.py (right):

https://codereview.chromium.org/11079010/diff/18001/tools/json_schema_compile...
tools/json_schema_compiler/json_parse.py:13: 
Can we incorporate this file into ordered_dict.py? I'd prefer there to be just a
single file with an OrderedDict class and I suppose these few little helper
method(s) (like Parse - would be nice to have that as a method on OrderedDict
but might be a contrivance to do so).

... import json_comment_eater

try:
  from collections import OrderedDict

  # Successfully imported, so we're Python >=2.7, and json.parse
  # supports object_pairs_hook.
  def Parse(json):
    return json.loads(json, object_pairs_hook=OrderedDict)

except ImportError:
  # Failed to import, so we're Python < 2.7, and json.parse doesn't
  # support object_pairs_hook. simplejson however does, but it's
  # slow. We also need to provide our own OrderedDict implementation.
  warn("using simplejson, so this will be slow!")
  ... import simplejson

  class OrderedDict(object):
     ... contents of ordered_dict.py goes here

  def Parse(json):
    return simplejson.loads(json, object_pairs_hook=OrderedDict)

[cduvall, 2012-11-17 01:55:11]

https://codereview.chromium.org/11079010/diff/18001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/api_data_source.py (right):

https://codereview.chromium.org/11079010/diff/18001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/api_data_source.py:23: if
item.get('nodoc', False):
On 2012/11/16 18:45:56, kalman wrote:
> might be cleaner exposing like json_parse.IsDict(...)

Done.

https://codereview.chromium.org/11079010/diff/18001/tools/json_schema_compile...
File tools/json_schema_compiler/json_parse.py (right):

https://codereview.chromium.org/11079010/diff/18001/tools/json_schema_compile...
tools/json_schema_compiler/json_parse.py:13: 
On 2012/11/16 18:45:56, kalman wrote:
> Can we incorporate this file into ordered_dict.py? I'd prefer there to be just
a
> single file with an OrderedDict class and I suppose these few little helper
> method(s) (like Parse - would be nice to have that as a method on OrderedDict
> but might be a contrivance to do so).
> 
> ... import json_comment_eater
> 
> try:
>   from collections import OrderedDict
> 
>   # Successfully imported, so we're Python >=2.7, and json.parse
>   # supports object_pairs_hook.
>   def Parse(json):
>     return json.loads(json, object_pairs_hook=OrderedDict)
> 
> except ImportError:
>   # Failed to import, so we're Python < 2.7, and json.parse doesn't
>   # support object_pairs_hook. simplejson however does, but it's
>   # slow. We also need to provide our own OrderedDict implementation.
>   warn("using simplejson, so this will be slow!")
>   ... import simplejson
> 
>   class OrderedDict(object):
>      ... contents of ordered_dict.py goes here
> 
>   def Parse(json):
>     return simplejson.loads(json, object_pairs_hook=OrderedDict)

Done.

[not at google - send to devlin, 2012-11-17 05:55:16]

lgtm

https://codereview.chromium.org/11079010/diff/34001/tools/json_schema_compile...
File tools/json_schema_compiler/json_schema.py (right):

https://codereview.chromium.org/11079010/diff/34001/tools/json_schema_compile...
tools/json_schema_compiler/json_schema.py:14: return (json_parse.IsDict(thing)
and thing.get('nocompile', False))
parens unnecessary?

[cduvall, 2012-11-19 20:53:41]

http://codereview.chromium.org/11079010/diff/34001/tools/json_schema_compiler...
File tools/json_schema_compiler/json_schema.py (right):

http://codereview.chromium.org/11079010/diff/34001/tools/json_schema_compiler...
tools/json_schema_compiler/json_schema.py:14: return (json_parse.IsDict(thing)
and thing.get('nocompile', False))
On 2012/11/17 05:55:16, kalman wrote:
> parens unnecessary?

Done.

[commit-bot: I haz the power, 2012-11-19 20:55:42]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/cduvall@chromium.org/11079010/39001

[commit-bot: I haz the power, 2012-11-19 20:55:51]

Presubmit check for 11079010-39001 failed and returned exit status 1.


Traceback (most recent call last):
  File "docs/server2/integration_test.py", line 27, in <module>
    from handler import Handler
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/handler.py",
line 14, in <module>
    from api_data_source import APIDataSource
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/api_data_source.py",
line 11, in <module>
    import third_party.json_schema_compiler.json_parse as json_parse
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/third_party/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson
Running presubmit commit checks ...

** Presubmit ERRORS **
IntegrationTest failed!

./sidenav_data_source_test.py failed!
Command ./sidenav_data_source_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2
Traceback (most recent call last):
  File "./sidenav_data_source_test.py", line 13, in <module>
    from sidenav_data_source import SidenavDataSource
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/sidenav_data_source.py",
line 11, in <module>
    from third_party.json_schema_compiler.model import UnixName
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/third_party/json_schema_compiler/model.py",
line 9, in <module>
    from json_parse import OrderedDict
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/third_party/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson


./samples_data_source_test.py failed!
Command ./samples_data_source_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2
Traceback (most recent call last):
  File "./samples_data_source_test.py", line 11, in <module>
    from samples_data_source import SamplesDataSource
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/samples_data_source.py",
line 13, in <module>
    import third_party.json_schema_compiler.model as model
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/third_party/json_schema_compiler/model.py",
line 9, in <module>
    from json_parse import OrderedDict
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/third_party/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson


./api_data_source_test.py failed!
Command ./api_data_source_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2
Traceback (most recent call last):
  File "./api_data_source_test.py", line 11, in <module>
    from api_data_source import (APIDataSource,
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/api_data_source.py",
line 11, in <module>
    import third_party.json_schema_compiler.json_parse as json_parse
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/third_party/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson


./model_test.py failed!
Command ./model_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
Traceback (most recent call last):
  File "./model_test.py", line 6, in <module>
    from json_schema import CachedLoad
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_schema.py",
line 9, in <module>
    import json_parse
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson


./cpp_type_generator_test.py failed!
Command ./cpp_type_generator_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
Traceback (most recent call last):
  File "./cpp_type_generator_test.py", line 6, in <module>
    from cpp_type_generator import CppTypeGenerator
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/cpp_type_generator.py",
line 6, in <module>
    from model import PropertyType
  File "/b/commit-queue/workdir/chromium/tools/json_schema_compiler/model.py",
line 9, in <module>
    from json_parse import OrderedDict
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson


./cpp_util_test.py failed!
Command ./cpp_util_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
Traceback (most recent call last):
  File "./cpp_util_test.py", line 6, in <module>
    import cpp_util
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/cpp_util.py", line
8, in <module>
    from model import PropertyType
  File "/b/commit-queue/workdir/chromium/tools/json_schema_compiler/model.py",
line 9, in <module>
    from json_parse import OrderedDict
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson


./json_parse_test.py failed!
Command ./json_parse_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
Traceback (most recent call last):
  File "./json_parse_test.py", line 9, in <module>
    from json_parse import OrderedDict
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson


./json_schema_test.py failed!
Command ./json_schema_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
Traceback (most recent call last):
  File "./json_schema_test.py", line 6, in <module>
    import json_schema
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_schema.py",
line 9, in <module>
    import json_parse
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson


./schema_util_test.py failed!
Command ./schema_util_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
Traceback (most recent call last):
  File "./schema_util_test.py", line 6, in <module>
    from schema_util import JsFunctionNameToClassName
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/schema_util.py",
line 7, in <module>
    import json_parse
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson


./idl_schema_test.py failed!
Command ./idl_schema_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
Traceback (most recent call last):
  File "./idl_schema_test.py", line 6, in <module>
    import idl_schema
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/idl_schema.py",
line 12, in <module>
    import schema_util
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/schema_util.py",
line 7, in <module>
    import json_parse
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 19, in <module>
    import simplejson
ImportError: No module named simplejson


Presubmit checks took 2.4s to calculate.

[commit-bot: I haz the power, 2012-11-19 21:45:33]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/cduvall@chromium.org/11079010/36003

[commit-bot: I haz the power, 2012-11-19 21:47:01]

Presubmit check for 11079010-36003 failed and returned exit status 1.


WARNING:root:Using simplejson to parse, this might be slow! Upgrade to Python
2.7.
Running presubmit commit checks ...

** Presubmit ERRORS **
./model_test.py failed!
Command ./model_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
WARNING:root:Using simplejson to parse, this might be slow! Upgrade to Python
2.7.
Traceback (most recent call last):
  File "./model_test.py", line 6, in <module>
    from json_schema import CachedLoad
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_schema.py",
line 9, in <module>
    import json_parse
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 49, in <module>
    import simplejson
ImportError: No module named simplejson


./cpp_type_generator_test.py failed!
Command ./cpp_type_generator_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
WARNING:root:Using simplejson to parse, this might be slow! Upgrade to Python
2.7.
Traceback (most recent call last):
  File "./cpp_type_generator_test.py", line 6, in <module>
    from cpp_type_generator import CppTypeGenerator
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/cpp_type_generator.py",
line 6, in <module>
    from model import PropertyType
  File "/b/commit-queue/workdir/chromium/tools/json_schema_compiler/model.py",
line 9, in <module>
    from json_parse import OrderedDict
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 49, in <module>
    import simplejson
ImportError: No module named simplejson


./cpp_util_test.py failed!
Command ./cpp_util_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
WARNING:root:Using simplejson to parse, this might be slow! Upgrade to Python
2.7.
Traceback (most recent call last):
  File "./cpp_util_test.py", line 6, in <module>
    import cpp_util
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/cpp_util.py", line
8, in <module>
    from model import PropertyType
  File "/b/commit-queue/workdir/chromium/tools/json_schema_compiler/model.py",
line 9, in <module>
    from json_parse import OrderedDict
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 49, in <module>
    import simplejson
ImportError: No module named simplejson


./json_parse_test.py failed!
Command ./json_parse_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
WARNING:root:Using simplejson to parse, this might be slow! Upgrade to Python
2.7.
Traceback (most recent call last):
  File "./json_parse_test.py", line 9, in <module>
    from json_parse import OrderedDict
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 49, in <module>
    import simplejson
ImportError: No module named simplejson


./json_schema_test.py failed!
Command ./json_schema_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
WARNING:root:Using simplejson to parse, this might be slow! Upgrade to Python
2.7.
Traceback (most recent call last):
  File "./json_schema_test.py", line 6, in <module>
    import json_schema
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_schema.py",
line 9, in <module>
    import json_parse
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 49, in <module>
    import simplejson
ImportError: No module named simplejson


./schema_util_test.py failed!
Command ./schema_util_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
WARNING:root:Using simplejson to parse, this might be slow! Upgrade to Python
2.7.
Traceback (most recent call last):
  File "./schema_util_test.py", line 6, in <module>
    from schema_util import JsFunctionNameToClassName
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/schema_util.py",
line 7, in <module>
    import json_parse
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 49, in <module>
    import simplejson
ImportError: No module named simplejson


./idl_schema_test.py failed!
Command ./idl_schema_test.py returned non-zero exit status 1 in
/b/commit-queue/workdir/chromium/tools/json_schema_compiler
WARNING:root:Using simplejson to parse, this might be slow! Upgrade to Python
2.7.
Traceback (most recent call last):
  File "./idl_schema_test.py", line 6, in <module>
    import idl_schema
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/idl_schema.py",
line 12, in <module>
    import schema_util
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/schema_util.py",
line 7, in <module>
    import json_parse
  File
"/b/commit-queue/workdir/chromium/tools/json_schema_compiler/json_parse.py",
line 49, in <module>
    import simplejson
ImportError: No module named simplejson


Presubmit checks took 82.6s to calculate.

[cduvall, 2012-11-19 22:40:36]

kalman: could you take another look at this? Mainly, do you think the changes I
made to simplejson are ok.

[commit-bot: I haz the power, 2012-11-20 00:06:56]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/cduvall@chromium.org/11079010/34011

[commit-bot: I haz the power, 2012-11-20 02:13:25]

Change committed as 168702