Moving prepopulated search engines to a JSON file.

tools/json_to_struct/json_to_struct.py

Index: tools/json_to_struct/json_to_struct.py
diff --git a/tools/json_to_struct/json_to_struct.py b/tools/json_to_struct/json_to_struct.py
new file mode 100644
index 0000000000000000000000000000000000000000..e35bb309b82ec14ac7e0b7e2f6b5b324a7af04e8
--- /dev/null
+++ b/tools/json_to_struct/json_to_struct.py
@@ -0,0 +1,198 @@
+#!/usr/bin/env python
+# Copyright 2012 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# Format for the JSON schema file:
+# {
+#   "type_name": "DesiredCStructName",
+#   "headers": [          // Optional list of headers to be included by the .h.
+#     "path/to/header.h"
+#   ],
+#   "schema": [           // Fields of the generated structure.
+#     {
+#       "field": "my_enum_field",
+#       "type": "enum",   // Either: int, string, string16, enum, array.
+#       "default": "RED", // Optional. Cannot be used for array.
+#       "ctype": "Color"  // Only for enum, specify the C type.
+#     },
+#     {
+#       "field": "my_int_array_field",
+#       "type": "array",
+#       "contents": {
+#         "type": "int"   // Either: int, string, string16, enum, array.
+#       }
+#     },
+#     ...
+#   ]
+# }
+#
+# Format for the JSON description file:
+# {
+#   "int_variables": {    // An optional list of constant int variables.
+#     "kDesiredConstantName": 45
+#   },
+#   "elements": {         // All the elements for which to create static
+#                         // initialization code in the .cc file.
+#     "my_const_variable": {
+#       "my_int_field": 10,
+#       "my_string_field": "foo bar",
+#       "my_enum_field": "BLACK",
+#       "my_int_array_field": [ 1, 2, 3, 5, 7 ],
+#     },
+#     "my_other_const_variable": {
+#       ...
+#     }
+#   }
+# }
+
+import json
+from datetime import datetime
+import os.path
+import sys
+import optparse
+_script_path = os.path.realpath(__file__)
+sys.path.insert(0, os.path.normpath(_script_path + "/../../"))

[not at google - send to devlin, 2012/11/12 18:38:13]

I think it'd be polite to remove this from the path after importing the required
modules. Inside try-finally preferably.

[beaudoin, 2012/11/13 18:44:26]

Done.

+import json_comment_eater
+import struct_generator
+import element_generator
+
+HEAD = """// Copyright (c) %d The Chromium Authors. All rights reserved.

[not at google - send to devlin, 2012/11/12 18:38:13]

I thought we weren't generating the (c) anymore?

[beaudoin, 2012/11/13 18:44:26]

Done.

+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+// GENERATED FROM THE SCHEMA DEFINITION AND DESCRIPTION IN
+//   %s
+//   %s
+// DO NOT EDIT.
+
+"""
+
+def GenerateHeaderGuard(h_filename):

[not at google - send to devlin, 2012/11/12 18:38:13]

Private functions should begin with an underscore, presumably that's all of the
functions in this file.

[beaudoin, 2012/11/13 18:44:26]

Done.

+  """Generates the string used in #ifndef guarding the header file.
+  """
+  return (('%s_' % h_filename)
+          .upper().replace(os.sep, '_').replace('/', '_').replace('.', '_'))
+
+def GenerateH(fileroot, head, namespace, schema, description):
+  """Generates the .h file containing the definition of the structure specified
+  by the schema.
+
+  Args:
+    fileroot: The filename and path of the file to create, without an extension.
+    head: The string to output as the header of the .h file.
+    namespace: A string corresponding to the C++ namespace to use.
+    schema: A dict containing the schema. See comment at the top of this file.
+    description: A dict containing the description. See comment at the top of
+        this file.
+  """
+
+  h_filename = fileroot + '.h'
+  with open(h_filename, 'w') as f:
+    f.write(head)
+    header_guard = GenerateHeaderGuard(h_filename)

[not at google - send to devlin, 2012/11/12 18:38:13]

You might find the Code class in tools/json_schema_compiler/code.py useful. It's
line-based, does indentation, 80 character limiting for comments (etc), and it's
composable.

[beaudoin, 2012/11/13 18:44:26]

Yeah, as noted elsewhere, I thought about using it but my use case what much
simpler (no >80 comments, no >2 space indents) so I went with the simple
'\n'.join() if you really thing there is much to be gained by going to Code it
can be done, but will require moving the code.py class to the tools/ directory
(like json_comment_eater.py).

[not at google - send to devlin, 2012/11/13 20:28:07]

See previous comment. Why does it require that?

+    f.write('#ifndef %s\n' % header_guard)
+    f.write('#define %s\n' % header_guard)
+    f.write('\n')
+
+    try:
+      for header in schema['headers']:
+        f.write('#include "%s"\n' % header)
+    except KeyError:
+      pass
+    f.write('\n')
+
+    if namespace:
+      f.write('namespace %s {\n' % namespace)
+      f.write('\n')
+
+    f.write(struct_generator.GenerateStruct(
+      schema['type_name'], schema['schema']))
+    f.write('\n')
+
+    try:
+      for var_name, value in description['int_variables'].items():
+        f.write('extern const int %s;\n' % var_name)
+      f.write('\n')
+    except KeyError:
+      pass
+
+    for element_name, element in description['elements'].items():
+      f.write('extern const %s %s;\n' % (schema['type_name'], element_name))
+
+    if namespace:
+      f.write('\n')
+      f.write('}  // namespace %s\n' % namespace)
+
+    f.write('\n')
+    f.write( '#endif  // %s\n' % header_guard)
+
+def GenerateCC(fileroot, head, namespace, schema, description):
+  """Generates the .cc file containing the static initializers for the
+  of the elements specified in the description.
+
+  Args:
+    fileroot: The filename and path of the file to create, without an extension.
+    head: The string to output as the header of the .cc file.
+    namespace: A string corresponding to the C++ namespace to use.
+    schema: A dict containing the schema. See comment at the top of this file.
+    description: A dict containing the description. See comment at the top of
+        this file.
+  """
+  with open(fileroot + '.cc', 'w') as f:
+    f.write(head)
+
+    f.write('#include <stdio.h>\n')
+    f.write('\n')
+    f.write('#include "%s"\n' % (fileroot + '.h'))
+    f.write('\n')
+
+    if namespace:
+      f.write('namespace %s {\n' % namespace)
+      f.write('\n')
+
+    f.write(element_generator.GenerateElements(schema['type_name'],
+        schema['schema'], description))
+
+    if namespace:
+      f.write('\n')
+      f.write('}  // namespace %s\n' % namespace)
+
+def Load(filename):
+  """Loads a JSON file int a Python object and return this object.
+  """
+  with open(filename, 'r') as handle:
+    result = json.loads(json_comment_eater.Nom(handle.read()))
+  return result
+
+if __name__ == '__main__':
+  parser = optparse.OptionParser(
+      description='Generates an C++ array of struct from a JSON description.',
+      usage='usage: %prog [option] -s schema description')
+  parser.add_option('-d', '--destdir',
+      help='root directory to output generated files.')
+  parser.add_option('-n', '--namespace',
+      help='C++ namespace for generated files. e.g search_providers.')
+  parser.add_option('-s', '--schema', help='path to the schema file, '
+      'mandatory.')
+  (opts, args) = parser.parse_args()
+
+  if not args:
+    sys.exit(0) # This is OK as a no-op

[not at google - send to devlin, 2012/11/12 18:38:13]

why? when can this happen?

[beaudoin, 2012/11/13 18:44:26]

Good question. Cut-and-pasted from the json_schema_compiler. :)

Got rid of it...

Done.

+  if not opts.schema:
+    parser.error('You must specify a --schema.')
+
+  description_filename = os.path.normpath(args[0])
+  root, ext = os.path.splitext(description_filename)
+  shortroot = os.path.split(root)[1]
+  if opts.destdir:
+    output_root = os.path.join(os.path.normpath(opts.destdir), shortroot)
+  else:
+    output_root = shortroot
+
+  schema = Load(opts.schema)
+  description = Load(description_filename)
+
+  head = HEAD % (datetime.now().year, opts.schema, description_filename)
+  GenerateH(output_root, head, opts.namespace, schema, description)

[not at google - send to devlin, 2012/11/12 18:38:13]

Passing head into these methods seems unnecessary, they could call a function
like GenerateCopyright themselves.

[beaudoin, 2012/11/13 18:44:26]

That GenerateCopyright function would need the opts.schema and
description_filename which would need to be passed down GenerateH and
GenerateCC. I went with the approach of passing head instead. Let me know if you
really prefer the other way.

[not at google - send to devlin, 2012/11/13 20:28:07]

My apologies, I didn't see that.

Different comment, and I alluded to this in another file: rather than threading
values through static function calls it's nice to bundle those up in a class and
call simple methods on them. E.g.

(_Generator(output_root, opts.namespace, schema, description)
    .GenerateH()
    .GenerateCC())

That would let you define GenerateCopyright on that object, etc.

+  GenerateCC(output_root, head, opts.namespace, schema, description)