Initial version of a serialization framework

pkg/serialization/lib/serialization.dart

+// Copyright (c) 2012, the Dart project authors.  Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+/**
+ * This provides a general-purpose serialization facility for Dart objects. A

[Jennifer Messerly, 2012/11/15 08:02:14]

btw, this comment is lovely. So much content!

[Alan Knight, 2012/11/15 20:51:03]

Thanks.

+ * [Serialization] is defined in terms of [SerializationRule]s. A simple
+ * example of usage is
+ *      var address = new Address();

[Jennifer Messerly, 2012/11/15 08:02:14]

general formatting: I think this might be easier to read with a newline before
and after each code block.

[Alan Knight, 2012/11/15 20:51:03]

Done.

+ *      address.street = 'N 34th';
+ *      address.city = 'Seattle';
+ *      var serialization = new Serialization()
+ *          ..addRuleFor(address);
+ *      String output = serialization.write(address);
+ * This creates a new serialization and adds a rule for address objects. Right
+ * now it has to be passed an address instance because we can't write Address
+ * as a literal. Then we ask the Serialization to write the address and we get
+ * back a String which is a JSON representation of the state of it and related

[Jennifer Messerly, 2012/11/15 08:02:14]

this could link to JSON, e.g. [JSON]

and add an import below:

import 'dart:json' show JSON; // for doc comment

[Alan Knight, 2012/11/15 20:51:03]

I'm not quite sure what you mean by the second part. The user doesn't actually
have to import JSON in order to use this, they just get back a string that they
can give back to a read() method.

[Jennifer Messerly, 2012/11/20 01:23:31]

Oh I meant that you need to add "import 'dart:json' show JSON;" below in this
very file. Otherwise DartDoc doesn't know what [JSON] is and it will render as
plain text.

[Alan Knight, 2012/11/20 12:17:54]

Hmm, even when I do that, it still seems to show as plain text.

+ * objects.
+ *
+ * The version above used reflection to automatically identify the public
+ * fields of the address object. We can also specify those fields explicitly.
+ *      var serialization = new Serialization()
+ *        ..addRuleFor(address,
+ *            constructor: "create",
+ *            constructorFields: ["number", "street"],
+ *            fields: ["city"]);
+ * This rule still uses reflection to access the fields, but not to calculate
+ * them. We can also allow it to calculate the fields, but tell it to ignore
+ * some fields that we don't want used.
+ *      var serialization = new Serialization()
+ *        ..addRuleFor(address,
+ *            constructor: "",
+ *            excludeFields: ["other", "stuff"]);
+ *
+ * We can also use a completely non-reflective rule to serialize and
+ * de-serialize objects.
+ *      addressToMap(a) => {"number" : a.number, "street" : a.street,
+ *          "city" : a.city};
+ *      createAddress(Map m) => new Address.create(m["number"], m["street"]);
+ *      fillInAddress(Map m, Address a) => a.city = m["city"];
+ *      var serialization = new Serialization()
+ *        ..addRule(
+ *            new ClosureToMapRule(anAddress.runtimeType,
+ *                addressToMap, createAddress, fillInAddress);
+ * Note that there are three different functions provided. The first one
+ * takes the fields we want serialized from the Address and puts them into a
+ * map. The second one creates a new address using a map like the one returned
+ * by the first function. And the third one fills in any remaining state in the
+ * created object. At the moment, however, this is more likely to cause problems

[Jennifer Messerly, 2012/11/15 08:02:14]

This is an intriguing sentence. It makes me want to know more about the problem
with cycles. Could you elaborate, maybe with another sentence or two?

[Alan Knight, 2012/11/15 20:51:03]

Done.

+ * with cycles.
+ *
+ * It is possible to give constructor fields values that aren't field names. If
+ * any value isn't a String, it will be treated as a constant. This allows you
+ * to provide constant values to a constructor that aren't obtained from fields.
+ *
+ * In some cases a non-constructor field cannot be properly set using field

[Jennifer Messerly, 2012/11/15 08:02:14]

suggestion: remove "properly" ... it is too easy to read as "property", and I
think the wording is clear without it

[Alan Knight, 2012/11/15 20:51:03]

Done.

+ * access or a setter, and you need to call a method. For example, it may not
+ * be possible to set a List field "foo", and you need to call an addFoo()
+ * method for each entry in the list. In these cases, if you are using a
+ * BasicRule for the object you can call the specialTreatmentFor() method.
+ *       s..addRuleFor(fooHolderInstance).specialTreatmentFor("foo",
+ *           (parent, value) => for (var each in value) parent.addFoo(value));
+ *
+ * To read and write objects, we use the read() and write() methods. There are
+ * currently two variations.

[Jennifer Messerly, 2012/11/15 08:02:14]

perhaps just say "There are two variations", then describe them, and wait for
the "we expect to generalize" to introduce the possibility of a third variation.

[Alan Knight, 2012/11/15 20:51:03]

Done. I think I keep wanting to apologize for the state of the output format
handling :-)

+ *       String output = serialization.write(someObject);
+ *       List output = serialization.writeFlat(someObject);
+ * The first uses a representation in which objects are represented as maps
+ * keyed by field name, but in which references between objects have been
+ * converted into Reference objects. This is then encoded as a JSON string.
+ *
+ * The second representation holds all the objects as a List of simple types.
+ * For practical use you may want to convert that to a JSON or other encoded
+ * representation as well.
+ *
+ * Both representations as primarily intended as proofs of concept for different

[Jennifer Messerly, 2012/11/15 08:02:14]

as primarily -> are primarily

[Alan Knight, 2012/11/15 20:51:03]

Done.

+ * types of representation, and we expect to generalize that to a pluggable
+ * mechanism for different representations.
+ *
+ * To read objects, the corresponding methods are [read] and [readFlat].

[Jennifer Messerly, 2012/11/15 08:02:14]

I think it would be helpful to use the markdown Headers to create a section for
"reading" and "writing", and perhaps one at the start for "setup". Something
like

<intro paragraph>

Setup
===

<discuss APIs for constructing serializer>

Reading
=====

<discuss APIs for reading>


Writing
=====

<discuss APIs for reading>

that way someone that cares about reading or writing can visually jump to the
right place.


(I can imagine the Setup section eventually splitting into "Mirrors" "Manual" if
it got big enough)

[Alan Knight, 2012/11/15 20:51:03]

Done.

+ *       List input = serialization.read(aString);
+ *       List input = serialization.readFlat(aList);
+ * There is also a convenience method for the case of reading a single object.
+ *       Object result = serialization.readOne(aString);
+ *       Object result = serialization.readOneFlat(aString);
+ *
+ * When reading, the serialization instance doing the reading must be configured
+ * with compatible rules to the one doing the writing. It's possible for the
+ * rules to be different, but they need to be able to read the same
+ * representation. For most practical purposes right now they should be the
+ * same. The simplest way to achieve this is by having the serialization
+ * variable [selfDescribing] be true. In that case the rules themselves are also
+ * stored along with the serialized data, and can be read back on the receiving
+ * end. Note that this does not yet work for ClosureToMapRule. The
+ * [selfDescribing] variable is true by default.
+ *
+ * When reading, some object references should not be serialized, but should be
+ * connected up to other instances on the receiving side. A notable example of
+ * this is when serialization rules have been stored. Instances of BasicRule
+ * take a ClassMirror in their constructor, and we cannot serialize those. So
+ * when we read the rules, we must provide a Map<String, Object> which maps from
+ * the simple name of classes we are interested in to a ClassMirror. This can

[Jennifer Messerly, 2012/11/15 08:02:14]

would be nice to link to classmirror, if possible

[Alan Knight, 2012/11/15 20:51:03]

Done.

+ * be provided either in the [externalObjects] variable of the Serialization,
+ * or as an additional parameter to the reading methods.
+ *     new Serialization()
+ *       ..addRuleFor(new Person(), constructorFields: ["name"])
+ *       ..externalObjects['Person'] = reflect(new Person()).type;
+ */
+

[Jennifer Messerly, 2012/11/15 08:02:14]

looks like extra newline here

[Alan Knight, 2012/11/15 20:51:03]

Done.

+library serialization;
+
+import 'src/mirrors_helpers.dart';
+import 'src/serialization_helpers.dart';
+import 'src/polyfill_identity_set.dart';
+import 'dart:json';
+
+part 'src/reader_writer.dart';
+part 'src/serialization_rule.dart';
+part 'src/basic_rule.dart';
+
+/**
+ * This class defines a particular serialization scheme, in terms of
+ * [SerializationRule] instances, and supports reading and writing them.
+ * See library comment for examples of usage.
+ */
+class Serialization {

[Jennifer Messerly, 2012/11/15 08:02:14]

random idea, could we call this Serializer?

[Alan Knight, 2012/11/15 20:51:03]

I really prefer calling things nouns rather than verbs if I can. I really didn't
have a better name for Reader and Writer, but I like Serialization much better
than Serializer. Also, it doesn't really do the work, it's more of a
configuration object. And grammatically - you define a serialization, and if you
want to read objects differently on the client side then you need a different
serialization on each side.

+
+  /**
+   * The serialization is controlled by the list of Serialization rules. These
+   * are most commonly added via [addRuleFor].
+   */
+  List rules = [];
+
+  /**
+   * When reading, we may need to resolve references to existing objects in

[Jennifer Messerly, 2012/11/15 08:02:14]

The part about class mirrors is not totally clear to me. why can't we construct
class mirrors? What kind of object is the value in that case? an example would
help

[Alan Knight, 2012/11/15 20:51:03]

We can't construct them because they're an abstract class with no constructor.
The only way I know of to get a ClassMirror that actually works right now is to
have an instance and do reflect(anInstance).type.  In theory,
reflect(anInstance.runtimeType) or reflect(Object) should work, but right now
the first one gives you an InstanceMirror on Type, and the second one doesn't
work. But rewrote this to give a more meaningful domain example.

+   * the system. Notably, when reading rule descriptions in a self-describing
+   * format we can't construct class mirrors, so we rely on the external objects
+   * giving us the ones we need. But this can be used for any object.
+   */
+  Map<String, dynamic> externalObjects = new Map();

[Jennifer Messerly, 2012/11/15 08:02:14]

if you use "{}" instead of new Map, I think the key type will be constrained to
string and checked in checked mode

[Alan Knight, 2012/11/15 20:51:03]

Done.

+
+  /**
+   * When we write out data using this serialization, should we also write
+   * out a description of the rules.
+   */
+  bool selfDescribing = true;
+
+  /**
+   * Creates a new serialization with a default set of rules for primitives
+   * and lists.
+   */
+  Serialization() {
+    _addDefaultRules();
+  }
+
+  /**
+   * Creates a new serialization with no default rules at all. The most common
+   * use for this is if we are reading self-describing serialized data and
+   * will populate the rules from that data.
+   */
+  Serialization.noDefaultRules() { }

[Jennifer Messerly, 2012/11/15 08:02:14]

Serialization.blank?

[Alan Knight, 2012/11/15 20:51:03]

Yes, much better. Done.

+
+  /**
+   * Create a [BasicRule] rule for the type of
+   * [someInstanceThatWeHaveToPassInBecauseWeCantUseLiteralTypes]. Optionally
+   * allows specifying a [constructor] name, the list of [constructorFields],
+   * and the list of [fields] not used in the constructor. Returns the new
+   * rule.
+   *
+   * If the optional parameters aren't specified, the default constructor will
+   * be used, and the list of fields will be computed. Alternatively, you can
+   * omit [fields] and provide [excludeFields], which will then compute the
+   * list of fields specifically excluding those listed.
+   *
+   * The fields can be actual public fields, but can also be getter/setter
+   * pairs or getters whose value is provided in the constructor. For the
+   * [constructorFields] they can also be arbitrary objects. Anything that is
+   * not a String will be treated as a constant value to be used in any
+   * construction of these objects.
+   *
+   * If the list of fields is computed, fields from the superclass will be
+   * included. However, each subclass needs its own rule, since the constructors
+   * are not inherited, and so may need to be specified separately for each
+   * subclass.
+   */
+  // TODO(alanknight): Take a type rather than an instance. Issue 6282.
+  BasicRule addRuleFor(
+      someInstanceThatWeHaveToPassInBecauseWeCantUseLiteralTypes,

[Jennifer Messerly, 2012/11/15 08:02:14]

haha :)

as funny as this is, I would give it a normal param name, like "instanceForType"
so it appears sane in user facing docs

[Alan Knight, 2012/11/15 20:51:03]

Awww. Done.

+      {String constructor,
+        List constructorFields,
+        List<String> fields,
+        List<String> excludeFields}) {
+
+    var rule;
+    rule = new BasicRule(
+        turnInstanceIntoSomethingWeCanUse(
+            someInstanceThatWeHaveToPassInBecauseWeCantUseLiteralTypes),
+        constructor, constructorFields, fields, excludeFields);
+    addRule(rule);
+    return rule;
+  }
+
+  /** By default we have rules for lists and primitives pre-populated. */
+  void _addDefaultRules() {
+    addRule(new PrimitiveRule());
+    addRule(new ListRule());
+    // Both these rules apply to lists, so unless otherwise indicated,
+    // it will always find the first one.
+    addRule(new ListRuleEssential());
+  }
+
+  /**
+   * Add a new SerializationRule [rule]. The addRuleFor method will probably
+   * handle most simple cases, but for adding an arbitrary rule, including
+   * a SerializationRule subclass which you have created, you can use this
+   * method.
+   */
+  void addRule(SerializationRule rule) {
+    rule.number = rules.length;
+    rules.add(rule);
+  }
+
+  /**
+   * This is the basic method to write out an object graph rooted at
+   * [object] and return the result. Right now this is hard-coded to return
+   * a String from a custom JSON format, but that is likely to change to be
+   * more pluggable in the near future.
+   */
+  String write(Object object) {
+    return newWriter().write(object);
+  }
+
+  /**
+   * Return a new [Writer] object for this serialization. This is useful if you
+   * want to do something more complex with the writer than just returning
+   * the final result.
+   */
+  Writer newWriter() => new Writer(this);
+
+  /**
+   * Write out the tree in a custom flat format, returning a list containing
+   * only "simple" types: num, String, and bool.
+   */
+  List writeFlat(Object object) {
+    return newWriter().writeFlat(object);
+  }
+
+  /**
+   * Read the serialized data from [input] and return a List of the root
+   * objects from the result. If there are objects that need to be resolved
+   * in the current context, they should be provided in [externals] as a
+   * Map from names to values. In particular, in the current implementation
+   * any class mirrors needed should be provided in [externals] using the
+   * class name as a key. In addition to the [externals] map provided here,
+   * values will be looked up in the [externalObjects] map.
+   */
+  List read(String input, [Map externals = const {}]) {
+    return newReader().read(input, externals);
+  }
+
+  /**
+   * In the most common case there is only a single root object to be read,
+   * and this method can be used to return just one object rather than
+   * a List. The [input] and [externals] parameters are the same as for the
+   * general [read] method.
+   */
+  Object readOne(String input, [Map externals = const {}]) {
+    return newReader().readOne(input, externals);
+  }
+
+  /**
+   * Return a new [Reader] object for this serialization. This is useful if
+   * you want to do something more complex with the reader than just returning
+   * the final result.
+   */
+  Reader newReader() => new Reader(this);
+
+  /**
+   * Return the list of SerializationRule that apply to [object]. For
+   * internal use, but public because it's used in testing.
+   */
+  List<SerializationRule> rulesFor(object) {
+    // This has a couple of edge cases.
+    // 1) The owning object may have indicated we should use a different
+    // rule than the default.
+    // 2) We may not have a rule, in which case we lazily create a BasicRule.
+    // 3) Rules are allowed to say mustBePrimary, meaning that they can be used
+    // iff no other rule was chosen first.
+    // TODO(alanknight): Can the mustBePrimary mechanism be removed or changed.
+    // It adds an order dependency to the rules, and is messy. Reconsider in the
+    // light of a more general mechanism for multiple rules per object.
+    // TODO(alanknight): Finding which rules apply seems likely to be a
+    // bottleneck, particularly with the current reflective implementation.
+    // Consider how to improve it. e.g. cache the list of rules by class. But
+    // be careful of issues like rules which have arbitrary predicates. Or
+    // consider having the arbitrary predicates be secondary to an initial

[Jennifer Messerly, 2012/11/15 08:02:14]

this seems reasonable to me.

[Alan Knight, 2012/11/15 20:51:03]

OK

+    // class-based lookup mechanism.
+    var target, candidateRules;
+    if (object is DesignatedRuleForObject) {
+      target = object.target;
+      candidateRules = object.possibleRules(rules);
+    } else {
+      target = object;
+      candidateRules = rules;
+    }
+    List applicable = candidateRules.filter((each) => each.appliesTo(target));
+
+    if (applicable.isEmpty) {
+      var newRule = addRuleFor(target);
+      return [newRule];

[Jennifer Messerly, 2012/11/15 08:02:14]

return [addRuleFor(target)]; ?

[Alan Knight, 2012/11/15 20:51:03]

Done. I often find myself pulling stuff apart into multiple lines like that so
the debugger will let me see it better.

+    }
+
+    if (applicable.length == 1) return applicable;
+    var finalRules = applicable.filter(
+        (x) => !(x != applicable[0] && x.mustBePrimary));

[Jennifer Messerly, 2012/11/15 08:02:14]

probably clearer as

!x.mustBePrimary || x == applicable[0]

out of curiosity, why is the first element special?

[Alan Knight, 2012/11/15 20:51:03]

If something has mustBePrimary, then it has to be the first rule. Otherwise it's
not primary. That whole business is quite ugly, as it forces an order dependency
in the rules, and I'd like to get rid of it. Right now it exists because of the
ListRule/EssentialListRule distinction, where both apply to Lists, but only one
should be used.

Done.

+
+    if (finalRules.isEmpty) throw new SerializationException(
+        'No valid rule found for object $object');
+    return finalRules;
+  }
+
+  /**
+   * Create a Serialization for serializing SerializationRules. This is used
+   * to save the rules in a self-describing format along with the data.
+   * If there are new rule classes created, they will need to be described
+   * here.
+   */
+  Serialization _ruleSerialization() {
+    // TODO(alanknight): There's an extensibility issue here with new rules.
+    // TODO(alanknight): How to handle rules with closures? They have to
+    // exist on the other side, but we might be able to hook them up by name,
+    // or we might just be able to validate that they're correctly set up
+    // on the other side.
+
+    // Make some bogus rule instances so we have something to feed rule creation
+    // and get their types. If only we had class literals implemented...
+    var closureRule = new ClosureToMapRule.stub([].runtimeType);
+    var basicRule = new BasicRule(reflect(null).type, '', [], [], []);
+
+    var meta = new Serialization()
+      ..selfDescribing = false
+      ..addRuleFor(new ListRule())
+      ..addRuleFor(new PrimitiveRule())
+      ..addRuleFor(new ListRuleEssential())
+      ..addRuleFor(basicRule,
+          constructorFields: ['typeWrapped',
+            'constructorName',
+            'constructorFields', 'regularFields', []],
+          fields: [])
+      ..addRule(new ClassMirrorRule());
+    meta.externalObjects = externalObjects;
+    return meta;
+  }
+}
+
+/**
+ * An exception class for errors during serialization.
+ */
+class SerializationException implements Exception {
+  final String message;
+  const SerializationException([this.message]);
+}