Add calculation of derived features for distillable page model

Add calculation of derived features for distillable page model

We've trained a simple model for detecting non-mobile distillable pages
based on a set of features derived from a webpage. This change adds the
main part of deriving those features (a bunch of values are calculated
based on some core features of the web page). It does not add the
extraction of those core features from a page.

A test is added that uses json-formatted core features and then
json-formatted output of the training pipelines derived feature
calculation. It then ensures that the calculation here matches that in
the training pipeline.

BUG=471439
Committed: https://crrev.com/54caa7821b5434ed5c410fe72161e5aa0302366f
Cr-Commit-Position: refs/heads/master@{#323554}

[cjhopman, 2015-03-28 02:19:06]

cjhopman@chromium.org changed reviewers:
+ nyquist@chromium.org

[cjhopman, 2015-03-28 02:19:07]

nyquist: *

[nyquist, 2015-04-01 01:19:14]

lgtm. I could only find one discrepancy between the C++ and the Python code.

https://codereview.chromium.org/1042053003/diff/40001/components/dom_distille...
File components/dom_distiller/core/page_features.cc (right):

https://codereview.chromium.org/1042053003/diff/40001/components/dom_distille...
components/dom_distiller/core/page_features.cc:76:
features.push_back(Contains(".phpbb", path));
The contains-statement here starts with '.', but the python does not.

https://codereview.chromium.org/1042053003/diff/40001/components/dom_distille...
File components/dom_distiller/core/page_features_unittest.cc (right):

https://codereview.chromium.org/1042053003/diff/40001/components/dom_distille...
components/dom_distiller/core/page_features_unittest.cc:77: if
(!derived_features->GetDouble(j * 2 + 1, &expected_value)) {
Nit: int value_index = j * 2 + 1;
Use here and below in assert?

https://codereview.chromium.org/1042053003/diff/40001/components/dom_distille...
components/dom_distiller/core/page_features_unittest.cc:82: std::string
entry_url;
Nit: Could this and filling it with the url string be moved outside of the inner
for-loop?

[cjhopman, 2015-04-01 20:12:14]

https://codereview.chromium.org/1042053003/diff/40001/components/dom_distille...
File components/dom_distiller/core/page_features.cc (right):

https://codereview.chromium.org/1042053003/diff/40001/components/dom_distille...
components/dom_distiller/core/page_features.cc:76:
features.push_back(Contains(".phpbb", path));
On 2015/04/01 01:19:14, nyquist wrote:
> The contains-statement here starts with '.', but the python does not.

Done.

https://codereview.chromium.org/1042053003/diff/40001/components/dom_distille...
File components/dom_distiller/core/page_features_unittest.cc (right):

https://codereview.chromium.org/1042053003/diff/40001/components/dom_distille...
components/dom_distiller/core/page_features_unittest.cc:77: if
(!derived_features->GetDouble(j * 2 + 1, &expected_value)) {
On 2015/04/01 01:19:14, nyquist wrote:
> Nit: int value_index = j * 2 + 1;
> Use here and below in assert?

Done.

https://codereview.chromium.org/1042053003/diff/40001/components/dom_distille...
components/dom_distiller/core/page_features_unittest.cc:82: std::string
entry_url;
On 2015/04/01 01:19:14, nyquist wrote:
> Nit: Could this and filling it with the url string be moved outside of the
inner
> for-loop?

Done.

[cjhopman, 2015-04-01 20:12:30]

The CQ bit was checked by cjhopman@chromium.org to run a CQ dry run

[cjhopman, 2015-04-01 20:12:31]

The patchset sent to the CQ was uploaded after l-g-t-m from nyquist@chromium.org
Link to the patchset: https://codereview.chromium.org/1042053003/#ps80001
(title: "Add OWNERS in components/test/data/dom_distiller")

[commit-bot: I haz the power, 2015-04-01 20:12:54]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1042053003/80001

[cjhopman, 2015-04-01 20:13:05]

cjhopman@chromium.org changed reviewers:
+ jam@chromium.org

[cjhopman, 2015-04-01 20:13:05]

jam@: for OWNERS of third_party/re2 (added to components/dom_distiller/DEPS)

[commit-bot: I haz the power, 2015-04-01 20:22:57]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-01 20:22:58]

Try jobs failed on following builders:
  chromium_presubmit on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

[cjhopman, 2015-04-01 22:42:54]

The CQ bit was checked by cjhopman@chromium.org to run a CQ dry run

[cjhopman, 2015-04-01 22:42:55]

The patchset sent to the CQ was uploaded after l-g-t-m from nyquist@chromium.org
Link to the patchset: https://codereview.chromium.org/1042053003/#ps100001
(title: "initialize vars")

[commit-bot: I haz the power, 2015-04-01 22:49:06]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1042053003/100001

[commit-bot: I haz the power, 2015-04-01 23:01:06]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-01 23:01:08]

Try jobs failed on following builders:
  chromium_presubmit on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

[jam, 2015-04-02 16:29:09]

On 2015/04/01 20:13:05, cjhopman wrote:
> jam@: for OWNERS of third_party/re2 (added to components/dom_distiller/DEPS)

rubberstamp lgtm

[cjhopman, 2015-04-02 16:30:09]

The CQ bit was checked by cjhopman@chromium.org

[commit-bot: I haz the power, 2015-04-02 16:30:56]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1042053003/100001

[commit-bot: I haz the power, 2015-04-02 16:42:35]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-04-02 16:42:36]

Try jobs failed on following builders:
  chromium_presubmit on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

[cjhopman, 2015-04-02 16:59:16]

The CQ bit was checked by cjhopman@chromium.org

[cjhopman, 2015-04-02 16:59:17]

The patchset sent to the CQ was uploaded after l-g-t-m from
nyquist@chromium.org, jam@chromium.org
Link to the patchset: https://codereview.chromium.org/1042053003/#ps120001
(title: "rebase")

[commit-bot: I haz the power, 2015-04-02 16:59:43]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1042053003/120001

[commit-bot: I haz the power, 2015-04-02 20:27:09]

Committed patchset #7 (id:120001)

[commit-bot: I haz the power, 2015-04-03 20:28:42]

Patchset 7 (id:??) landed as
https://crrev.com/54caa7821b5434ed5c410fe72161e5aa0302366f
Cr-Commit-Position: refs/heads/master@{#323554}