Defining the test expectations object

tools/telemetry/telemetry/page/page_runner.py

Index: tools/telemetry/telemetry/page/page_runner.py
diff --git a/tools/telemetry/telemetry/page/page_runner.py b/tools/telemetry/telemetry/page/page_runner.py
index 9ec2d348dc654ad955b6ad52f8ae083897ba6601..009db6830b3af6df413f04bf949771d6221004c9 100644
--- a/tools/telemetry/telemetry/page/page_runner.py
+++ b/tools/telemetry/telemetry/page/page_runner.py
@@ -230,8 +230,12 @@ def Run(test, page_set, options):
           if options.profiler_dir:
             state.StartProfiling(page, options)
 
+          expectation = page_set.expectations.GetExpectationForPage(

[dtu, 2013/07/16 09:50:49]

If expectations are tied to the Test instead of the PageSet (see my comment in
PageSet), then you can't get page_set.expectations. Better to pass it in from
Run(test, page_set, expectations, options)

+              state.browser.platform, page)
+
           try:
-            _RunPage(test, page, state.tab, results_for_current_run, options)
+            _RunPage(test, page, state.tab, results_for_current_run, options,
+                expectation)
             _CheckThermalThrottling(state.browser.platform)
           except exceptions.TabCrashException:
             stdout = ''
@@ -345,8 +349,10 @@ def _CheckArchives(page_set, pages, results):
           pages_missing_archive_path + pages_missing_archive_data]
 
 
-def _RunPage(test, page, tab, results, options):
-  if not test.CanRunForPage(page):
+def _RunPage(test, page, tab, results, options, expectation):
+  if (expectation == "skip" or
+      expectation == "fail" or

[bajones, 2013/07/15 23:37:17]

In this revision of the code "fail" is being treated as a skip for simplicity.
This is not correct. "fail" should indicate that the test is run as usual but
that failures are ignored/overridden, and if the test completes successfully we
should probably raise a flag about it.

It's not clear to me what the best way to handle this is, and advice on the
matter is appreciated!

[Ken Russell (switch to Gerrit), 2013/07/15 23:52:59]

In the "except page_test.Failure" clause below, can the result be added as a
success if the expectation is "fail", and otherwise, be added as a failure? That
would match how the Blink layout test harness reports its results.

[dtu, 2013/07/16 09:50:49]

Yep, that's the way to do it.

+      not test.CanRunForPage(page)):
     logging.warning('Skipping test: it cannot run for %s', page.url)
     results.AddSkip(page, 'Test cannot run')
     return