change status files to make it possible to run test.py -c frog,dart2js -r none.

change status files to make it possible to run test.py -c frog,dart2js -r none.

Except for standalone, this mostly skips all other tests. The configurations
frog_none and dart2js_none these have been ignored for now by the bots, enabling
those tests should be done as needed.

Committed: https://code.google.com/p/dart/source/detail?r=6382

[Siggi Cherem (dart-lang), 2012-04-10 18:14:53]

After this change, you should be able to write:

test.py -c frog,dart2js -r d8,none,drt

and not get errors. Most tests will be skipped when $runtime == none, but we can
later change that.

Peter - could you help us follow up with the status files in co19?

[ahe, 2012-04-10 18:20:34]

I'm not sure I understand the purpose of this CL.

https://chromiumcodereview.appspot.com/10042001/diff/1/client/tests/client/cl...
File client/tests/client/client-leg.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/1/client/tests/client/cl...
client/tests/client/client-leg.status:29: [ $compiler == dart2js && $runtime ==
none ]
We want to test this combination, so the tests should not be skipped.

[Siggi Cherem (dart-lang), 2012-04-10 18:40:17]

On 2012/04/10 18:20:34, ahe wrote:
> I'm not sure I understand the purpose of this CL.
> 

Sorry - here is the full story:

We recently changed the test.dart configurations to have a cross product of
compilers and runtimes.

Before the change something like frog_none and dart2js_none was invalid.
Following your suggestion, we made it valid (I think you mentioned that this
will be useful to test that we can parse code even if we don't generate code for
it with dart2js).

When we made them valid, we didn't update the .status files to indicate what
happens. So, since we switched to the new configurations, many tests are failing
and we can no longer run tests by specifying every possible compiler/runtime
combination in a single call to test.py. Instead we have to run tests in
segments.

The purpose of this change is to be able to run all tests together. The
($compiler == frog || $compiler == dart2js) && $runtime == none configurations
are not being used in our bots, so to start I'm marking them as skipped. We
should later mark them with more precise results, but that takes more time and I
rather we do that in stages.


>
https://chromiumcodereview.appspot.com/10042001/diff/1/client/tests/client/cl...
> File client/tests/client/client-leg.status (right):
> 
>
https://chromiumcodereview.appspot.com/10042001/diff/1/client/tests/client/cl...
> client/tests/client/client-leg.status:29: [ $compiler == dart2js && $runtime
==
> none ]
> We want to test this combination, so the tests should not be skipped.

Yes, but I think we should enable those in a separate CL. Currently our bots are
not running them (they call test.py only with runtime == drt, d8, or a specific
browser). As a result, the status of some of them needs to be changed.

[ahe, 2012-04-10 18:58:28]

Hi Siggi,

Thanks for doing this!

I prefer not skipping tests. For dart2js, could you instead use:

*: Fail, Pass # TODO(ahe): Triage these tests.

Cheers,
Peter


On 2012/04/10 18:40:17, sigmund wrote:
> On 2012/04/10 18:20:34, ahe wrote:
> > I'm not sure I understand the purpose of this CL.
> > 
> 
> Sorry - here is the full story:
> 
> We recently changed the test.dart configurations to have a cross product of
> compilers and runtimes.
> 
> Before the change something like frog_none and dart2js_none was invalid.
> Following your suggestion, we made it valid (I think you mentioned that this
> will be useful to test that we can parse code even if we don't generate code
for
> it with dart2js).
> 
> When we made them valid, we didn't update the .status files to indicate what
> happens. So, since we switched to the new configurations, many tests are
failing
> and we can no longer run tests by specifying every possible compiler/runtime
> combination in a single call to test.py. Instead we have to run tests in
> segments.
> 
> The purpose of this change is to be able to run all tests together. The
> ($compiler == frog || $compiler == dart2js) && $runtime == none configurations
> are not being used in our bots, so to start I'm marking them as skipped. We
> should later mark them with more precise results, but that takes more time and
I
> rather we do that in stages.
> 
> 
> >
>
https://chromiumcodereview.appspot.com/10042001/diff/1/client/tests/client/cl...
> > File client/tests/client/client-leg.status (right):
> > 
> >
>
https://chromiumcodereview.appspot.com/10042001/diff/1/client/tests/client/cl...
> > client/tests/client/client-leg.status:29: [ $compiler == dart2js && $runtime
> ==
> > none ]
> > We want to test this combination, so the tests should not be skipped.
> 
> Yes, but I think we should enable those in a separate CL. Currently our bots
are
> not running them (they call test.py only with runtime == drt, d8, or a
specific
> browser). As a result, the status of some of them needs to be changed.

[Siggi Cherem (dart-lang), 2012-04-10 19:09:14]

On 2012/04/10 18:58:28, ahe wrote:
> Hi Siggi,
> 
> Thanks for doing this!
> 
> I prefer not skipping tests. For dart2js, could you instead use:
> 
> *: Fail, Pass # TODO(ahe): Triage these tests.
> 
No problem. Updated the status files to indicate this for dart2js.

> Cheers,
> Peter
> 
> 
> On 2012/04/10 18:40:17, sigmund wrote:
> > On 2012/04/10 18:20:34, ahe wrote:
> > > I'm not sure I understand the purpose of this CL.
> > > 
> > 
> > Sorry - here is the full story:
> > 
> > We recently changed the test.dart configurations to have a cross product of
> > compilers and runtimes.
> > 
> > Before the change something like frog_none and dart2js_none was invalid.
> > Following your suggestion, we made it valid (I think you mentioned that this
> > will be useful to test that we can parse code even if we don't generate code
> for
> > it with dart2js).
> > 
> > When we made them valid, we didn't update the .status files to indicate what
> > happens. So, since we switched to the new configurations, many tests are
> failing
> > and we can no longer run tests by specifying every possible compiler/runtime
> > combination in a single call to test.py. Instead we have to run tests in
> > segments.
> > 
> > The purpose of this change is to be able to run all tests together. The
> > ($compiler == frog || $compiler == dart2js) && $runtime == none
configurations
> > are not being used in our bots, so to start I'm marking them as skipped. We
> > should later mark them with more precise results, but that takes more time
and
> I
> > rather we do that in stages.
> > 
> > 
> > >
> >
>
https://chromiumcodereview.appspot.com/10042001/diff/1/client/tests/client/cl...
> > > File client/tests/client/client-leg.status (right):
> > > 
> > >
> >
>
https://chromiumcodereview.appspot.com/10042001/diff/1/client/tests/client/cl...
> > > client/tests/client/client-leg.status:29: [ $compiler == dart2js &&
$runtime
> > ==
> > > none ]
> > > We want to test this combination, so the tests should not be skipped.
> > 
> > Yes, but I think we should enable those in a separate CL. Currently our bots
> are
> > not running them (they call test.py only with runtime == drt, d8, or a
> specific
> > browser). As a result, the status of some of them needs to be changed.

[ahe, 2012-04-10 19:45:31]

https://chromiumcodereview.appspot.com/10042001/diff/6001/client/tests/dartc/...
File client/tests/dartc/dartc.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/client/tests/dartc/...
client/tests/dartc/dartc.status:19: 
No need to modify this file, I think.

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/leg_only...
File frog/tests/leg_only/leg_only.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/leg_only...
frog/tests/leg_only/leg_only.status:16: [ $compiler == frog || $compiler ==
frogsh || $runtime == drt || $runtime == none]
We don't want to skip these tests for dart2js.

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/standalone/st...
File tests/standalone/standalone-leg.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/standalone/st...
tests/standalone/standalone-leg.status:19: FailTest: Pass # fails only at
runtime, overrides default expectation from standalone.status
This will make the test appear to be flaky. So in this case I prefer "Skip" of
the two evils :-(

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/utils/utils.s...
File tests/utils/utils.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/utils/utils.s...
tests/utils/utils.status:32: [ $runtime == none ]
This will skip the tests on dart2js. That's not what we want.

https://chromiumcodereview.appspot.com/10042001/diff/6001/utils/tests/css/css...
File utils/tests/css/css.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/utils/tests/css/css...
utils/tests/css/css.status:11: [ ($compiler == dart2js || $compiler == frog ||
$compiler == frogsh) && $runtime == none ]
We don't want to skip tests on dart2js.

[Siggi Cherem (dart-lang), 2012-04-10 20:01:08]

https://chromiumcodereview.appspot.com/10042001/diff/6001/client/tests/dartc/...
File client/tests/dartc/dartc.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/client/tests/dartc/...
client/tests/dartc/dartc.status:19: 
On 2012/04/10 19:45:31, ahe wrote:
> No need to modify this file, I think.

Done.

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/leg_only...
File frog/tests/leg_only/leg_only.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/leg_only...
frog/tests/leg_only/leg_only.status:16: [ $compiler == frog || $compiler ==
frogsh || $runtime == drt || $runtime == none]
On 2012/04/10 19:45:31, ahe wrote:
> We don't want to skip these tests for dart2js.

Done.

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/standalone/st...
File tests/standalone/standalone-leg.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/standalone/st...
tests/standalone/standalone-leg.status:19: FailTest: Pass # fails only at
runtime, overrides default expectation from standalone.status
On 2012/04/10 19:45:31, ahe wrote:
> This will make the test appear to be flaky. So in this case I prefer "Skip" of
> the two evils :-(

Done.

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/utils/utils.s...
File tests/utils/utils.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/utils/utils.s...
tests/utils/utils.status:32: [ $runtime == none ]
On 2012/04/10 19:45:31, ahe wrote:
> This will skip the tests on dart2js. That's not what we want.

Done.

https://chromiumcodereview.appspot.com/10042001/diff/6001/utils/tests/css/css...
File utils/tests/css/css.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/utils/tests/css/css...
utils/tests/css/css.status:11: [ ($compiler == dart2js || $compiler == frog ||
$compiler == frogsh) && $runtime == none ]
On 2012/04/10 19:45:31, ahe wrote:
> We don't want to skip tests on dart2js.

Done. Seems that frog and dart2js were passing, so I didn't add the triage TODO
here.

[Emily Fortuna, 2012-04-10 20:30:51]

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/await/aw...
File frog/tests/await/await.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/await/aw...
frog/tests/await/await.status:18: [ $runtime == vm || $compiler == dartc ||
$runtime == drt || $compiler == dart2js || $runtime == none ]
You can get rid of $compiler == dartc since if $compiler == dartc, then the
$runtime has to == none.

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/frog/fro...
File frog/tests/frog/frog.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/frog/fro...
frog/tests/frog/frog.status:30: [ $compiler == dart2js || $compiler == dartc ]
Is this change what you want? Seems like we'd be skipping $compiler == dart2js
and $runtime == d8 now

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/corelib/corel...
File tests/corelib/corelib-leg.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/corelib/corel...
tests/corelib/corelib-leg.status:25: *: Fail, Pass # TODO(ahe): Triage these
tests.
(Can these tests with no runtime crash? If so, we should start with skip, and
then change to Pass, Fail or the specific designation after they have been
specifically tested.

https://chromiumcodereview.appspot.com/10042001/diff/6005/utils/tests/css/css...
File utils/tests/css/css.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6005/utils/tests/css/css...
utils/tests/css/css.status:5: [ ($compiler == dart2js || $compiler == frog) &&
($runtime == ie || $runtime == drt || $runtime == chrome || $runtime == ff ||
$runtime == safari) ]
But you probably want to skip if $compiler == frog or frogsh and $runtime ==
none, right? I'm looking at the change here:
https://chromiumcodereview.appspot.com/10042001/diff2/6001:6005/utils/tests/c...

[Siggi Cherem (dart-lang), 2012-04-10 20:47:20]

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/await/aw...
File frog/tests/await/await.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/await/aw...
frog/tests/await/await.status:18: [ $runtime == vm || $compiler == dartc ||
$runtime == drt || $compiler == dart2js || $runtime == none ]
On 2012/04/10 20:30:51, Emily Fortuna wrote:
> You can get rid of $compiler == dartc since if $compiler == dartc, then the
> $runtime has to == none.

Done.

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/frog/fro...
File frog/tests/frog/frog.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/frog/tests/frog/fro...
frog/tests/frog/frog.status:30: [ $compiler == dart2js || $compiler == dartc ]
On 2012/04/10 20:30:51, Emily Fortuna wrote:
> Is this change what you want? Seems like we'd be skipping $compiler == dart2js
> and $runtime == d8 now

Yes - they were actually failing and not being run, all these tests were using
'native' which is very frog-specific.

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/corelib/corel...
File tests/corelib/corelib-leg.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/corelib/corel...
tests/corelib/corelib-leg.status:25: *: Fail, Pass # TODO(ahe): Triage these
tests.
On 2012/04/10 20:30:51, Emily Fortuna wrote:
> (Can these tests with no runtime crash? If so, we should start with skip, and
> then change to Pass, Fail or the specific designation after they have been
> specifically tested.

Seems we can leave it as it is. I ran all these tests and I see no crashes.

[Emily Fortuna, 2012-04-10 20:48:42]

lgtm!

[ahe, 2012-04-10 21:17:04]

LGTM!

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/corelib/corel...
File tests/corelib/corelib-leg.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/6001/tests/corelib/corel...
tests/corelib/corelib-leg.status:25: *: Fail, Pass # TODO(ahe): Triage these
tests.
On 2012/04/10 20:30:51, Emily Fortuna wrote:
> (Can these tests with no runtime crash? If so, we should start with skip, and
> then change to Pass, Fail or the specific designation after they have been
> specifically tested.

The problem is: we don't want to skip tests. This leaves us with two options:

1. Use "Fail, Pass".

2. Add the correct status.

It takes me about 20 minutes to add the correct status, but I think it may take
longer for someone not used to updating co19, so I asked Siggi to use Fail,
Pass. I expect that I'll deal with these TODOs real soon.

https://chromiumcodereview.appspot.com/10042001/diff/8005/frog/tests/await/aw...
File frog/tests/await/await.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/8005/frog/tests/await/aw...
frog/tests/await/await.status:6: *: Skip  # compiler API evolved and this codea
and tests are not maintained.
"codea"

https://chromiumcodereview.appspot.com/10042001/diff/8005/frog/tests/await/aw...
frog/tests/await/await.status:9: #WhileLoopWithBreakTest: Fail    # break within
a loop
Why keep these comments?

[Siggi Cherem (dart-lang), 2012-04-10 21:32:03]

https://chromiumcodereview.appspot.com/10042001/diff/8005/frog/tests/await/aw...
File frog/tests/await/await.status (right):

https://chromiumcodereview.appspot.com/10042001/diff/8005/frog/tests/await/aw...
frog/tests/await/await.status:9: #WhileLoopWithBreakTest: Fail    # break within
a loop
On 2012/04/10 21:17:04, ahe wrote:
> Why keep these comments?

I was using these tests as a way to indicate the status of the implementation in
terms of features, so this was a compact way of saying which parts still had to
be implemented (e.g. which would be relevant again if we fix things up)