Represent tokens as a compressed stream instead of an array.

Represent tokens as a compressed stream instead of an array.

On x86:

Original sizes:
Size of isolate snapshot = 925655
New space (0k of 32768k) Old space (1700k of 1792k) Code space (0k of 0k)
out/ReleaseIA32/run_vm_tests --benchmarks
CorelibCompileAll(RunTime): 80944
CorelibIsolateStartup(RunTime): 5636
UseDartApi(RunTime): 404620
Dart2JSCompileAll(RunTime): 1379683
FrameLookup(RunTime): 45

Size with the new TokenStream implementation:
Size of isolate snapshot = 851352
New space (0k of 32768k) Old space (1368k of 1536k) Code space (0k of 0k)
out/ReleaseIA32/run_vm_tests --benchmarks
CorelibCompileAll(RunTime): 76918
CorelibIsolateStartup(RunTime): 4056
UseDartApi(RunTime): 502111
Dart2JSCompileAll(RunTime): 1251646
FrameLookup(RunTime): 45

(saves about 332k out of the original TokenStream size of 410k)

On x64:

Original sizes:
Size of isolate snapshot = 924865
New space (0k of 32768k) Old space (3234k of 3444k) Code space (0k of 0k)
out/ReleaseX64/run_vm_tests --benchmarks
CorelibCompileAll(RunTime): 109660
CorelibIsolateStartup(RunTime): 3985
UseDartApi(RunTime): 463009
Dart2JSCompileAll(RunTime): 2170309
FrameLookup(RunTime): 35

Size with the new TokenStream implementation:
Size of isolate snapshot = 849675
New space (0k of 32768k) Old space (2436k of 2560k) Code space (0k of 0k)
out/ReleaseX64/run_vm_tests --benchmarks
CorelibCompileAll(RunTime): 98716
CorelibIsolateStartup(RunTime): 2938
UseDartApi(RunTime): 509149
Dart2JSCompileAll(RunTime): 2042273
FrameLookup(RunTime): 39

(saves about 798k out of the original TokenStream size of 810k)

Token stats:
Number of tokens:   101470
  Literal tokens:   4550
    Ident tokens:   33020
Committed: https://code.google.com/p/dart/source/detail?r=9539

[Ivan Posva, 2012-07-02 21:48:32]

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/object.cc
File vm/object.cc (right):

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/object.cc#newco...
vm/object.cc:4578: stream_(tokens.EntryAddr(0), tokens.Length()),
This is not GC safe.

[Ivan Posva, 2012-07-02 22:56:05]

Some more comments.

-Ivan

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/object.cc
File vm/object.cc (right):

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/object.cc#newco...
vm/object.cc:4493: intptr_t FindIdentIndex(String* ident) {
We should also investigate unique indices for literals.

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/snapshot_test.cc
File vm/snapshot_test.cc (right):

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/snapshot_test.c...
vm/snapshot_test.cc:758: TokenStreamIterator expected_iterator(expected_tokens,
0);
I am becoming more and more convinced that this should be TokenStream::Iterator.

[siva, 2012-07-04 01:43:02]

PTAL.

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/object.cc
File vm/object.cc (right):

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/object.cc#newco...
vm/object.cc:4493: intptr_t FindIdentIndex(String* ident) {
On 2012/07/02 22:56:05, Ivan Posva wrote:
> We should also investigate unique indices for literals.

Done.

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/object.cc#newco...
vm/object.cc:4578: stream_(tokens.EntryAddr(0), tokens.Length()),
Removed the stream_ field and instead use EntryAddr(pos) for access to the
stream bytes.

On 2012/07/02 21:48:32, Ivan Posva wrote:
> This is not GC safe.

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/snapshot_test.cc
File vm/snapshot_test.cc (right):

https://chromiumcodereview.appspot.com/10697055/diff/12004/vm/snapshot_test.c...
vm/snapshot_test.cc:758: TokenStreamIterator expected_iterator(expected_tokens,
0);
On 2012/07/02 22:56:05, Ivan Posva wrote:
> I am becoming more and more convinced that this should be
TokenStream::Iterator.

Done.

[Ivan Posva, 2012-07-09 20:41:20]

LGTM -ip