|
|
Created:
5 years, 7 months ago by Petr Hosek Modified:
5 years, 5 months ago Reviewers:
Mark Seaborn CC:
native-client-reviews_googlegroups.com Base URL:
https://chromium.googlesource.com/native_client/pnacl-llvm.git@master Target Ref:
refs/heads/master Visibility:
Public. |
DescriptionUse -mc-relax-all flag for sandboxed translator
This enables LLVM MC to write instruction padding directly into
fragments reducing memory usage of the sandboxed translator.
BUG=https://code.google.com/p/nativeclient/issues/detail?id=4063
TEST=toolchain trybots
R=mseaborn@chromium.org
Committed: https://chromium.googlesource.com/native_client/pnacl-llvm/+/56b07283d8febb0bbb5bf2c96ca03ffd930dfba3
Patch Set 1 #
Total comments: 2
Patch Set 2 : Add a comment explaining the change #Patch Set 3 : Rebase #Messages
Total messages: 12 (1 generated)
phosek@chromium.org changed reviewers: + mseaborn@chromium.org
LGTM once the trybot runs pass. https://chromiumcodereview.appspot.com/1129013004/diff/1/tools/pnacl-llc/pnac... File tools/pnacl-llc/pnacl-llc.cpp (right): https://chromiumcodereview.appspot.com/1129013004/diff/1/tools/pnacl-llc/pnac... tools/pnacl-llc/pnacl-llc.cpp:709: Options.MCOptions.MCRelaxAll = true; Can you add a comment explaining that this reduces memory use (citing the issue perhaps) and why we enable it only for the sandboxed build of pnacl-llc?
https://chromiumcodereview.appspot.com/1129013004/diff/1/tools/pnacl-llc/pnac... File tools/pnacl-llc/pnacl-llc.cpp (right): https://chromiumcodereview.appspot.com/1129013004/diff/1/tools/pnacl-llc/pnac... tools/pnacl-llc/pnacl-llc.cpp:709: Options.MCOptions.MCRelaxAll = true; On 2015/05/15 14:03:42, Mark Seaborn wrote: > Can you add a comment explaining that this reduces memory use (citing the issue > perhaps) and why we enable it only for the sandboxed build of pnacl-llc? Done.
The trybots are failing with validation errors on x86-32 and x86-64. I tried comparing output with and without passing "-mc-relax-all" to llc. On a minimal test case, I noticed that the string table references end up being wrong with "-mc-relax-all". Try this minimal example with -O0: void foo(const char *arg1, const char *arg2, const char *arg3) { } void _start() { foo("hello1", "hello2", "hello3"); } Without "-mc-relax-all", llc outputs this: 43: c7 44 24 08 0e 00 00 movl $0xe,0x8(%esp) 4a: 00 47: R_386_32 .rodata 4b: c7 44 24 04 07 00 00 movl $0x7,0x4(%esp) 52: 00 4f: R_386_32 .rodata 53: c7 04 24 00 00 00 00 movl $0x0,(%esp) 56: R_386_32 .rodata With "-mc-relax-all", llc outputs this: 43: c7 44 24 08 15 00 00 movl $0x15,0x8(%esp) 4a: 00 47: R_386_32 .rodata 4b: c7 44 24 04 15 00 00 movl $0x15,0x4(%esp) 52: 00 4f: R_386_32 .rodata 53: c7 04 24 00 00 00 00 movl $0x0,(%esp) 56: R_386_32 .rodata Notice the use of offset 15 twice, referring to the *same* offset in the .rodata section twice, which can't be right. I don't know whether this explains the validation failure, but it might do if there is a problem with forward references. Any ideas? Cheers, Mark -- You received this message because you are subscribed to the Google Groups "Native-Client-Reviews" group. To unsubscribe from this group and stop receiving emails from it, send an email to native-client-reviews+unsubscribe@googlegroups.com. To post to this group, send email to native-client-reviews@googlegroups.com. Visit this group at http://groups.google.com/group/native-client-reviews. For more options, visit https://groups.google.com/d/optout.
Hi Petr, I was able to reproduce the validation failures by adding "-mc-relax-all" into pnacl-translate.py and building hello_world.nexe. I'm seeing two problems: * Instructions crossing bundle boundaries, e.g. 2019f: f6 c4 02 test $0x2,%ah 201a2: 0f 85 2c 00 00 00 jne 201d4 <Function+0xf4> * CALL instructions whose ends are 16-byte-aligned but not 32-byte-bundle-aligned, e.g. 201b0: 66 66 66 66 66 66 2e data32 data32 data32 data32 data32 nopw %cs:0x0(%eax,%eax,1) 201b7: 0f 1f 84 00 00 00 00 201be: 00 201bf: 66 66 66 2e 0f 1f 84 data32 data32 nopw %cs:0x0(%eax,%eax,1) 201c6: 00 00 00 00 00 201cb: e8 f0 0f 00 00 call 211c0 <Function15> 201d0: 0f b7 47 0c movzwl 0xc(%edi),%eax It looks like the nop-inserting logic might be getting confused about its current offset within a section. The following function starts off OK, but then padding is inserted at unnecessary places: 000200e0 <Function>: 200e0: 57 push %edi 200e1: 56 push %esi 200e2: 83 ec 34 sub $0x34,%esp 200e5: 65 a1 00 00 00 00 mov %gs:0x0,%eax 200eb: 8b b0 78 fb ff ff mov -0x488(%eax),%esi 200f1: b8 38 01 02 10 mov $0x10020138,%eax 200f6: e9 06 00 00 00 jmp 20101 <Function+0x21> 200fb: 0f 1f 44 00 00 nopl 0x0(%eax,%eax,1) 20100: 40 inc %eax 20101: a8 03 test $0x3,%al 20103: 0f 84 17 00 00 00 je 20120 <Function+0x40> 20109: 80 38 00 cmpb $0x0,(%eax) 2010c: 0f 85 ee ff ff ff jne 20100 <Function+0x20> 20112: e9 44 00 00 00 jmp 2015b <Function+0x7b> 20117: 66 0f 1f 84 00 00 00 nopw 0x0(%eax,%eax,1) 2011e: 00 00 20120: 8b 08 mov (%eax),%ecx 20122: 89 ca mov %ecx,%edx 20124: f7 d2 not %edx 20126: 8d b9 ff fe fe fe lea -0x1010101(%ecx),%edi 2012c: 21 d7 and %edx,%edi 2012e: 83 c0 04 add $0x4,%eax 20131: f7 c7 80 80 80 80 test $0x80808080,%edi 20137: 0f 84 e3 ff ff ff je 20120 <Function+0x40> 2013d: 83 c0 fc add $0xfffffffc,%eax 20140: e9 0e 00 00 00 jmp 20153 <Function+0x73> // This nop didn't need to be inserted: 20145: 66 66 2e 0f 1f 84 00 data32 nopw %cs:0x0(%eax,%eax,1) 2014c: 00 00 00 00 20150: 40 inc %eax 20151: 8a 08 mov (%eax),%cl 20153: 84 c9 test %cl,%cl 20155: 0f 85 f5 ff ff ff jne 20150 <Function+0x70> 2015b: b9 38 01 02 10 mov $0x10020138,%ecx // By chance, no nop padding needed before this bundle start: 20160: 29 c8 sub %ecx,%eax 20162: c7 44 24 18 38 01 02 movl $0x10020138,0x18(%esp) 20169: 10 2016a: 89 44 24 1c mov %eax,0x1c(%esp) // This nop didn't need to be inserted: 2016e: 66 90 xchg %ax,%ax 20170: c7 44 24 20 5d 01 02 movl $0x1002015d,0x20(%esp) 20177: 10 20178: c7 44 24 24 01 00 00 movl $0x1,0x24(%esp) 2017f: 00 // By chance, no nop padding needed before this bundle start: 20180: 40 inc %eax 20181: 89 44 24 30 mov %eax,0x30(%esp) 20185: 8d 44 24 18 lea 0x18(%esp),%eax 20189: 89 44 24 28 mov %eax,0x28(%esp) 2018d: 0f 1f 00 nopl (%eax) 20190: c7 44 24 2c 02 00 00 movl $0x2,0x2c(%esp) 20197: 00 20198: 8b 7e 08 mov 0x8(%esi),%edi 2019b: 0f b7 47 0c movzwl 0xc(%edi),%eax // This instruction crosses a bundle boundary: 2019f: f6 c4 02 test $0x2,%ah 201a2: 0f 85 2c 00 00 00 jne 201d4 <Function+0xf4> Cheers, Mark On 15 May 2015 at 23:52, Mark Seaborn <mseaborn@chromium.org> wrote: > The trybots are failing with validation errors on x86-32 and x86-64. > > I tried comparing output with and without passing "-mc-relax-all" to llc. > On a minimal test case, I noticed that the string table references end up > being wrong with "-mc-relax-all". Try this minimal example with -O0: > > void foo(const char *arg1, const char *arg2, const char *arg3) { > } > > void _start() { > foo("hello1", "hello2", "hello3"); > } > > Without "-mc-relax-all", llc outputs this: > > 43: c7 44 24 08 0e 00 00 movl $0xe,0x8(%esp) > 4a: 00 > 47: R_386_32 .rodata > 4b: c7 44 24 04 07 00 00 movl $0x7,0x4(%esp) > 52: 00 > 4f: R_386_32 .rodata > 53: c7 04 24 00 00 00 00 movl $0x0,(%esp) > 56: R_386_32 .rodata > > With "-mc-relax-all", llc outputs this: > > 43: c7 44 24 08 15 00 00 movl $0x15,0x8(%esp) > 4a: 00 > 47: R_386_32 .rodata > 4b: c7 44 24 04 15 00 00 movl $0x15,0x4(%esp) > 52: 00 > 4f: R_386_32 .rodata > 53: c7 04 24 00 00 00 00 movl $0x0,(%esp) > 56: R_386_32 .rodata > > Notice the use of offset 15 twice, referring to the *same* offset in the > .rodata section twice, which can't be right. > > I don't know whether this explains the validation failure, but it might do > if there is a problem with forward references. > > Any ideas? > > Cheers, > Mark > > -- You received this message because you are subscribed to the Google Groups "Native-Client-Reviews" group. To unsubscribe from this group and stop receiving emails from it, send an email to native-client-reviews+unsubscribe@googlegroups.com. To post to this group, send email to native-client-reviews@googlegroups.com. Visit this group at http://groups.google.com/group/native-client-reviews. For more options, visit https://groups.google.com/d/optout.
On 20 May 2015 at 23:02, Mark Seaborn <mseaborn@chromium.org> wrote: > It looks like the nop-inserting logic might be getting confused about its > current offset within a section. > Yes, specifically it's getting confused by ".align 16" directives that pnacl-llc emits before labels. Here is a minimal reproducer: $ cat asm_nacl_test.S // Bundle 1 hlt // 1 byte .align 16 movl $0x11111111, (%esp) // 7 bytes movl $0x11111111, (%esp) // 7 bytes // Bundle 2 movl $0x22222222, 0x11(%esp) // 8 bytes movl $0x22222222, 0x11(%esp) // 8 bytes movl $0x22222222, 0x11(%esp) // 8 bytes movl $0x22222222, (%esp) // 7 bytes // Bundle 3 movl $0x33333333, (%esp) // 7 bytes movl $0x33333333, (%esp) // 7 bytes $ .../native_client/toolchain/linux_x86/pnacl_newlib/bin/llvm-mc -triple i686-nacl-gnu asm_nacl_test.S -filetype=obj -o asm_nacl_test.o -mc-relax-all $ objdump -d asm_nacl_test.o ... 00000000 <.text>: 0: f4 hlt 1: 66 66 66 66 66 66 2e data32 data32 data32 data32 data32 nopw %cs:0x0(%eax,%eax,1) 8: 0f 1f 84 00 00 00 00 f: 00 10: c7 04 24 11 11 11 11 movl $0x11111111,(%esp) 17: c7 04 24 11 11 11 11 movl $0x11111111,(%esp) 1e: c7 44 24 11 22 22 22 movl $0x22222222,0x11(%esp) 25: 22 26: c7 44 24 11 22 22 22 movl $0x22222222,0x11(%esp) 2d: 22 2e: 66 90 xchg %ax,%ax 30: c7 44 24 11 22 22 22 movl $0x22222222,0x11(%esp) 37: 22 38: c7 04 24 22 22 22 22 movl $0x22222222,(%esp) 3f: c7 04 24 33 33 33 33 movl $0x33333333,(%esp) 46: c7 04 24 33 33 33 33 movl $0x33333333,(%esp) We got nop-padding before 0x30, but it should have been before 0x20 and 0x40. Cheers, Mark -- You received this message because you are subscribed to the Google Groups "Native-Client-Reviews" group. To unsubscribe from this group and stop receiving emails from it, send an email to native-client-reviews+unsubscribe@googlegroups.com. To post to this group, send email to native-client-reviews@googlegroups.com. Visit this group at http://groups.google.com/group/native-client-reviews. For more options, visit https://groups.google.com/d/optout.
Petr wrote: > I run the change through pnacl toolchain trybots, but I'm seeing some > strange errors, see https://chromiumcodereview.appspot.com/1158363002/ > It's failing inside gold for some reason. (This is when testing with your proposed fix, http://reviews.llvm.org/D10044, applied.) I notice that all of the nexes that Gold is failing to link are *not* built from ABI-stable pexes: scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/gdb/gdb_test_guest.nexe failed: Error 1 scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/gdb/gdb_test_guest_thread.nexe failed: Error 1 scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/mmap_prot_exec/mmap_prot_exec.nexe failed: Error 1 scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/sysbasic/sysbasic.nexe failed: Error 1 scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/syscalls/sysbrk_test.nexe failed: Error 1 scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/toolchain/frame_addresses.nexe failed: Error 1 scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/toolchain/return_address.nexe failed: Error 1 -- from http://build.chromium.org/p/tryserver.nacl/builders/nacl-toolchain-linux-pnac... The error from Gold is: /mnt/data/b/build/slave/nacl-toolchain/build/native_client/toolchain/linux_x86/pnacl_newlib_raw/bin/le32-nacl-ld.gold: internal error in output_address, at ../../../src/binutils/gold/output.cc:3009 So there is some feature that isn't exposed by ABI-stable pexes that is causing this failure. DWARF debug info seems like a likely candidate. Can you reproduce these failures locally? Do they go away if you remove "-g" from nacl_env in SConstruct? Maybe "-mc-relax-all" is causing some bundle alignment padding to wrongly be added to DWARF debug info? From looking at the new fix (http://reviews.llvm.org/D10044) it seems unlikely that that would cause this linker failure. Perhaps this is caused by the earlier change (http://reviews.llvm.org/D8072)? Does the linker failure happen without the new change, or with the earlier "-mc-relax-all" change reverted too? Cheers, Mark -- You received this message because you are subscribed to the Google Groups "Native-Client-Reviews" group. To unsubscribe from this group and stop receiving emails from it, send an email to native-client-reviews+unsubscribe@googlegroups.com. To post to this group, send email to native-client-reviews@googlegroups.com. Visit this group at http://groups.google.com/group/native-client-reviews. For more options, visit https://groups.google.com/d/optout.
On 28 May 2015 at 12:37, Mark Seaborn <mseaborn@chromium.org> wrote: > Petr wrote: > >> I run the change through pnacl toolchain trybots, but I'm seeing some >> strange errors, see https://chromiumcodereview.appspot.com/1158363002/ >> It's failing inside gold for some reason. > > > (This is when testing with your proposed fix, > http://reviews.llvm.org/D10044, applied.) > > I notice that all of the nexes that Gold is failing to link are *not* > built from ABI-stable pexes: > > scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/gdb/gdb_test_guest.nexe > failed: Error 1 > scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/gdb/gdb_test_guest_thread.nexe > failed: Error 1 > scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/mmap_prot_exec/mmap_prot_exec.nexe > failed: Error 1 > scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/sysbasic/sysbasic.nexe > failed: Error 1 > scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/syscalls/sysbrk_test.nexe > failed: Error 1 > scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/toolchain/frame_addresses.nexe > failed: Error 1 > scons-out/nacl-x86-64-pnacl-pexe-clang/obj/tests/toolchain/return_address.nexe > failed: Error 1 > > -- from > http://build.chromium.org/p/tryserver.nacl/builders/nacl-toolchain-linux-pnac... > > The error from Gold is: > /mnt/data/b/build/slave/nacl-toolchain/build/native_client/toolchain/linux_x86/pnacl_newlib_raw/bin/le32-nacl-ld.gold: > internal error in output_address, at > ../../../src/binutils/gold/output.cc:3009 > > So there is some feature that isn't exposed by ABI-stable pexes that is > causing this failure. > > DWARF debug info seems like a likely candidate. > > Can you reproduce these failures locally? Do they go away if you remove > "-g" from nacl_env in SConstruct? > > Maybe "-mc-relax-all" is causing some bundle alignment padding to wrongly > be added to DWARF debug info? > > From looking at the new fix (http://reviews.llvm.org/D10044) it seems > unlikely that that would cause this linker failure. Perhaps this is caused > by the earlier change (http://reviews.llvm.org/D8072)? Does the linker > failure happen without the new change, or with the earlier "-mc-relax-all" > change reverted too? > I checked: The linker failure still happens when using tip-of-tree pnacl-llvm, without your new change. Removing "-g" from SConstruct won't help because there's still debugging info in the pre-built standard libraries. This minimal test also reproduces the linker failure (with "-mc-relax-all" added in pnacl-translate.py): $ cat minimal.c int _start() { return 123; } $ ./toolchain/linux_x86/pnacl_newlib_raw/bin/pnacl-clang -g minimal.c -o minimal.pexe -nostdlib $ ./toolchain/linux_x86/pnacl_newlib_raw/bin/pnacl-translate -arch x86-32 minimal.pexe -o minimal.nexe --allow-llvm-bitcode-input /home/mseaborn/devel/nacl/native_client/toolchain/linux_x86/pnacl_newlib_raw/bin/le32-nacl-ld.gold: internal error in output_address, at ../../../src/binutils/gold/output.cc:3009 If I remove "-g" from that command, the failure goes away, so debugging info is involved. Cheers, Mark -- You received this message because you are subscribed to the Google Groups "Native-Client-Reviews" group. To unsubscribe from this group and stop receiving emails from it, send an email to native-client-reviews+unsubscribe@googlegroups.com. To post to this group, send email to native-client-reviews@googlegroups.com. Visit this group at http://groups.google.com/group/native-client-reviews. For more options, visit https://groups.google.com/d/optout.
On 28 May 2015 at 16:27, Mark Seaborn <mseaborn@chromium.org> wrote: > I checked: The linker failure still happens when using tip-of-tree > pnacl-llvm, without your new change. > > Removing "-g" from SConstruct won't help because there's still debugging > info in the pre-built standard libraries. > > This minimal test also reproduces the linker failure (with "-mc-relax-all" > added in pnacl-translate.py): > > $ cat minimal.c > int _start() { > return 123; > } > > $ ./toolchain/linux_x86/pnacl_newlib_raw/bin/pnacl-clang -g minimal.c -o > minimal.pexe -nostdlib > $ ./toolchain/linux_x86/pnacl_newlib_raw/bin/pnacl-translate -arch x86-32 > minimal.pexe -o minimal.nexe --allow-llvm-bitcode-input > /home/mseaborn/devel/nacl/native_client/toolchain/linux_x86/pnacl_newlib_raw/bin/le32-nacl-ld.gold: > internal error in output_address, at > ../../../src/binutils/gold/output.cc:3009 > > If I remove "-g" from that command, the failure goes away, so debugging > info is involved. > I think the problem is the one I identified earlier, where references to a data section (a string table) are wrong: see https://chromiumcodereview.appspot.com/1129013004/#msg5. This problem is affecting references within debug info too. For example, the assembly output (for minimal.pexe above) contains this for debug info: .section .debug_pubnames,"",@progbits .long .LpubNames_end0-.LpubNames_begin0 # Length of Public Names Info .LpubNames_begin0: If I disassemble the .o files produced with and without "-mc-relax-all", the diff between the two versions includes this: Disassembly of section .debug_pubnames: 00000000 <.debug_pubnames>: - 0: 19 00 sbb %eax,(%eax) + 0: 00 00 add %al,(%eax) 2: 00 00 add %al,(%eax) So the constant ".LpubNames_end0-.LpubNames_begin0" isn't getting outputted correctly. Cheers, Mark -- You received this message because you are subscribed to the Google Groups "Native-Client-Reviews" group. To unsubscribe from this group and stop receiving emails from it, send an email to native-client-reviews+unsubscribe@googlegroups.com. To post to this group, send email to native-client-reviews@googlegroups.com. Visit this group at http://groups.google.com/group/native-client-reviews. For more options, visit https://groups.google.com/d/optout.
Here is a more direct reproducer for the string table misassembly: $ cat asm_reloc_test3.ll @.str1 = internal constant [7 x i8] c"hello1\00", align 1 @.str2 = internal constant [7 x i8] c"hello2\00", align 1 @.str3 = internal constant [7 x i8] c"hello3\00", align 1 declare void @callee(i8*, i8*, i8*) define void @myfunc() { call void @callee( i8* getelementptr inbounds ([7 x i8]* @.str1, i32 0, i32 0), i8* getelementptr inbounds ([7 x i8]* @.str2, i32 0, i32 0), i8* getelementptr inbounds ([7 x i8]* @.str3, i32 0, i32 0)) ret void } $ toolchain/linux_x86/pnacl_newlib/bin/pnacl-llc -mtriple=i686-nacl-gnu -filetype=obj asm_reloc_test3.ll -o asm_reloc_test3.o -mc-relax-all $ objdump -dr asm_reloc_test3.o ... 00000000 <myfunc>: 0: 83 ec 0c sub $0xc,%esp 3: c7 44 24 08 15 00 00 movl $0x15,0x8(%esp) a: 00 7: R_386_32 .rodata b: c7 44 24 04 15 00 00 movl $0x15,0x4(%esp) 12: 00 f: R_386_32 .rodata 13: c7 04 24 00 00 00 00 movl $0x0,(%esp) 16: R_386_32 .rodata ... Cheers, Mark On 15 May 2015 at 23:52, Mark Seaborn <mseaborn@chromium.org> wrote: > The trybots are failing with validation errors on x86-32 and x86-64. > > I tried comparing output with and without passing "-mc-relax-all" to llc. > On a minimal test case, I noticed that the string table references end up > being wrong with "-mc-relax-all". Try this minimal example with -O0: > > void foo(const char *arg1, const char *arg2, const char *arg3) { > } > > void _start() { > foo("hello1", "hello2", "hello3"); > } > > Without "-mc-relax-all", llc outputs this: > > 43: c7 44 24 08 0e 00 00 movl $0xe,0x8(%esp) > 4a: 00 > 47: R_386_32 .rodata > 4b: c7 44 24 04 07 00 00 movl $0x7,0x4(%esp) > 52: 00 > 4f: R_386_32 .rodata > 53: c7 04 24 00 00 00 00 movl $0x0,(%esp) > 56: R_386_32 .rodata > > With "-mc-relax-all", llc outputs this: > > 43: c7 44 24 08 15 00 00 movl $0x15,0x8(%esp) > 4a: 00 > 47: R_386_32 .rodata > 4b: c7 44 24 04 15 00 00 movl $0x15,0x4(%esp) > 52: 00 > 4f: R_386_32 .rodata > 53: c7 04 24 00 00 00 00 movl $0x0,(%esp) > 56: R_386_32 .rodata > > Notice the use of offset 15 twice, referring to the *same* offset in the > .rodata section twice, which can't be right. > > I don't know whether this explains the validation failure, but it might do > if there is a problem with forward references. > > Any ideas? > > Cheers, > Mark > > -- You received this message because you are subscribed to the Google Groups "Native-Client-Reviews" group. To unsubscribe from this group and stop receiving emails from it, send an email to native-client-reviews+unsubscribe@googlegroups.com. To post to this group, send email to native-client-reviews@googlegroups.com. Visit this group at http://groups.google.com/group/native-client-reviews. For more options, visit https://groups.google.com/d/optout.
Message was sent while issue was closed.
Committed patchset #3 (id:40001) manually as 56b07283d8febb0bbb5bf2c96ca03ffd930dfba3 (presubmit successful). |