I was discussing with the Debian Reproducible Builds people some problem that I've noticed (only) with LTO builds. The build id (and debug link) was changing all the time and Jérémy Bobbio suggested that one reason for this problem are the odd filenames in the debug sections. I was able to workaround this special problem by using -flto-partition=none But now I still have differences in the debug sections. I tried to narrow it down, uncompressed it and found out that both files have the same size. And things like the debug_str sections have common parts but oddly (random?) ordered. This currently makes LTO unsuitable for reproducible builds on Debian (amd64, and most likely all other architectures). I was not able to find any related bug in the meta bug #47819. Maybe it should be added there. My discussion of the findings can be found at http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20150209/000933.html
Confirmed. int main() { return 0; } gcc t.c -g -flto produces <0><166>: Abbrev Number: 1 (DW_TAG_compile_unit) <167> DW_AT_producer : (indirect string, offset: 0x181): GNU GIMPLE 5.0 .0 20150128 (experimental) [trunk revision 220205] -mtune=generic -march=x86-64 -mtune=generic -march=x86-64 -g -fmath-errno -fsigned-zeros -ftrapping-math -fno -trapv -fno-strict-overflow -fno-openmp -fno-openacc -fltrans-output-list=/tmp/c cmHsiT8.ltrans.out -fltrans <16b> DW_AT_language : 12 (ANSI C99) <16c> DW_AT_name : (indirect string, offset: 0x29b): /tmp/ccmHsiT8.ltrans0.o which has both tmpfile names in DW_AT_producer and DW_AT_name. I have a fix for the first issue, but I'm not sure what to use for DW_AT_name. For -flto-partition=none we get sth like /tmp/ccxWIAkj.o while for regular LTO we get /tmp/cctZHL25.ltrans0.o. I can use simple plain "WPA" and "LTRANS".
Created attachment 34724 [details] Mini Testcases Current results (the -COMPILE things are just out of curiosity): SIMPLE-COMPILE: OK SIMPLE-LINK: OK LTO-COMPILE: FAIL LTO-LINK: FAIL LTO-OBJDUMP: OK LTO-EXTERNAL-DEBUG: FAIL LTO-STRIP-LINK: FAIL LTO-STRIP-EXTERNAL-DEBUG: FAIL LTO-BUILDID-LINK: FAIL LTO-SAVETEMPS-LINK: OK LTO-SAVETEMPS-EXTERNAL-DEBUG: OK The -save-temps was a suggestion by Jérémy Bobbio and shows that there is still a problem with the random file names when trying to use LTO to create reproducible builds. At least for the simple one file case
Author: rguenth Date: Wed Feb 11 12:00:44 2015 New Revision: 220613 URL: https://gcc.gnu.org/viewcvs?rev=220613&root=gcc&view=rev Log: 2015-02-11 Richard Biener <rguenther@suse.de> PR lto/65015 * dwarf2out.c (gen_producer_string): Drop -fltrans-output-list and -fresolution. Modified: trunk/gcc/ChangeLog trunk/gcc/dwarf2out.c
(In reply to conchur from comment #2) > Created attachment 34724 [details] > Mini Testcases > > Current results (the -COMPILE things are just out of curiosity): > > SIMPLE-COMPILE: OK > SIMPLE-LINK: OK > LTO-COMPILE: FAIL that's expected to fail > LTO-LINK: FAIL > LTO-OBJDUMP: OK > LTO-EXTERNAL-DEBUG: FAIL > LTO-STRIP-LINK: FAIL > LTO-STRIP-EXTERNAL-DEBUG: FAIL > LTO-BUILDID-LINK: FAIL > LTO-SAVETEMPS-LINK: OK > LTO-SAVETEMPS-EXTERNAL-DEBUG: OK > > The -save-temps was a suggestion by Jérémy Bobbio and shows that there is > still a problem with the random file names when trying to use LTO to create > reproducible builds. At least for the simple one file case -save-temps makes some of the names non-random. I still expect that the patch I just committed plus Index: gcc/dwarf2out.c =================================================================== --- gcc/dwarf2out.c (revision 220613) +++ gcc/dwarf2out.c (working copy) @@ -24521,8 +24521,11 @@ dwarf2out_finish (const char *filename) gen_remaining_tmpl_value_param_die_attribute (); /* Add the name for the main input file now. We delayed this from - dwarf2out_init to avoid complications with PCH. */ - add_name_attribute (comp_unit_die (), remap_debug_filename (filename)); + dwarf2out_init to avoid complications with PCH. + Avoid doing this for LTO produced units as it adds random + tempfile names. */ + if (!in_lto_p) + add_name_attribute (comp_unit_die (), remap_debug_filename (filename)); if (!IS_ABSOLUTE_PATH (filename) || targetm.force_at_comp_dir) add_comp_dir_attribute (comp_unit_die ()); else if (get_AT (comp_unit_die (), DW_AT_comp_dir) == NULL) will fix all cases. Indeed: > ./t.sh SIMPLE-COMPILE: OK SIMPLE-LINK: OK LTO-COMPILE: FAIL LTO-LINK: OK LTO-OBJDUMP: OK LTO-EXTERNAL-DEBUG: OK LTO-STRIP-LINK: OK LTO-STRIP-EXTERNAL-DEBUG: OK LTO-BUILDID-LINK: OK LTO-SAVETEMPS-LINK: OK LTO-SAVETEMPS-EXTERNAL-DEBUG: OK as expected.
Thanks for the patches. I've rebuild the gcc package (which took the whole afternoon + night on my machine) and can verify that the mini testcases are now working perfectly fine. I've also tried this with some other projects (shared objects and executables) and it now seems to work even without -flto-partition=none Is there any problem with the uncommitted patch that I should know about? Could this patch theoretically be applied in Debian or there problems which the Debian gcc-4.9 maintainers should know about?
(In reply to conchur from comment #5) > Thanks for the patches. I've rebuild the gcc package (which took the whole > afternoon + night on my machine) and can verify that the mini testcases are > now working perfectly fine. > > I've also tried this with some other projects (shared objects and > executables) and it now seems to work even without -flto-partition=none > > Is there any problem with the uncommitted patch that I should know about? > Could this patch theoretically be applied in Debian or there problems which > the Debian gcc-4.9 maintainers should know about? I've posted an alternative patch for review as some people thought there might be DWARF consumers that don't like dropping DW_AT_name, thus the alternative is to set it to sth like <artificial> (like in the alternate patch). The patch isn't reviewed yet. Once it's approved I plan to backport the patch to relevant branches.
Author: rguenth Date: Fri Feb 13 09:35:57 2015 New Revision: 220678 URL: https://gcc.gnu.org/viewcvs?rev=220678&root=gcc&view=rev Log: 2015-02-13 Richard Biener <rguenther@suse.de> PR lto/65015 * dwarf2out.c (dwarf2out_finish): Use <artificial> as DW_AT_name for LTO produced CUs. Modified: trunk/gcc/ChangeLog trunk/gcc/dwarf2out.c
Fixed for GCC 5 sofar.
Created attachment 34746 [details] Mini Testcases Thanks a lot. I was playing around with your new patches. And I was looking at the buildid the whole time (assuming that it is the hash over the whole binary). This doesn't seem to be and was therefore a wrong assumption by me. What does it mean: * your patches work fine with -flto -flto-partition=none (which is awesome) * my conclusion that it also works with plain -flto as I said in the Debian bug #777753 was wrong The testcases are updated to reflect this. SIMPLE-COMPILE: OK SIMPLE-LINK: OK LTO-LINK: FAIL LTO-OBJDUMP: FAIL LTO-EXTERNAL-DEBUG: FAIL LTO-STRIP-LINK: FAIL LTO-STRIP-EXTERNAL-DEBUG: FAIL LTO-BUILDID-LINK: FAIL LTO-SAVETEMPS-LINK: FAIL LTO-SAVETEMPS-EXTERNAL-DEBUG: FAIL NOPART-LTO-LINK: OK NOPART-LTO-OBJDUMP: OK NOPART-LTO-EXTERNAL-DEBUG: OK NOPART-LTO-STRIP-LINK: OK NOPART-LTO-STRIP-EXTERNAL-DEBUG: OK NOPART-LTO-BUILDID-LINK: OK NOPART-LTO-SAVETEMPS-LINK: OK NOPART-LTO-SAVETEMPS-EXTERNAL-DEBUG: OK
Difference in readelf -a is --- x 2015-02-13 12:04:07.526383914 +0100 +++ y 2015-02-13 12:04:10.158414178 +0100 @@ -234,7 +234,7 @@ 42: 0000000000600df8 0 OBJECT LOCAL DEFAULT 19 __do_global_dtors_aux_fin 43: 0000000000400580 0 FUNC LOCAL DEFAULT 13 frame_dummy 44: 0000000000600df0 0 OBJECT LOCAL DEFAULT 18 __frame_dummy_init_array_ - 45: 0000000000000000 0 FILE LOCAL DEFAULT ABS cc5R4C1Y.ltrans0.o + 45: 0000000000000000 0 FILE LOCAL DEFAULT ABS ccqmAj6c.ltrans0.o 46: 00000000004005b1 11 FUNC LOCAL DEFAULT 13 helper 47: 0000000000000000 0 FILE LOCAL DEFAULT ABS elf-init.c 48: 0000000000000000 0 FILE LOCAL DEFAULT ABS crtstuff.c that is, somehow .symtab contains a reference to the ltrans object file name, probably because of .file "ccqmAj6c.ltrans0.o" we emit via output_file_directive. Unfortunately omitting that doesn't help. It seems that the linker itself adds those symbols!? HJ? Gold doesn't do that (thus, it works with -fuse-ld=gold).
For reference, the following patch doesn't work: Index: gcc/varasm.c =================================================================== --- gcc/varasm.c (revision 220677) +++ gcc/varasm.c (working copy) @@ -7042,7 +7042,9 @@ default_file_start (void) && !(flag_verbose_asm || flag_debug_asm || flag_dump_rtl_in_asm)) fputs (ASM_APP_OFF, asm_out_file); - if (targetm.asm_file_start_file_directive) + if (targetm.asm_file_start_file_directive + /* LTO produced units have no meaningful main_input_filename. */ + && !in_lto_p) output_file_directive (asm_out_file, main_input_filename); } and -g is not necessary to trigger the bogus tempfile name in .symtab
Created attachment 34747 [details] Mini Testcases > Gold doesn't do that (thus, it works with -fuse-ld=gold). Weird, my gnu gold 2.25-4 (debian version) still shows the symptoms: SIMPLE-LINK: OK SIMPLE-OBJDUMP: OK SIMPLE-EXTERNAL-DEBUG: OK BFD-LTO-LINK: FAIL BFD-LTO-OBJDUMP: FAIL BFD-LTO-EXTERNAL-DEBUG: FAIL NOPART-BFD-LTO-LINK: OK NOPART-BFD-LTO-OBJDUMP: OK NOPART-BFD-LTO-EXTERNAL-DEBUG: OK GOLD-LTO-LINK: FAIL GOLD-LTO-OBJDUMP: FAIL GOLD-LTO-EXTERNAL-DEBUG: FAIL NOPART-GOLD-LTO-LINK: OK NOPART-GOLD-LTO-OBJDUMP: OK NOPART-GOLD-LTO-EXTERNAL-DEBUG: OK Maybe you have a newer gold version installed which already includes a patch which resolves this problem? Diff of bfd: 221c221 < 41: 0000000000000000 0 FILE LOCAL DEFAULT ABS ccsmMmrk.ltrans0.o --- > 41: 0000000000000000 0 FILE LOCAL DEFAULT ABS cctSjrTo.ltrans0.o Diff of gold: 201c201 < 14: 0000000000000000 0 FILE LOCAL DEFAULT ABS cc4lSDuA.ltrans0.o --- > 14: 0000000000000000 0 FILE LOCAL DEFAULT ABS ccxXItKB.ltrans0.o 245c245 < Build ID: a9117c8c1852ec657a0cdd6e206427571bae9b31 --- > Build ID: d65e429c3314089118f114fee1625276ef91835c
We are generating .file "cc8zpoWj.ltrans0.o" ^^^^^^^^^^^^^^^^^^^^^ We shouldn't do it. .text .Ltext0: .type helper, @function helper: .LFB0: .file 1 "dependency.c"
On February 13, 2015 2:42:45 PM CET, "hjl.tools at gmail dot com" <gcc-bugzilla@gcc.gnu.org> wrote: >https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65015 > >--- Comment #13 from H.J. Lu <hjl.tools at gmail dot com> --- >We are generating > > .file "cc8zpoWj.ltrans0.o" > ^^^^^^^^^^^^^^^^^^^^^ >We shouldn't do it. Huh, but then my patch should have worked. Maybe I botched testing. > .text >.Ltext0: > .type helper, @function >helper: >.LFB0: > .file 1 "dependency.c"
I opened: https://sourceware.org/bugzilla/show_bug.cgi?id=17973
(In reply to Richard Biener from comment #11) > For reference, the following patch doesn't work: > > Index: gcc/varasm.c > =================================================================== > --- gcc/varasm.c (revision 220677) > +++ gcc/varasm.c (working copy) > @@ -7042,7 +7042,9 @@ default_file_start (void) > && !(flag_verbose_asm || flag_debug_asm || flag_dump_rtl_in_asm)) > fputs (ASM_APP_OFF, asm_out_file); > > - if (targetm.asm_file_start_file_directive) > + if (targetm.asm_file_start_file_directive > + /* LTO produced units have no meaningful main_input_filename. */ > + && !in_lto_p) > output_file_directive (asm_out_file, main_input_filename); > } > > and -g is not necessary to trigger the bogus tempfile name in .symtab This patch works together with BFD linker patch: https://sourceware.org/ml/binutils/2015-02/msg00193.html
This patch works wit the existing linker: diff --git a/gcc/varasm.c b/gcc/varasm.c index 0211306..f3241a8 100644 --- a/gcc/varasm.c +++ b/gcc/varasm.c @@ -7043,7 +7043,12 @@ default_file_start (void) fputs (ASM_APP_OFF, asm_out_file); if (targetm.asm_file_start_file_directive) - output_file_directive (asm_out_file, main_input_filename); + { + if (in_lto_p) + output_file_directive (asm_out_file, "<artificial>"); + else + output_file_directive (asm_out_file, main_input_filename); + } } /* This is a generic routine suitable for use as TARGET_ASM_FILE_END
Created attachment 34753 [details] A patch
(In reply to H.J. Lu from comment #18) > Created attachment 34753 [details] > A patch even for -flto-partition=none we produce 45: 0000000000000000 0 FILE LOCAL DEFAULT ABS ccPyi2gu.o In theory we can also emit a .file directive before each variable/function we output (from location info). I suppose the debugger consumes them for backtraces? Thus produce .file "test.c" .text .globl main .type main, @function main: .LFB0: .cfi_startproc pushq %rbp .cfi_def_cfa_offset 16 .cfi_offset 6, -16 movq %rsp, %rbp .cfi_def_cfa_register 6 call helper popq %rbp .cfi_def_cfa 7, 8 ret .cfi_endproc .LFE0: .size main, .-main .file "dependency.c" .type helper, @function helper: .LFB1: .cfi_startproc pushq %rbp .cfi_def_cfa_offset 16 .cfi_offset 6, -16 movq %rsp, %rbp .cfi_def_cfa_register 6 movl $1, %eax popq %rbp .cfi_def_cfa 7, 8 ret .cfi_endproc .LFE1: .size helper, .-helper .ident "GCC: (GNU) 5.0.0 20150213 (experimental)" .section .note.GNU-stack,"",@progbits but appearantly as/ld doesn't handle multiple global .file directives well? I get 45: 0000000000000000 0 FILE LOCAL DEFAULT ABS dependency.c 46: 0000000000000000 0 FILE LOCAL DEFAULT ABS test.c 47: 00000000004005b1 11 FUNC LOCAL DEFAULT 13 helper ... 71: 00000000004005a6 11 FUNC GLOBAL DEFAULT 13 main so 'helper' is associated with test.c wrongly(?) Patch I was playing with (for functions only): Index: gcc/varasm.c =================================================================== --- gcc/varasm.c (revision 220677) +++ gcc/varasm.c (working copy) @@ -1713,6 +1713,10 @@ assemble_start_function (tree decl, cons char tmp_label[100]; bool hot_label_written = false; + if (targetm.asm_file_start_file_directive + && in_lto_p) + output_file_directive (asm_out_file, DECL_SOURCE_FILE (decl)); + if (flag_reorder_blocks_and_partition) { ASM_GENERATE_INTERNAL_LABEL (tmp_label, "LHOTB", const_labelno); @@ -7042,7 +7046,9 @@ default_file_start (void) && !(flag_verbose_asm || flag_debug_asm || flag_dump_rtl_in_asm)) fputs (ASM_APP_OFF, asm_out_file); - if (targetm.asm_file_start_file_directive) + if (targetm.asm_file_start_file_directive + /* LTO produced units have no meaningful main_input_filename. */ + && !in_lto_p) output_file_directive (asm_out_file, main_input_filename); } which shows an alternative to <artificial> by picking a random source file name via output_file_directive (asm_out_file, DEC_SOURCE_FILE (symtab->first_defined_symbol ()->decl)); That said, I think we can go with <artificial> for now and try to improve that later. I'm going to test an adjusted patch using in_lto_p again.
The assember already produces Symbol table '.symtab' contains 11 entries: Num: Value Size Type Bind Vis Ndx Name 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND 1: 0000000000000000 0 FILE LOCAL DEFAULT ABS dependency.c 2: 0000000000000000 0 FILE LOCAL DEFAULT ABS test.c 3: 0000000000000000 0 SECTION LOCAL DEFAULT 1 4: 0000000000000000 0 SECTION LOCAL DEFAULT 2 5: 0000000000000000 0 SECTION LOCAL DEFAULT 3 6: 000000000000000b 11 FUNC LOCAL DEFAULT 1 helper 7: 0000000000000000 0 SECTION LOCAL DEFAULT 5 8: 0000000000000000 0 SECTION LOCAL DEFAULT 6 9: 0000000000000000 0 SECTION LOCAL DEFAULT 4 10: 0000000000000000 11 FUNC GLOBAL DEFAULT 1 main so not sure if multiple global .file directives are really supported.
Finally fixed for 5.0 with reasonably backportable patches.
Author: rguenth Date: Mon Feb 16 14:53:23 2015 New Revision: 220735 URL: https://gcc.gnu.org/viewcvs?rev=220735&root=gcc&view=rev Log: 2015-02-16 Richard Biener <rguenther@suse.de> PR lto/65015 * varasm.c (default_file_start): For LTO produced units emit <artificial> as file directive. Modified: trunk/gcc/ChangeLog trunk/gcc/varasm.c
Awesome :) Just tested it and it seems to work quite well here.
Author: rguenth Date: Tue Feb 24 09:09:11 2015 New Revision: 220935 URL: https://gcc.gnu.org/viewcvs?rev=220935&root=gcc&view=rev Log: 2015-02-24 Richard Biener <rguenther@suse.de> Backport from mainline 2015-02-11 Richard Biener <rguenther@suse.de> PR lto/65015 * dwarf2out.c (gen_producer_string): Drop -fltrans-output-list and -fresolution. 2015-02-13 Richard Biener <rguenther@suse.de> PR lto/65015 * dwarf2out.c (dwarf2out_finish): Use <artificial> as DW_AT_name for LTO produced CUs. 2015-02-16 Richard Biener <rguenther@suse.de> PR lto/65015 * varasm.c (default_file_start): For LTO produced units emit <artificial> as file directive. 2015-01-17 Jan Kratochvil <jan.kratochvil@redhat.com> * dwarf2out.c (gen_producer_string): Ignore also OPT_fpreprocessed. Modified: branches/gcc-4_9-branch/gcc/ChangeLog branches/gcc-4_9-branch/gcc/dwarf2out.c branches/gcc-4_9-branch/gcc/varasm.c
Backported for 4.9.3.
Fixed. Thanks for reporting!
Author: rguenth Date: Tue Feb 24 11:27:43 2015 New Revision: 220938 URL: https://gcc.gnu.org/viewcvs?rev=220938&root=gcc&view=rev Log: 2015-02-24 Richard Biener <rguenther@suse.de> Backport from mainline 2015-02-11 Richard Biener <rguenther@suse.de> PR lto/65015 * dwarf2out.c (gen_producer_string): Drop -fltrans-output-list and -fresolution. 2015-02-13 Richard Biener <rguenther@suse.de> PR lto/65015 * dwarf2out.c (dwarf2out_finish): Use <artificial> as DW_AT_name for LTO produced CUs. 2015-02-16 Richard Biener <rguenther@suse.de> PR lto/65015 * varasm.c (default_file_start): For LTO produced units emit <artificial> as file directive. 2015-01-17 Jan Kratochvil <jan.kratochvil@redhat.com> * dwarf2out.c (gen_producer_string): Ignore also OPT_fpreprocessed. Modified: branches/gcc-4_8-branch/gcc/ChangeLog branches/gcc-4_8-branch/gcc/dwarf2out.c branches/gcc-4_8-branch/gcc/varasm.c