xorriso in ghostbsd-build

[scdbackup]

Hi,

i will study the compiler warnings. They look suspicious.


In the hope to get a more realistic tree of input files, i downloaded
ftp://ghostbsd.org/pub/GhostBSD/release ... -amd64.iso
but this lacks any EFI or HFS+ stuff.

Nevertheless
ftp://ghostbsd.org/pub/GhostBSD/release ... -amd64.iso
has the files which the xorriso run expects to find in /mnt/iso.

Again, no error found by valgrind. (Shall i be happy or shall i be sad ?)

---------------------------------------------------------------------------
Consider to additionally give on
ftp://ghostbsd.org/pub/GhostBSD/release ... 64.iso.md5
the image size in blocks of 2048 bytes. This would enable users to verify
the image when it is already on plain storage devices which yield more
bytes when read than the image file does. Like:

Code: Select all

dd if=/dev/cd0 bs=2048 count=1150638 | md5

---------------------------------------------------------------------------

Have a nice day

Thomas

[scdbackup]

Hi,

CLANG plus peculiarities in the HFS+ code contributed to libisofs
by Vladimir Serbinenko are the best suspects for now.

libisofs/hfsplus.c:618

Code: Select all

  set_time (&sb.ctime, t->now);

"sb" is of type

Code: Select all

  struct hfsplus_volheader {
    uint16_t magic;
    uint16_t version;
    uint32_t attributes;
    uint32_t last_mounted_version;
    uint32_t journal;
    uint32_t ctime;
    uint32_t utime;
    ...
  } __attribute__ ((packed));

Vladimir will not have used this without a reason, i fear.
The struct goes unchanged into the ISO result byte stream:

libisofs/hfsplus.c:656

Code: Select all

      ret = iso_write(t, &sb, sizeof (sb));

(Those who started their programmer's life with C rather than BASIC
are hard to convince that it is unwise to do that.)

I do not really understand the mapping from struct to byte array.
Shouldn't one of the two neighbors

Code: Select all

    uint32_t ctime;
    uint32_t utime;

be aligned to 64 bit ?
Why do both cause a warning ?

Well, in any case one of the two should be unaligned, indeed.

The warnings about line 853 ff. are about struct hfsplus_catfile_common
with similar gestures as in line 656 ff.

The code in xorriso/cmp_update.c:243 and test/compare_file.c:246
is not supposed to be visited by a run for grub-mkrescue.

---------------------------------------------------------------------

A remedy could be to replace this gesture

Code: Select all

    set_time (&sb.utime, t->now);

by

Code: Select all

    uint32_t timebuf;
    ...
    set_time (&timebuf, t->now);
    memcpy((char *) &sb.utime, (char *) timebuf, 4);

If you can reproduce the SIGSEGV with above struct in a small test program,
then please try whether my idea of a remedy would prevent the SIGSEGV.

(Grmpf. HFS+ is hard for me to test. Linux hates HFS+ block size 2048.
I would have to try making a reproducible ISO without the change and
then reproduce it with the code change. Not impossible.)

---------------------------------------------------------------------

Have a nice day

Thomas

[ASX]

I did something different, more raw, for sure: I removed the "packed" attribute from all structs in hfsplus.h, at cost of creating an invalid ISO image eventually.

Rebuilding xorriso this time doesn't complaint about unaligned pointers, yet once installed it fail with same signal error, which means we are not on the right track. must be something else.

I'm out of time right now, and will continue as soon as possible.

[scdbackup]

Hi,

i already made a test program before reading about your result without
"packed". It is now on the shelf for the case that we run out of ideas.
A stack trace of the SIGSEGV would help to get new ones.

Another bold test would be to compile xorriso by gcc. Just to make sure
that CLANG plays a role.

Have a nice day

Thomas

[ASX]

Another bold test would be to compile xorriso by gcc. Just to make sure
that CLANG plays a role.

Bingo!!!

Compiled with gcc6 and indeed it just works!

Considering it was working in 10.3, and the issue appeared in 11.0 could be a regression introduced in clang 3.8 that was shipped starting from 11.0.

[scdbackup]

Hi,

the success with gcc at least explains why my best efforts cannot
find anything.

But what next ? Any idea what it could be, except the alignment ?

My only idea is the wish for a stack trace.


Here is my little test program for the alignment suspicion.
Let's call it "t.c":

Code: Select all

/*
  Compile by
    cc -g -o t t.c
  or
    cc -g -DWITH_MY_REMEDY_PROPOSAL -o t t.c
  and run
    ./t
*/

/* for uintNN_t */
#include <stdint.h>

/* for memcpy */
#include <string.h>


struct hfsplus_volheader
{
  uint16_t magic;
  uint16_t version;
  uint32_t attributes;
  uint32_t last_mounted_version;
  uint32_t journal;
  uint32_t ctime;
  uint32_t utime;
  uint32_t backup_time;
  uint32_t fsck_time;
} __attribute__ ((packed));


void iso_msb(uint8_t *buf, uint32_t num, int bytes)
{
    int i;

    for (i = 0; i < bytes; ++i)
        buf[bytes - 1 - i] = (num >> (8 * i)) & 0xff;
}


static void set_time (uint32_t *tm, uint32_t t)
{
  iso_msb ((uint8_t *) tm, t + 2082844800, 4);
}


int main()
{
    struct hfsplus_volheader sb;
    uint32_t ts = 1506011781; /* from date '+%s' today */

#ifdef WITH_MY_REMEDY_PROPOSAL

    uint32_t timebuf;

    set_time(&timebuf, ts);
    memcpy((char *) &sb.ctime, (char *) &timebuf, 4);
    memcpy((char *) &sb.utime, (char *) &timebuf, 4);

#else

    set_time (&sb.ctime, ts);
    set_time (&sb.utime, ts);

#endif

    return(0);
}

If the suspicion is right, then the binary made without
-DWITH_MY_REMEDY_PROPOSAL should crash and the one with the macro should
end with exit value 0.

If it does not crash, then we need to make it more similar to
libisofs/hfsplus.h and libisofs/hfsplus.c.

------------------------------------------------------------------------

We will need to decomission HFS+ before the time stamps roll over
as 32 bit unsigned values by the added constant 2082844800 to the
seconds since 1970.
So independently of the SIGSEGV it might be wise to mute option -hfsplus.
(See
https://dev.lovelyhq.com/libburnia/libi ... cue-sed.sh
for my own spy-and-manipulate script for grub-mkrescue.
My favorite is a no-fuzz MBR based BIOS+EFI layout named "mbr_only",
together with xorrisofs option -partition_offset 16.)

Have a nice day

Thomas

[ASX]

Here is my little test program for the alignment suspicion.

Doesn't crash. I made a similar test before removing the "packed" attribute, but forgot to report.

Best course of action is to instrument a debugger, and get a stack trace, but to do that I need to go into grub-mkrescue details, which is the program calling xorriso ... I will try to do that.

[ASX]

Just another note:

the problem arise just before or a normal excution would print out something like this, no idea if this can be of some help:

libisofs: NOTE : Automatically adjusted MBR geometry to 1017/142/32
xorriso : UPDATE : 0.32% done
xorriso : UPDATE : 0.65% done
xorriso : UPDATE : 1.00% done, estimate finish Thu Sep 21 21:22:36 2017
xorriso : UPDATE : 1.48% done, estimate finish Thu Sep 21 21:22:28 2017
...

[scdbackup]

Hi,

> I need to go into grub-mkrescue details

It artfully makes MBR code, EFI System Partition image, and HFS+ specific
files. While doing this, it composes a little pile of expert options for
xorriso's -as mkisofs emulation.

We can take advantage of its work by using an ISO made by it.
The GhostBSD ISO made by FreeBSD 10.3 contains all data we need.

Just mount GhostBSD10.3-RELEASE-20160827-182745-xfce-amd64.iso
to a directory of your choice and run something like

Code: Select all

    export mount_point=/...path.where.BSD10.3.iso.is.mounted...

    export grub_mbr_file=/usr/lib/grub/i386-pc/boot_hybrid.img

    gdb /...some.path.../xorriso
    (gdb) run -as mkisofs \
       -graft-points \
       --modification-date=2017092114161100 \
       -b boot/grub/i386-pc/eltorito.img \
          -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info \
       --grub2-mbr "$grub_mbr_file" \
       -hfsplus -apm-block-size 2048 \
       -hfsplus-file-creator-type chrp tbxj \
                                  /System/Library/CoreServices/.disk_label \
       -hfs-bless-by i /System/Library/CoreServices/boot.efi \
       --efi-boot efi.img -efi-boot-part --efi-boot-image \
       --protective-msdos-label \
       -o test.iso \
       -r --sort-weight 0 / \
       --sort-weight 1 /boot \
       "$mount_point" \
       -- \
       -volid GhostBSD

If your debugger does not support exported environment variables,
then insert "$mount_point" and "$grub_mbr_file" manually.

If you do not have
/usr/lib/grub/i386-pc/boot_hybrid.img
installed, then you can cut out the first 512 bytes of the 10.3 ISO:

Code: Select all

dd if=GhostBSD10.3-RELEASE-20160827-182745-xfce-amd64.iso \
   bs=512 count=1 of=/...path.of.your.choice.../grub_iso_mbr.img

export grub_mbr_file=/...path.of.your.choice.../grub_iso_mbr.img

One may well replace the output address "test.iso" by "/dev/null".
The treatment in libburn is a little bit different. But that's far away
from -hfsplus in libisofs.

If you use
http://scdbackup.sourceforge.net/xorriso-1.4.6.tar.gz
then you may set helpful compiler flags by variable CFLAGS before running
script ./configure.
(As most autotools users i do not really know what the 480 KB of
./configure do.)

I took above options from your earlier post in july.
But before doing that i compared the options with the analysis reported by:

Code: Select all

xorriso -hfsplus on \
        -indev GhostBSD10.3-RELEASE-20160827-182745-xfce-amd64.iso \
        -report_el_torito as_mkisofs

The options are the same, except that xorriso proposes to use this extra expert
option for the MBR:

Code: Select all

--grub2-mbr --interval:local_fs:0s-15s:zero_mbrpt,zero_gpt,zero_apm:'GhostBSD10.3-RELEASE-20160827-182745-xfce-amd64.iso'

which would copy the MBR directly from the original ISO. No need for "dd".

Have a nice day

Thomas

[ASX]

I will look into that tomorrow.