GhostBSD website access, file sizes

[kraileth]

The project's starting page has some nice hi-res png images which I do like both optically and technically (going with the open file format PNG instead of the proprietary and ugly JPEG or something). However they are quite big and take really long to load when you're on a slow internet connection (mine isn't that slow but I noticed it when my bandwidth was eaten by e.g. a parallel download) and it's probably no fun at all with a mobile connection (haven't checked that).

There are various "png optimizers" out there of which to my knowledge "pngout" used to provide the best results (it has been some years since I actually compared those programs). It isn't open source (which is a pitty) but at least it's free software. Ken Silverman (the guy behind the Duke Nukem 3D engine of the '90s) wrote them and releases them for Windows. A friend of him (I think) ports a lot of his work over to Linux, dito for pngout. The cool thing here: He also provides binaries compiled natively for FreeBSD which is certainly cool!

I used that tool to optimize the big PNG files that are used on the starting page. Actually I did that last year but then forgot about it. Now I've run across them again and even if 11.0 is not too far away anymore and will certainly have new pictures (btw., does it have a codename, yet?) IMO it makes enough of a difference to use the optimized ones until then. And of course I can optimize the new ones when they are ready, too. It only takes a bit of trial and error, not too much time. Currently all 7 pictures together are 9.3 MB in size, the optimized versions save about 540K. Altogether it may not be that much but then again half a meg out of less than 10 is not that bad, either. And we're talking about the web here with "time to render" and stuff like that. BTW: Google tools rate our site with 48/100 for desktop and even 39/100 for mobile! So those optimized files may just be a first start in getting a better scoring website for GhostBSD...

@Eric: I uploaded a file called "optimized_pngs.tar" and put it in /root on the Canadian server as I didn't know where the files needed to go. Could you take a look at that? Oh, and ASX asked me a while back to register for an account on the website backend which I did. But obviously I made a typo on that day and cannot log in. In fact this may be a good thing since it made me try password recovery - only to find that it is probably configured wrongly: It says that it sent a mail (I tried this out a couple of times on different days) but nothing ever arrives. Could you just delete my account by hand and I'll recreate it?

[kraileth]

And about another related issue: You've released Alpha1 of 11.0 (thanks for that!) with ISOs for MATE and Xfce for both amd64 and i386. This totally makes sense. But of course that's a lot of data ( I need about 1 hour to download MATE 64 bit). ASX mentioned once that he's currently online only via mobile connection and would easily exceed his monthly allowed traffic if he downloaded multiple ISOs. I'd imagine that there are more people out there who'd be happy if they could download smaller images.

So after downloading I thought that I'd give it a try and compress the ISOs to see if they contain data that compresses well. I would have expect that it doesn't make much sense to compress them or we'd probably already do this. Compression yielded some considerable savings however! Here are the detailed results:

GhostBSD11.0-ALPHA1-20170209-200437-mate-amd64.iso

• original: 2056589312
• gzip: 1853160753
• bzip2: 1850511004
• zstd: 1804963063
• xz: 1792697960

GhostBSD11.0-ALPHA1-20170209-202334-xfce-i386.iso

• original: 1797920768
• gzip: 1620574993
• bzip2: 1617185248
• zstd: 1579776856
• xz: 1570547396

With savings of about 260 MB (MATE) or 225 MB (Xfce) respectively with the strongest compression and still about 200 or 180 MB even with the old gzip, we should consider if releasing compressed ISOs, too, might be an option.

@Eric: What does the release process currently look like? You build the releases directly on the Canadian server, I guess, calculate the hashes and copy the files into a directory of the FTP server?

I'm thinking about a "do_release" script which would scan the default build dirs and check if all four ISOs are present. It could then automatically compress the ISOs, calculate sums for both the uncompressed and compressed ISOs and move them over into the FTP dir. And eventually it could push the most strongly compressed one (scp?) over to the European server to a fixed location. There we should have a cron job scaning that dir for new releases and act upon discovering one (check hash, decompress, recompress to create the missing formats and make the release available via FTP, too). What do you think about something like this?

[ASX]

kraileth, the ISOs already contains a compressed "/usr" filesystem, that's why they don't compress further.

[kraileth]

kraileth, the ISOs already contains a compressed "/usr" filesystem, that's why they don't compress further.

Actually I expected that the compressed files would be 99% the size but it seems like there actually is enough uncompressed data to allow for relevant savings. Probably even better compression could be achieved using some tricks, but I'd say that 260 MB of 1.9 GB could be worth the effort.

[ericbsd]

kraileth I did delete your account you can redo a new one.

[ericbsd]

I would say pretty much the same thin then ASX about the image, it is compress to is limit, sure there is tweak that could reduce the ISO size like reducing the icon them one set of icon, replacing xorg by xorg-lite, LibreOffice, Firefox use lot of space. I looking to replace Firefox but not on 11.0, Firefox is not Good Browser anymore.

On the website right now there is to image that does not display on the Sponsor widget, look like does 2 website can be reachable anymore. I am gonna replace it with text until the guy fix is problem. Also I might replace the slide later.

[kraileth]

Ok, it seems like you don't think the savings of 200 Megs are worth the effort, am I right here? In any case, I tried harder and have a better result. How does saving 900 MB out of 2 GB sound? Yes, I'm serious, the resulting file is 1153480898 bytes in size and can be restored to the original ISO (matching the md5 hash).

I've used a little tool written by friend of mine. It's not currently available from ports but it's open source, Apache licensed (he intended to make it GPL but fortunately I could convince him not to go down that road!) and current versions compile on FreeBSD without any patches. I could try to make and submit a port. Does saving almost a Gigabyte sound better than my previous attempt? This tool might be perfect for various ISOs and GhostBSD could be the first one to adopt it. What do you think about this?

[ASX]

I would not say that the current ISOs /usr is compressed to the limits, in fact actually it is a tradeoff between compression and performance (both creating the compressed folder, and expanding it later for live sessions).

/usr is (was?) compressed using zip compression, I remember that me and convbsd made a few experiment using xz compression in DesktopBSD, but don't know if that change was ported to GhostBSD too.

The base system is totally uncompressed, that's why you can get ~250 MB reduction.

The better compression (xz) come at a price: longer build time, larger memory requirements while running in live mode, this can be a problem for i386 arch based systems.

kraileth, if you can really achieve such compression, definitely do it, as long as a user can easily decompress the images, but in my opinion is not a top priority thing.

ericbsd, please define "Good Browser", because I'm unable to find one that fit the definition:
chromium in FreeBSD has issues
firefox is bloated and full of "telemetry"
qupzilla run fine for me, but it is QT based, (and however is not fully featured as firefox)
netsurf is light but lack of javascript, work well only on relatively simple sites
iceweasel (the firefox derived browser made from debian) would be great but is not available in ports/pkgs.
I'm going to try seamonkey .... but so far firefox remain the best option.

[kraileth]

I would not say that the current ISOs /usr is compressed to the limits, in fact actually it is a tradeoff between compression and performance (both creating the compressed folder, and expanding it later for live sessions).

/usr is (was?) compressed using zip compression, I remember that me and convbsd made a few experiment using xz compression in DesktopBSD, but don't know if that change was ported to GhostBSD too.

I'm pretty sure that it's any sort of zlib compression (e.g. "zip") as that were most streams which were detected and could be re-compressed by the tool that I use.

The better compression (xz) come at a price: longer build time, larger memory requirements while running in live mode, this can be a problem for i386 arch based systems.

Yes, I like xz a lot but I'd definitely vote against it for live media. It may work reasonably well on modern hardware but it's not really a good idea IMO.

kraileth, if you can really achieve such compression, definitely do it, as long as a user can easily decompress the images, but in my opinion is not a top priority thing.

I definitely can. And by now I also have a second ISO (32 bit Xfce) ready: 1797920768 Bytes -> 1007328305. 1.8 GB down to 1 GB is definitely a huge saving, but since it's accomplished by brute forcing, it takes a while even for the smaller ISO. On my system it took a little over 1 hour to compress. Of course restoring the original only takes a couple of minutes (10 almost exactly) but I expect this time to drop massively in a coming version (currently it uses temp files for all the streams... Doing this in memory would skyrocket the performance).

Just as a quick intro so you know what tool I'm talking about: Precomp works by looking for compressed streams in a file and trying to decompress those. If it succeeds, it tries to make a sensible guess but then basically brute forces known compression parameters (zlib levels, dictionary sizes, etc.) in trying to recreate the exact original stream (byte by byte identical). If it finds a match, it records the compression parameters and can use the decompressed stream. It can also do recursion and thus is able to decompress streams within streams that it decompressed. The resulting file usually is a lot bigger but can be made smaller by applying stronger compression (64 bit MATE ISO was precompressed to 7 GB before being re-compressed to 1.1 GB). Current versions of Precomp also allow for lossless compression of other media like JPEG, MP3, ...

ericbsd, please define "Good Browser", because I'm unable to find one that fit the definition:

I totally agree with you here, ASX. The situation is not nice at all when it comes to browsers. I really wish that somebody would at least succeed in fixing Chromium. But let's see what happens to Firefox now that they start pushing the Rust components...