Using Synth

[kraileth]

This is a first draft for a wiki article on Synth.

----------

Synth is one of two tools that can be used to create local package repositories for FreeBSD (the other being Poudriere). It is quite fast due to building packages in RAM.

Requirements

To use Synth you need a ports tree in place (in /usr/ports). If you don't have it, fetch it using the tool portsnap or check it out from SVN:

Code: Select all

# portsnap fetch extract

OR

Code: Select all

# svnlite co svn://svn.freebsd.org/ports/head /usr/ports

If you want the stable ports branch, checkout that one instead of head.

Synth also needs a distfiles directory. Either create it in the default location or, as Synth suggests, feel free to put it somewhere else as in that case you can always delete your ports directory and keep the distfiles (remember to adjust configuration in that case!):

Code: Select all

# mkdir -p /usr/ports/distfiles

Installing Synth

To install Synth either use pkg or build from ports:

Code: Select all

pkg install synth

OR

Code: Select all

make -C /usr/ports/ports-mgmt/synth

Configure Synth

Now you need to configure Synth:

Code: Select all

# synth configure

Take a look at the directories (A - G) and change them if desired.

Synth will suggest a certain number of builders (I) and jobs (J) based on your hardware (CPU and RAM available). Feel free to tune these numbers! Depending on several factors (and depending on what you want to build) other configurations may fit your needs better. Builders are the number of packages that are build concurrently. Jobs are the number of threads used for compilation of every builder. Example: If you have a four core CPU with hyperthreading that means you have eight threads available. To theoretically saturate your CPU you could use 1 builder and 8 jobs. This would be the equivalent to building software with "make -j 8". You could also decide to use 2 builders with 4 jobs each or 4 builders with 2 jobs. Since package building consists of more steps than just compiling, your CPU usually won't be under maximum load all the time. For that reason it can make sense for example to configure Synth to use 4 builders with 3 jobs even if you don't actually have 12 threads! Play with this number and find out what works best for your machine and your workload.

You should leave K and L enabled unless you have very little RAM. If you experience that Synth is using high amounts of swap all the time, first consider building less packages concurrently or with less jobs. As a last resort disable K and/or L.

Disable M if you can't stand the curses interface.

N gives you the option to fetch prebuilt packages (from the FreeBSD repo) to satisfy dependencies instead of building those locally, too.

And then there's H which will be discussed separately.

Using Synth

To be written...

[ASX]

please allow me a couple of notes:

instead of:

Code: Select all

# svnlite co svn://svn.freebsd.org/ports/head

I think is preferable something like, (which will correspond to portsnap results):

Code: Select all

# svnlite co svn://svn.freebsd.org/ports/head  /usr/ports

Also, distfiles are better placed outside /usr/ports tree, reason is: if something go wrong while fetching ports updates you can always delete /usr/ports and re-fetch, saving the distfiles, and avoid to re-download them again.

[kraileth]

please allow me a couple of notes:

instead of:

Code: Select all

# svnlite co svn://svn.freebsd.org/ports/head

I think is preferable something like, (which will correspond to portsnap results):

Code: Select all

# svnlite co svn://svn.freebsd.org/ports/head  /usr/ports

Totally! That path was was meant to be there and I simply lost it somehow.

Also, distfiles are better placed outside /usr/ports tree, reason is: if something go wrong while fetching ports updates you can always delete /usr/ports and re-fetch, saving the distfiles, and avoid to re-download them again.

I'll add the reason to the text. A little explanation never hurts.

[NevilleGoddard]

Thanks ASX and kraileth for your helpful posts. This is what I did.

I already had the ports tree so I moved it to another location.
I ran the command:

# svnlite co svn://svn.freebsd.org/ports/head /usr/ports

then:

# mkdir -p /usr/ports/distfiles

then:

pkg install synth

but perhaps better is:

make -C /usr/ports/ports-mgmt/synth

I then ran the command:

synth upgrade-system

Everything went fine until synth got to the synth port and it of course upgraded itself but then everything stopped working and my machine froze.
Perhaps if I installed synth from ports this wouldn't have happened as it would have already been upgraded so it wouldn't have upgraded and deinstalled itself?

Or did I make a mistake somewhere?

Anyway I rebooted and started again and everything seems fine.

[ASX]

Everything went fine until synth got to the synth port and it of course upgraded itself but then everything stopped working and my machine froze.
Perhaps if I installed synth from ports this wouldn't have happened as it would have already been upgraded so it wouldn't have upgraded and deinstalled itself?

synth itself doesn't perform the upgrade, it build a "Sinth" local repository, and there is no way this can cause problems while updating itself. In fact you could safely split the command you used in two:

Code: Select all

synth prepare-system
pkg upgrade -r Synth

Or did I make a mistake somewhere?

I think no, probably not: may be your system run out of ram + swap ? That would be a case where your system would loose resposiveness completely, at which point you would not even able to kill a process.

Anyway I rebooted and started again and everything seems fine.

Very likely your system run out of swap prior to reboot, this is configured mainly by "use tmpfs" along with the number of parallel builders.

[NevilleGoddard]

You're right ASX. Twice my machine has run out swap while using Synth. I reconfigured synth to not use swap and it's running well but much slower.

[ASX]

You're right ASX. Twice my machine has run out swap while using Synth. I reconfigured synth to not use swap and it's running well but much slower.

You should not remove swap, (eventually turn off use of tmpfs for local base and work area).
For your info, many packages are very small and require little ram, some are very large and require a lot of RAM (in form of tmpfs filesystems):
- chromium: ~8 GB
- libreoffice: ~ 10 B
- *-webkit-* ~ 8 GB
- openoffice ~ 14 GB (this one doesn't build at all, because synth configure tmpfs max size to 12 GB)

[NevilleGoddard]

Interesting. I turned off swap in Synth and /lang/rust went without a problem but it took about 2 hours. So did openjdk7. But with swap turned on, it failed.
I have 8gb of RAM and 3.9gb of swap. Should I create a swap file?

[NevilleGoddard]

if I run the commands:

synth prepare-system
pkg upgrade -r Synth

I get:


Updating Synth repository catalogue...
Synth repository is up to date.
All repositories are up to date.
Updating database digests format: 100%
Checking for upgrades (405 candidates): 100%
Processing candidates (405 candidates): 100%
Checking integrity... done (0 conflicting)
The following 76 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED: <-------
cheese-3.18.1
gvfs-1.26.3_4
libreoffice-5.2.6_1
midori-0.5.11
webkit-gtk2-2.4.11_9
ImageMagick-6.9.6.4_1,1
xfce-4.12_1
orage-4.12.1_1
shotwell-0.26.0_1
webkit2-gtk3-2.8.5_8
libgdata-0.17.4
evolution-data-server-3.18.5_6
gnome-online-accounts-3.18.6_1
nautilus-3.18.5
gksu-2.0.2_6
py27-qt5-gui-5.6
qt5-imageformats-5.7.1
Thunar-1.6.11
xfce4-desktop-4.12.3_2
py27-webkitgtk-1.1.8_6
py27-qt5-widgets-5.6
hplip-3.16.11_2
catfish-1.4.1

New packages to be INSTALLED:
python36: 3.6.1_1 [Synth]

Installed packages to be UPGRADED:
youtube_dl: 2017.03.24 -> 2017.05.01 [Synth]
wget: 1.19 -> 1.19.1 [Synth]
webp: 0.5.2_1 -> 0.6.0_1 [Synth]
sqlite3: 3.17.0 -> 3.18.0 [Synth]
shared-mime-info: 1.5 -> 1.8 [Synth]
ruby: 2.3.3_2,1 -> 2.3.4,1 [Synth]
readline: 6.3.8 -> 6.3.8_1 [Synth]
py27-sip: 4.18,1 -> 4.19.2,1 [Synth]
py27-qt5-dbussupport: 5.6 -> 5.7.1 [Synth]
py27-pillow: 3.4.2 -> 3.4.2_1 [Synth]
perl5: 5.24.1 -> 5.24.1_1 [Synth]
pciids: 20170316 -> 20170423 [Synth]
net-snmp: 5.7.3_14 -> 5.7.3_16 [Synth]
nano: 2.8.0 -> 2.8.1 [Synth]
libwps: 0.4.6 -> 0.4.6_1 [Synth]
libvisio01: 0.1.5_8 -> 0.1.5_9 [Synth]
libunwind: 20121006_2 -> 20170113 [Synth]
libsigc++: 2.4.1 -> 2.10.0 [Synth]
librevenge: 0.0.4_3 -> 0.0.4_4 [Synth]
libraw: 0.18.1_1 -> 0.18.2 [Synth]
libpagemaker: 0.0.3_2 -> 0.0.3_3 [Synth]
liborcus: 0.12.1_1 -> 0.12.1_2 [Synth]
libodfgen01: 0.1.6_2 -> 0.1.6_3 [Synth]
libnghttp2: 1.21.0 -> 1.22.0 [Synth]
libmwaw03: 0.3.10 -> 0.3.10_1 [Synth]
libmspub01: 0.1.2_9 -> 0.1.2_10 [Synth]
libical: 1.0.1 -> 2.0.0 [Synth]
libgd: 2.2.4,1 -> 2.2.4_1,1 [Synth]
libetonyek01: 0.1.6_4,1 -> 0.1.6_5,1 [Synth]
libedit: 3.1.20150325_2,1 -> 3.1.20170329_2,1 [Synth]
libcmis: 0.5.1_2 -> 0.5.1_3 [Synth]
libcdr01: 0.1.3_6 -> 0.1.3_7 [Synth]
libcdio-paranoia: 10.2+0.93+1 -> 10.2+0.94+1 [Synth]
libabw: 0.1.1_4 -> 0.1.1_5 [Synth]
icu: 58.2,1 -> 58.2_1,1 [Synth]
gpgme: 1.8.0_1 -> 1.9.0 [Synth]
gnutls: 3.5.9 -> 3.5.11 [Synth]
gnupg: 2.1.19_1 -> 2.1.20 [Synth]
glibmm: 2.44.0,1 -> 2.50.1,1 [Synth]
glib: 2.46.2_5 -> 2.50.2_1,1 [Synth]
git: 2.12.1 -> 2.12.1_1 [Synth]
foomatic-db: 20161209 -> 20170413 [Synth]
fftw3-float: 3.3.6.p1_1 -> 3.3.6.p2 [Synth]
fftw3: 3.3.6.p1_1 -> 3.3.6.p2 [Synth]
desktop-file-utils: 0.22_4 -> 0.23 [Synth]
dbus-glib: 0.104 -> 0.108 [Synth]
dbus: 1.10.14_2 -> 1.10.16_1 [Synth]
clucene: 2.3.3.4_9 -> 2.3.3.4_10 [Synth]
cantarell-fonts: 0.0.24 -> 0.0.25 [Synth]
boost-libs: 1.63.0_1 -> 1.64.0 [Synth]

Installed packages to be REINSTALLED:
python3-3_3 [Synth] (direct dependency changed: python36)
fish-2.5.0 [Synth] (direct dependency changed: python36)

Number of packages to be removed: 23
Number of packages to be installed: 1
Number of packages to be upgraded: 50
Number of packages to be reinstalled: 2

The operation will free 542 MiB.

Proceed with this action? [y/N]:

Do I have to update Synth's repositories?

[ASX]

Interesting. I turned off swap in Synth and /lang/rust went without a problem but it took about 2 hours. So did openjdk7. But with swap turned on, it failed.
I have 8gb of RAM and 3.9gb of swap. Should I create a swap file?

Hmm ... I suspect you turned off use of tmpfs too, else your result aren't consistent.

Preferably you should have a swap partition, if possible something 16 GB.
A swap file can be considered, except if you are using ZFS, swap file on top of ZFS doesn't work well at all.

I have not tried a swap file on UFS, but I think it should work, anyay the result will be similar to running synth without using tmpfs.

The question is: how many builder are you running in parallel ?
1 builder will be safe, then configure the number of jobs to make use of all your cores.
2 builders will be fine most of the time, except when two large packages will be built simultaneously, that is the case when synth would need more that 8 + 4 GB of ram + swap.

Alternatively, with your current setup (8 + 4 GB) you could:
- turn off tmpfs for work area
- turn on tmpfs for local base
- assuming your cpu is a 2 cores 4 threads you could try something like 2 builders x 3 jobs, or 2 x 4

See also:
https://forums.freebsd.org/threads/59107/