infrastructure

[ASX]

Sharing a couple thoughs.

We were talking about our machines, particularly about the builder setup.

Within few days will expire the subscription of our Xeon E5-1650/64G/2x3TB (the builder), which cost 100 CAD per month.

The idea was to substitute that machine with two E3-1245/32G/2x2TB, 56 CAD month each one. (this is also the same machine we currently use for our website/forum/issue tracker/wiki, aka the webserver), although this machine is a little slower than the E5-1650, having two of them will allow to run in parallel, one for i386 and one for amd64.

The need for SSD disks has been successfully overcome by using a ZFS RAID 0 (stripe) setup.

Also, in my mind I think we should build ISOs directly on the rispective builder, thus using the 'local' repository, that should allow us to build 'weekly' ISOs.

I think we could/should serve the pkgs and the ISOs directly from the builder machines, (after all those machines should build for 3 days each two weeks), but I'm not sure it is the right choice.
We could for example transfer the pkgs and the ISO with ZFS send/receive from our builder to the webserver.
Note that in the latter case, we could successfully make use of one builder only, and thus maintain only the current E5-1650.

We can choose between machines located in Canada datacenter or in France datacenter, I'm of the idea to keep one in Canada and one in France, if we go for two builders.

Your thoughts ? (this thread will be closed in two days)

for reference and avilability: https://www.soyoustart.com/ca/en/essential-servers/

[ASX]

Adding a couple of consideration:

The webserver is currently setup with a mirrored ZFS filesystem, for a total of 2 TB, overdimensioned to host our websites, forum, wiky, tracker, and surely more than enough to deploy also ISOs and packages.


scenario 1)
deploying packages and ISOs using our "webserver"

In this scenario, we could choose a SSD based machine: 3x120 GB (stripe)
https://www.soyoustart.com/ca/en/offers/e3-ssd-3.xml
to be used to build one arch repo, and upon completation send it to the webserver using zfs send/receive.

with this setup we can build both arch on two machine in parallel, but the storage is barely sufficient, I can think we can drop the cache and the needs will be around 75 GB for the repo, and 105 GB for the distfiles + 20 GB for other things.

scenario 2)
deploying ISOs and packages each arch from its own "builder"
In this case we need more storage (300 GB for 4 repos, + 105 GB distfiles + 100 GB ccache),
and therefore we could stripe two 2 TB disks:
https://www.soyoustart.com/ca/en/offers/e3-sat-3.xml

This setup has a few pro and a few cons:
pro:
- additional bandwith, limited per arch instead of globally
- no need to transfer the repo up to the webserver
- easy traffic stats per arch

cons:
- the build process may badly affect the download
- the download may badly affect the build process

[ericbsd]

Scenario 1 look interesting, but It is true that the storage will be really tight.

[ASX]

Scenario 1 look interesting, but It is true that the storage will be really tight.

yeah, unfortunately SYS doesn't have any 2x300 SSD (or 2x240) available ...

I'm thinking about renting only one machine with 3x120 SSD, considering that we could still build one arch at a time. That would make it a pure builder, upon completation the repo is immediately transferred to the webserver.
At least for the first month.

Also, trying this one week would be interesting:
https://www.ovh.com/ca/en/dedicated-ser ... 173sp2.xml
What is attracting me here is the RAM speed 2133 MHz vs. 1333 Mhz, because synth works entirely in RAM.

[kraileth]

Interesting choices here! Here are some of my thoughts:

If we're going for the two server setup, having one in Canada and one in France sounds like a good idea. To be really useful however, in that case both servers would have to be full mirrors. If they aren't, there's not too much benefit to it and it makes the whole infrastructure more complex.

About the space issue when going for the one machine option: We obviously have plenty of storage on the webserver. Have you considered using some of that space for the builder? If both servers are in the same datacenter in CAN that shouldn't be much of a problem really. We could either go the "classical" way of setting up NFS exports on the webserver and use those on the buildserver to store the packages. Or we could create zvols and export those via iSCSI targets. However I've never done the latter and I think (might be wrong) that it does not allow accessing the files on the hostmachine because there can only be one iSCSI initiator attached to each target. But maybe there's a way to get around this.

[ASX]

Have you considered using some of that space for the builder? If both servers are in the same datacenter in CAN that shouldn't be much of a problem really.

Yes, I have considered that, but that option is available only from OVH servers (more expensive), not from SYS.

I was also thinking at a single big machine, that would be viable if there is a way to assign/reserve some cpu cores for each task, i.e.:
- 4 cores for webserver
- 8 cores for builder
May be this can be feasible using some hypervisor ... (SmartOS ?). Not really an expert in this area.

[ericbsd]

Scenario 1 look interesting, but It is true that the storage will be really tight.

yeah, unfortunately SYS doesn't have any 2x300 SSD (or 2x240) available ...

I'm thinking about renting only one machine with 3x120 SSD, considering that we could still build one arch at a time. That would make it a pure builder, upon completation the repo is immediately transferred to the webserver.
At least for the first month.

Also, trying this one week would be interesting:
https://www.ovh.com/ca/en/dedicated-ser ... 173sp2.xml
What is attracting me here is the RAM speed 2133 MHz vs. 1333 Mhz, because synth works entirely in RAM.

Also worth to mention it is DDR4 and that the process is aE3-1270v6 which his as good that the current builder.

[ericbsd]

The best optimal solution would be https://www.ovh.com/ca/en/dedicated-ser ... 160mg1.xml. The incomes that we gain monthly is not enough to provide something like this that for a long time at best five months.

I think for now we need to go with the best we can go with for what we have. We get over 1000 CAD to play with, and we get in average more than 200 CAD per month.

Now I would like to try something to see, but time is also going fast, and this release has been delayed for build testing. Right now we have to make a choice and stick to it whatever happen, I have done a lot of change, and I have no idea if any will work. I need a repository to work with.

No machine will perform well without SSD. I think it start to be clear and the CPU and RAM also play a significant role in that too. But I don't think right now that we have a lot of solutions. We could go with OVH solution for 115 + tax CAD, but on all what I I have seen, I doubt that we will go over 700 pkg/hour.

[ASX]

I know you are waiting for the final repo, and that is why I built amd64 first, hopefully will be able to add own ports today, ideally tomorrow you will have the complete amd64 repo available on the webserver.

Right now we have to make a choice and stick to it whatever happen,

Not really. SYS rent server on a montly basis, and that's good

Setting up a build server require less than one day, if needed we can switch again one month later.

I propose to take two servers, for the upcoming month:
https://www.soyoustart.com/ca/en/offers/e3-ssd-3.xml
https://www.soyoustart.com/ca/en/offers/e3-sat-3.xml

and the idea is to go with scenario (1) ... at the end of the rent month we will replace one of the two server with what we will have found is better for us.

The scenario 3, eventually, is to stick with the current e5-1650 for another month, this is also an open option.
(I'm adding this because there are no other 64 GB RAM machines available from SYS).

[ericbsd]

Yea scenario 3 would be easy but, the performance is really poor.

We could go with your subjection for 2 servers it will show us how much SSD play a role in performance.