ISSUE TRACKER IS DOWN

[ASX]

preliminary examination of log files on the server host system:

Code: Select all

root@server:/var/log2/log # grep pkg messages*
messages.0:Jan 29 09:58:18 server pkg: pkg upgraded: 1.8.7_3 -> 1.9.4_1 
messages.0:Jan 29 09:58:23 server pkg: ccache-3.2.5_2 installed
messages.0:Jan 29 10:02:24 server pkg: synth-1.66 installed

The above is consistent with my actions, installing synth on the server.

now, from inside the jail:

Code: Select all

root@ghostbsd_www:/var/log.as/log # grep pkg messages*
messages.0:Feb  4 18:49:09 ghostbsd_www pkg: pkg upgraded: 1.8.8 -> 1.9.4_1 
messages.0:Feb  4 18:49:49 ghostbsd_www pkg: perl5-5.24.1.r4_1 installed
messages.0:Feb  4 18:49:50 ghostbsd_www pkg: perl5-5.20.3_15 deinstalled
messages.0:Feb  4 18:49:52 ghostbsd_www pkg: ImageMagick upgraded: 6.9.4.3,1 -> 6.9.6.4,1 
messages.0:Feb  4 18:49:55 ghostbsd_www pkg: perl5.20-5.20.3_15 installed

and:

Code: Select all

root@ghostbsd_www:/var/log.as/log # pkg info | grep perl
perl5-5.24.1.r4_1              Practical Extraction and Report Language
perl5.20-5.20.3_15             Practical Extraction and Report Language

who did this update on Feb 4 ?
why pkg was at 1.8.8 on jail and 1.8.7 on host ?

[ASX]

Fixed by downgrading ImageMagick package to ImageMagick-6.9.4.3,1

[ASX]

Let me add a little rant here:

- while I was investigating this issue, among other things I though at a possible break-in, therefore I take a good look at various log files: the saved auth.log go down up to 16 Feb only... that because of the many failed login tentative logged and consequent reach of auth.log size limit and rotation.

As you can see that is insufficient to track who logged in on Feb 4 when the package was installed. Lucklily this ImageMagick issue was only an oversight on our side, but I if it would have been a reak break in that would be a major issue by now.

Therefore we should change our log settings to extend the time-range of the logged events.
Additionally, It could be a good idea to transfer the log files elsewhere, also regularly, say once a day.

- I don't know if we have backup, as far as I know we don't and we should make it regularly and act very soon.
I would take the chance to put in good use the EU server that kraileth company kindly offered us, and for a start we could transfer there some backup of the jail, regularly, say once a week or so.
Obviously we need an agreement on that proposal.

- Finally and last, I would like to put in place a policy (and instructions) to update the server and the jails, I guess we have all interest in keeping it up, running and patched/upgraded regularly.

[ericbsd]

I will be for the motion of that policy.

That update was my fault, I thought image magic was not installed but it was i did not know that would done that.

And for the login tentative should we set ssh of the jail to only ssh key?

[ASX]

I will be for the motion of that policy.

ok

That update was my fault, I thought image magic was not installed but it was i did not know that would done that.

OK, beware that redmine is particularly picky, if unsure just have a good backup before performing any update.

And for the login tentative should we set ssh of the jail to only ssh key?

YES, on jail AND on host too. Note that both the kail and the host already have our public keys set. (you, me, and kraileith).

One thing I don't know and would like to know is: do you have any backup access to the machine in case something go wrong with the ssh setup ?

[kraileth]

Yes, we totally need to improve the situation with the server. Yet it is not much fun to try and fix an unknown server in general - it's a time consuming task and you never know if you overlooked something and break it accidentally. I assume that we don't have any useful server documentation, right? If that doesn't exist, I strongly suggest that creating this should be the very first step! It could be as simple as that:

Host: FQDN
Specs: ...
Functions:
- function a:
Provided by service b in jail c
Comment: Caution when updating, package x is known to break y!
- function d:
Done via cronjob (root's crontab)
Comment: Works but needs further optimizing
- function e:
...

Things like that which would quickly show any newcomer what the server does and give him an idea of how. BTW: Doing this is not at all a waste of time! We're going to make use of the EU mirror server and we're discussing a third box as a build server. That's three machines already. Complexity will rise and things are likely to get more and more messy if not at least some minimal formal documentation exists. According to my experience, documenting things after implementing them also helps quality, since you're thinking things over again when you write them down. Also other team members are more likely to ask questions or make suggestions.

Can we agree on creating some kind of documentation per server? I'm ok with just about any form, the above is just a very simple example. But without doing anything I predict that we're going to shoot ourselves in the foot one day.

[ASX]

ktaieith, fully agree!

[ASX]

FYI: I deleted from the server an old backup I made when we moved /usr under its own ZFS dataset, therefore the total used storage decreased now.