CAcert System Admins discussion list

[Jan Dittberner <jandd AT cacert.org>]

Dear CAcert community,

I will perform a major system upgrade of our infrastructure host (infra02
[1]) tomorrow starting at 8am UTC/10am CEST. Wytze van der Raay of the
critical infrastructure team will assist me via remote console if necessary.
I expect the upgrade to run for at least 4 hours and some services might
need fixes that will require even longer.

[1] https://infradocs.cacert.org/systems/infra02.html

Impact
------

Most services will be unavailable at least for parts of the upgrade session.
Effected systems are:

- infra02 itself
- blog.cacert.org
- board.cacert.org
- bugs.cacert.org
- cats.cacert.org
- email.cacert.org
- emailout.cacert.org
- git.cacert.org
- irc.cacert.org
- issue.cacert.org (OTRS)
- jenkins.cacert.org
- lists.cacert.org
- monitor.cacert.org
- motion (no public services yet)
- pgpkeys (currently not in public use)
- proxyin.cacert.org
- proxyout.cacert.org
- puppet (internal Puppet master)
- svn.cacert.org
- test.cacert.org
- test2.cacert.org
- test3.cacert.org
- testmgr.cacert.org
- translations.cacert.org
- web.cacert.org (Reverse Proxy for Jenkins, codedocs.cacert.org,
  funding.cacert.org and infradocs.cacert.org)
- webmail.cacert.org (Community webmail)
- webstatic (Host for codedocs, funding and infradocs)
- wiki.cacert.org

I will try to keep the downtime of essential services (email, emailout,
lists, blog, wiki) as short as possible.


Technical Details
-----------------

The operating system on the machine is currently a unsupported Debian 7
(Wheezy) system and hinders a lot of progress on the infrastructure side.

I will upgrade the system in multiple steps:

- Debian 7 (Wheezy) to Debian 8 (Jessie) (LXC 0.8.0 -> LXC 1.0.6)
- Debian 8 (Jessie) to Debian 9 (Stretch) (LXC 1.0.6 -> LXC 2.0.7)
- Debian 9 (Stretch) to Debian 10 (Buster) (LXC 2.0.7 -> LXC 3.0.1)

I will stop all LXC containers before each of the upgrades and will only
resume email, emailout, lists, blog and wiki after each reboot to

a) see that the containers still boot and record the necessary changes for
   required fixes
b) make sure that the service interruption is as short as possible

I will document all changes and send another announcement when we finish the
upgrades. Infradocs documentation will be updated too. I try to write some
ansible automation code to help me during the session tomorrow and serve as
guidance to make future upgrade smoother.


I hope to not cause to many inconvenience but we cannot wait longer to
perform these long needed update. The Debian Buster stable release last week
and the recently acquired knowledge on how to use the remote console system
of infra02 inspired us to perform the upgrade now.


Kind regards
Jan

-- 
Jan Dittberner - CAcert Infrastructure Team Lead
Software Architect, Debian Developer
GPG-key: 4096R/0xA73E0055558FB8DD 2009-05-10
         B2FF 1D95 CE8F 7A22 DF4C  F09B A73E 0055 558F B8DD
https://jan.dittberner.info/

Attachment: smime.p7s
Description: S/MIME cryptographic signature