CAcert System Admins discussion list

[Daniel Black <daniel AT cacert.org>]

As our offer of hosting is withdrawn, personal allegations of my allegiance 
to 
CAcert are made, and board talk is starting to happen about requesting team 
reports for an Annual Report, I've decided to write this up to justify my 
passion towards hosting offers as my interest/capacity for CAcert diminishes. 

Let me know if it is missing anything. (?) bits i still need to check.

CAcert Infrastructure Report 2010

The year began slowly. In January/February Brian Henson started and finished 
some major work to get a puppet centralised management ready for CAcert. 
Daniel Black did some planning to see what will be needed for CAcert in the 
foreseeable future. Some testing began with Ksplice as a mitigation for 
kernel 
vulnerabilities without having to reboot servers specificity virtual host 
servers.

February hit and the effects of CVE-2009-3555 SSL renegotiation started to 
hit 
as browsers broke a previously permitted behaviour. The previous approach of 
optional/mandatory client certificate authentication was on a directory basis 
which would require a SSL renegotiation. Some interim work was done to 
lists.cacert.org/community.caert.org and blog.cacert.org(?) to require 
certificate authentication before a long term solution.

In March Mario Lipinski got restructured text working on the wiki.

April, Andreas Bürki got a proposal together with a hosting provider that 
covers our current and future requirements and put it to the board.

In May after a 3 month trial of KSplice the board approved to fund it for a 
year (m20100420.2). Thank you board. The gains of this in terms of uptime, 
security and lower sysadmin effort is much appreciated.

June saw some internal movements within BIT data centre. Thank you Wytze van 
der Raay for all the coordination and movement. Thanks also for getting all 
of 
our infrastructure services started due to our configuration problem.

Also in June, Jan Dittberner solved the CVE-2009-3555 issue. By packaging up 
a 
newer Apache version with SNI, using virtual hosting and certificates with 
subject alternate names we will be able to provide certificate authentication 
services, handle the idiosyncrasies of Safari, the poor error messages in 
Firefox.Jan also prepared a fully client certificate SVN server with client 
instructions.

July saw the withdrawal of infrastructure offer after no decision was reached 
by the board before the end of June deadline.

Current state of Infrastructure:

Currently there are far too many VMs on Debian 4.0 Etch that finished 
security 
support on February 14 2010. Those that can be easily updated have been.  A 
number of VMs have had adhoc packages installed that make an in-place upgrade 
is too risky an option with no reasonable blackout plan. The flexibility of 
the 
current managed gateway has made it undesirable to create and manage test VMs 
within the current for upgrading installations. 

As indicated by Jan's recent work on SNI testing new opportunities exist for 
developing better client certificate based infrastructure services. Ideally 
this should be tested on independent VMs and a migration strategy deployed.

The ability to deploy new testing services is not conducive in BIT which is 
managed gateway designed around production systems. The hassle with 
organising 
accounts with the critical admin team, as helpful as they are, and the delays 
in Tunix firewall changes make this an unsuitable location for dynamic 
infrastructure.

In short - new infrastructure is needed to move existing services to a 
stable, 
secure and sustainable state.

Regarding specific services:

wiki - on Debian Lenny. Looking for staff effort to migrate to a certificate 
auth 
and mitigate some spam.

Blog - on Debian Lenny. Fairly good state.

irc - is a mess of custom installed packages on what appears to be a Debian 
Etch host.

SVN - new Debian Lenny server prepared with full certificate authentication. 
Just needs to find a place to deploy to and then migration can happen.

bugs - on Debian Etch - not much effort/interest/investigation has been 
performed on this server.

lists - on Debian Etch - a number of custom fixes/packages are in place 
preventing an easy upgrade - particularly due to the criticality of the 
system. Volunteer effort for migration has been identified.

email - on Debian Etch - has a moderate amount of custom packages and 
configuration that will not survive and easy upgrade.

webmail/community.cacert.org - on Debian etch(?). Possibly upgradeable with 
some extreme care.

test2 - recently upgraded by Philipp(?)

hashserver.cacert.org - abandoned service

translingo - Etch server of unknown state. Crudely working but internals are 
unknown.

CATS - etch server. Class3 authentication is broken. Possibly upgradeable.

issue - Lenny server - working well and serving support teams well

logging - abandoned effort - logging achieved centrally using different 
mechanism.

forum - abandoned effort.

cod - documentation server - abandoned effort

emailout - working well as automated outbound services for wiki/issue 
tracking 
notices.

State of Staffing:

From a bulk recruitment that happened August last year only a few admins 
still 
remain. Some have formally resigned and others have faded from existence. 
While goals were set initially the crux of the problem is that flexible 
infrastructure is needed to deploy/test and migrate services.

Recently some new volunteers have offered to prepare Sympa6 and Mediawiki 
services in order to update our existing list and wiki services hopefully 
correcting a number of outstanding feature request/bugs. Without hosting 
there 
will be no place to provide these services.

Of concern is community projects that host important CAcert services like the 
main CAcert test/development site and co-auditing. These are occurring 
without 
the benefit of having CAcert ownership, backup, and monitoring. With no 
infrastructure hosting to offer these community teams the community assets 
they 
build are at risk from technical, relationship and management failures and 
may 
eventually be lost to the CAcert community.

So looking to the future the infrastructure team hopes to find a donor of 
infrastructure services who is willing to work with the CAcert board to 
provide reliable hardware and hosting so our aging systems can be migrated, 
and reinvigorated, new systems can appear, auditing critical systems will 
become easier (and less hassle for the critical team) and our staffing 
volunteer 
effort can be utilised.


-- 
Daniel Black
Infrastructure Team Lead
CAcert