CAcert System Admins discussion list

[Wytze van der Raay <wytze AT cacert.org>]

On 19.10.2013 20:31, Andre Klärner wrote:
> On Fri 18.10.2013 11:49:19, Wytze van der Raay wrote:
>> On 17.10.2013 20:28, Andre Klärner wrote:
>>> What's about using a cloudflare service to distribute the crl.cacert.org
>>> site?
>>
>> That's an interesting suggestion! I didn't know about the free CloudFlare
>> services. After having checked their website, it isn't entirely clear to
>> me though where this would work for crl.cacert.org. Do you (or someone 
>> else)
>> have any experience with the service?
> 
> Well, my suggestion was mainly to start a discussion about using a CDN for
> such bandwidth-intensive content.
> 
> After a bit more research from my side, I think that CloudFlare wouldn't be
> the way to go, unless we want to hit this problem with vast amounts of cash.

Well, that's simple, we don't have vast amounts of cash.

> The more open-source approach would be something like CoralCDN. Their
> approach also sounds quite easy to adopt, their upper limit according to
> the FAQ is about 250GB/day. Essentially the changes to our side would be
> redirecting clients to another URL that runs over Coral.

Hmm ... just tried it, but it didn't work very well. The download redirected
to CoralCDN stalls after 98304 bytes. And on a repeat, it stalls even
earlier. Any idea why??

> What might be interesting in this case: are there any anonymized logs for
> crl.cacert.org? I'd like to consolidate the logs to check for patterns in
> the traffic, like reoccuring user agents, frequent requests from a limited
> set of hosts etc. Maybe we could even smart up the delivery by clever
> caching outside the firewall, or redirecting specificly annoying clients to
> another host that can handle the load better or to a small CDN.

There are *very* anonymized logs, in the sense that the origin IP is always
replaced by the internal address of our proxying firewall. But the user
agent field and referer are still available in these logs. If you have
some scripts available that would be useful to run on these logs, let me
know, and I can run them for you and publish the results.
The logfile format is:

LogFormat "%h %l %u %t \"%r\" %>s %b \
\"%{Referer}i\" \"%{User-Agent}i\""                     combined

but note that %h is fixed at 172.16.3.1. Typically there will be 40.000 to
60.000 lines per day.

By the way, there is also an rsync alternative underway, for people who are
willing to use this for regular refreshing their copy of the CRLs, the
performance will be way way better than anything else. Stay tuned ...

Regards,
-- wytze