[M3devel] changes in CM3 continuous integration setup

[Jay K]

Olaf, ssh and cvs are usually failing now. It is quite bad now.
I have to retry cvs -z3 diff and cvs -z3 commit multiple times to get them to work.

 - Jay


> Date: Mon, 23 Aug 2010 13:41:40 +0200
> From: wagner at elegosoft.com
> To: jay.krell at cornell.edu
> CC: m3devel at elegosoft.com; m3-support at elego.de
> Subject: RE: [M3devel] changes in CM3 continuous integration setup
> 
> Quoting Jay K <jay.krell at cornell.edu>:
> 
> > I don't really understand all of this.
> > Surely CVS can tolerate more than 10 "active" users?
> > Or is it really awful?
> >   (or, most likely, both)
> > What if they all poll much less than once an hour?
> 
> It's not just CVS. The machine is running our WWW services, FTP,
> CVSup, backup for several hosts and secondary mail services.
> 
> > Let's say we have 12 machines.
> > Have them each poll 4 times per day: 48 polls.
> > So that's just one operation every 30 minutes.
> >  One poll, possibly followed by one update.
> 
> That would be an individual setup for all jobs regarding polling.
> It's of course possible, but more tedious work again.
> 
> > Or, didn't you set it up so only one machine polled and all the   
> > other builds would follow it, serially?
> > If not, that makes sense.
> > Have one machine poll every 30 minutes.
> > If it finds something, it builds, and the next one builds.
> > If it finds nothing, then nothing occurs.
> > Surely this would work ok?
> > It merely serializes all of our Hudson jobs?
> >   Which is, granted, rather unfortunate.
> >   Adding machines should allow great parallelism.
> >   And CVS is read-mostly, so should handle plenty of load.   
> > "Read-mostly" being
> >   a property that makes systems easier to scale.
> 
> The current setup is suboptimal.
> I've changed things in a way that could be done quickly.
> But it's not easy to foretell how any change will affect the load
> and responsiveness, as it isn't even clear where the limits are; and
> other services are continually competing for resources, too.
> 
> I've set up continuous logging of system load, top and iotop.
> You can get a quick summary with this job:
> 
> http://hudson.modula3.com:8080/view/zzz/job/sysload
> 
> We'll monitor this for some days and see how our changes affect
> the situation.
> 
> Complete logs are written to Hudson's ~/log/ directory.
> 
> > I assume if I bothered to learn to use cvsup, it would help much?
> > We'd have a few mirrors and it'd "fan out well"?
> > e.g. 5 cvsup mirrors could each serve 5 cvs clients, 25 for the price of 5?
> 
> I think we'd only need one CVS mirror that the continous integration
> could use. Unfortunately, our system administrator is on vacance (his wife's
> expecting their second baby) right now, and I haven't got any machine
> for that readily available.
> 
> We'll sort that out, though; don't worry too much. It may need some
> weeks, but we'll find an acceptable solution without much limitations.
> 
> Olaf
> -- 
> Olaf Wagner -- elego Software Solutions GmbH
>                 Gustav-Meyer-Allee 25 / Gebäude 12, 13355 Berlin, Germany
> phone: +49 30 23 45 86 96  mobile: +49 177 2345 869  fax: +49 30 23 45 86 95
>     http://www.elegosoft.com | Geschäftsführer: Olaf Wagner | Sitz: Berlin
> Handelregister: Amtsgericht Charlottenburg HRB 77719 | USt-IdNr: DE163214194
> 
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://m3lists.elegosoft.com/pipermail/m3devel/attachments/20100824/b6eee1a2/attachment-0002.html>