PROPOSED, change of stance, release metronome

[Hal Murray]

verm at darkbeer.org said:
> This can get complicated real fast.  I would encourage distros to do rolling
>  testing but we cannot pin our releases to their schedule in any way.

> As long as we have our test results online and they can pinpoint the 'best'
> version to run they can go back a release or two in order to find one they
> are comfortable with.

I'm not sure what you have in mind for "test results".  How is a non-wizard 
going to be able to evaluate anything other than a yes/no, and we will 
already have filtered out the no-s.

> This also encourages us to release more frequently.  I don't believe we have
> any reason to have 10 releases in a month but we should always have at
> least 2 and no fewer than that.  This is to avoid stacking changes and lets
> things get tested in the test system.

What is the purpose of a release?

I'm assuming it's a tag on a pile of bits so we have a way to talk about them 
and a way to focus testing and usage on a smaller number of things to keep 
track of and support.


> We could also solve this by having a 'recommended' version on our website
> and point to test results.  It would be good to keep distros at least 30
> days behind our development cycle.  I expect the full run of run-time
> testing to take at least one week per built release.  I have a bunch of
> RPIs here that I will run ntpd on to make sure it keeps time properly one
> that test passes then we can say that is a 'stable' version of a release
> and recommend it as the next version to try. 

The idea of "recommended" sounds good.  It looks like you are automating the 
2 week-ish release cycle.  We still have to work out which releases to 
support long term and how long.


In terms of testing...

I look at a lot of graphs by hand.  I think I could automate some of that, 
but that would still leave a lot of work.

It may not be possible (or worth the effort) to totally automate the testing.

This will probably change when Eric's TESTFRAME starts working.  (I'm 
assuming we can run some real servers setup to collect data so we can gather 
a bunch of test cases when "interesting" things happen.)

We also have to test refclocks.

We can get real traffic by putting up pool servers.


-- 
These are my opinions.  I hate spam.