Precision and accuracy

[Gary E. Miller]

Yo Kurt!

On Sat, 21 Jan 2017 01:23:47 +0100
Kurt Roeckx <kurt at roeckx.be> wrote:

> On Fri, Jan 20, 2017 at 12:44:38PM -0800, Gary E. Miller wrote:
> > Imagine in front of you are two handheld voltmeters, and a super
> > precision voltage source.  
> 
> So let me correct all those terms to what people normally use them
> for.

You use your terms, I'll use mine.  As long as we understand each other.

You left out the part where I started with:

    "Many will disagree with my terms, for those that do, just think
    of the names I use as random variable names and try to follow the
    concept."

I really did try to find some online definitions of these that are
widely agreed on, and failed.  Everyone has their 'one true' dictionary,
but they can never seem to find it.  If you can find one I'd like to see
it.

> Suppose you have a reference voltage that has a voltage of
> exactly 1.000000000 V. And you measure that voltage with your
> voltmeter, and it shows you 0.995, 0.994, 0.993, 0.994, 0.992,
> 0.994, 0.995, and so on. Let's assume the average value is 0.994
> V, that it's a normal distribution and that the standard deviation
> is 1 mV. People could say that that 1 mV is the precision, and 6
> mV is the accuracy.

Ah, but now you have added in jitter.  I said nothing about jitter,
which not really a thing in real DVMs.  I have tested a lot of
NoName DVMs and expensive DVMs and never seen them jitter.

And standard deviation is only for when you do not know the real error
processes.

> Suppose you take a different voltmeter, and that one always shows
> 1.002 V. People might claim the precision is 0, but a better
> number would be halve the resolution, so 0.5 mV. An even better
> number could be 0.43 mV. That's sqrt(3) / 2 * halve the resolution. I
> has at least a rectangular distribution, and I think that's the
> correct formule to match the coverage of 1 standard deviation.

Which, since standard deviation is usefull only when you are already 
lost, and only for one particular type of noise, not useful.

Time and time again I have heard: you only use standard deviation
when you are already lost.  I 100% agree with that.
 
> But the 2nd example already shows why the first's precision is
> actually wrong, and they both need to get added together, and the
> first would be 1.1 mV (sqrt(1^2+0.43^2)).

In the ditch now.

> It's also possible that that 2nd meter is averaging over a longer
> period of time than that first one, and in that way improve it's
> precision.

Cheap meters do not average, even expensive DVMs do not average.

If they did the averaging only helps for certain types of noise, and
is misleading for other types.

Out of ditch and into the river.

> And if you really start to care about putting numbers of what the
> accuracy and precision is, you get into troubles. Which is why
> they're never talked about like that.

I agree you've gone off the rail, let's not talk like that.

> You don't start from a
> measurement and then come up with what the precision and accuracy
> is. Because you're probably forgetting all kinds of errors like
> the drift. 

Now are you bringing it time varying results.  Notice my demo was at
one point in time and NIST traceable.  You change my experiment then of
course you see different things.

Then go looking into the NTP data and you will see 10 times more
different terms with unique, non-standard, often contradictory
definitions.  We could both keep adding error terms all night.  Let's
not go there.  Let's clean up NTP.

> But you can ussually find values for some of those in
> the datasheets of your device, and they're ussually not even clear
> what they really mean.

Having worked in ATE for years, I can 100% agree with that.  But
with most reputable vendors, with some effort, their data could be
replicated.  At least when you were buying in the large volumnes my
ATE clients were.  You buy enough of their parts they will cooperate.
Otherwise they could be left unpaid, or worse, sued for fraud.

> If you go and look at the calibration sheet of your reference
> voltage, or one of those 8 digit volt meters, you're not going to
> find a mention of the accuracy and precision. Instead they're
> going to talk about the uncertainty, and say your reference voltage
> gives a voltage of 1.000000001 +/- 3 nV, where the 3 nV is an
> expanded uncertainty, maybe that it's a normal distribution,
> and that it has a k factor of 2 to have a 95% coverage.

We must be buying from a different class of vendors.  I have only
rarely seen normal or standard deviations in the real world.

> > Now plug in the Fluke meter.  I feed in the same 1.000000000 Volts
> > and I read on the meter 1.000 Volts.  How accurate, NIST traceable,
> > would you say the Fluke meter is?  You guess 0.1%?  Maybe, maybe
> > not.
> > 
> > Now I set the calibrator to 1.000490000 and the meter still reads
> > 1.000.  I set the meter to 1.00050000 and the meter changes to 
> > read 1.001.  How accurate is the meter?  I say the meter is accurate
> > to 0.001%  
> 
> That's some very nice fluke meter you got there. The best I could
> find claims an accuracy of 0.05% (I assume of reading, but could
> also be of range) + 1 digit, which would be at least 1.5 mV. All
> the others are worse with most doing 0.5% + 2 digits.

Yes, John Fluke Mfg. Co., Inc. made damn fine stuff.

> > The Fluke meter can in fact reliably, repeatably, NIST traceably,
> > differentiate between two inputs that are only 0.001% apart!  When
> > that Fluke meter said a voltage was 1.000, you know it was between
> > 0.99951 Volts and 1.00049 Volts.  You could in fact use it to
> > accurately, repeatedly, NIST traceably tune a voltage divider to
> > 1.000 Volts +/-0.001%  
> 
> That error would still be 0.5 mV, or 0.05%.

Sorry, I did slip a digit, but I was not giving a math lesson.  And you
do seem to have got my point.  Agreed or not.

> > Compare that to the cheap meter that is only 5%.  And yet they both
> > have a precision of 0.1%.  
> 
> That cheap meter probably also does 0.5%.

Not even close, at least in accuracy.  Just in, to use your preferred term:
resolution.  Don't throw numbers without defining them.  That's how
we get into these confusions, not defining the thing being calculated.

> (And they both have the
> same resolution, not precision.)

You use your English, I'll use mine, at least we now understand the 
previous confusion.

Which all circles back to that NTP also uses terms in non-standard often
contradictory ways.  Which do not relaate at all to what either of us
have just been saying.  Before we can fix that we need to figure out
what all those NTP key words mean.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	gem at rellim.com  Tel:+1 541 382 8588

	    Veritas liberabit vos. -- Quid est veritas?
    "If you can’t measure it, you can’t improve it." - Lord Kelvin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 455 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ntpsec.org/pipermail/devel/attachments/20170120/5460cb78/attachment.bin>