P-S-01 — Paired difference experiment procedure

Category: Statistical methods
Bulletin: P-S-01
Document(s): S-S-01
Issue Date:
Effective Date:


Table of content


1.0 Purpose

1.1 The purpose of this procedure is to provide instructions that may be used for performing an experiment to assess the effect of a change on a characteristic of a sample unit, using the sample unit as its own control.

1.2 This procedureFootnote 1 can be used as a reference by organizations to develop their own procedures for performing a replicated paired difference experiment.

1.3 This procedure assumes that the minimal conditions for applying ISO 3301:1975 have been met, but additional criteria can be used.

2.0 Scope

2.1 This procedure applies where:

  1. a replicated paired difference experiment may be conducted on retrofitted diaphragm gas meters in service with an automatic meter reading (AMR) device and/or register;
  2. the meters tested are similar in all respects except for the systematic difference which is being evaluated;
  3. the analysis of the test results is based on ISO 3301:1975; and
  4. a single operator is involved in effecting the change; or
  5. more than one operator is involved in effecting the change and a technical professional thoroughly examines the procedure and determines that there is no possibility for inter‑operator variability.

3.0 Reference

3.1 S-S-01 – Specifications for Random Sampling and Randomization

4.0 Terminology

For the purposes of this procedure, the following terms and definitions and those in the normative reference apply.

Experiment (designed experiment)

Experimental plan selected so as to meet a specific objective.

Replication

Repetition of an experiment more than once for a given group of variables used in regression to predict others variables.

Operator

Person performing the operation of interest in the experiment.

Grand mean

Mean of a set of means.

5.0 Symbols

For the purposes of this procedure, the symbols in the normative reference and the following symbols apply.

d

the signed difference between paired observations

m

the number of measurement replications

n

the number of units in the sample

x

the measured value of a characteristic under the initial set of conditions

y

the measured value of a characteristic following the introduction of the change being evaluated

x ij

the value of x corresponding to the jth replicate of the ith sample unit

x ¯ l

the mean value of xi for the sample of n units

S 2

the estimate from the sample of the variance

v

the number of degrees of freedom

the sum of

Σ i = 1 n x i

the sum of all the x values when i takes integral values from 1 to n

g

the absolute value of an arbitrary variable g

6.0 Procedure

6.1 Planning

  1. Define the population or universe to which the results of the experiment will apply.
  2. Define the nature of the change that will be introduced to the population under study.
  3. Specify the characteristics of the population units which will be measured.
  4. For each sample unit, establish the size of the sample to be chosen from the population and the number of replicated measurements to be performed.
  5. Maintain a record of the data obtained in steps (a) to (d).

6.2 Sample selection and preparation

  1. Randomly select a representative sample from the defined population or universe in accordance with the reference cited in section 3.1.
  2. Condition the units of the sample to stabilize the characteristics to be measured.

6.3 Sample evaluation

  1. Measure the characteristics specified for each of the sample units, replicating the measurements the same number of times for each characteristic.
  2. Record all measurement results on the form supplied in Appendix A.
  3. Introduce the change to be evaluated to each of the sample units.
  4. Repeat steps (a) and (b).

6.4 Statistical calculations

The following calculations are to be applied when using Appendix A.

  1. For each characteristic of subject i under controlled conditions, calculate the mean ( x ¯ i ) of each set of replicated measurements in accordance with the following equation:
    x ¯ l = Σ j = 1 m x ij m
  2. From the means obtained in step (a), calculate the grand mean in accordance with the following equation:
    x = = Σ i = 1 n x ¯ l n
  3. For each characteristic of subject i under experimental conditions, calculate the mean ( y ¯ l ) of each set of replicated measurements in accordance with the following equation:
    y ¯ l = Σ j = 1 m y ij m
  4. From the means obtained in step (c), calculate the grand mean in accordance with the following equation:
    y = = Σ i = 1 n y ¯ l n
  5. Calculate the signed differences (di) between the means of the replicated measurements taken before and after the introduced change for each characteristic in accordance with the following equation:
    d i = x ¯ l y ¯ l
  6. Calculate the mean ( d ¯ l ) of the signed differences (di) for each characteristic in accordance with the following equation:
    d ¯ l = Σ i = 1 n d i n
  7. Calculate the variance ( S d 2 ) and the associated degrees of freedom (v) of the signed differences (di) for each characteristic in accordance with the following equations:
    S d 2 = Σ i = 1 n ( d i d ¯ ) 2 n 1
    v = n 1
  8. Calculate the ratio of Student's t-distribution for the sample size n in accordance with the following equation:
    t 0.95 ( v ) n
  9. Calculate the tolerance adjustment for the confidence interval and the variance of the sample for each characteristic in accordance with the following equation:
    A 1 = t 0.95 ( v ) n S d 2
  10. Maintain a record of the results of all statistical calculations performed in steps (a) through (i).

6.5 Analysis of results

6.5.1 Statistically significant mean paired difference experiment

  1. Determine if a statistically significant mean paired difference is within upper and lower bias limits using the two one‑sided cases below:

    Case 1

    The hypothesis that the population mean of the differences for each characteristic is at least equal to the upper tolerance limit (d1) (null hypothesis) is rejected if:

    d ¯ < d 1 A 1

    Case 2

    The hypothesis that the population mean of the differences for each characteristic is at most equal to the lower tolerance limit (d2) (null hypothesis) is rejected if:

    d ¯ < d 2 A 1

  2. There is insufficient evidence that a mean paired difference outside a ±0.2% limit exists if the null hypothesis is rejected for Case 1 and Case 2 for each characteristic.

Appendix A – Test results forms

A.1 Test results form—Controlled conditions

Test results form—Controlled conditions
Sample
Unit
Controlled conditions (before change introduced)
xi1 xi2 xi3 xi4 xi5 i
Sample units 1 to 30 Enter the meter performance test results in percent error. Enter the meter performance test results in percent error. Enter the meter performance test results in percent error. Enter the meter performance test results in percent error. Enter the meter performance test results in percent error. Enter the lot number.
Enter the grand mean of the means for the sample units.

A.2 Test Results form—Experimental conditions

Test results form—Experimental conditions
Sample
Unit
Experimental conditions (after change introduced)
yi1 yi2 yi3 yi4 yi5 i
Sample units 1 to 30 Enter the meter performance test results in percent error. Enter the meter performance test results in percent error. Enter the meter performance test results in percent error. Enter the meter performance test results in percent error. Enter the meter performance test results in percent error. Enter the lot number.
Enter the grand mean of the means for the sample units.

Appendix B – Analytical results form

Analytical results form
Description Symbol Characteristic 1 Characteristic 2 Critical value
Sample size n Enter the number of units in the sample. Enter the number of units in the sample. -
Replicates m Enter the number of measurement replications. Enter the number of measurement replications. -
Degrees of freedom v = n 1 Enter the degrees of freedom. Enter the degrees of freedom. -
Controlled conditions x = Enter the grand mean of the means for the sample units. Enter the grand mean of the means for the sample units. -
Experimental conditions y = Enter the grand mean of the means for the sample units. Enter the grand mean of the means for the sample units. -
Confidence interval (%) - - - 95
Difference statistics d ¯ Enter the mean of the signed differences between paired observations. Enter the mean of the signed differences between paired observations. -
S d 2 Enter the variance of the signed differences between paired observations. Enter the variance of the signed differences between paired observations. -
t 0.95 ( v ) n Enter the ratio of Student's t-distribution for the sample size. Enter the ratio of Student's t-distribution for the sample size. -
A 1 Enter the tolerance adjustment for the confidence interval and the variance of the sample between paired observations. Enter the tolerance adjustment for the confidence interval and the variance of the sample between paired observations. -
Analysis of dependent variables based on ISO 3301:1975 d 1 - - 0.2
d 2 - - -0.2
Case 1 (at most) Inside/Outside Inside/Outside -
Case 2 (at least) Inside/Outside Inside/Outside -
Conclusion (95%) Inside/Outside Inside/Outside Inside/Outside
tolerance

Footnotes

Footnote 1

The use of this procedure entails inherent risks and limitations with regard to the conclusions that may be drawn from it. Meter owners are therefore advised that, although conformity with the requirements of this procedure may qualify meter owners to seek accreditation for the installation of AMR devices on diaphragm meters in situ, relying solely on the use of this procedure will not provide meter owners with an assurance of compliance with the metering accuracy obligations prescribed under the Electricity and Gas Inspection Act. Contractors are ultimately responsible for ensuring the performance quality of the in-service meter lots which they own.

Return to footnote 1 referrer