Super-script     ~/TD/a/Allcamps/do-project-statistics

#_____ This is an example ______#
# setup for an analysis variant #
#                               #
  setenv VAR -OO-ndr-fsg-H      #
  setenv DVAR -ochy             #
  setenv DVAR -ochy1d           #
  setenv UVAR $VAR$DVAR         #
  setenv OVAR -ov0              #
#################################
# The comments here and those in the script differ a bit.
#
Wonder about data files produced under this roof? Look here.

Prepare

cd ~
. .4tides
. .4ag
g C
. .4recent

Do

Pre-requisit: You have campaign condumps, a merged condump, a feature table, and a jd-file for analysis variant $VAR , and a drift solution designated $DVAR
i.e. you have passed the stage where urtapm and urtapt produce solutions
(includes preparation stages make-ts4VAR and make-tse4urtapm ).

Depending on $VAR you'll need a collecting directory, also used for some intermediary result files:
setenv ODIR tmp$UVAR

Stage "Campaigns"

You will have obtained a residual with one or the other project being far from the ensemble mean. We are going to compute partial derivatives of the residuals w.r.t. offsets one campaign at a time. Originally we use the instrument-specific offsets from the international comparison campaigns, which may encompass a sequence of campaigns at OSO. The script that alters the offsets at this processing stage is  tmp/cvaroffsets2env (a backup original should be kept in safe/ ). The offsets from the international comparisons are in safe/offsets2env.orig .
The process is controlled by a source script do-offset-variations .

do-offset-variations :
With switch -c it enacts campaign-wise processing. It involves a loop running urtapm and urtapt for each offset. The sequence number creates a file name mark $OVAR, here -cv0 (no variation, reference case), and -cv1 ... -cv14 for the campaign nudged.

With switch -s it varies only selected campaigns, however together. The file name mark will be -ovs   

Without a switch, all campaigns formally depending on the ICC's will be varied one-by-one. The file name mark will be -ov0 ... -ov9.

The scripts doing the variations are
-s : svaroffsets2env
-c : cvaroffsets2env
default: varoffsets2env
There are copies in subdirs. The one going to be used must be placed/symlinked in ~/TD/a/Allcamps/tmp/
The originals are in   ~/TD/a/Allcamps/safe/?varoffsets2env
At every pass of the loop, a tp-ts-file is created, tmp/meters${OVAR}.ts 
set do_prep=1
set do_solve=1

. do-offset-variations -c

plot-project-statistics D -c

plot-project-statistics . -c
$campaigns

This creates files with project-means and mean-differences-from-cv0  in $ODIR
The output that that project-statistics creates from tsex $odir/wstats.tse,S
contain MEAN, WMEAN and WRMS

Stage "Adjustment"

We collect the multi-campaign mean and RMS differences as follows:
touch $odir/campstats-diff-cv.smry
foreach i ( `N 0 $#campaigns` )
   setenv OVAR -cv$i
   project-statistics I -c >> $odir/campstats-diff-cv.smry
end
and sort them by decreasing RMS:

#================================================================================
    sort -r -n -k6 $odir/campstats-diff-cv.smry >! $odir/campstats-diff-cv.sort
#================================================================================

Finally we collect the g-rate results from the urtapt -cv protocol files:
awk '/SSCG/&&/year/{s=FILENAME;sub(/.*-cv/,"-cv",s);sub(/\.prl/,"",s); \
    printf "%-5s %8.2f +- %8.2f\n", s,$6,$8}' \
    o/scg-cal-merged$UVAR-cv*.prl >! $ODIR/rates-cv.dat
Now, using the differences w.r.t. the reference case, a number of campaigns can be adjusted simultaneously to lower the residual. In the example following we use up to five campaigns in descending order of misfit RMS.
superc-pp @ superc-pp.ins :FSG >! logs/superc-pp$UVAR.log

. do-offset-variations -s 5

plot-project-statistics D -s 5
plot-project-statistics . -s 5 $campaigns
# So far the super-script

Aftermath:
To list the original and the adjusted offsets,
rm -f tmp/coffs.dat
touch tmp/coffs.dat 
foreach c ( $campaigns )
   eval echo \$YOFFS_$c \>\> tmp/coffs.dat  
end
foreach i ( `N 1 5` )
   awk '/>ADD/{printf "%8.2f :: %s\n",$3,$9}' logs/varoffsets2env-sv$i.log |\
    paste -d' ' tmp/coffs.dat - >! $ODIR/adjusted-campaign-offsets-sv$i.rsl
end
paste -d' ' $ODIR/adjusted-campaign-offsets-sv1.rsl $ODIR/adjusted-campaign-offsets-sv5.rsl
10.0    10.00 :: 200907a 10.0    10.00 :: 200907a
17.0    17.00 :: 200911a 17.0    17.00 :: 200911a
17.0    17.00 :: 201004a 17.0    17.00 :: 201004a
47.0    47.00 :: 201006a 47.0    70.70 :: 201006a
47.0    47.00 :: 201009a 47.0    47.00 :: 201009a
47.0    47.00 :: 201106a 47.0    47.00 :: 201106a
11.0    11.00 :: 201106b 11.0    -2.79 :: 201106b
11.0    11.00 :: 201106c 11.0    -2.79 :: 201106c
22.0    22.00 :: 201304a 22.0    36.05 :: 201304a
22.0    22.00 :: 201405a 22.0    23.92 :: 201405a
52.0    52.00 :: 201405b 52.0    52.00 :: 201405b
52.0    52.00 :: 201502b 52.0    52.00 :: 201502b
25.3    25.30 :: 201505a 25.3    25.30 :: 201505a
25.3   -20.58 :: 201606a 25.3    -3.70 :: 201606a
10.0    10.00 :: 201707a 10.0    10.00 :: 201707a

awk '{if($2!=$1){print}}' $ODIR/adjusted-campaign-offsets-sv6.rsl
47.0    65.99 :: 201006a
11.0    -1.06 :: 201106b
11.0    -1.06 :: 201106c
22.0    35.31 :: 201304a
22.0    24.79 :: 201405a
52.0    74.78 :: 201502b
25.3    -0.05 :: 201606a

To list the g-rate results - remember we have a rate bias in the tide analysis, so we add it here:
set ratebias=`awk '/<ASlope>>> Sum/{print $4}' ~/Ttide/SCG/logs/urtap-openend-lpt-ochy1d.log`

awk -v rb=$ratebias '/SSCG/&&/year/{s=FILENAME;sub(/.*-sv/,"-sv",s);sub(/\.prl/,"",s); \

    printf "%-5s %8.2f +- %8.2f\n", s,$6+rb,$8}' \
    o/scg-cal-merged$UVAR-sv*.prl >! $ODIR/adjusted-rates-sv.dat

-sv0    -10.99 +-     0.13
-sv1     -2.45 +-     0.13
-sv2     -2.54 +-     0.13
-sv3     -3.04 +-     0.13
-sv4     -2.94 +-     0.13
-sv5     -2.94 +-     0.13






Files, columns, permissible values, results to verify

Under plot-project-statistics D, .rsl files, e.g.  ( OVAR = -cv9 , odir = tmp-OO-ndr-H-ochy )
which calls  project-statistics W 
$odir/project-stats$OVAR.rsl  - statistics evaluated on the basis of $ODIR/ra.mc
       Mean        WMean        WRMS                                               %RMS        %XSQ          %NVS
     18.72532     18.65941      1.211 :: Onsala_AS_20090702a.drop.txt FG5-233    42.18837    41.95518        1200 AS

     18.55008     18.42779      2.192 :: Onsala_AA_20090703a.drop.txt FG5-233    77.76180    75.79336        1196 AA
     ...
where  WRMS = R`%XSQ %RVS /` and %RVS = sqrt(%NVS), so that WRMS is the 1-sigma uncertainty of the mean.

$odir/project-stats-diff$OVAR.rsl  - difference of $OVAR w.r.t. -cv0
       Mean        WMean        WRMS                                                  %RMS     %XSQ   %NVS
     1.27643      1.2532      1.2110 :: Onsala_AS_20090702a.drop.txt        FG5-233 42.18837 41.95518 1200 AS

    -0.34683     -0.3536      2.1920 :: Onsala_AA_20090703a.drop.txt        FG5-233 77.76180 75.79336 1196 AA


and, under  project-statistics O  
$odir/project-coffs$OVAR.rsl - applied offsets = official values and one variation
                               (here, for -cv9 the extra offset is not visible since it's for campaign 201707a,
                               check with tmp/cvaroffsets2env )

     10.00000   :: Onsala_AS_20090702a.drop.txt FG5-233 AS
     10.00000   :: Onsala_AA_20090703a.drop.txt FG5-233 AA


Preparing the input to superc-pp :
set rsl=$odir/campstats-diff
What  project-statistics I  writes to STDIN is the summary of the  statistics internally saved in $rsl$OVAR.dat  
$rsl$OVAR.dat
         Mean         WMean        WRMS                                                      %RMS
        1.27643      1.25323      1.21100 :: Onsala_AS_20090702a.drop.txt          FG5-233 42.18837

       -0.34683     -0.35364      2.19200 :: Onsala_AA_20090703a.drop.txt          FG5-233 77.76180

An awk-script sums this up according to

   set means = ( `awk '{su+=$1;sw+=$2;n++} END{print su/n,sw/n}' ${rsl}$OVAR.dat` )
i.e. the means of the project means, unweighted and weighted, respectively,

   awk -v su=$means[1] -v sw=$means[2] -v s=$OVAR \

      '{ru+=($1-su)**2;rw+=($1-sw)**2;n++} \
      END{printf "Summary: %-5s %12.3f%10.3f%10.3f%10.3f\n",s,su,sw,sqrt(ru/n),sqrt(rw/n)}' ${rsl}$OVAR.dat

e.g. for OVAR = -cv9
                      Mean      WMean     Sdev(M)   Sdev(WM)
Summary: -cv9         0.841     0.821     4.076     4.074




.bye