urtap merging and solution processing super-script

urtap-merge-pp
and utilities
     stats-of-merged  make-residual-tpmc  plot-tptight
     tpindex-for-campaign 
tpindex-for-project  ts2tpmc 


There are four alternatives to treat the drift of the SCG, and two analysis methods to account for them in the merged solutions
DVAR = -expf -dsyn -dwwf -ochy  - drift variants
VAR  = -O  -O-sdr               - analysis variants (the `-O´ stands for orientation offsets by instrument)
-expf  - solution from expfitbm
-dsyn  - solution from standard open-end tide analysis
-dwwf  - solution from extended open-end tide analysis
-ochy  -
solution from extended tide analysis with nontidal ocean and hydrology

-O     - subtracting the drift series - urtapm-big-O-ndr.ins urtap-merged-O-ndr.ins
                                        and symlinks without `-ndr´
-O-sdr - solving a drift coefficient  - urtapm-big-O-sdr.ins urtap-merged-O-sdr.ins
  
Drift solutions must have been prepared in advance.
expfitbm determines the exponential decay parameters.
They are used in the standard and extended tide solutions. They are long-term stable
unless a new event has to be accounted for. However, expfitbm determines also the
long-term linear and offset terms, so that re-running expfitbm is advisable from time to time.

urtap-merge-pp branches off at make-drift-tsf in the processing of SG-AG-data

It comprises the following stages:
1. make-drift-tsf
2. mc4campaigns
3. make-urtap-ins
4. urtapt on one campaign at a time, optionally with post-fit RMS-based reweighting
5. urtapm merging step for a specific combination of analysis and drift variants.
6. Message announcing that urtapt can be run on the merged problem.
urtapm and what to do next (under construction)

Ad 4:
In the first series of campaign solutions, if a job has failed, a common reason is a project that contains too little data (e.g. many outliers get edited away). To remove such a project, update the associated jump file. The log file will inform which column, say n, of the design matrix that has no signal.
The incriminating slope and offset definition is group n-1 (the first column contains the SCG-series).
Rerun with directive U
Stage 4 implies re-iteration for outliers.

The files campaign.weights and campaign.var :
The first solution produces weight factors, one for each campaign, specifying how much the AG uncertainties can be reduced to arrive at a normalized chi^2 of unity. The average (RMS) a-priori uncertainty of the merged files is computed and added as the last line of campaign.weights
...
%W 201606a -dwwf 3.4368E-01
%W 201606a -dsyn 3.4368E-01
%W 201606a -expf 3.4368E-01
%W 201606a -201606a 3.4368E-01
%E all  -expf 220.379
%E all  -dsyn 220.379
%E all  -dwwf 220.379

The sum-square a-priori uncertainty of each campaign is written to campaign.var
                    Sum sig^2   N
...
%V  201606a -dwwf 0.26494E+05 53225
%V  201606a -dsyn 0.26494E+05 53225
%V  201606a -expf 0.26494E+05 53225

Reweighting makes it slightly more complicated to compute a-priori offsets for campaigns or projects.
Read on here.
 

Ad 5:
The list of input .cdmp-files is created manually as is the whole .ins-file. Check that the
list contains the recent .cdmp-files from stage 4.

You can introduce qualified a-priori offsets. Read on here.
 

A-priori offsets for campaigns or projects
You will probably introduce an offset in experimenting with solutions after you have obtained the first urtap-merged result. Thus, the feature table can be used to find out the indexes. Campaign-wide offsets can be introduced in a simple way: This procedure invokes make-urtapm-tse which prepares a section YOFFS.
setenv YOFFS_201004a dg/w
where dg is the offset in nm/s^2 and w the effective mean uncertainty of the campaign. Use the calculator script
setenv YOFFS_<campaign> `eff-yoffs <campaign> dg`
Project-wide offsets can/should be added in this block manually.
eff-yoffs -p <project> dg
will return the tse-command.
$ eff-yoffs -p Onsala_AC_20160628 20
ADD 0.357521 From #129785 To #139276  ; 201606a Onsala_AC_20160628

"Qualified" offsets
If you suspect that projects are systematically offset, here's an example how to go about.
Assume you have run both urtapm and urtapt on the merged problem.
Run
stats-of-merged DE
and check the results (by default written to o/yoffs4prjs.tse). Example: (under construction)
ADD  F`3.9188 * ...` From #000000 To #001183 :: Onsala_AS_20090630a FG5-233
ADD  F`0.7266 * ...` From #001184 To #002383 :: Onsala_AS_20090702a FG5-233
ADD  F`-3.2576 * ...` From #002384 To #003569 :: Onsala_AA_20090703a FG5-233
...
The offset values to ADD are the differences Median minus Mean in nm/s^2
You suspect that these differences are big and significant.

Add a line
${CONT_YOFFS:[; ]
in the block headed by TSF EDIT YOFFS in urtapm$VAR.tse
and activate the offset section using
setenv CONT_YOFFS "CONT ?? R o/yoffs4prjs.tse"
Then run utapm and urtapt again.



 USAGE:   
           urtap-merge-pp [DIRECTIVES] [options] [<campaign> <campaign> ...]

 ALIAS:   
           ts4merge-superscript

 
 PURPOSE:
           Prepare observation mc-files
           Prepare and exec urtap-mrg.ins
              for different alternatives concerning drift;
              iterates until number of outliers is zero,.
           Prepare urtapm jobs.
           Exec urtapm for a specified drift alternative,
 
           Default campaigns (unless option -c is used):
           The campaigns in file campaigns.lst  
           Drift alternatives:
 
            DVAR  label        drift-file
           -dsyn SGD+DSYN from d/synth-drift-simp.ts
           -dwwf SGD+DWWF from d/synth-drift-extd.ts
           -expf SGD+EXPF from d/synth-drift-expfit.ts
           -ochy SGD+OCHY
from d/synth-drift-ochy.ts

 
           Keep OQUILTUR and M directives to separate jobs
  
          
 
 DIRECTIVES:
   O      - Observations. Prepare campaign mc-files with
              extra columns for drift alternatives.
   Os     - Skip the drift-file renewal.

   Q      - Inspect the mc-files for their columns content (labels)
 
       - Produce urtap-mrg.ins files. The precise file names are
               found in ins4mrg.lst .
   L      - Re-produce ins4mrg.lst .
 
       - Create symlinks for outlier tse-files
 
       - Exec urtapt for all ins-files in ins4mrg.lst
   Uw     - add a reweighting step so the input to
            urtapm is more balanced. All drops in a campaign
            are up- or down-weighted by the same amount.
 
       - Show results. See option -show
 
   M      - Run the merging job. Needs options -v and -d
            Checks integrity of pre-processing jobs.
   Mi     - Exit after integrity check.
 
 OPTIONS:

 -show <searchstring>  - With directive R , shows protocol lines

                         containing the string. Default is " SCG "

 -s <var>[,<var>...]   - Under directives OQUILTUR , select the drift variants
                         for which the perparatory jobs are issue; any of
                         expf dsyn dwwf ochy
                         All four are the default.
 
 -c <listfile>         - A file with the names of the campaigns instead of
                         specifying campaign names on the command line.
                         Uses pre-existing files urtap-*-mrg.ins
                 OBS!    Do not use with directive  I ,
                         since the ins-files must keep a unique
                         setup of slope features. All campaigns
                         must be included ( source ~/.4ag )

                         Best to first specify all campaign names on the
                         command line with directive I and then
narrow  
                         the range of campaigns by editing the list file.

 
-i <ins-file>         - Take the campaigns from an ins-file for urtapm
                         e.g. urtapm-big-O-ndr.ins
                         This option takes precedence over -c
 

 -v <var>              - Required under M : job variant.
                      O     for distinguishing orientation offsets
                            by instrument.
                            Implies drift subtraction.
                            The ins-file for the merged solution
                            will be urtap-merged-O-ndr.ins
                      O-sdr to solve a drift admittance coefficient
                            instead of subtracting the drift.

                            The ins-file for the merged solution
                            will be urtap-merged-O-sdr.ins

 -d <dvar>             - Required under M : drift variant
                         dsyn dwwf expf ochy
 
                         Both -v and -d are mercyful as to a
                         leading `-´ in the argument. 
 
                         The ins-file for urtapm is urtapm-big-$var.ins
 
 -sel <str>            - Select urtap ins-files (fgrep str ins4mrg.lst)


 EXAMPLES:

            urtap-merge-pp OQILTUR $campaigns

            urtap-merge-pp OQILTUR -s ochy $campaigns

            urtap-merge-pp M -v O-sdr -d dsyn