tsf2ts-decim [options] in-dir pre yymm

       GWR gravity and barometer data conversion from
       ascii 1-sec data to separate ts-files with e.g. 10 min
       sampling interval.
       Output to ascii and/or .ts or .mc
       where ${c} is the column number
       Typically  d/G1_1_090701.dat     (ascii)
                  d/G1_1_090701-600s.ts (bin)
                  d/         ( use -O labels dir/filename )

       OBS! Uses a temporary file ./tmp.dat

       Concatenation of files over midnight boundary
       using, intermediate output to tmp.dat
       each day.
       Low-pass filtering and subsampling, output appending to
       ascii dat file, using tslist
       Conversion ascii to binary ts using tslist.

       Does a check on the existence of the input data files first.

       A tsfedit-control file named deci.tse
and a few other files and routines, cf end of this doc.
PARAMETERS (compulsory):
in-dir     NOTE! If -d or -dm is given without argument, a `- must be
                   inserted if it's the last option before in-dir
  in-dir         - input directory, e.g. RAW_o054/
  pre            - file name prefix (characters before date part),
                   e.g. G1
  yymm           - year and month part of file names, e.g. 0906

  -G             - Process monthly .GGP files.
                   Then, -r
#from[ #to] specify a range of months
                   and last parameter yymm signifies only the year yy
                   Default are daily files.

  -g [hfile]     - Create GGS-files.
                   With -w uses hfile for the header (default: model.ggs).
                   -g must appear before -w, else an initial block
                   separator (7777777) will not be written.
                   Implies -a -d 1, denies -t and -W
                   This is preliminary as tslist writes GGS records
                   currently in fixed format, two data columns,
                   gravity [V] and BaroPress [hPa]

  -t             - do only   the final ascii-to-ts conversion
  -a             - do not do the final ascii-to-ts conversion
                   Thus you can split the job into two tasks.
  -l             - use tsl, not tslist, for the final ascii-to-ts conversion
                   (will produce a log file in ~/tslist-logs/).

  -E f.tse,lab   - (may be repeated) tsfedit options: Call tsfedit at the stage
                   where the tmp.dat file is produced. File f.tse label lab.


  -X ext         - Input file name extension. Default=054. If GGP is
                   to be changed, -X must be given after -G.

                   A mercy-option might be welcome.

  -w             - wipe out pre-existent collecting ascii files
  -W             - wipe out collecting ascii files at end
  -z             - bzip2 the data source

  -k[t][c]       - keep ascii data (t: tmp.dat, c: collecting ascii files)
                   -k is interpreted as -ktc
                   If the c option is not given, the routine prompts for
                   a decision.

  -td dir        - A temporary directory where to write ascii data (tmp.dat)
                   binary data ( and if necessary a tmp.tse tile
                   Default = tmp


  -d #rate[,#move][,label] - decimation. Rate = factor on input sampling interval.
is a label in deci.tse, default label = #rate (!)
                   Exception: -d 1  meaning no decimation.
                   Default = 600 (1s to 10min).
                   Move the first epoch by #move (in sample units of the input
                   file), default=0

  -df file       - File name where the decimation instructions are found,
                   default = ./deci.tse

  -dm #rate,#move  decimation, unprepared tse-file. In this variant,
                   a temporary file tmp.tse is generated
                   (-df is ignored unless it precedes the -dm option).
                   A simple
moving-average filter will be used.
                   Meaning of #rate,#move parameters like under -d.

  -n #ovrl       - next-day overlap, default = auto-detect or 0.
                   The program determines the overlap from the tslist protocol.
                   A value is computed that warrants against gaps in daily output.
                   Depending upon other applications, the overlap can
                   be changed; then, however, it must be specified
                   after the -d option.

                   The program will stop
if there is an overlap problem.

  -T [m|s]dt     - Manual supply of the -r option to tslist (sampling rate).
                   dt = output sampling rate in hours or minutes (m) or seconds (s),
                   to be imposed in order to prevent rounding errors
                   when tslist tries to determine it from ascii data.
                   Default: no -r option issued; let tslist try for itself.
                   Placement of s|m: Can be written as  -T 30s  for example.

  -r #from[ #to] - day range, default = 1 31 of month mm
where mm is taken from the last argument.

                   Default #to = #from.
                   Can also be coded with comma: #from,#to
                   Month boundaries: Don't hesitate to code
                   e.g. -r 25,35 ... 0908
                   to request data from Aug 25 to Sep 04
  -r #from +#n     Alternative: start with day #from, work through #n days.

  -I  -i -ii       Ignore missing input files. A data gap and/or
                   a premature file ending may ensue.
                -I will make the program stop at the first missing file.
                   A suggestion will be printed.
                -i program tests for all missing-file but will stop
                   before processing.
               -ii will let the program pass all missing-file tests
                   and process the eventually incomplete set.
                   Default behaviour is -I .

  -c #n1,[#n2..] - column numbers to process. Default = 1,3
                   (gravity and barometer in G1-files).
                   A consecutive series of columns may be coded
                   as #na-#nz
The two forms can be mixed.
-c name1,[name2..]                 Alternate form:
                   Name is a channel header. Useful when column assignment
                   in the files changes over time. We have no tool as of yet
                   to assist in the coding. Names must be given one for each
                   channel, like in  -c TX-Pwr,TX-Bal,

  -c help        - will print the channels and exit. Minimum parameters must
                   specify the day (by -r #d), the file directory (indir/),
                   the file type (pre = letter+number like A2 or G1),
                   and year+month (yymm):
-decim -r #from -c help  indir/  pre  yymm
                   If there is no file for the day following, add option -d 1
  tsf2ts-decim -d 1 -r #from -c help  indir/  pre  yymm
                   You can also use
                     xchan-units -c {indir}/{pre}yymmdd.054

  -u ...         - like -c, but with units conversion in xchan-units
  -U ... string  - like -u, but with user's conversion factors, see
                   xchan-units, -C option, how to code this.

             NOTE! with -u and -c you cannot duplicate channels, one
                   with and one without conversion. The converted
                   channels will overwrite the direct ones.
                   Run tst2ts-decim twice and use different destination

  -o outdir      - output directory. Default = .

                   outdir may be a subdirectory name or
                   subdir/filename ; only one level is allowed!
                   A `## in the filename will be replaced by the
                   column number (c.f. -c option)
  -of outdir dn  - like -o, dn is the date part of the output file name
                   (instead of the default yymm#from)
                   Use this for append, but then without the -w option!

  -ON            - Use channel names for output file name part e.g.
The channel name is retrieved
                   from the input source.

  -O 'label label ...' outdir/[fnh]

                   for labelled output. Labels in the order of
                   retrieved columns,
                   -c columns first, -u columns last, see below.
                   outdir may be a subdirectory name or
                   subdir/filename-head ; only one level is allowed!
                   Example for this:
                   tsf2ts-deci -W -w -c Dewr-P,Dwr-Htr -O 'DWPRES,VAL DWRHEAT,VAL' d/A2_dewr_ \
                               -d 1 -r 29 29 RAW_o054/ A2 1507

It would be nice to have an auto-label option, using the channel names!

  -oo option     - (may be repeated) Additional options to the final call to
                   tslist, producing the bin ts files.
                   Useful e.g.
                   -oo -BHc2010,06,27,23,01,15 -oo -U2010,06,30,15,16,15

                   to cut the beginning and the end.

           For specifying the last three parameters on the call line, file
           names must be deconstructed according to the following model:

              tsf2ts-decim ... -r range_of_days ... in-dir pre yymm 
           where yymmdd is computed with
              jdc -A$d -FS -fs ${yymm}00
           foreach d ( `fromto $range_of_days[1] $range_of_days[2]` )

           to  ${outdir}/${pre}
_${c}_${yymmdd}-${rate}s.ts   (bin)
           and ${outdir}/${pre}_${c}_${yymmdd}.dat           (ascii)
           where ${c} is the column number and dd the first day

REQUIRES:  /home/hgs/bin/ beak  fromto  jdc  xchan-units  xggp  tslist  ./deci.tse

TEMPFILES: ./tmp.dat   (still hard-wired)

(1)        tsf2ts-decim -w -r 11,30 -d 600,60 -o d/ RAW_o054/ G1 0906
       reads     RAW_o054/G1090611.054 through RAW_o054/G1090630.054
       generates G110906.dat G110906-600s.ts (gravity)
                 G130906.dat G130906-600s.ts (barometer)
       Data from the day range 11 through 30 of 0906 (June 2009) is
       collected in one file for each channel (1 and 3 by default)

       For a really long series, ascii output only:
           tsf2ts-decim -r 1,1415 -w -W -a -d 600,60 -o d/ RAW_o054/ G1 0907
       To add at the end, remove the
-w and -W options and add -kc .

           tsf2ts-decim -W -w -r 15,92
-d 3600,3272 -n 40000 -o tmp/ RAW_o054/ G1 0906
       creates 1h samples at minute/second 0/0 proper from mid June to end of August 2009.
       Saves results in tmp/ G1_1_090615-3600s.ts [V] and G1_3_090615-3600s.ts [hPa]. 

(2)        tsf2ts-decim -r 11,30 -w -a -o d/ RAW_o054/ G1 0906
           tsf2ts-decim -r 1,5   -w -a -o d/ RAW_o054/ G1 0907
           foreach c ( 1 3 )
             cat d/G1_${c}_0907.dat >> d/G1_${c}_0906.dat
           tsf2ts-decim -t -o d/ RAW_o054/ G1 0906
       concatenates across a month boundary. This model may be
       useful to introduce gaps on purpose. Else: Sorry for the

       inconvenience of the foreach loop. But you know that we can
specify exceedingly large ending-day numbers:

tsf2ts-decim -r 11,35 -o d/ RAW_o054/ G1 0906
       and far beyond!

tsf2ts-decim -r 1,1416 -o d/ -d 600,0,NEW600 RAW_o054/ G1 0907
for obtaining 10-min data up to May 15,2013 with the new decimation scheme.
       This will be boringly slow, prefer to use the script mc4tideanalysis.

(3)        tsf2ts-decim -w -W -u 13,14 -r 1,26 -ii -d 600,60 -o d/ RAW_o054/ A2 1310
       prepares Neck-T1 and Neck-T2 (downsampled to 600s) for October 2013,
       with an incomplete last day (option -ii). Output units = Kelvin.

(4)        tsf2ts-decim -r 5,6 -c 1 -o cal/ -a -w -d 13,288 RAW_o054/ G1 0907

       prepares a series suitable to compare with the FG5 for
       calibration. The decimation scheme uses a 25-samples moving
       average, the output series is subsampled at 600 s.

(5)        tsf2ts-decim -c help -d 1 RAW_o054/ A2 0907

       gives a list of data columns of file RAW_o054/A2090701.054

(6)        tsf2ts-decim -G -a -w -W -r 6,14 -c 1,3 -d 60,0 -o tst/ MON_o054/ GW 09

       retrieves gravity and barometer from monthly GGP files (1-min sampling)
       starting in June 2009 and ending with February 2010 (-r 6,14),
       reduces sampling interval to 60-min, and leaves when tst/GW_?_0906.dat
       have been produced. The script protocol ends with a suggestion:

             tslist tst/GW_3_0906.dat -A -gI5,2i3,1x,3i3,t37,e19.0 \
                    -k3 -I  \
                    -o tst/GW_3_0906-60m.ts
             would have been issued had the -a option not been given.

        If this command line is augmented with -Bc2009,6,15 the
        resulting ts-file will be gap-free!

(7)        tsf2ts-decim -g -w -r 28 59 -d 1 -o GGS/ RAW_o054/ G1 1002

        creates a 1-s GGS file for Feb 28 through Mar 31.
        -w generates the header from ./model.ggs

(8)        tsf2ts-decim -w -d 60 -c Dwr-Htr -r 1 370 -o d/ RAW_o054/ A1 1101

        One year of Dewar Heater signal downsampled to 1 h across the computer swap
        in December 2011. 

(9.1)      tsf2ts-decim -W -w -c G1-Sig,TX-Pwr,TY-Pwr \
             -O 'GRAV,VAL TXPWR,VAL TYPWR,VAL' \
                 d/ -d 10 -r 07 07 \
             RAW_o054/ A2 1212

        One day of gravity and tilt, downsampled to 10 s

(9.2)      tsf2ts-decim -a -kc -c TX-Pwr,TY-Pwr -d 60 -r 1 1670 -o d/ RAW_o054/ A1 0907
           tail -1 d/A1_TX-Pwr_090701.dat
           2014 01 26  02 00 00   56683.083333   6.3472992843E+00
           jdc -A-`jdc -A-1 -d 2009 07 01` -di `tail -1 d/A1_TX-Pwr_090701.dat`
tsf2ts-decim -a -kc -c TX-Pwr,TY-Pwr -d 60 -r 1671 1671 -o d/ RAW_o054/ A1 0907
        Collects first all tilt data into an ascii file, then adds another day.
        Decimates to 1-h. Finally, make MC-files, cleaned for outliers
           setenv DELLIMIT 1.0
           tslist d/A1_TX-Pwr_090701.dat -gi4,2i3,i4,t36,d19.0 -k1 -I -Etiltclean.tse,C -O:`label TXPOW,VAL` d/
           tslist d/A1_TY-Pwr_090701.dat -gi4,2i3,i4,t36,d19.0 -k1 -I -Etiltclean.tse,C -O:`label TYPOW,VAL` d/

(9.2)      xchan-units -c RAW_o054/A1120101.054
           tsf2ts-decim -w -W -r 1 366 -d 1,0 -u 34 -o d/ RAW_o054/ A1 1201
Channel 34 is Temp-G2

(9.3)      tsf2ts-decim -c help RAW_o054/ A1 1002

(10)    A useful loop to obtain one month worth of daily mc-files 1 Sps, here for July 2018 and the two tilt components:
           foreach d ( `fromto 1 31` )
             tsf2ts-decim -w -W -r $d $d -d 1 -u 11,18 -O 'TX-Pwr TY-Pwr' d/T2
RAW_o054/ A2 1807
        If you need more than one month's data, you must code an outer loop for it.

            tsf2ts-decim -w -W -r $d,$d -u Dewr-P,Dwr-Htr -d 1 -o d/DW## -ON RAW_o054/ A2 1810
        will generate two ts-files with the respective channel names,
        d/DW_Dewr-P_181006-1s.ts and d/DW_Dwr-Htr_181006-1s.ts

        ... and actually (181005):
            tsf2ts-decim -w -W -r $d,$d -u Dewr-P,Dwr-Htr -d 1 -O 'Dewr-P Dwr-Htr' d/TPC2_ RAW_o054/ A2 1810