USAGE:
tsf2ts-decim [options] in-dir pre yymm
PURPOSE:
GWR gravity and barometer data conversion from
ascii 1-sec data to separate ts-files with e.g. 10 min
sampling interval.
Output to ascii and/or .ts or
.mc
${outdir}/${pre}_${c}_${yymmdd}-${deci}u.ext
where ${c} is the column
number
Typically
d/G1_1_090701.dat (ascii)
d/G1_1_090701-600s.ts (bin)
d/whatever.mc ( use -O labels
dir/filename )
OBS! Uses a temporary file ./tmp.dat
Does:
Concatenation of files over midnight boundary
using xchan-units.pl,
intermediate output to tmp.dat
each day.
Low-pass filtering and subsampling, output appending to
ascii dat file, using tslist
Conversion ascii to binary ts using tslist.
Does a check on the existence of
the input data files first.
Needs:
A tsfedit-control file named
deci.tse
and a few other
files and routines, cf end of this doc.
PARAMETERS
(compulsory):
in-dir NOTE! If -d or -dm is given without argument,
a `-´ must be
inserted if it's the last option before in-dir
in-dir
- input directory, e.g. RAW_o054/
pre
- file name prefix (characters before date
part),
e.g. G1
yymm
- year and month part of file names, e.g. 0906
OPTIONS:
-G
- Process monthly .GGP files.
Then, -r #from[ #to] specify a range of
months
and last parameter yymm
signifies only the year yy
Default are daily files.
-g [hfile]
- Create GGS-files.
With -w uses hfile for the header
(default: model.ggs).
-g must appear before -w, else an initial block
separator (7777777) will not be written.
Implies -a -d 1, denies -t and -W
This is preliminary as
tslist writes GGS records
currently in fixed format, two data columns,
gravity [V] and BaroPress [hPa]
-t
- do only the final ascii-to-ts conversion
-a
- do not do the final ascii-to-ts conversion
Thus you can split the job into two tasks.
-l
- use tsl, not tslist, for the final ascii-to-ts conversion
(will produce a log file in ~/tslist-logs/).
-E f.tse,lab - (may be
repeated) tsfedit options: Call tsfedit at the stage
where the tmp.dat file is produced. File f.tse label lab.
FILES:
-X ext
- Input file name extension. Default=054. If GGP
is
to be changed, -X
must be given after -G.
A mercy-option might be welcome.
-w
- wipe out pre-existent collecting ascii files
-W
- wipe out collecting ascii files at end
-z
- bzip2 the data source
-k[t][c]
- keep ascii data (t:
tmp.dat, c: collecting
ascii files)
-k is interpreted as -ktc
If the c option is not
given, the routine prompts for
a decision.
-td dir
- A temporary directory where to write ascii data (tmp.dat)
binary data (tmp.mc) and if necessary a tmp.tse tile
Default = tmp
DECIMATION:
-d #rate[,#move][,label] - decimation. Rate =
factor on input sampling interval.
label is a label in deci.tse, default label = #rate (!)
Exception: -d 1
meaning no decimation.
Default = 600 (1s to 10min).
Move the first epoch by #move
(in sample units of the input
file), default=0
-df file - File
name where the decimation instructions are found,
default = ./deci.tse
-dm #rate,#move decimation,
unprepared tse-file. In this variant,
a temporary file tmp.tse is generated
(-df
is
ignored
unless
it
precedes
the
-dm
option).
A simple moving-average
filter
will
be used.
Meaning of #rate,#move parameters like under -d.
-n #ovrl -
next-day overlap, default = auto-detect or 0.
The program determines the overlap from the tslist protocol.
A value is computed that warrants against gaps in daily output.
Depending upon other applications, the overlap can
be changed; then, however, it must be specified
after the -d option.
The program will stop
if there is an overlap problem.
-T [m|s]dt
- Manual supply of the -r option
to tslist (sampling rate).
dt = output sampling
rate in hours or minutes (m)
or seconds (s),
to be imposed in order to prevent rounding errors
when tslist tries to determine it from ascii data.
Default: no -r option issued; let tslist try for itself.
Placement of s|m: Can be written as -T 30s for example.
TEMPORAL EXTENT:
-r #from[ #to] - day range, default = 1 31 of
month mm
where mm is
taken from the last argument.
Default #to = #from.
Can also be coded with comma: #from,#to
Month boundaries: Don't hesitate to code
e.g. -r 25,35 ... 0908
to request data from Aug 25 to Sep 04
-r #from +#n Alternative: start
with day #from, work
through #n days.
INPUT / OUTPUT:
-I -i -ii
Ignore missing input files. A data
gap and/or
a premature file ending may ensue.
-I will make the program stop at the first missing file.
A suggestion will be printed.
-i program tests for all
missing-file but will stop
before processing.
-ii will let the program
pass all missing-file tests
and process the eventually incomplete set.
Default behaviour is -I .
-c #n1,[#n2..] - column numbers to process. Default =
1,3
(gravity and barometer in G1-files).
A consecutive series of columns may be coded
as #na-#nz
The two forms can be mixed.
-c name1,[name2..]
Alternate form:
Name is a channel header. Useful when column assignment
in the files changes over time. We have no tool as of yet
to assist in the coding. Names must be given one for each
channel, like in -c
TX-Pwr,TX-Bal,TY-Pwr,TY-Bal
-c help
- will print the channels and exit. Minimum parameters must
specify the day (by -r #d), the file directory (indir/),
the file type (pre = letter+number like A2
or G1),
and year+month (yymm):
tsf2ts-decim -r #from -c help indir/ pre yymm
If there is no file for the day following, add option -d 1
tsf2ts-decim -d 1
-r #from -c help indir/ pre yymm
You can also use
xchan-units -c {indir}/{pre}yymmdd.054
-u
... - like -c, but with units
conversion in xchan-units
-U ... string - like -u, but with user's
conversion factors, see
xchan-units,
-C option, how to code
this.
NOTE! with -u
and -c you cannot duplicate channels, one
with and one without conversion. The converted
channels will overwrite the direct ones.
Run tst2ts-decim twice and use
different destination
directories.
-o outdir
- output directory. Default = .
outdir may be a
subdirectory name or
subdir/filename ; only one level is allowed!
A `##´ in the filename
will be replaced by the
column number (c.f. -c option)
-of outdir dn - like -o, dn is the date part of the
output file name
(instead
of
the
default yymm#from)
Use this for append, but then
without the -w option!
-ON
- Use channel names for output file name part e.g.
A1_T3-Ext_yymmdd-rr.ts. The channel
name is retrieved
from the input source.
-O 'label label ...'
outdir/[fnh]
for labelled output. Labels in the order of
retrieved columns,
-c columns first, -u columns last, see below.
outdir may be a
subdirectory name or
subdir/filename-head ; only one level is
allowed!
Example for this:
tsf2ts-deci -W -w -c Dewr-P,Dwr-Htr -O
'DWPRES,VAL DWRHEAT,VAL' d/A2_dewr_ \
-d 1 -r 29 29 RAW_o054/ A2 1507
-L
- In the case of MC-output (-O), add labelled columns
(default: begin a new mc-file).
It
would be nice to have an auto-label option, using the channel names!
MISCELLANEOUS:
-oo option
- (may be repeated) Additional options to the final call to
tslist, producing the bin ts files.
Useful e.g.
-oo -BHc2010,06,27,23,01,15 -oo -U2010,06,30,15,16,15
to cut the beginning and the end.
INPUT:
For
specifying the last three parameters on the call line, file
names
must be deconstructed according to the following model:
${in-dir}/${pre}$yymmdd.054
tsf2ts-decim ... -r range_of_days ... in-dir pre
yymm
where
yymmdd is computed with
jdc -A$d -FS -fs ${yymm}00
foreach d ( `fromto $range_of_days[1] $range_of_days[2]` )
OUTPUT:
to ${outdir}/${pre}_${c}_${yymmdd}-${rate}s.ts
(bin)
and ${outdir}/${pre}_${c}_${yymmdd}.dat
(ascii)
where ${c} is the column number and dd the first
day
REQUIRES:
/home/hgs/bin/ beak fromto jdc xchan-units
xggp tslist ./deci.tse
TEMPFILES:
./tmp.dat (still hard-wired)
EXAMPLES:
(1)
tsf2ts-decim -w -r 11,30 -d 600,60 -o d/ RAW_o054/ G1 0906
reads RAW_o054/G1090611.054 through
RAW_o054/G1090630.054
generates G110906.dat G110906-600s.ts (gravity)
G130906.dat G130906-600s.ts (barometer)
Data
from
the
day
range
11
through
30
of
0906
(June
2009) is
collected
in
one
file
for
each
channel
(1
and
3
by
default)
For a really long series,
ascii output only:
tsf2ts-decim -r 1,1415 -w -W -a -d 600,60 -o d/ RAW_o054/ G1 0907
To add at the end, remove the
-w and -W options and add
-kc .
tsf2ts-decim
-W -w -r 15,92 -d 3600,3272 -n
40000 -o tmp/ RAW_o054/ G1 0906
creates 1h samples at
minute/second 0/0 proper from mid June to end of August 2009.
Saves results in tmp/
G1_1_090615-3600s.ts [V] and G1_3_090615-3600s.ts [hPa].
(2)
tsf2ts-decim -r 11,30 -w -a -o d/ RAW_o054/ G1 0906
tsf2ts-decim -r 1,5 -w -a -o d/ RAW_o054/ G1 0907
foreach c ( 1 3 )
cat d/G1_${c}_0907.dat >> d/G1_${c}_0906.dat
end
tsf2ts-decim -t -o d/ RAW_o054/ G1 0906
concatenates across a month boundary. This model may be
useful to introduce gaps on purpose.
Else: Sorry for the
inconvenience
of
the
foreach
loop.
But
you
know
that
we
can
specify exceedingly large ending-day
numbers:
tsf2ts-decim -r 11,35 -o d/
RAW_o054/ G1 0906
and far beyond!
tsf2ts-decim -r 1,1416 -o d/ -d
600,0,NEW600 RAW_o054/ G1 0907
for obtaining 10-min
data up to May 15,2013 with the new decimation scheme.
This will be boringly slow,
prefer to use the script mc4tideanalysis.
(3) tsf2ts-decim
-w -W -u 13,14 -r 1,26 -ii -d 600,60 -o d/ RAW_o054/ A2 1310
prepares Neck-T1 and Neck-T2
(downsampled to 600s) for October 2013,
with an incomplete last day
(option -ii). Output units = Kelvin.
(4)
tsf2ts-decim -r 5,6 -c 1 -o cal/ -a -w -d 13,288 RAW_o054/ G1
0907
prepares a series suitable to compare with the FG5 for
calibration. The decimation scheme uses a 25-samples moving
average, the output series is subsampled at 600 s.
(5)
tsf2ts-decim -c help -d 1 RAW_o054/ A2 0907
gives a list of data columns
of file RAW_o054/A2090701.054
(6) tsf2ts-decim -G
-a -w -W -r 6,14 -c 1,3 -d 60,0 -o tst/ MON_o054/ GW 09
retrieves gravity and
barometer from monthly GGP files (1-min sampling)
starting in June 2009 and
ending with February 2010 (-r
6,14),
reduces sampling interval to
60-min, and leaves when tst/GW_?_0906.dat
have been produced. The script
protocol ends with a suggestion:
tslist tst/GW_3_0906.dat -A
-gI5,2i3,1x,3i3,t37,e19.0 \
-k3 -I \
-o
tst/GW_3_0906-60m.ts
would
have
been
issued
had
the
-a
option
not
been
given.
If this command line is augmented with -Bc2009,6,15 the
resulting ts-file
will be gap-free!
(7) tsf2ts-decim
-g -w -r 28 59 -d 1 -o GGS/ RAW_o054/ G1 1002
creates a 1-s GGS
file for Feb 28 through Mar 31.
-w generates the
header from ./model.ggs
(8) tsf2ts-decim
-w -d 60 -c Dwr-Htr -r 1 370 -o d/ RAW_o054/ A1 1101
One year of Dewar
Heater signal downsampled to 1 h across the computer swap
in December
2011.
(9.1) tsf2ts-decim -W -w -c
G1-Sig,TX-Pwr,TY-Pwr \
-O 'GRAV,VAL TXPWR,VAL TYPWR,VAL' \
d/A2_gxy_120707-10s.mc -d 10 -r 07 07 \
RAW_o054/ A2 1212
One day of gravity
and tilt, downsampled to 10 s
(9.2) tsf2ts-decim -a -kc -c
TX-Pwr,TY-Pwr -d 60 -r 1 1670 -o d/ RAW_o054/ A1 0907
tail -1 d/A1_TX-Pwr_090701.dat
2014 01 26 02 00 00
56683.083333 6.3472992843E+00
jdc
-A-`jdc -A-1 -d 2009 07 01` -di `tail -1
d/A1_TX-Pwr_090701.dat`
01671
tsf2ts-decim
-a -kc -c TX-Pwr,TY-Pwr -d 60 -r 1671 1671 -o d/
RAW_o054/ A1 0907
Collects first all tilt data into an ascii file, then adds
another day.
Decimates to 1-h.
Finally, make MC-files, cleaned for outliers
setenv
DELLIMIT 1.0
tslist
d/A1_TX-Pwr_090701.dat -gi4,2i3,i4,t36,d19.0 -k1 -I
-Etiltclean.tse,C -O:`label TXPOW,VAL`
d/A1_t_090701-140126-1h.mc
tslist d/A1_TY-Pwr_090701.dat -gi4,2i3,i4,t36,d19.0 -k1 -I
-Etiltclean.tse,C -O:`label TYPOW,VAL`
d/A1_t_090701-140126-1h.mc
(9.2)
xchan-units -c
RAW_o054/A1120101.054
tsf2ts-decim -w -W -r 1 366 -d 1,0 -u 34 -o d/ RAW_o054/ A1
1201
Channel 34
is Temp-G2
(9.3) tsf2ts-decim -c help
RAW_o054/ A1 1002
(10) A useful loop to obtain one month
worth of daily mc-files 1 Sps, here for July 2018 and the two
tilt components:
foreach d
( `fromto 1 31` )
tsf2ts-decim -w -W -r $d $d -d 1 -u
11,18 -O 'TX-Pwr TY-Pwr' d/T2 RAW_o054/
A2 1807
end
If you need
more than one month's data, you must code an outer loop for
it.
tsf2ts-decim -w -W -r $d,$d -u Dewr-P,Dwr-Htr -d 1 -o
d/DW## -ON RAW_o054/ A2 1810
will generate two
ts-files with the respective channel names,
d/DW_Dewr-P_181006-1s.ts and
d/DW_Dwr-Htr_181006-1s.ts
... and actually (181005):
tsf2ts-decim -w -W -r $d,$d -u Dewr-P,Dwr-Htr -d 1 -O 'Dewr-P
Dwr-Htr' d/TPC2_ RAW_o054/ A2 1810
.bye