Date   
Re: file_delivery_speed

Ulrich G. Kliegis
 

An: MSG-1@...
Von: "ernstlobsiger" <ernst.lobsiger@...>
Datum: Mon, 07 Mar 2011 19:24:53 -0000
Betreff: [MSG-1] Re: file_delivery_speed
Antwort an: MSG-1@...

It might well be that you have never seen quite a number of parameters
that can actually be set either in recv.ini others in
recev-channels.ini. Nevertheless these parameters exist and may assume
default values if they don't show up in the *.ini files.
For the last hour or so, I've asked myself how I could receive anything
without seeing the obviously invisible - namely the missed data... Punned
enough, Ernst, your students must be foll of good words about your didactic
capabilities :)

I studied the parameter description on the tellicast webshell (admittedly) for
the first time (my aversion against pink?), and I can only recommend to read
all paragraphs about tmp_directory, tmp-suffix and temp-prefix and their
function. In hardware, these *fixes would be called handshakelines, their
function is to make sure the data have arrived completely at their destination
before the next software instance grabs them. And since this is single-
sewed, there is this additional parameter called delivery speed (which surely
will not be a wait loop to be counted down before transferring the next byte,
but rather an estimator how long a flag data ready should be delayed after
the transfer aout of any buffer is started. A somewhat strnage philolsophy,
but if one considers the origins of Deutsche Telekom who were the
company mother of T-Systems before they sold it to Spain, their origin was
Deutsche Bundespost, who were often compared to tournament snails at
their time. Oh well.

I set the parameter to 5.000.000 now - which is still faster than the average
over-all-transfer rate . The missed packets rates have varied between 0 and
31 since / per hour), all recovered. At the same time, I moved the tmp_dirs
back to where their respective target_dirs are located. I'll try to lower the rate
a bit more now and then will try to add the excluded channels again one by
one - just to see where the limits are.

Ernst, really herzlichen Dank for the time being, keeping my fingers crossed,
or, as we say, pressing my thumbs!

Cheers,
U.

Re: file_delivery_speed

ernstlobsiger
 

--- In MSG-1@..., Alan Sewards <alan@...> wrote:

Hi, I have followed this thread with great interest
even though I don't run MRTG and don't intend to.

I have never seen an entry file_delivery_speed in either recv.ini
or recv-channels.ini. What would be the point of it?
Hello Alan

It might well be that you have never seen quite a number of parameters that can actually be set either in recv.ini others in recev-channels.ini. Nevertheless these parameters exist and may assume default values if they don't show up in the *.ini files.

e.g.

file_delivery_speed=20000000 (default is 10000000)
file_delivery_counter=50 (default is 25 as far as I remember)

Both in recv.ini might be such parameters. When I set them to these numbers (commented out in the distribution recv.ini) my old GNU/Linux boxes just fade away without any log entry at all! I found that I can increase either one parameter and let the other commented (default).
The point here ist to get the files as fast as possible from segments in FSY on the RAM disk into real data files on the hard disk(s).

To find all parameters possible and exhaustive explanations go to the TelliCast WEB Interface of your receiver and look under Menu Help, File Formats ...




- Alan Sewards says he still uses 2.4.4a with "temp_directory"
- David Taylor says your client 2.4.4B is older than 2.4.4a.
I think it likely that tc.recv can handle the different
tmp/temp when parsingthe file lines - it would be a
common-sense thing to build in when you had made a minor
version change to a widely-used program.
Yes it might well be the case and common-sense that "temp_directory" and "tmp_directory" are both accepted by the TelliCast Client. As I only started EUMETCast reception one year ago I didn't know about the "temp" variant until now. Windows also has %temp% and %tmp% system variables but these may mean different things ...


Best regards - Alan S
Cheers
Ernst

Re: New to Metop

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

David

Inspite of a great deal of help form Alan and others, for which I am very
grateful,I am still in deep trouble with Metop.

I am wondering if it would be best to uninstall every thing including
T-Systems and start again from scratch.

My trail time runs out tomorrow and I wondered if it would be possible to
have it extended ?

regards...........Geoffrey
Geoffrey,

I'm sorry, but the trial period lasts 30 days, fixed. All I can suggest is that you order the software, and continue to try and get the receive side working. Many people are receiving Metop successfully, so it should be possible for you as well. You can order here:

http://www.satsignal.eu/software/reg-metop-manager.html

If there ends up being a genuine problem with my software, I'm sure we can sort something out.

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

New to Metop

geoffreygee90
 

David

Inspite of a great deal of help form Alan and others, for which I am very grateful,I am still in deep trouble with Metop.

I am wondering if it would be best to uninstall every thing including T-Systems and start again from scratch.


My trail time runs out tomorrow and I wondered if it would be possible to have it extended ?

regards...........Geoffrey

Re: not yet perfect, but...

Ulrich G. Kliegis
 

An: <MSG-1@...>
Von: "David J Taylor" <gm8arv@...>
Datum: Mon, 7 Mar 2011 15:28:56 -0000
Betreff: Re: [MSG-1] Re: not yet perfect, but...
Antwort an: MSG-1@...

Your losses now seem to be back to near what they were on Thursday -
before we started all this!
Yes, but then EPS 5 and 11 were not yet commented off.

The 26 minutes rhythm? no idea, no clue? Let me observe that a bit longer.

My audiovisual (disk noise, colored pixels all over the place!) impression is
still that it has something to do with NOAA and / or METOP data bursting in.

What are te housekeeping timings of your programs, purging old files, etc.?

The only service I have running except the satellite stuff here is the ftp
upload (every 10 mins) - and the "viewing PC" fetching data during
(expanded) office hours. You see the silent ethernet phase at night-
obviously no influence on the missed packets incidence.

I now also switched off screen protection (which just switched off the video
signal after 20 mins).

And I also deactivated the only remaining indexing for drive C:

Cheers,
U.

Re: not yet perfect, but...

Ulrich G. Kliegis
 

An: MSG-1@...
Von: "ernstlobsiger" <ernst.lobsiger@...>
Datum: Mon, 07 Mar 2011 16:56:43 -0000
Betreff: [MSG-1] Re: not yet perfect, but...
Antwort an: MSG-1@...

Ernst,

Here my last hints. As a teacher you don't only have this sharp eye
but you ask again, if you feel your questions have not been answered.
and a good teacher, if I may say so, adds some additional information to
make the student capable to understand more of the foggy world around...

A good example follows:

in FSY) it is written out to the "tmp_directory" with the speed you
have chosen with "file_delivery_speed". Be sure to set this to
10000000 - 20000000 as discussed before. On the other hand 20E6
Bytes/s is certainly slow for a well working RAM drive.
I was not really aware of the function of this parameter - although I
remember now to have heard about it before...
So, there is still ample head- or rather footroom to experiment a bit. I set it to
20.000.000 right now (res. yesterday)


So it still might be worth looking up in your WEB interface what
they propose under Help, File Formats whether "tmp" or "temp".
(Maybe you have done that and I just didn't get it again ...)
They fill up and empty, so my conclusion is, that they work.

I pushed your recv.log through TClogSummary. The problem is not only
EPS but all sorts of channels ... There is still your fast growing FSY
file. This may either be caused by not being emptied as expected (Hard
Disks) or by not being filled like expected (lots of missing
packets/segments caused by the DVB card, TelliCast Client is waiting
for files to complete and keeps filling in other packets, see below).


===============================================
TClogSummary Version 0.91

Windows EUMETCast receiver WINDBUEDEL
Windows TelliCast Client Version 2.4.4.B

TelliCast Client Event Summary from 2011-03-07
Head time 13:59:56.4 -UTC- Tail time 15:20:11.3

A CMD script by Ernst Lobsiger
With ideas from Arne van Belle
===============================================
Number of messages of category ERR: 0
===============================================
Number of messages of category WRN: 0
===============================================
Number of messages of category MSG: 3611
===============================================
Number of *Starting* (what) events: 6
MSG:2011-03-07 13:59:56.412:Watchdog starting... [1832]
MSG:2011-03-07 13:59:56.582:Starting new child...
MSG:2011-03-07 13:59:56.763:tc-recv.exe starting... [2108]
MSG:2011-03-07 14:28:34.292:Watchdog starting... [2124]
MSG:2011-03-07 14:28:36.085:Starting new child...
MSG:2011-03-07 14:28:38.068:tc-recv.exe starting... [2392]
===============================================
Number of *Missed* (file) messages: 17
MSG:2011-03-07 14:00:21.678:Missed file
`sat01-noa_k4_cenoa_p_000_000-1103071338-ofl---tiff' id
4d74e4ef002171a7 from channel `DWDSAT' MSG:2011-03-07
14:00:21.678:Missed file
`sat01-noa_k1_cenoa_p_000_000-1103071338-ofl---tiff' id
4d74e4ef002171a8 from channel `DWDSAT' MSG:2011-03-07
14:00:21.678:Missed parts of file
`sat01-noa_k4_eunoa_p_000_000-1103071338-ofl---tiff' id
4d74e4ef002171a9 from channel `DWDSAT' MSG:2011-03-07
14:06:23.028:Missed file `thin_MOD03.P2011066.1255.hdf' id
4d74e41900216cb1 from channel `EUMETSAT Data Channel 4' MSG:2011-03-07
14:06:23.028:Missed parts of file `thin_MOD03.P2011066.1240.hdf' id
4d74e41900216cb2 from channel `EUMETSAT Data Channel 4' MSG:2011-03-07
14:13:49.670:Missed parts of file
`mhs_20110307_123951_metopa_22728_eps_o.l1_bufr' id 4d74e74d002181bb
from channel `EPS-8' MSG:2011-03-07 14:26:43.333:Missed parts of file
`H-000-MSG1__-MSG1_RSS____-IR_120___-000006___-201103071425-C_' id
4d74eb2e00219947 from channel `EUMETSAT Data Channel 5' MSG:2011-03-07
14:26:58.444:Missed parts of file
`H-000-MSG2__-MSG2________-IR_134___-000007___-201103071415-C_' id
4d74eb2b00219926 from channel `EUMETSAT Data Channel 2' MSG:2011-03-07
14:47:02.596:Missed parts of file `thin_MOD021KM.P2011066.1255.hdf' id
4d74eb3100219965 from channel `EUMETSAT Data Channel 4' MSG:2011-03-07
15:13:34.014:Missed parts of file
`H-000-MSG1__-MSG1_RSS____-VIS008___-000007___-201103071510-C_' id
4d74f6350021e3d2 from channel `EUMETSAT Data Channel 5' MSG:2011-03-07
15:13:34.014:Missed file
`H-000-MSG1__-MSG1_RSS____-WV_062___-000007___-201103071510-C_' id
4d74f6350021e3d3 from channel `EUMETSAT Data Channel 5' MSG:2011-03-07
15:13:34.014:Missed file
`H-000-MSG1__-MSG1_RSS____-WV_073___-000007___-201103071510-C_' id
4d74f6350021e3d4 from channel `EUMETSAT Data Channel 5' MSG:2011-03-07
15:13:34.014:Missed file
`H-000-MSG1__-MSG1_RSS____-HRV______-000021___-201103071510-C_' id
4d74f6350021e3d5 from channel `EUMETSAT Data Channel 5' MSG:2011-03-07
15:13:34.014:Missed file
`H-000-MSG1__-MSG1_RSS____-HRV______-000022___-201103071510-C_' id
4d74f6350021e3d6 from channel `EUMETSAT Data Channel 5' MSG:2011-03-07
15:15:51.762:Missed parts of file
`L-000-MTP___-MET7________-00_7_057E-000004___-201103071530-C_' id
4d74f6140021e2c7 from channel `EUMETSAT Data Channel 3' MSG:2011-03-07
15:15:51.762:Missed file
`L-000-MTP___-MET7________-00_7_057E-000003___-201103071530-C_' id
4d74f6140021e2c8 from channel `EUMETSAT Data Channel 3' MSG:2011-03-07
15:15:51.762:Missed parts of file
`L-000-MTP___-MET7________-11_5_057E-000002___-201103071530-C_' id
4d74f6140021e2c9 from channel `EUMETSAT Data Channel 3'
=============================================== Number of *lost*
messages/channels: 110/3 MSG:2011-03-07 15:13:32.352:Disconnect from
data channel `EUMETSAT Data Channel 5', address 224.223.222.29:3111
completed (channel lost) MSG:2011-03-07 15:13:32.392:Disconnect from
announcement channel `TSL Announcement Channel', address
224.223.222.223:4711 completed (channel lost) MSG:2011-03-07
15:13:34.144:Reconnected to announcement channel `TSL Announcement
Channel', address 224.223.222.223:4711 (channel was lost)
=============================================== Lower priority *ended*
Lists/Files: 8/12 ===============================================
Priority *interrupted* Lists/Files: 64/108
=============================================== Received and delivered
Lists/Files: 991/2364 ===============================================
Treated number of recv Files/Lines: 1/3612
===============================================


5)
Instead of using "tmp_directories" in "recv-channels.ini" at all
you could use an entry in "recv.ini"under "[parameters]"
"tmp_suffix=.tmp". This is from my GNU/Linux version, so please
check that this exists with the same syntax in your Windows Client
(Again check on your TelliCast WEB interface, Help, File Formats,
...).
But only if I collect everything on one disk - which is not given
right now. I am considering to replace the two disks with one large
one, merging te directories also into one. But not yet.
From what I read "tmp_suffix" should work for more than one disk.
Again check the explanation under your TelliCast WEB interface, Help,
File Formats.



As David pointed out we are pretty much where we started.

Further suggestions:

- Is your RAM disk working well?

- Is the SkyStar 2.6D broken (reposition it in the slot or even move
it to another slot). Buy another SkyStar 2.6D on eBay.de for 20 Euros.

- You seem to have 2 SATA drives. If these drives do not run under
AHCI (needs a special driver and cannot easily be changed in the Bios
once the Software is installed), chances are that they "emulate" IDE
in a way that sets us back to the Master/Slave problem. Here is a post
in german that tries to explain this. Using two disks as data disks in
Master/Slave configuration would be less than suboptimal ...

<ZITAT>
...

2. Der AHCI-Modus. Er benötigt andere Treiber, welche erst ab Vista
schon ins OS integriert sind. In diesem Modus stehen alle Sata-II
features wie NCQ sowie die volle Bandbreite zur Verfügung.
My PC runs under XP Pro. I am not aware of the AHCI mode, if it is active or
what. I did at least never do it consciously myself. The SATA disks hang on
a simple DeLock PCI interface. The BIOS does not list them as IDE drives.

Ein gutes Schlusswort!
Indeed. For the time being, there are more options than to be tested in one
evening.

A typical RAM error would probably either lead to a bluescreen or, if the
RAMdisk ist affected, probably to some similar event. But there were
absolute no indications of any malfunctions.

Thanks for your patient assistance!

Cheers,
U.

Re: not yet perfect, but...

Alan Sewards
 

Hi,
I have followed this thread with great interest even though I don't run MRTG and don't intend to.

On 07/03/2011 5:56 PM, ernstlobsiger wrote:
Ulrich
*snip=========

The multiplexed image file packets are stored and numbered on your RAM
disk in the FSY file. If a file is complete (all packets of the file in
FSY) it is written out to the "tmp_directory" with the speed you have
chosen with "file_delivery_speed". Be sure to set this to 10000000 -
20000000 as discussed before. On the other hand 20E6 Bytes/s is
certainly slow for a well working RAM drive.
I have never seen an entry file_delivery_speed in either recv.ini or
recv-channels.ini. What would be the point of it?



> > So here is some points you can follow if not already done long ago:
> >
> > 1)
> > It seems that older TelliCast Clients really used "temp_directory"
> > which is now "tmp_directory".
> Done yesterday.
> > So double check what it is for your
> > TelliCast Client I don't know the version of. Look under the TelliCast
> > WEB interface, Help, File Formats.
> Will look again, from memory: 2.4.4 A - but I'll check.

You did look up the client number. So far so good, BUT:

- Alan Sewards says he still uses 2.4.4a with "temp_directory"
- David Taylor says your client 2.4.4B is older than 2.4.4a.
I think it likely that tc.recv can handle the different tmp/temp when parsing
the file lines - it would be a common-sense thing to build in when you had made a minor version change to a widely-used program.

*snip==========
Best regards - Alan S

--
Alan Sewards
email: alan@...
web site: http://asewards.free.fr

Re: not yet perfect, but...

ernstlobsiger
 

Ulrich

Here my last hints. As a teacher you don't only have this sharp eye but you ask again, if you feel your questions have not been answered. Having said that ...


An: MSG-1@...
Von: "ernstlobsiger" <ernst.lobsiger@...>
Datum: Mon, 07 Mar 2011 12:04:37 -0000
Betreff: [MSG-1] Re: not yet perfect, but...
Antwort an: MSG-1@...

Ernst,
again, thanks for your comments and helpful hints.


Ulrich

I may be wrong but I don't think putting the "tmp_directory" on a RAM
disk does any good. The data must be moved to the hard disk E: or F:
later anyway. This move will then be a full data copy that you
postponed with "tmp_directory" on the RAM disk only.
Well, that's true, the data have to be moved. But what happens in the tmp-
phase? The final blocks are composed from fragments dropping in in a
fuzzy sequence and not necessarily well ordered. Composing them piece by
piece on a hard disk takes - at least if my image of this composition process
is correct - a number of disk accesses. These are the most time consuming
when working with HDDs - I am not sure how a writing cache buffer will
handle such portioned write actions if the file is kept open by the OS.
So, the question is: is the composing part consuming more time and
ressources than the shift of the data from RAM to a physical, classical
HDD? (bypass thought: would the new SSDs be able to handle the job, or
are they not yet ready for the very high troughput?)
The multiplexed image file packets are stored and numbered on your RAM disk in the FSY file. If a file is complete (all packets of the file in FSY) it is written out to the "tmp_directory" with the speed you have chosen with "file_delivery_speed". Be sure to set this to 10000000 - 20000000 as discussed before. On the other hand 20E6 Bytes/s is certainly slow for a well working RAM drive.





So here is some points you can follow if not already done long ago:

1)
It seems that older TelliCast Clients really used "temp_directory"
which is now "tmp_directory".
Done yesterday.
So double check what it is for your
TelliCast Client I don't know the version of. Look under the TelliCast
WEB interface, Help, File Formats.
Will look again, from memory: 2.4.4 A - but I'll check.
You did look up the client number. So far so good, BUT:

- Alan Sewards says he still uses 2.4.4a with "temp_directory"
- David Taylor says your client 2.4.4B is older than 2.4.4a.

So it still might be worth looking up in your WEB interface what
they propose under Help, File Formats whether "tmp" or "temp".
(Maybe you have done that and I just didn't get it again ...)


4)
In recv.ini under [logging] set "log_level=normal" and restart with an
empty FSY (reboot). Then wait some minutes and look what the
"recv.log" file says ...
That's a point! :) Will do that next.
I pushed your recv.log through TClogSummary. The problem is not only EPS but all sorts of channels ... There is still your fast growing FSY file. This may either be caused by not being emptied as expected (Hard Disks) or by not being filled like expected (lots of missing packets/segments caused by the DVB card, TelliCast Client is waiting for files to complete and keeps filling in other packets, see below).


===============================================
TClogSummary Version 0.91

Windows EUMETCast receiver WINDBUEDEL
Windows TelliCast Client Version 2.4.4.B

TelliCast Client Event Summary from 2011-03-07
Head time 13:59:56.4 -UTC- Tail time 15:20:11.3

A CMD script by Ernst Lobsiger
With ideas from Arne van Belle
===============================================
Number of messages of category ERR: 0
===============================================
Number of messages of category WRN: 0
===============================================
Number of messages of category MSG: 3611
===============================================
Number of *Starting* (what) events: 6
MSG:2011-03-07 13:59:56.412:Watchdog starting... [1832]
MSG:2011-03-07 13:59:56.582:Starting new child...
MSG:2011-03-07 13:59:56.763:tc-recv.exe starting... [2108]
MSG:2011-03-07 14:28:34.292:Watchdog starting... [2124]
MSG:2011-03-07 14:28:36.085:Starting new child...
MSG:2011-03-07 14:28:38.068:tc-recv.exe starting... [2392]
===============================================
Number of *Missed* (file) messages: 17
MSG:2011-03-07 14:00:21.678:Missed file `sat01-noa_k4_cenoa_p_000_000-1103071338-ofl---tiff' id 4d74e4ef002171a7 from channel `DWDSAT'
MSG:2011-03-07 14:00:21.678:Missed file `sat01-noa_k1_cenoa_p_000_000-1103071338-ofl---tiff' id 4d74e4ef002171a8 from channel `DWDSAT'
MSG:2011-03-07 14:00:21.678:Missed parts of file `sat01-noa_k4_eunoa_p_000_000-1103071338-ofl---tiff' id 4d74e4ef002171a9 from channel `DWDSAT'
MSG:2011-03-07 14:06:23.028:Missed file `thin_MOD03.P2011066.1255.hdf' id 4d74e41900216cb1 from channel `EUMETSAT Data Channel 4'
MSG:2011-03-07 14:06:23.028:Missed parts of file `thin_MOD03.P2011066.1240.hdf' id 4d74e41900216cb2 from channel `EUMETSAT Data Channel 4'
MSG:2011-03-07 14:13:49.670:Missed parts of file `mhs_20110307_123951_metopa_22728_eps_o.l1_bufr' id 4d74e74d002181bb from channel `EPS-8'
MSG:2011-03-07 14:26:43.333:Missed parts of file `H-000-MSG1__-MSG1_RSS____-IR_120___-000006___-201103071425-C_' id 4d74eb2e00219947 from channel `EUMETSAT Data Channel 5'
MSG:2011-03-07 14:26:58.444:Missed parts of file `H-000-MSG2__-MSG2________-IR_134___-000007___-201103071415-C_' id 4d74eb2b00219926 from channel `EUMETSAT Data Channel 2'
MSG:2011-03-07 14:47:02.596:Missed parts of file `thin_MOD021KM.P2011066.1255.hdf' id 4d74eb3100219965 from channel `EUMETSAT Data Channel 4'
MSG:2011-03-07 15:13:34.014:Missed parts of file `H-000-MSG1__-MSG1_RSS____-VIS008___-000007___-201103071510-C_' id 4d74f6350021e3d2 from channel `EUMETSAT Data Channel 5'
MSG:2011-03-07 15:13:34.014:Missed file `H-000-MSG1__-MSG1_RSS____-WV_062___-000007___-201103071510-C_' id 4d74f6350021e3d3 from channel `EUMETSAT Data Channel 5'
MSG:2011-03-07 15:13:34.014:Missed file `H-000-MSG1__-MSG1_RSS____-WV_073___-000007___-201103071510-C_' id 4d74f6350021e3d4 from channel `EUMETSAT Data Channel 5'
MSG:2011-03-07 15:13:34.014:Missed file `H-000-MSG1__-MSG1_RSS____-HRV______-000021___-201103071510-C_' id 4d74f6350021e3d5 from channel `EUMETSAT Data Channel 5'
MSG:2011-03-07 15:13:34.014:Missed file `H-000-MSG1__-MSG1_RSS____-HRV______-000022___-201103071510-C_' id 4d74f6350021e3d6 from channel `EUMETSAT Data Channel 5'
MSG:2011-03-07 15:15:51.762:Missed parts of file `L-000-MTP___-MET7________-00_7_057E-000004___-201103071530-C_' id 4d74f6140021e2c7 from channel `EUMETSAT Data Channel 3'
MSG:2011-03-07 15:15:51.762:Missed file `L-000-MTP___-MET7________-00_7_057E-000003___-201103071530-C_' id 4d74f6140021e2c8 from channel `EUMETSAT Data Channel 3'
MSG:2011-03-07 15:15:51.762:Missed parts of file `L-000-MTP___-MET7________-11_5_057E-000002___-201103071530-C_' id 4d74f6140021e2c9 from channel `EUMETSAT Data Channel 3'
===============================================
Number of *lost* messages/channels: 110/3
MSG:2011-03-07 15:13:32.352:Disconnect from data channel `EUMETSAT Data Channel 5', address 224.223.222.29:3111 completed (channel lost)
MSG:2011-03-07 15:13:32.392:Disconnect from announcement channel `TSL Announcement Channel', address 224.223.222.223:4711 completed (channel lost)
MSG:2011-03-07 15:13:34.144:Reconnected to announcement channel `TSL Announcement Channel', address 224.223.222.223:4711 (channel was lost)
===============================================
Lower priority *ended* Lists/Files: 8/12
===============================================
Priority *interrupted* Lists/Files: 64/108
===============================================
Received and delivered Lists/Files: 991/2364
===============================================
Treated number of recv Files/Lines: 1/3612
===============================================


5)
Instead of using "tmp_directories" in "recv-channels.ini" at all you
could use an entry in "recv.ini"under "[parameters]"
"tmp_suffix=.tmp". This is from my GNU/Linux version, so please check
that this exists with the same syntax in your Windows Client (Again
check on your TelliCast WEB interface, Help, File Formats, ...).
But only if I collect everything on one disk - which is not given right now. I
am considering to replace the two disks with one large one, merging te
directories also into one. But not yet.
From what I read "tmp_suffix" should work for more than one disk. Again check the explanation under your TelliCast WEB interface, Help, File Formats.



As David pointed out we are pretty much where we started.

Further suggestions:

- Is your RAM disk working well?

- Is the SkyStar 2.6D broken (reposition it in the slot or even move it to another slot). Buy another SkyStar 2.6D on eBay.de for 20 Euros.

- You seem to have 2 SATA drives. If these drives do not run under AHCI (needs a special driver and cannot easily be changed in the Bios once the Software is installed), chances are that they "emulate" IDE in a way that sets us back to the Master/Slave problem. Here is a post in german that tries to explain this. Using two disks as data disks in Master/Slave configuration would be less than suboptimal ...

<ZITAT>

NoxCivi (Profi Full Member)
Bitte schreib deine Konfiguration in deine Signatur. Ohne diese können wir dir nur schwer helfen.

Um es dir aber zu verdeutlichen. Für den Sata-Controller und damit für die Geräte gibt es 3 mögliche Modi.

1. Der emulierte IDE-Modus.
Hier behandelt das OS alle die Sata-Anschlüsse wie IDE-Anschlüsse. Je zwei Sata-Ports werden zu einem IDE-Channel zugewiesen inkl. Master/Slave-Zuweisung. Die Geräte sind auf die Bandbreite von Sata-I beschränkt (1,5GBit/s). In diesem Modus gibt es zudem das IDE-typische Problem, dass immer nur ein Gerät pro Channel angesprochen werden kann. Das kann man weitestgehend vermeiden, wenn man je virtuellem IDE-Kanal nur einen Sata-Port nutzt.

2. Der AHCI-Modus. Er benötigt andere Treiber, welche erst ab Vista schon ins OS integriert sind. In diesem Modus stehen alle Sata-II features wie NCQ sowie die volle Bandbreite zur Verfügung.

3. Der RAID-Modus. Ist prinzipell mit dem AHCI-Modus identisch, aber hier können mehrere HDDs zu einem RAID zusammengeschaltet werden. Nicht eingebundene HDDs bzw. DVD-Laufwerde etc. laufen im AHCI-Modus.

Je nach Board gibt es neben der Intel-Soutbridge noch weitere Controller, die die lila Sata-Ports bedienen. Für diese gibt es zusätzlich die "Disabled"-Funktion. Alles weitere ist wie oben.

Alle Klarheiten beseitigt? ;)

</ZITAT>

Ein gutes Schlusswort!

Cheers
Ernst

Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

So, the question is: is the composing part consuming more time and
ressources than the shift of the data from RAM to a physical, classical
HDD? (bypass thought: would the new SSDs be able to handle the job, or
are they not yet ready for the very high troughput?)
[]
Cheers,
U.
I meant to comment earlier...

SSDs have a limited number of writes, so I would not employ them in a path with lots of writing. Lots of reading suits them very well indeed.

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

The log file is presently mirrored to this URL:
http://eumetmon.rummelecke.de/recv.log

Update every 10 minutes.

Cheers,
U.
Ulrich, I see nothing obvious, but you are getting a bunch of "message lost" about every 26 minutes (well: 1426 and 14:52). Is anything scheduled to run around that interval or time?

Losses do seem to be lower immediately after a restart - there must be a clue there!

Your losses now seem to be back to near what they were on Thursday - before we started all this!

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

So double check what it is for your TelliCast

2.4.4.B (!)

Cheers,
U.
That's fine - 2.4.4 B is slightly earlier than 2.4.4a (don't ask, must be Unix logic!), but both versions have the same core and either is fine.

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

Re: not yet perfect, but...

Ulrich G. Kliegis
 

An: <MSG-1@...>
Von: "David J Taylor" <gm8arv@...>
Datum: Mon, 7 Mar 2011 12:14:44 -0000
Betreff: Re: [MSG-1] Re: not yet perfect, but...
Antwort an: MSG-1@...


Yes, I had been assuming that Ulrich had checked the log file for
obvious errors like; "No space in FSY database", but whether I asked
or not I now forget!
The log file is presently mirrored to this URL:
http://eumetmon.rummelecke.de/recv.log

Update every 10 minutes.

Cheers,
U.

Re: not yet perfect, but...

Ulrich G. Kliegis
 

So double check what it is for your TelliCast

2.4.4.B (!)

Cheers,
U.

Re: not yet perfect, but...

Ulrich G. Kliegis
 

An: MSG-1@...
Von: "ernstlobsiger" <ernst.lobsiger@...>
Datum: Mon, 07 Mar 2011 12:04:37 -0000
Betreff: [MSG-1] Re: not yet perfect, but...
Antwort an: MSG-1@...

Ernst,
again, thanks for your comments and helpful hints.


Ulrich

I may be wrong but I don't think putting the "tmp_directory" on a RAM
disk does any good. The data must be moved to the hard disk E: or F:
later anyway. This move will then be a full data copy that you
postponed with "tmp_directory" on the RAM disk only.
Well, that's true, the data have to be moved. But what happens in the tmp-
phase? The final blocks are composed from fragments dropping in in a
fuzzy sequence and not necessarily well ordered. Composing them piece by
piece on a hard disk takes - at least if my image of this composition process
is correct - a number of disk accesses. These are the most time consuming
when working with HDDs - I am not sure how a writing cache buffer will
handle such portioned write actions if the file is kept open by the OS.
So, the question is: is the composing part consuming more time and
ressources than the shift of the data from RAM to a physical, classical
HDD? (bypass thought: would the new SSDs be able to handle the job, or
are they not yet ready for the very high troughput?)

I just tested it empirically and undoumented, so don't beat me! :)
I first moved all EPS related handling of tmp-stuff to the RAMdisk. Worse
results than keeping all linked paths on the same partition (=physical drive).

Then vice versa, all EPS stuff back to disk, non EPS-tmp-stuff on the
RAMdisk. Much more quiecy, but still considerable missed packets counts.

Next, I tested EPS on its disk, non-EPS on drive D: missed-packet-wise, no
siginificant difference or improvement.

Comment-switching off EPS 5 and 11 reduced the traffic, of course, and
also the error rate. But again, I am miles away from your nearly zero error
rate.


As David pointed out, you have a high troughput on your DVB interface.
I had a longer discussion with David about what throughput means on
Windows and GNU/Linux. If my brain serves me right I think the bottom
line was that under Windows with a SkyStar and MRTG you see the
trougput that finally (maybe with some TelliCast decompression) will
reach your hard disks. With an USB box under Windows and MRTG
throughput seems to be as with GNU/Linux, see below.

If this is right so far, it means that you have licensed a real lot of
data with EUMETSAT and try to get them all. The recv-channels.ini you
posted is allmost like take all ("*").
Yes, and until a few days ago, I had no idea that I had any problems getting
them... :)

....

[ ... ]

So here is some points you can follow if not already done long ago:

1)
It seems that older TelliCast Clients really used "temp_directory"
which is now "tmp_directory".
Done yesterday.
So double check what it is for your
TelliCast Client I don't know the version of. Look under the TelliCast
WEB interface, Help, File Formats.
Will look again, from memory: 2.4.4 A - but I'll check.


2)
Try to direct everything to one HD (F:?) before using both (F: and
E:). Newer SATA drives use AHCI and the like ... As I understood it
worked on F: before without problems.
I've had the bifurcation for the data now for a couple of years.


3)
Are the tmp_directories really generated and used? Is there a slight
chance that your client has an access problem because it has not
sufficient rights somewhere? It seems you had a problem of that kind
with MRTG before?! Same when run as administrator?
All run with admin rights. The MRTG-problems came for a buggy version I
happened to get just after it had come out a few days ago. Corrected, no
more problems on that side.


4)
In recv.ini under [logging] set "log_level=normal" and restart with an
empty FSY (reboot). Then wait some minutes and look what the
"recv.log" file says ...
That's a point! :) Will do that next.


5)
Instead of using "tmp_directories" in "recv-channels.ini" at all you
could use an entry in "recv.ini"under "[parameters]"
"tmp_suffix=.tmp". This is from my GNU/Linux version, so please check
that this exists with the same syntax in your Windows Client (Again
check on your TelliCast WEB interface, Help, File Formats, ...).
But only if I collect everything on one disk - which is not given right now. I
am considering to replace the two disks with one large one, merging te
directories also into one. But not yet.

Happy testing.
Well, yes... Thanks! :)


P.S.
Your post of the recv.ini and recv-channels.ini showed "[recv.ini]"
"[recv-channels.ini]" as head and "[EOF]" as tail. You probably added
that to clarify the post, it should not be in these ini files
otherwise ...
Exactly. If I understand it right, you are a teacher. And obviously one with an
eagle's eye for small glitches... :)

I know, my wife is a teacher too...

Cheers,
U.

Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

I may be wrong but I don't think putting the "tmp_directory" on a RAM disk does any good. The data must be moved to the hard disk E: or F: later anyway. This move will then be a full data copy that you postponed with "tmp_directory" on the RAM disk only.
Ernst, it does help with the 2.5.17 client as the tmp_directory is used instead of the FSY files, but whilst it is an extreme measure, writing to the RAMdisk will be faster and use less resources than writing to HD. It /may/ be worth trying.

[]
I noted one more thing: It appears to me that your FSY file on RAM disk reaches it's top very fast all the time. You posted something like seeing error free reception for some minutes when you started. If you rebooted and your FSY file was recreated with 0 bytes first, this could have been some 5-7 minutes until your 325MB FSY file is full if it cannot be emptied to the tmp_directories as normal.
[]
4)
In recv.ini under [logging] set "log_level=normal" and restart with an empty FSY (reboot). Then wait some minutes and look what the "recv.log" file says ...
[]
Happy testing.

Cheers
Ernst
Yes, I had been assuming that Ulrich had checked the log file for obvious errors like; "No space in FSY database", but whether I asked or not I now forget!

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

Re: not yet perfect, but...

ernstlobsiger
 

--- In MSG-1@..., "Ulrich G. Kliegis" <Ulrich.Kliegis@...> wrote:



An: <MSG-1@...>
Von: "David J Taylor" <gm8arv@...>
Datum: Mon, 7 Mar 2011 07:42:33 -0000
Betreff: Re: [MSG-1] not yet perfect, but...
Antwort an: MSG-1@...

No chance you've left the "*" entry somewhere in recv-channels.ini?
Nope, even not in white print on white paper.

I am just experimenting a bit now - directed the tmp to 50 MB of free space
on the RAM disk for Eumetsat Channels 1 ad 2 - and commented out EPS-
5.

Cheers,
U.
Ulrich

I may be wrong but I don't think putting the "tmp_directory" on a RAM disk does any good. The data must be moved to the hard disk E: or F: later anyway. This move will then be a full data copy that you postponed with "tmp_directory" on the RAM disk only.

As David pointed out, you have a high troughput on your DVB interface. I had a longer discussion with David about what throughput means on Windows and GNU/Linux. If my brain serves me right I think the bottom line was that under Windows with a SkyStar and MRTG you see the trougput that finally (maybe with some TelliCast decompression) will reach your hard disks. With an USB box under Windows and MRTG throughput seems to be as with GNU/Linux, see below.

If this is right so far, it means that you have licensed a real lot of data with EUMETSAT and try to get them all. The recv-channels.ini you posted is allmost like take all ("*").

Under GNU/Linux I see the whole traffic on my dummy interface which is even 50% higher than yours but I only take about 30% of it to disk. You can look at my receivers under

See http://www.gymalp.ch/~eumetcast

at the bottom of the page. Monitoring is done with RRDTool/Eluna here (not MRTG).

This all means that the SkyStar 2.6D can cope with the full traffic without problems while the TelliCast Client may get in trouble assembling, decompressing and writing everything to disk. So David's advice to slowly build up your recv-channels.ini is certainly a good thing.

I noted one more thing: It appears to me that your FSY file on RAM disk reaches it's top very fast all the time. You posted something like seeing error free reception for some minutes when you started. If you rebooted and your FSY file was recreated with 0 bytes first, this could have been some 5-7 minutes until your 325MB FSY file is full if it cannot be emptied to the tmp_directories as normal.


So here is some points you can follow if not already done long ago:

1)
It seems that older TelliCast Clients really used "temp_directory" which is now "tmp_directory". So double check what it is for your TelliCast Client I don't know the version of. Look under the TelliCast WEB interface, Help, File Formats.

2)
Try to direct everything to one HD (F:?) before using both (F: and E:). Newer SATA drives use AHCI and the like ... As I understood it worked on F: before without problems.

3)
Are the tmp_directories really generated and used? Is there a slight chance that your client has an access problem because it has not sufficient rights somewhere? It seems you had a problem of that kind with MRTG before?! Same when run as administrator?

4)
In recv.ini under [logging] set "log_level=normal" and restart with an empty FSY (reboot). Then wait some minutes and look what the "recv.log" file says ...

5)
Instead of using "tmp_directories" in "recv-channels.ini" at all you could use an entry in "recv.ini"under "[parameters]" "tmp_suffix=.tmp". This is from my GNU/Linux version, so please check that this exists with the same syntax in your Windows Client (Again check on your TelliCast WEB interface, Help, File Formats, ...).


Happy testing.

Cheers
Ernst


P.S.
Your post of the recv.ini and recv-channels.ini showed "[recv.ini]" "[recv-channels.ini]"
as head and "[EOF]" as tail. You probably added that to clarify the post, it should not be in these ini files otherwise ...

Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

No chance you've left the "*" entry somewhere in recv-channels.ini?
Nope, even not in white print on white paper.

I am just experimenting a bit now - directed the tmp to 50 MB of free space
on the RAM disk for Eumetsat Channels 1 ad 2 - and commented out EPS-
5.

Cheers,
U.
Ulrich,

If you are taking MODIS-L1 data (or VGT4Africa) 50MB may not be enough - I have seen files up to 85MB. But for a test, it's a start. [EPS-11] (IASI) is another major data throughput source.

I might suggest starting small and building up - a fairly comprehensive list would be something like:

[DWDSAT]
[EPS-10]
[EPS-15]
[EPS-18]
[EUMETSAT Data Channel 1]
[EUMETSAT Data Channel 2]
[EUMETSAT Data Channel 3]
[EUMETSAT Data Channel 4]
[EUMETSAT Data Channel 11]
[Info-Channel-1]
[Info-Channel-2]
[SAF-Africa]
[SAF-Americas]
[SAF-Europe]
[SAF-Global]

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

Re: not yet perfect, but...

Ulrich G. Kliegis
 

An: <MSG-1@...>
Von: "David J Taylor" <gm8arv@...>
Datum: Mon, 7 Mar 2011 07:42:33 -0000
Betreff: Re: [MSG-1] not yet perfect, but...
Antwort an: MSG-1@...

No chance you've left the "*" entry somewhere in recv-channels.ini?
Nope, even not in white print on white paper.

I am just experimenting a bit now - directed the tmp to 50 MB of free space
on the RAM disk for Eumetsat Channels 1 ad 2 - and commented out EPS-
5.

Cheers,
U.

Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

David,
Thank you for the info on the Tellicast versions. I have checked and am
running 2.4.4a, which gives me no trouble and as it ain't broke, I'm not
going to fix it. When the new release is finally made with all the
improvements, I will probably upgrade.

Best regards - Alan S
2.4.4a is the latest released version, Alan, so there's no suggestion that you should upgrade. I did not mean to give that impression. There /will/ be a new version at some point in the future, and it's possible that it will eventually become a mandatory update some time after its release.

2.4.4a works a little better than the 2.5.17 version, in fact!

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

What could have happened when the T-Systems-software didn't find it's
t(e)mp folders? It works with tmp... now, but also worked with temp - and
since I upgraded to the Big Toro dish, I had no missing frames any more
(had them with the old Thomson dish).

But this doesn't cure my (diagnosis-induced) symptoms yet...
Sorry, can't deny my background... ;/

Cheers,

U.
Ulrich, without the tmp_directory, TelliCast will build up the final files in situ, so that there is a danger that programs will try and use the data before it is complete. My own software tries to guard against this, of course, but having the tmp_directory helps ensure file integrity.

On one underpowered PC I have put the tmp_directory on RAMdisk to reduce losses, but that was a single-core PC receiving all the data, including a full Metop stream. (This with the newer 2.5.17 client which does not use FSY files).

Your losses seem to be strongly correlated with data throughput. I do notice that your throughput seems a lot higher than I would expect for a SkyStar card. My main PC, which takes Meteosat-9 full scan, EARS AVHRR, Metop AVHRR, and MODIS-L1 data, has a daily throughput of 400kB/s, whereas yours seems to be in excess of 800kB/s.

http://www.satsignal.eu/mrtg/feenix_dvb.html
http://eumetmon.rummelecke.de/windbuedel_dvb.html

No chance you've left the "*" entry somewhere in recv-channels.ini? It seems unlikely, as you would rapidly run out of disk space! But why is your throughput twice mine? If you aren't using IASI or GOME data, I would suggest commenting out the entries in recv-channels.ini for the moment.

What version of the SkyStar and TelliCast software do you have?

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...