Date   

Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

David (and others!),
last night, I checked the settings of my receiver again. Then, one idea hit me
- you may remember that I had to freshly install it last year. Did I think of
switching Windows' indexing function off? Nope. What followed, was two
hours of intensive re-asserting file properties on two HDDs. While that was
running, I saw that there were tons of old directories on the two disks (one
works for MSG, the other for NOAA and Metop). Purging those put some
additional strain on the r/w-heads in the disk drives. You see these activities
reflected in the missed packets graph of last night. Then - silence. Quiet
disks, and no missed packets in the graph. Well. That lasted a few minutes.
Then the grass started growing again, and I am back to normal now. At
least, the CPU usage has decreased, while it rested on 100% for a second
or two yesterday, about every few seconds, then fell back to 0 to 5%, 100%
appears in short peaks now only.

I also re-arranged the virtual memory, it is on a otherwise not used partition
right now (but physically on the same drive like C:\).

Acronis Backup operates only once per week, so, that can be ruled out too.
No virus protection, no service that eats ressources.

I can't believe yet that my system is insufficient for this job.

Cheers,
U.
Yes, Ulrich, I can see the events on your graphs. Be careful with multiple partitions on the same disk. If the disk head needs to move half way across the disk to access the other partition, that could be a greater penalty than having those files just one the same partition. Using two physical disks is a much better solution - perhaps system files and my software on one disk, EUMETCast and processed data files on a second disk. In any case, get a disk with good performance - with a 32MB or 64MB buffer.

I would recommend having a real-time defragmenter running - with the TelliCast 2.4.4 client software we have found mstDefrag to be very acceptable, and have almost zero effect on received data (i.e. we could not attribute any extra losses to running the defragmenter). I've used version 2 and 3, but I see they are up to version 7 now. How did they manage that?

http://www.mstsoftware.com/mst-Defrag

There are a couple of free programs which we've used with success as well:

http://www.auslogics.com/en/software/disk-defrag/
http://www.iobit.com/iobitsmartdefrag.html

Both will defrag "when needed" by monitoring the disk state. Auslogics will defrag no more than once per 12 hours, and Smart Defrag not more than every three hours. If you haven't defragged your disk, that could be the issue.

From the TelliCast configuration standpoint:

- you have checked that the FSY files are actually on your RAMdisk, I suppose? But judging from your FSY graph they must be!

- I recommend using the common tmp_directory entry in the recv.ini file, rather than individual entries in the recv-channels.ini file - it is cleaner and makes the channels file simpler. Something like:

______________________________
[parameters]
tmp_directory=temp
______________________________


The directory should be on the same partition as the \received\ directories. As Alan mentions, all this should be /outside/ the Program Files tree. I installed to C:\EUMETCast\ in one Windows XP installation. My recv-channels.ini file then has lines like:

______________________________
[EUMETSAT Data Channel 1]
target_directory=received\Data Channel 1

[EUMETSAT Data Channel 2]
target_directory=received\Data Channel 2

[EUMETSAT Data Channel 3]
target_directory=received\Data Channel 3
______________________________


Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...


Re: not yet perfect, but...

Ulrich G. Kliegis
 

Try this simple program to see what's eating up CPU & Disk time.
OK, basically the same as the windows-own processes and services
monitor, minus CPU usage and memory consumption. But good for a quick
overview who is there at all.

Or did I miss something?

Cheers,
U.


Re: not yet perfect, but...

Ulrich G. Kliegis
 

Other thoughts.
Is the disk too full? need to keep it less than 75% even better 50%.
Well, about 30% free, pretty stable.

Is the Hard Drive dying?
Nope, no signs of that, no read-write-problems that normally signify that.

What I cannot exclude is fragmentation, but defragging a 250 G disk is
questionable anyway. On the previous machine, I had a continuous
defragmenter running. It cost more ressources than it could contribute -
switching it off finally reduced the missing files count. But that was on the
old, slow box.

Keep Tellicast on a seperate drive to
processing programs.
You mean the Tellicast programs or its digestive tract?
First, I have a ramdisk running.
then, the satellite-specific receiving/ and received/ directories are on the
same drives (physically) as the target folders are, so, data don't have to be
moved, just re-adjustments of the file pointers in the disk administration. All
disks are NTFS.


Keep processing programs in their own folder and
not default 'Program Files'.
That one is new to me. You say, I should move, say, MSGadmin out of its
\programs\DJTsoftware\MSG1\ context to, say, ..\MSG1\ ?

What is the difference?

Thanks, and cheers!

U.


Re: not yet perfect, but...

Ulrich G. Kliegis
 

Try this simple program to see what's eating up CPU & Disk time.
May give you some clues.
http://www.itsth.com/en/produkte/Whats-my-computer-doing.php?fromwmc
d

Thanks, Alan. Will run that and let you know about the findings.

Cheers,

U.


Re: not yet perfect, but...

Alan Banks <alan@...>
 

Other thoughts.
Is the disk too full? need to keep it less than 75% even better 50%.
Is the Hard Drive dying?
Keep Tellicast on a seperate drive to processing programs.
Keep processing programs in their own folder and not default 'Program Files'.

Alan

On 04/03/2011 09:16, Alan Banks wrote:
Ulrich
Try this simple program to see what's eating up CPU& Disk time.
May give you some clues.
http://www.itsth.com/en/produkte/Whats-my-computer-doing.php?fromwmcd
Cheers

Alan


Re: not yet perfect, but...

Alan Banks <alan@...>
 

Ulrich
Try this simple program to see what's eating up CPU & Disk time.
May give you some clues.
http://www.itsth.com/en/produkte/Whats-my-computer-doing.php?fromwmcd
Cheers

Alan


On 04/03/2011 09:08, Ulrich G. Kliegis wrote:

I can't believe yet that my system is insufficient for this job.

Cheers,
U.


Re: not yet perfect, but...

Ulrich G. Kliegis
 


May well help - I try and run as much stuff as I can at lower priority
here.
David (and others!),
last night, I checked the settings of my receiver again. Then, one idea hit me
- you may remember that I had to freshly install it last year. Did I think of
switching Windows' indexing function off? Nope. What followed, was two
hours of intensive re-asserting file properties on two HDDs. While that was
running, I saw that there were tons of old directories on the two disks (one
works for MSG, the other for NOAA and Metop). Purging those put some
additional strain on the r/w-heads in the disk drives. You see these activities
reflected in the missed packets graph of last night. Then - silence. Quiet
disks, and no missed packets in the graph. Well. That lasted a few minutes.
Then the grass started growing again, and I am back to normal now. At
least, the CPU usage has decreased, while it rested on 100% for a second
or two yesterday, about every few seconds, then fell back to 0 to 5%, 100%
appears in short peaks now only.

I also re-arranged the virtual memory, it is on a otherwise not used partition
right now (but physically on the same drive like C:&#92;).

Acronis Backup operates only once per week, so, that can be ruled out too.
No virus protection, no service that eats ressources.

I can't believe yet that my system is insufficient for this job.

Cheers,
U.


Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

Note that we are in the solar outage period right now, so the peak in
missed packets on my system in the last day or two is due to solar
noise getting into the receiving chain, in my case around 11:30 UTC.
That's about 11:42 here (which is only intuitively paradox, gemetrically
correct, if my navigating intuition doesn't fail...)
Yes, it took me a moment, but as I am further west, my antenna looks further east, so the sun will be in the beam of my antenna before it's in the beam of your more west-facing antenna. EB-9 is at 165 degrees azimuth from Edinburgh.

Yes, I'll trace that for another day or two. It can be the backup program, I
could run that with a reduced priority.
May well help - I try and run as much stuff as I can at lower priority here.

Speaking of priorities, I happened to look at the active processes in sysmon
on the receiver. First, I saw that server4 PC exists there twice, both
instances running with priority set to normal. tc-recv is there twice as well,
one running with prio set to 'high', the other 'higher than normal'. I never
changed anything there, so that's probably standard.
Yes, that's normal.

All PCs run at 100 Mb/s here.

The LAN activities seem to be pretty low, just the peak after switching on the
viewing machine is prominent, the rest rather unspectacular, I think.

I'll re-check the recv.ini settings. The totally constant FSY size puzzles me.

I think, the front end (skystar) does not need much more attention, otherwise
the BER should be much higher.

It is really surprising how much analysis can be done from remote based on
that few pixels... :) Highly complex communication with lots of a priori
knowledge - and much more a posteriori to come... :)

Cheers,

U.
The FSY size will be constant, or at least monotonically increasing in normal use. Every time there is some data it needs to store, it puts it into one or other of the FSY files, and it never fives that space back to the OS. If some larger data comes in (e.g. VGT4Africa) the FSY size will increase, but not drop back afterwards. The newer 2.5.17 TelliCast client handles things completely differently, but has a few bugs in handing back space...

I find with the 2.6x cards that the signal has to be very bad for a non-zero BER reading - that's why the BER section of my graphs is restricted (mostly) to 2.3 cards which had either a more sensitive BER indication or were poorer performers, or both!

I'm glad that you've found the effort you put into monitoring is now giving some worthwhile feedback.

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...


Re: not yet perfect, but...

Ulrich G. Kliegis
 

Hi David,

nice to know you are around all the time:)

I'll readjust the dish today (hopefully). Should help with the
missed packets. Or?

Judging by your graphs, Ulrich, it seems you may have made an
adjustment around 13:00 UTC.
That was the first attempt, yes. A bit tricky here, the rec PC lives in the
basement, and since I don't have a helper here right now, I have to transfer
the audio signal from server4pc to the dish on the balcony (now ice-free) by
means of the house-phone.

I losened and refastened the screws keeping the stand together, since it
looked like the parts had shifted a bit there - this led to a 2 % higher signal
strength, a just visible increase in the SNR, and about 8% in signal quality.

The two quad-LNBs for Astra and Hotbird 6 sit on the same rail like the
single LNB for EB9. I had optimized the dish position and adjustments for
the latter, since it carries the weakest signal.

The TV channels ar still beyond their previous quality (but totally dead
before the first adjustment today), although they still sit in the same position
on the rail. It would be interesting to hear from Arne who has the same dish,
if it can deform in vfa (very fast air).
Being not yet happy with the result, I loosened the screws that hold the
whole tilting mechanics on the pole and gave the mirror a light push - again
listening to the audio-feedback. Re-fastened it, again, a very slight increase
in strength, but nothing else.

Missed packets are usually caused by too high a load on your system,
in particular:

- network IO
- CPU load
- disk I/O

For most systems, adding a RAMdisk on which to store the FSY files
reduces the susceptibility to these overloads down to a level where
there are but a few missed packets a day, as you can see here:
I have a 349 MB sized RAM disk (some commercial solution), you see the
fsy file size - totally constant - on my eumetmon - page.


http://www.satsignal.eu/mrtg/performance_eumetcast-europe-packets.ph
p

Note that we are in the solar outage period right now, so the peak in
missed packets on my system in the last day or two is due to solar
noise getting into the receiving chain, in my case around 11:30 UTC.
That's about 11:42 here (which is only intuitively paradox, gemetrically
correct, if my navigating intuition doesn't fail...)


See whether the graph of missed packets has peaks that correspond to
other activity on the PC. On your combined graph, for example, shift
the missed packet graph to be under the throughput graph, and I think
you will see a correlation.

http://eumetmon.rummelecke.de/

The peaks at about 06:30 UTC and 12:55 UTC look to be due to a
different activity.
Yes, I'll trace that for another day or two. It can be the backup program, I
could run that with a reduced priority.

Speaking of priorities, I happened to look at the active processes in sysmon
on the receiver. First, I saw that server4 PC exists there twice, both
instances running with priority set to normal. tc-recv is there twice as well,
one running with prio set to 'high', the other 'higher than normal'. I never
changed anything there, so that's probably standard.


If you have a two-PC setup with a LAN connection, you might want to
check that the LAN on the receiver PC is set to 100Mb/s rather than
1Gb/s, as I've found that it can cause problems working at the
higher-speed, possibly due to "interrupt coalescing". Note that
changing the LAN speed /may/ cause the interface number to change, so
a power-down reboot after that would restore the old numbers.
All PCs run at 100 Mb/s here.

The LAN activities seem to be pretty low, just the peak after switching on the
viewing machine is prominent, the rest rather unspectacular, I think.

I'll re-check the recv.ini settings. The totally constant FSY size puzzles me.

I think, the front end (skystar) does not need much more attention, otherwise
the BER should be much higher.

It is really surprising how much analysis can be done from remote based on
that few pixels... :) Highly complex communication with lots of a priori
knowledge - and much more a posteriori to come... :)

Cheers,

U.


Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

I'll readjust the dish today (hopefully). Should help with the missed packets.
Or?

Cheers,

U.
Judging by your graphs, Ulrich, it seems you may have made an adjustment around 13:00 UTC.

Missed packets are usually caused by too high a load on your system, in particular:

- network IO
- CPU load
- disk I/O

For most systems, adding a RAMdisk on which to store the FSY files reduces the susceptibility to these overloads down to a level where there are but a few missed packets a day, as you can see here:

http://www.satsignal.eu/mrtg/performance_eumetcast-europe-packets.php

Note that we are in the solar outage period right now, so the peak in missed packets on my system in the last day or two is due to solar noise getting into the receiving chain, in my case around 11:30 UTC.

See whether the graph of missed packets has peaks that correspond to other activity on the PC. On your combined graph, for example, shift the missed packet graph to be under the throughput graph, and I think you will see a correlation.

http://eumetmon.rummelecke.de/

The peaks at about 06:30 UTC and 12:55 UTC look to be due to a different activity.

If you have a two-PC setup with a LAN connection, you might want to check that the LAN on the receiver PC is set to 100Mb/s rather than 1Gb/s, as I've found that it can cause problems working at the higher-speed, possibly due to "interrupt coalescing". Note that changing the LAN speed /may/ cause the interface number to change, so a power-down reboot after that would restore the old numbers.

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...


Re: not yet perfect, but...

Ulrich G. Kliegis
 

I'll add a www.-alias, nevertheless, to the server routing.
Ooops, not that easily feasible. So, no www.

I'll readjust the dish today (hopefully). Should help with the missed packets.
Or?

Cheers,

U.


Re: not yet perfect, but...

Ulrich G. Kliegis
 

An: <MSG-1@...>
Von: "David J Taylor" <gm8arv@...>
Datum: Thu, 3 Mar 2011 10:55:00 -0000
Betreff: Re: [MSG-1] not yet perfect, but...
Antwort an: MSG-1@...

I'm sure you meant:
http://eumetmon.rummelecke.de/
Sure!

I'll add a www.-alias, nevertheless, to the server routing.

Cheers,
U.


EnviHam File Browser

fbreame
 

Although not strictly a MSG topic, we may have some EnviHam (an experimental project for reception of EnviSat images similar to EUMETCast) users on this group, so this may be of interest.

As I mentioned on the GEO-Subscribers group recently, I've written a browser for EnviHam files, called, unimaginatively, EnviHamBrowser. This makes it quicker and easier to see their coverage, and then to select one or more to be opened in the ESA-supplied VISAT viewer. Unfortunately VISAT, although it does show coverage, is rather slow in opening files (at least on my setup).

EnviHamBrowser is available for download on my www site at

http://www.vf0123.btinternet.co.uk

together with its user guide, if anyone would like to try it. Please read the user guide for installation instructions.

(If you have already downloaded the original version 0.1.0, please update to the new version 0.1.1 which fixes a problem with incorrect views on images longer than 1 orbit.)

I haven't had the opportunity to test it on a lot of systems, so you will be doing a valuable beta test. Please let me know of any problems or comments.

Francis Breame


Re: re- cma

Alan Sewards
 

Geoffrey,

In your mail of 24 Feb, you gave your recv-channels.ini file and the relevant block is:

#METOP AVHRR Data channel EPS-10
[EPS-10]
target_directory=C:&#92;received&#92;EPS-10

In my opinion, this block is quite correct. The title line (there for humans) is commented with the #, so is ignored by the software. The crucial two lines required for the computer are present and correct. The only thing following from this is that you must look for incoming files in the folder C:&#92;received&#92;EPS-10, and NOT in the folder with the same name in the C:&#92;Program Files&#92;T Systems&#92;Business TV-IP tree.

Best regards - Alan S

On 03/03/2011 10:44 AM, Geoffrey Gee wrote:
Alan

In rec channels.ini the line Metop AVHRR is not commented . Is this
relevant to my troubles ?

No useful news from Eumetsat as yet..

Best Regards.....Geoffrey



------------------------------------

Unsure what a term means? Check the Glossary at:
http://www.satsignal.eu/wxsat/glossary.htm
Yahoo! Groups Links



--
Alan Sewards
email: alan@...
web site: http://asewards.free.fr


Re: not yet perfect, but...

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

Just added FSY size and missed packets graphs. See
www.eumetmon.rummelecke.de
I just tried, and got: "Hier entsteht eine neue Internetpr�senz! ", but I'm sure you meant:
http://eumetmon.rummelecke.de/

I've added your packet loss (which should be much nearer zero, by the way).

Is there a simple and not too resources-hungry way to determine the free
space on the HDDs? I'd like to plot that too, if possible.

Cheers,
U.

(if you wonder what "rummelecke" is and don't find in your dictionary - that's
right. It is a family-idiom, standing for corners where unsorted items
sedimentate. My office is such a place...)
Danke!

There are some notes on disk space here:

http://www.satsignal.eu/mrtg/performance_howto.php#DiskSpace

and it's not really satisfactory. You can get the disk used, but not the percent used, and not the disk free. Even worse, the entries are an index into the disk table, and those entry numbers change if, for example, you plug in a USB memory stick. I've not found an easy way to do this in Perl (without add 3rd party modules), so I really should write a small Delphi program with takes one or two HD names as parameters (e.g. C: E:), and has an option to return percent used, percent free, MB used or MB free as the integer result. Maybe one day? Maybe someone else would write something?

I upload at 15 minute intervals - sort of consistent with Meteosat-9 scans, but the more frequent the better.

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...


Re: re- cma

Hans-Joachim Empel
 

In rec.channels.ini the METOP AVHRR is EPS10.

My rec channels.ini looks like this :

[EPS-10]
target_directory=D : EPS-10
tmp_directory=D : received / tmp

Hope that helps

Regards from Germany
Joachim

----- Original Message -----
From: "Geoffrey Gee" <geogee@...>
To: <MSG-1@...>
Sent: Thursday, March 03, 2011 10:44 AM
Subject: Re: [MSG-1] re- cma


Alan
In rec channels.ini the line Metop AVHRR is not commented . Is this relevant to my troubles ?
No useful news from Eumetsat as yet..
Best Regards.....Geoffrey ------------------------------------
Unsure what a term means? Check the Glossary at: http://www.satsignal.eu/wxsat/glossary.htm
Yahoo! Groups Links


Re: re- cma

geoffreygee90
 

Alan

In rec channels.ini the line Metop AVHRR is not commented . Is this relevant to my troubles ?

No useful news from Eumetsat as yet..

Best Regards.....Geoffrey


Re: not yet perfect, but...

Ulrich G. Kliegis
 

I know that EUMETSAT do look at the
pages and take an interest in them.
I forgot to mention that I changed the upload interval from 20 to 10 mins. My
data seemed to lag a bit behind most others'.

Cheers,
U.


Re: not yet perfect, but...

Ulrich G. Kliegis
 

Many thanks for setting this up. I know that EUMETSAT do look at the
pages and take an interest in them.
Just added FSY size and missed packets graphs. See
www.eumetmon.rummelecke.de

Is there a simple and not too resources-hungry way to determine the free
space on the HDDs? I'd like to plot that too, if possible.

Cheers,
U.

(if you wonder what "rummelecke" is and don't find in your dictionary - that's
right. It is a family-idiom, standing for corners where unsorted items
sedimentate. My office is such a place...)


Re: re- cma

Alan Sewards
 

Duncan,
Line Metop AVHRR should be commented, like #Metop AVHRR
line NOAA 19 GAC should be commented, like #NOAA 19 GAC.

I'm not sure if it is obligatory, but I use a temp_directory= xxx line for each of the channels, where xxx is receiving&#92;tmp for those channels where your target directory is received, and receiving&#92;EPS-10 etc for the others. Also I don't know where these files are installed as you do not give the path - are your files in a root directory or in the Business TV-IP directory?

And finally, please punctuate your postings. They are difficult to read with no full stops and no paragraphing.

Best regards - Alan S

On 02/03/2011 5:26 PM, Duncan Edwards wrote:
Just as a precaution here is a copy of my recv-channels. As i only
receive a small portion of the data compaired to others i am wondering
as Ian said "my ramdisk is filling up fast" if i may after all this time
of recieving the data i may have something wrong here any comments would
be apreciated. Best Regards Duncan

recv-channels.ini - TelliCast+TelliVision Client channel configuration file
#
# 2-4-3 10-10-2005
#
[EUMETSAT Data Channel 1]
target_directory=received

[EUMETSAT Data Channel 2]
target_directory=received

[EUMETSAT Data Channel 3]
target_directory=received

[EUMETSAT Data Channel 4]
target_directory=received

[EUMETSAT Data Channel 11]
target_directory=received

Metop AVHRR
[EPS-10]
target_directory=received&#92;EPS-10

NOAA 19 GAC
[EPS-15]
target_directory=received&#92;EPS-15

# target_directory: Directory for received files. The target_directory name
# is relative to the current working directory.
# syntax: target_directory=<directory>
# default: received

tmp_directory=receiving/tmp/files
# tmp_directory: If this parameter is given tc-recv will internally
# store/compose all received files within the given directory and
# later move them to the target directory.
# While this option requires an additional move operation it
# guarantees that all files appearing in the target directory are
# fully written to the file system.
# syntax: tmp_directory=<directory>
# default: undefined=off, but default value may be redefined in
# recv.ini

# Section per channel or channel group identified by channel name (wildcard
# "*" allowed at end of name)

#activate_ftp_forwarding=1
# activate_ftp_forwarding: Will activate FTP forwarding for the correctly
# received files according to the recv.ini [ftp_forwarding]
# parameters.
# syntax: activate_ftp_forwarding=<value of range>
# range: 0/1
# default: 0=off

#tmp_directory=receiving/tmp/files
# tmp_directory: If this parameter is given tc-recv will internally
# store/compose all received files within the given directory and
# later move them to the target directory.
# While this option requires an additional move operation it
# guarantees that all files appearing in the target directory are
# fully written to the file system.
# syntax: tmp_directory=<directory>
# default: undefined=off, but default value may be redefined in
# recv.ini

#tmp_prefix=.
# tmp_prefix: Like tmp_directory, but files are composed within the final
# target directories but stored with the given prefix in front of
# the normal filenames until the files are fully written.
# syntax: tmp_prefix=<file name start>
# default: undefined=off, but default value may be redefined in
# recv.ini

#tmp_suffix=.tmp
# tmp_suffix: Like tmp_prefix, but adding a suffix to the normal filenames
# until the files are fully written.
# syntax: tmp_suffix=<file suffix>
# default: undefined=off, but default value may be redefined in
# recv.ini

--
Alan Sewards
email: alan@...
web site: http://asewards.free.fr

18001 - 18020 of 33137