Date   
Re: re- cma

geoffreygee90
 

Alan

In rec channels.ini the line Metop AVHRR is not commented . Is this relevant to my troubles ?

No useful news from Eumetsat as yet..

Best Regards.....Geoffrey

Re: re- cma

Hans-Joachim Empel
 

In rec.channels.ini the METOP AVHRR is EPS10.

My rec channels.ini looks like this :

[EPS-10]
target_directory=D : EPS-10
tmp_directory=D : received / tmp

Hope that helps

Regards from Germany
Joachim

----- Original Message -----
From: "Geoffrey Gee" <geogee@...>
To: <MSG-1@...>
Sent: Thursday, March 03, 2011 10:44 AM
Subject: Re: [MSG-1] re- cma


Alan
In rec channels.ini the line Metop AVHRR is not commented . Is this relevant to my troubles ?
No useful news from Eumetsat as yet..
Best Regards.....Geoffrey ------------------------------------
Unsure what a term means? Check the Glossary at: http://www.satsignal.eu/wxsat/glossary.htm
Yahoo! Groups Links

Re: not yet perfect, but...

David J Taylor
 

Just added FSY size and missed packets graphs. See
www.eumetmon.rummelecke.de
I just tried, and got: "Hier entsteht eine neue Internetpr�senz! ", but I'm sure you meant:
http://eumetmon.rummelecke.de/

I've added your packet loss (which should be much nearer zero, by the way).

Is there a simple and not too resources-hungry way to determine the free
space on the HDDs? I'd like to plot that too, if possible.

Cheers,
U.

(if you wonder what "rummelecke" is and don't find in your dictionary - that's
right. It is a family-idiom, standing for corners where unsorted items
sedimentate. My office is such a place...)
Danke!

There are some notes on disk space here:

http://www.satsignal.eu/mrtg/performance_howto.php#DiskSpace

and it's not really satisfactory. You can get the disk used, but not the percent used, and not the disk free. Even worse, the entries are an index into the disk table, and those entry numbers change if, for example, you plug in a USB memory stick. I've not found an easy way to do this in Perl (without add 3rd party modules), so I really should write a small Delphi program with takes one or two HD names as parameters (e.g. C: E:), and has an option to return percent used, percent free, MB used or MB free as the integer result. Maybe one day? Maybe someone else would write something?

I upload at 15 minute intervals - sort of consistent with Meteosat-9 scans, but the more frequent the better.

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

Re: re- cma

Alan Sewards
 

Geoffrey,

In your mail of 24 Feb, you gave your recv-channels.ini file and the relevant block is:

#METOP AVHRR Data channel EPS-10
[EPS-10]
target_directory=C:&#92;received&#92;EPS-10

In my opinion, this block is quite correct. The title line (there for humans) is commented with the #, so is ignored by the software. The crucial two lines required for the computer are present and correct. The only thing following from this is that you must look for incoming files in the folder C:&#92;received&#92;EPS-10, and NOT in the folder with the same name in the C:&#92;Program Files&#92;T Systems&#92;Business TV-IP tree.

Best regards - Alan S

On 03/03/2011 10:44 AM, Geoffrey Gee wrote:
Alan

In rec channels.ini the line Metop AVHRR is not commented . Is this
relevant to my troubles ?

No useful news from Eumetsat as yet..

Best Regards.....Geoffrey



------------------------------------

Unsure what a term means? Check the Glossary at:
http://www.satsignal.eu/wxsat/glossary.htm
Yahoo! Groups Links



--
Alan Sewards
email: alan@...
web site: http://asewards.free.fr

EnviHam File Browser

fbreame
 

Although not strictly a MSG topic, we may have some EnviHam (an experimental project for reception of EnviSat images similar to EUMETCast) users on this group, so this may be of interest.

As I mentioned on the GEO-Subscribers group recently, I've written a browser for EnviHam files, called, unimaginatively, EnviHamBrowser. This makes it quicker and easier to see their coverage, and then to select one or more to be opened in the ESA-supplied VISAT viewer. Unfortunately VISAT, although it does show coverage, is rather slow in opening files (at least on my setup).

EnviHamBrowser is available for download on my www site at

http://www.vf0123.btinternet.co.uk

together with its user guide, if anyone would like to try it. Please read the user guide for installation instructions.

(If you have already downloaded the original version 0.1.0, please update to the new version 0.1.1 which fixes a problem with incorrect views on images longer than 1 orbit.)

I haven't had the opportunity to test it on a lot of systems, so you will be doing a valuable beta test. Please let me know of any problems or comments.

Francis Breame

Re: not yet perfect, but...

Ulrich G. Kliegis
 

An: <MSG-1@...>
Von: "David J Taylor" <gm8arv@...>
Datum: Thu, 3 Mar 2011 10:55:00 -0000
Betreff: Re: [MSG-1] not yet perfect, but...
Antwort an: MSG-1@...

I'm sure you meant:
http://eumetmon.rummelecke.de/
Sure!

I'll add a www.-alias, nevertheless, to the server routing.

Cheers,
U.

Re: not yet perfect, but...

Ulrich G. Kliegis
 

I'll add a www.-alias, nevertheless, to the server routing.
Ooops, not that easily feasible. So, no www.

I'll readjust the dish today (hopefully). Should help with the missed packets.
Or?

Cheers,

U.

Re: not yet perfect, but...

David J Taylor
 

I'll readjust the dish today (hopefully). Should help with the missed packets.
Or?

Cheers,

U.
Judging by your graphs, Ulrich, it seems you may have made an adjustment around 13:00 UTC.

Missed packets are usually caused by too high a load on your system, in particular:

- network IO
- CPU load
- disk I/O

For most systems, adding a RAMdisk on which to store the FSY files reduces the susceptibility to these overloads down to a level where there are but a few missed packets a day, as you can see here:

http://www.satsignal.eu/mrtg/performance_eumetcast-europe-packets.php

Note that we are in the solar outage period right now, so the peak in missed packets on my system in the last day or two is due to solar noise getting into the receiving chain, in my case around 11:30 UTC.

See whether the graph of missed packets has peaks that correspond to other activity on the PC. On your combined graph, for example, shift the missed packet graph to be under the throughput graph, and I think you will see a correlation.

http://eumetmon.rummelecke.de/

The peaks at about 06:30 UTC and 12:55 UTC look to be due to a different activity.

If you have a two-PC setup with a LAN connection, you might want to check that the LAN on the receiver PC is set to 100Mb/s rather than 1Gb/s, as I've found that it can cause problems working at the higher-speed, possibly due to "interrupt coalescing". Note that changing the LAN speed /may/ cause the interface number to change, so a power-down reboot after that would restore the old numbers.

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

Re: not yet perfect, but...

Ulrich G. Kliegis
 

Hi David,

nice to know you are around all the time:)

I'll readjust the dish today (hopefully). Should help with the
missed packets. Or?

Judging by your graphs, Ulrich, it seems you may have made an
adjustment around 13:00 UTC.
That was the first attempt, yes. A bit tricky here, the rec PC lives in the
basement, and since I don't have a helper here right now, I have to transfer
the audio signal from server4pc to the dish on the balcony (now ice-free) by
means of the house-phone.

I losened and refastened the screws keeping the stand together, since it
looked like the parts had shifted a bit there - this led to a 2 % higher signal
strength, a just visible increase in the SNR, and about 8% in signal quality.

The two quad-LNBs for Astra and Hotbird 6 sit on the same rail like the
single LNB for EB9. I had optimized the dish position and adjustments for
the latter, since it carries the weakest signal.

The TV channels ar still beyond their previous quality (but totally dead
before the first adjustment today), although they still sit in the same position
on the rail. It would be interesting to hear from Arne who has the same dish,
if it can deform in vfa (very fast air).
Being not yet happy with the result, I loosened the screws that hold the
whole tilting mechanics on the pole and gave the mirror a light push - again
listening to the audio-feedback. Re-fastened it, again, a very slight increase
in strength, but nothing else.

Missed packets are usually caused by too high a load on your system,
in particular:

- network IO
- CPU load
- disk I/O

For most systems, adding a RAMdisk on which to store the FSY files
reduces the susceptibility to these overloads down to a level where
there are but a few missed packets a day, as you can see here:
I have a 349 MB sized RAM disk (some commercial solution), you see the
fsy file size - totally constant - on my eumetmon - page.


http://www.satsignal.eu/mrtg/performance_eumetcast-europe-packets.ph
p

Note that we are in the solar outage period right now, so the peak in
missed packets on my system in the last day or two is due to solar
noise getting into the receiving chain, in my case around 11:30 UTC.
That's about 11:42 here (which is only intuitively paradox, gemetrically
correct, if my navigating intuition doesn't fail...)


See whether the graph of missed packets has peaks that correspond to
other activity on the PC. On your combined graph, for example, shift
the missed packet graph to be under the throughput graph, and I think
you will see a correlation.

http://eumetmon.rummelecke.de/

The peaks at about 06:30 UTC and 12:55 UTC look to be due to a
different activity.
Yes, I'll trace that for another day or two. It can be the backup program, I
could run that with a reduced priority.

Speaking of priorities, I happened to look at the active processes in sysmon
on the receiver. First, I saw that server4 PC exists there twice, both
instances running with priority set to normal. tc-recv is there twice as well,
one running with prio set to 'high', the other 'higher than normal'. I never
changed anything there, so that's probably standard.


If you have a two-PC setup with a LAN connection, you might want to
check that the LAN on the receiver PC is set to 100Mb/s rather than
1Gb/s, as I've found that it can cause problems working at the
higher-speed, possibly due to "interrupt coalescing". Note that
changing the LAN speed /may/ cause the interface number to change, so
a power-down reboot after that would restore the old numbers.
All PCs run at 100 Mb/s here.

The LAN activities seem to be pretty low, just the peak after switching on the
viewing machine is prominent, the rest rather unspectacular, I think.

I'll re-check the recv.ini settings. The totally constant FSY size puzzles me.

I think, the front end (skystar) does not need much more attention, otherwise
the BER should be much higher.

It is really surprising how much analysis can be done from remote based on
that few pixels... :) Highly complex communication with lots of a priori
knowledge - and much more a posteriori to come... :)

Cheers,

U.

Re: not yet perfect, but...

David J Taylor
 

Note that we are in the solar outage period right now, so the peak in
missed packets on my system in the last day or two is due to solar
noise getting into the receiving chain, in my case around 11:30 UTC.
That's about 11:42 here (which is only intuitively paradox, gemetrically
correct, if my navigating intuition doesn't fail...)
Yes, it took me a moment, but as I am further west, my antenna looks further east, so the sun will be in the beam of my antenna before it's in the beam of your more west-facing antenna. EB-9 is at 165 degrees azimuth from Edinburgh.

Yes, I'll trace that for another day or two. It can be the backup program, I
could run that with a reduced priority.
May well help - I try and run as much stuff as I can at lower priority here.

Speaking of priorities, I happened to look at the active processes in sysmon
on the receiver. First, I saw that server4 PC exists there twice, both
instances running with priority set to normal. tc-recv is there twice as well,
one running with prio set to 'high', the other 'higher than normal'. I never
changed anything there, so that's probably standard.
Yes, that's normal.

All PCs run at 100 Mb/s here.

The LAN activities seem to be pretty low, just the peak after switching on the
viewing machine is prominent, the rest rather unspectacular, I think.

I'll re-check the recv.ini settings. The totally constant FSY size puzzles me.

I think, the front end (skystar) does not need much more attention, otherwise
the BER should be much higher.

It is really surprising how much analysis can be done from remote based on
that few pixels... :) Highly complex communication with lots of a priori
knowledge - and much more a posteriori to come... :)

Cheers,

U.
The FSY size will be constant, or at least monotonically increasing in normal use. Every time there is some data it needs to store, it puts it into one or other of the FSY files, and it never fives that space back to the OS. If some larger data comes in (e.g. VGT4Africa) the FSY size will increase, but not drop back afterwards. The newer 2.5.17 TelliCast client handles things completely differently, but has a few bugs in handing back space...

I find with the 2.6x cards that the signal has to be very bad for a non-zero BER reading - that's why the BER section of my graphs is restricted (mostly) to 2.3 cards which had either a more sensitive BER indication or were poorer performers, or both!

I'm glad that you've found the effort you put into monitoring is now giving some worthwhile feedback.

Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

Re: not yet perfect, but...

Ulrich G. Kliegis
 


May well help - I try and run as much stuff as I can at lower priority
here.
David (and others!),
last night, I checked the settings of my receiver again. Then, one idea hit me
- you may remember that I had to freshly install it last year. Did I think of
switching Windows' indexing function off? Nope. What followed, was two
hours of intensive re-asserting file properties on two HDDs. While that was
running, I saw that there were tons of old directories on the two disks (one
works for MSG, the other for NOAA and Metop). Purging those put some
additional strain on the r/w-heads in the disk drives. You see these activities
reflected in the missed packets graph of last night. Then - silence. Quiet
disks, and no missed packets in the graph. Well. That lasted a few minutes.
Then the grass started growing again, and I am back to normal now. At
least, the CPU usage has decreased, while it rested on 100% for a second
or two yesterday, about every few seconds, then fell back to 0 to 5%, 100%
appears in short peaks now only.

I also re-arranged the virtual memory, it is on a otherwise not used partition
right now (but physically on the same drive like C:&#92;).

Acronis Backup operates only once per week, so, that can be ruled out too.
No virus protection, no service that eats ressources.

I can't believe yet that my system is insufficient for this job.

Cheers,
U.

Re: not yet perfect, but...

Alan Banks <alan@...>
 

Ulrich
Try this simple program to see what's eating up CPU & Disk time.
May give you some clues.
http://www.itsth.com/en/produkte/Whats-my-computer-doing.php?fromwmcd
Cheers

Alan


On 04/03/2011 09:08, Ulrich G. Kliegis wrote:

I can't believe yet that my system is insufficient for this job.

Cheers,
U.

Re: not yet perfect, but...

Alan Banks <alan@...>
 

Other thoughts.
Is the disk too full? need to keep it less than 75% even better 50%.
Is the Hard Drive dying?
Keep Tellicast on a seperate drive to processing programs.
Keep processing programs in their own folder and not default 'Program Files'.

Alan

On 04/03/2011 09:16, Alan Banks wrote:
Ulrich
Try this simple program to see what's eating up CPU& Disk time.
May give you some clues.
http://www.itsth.com/en/produkte/Whats-my-computer-doing.php?fromwmcd
Cheers

Alan

Re: not yet perfect, but...

Ulrich G. Kliegis
 

Other thoughts.
Is the disk too full? need to keep it less than 75% even better 50%.
Well, about 30% free, pretty stable.

Is the Hard Drive dying?
Nope, no signs of that, no read-write-problems that normally signify that.

What I cannot exclude is fragmentation, but defragging a 250 G disk is
questionable anyway. On the previous machine, I had a continuous
defragmenter running. It cost more ressources than it could contribute -
switching it off finally reduced the missing files count. But that was on the
old, slow box.

Keep Tellicast on a seperate drive to
processing programs.
You mean the Tellicast programs or its digestive tract?
First, I have a ramdisk running.
then, the satellite-specific receiving/ and received/ directories are on the
same drives (physically) as the target folders are, so, data don't have to be
moved, just re-adjustments of the file pointers in the disk administration. All
disks are NTFS.


Keep processing programs in their own folder and
not default 'Program Files'.
That one is new to me. You say, I should move, say, MSGadmin out of its
&#92;programs&#92;DJTsoftware&#92;MSG1&#92; context to, say, ..&#92;MSG1&#92; ?

What is the difference?

Thanks, and cheers!

U.

Re: not yet perfect, but...

Ulrich G. Kliegis
 

Try this simple program to see what's eating up CPU & Disk time.
May give you some clues.
http://www.itsth.com/en/produkte/Whats-my-computer-doing.php?fromwmc
d

Thanks, Alan. Will run that and let you know about the findings.

Cheers,

U.

Re: not yet perfect, but...

Ulrich G. Kliegis
 

Try this simple program to see what's eating up CPU & Disk time.
OK, basically the same as the windows-own processes and services
monitor, minus CPU usage and memory consumption. But good for a quick
overview who is there at all.

Or did I miss something?

Cheers,
U.

Re: not yet perfect, but...

David J Taylor
 

David (and others!),
last night, I checked the settings of my receiver again. Then, one idea hit me
- you may remember that I had to freshly install it last year. Did I think of
switching Windows' indexing function off? Nope. What followed, was two
hours of intensive re-asserting file properties on two HDDs. While that was
running, I saw that there were tons of old directories on the two disks (one
works for MSG, the other for NOAA and Metop). Purging those put some
additional strain on the r/w-heads in the disk drives. You see these activities
reflected in the missed packets graph of last night. Then - silence. Quiet
disks, and no missed packets in the graph. Well. That lasted a few minutes.
Then the grass started growing again, and I am back to normal now. At
least, the CPU usage has decreased, while it rested on 100% for a second
or two yesterday, about every few seconds, then fell back to 0 to 5%, 100%
appears in short peaks now only.

I also re-arranged the virtual memory, it is on a otherwise not used partition
right now (but physically on the same drive like C:&#92;).

Acronis Backup operates only once per week, so, that can be ruled out too.
No virus protection, no service that eats ressources.

I can't believe yet that my system is insufficient for this job.

Cheers,
U.
Yes, Ulrich, I can see the events on your graphs. Be careful with multiple partitions on the same disk. If the disk head needs to move half way across the disk to access the other partition, that could be a greater penalty than having those files just one the same partition. Using two physical disks is a much better solution - perhaps system files and my software on one disk, EUMETCast and processed data files on a second disk. In any case, get a disk with good performance - with a 32MB or 64MB buffer.

I would recommend having a real-time defragmenter running - with the TelliCast 2.4.4 client software we have found mstDefrag to be very acceptable, and have almost zero effect on received data (i.e. we could not attribute any extra losses to running the defragmenter). I've used version 2 and 3, but I see they are up to version 7 now. How did they manage that?

http://www.mstsoftware.com/mst-Defrag

There are a couple of free programs which we've used with success as well:

http://www.auslogics.com/en/software/disk-defrag/
http://www.iobit.com/iobitsmartdefrag.html

Both will defrag "when needed" by monitoring the disk state. Auslogics will defrag no more than once per 12 hours, and Smart Defrag not more than every three hours. If you haven't defragged your disk, that could be the issue.

From the TelliCast configuration standpoint:

- you have checked that the FSY files are actually on your RAMdisk, I suppose? But judging from your FSY graph they must be!

- I recommend using the common tmp_directory entry in the recv.ini file, rather than individual entries in the recv-channels.ini file - it is cleaner and makes the channels file simpler. Something like:

______________________________
[parameters]
tmp_directory=temp
______________________________


The directory should be on the same partition as the &#92;received&#92; directories. As Alan mentions, all this should be /outside/ the Program Files tree. I installed to C:&#92;EUMETCast&#92; in one Windows XP installation. My recv-channels.ini file then has lines like:

______________________________
[EUMETSAT Data Channel 1]
target_directory=received&#92;Data Channel 1

[EUMETSAT Data Channel 2]
target_directory=received&#92;Data Channel 2

[EUMETSAT Data Channel 3]
target_directory=received&#92;Data Channel 3
______________________________


Cheers,
David
--
SatSignal software - quality software written to your requirements
Web: http://www.satsignal.eu
Email: david-taylor@...

Re: not yet perfect, but...

Alan Banks <alan@...>
 

On 04/03/2011 10:02, Ulrich G. Kliegis wrote:

Keep processing programs in their own folder and
not default 'Program Files'.
That one is new to me. You say, I should move, say, MSGadmin out of its
&#92;programs&#92;DJTsoftware&#92;MSG1&#92; context to, say, ..&#92;MSG1&#92; ?

What is the difference?

See David's reply.
I have the following structure and have no losses.
AMD Sempron 1800MHz.
Disk 0 (system) 80Gb
Disk 1 (Data) 200GB
2Gb RAM
320MB Ramdrive
and Win XP Home

Disk 0 has following folders
Windows
Program Files inc TechniSat DVB, Microsoft Security Essentials, Auslogics defrag (run once daily)

David_J_Taylor with GSS, 2 iterations of MSGDM, MSGAnimator
MSG-1&#92;images&#92;**
mrtg-2.16.3
Documents & Settings
myweb/documents

Disk 1
T-Systems
Received folders
I also use it to store a drive image of Disk 0 and a years worth of particular HRIT files I wanted and as a backup for some old files. Once defragged and optimized the unused stuff tends to be on the slower part of the drive and the head has less work to do write/erase the received folders (that's the theory anyway)

My feeling is that the writing from ramdrive to received folders is the critical issue. The less that harddrive has to do the better and the bigger the cache the better. I think mine is 16MB.

Also TURN OFF SYSTEM RESTORE
Sorry for the shouting, but that can take up a lot of resources.
Use one of the free disk imaging utilities and you don't need sytem restore.
See here for a choice of what's available
http://www.techsupportalert.com/best-free-drive-imaging-program.htm
Cheers

Alan

Re: not yet perfect, but...

Ulrich G. Kliegis
 

Yes, Ulrich, I can see the events on your graphs. Be careful with
multiple partitions on the same disk. If the disk head needs to move
half way across the disk to access the other partition, that could be
a greater penalty than having those files just one the same partition.
Yes. I thought about moving the virtual memory to the RAM disk, but then,
after some deep thinking... :)

Nope, not really.
I have a SSD on one of my new systems in the office, but that's reserved for
other stuff.

Using two physical disks is a much better solution - perhaps system
files and my software on one disk, EUMETCast and processed data files
on a second disk. In any case, get a disk with good performance - with
a 32MB or 64MB buffer.
See, I have three HDDs: C: has system and your software, D: only things
like setup-files, E: all MSG stuff (including the MSG-related receiving- and
received folders, F. the same for NOAA-AVHRR and Metop.

FSY of course on the RAM disk.

I would recommend having a real-time defragmenter running - with the
TelliCast 2.4.4 client software we have found mstDefrag to be very
acceptable, and have almost zero effect on received data (i.e. we
could not attribute any extra losses to running the defragmenter).
I've used version 2 and 3, but I see they are up to version 7 now.
How did they manage that?
Too many software fragments... :)

And since Windows 7, it's a fashionable must to have some 7 in the brand.

I follow the not more than a few steps at once strategy now, installed an
older mst defrag that I still had here (3.6) and started it. Well, it's doing its
work, and you all can see of it will decrease the missed packets count. The
degree of fragmentation on E: and F: was considerable...




http://www.mstsoftware.com/mst-Defrag
Both will defrag "when needed" by monitoring the disk state.
Auslogics will defrag no more than once per 12 hours, and Smart Defrag
not more than every three hours. If you haven't defragged your disk,
that could be the issue.
mst reagards idle times - although these do not really happen, but it seems
to stay rather politely in the background.

From the TelliCast configuration standpoint:

- you have checked that the FSY files are actually on your RAMdisk, I
suppose? But judging from your FSY graph they must be!
Well, I suppose they are the reason for the FSY files growing (after the
reboot last night) - there is no other explanation.


- I recommend using the common tmp_directory entry in the recv.ini
file, rather than individual entries in the recv-channels.ini file -
it is cleaner and makes the channels file simpler. Something like:

______________________________
[parameters]
tmp_directory=temp
______________________________
My rationale was: If I dispatch the data already in the beginning to the
correct partition, they don't have to be pushed around again. And the disk
heads have it a bit easier, maybe.


The directory should be on the same partition as the &#92;received&#92;
directories. As Alan mentions, all this should be /outside/ the
Program Files tree. I installed to C:&#92;EUMETCast&#92; in one Windows XP
installation. My recv-channels.ini file then has lines like:
Like in my case, see above.


Thanks again,

cheers,

U.

Re: not yet perfect, but...

Ulrich G. Kliegis
 

Alan,

thanks for the explicit description. I hope it will help others too. As you see
from my parallel posting, my setup is not that much different from yours.
Streamlining the data paths is essential, I remember the discussions about
this here some years ago.


Also TURN OFF SYSTEM RESTORE
Sorry for the shouting, but that can take up a lot of resources.
Use one of the free disk imaging utilities and you don't need sytem
restore.
Very good point (potentially, did not yet check that). Since I have acronis
true image taking a picture of the system disk each week, (and that is not
stored on the same drive!) I feel pretty comfortable, backup-wise (pun
intended).

After the defragmentation, I will do the next steps.

Cheers,
U.