Date   
Re: Lossless HRV data

Douglas Deans <douglas@...>
 

----- Original Message -----
From: "David Taylor" <david-taylor@...>
To: <MSG-1@...>
Sent: Wednesday, October 29, 2003 2:36 PM
Subject: [MSG-1] Lossless HRV data


For 24 hours from 1300 today, EUMETSAT are transmitting HRV with
lossless Wavelet compression.

My initial impressions are:

- even with JPEG (quality 82) there was a noticeable improvement on
images which were histogram equalised to make full use of the
dynamic range.

- I needed to use PNG save format rather than JPEG to see the full
extent of the differences

- there is a definite improvement - detail is now visible that could
not be seen before and the blockiness is completely gone.

- it also shows that if you use Save visible as JPEG in the MSG Data
Manager then you maybe need a higher quality setting than the
default value of 82.

- The greater bandwidth consumed is obvious on the graphs! The 30-
minute average has gone from something over 108KB/s to around
138KB/s.

http://www.david-taylor.myby.co.uk/mrtg/hermes_dvb.html

Cheers,
David

Out of curiosity I turned the JPEG quality up to 100. Added nearly 12Mb on
to the save file size. Didn't realise it would be that dramatic. I
suppose 100% on JPEG should be about the same as PNG... it more or less is
from a file size point of view.


Regards
Douglas.

Lossless HRV data

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

For 24 hours from 1300 today, EUMETSAT are transmitting HRV with
lossless Wavelet compression.

My initial impressions are:

- even with JPEG (quality 82) there was a noticeable improvement on
images which were histogram equalised to make full use of the
dynamic range.

- I needed to use PNG save format rather than JPEG to see the full
extent of the differences

- there is a definite improvement - detail is now visible that could
not be seen before and the blockiness is completely gone.

- it also shows that if you use Save visible as JPEG in the MSG Data
Manager then you maybe need a higher quality setting than the
default value of 82.

- The greater bandwidth consumed is obvious on the graphs! The 30-
minute average has gone from something over 108KB/s to around
138KB/s.

http://www.david-taylor.myby.co.uk/mrtg/hermes_dvb.html

Cheers,
David

Re: resolution

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

--- In MSG-1@..., "Guy Martin" <agm@t...> wrote:
I may have missed the discussion about this but when animating
HRV, as the terminator moves across and the pictures descends into
darkness the resolution goes to 'pot', square edged chunks of dark
grey and black. I can see this effect on some of the LRIT images as
well. Is this an effect of the lossy compression or the fact that
graphics cards only use 8 bits even for a greyscale.

Guy,

Both LRIT and HRV normally use lossy JPEG compression. This may
account for the artefacts you are seeing. See my post on HRV.

Cheers,
David

Re: Single-machine statistics

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

Sorry this is rather long, but I felt the data might be useful
and of general interest.

Best regards - Alan

Alan Sewards
Alan,

Thanks for that. I haven't digested it all, by any means, but Sam
Elsdon has also been looking with Performance Monitor and I hope he
write up his findings!

Some checks:

- ensure that you minimise the number of HRIT channels you collect.
Only collect what you need.
- Don't take LRIT if you don't need it.
- be sure to defrag your disks regularly
- use NTFS and not FAT for the disk structure.


Having said that, with the current beta version I am currently
collecting all 12 channels, and I see a Commit charge in task
manager of 530MB as I write. In addition to the MSG Data Manager, I
have running:

- Explorer
- Internet Explorer
- Speaking clock
- MRTG (runs under Perl)
- Outlook Express
- Front Page
- Zome Alarm
- Anti-virus

I often run Borland's Delphi for development, and various office and
imaging products as part of the development process. All this under
Windows XP SP1. The machine only seems to be affected by saving the
HRV image (in JPEG format), and I'm almost sure it didn't used to do
that, and suspect that one of the security patches is responsible.
Saving an HRV image in PNG format does take time - I've just been
doing that as part of the current "lossless HRV" tests.

I don't see why you are quite so worried by the commit charge value
you are seeing.

As to your recommendations, I believe the Web site already says to
have 1GB if you want to process HRV (channel 12) data, you fidings
do support that.

Cheers,
David

Re: Missing HRV Sector 17

Peter Benney <tugboat@...>
 

Task Scheduler was running Symantec NetDetect at approx 13-15 minutes past the hour.
Should read "running NetDetect every 5 minutes commencing 1315 past the hour"

Peter

Re: Missing HRV Sector 17

Peter Benney <tugboat@...>
 

If it's not the MSG Data Manager - do you have a process running
every 15 minutes on your PC which might be interfering with
reception.
Task Scheduler was running Symantec NetDetect at approx 13-15 minutes past the hour. I was unaware that Task Scheduler was running and have now disabled it.

Fingers crossed !

Peter

Single-machine statistics

Alan Sewards <alan.sewards@...>
 

Hi David,
As you know, I have been persevering with a single-machine setup, but
recently, I have been forced to make some changes. I thought you would be
interested in the results. I am running the 2.3.0a version of the
SkyStar2/Tellique software, MDM 1.2.4.255 and Animator 1.1.8.33.

I have been aware for some time that the wxsat machine has been
sluggish, with waits for the menus to pop up, jerky animations, etc. but it
all got too much last week and I checked out the Performance Monitor and
found to my horror that the pages writes per second were up in the hundreds
if not thousands! The Average Disk Queue was getting pretty long too but the
CPU was still handling things OK. So I decided to order another 512 MB RAM.
I expected that would arrive within a couple of days, but when it did not I
decided to connect my systems to run as a two-machine setup, with one PC for
the receive function and MDM, Animator, etc on the second. This proved to
have surprisingly little effect, the machine with MDM (but only 512 MB RAM)
was almost as bogged down as the single-machine had been, and as this
machine is my daily workhorse, this was not acceptable! Typical stats for
this (two-machine) setup are: Pages/sec: Avg 880, Disk Q: 2.5 Avg, CPU: 28
Avg. After a few days with still no new RAM, I changed back to the single
machine again (with 512 MB RAM). Stats for this were: Pages/sec: 1247 Avg,
Disk Q: 0.1 Avg, and CPU: 18 Avg. Finally the RAM arrived and I installed
it with these results: Pages/sec: 0.4 Avg, Disk Q: 0.009 Avg, CPU: 23 Avg. I
was surprised to find that the Commit Charge with 1GB RAM was still as high
as 404 MB, as the sum of memory needs for all processes did not seem to come
anywhere near 1024 MB. Snapshots later and following morning gave the
following: Pages: 3 Avg, Disk Q: 0.2 Avg, CPU 35 Avg; and pages: 0.4, Disk
Q: 0.005, CPU 18. Later I ran the Animator and the following resulted:
Pages: 0.4, Disk Q: 0.011, CPU: 27. Commit Charge went up to 490 MB. A bit
later (just after the big save at the end of a cycle) I got the following:
Pages: 578, Disk Q: 3, CPU: 78, but within a few seconds things were back to
the more normal figures quoted above.

My comments on these figures are as follows:
1. There is a HUGE difference between a machine with 512 MB and 1024 MB when
it comes to running MDM. With only (!) 512 MB RAM, the machine quickly
becomes bogged down in paging to disk, not perhaps to the point of losing
data, but certainly to the point of becoming a pain to use!
2. It seems essential that the machine running MDM has 1 GB of RAM. If it
does, and is fast enough (mine is Athlon XP2400+), it can handle as well the
receive function without noticing.
3. I don't understand why with 1 GB RAM, such a large Commit Charge is
required. Where is all this memory being used? MDM is shown as taking up
200-350 MB depending on when you look, but the sum total of all the other
processes running does not reach that figure.

Sorry this is rather long, but I felt the data might be useful and of
general interest.

Best regards - Alan

Alan Sewards
email: alan.sewards@...
web site: http://asewards.free.fr

Re: Missing HRV Sector 17

Peter Benney <tugboat@...>
 

In all of my recent testing I have both the Upper and Lower boxes
checked for the Centre HRV window option on the Advanced setup.
It might be interesting to know what you have set, and if setting those
options removes the problem.
Upper and lower boxes checked.

If it's not the MSG Data Manager - do you have a process running
every 15 minutes on your PC which might be interfering with
reception.
I have Zone Alarm running on the RX PC.

If it _was_ the MSG Data Manager, I would have thought
that other people would have seen the problem as well.
Definately not the Data Manager David as the Tellique log on the RX PC shows that the files were not received.On the 0615 cycle two files were reported missing or incomplete and I would have expected to see this on the other cycles.

Peter

Re: Missing HRV Sector 17

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

--- In MSG-1@..., "Peter Benney" <tugboat@g...> wrote:
I seem to have a problem receiving this HRV sector. This morning
it is missing from the 0530, 0600, 0615 and 0700 cycles. The
Tellique log appears normal except for the 0615 cycle when a missing
or incomplete file warning is logged between the reception of
sectors 16 and 18. No clues in the DM log. This is happening quite
regularly with missing sector 17's on most days.

Any ideas ?
Peter,

That sector is typically the one where the switch is made from the
lower window of the scan to the upper. In all of my recent testing
I have both the Upper and Lower boxes checked for the Centre HRV
window option on the Advanced setup. It might be interesting to
know what you have set, and if setting those options removes the
problem.

If it's not the MSG Data Manager - do you have a process running
every 15 minutes on your PC which might be interfering with
reception. If it _was_ the MSG Data Manager, I would have thought
that other people would have seen the problem as well.

Cheers,
David

Missing HRV Sector 17

Peter Benney <tugboat@...>
 

I seem to have a problem receiving this HRV sector. This morning it is missing from the 0530, 0600, 0615 and 0700 cycles. The Tellique log appears normal except for the 0615 cycle when a missing or incomplete file warning is logged between the reception of sectors 16 and 18. No clues in the DM log. This is happening quite regularly with missing sector 17's on most days.

Any ideas ?

Peter

resolution

Guy Martin <agm@...>
 

I may have missed the discussion about this but when animating HRV, as the terminator moves across and the pictures descends into darkness the resolution goes to 'pot', square edged chunks of dark grey and black. I can see this effect on some of the LRIT images as well. Is this an effect of the lossy compression or the fact that graphics cards only use 8 bits even for a greyscale.

regards, Guy

----------

http://www.gordano.com - Messaging for educators.

Re: IMPF

Luca Bertagnolio <lucaberta@...>
 

Actually what happened was, they skipped all the 7th HRIT segments
and HRV segments 19 thru 23, so the final segments were all
disseminated, and MDM recorded the image normally.

Bye, Luca

--- In MSG-1@..., "Douglas Deans" <douglas@d...> wrote:
As advised the IMPF changed occurred at 13.00. As suggested the
data is
definitely earlier, and to allow for that the 12.45 UTC scan was
cleverly
ended early by, I think, sending blank final sectors. So David's
program
did not need to wait.
Looks good.

Douglas.

Re: IMPF

Luca Bertagnolio <lucaberta@...>
 

--- In MSG-1@..., "David Taylor" <david-taylor@b...>
wrote:
Luca, that usually means that the OS has created the file, but
nothing has yet been written to it. Unusual, yes. Perhaps
Tellique
took its time writing the file data?
sure, that's what I thought, but the odd thing was that the 0k file
was surrounded by other files with reasonable lentgh, and there were
some three 0k files in a couple of minutes time.

We'll see later, when I turn on MDM, if those files are OK or not.
Running the usual 8% signal strength today... ;-)

Bye, Luca

Re: Tellique software...........

John Parkins <kvp@...>
 

Hello Alan,

No the log didn't help at all on one line it was downloading a file
the next line was when I restarted it! Oh well at least I know it's
not just me. In fact I had been rather impressed with that machine, it
had been running for nearly a month since it's last re-boot.

Tuesday, October 28, 2003, 3:39:25 PM, you wrote:

AS> John,
AS> I have had this happen twice, both times I reported it here, the second
AS> email is quoted below FYI (17/10/2003).

AS> Hi all,
AS> This morning at 0653 UTC, the tellique software shut itself down after
AS> having problems with the child process, almost identical to that which I had
AS> a few days ago and reported here. I did not notice this until a couple of
AS> hours later, and tried to restart tellique but it would not restart. I had
AS> to shut down the computer and restart it, whereupon tqrecv would start and
AS> run and everything else could be got back running again. I wish I knew the
AS> cause of this, but whatever the cause it does show that tqrecv is not
AS> robust. Perhaps the newer version (is it officially approved by Eumetsat
AS> yet?) may be better.

AS> Oh, and HRV is back over Europe, thank you!

AS> Best regards - Alan
AS> BTW: the Tellique log showed the shutting down of tqrecv when it had lost
AS> its child process. what did your log say?

AS> Best of luck - Alan

--
Best regards,
John mailto:kvp@...

Re: Tellique software...........

Alan Sewards <alan.sewards@...>
 

John,
I have had this happen twice, both times I reported it here, the second
email is quoted below FYI (17/10/2003).

Hi all,
This morning at 0653 UTC, the tellique software shut itself down after
having problems with the child process, almost identical to that which I had
a few days ago and reported here. I did not notice this until a couple of
hours later, and tried to restart tellique but it would not restart. I had
to shut down the computer and restart it, whereupon tqrecv would start and
run and everything else could be got back running again. I wish I knew the
cause of this, but whatever the cause it does show that tqrecv is not
robust. Perhaps the newer version (is it officially approved by Eumetsat
yet?) may be better.

Oh, and HRV is back over Europe, thank you!

Best regards - Alan
BTW: the Tellique log showed the shutting down of tqrecv when it had lost
its child process. what did your log say?

Best of luck - Alan


----- Original Message -----
From: "John Parkins" <kvp@...>
To: "MSG-1 Mailing List" <MSG-1@...>
Sent: Tuesday, October 28, 2003 4:17 PM
Subject: [MSG-1] Tellique software...........


Hello,

Erm.... just noticed that for some reason the Tellique software just
stopped at 23:56 last night. It's running on it's own computer
(1800+AMD WinXP) and hasn't done this before. Anyone else had the
software just stop? No indications in the log and the computer and
Skystar card are working fine.



--
Best regards,
John mailto:kvp@...





To unsubscribe from this group, send an email to:
MSG-1-unsubscribe@...



Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/



Re: IMPF

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

hmmm... how strange, I had a bunch of files with 0k length, but I
just checked now and they are of proper length. Just hit F5 and
their correct size appeared. Never seen this behaviour before.

FYI, I monitor my receiving PC at home via VNC over my VPN link
between the router at home and the office, but as I said I had
never
seen a 0k file before.
Luca, that usually means that the OS has created the file, but
nothing has yet been written to it. Unusual, yes. Perhaps Tellique
took its time writing the file data?

Ciao,
David

Tellique software...........

John Parkins <kvp@...>
 

Hello,

Erm.... just noticed that for some reason the Tellique software just
stopped at 23:56 last night. It's running on it's own computer
(1800+AMD WinXP) and hasn't done this before. Anyone else had the
software just stop? No indications in the log and the computer and
Skystar card are working fine.



--
Best regards,
John mailto:kvp@...

Re: IMPF

Douglas Deans <douglas@...>
 

----- Original Message -----
From: "David Taylor" <david-taylor@...>
To: <MSG-1@...>
Sent: Tuesday, October 28, 2003 2:44 PM
Subject: [MSG-1] Re: IMPF


Channel 6 HRIT also looks a lot better. The interpolation
artefacts are almost gone now but at the expense of quite a loss on
the west side.

Presumably, you don't have the "Calibrate thermal" set for channel 6?
Oops you are right !!

Hadn't noticed the west side loss.
Still a nice balanced disc now.
HRV outlines now spot on.
The change has certainly rounded off the few remaining sharp edges...
excellent.

Regards
Douglas.

Re: IMPF

Luca Bertagnolio <lucaberta@...>
 

--- In MSG-1@..., "David Taylor" <david-taylor@b...>
wrote:
No, I didn't see that. What were the file names? (There should
never be zero length files, as all files should have a header).
hmmm... how strange, I had a bunch of files with 0k length, but I
just checked now and they are of proper length. Just hit F5 and
their correct size appeared. Never seen this behaviour before.

FYI, I monitor my receiving PC at home via VNC over my VPN link
between the router at home and the office, but as I said I had never
seen a 0k file before.

Bye, Luca

Re: IMPF

David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺
 

I am not running the decode PC now, so I am just monitoring the
downloaded data. Did you noticed to 0k files in both HRV and HRIT
channels around 14:12?

Bye, Luca
No, I didn't see that. What were the file names? (There should
never be zero length files, as all files should have a header).

Cheers,
David