Good morning everyone,

I'm looking for a pc that is economical in terms of power consumption and is still suitable for processing eumetcast data. Does anyone have a recommendation?

Best wishes,
Michael

Michael,

It's impossible to answer your question. You must be far more specifc:

What OS do you run?
What processing software do you (intend to) use?
Do you want to store raw satellite data for some days?
Is this PC also an EUMETCast receiver or a processing PC only?
What EUMETCast data do you want to process (we already test MTI-1 data)?

Regards,
Ernst

Good morning Ernst,

thanks for your answer. I run Windows 10 on my PC using David's software for processing. I normally store two days of raw satellite data. The PC is used for receiving and processing. The new hardware should be save to process future channel data as well.

At the moment my PC needs 4 KWh in 24 hours. (AMD Ryzen 1950, NVIDIA TITAN V and several SSD drives) The speed is perfect, but this should go more economical. 

 

Best wishes, Michael

Michael,

as all my receivers run headless under GNU/Linux and on rather old surplus HW I cannot recommend a certain recent PC model for a Windows all in one system. From my own experience and from what I see an I5 processor with 16 or better 32GB of RAM is enough to receive and process all data of (my and other amateurs) interest of all three current EUMETCAST Europe services. If you use big HDDs make sure that these use CMR (and never SMR!) recording. This has been discussed a couple of times on this MSG-1 list starting here:

https://groups.io/g/MSG-1/topic/75612642#29669

EUMETSAT recommends to strictly not do image processing on a receiver PC but on a well designed system it can be done at large as my systems show for GNU/Linux and among others Thorsten Miglus shows for Windows. There is actually little consensus on what the best way to setup an EUMETCast receiver systems is. It may also depend on whether you use DVB routers or PCIe cards.

https://groups.io/g/MSG-1/topic/9673430#26177

Clearly your system seems an overkill and about half of the energy you use should do. Your video card alone costs more than all my last 40 PCs :-) and will burn an important share of the electricity you need. Christian Peters has used low power systems under GNU/Linux and AFAIK Alan Curnow uses mini PCs and remote desktop under Windows 10.

As for the processing software be aware that David Taylor has said more than once that he will not support all the new stuff that's comming rather soon (MTG, Metop 2nd Generation). One route to go -- albeit with a certain learning curve -- is PyTROLL/Satpy. It works under GNU/Linux and Windows. It has been shown to run on Mac OS and even on Raspberry PIs.

Cheers,
Ernst

Good morning Ernst,

On Wed, Aug 10, 2022 at 01:35 AM, Michael Mehle wrote:

You think it is possible to run Satpy on a Pi?

Michael,

Christian Peters has done it on a Pi under GNU/Linux. In spring 2021 -- with Christian's help -- Nigel Heasman even made an image with my 3.0 Kit on a Pi with 2GB RAM only (image attached as proof of concept). There is no conda based PyTROLL/Satpy install though, you have to use virtualenv and pip as described in my HOWTO 2.0. And you should have 8GB of RAM. Unfortunately there is currently no way to run the TelliCast client on a Pi as there is no ARM compiled version of this COTS software.

Regards,
Ernst

... and here come the attachment. Sorry, my short time memory isn't what it used to be.

Hello,

All the info below refers to PyTroll.

I've just been researching this subject last night. My existing receiving+processing PC is an i5-4570 with 16GB of RAM currently and it's doing fine. Initially I want to test out moving to Python 3.11 which is supposedly faster (less time to process, less energy use). To quote them "Python 3.11 is up to 10-60% faster than Python 3.10". Ernst, perhaps you tested this already?

However, in terms of power efficiency, an entirely new PC would definitely do better, considering soon we'll have to deal with MTG data. Maybe standby power is going to improve a bit too (currently I'm getting around 35W when just receiving).

In this regard, I was looking at the options at both Intel and AMD. Since a graphics card is not really wanted due to increased power draw, I was exclusively looking at CPUs with an integrated GPU. It seems that AMD has a better price/performance ratio but due to the lack of lower end 'G' models, it is definitely more costly. As an entry level a Pentium Gold G7400 or maybe an i3-12100 would be ok, I think? For the higher end, I was looking at a Ryzen 5700G. Considering that in a few years, the MTG system is going to expand to its full potential, this kind of processing power might be necessary for all the imaging and sounding data. It might be better to invest in a higher end system now, rather than make e-waste later and buy a new one.

As for RAM requirements, I realized 8 GB is definitely not enough when using multithreading. Maybe for MTG data, 32 GB would become necessary? I still need to conduct a test with the test data provided.

I could definitely use some constructive feedback, since this is just my preliminary guesswork based on my current system.

Regards,
George

On Wed, Aug 10, 2022 at 06:02 AM, George Sz wrote:

To quote them "Python 3.11 is up to 10-60% faster than Python 3.10". Ernst, perhaps you tested this already?

George,

I have not tested Python 3.11 yet, I currently still use Python 3.8.13.
Satpy 0.37 just came out. No idea whether it runs with Python 3.11.
But I have heavily improved and speeded up my own scripts by using:

- in tmpdir caching of decompressed GEO slot data (MSGX, HIMA8)
- using whole lists of several composites generated in one go
- using delayed resampling (each channel is only resampled once)
- using GEO and OVR caching (the latter has been improved by Dave).

My next generation scripts also use a couple of fixes that I made to
Pycoast 1.5. Unfortunately there is no new Pycoast conda package yet.

The scripts run up to twice as fast under Devuan GNU/Linux compared to
the same scripts run under my Windows 10 PRO on similar hardware.

Regards,
Ernst

Hello,

I'm running a single receiving/processing PC (Windows 10) using a Ryzen 3200G and 16GB of RAM with a single 240GB SSD and 1TB HDD. I have no problems on BAS/HVS1 and HVS2 using SatSignal software even when performing additional processing tasks. Initially, I had hoped that I would not need a graphics card to keep the power consumption down but I experienced unacceptably high packet loss with an internal TBS 6983 card until I added an entry level passively cooled Nvidia card (GT710 based). I don't know if the situation would have been different with an external USB or Ethernet based EUMETCast receiver.  Average power consumption is now about 50W. One point to bear in mind is that a good quality power supply is needed that has high efficiency over the power range of interest to keep running costs down.

Noting that MTG is coming down the track, I tried Rob Alblas' MTG Satpy script with the MTG test data whilst monitoring memory usage. Although I produced an image successfully, I rapidly came to the conclusion that 16GB is not enough as virtually all available memory was consumed and that 32GB is definitely recommended for future use. Whether my existing processing power will be sufficient, only time will tell.....

Regards,
Richard

Thank you, Richard, Ernst, Kenan, George and Andreas for your insights due the topic. I think first I have to give PyTroll a try and separate receiving and processing on different machines with less processor power.

It would be interesting what will additional come with MTI-1 Data and how much additional precessing power for that will be necessary in the next years...

 

Best wishes,
Michael

 

On Fri, Aug 19, 2022 at 12:00 PM, Michael Mehle wrote:

It would be interesting what will additional come with MTI-1 Data and how much additional precessing power for that will be necessary in the next years...

Michael,

that will heavily depend on what you want to receive and process from EUMETCast. Already now I can overload all my processing power making GOES16/18 full resolution videos.
But it's of no use to make 5500 x 5500 pixel videos if I have only my FullHD monitor. Looking at the latest MTI1 testdata I expect to be safe with GNU/Linux, Satpy and 16GB RAM.

This link might be kind of frightening ...

https://www.eumetsat.int/evolutions-eumetcast-services-mtg-era

... but let's cross that bridge when we get to it.

Ernst

Ernst,

A while ago I realized it's not worth making videos out of geo images. Instead, I opted to saving JPEGs (with 4:4:4 sampling) and playing them in sequence using a simple web page. This way no re-encoding is necessary, and the "movie" plays in your browser. What I did not find yet is an off-the-shelf web player that supports seeking, pause, speed change, ping-pong etc.

As an alternate, (CPU) power efficient method, I imagine it would be possible to put these JPEGs into a movie file directly, without resorting to encoding. I'm almost certain ffmpeg can take a list of pictures and put them into an M-JPEG.

Regards,
George

George,

my direction to go is to define special satpy rss (frame) areas that have
about the pixel size of my monitor. Then I encode with ffmpeg to webm.
This is not especially CPU intensive and webm runs in every current
browser with the usual controls just a right click away. Webm videos
can also easily be integrated in HTML5. Here are some MSG3 webm movies
I make on a 13 years old receiver PC with a Core2 Duo proc and 8GB RAM.
All these movies are updated every hour. Hourly full resolution GEO
and LEO images for DAY and NIG are pictured as well. Ffmpeg can also
take full size GEO images and reduce frame sizes while encoding ...

https://we.tl/t-NeYo3XsKP1

wetransfer, link active one week


Cheers,
Ernst

Further to my previous post, I have since upgraded both my CPU and memory in anticipation of the need for more processing power with future EUMETCast services The CPU is now a Ryzen 5 5500 which has provided a noticeable improvement in overall performance. Surprisingly, the average power consumption of the PC when just receiving has dropped from about 50W to 40W. Using Satpy to decode MTG test data causes the CPU to hit 100% utilisation according to Task Manager but does not cause any packet loss. 

After installing the new CPU, I started to suffer significant packet loss at random intervals. Investigation revealed that I had fallen foul of a bug in Ryzen processors that can cause them to "stutter" and stop processing for a second or more which is disastrous when decoding a real-time data stream such as Tellicast. The bug is related to the internal Trusted Platform Module (TPM). One fix is to disable the TPM in the BIOS (a possible problem for Windows 11 users) and AMD has also provided a fix but this needs to be incorporated in the BIOS of a motherboard by an update. Follow this link for more information (AMD's Bugfix for Ryzen Stuttering Now Widely Available | Tom's Hardware (tomshardware.com)). I have applied both fixes (I don't use Windows 11) and the problem has stopped.

I did consider using an older PC as a separate receiving station. However, as this consumed significantly more mains power than the current receiving/image processing PC, I decided against it. The newer technology appears to be more power efficient.

Richard

Good morning Ernst,

I think about installing a complete new system based on linux, one computer for eumetcast reception, the other one for data precessing under PyTroll.

Are there existing preinstalled images? (I am no so familiar with linux at the moment. :)

Do you have an advice for the hardware of the two computers for me?

Have you heard about receiving and processing the date on two raspberry Pi's?

Thanks's for help

Best wishes,
Michael

Michael,

I'm not sure what you mean with "preinstalled images".
GNU/Linux comes in many flavours (= "Distributions").
You cannot go wrong if you take "Debian" that I use.

Download the  debian-11.4.0-amd64-netinst.iso  from:

https://www.debian.org/download

Burn it to a standard CD or make a bootable USB-disk.
Boot an old surplus PC and make "install" no graphics.
Install mc and get familiar with this CLI system and bash.
Running sshd you can admin that system from Windows 10
using a common ssh client program (PuTTY recommended).
If all this is rather "easy going" you are ready to move on.

Regards,
Ernst

P.S. I've answered your question re Raspberry PI further up.

On 07/09/2022 10:16, Ernst Lobsiger via groups.io wrote:

I'm not sure what you mean with "preinstalled images".
GNU/Linux comes in many flavours (= "Distributions").
You cannot go wrong if you take "Debian" that I use.

Ernst,

Likely he means one image to install Linux and have EUMETCast reception and
processing ready to go.

This is often done in e.g. Amateur Radio circles so that all the software
required (e.g. for multiple SDR programs and hardware support) is ready to run.
Typically for a Raspberry Pi requiring a 32 GB SD card to be flashed and
placed into the RPi. Easy!

Could be a popular choice!

Cheers,
David
--
SatSignal Software - Quality software for you
Web: https://www.satsignal.eu
Email: david-taylor@...
Twitter: @gm8arv