Reducing packet loss
As you know I use GNU/Linux on ancient hardware with 8GB RAM only.
With these PCs I am able to receive all 3 services with good results.
Having a reasonable reception SNR is not what we are talking about.
On my PCs I had to
1) Disable all kind of C-States in the BIOS
2) Use two HDDs one for OS + locally processed images
and one for EUMETCast Basic, HVS-1 and HVS-2 (T1 + T2) raw data
3) I used a 3.5GB RAM disk for tmp and received + my script "mvmsg"
to move received to data HDD and to take care of damaged tmp files
4) I recently gave up the RAM disk taking care not to stress the data
HDD (now including tmp + received) too much anymore. I do this by:
- Omitting as far as possible du and find (re WINDOWS your trimtree
comes to mind and of course things like defrag and virus scanners)
- Sorting the data in a Channel/YYYY/mm/DD structure and tidy by day
- Using the big 8000000 socket channel buffers now proposed by EUMETSAT
- Having now most of the 8GB RAM for HDD raed/write buffering by Linux
5) A further significant step was to pin my busiest interrupts to seperate
processor cores. For those using routers I propose to pin the NIC
interrupt and all HDD interrupts to separate cores. It's rather easy
under Linux (I published a script on this list recently). Maybe someone
can come up with a similar solution and demonstrate results for WINDOWS.
I always had 2 PC systems though I still locally process all OLCI EFR data of
Sentinel-3A and Sentinel-3B on the receiver. The idea is not to move data
around too much. My receivers NFS export data for processing with PyTroll/SatPy
and EUMETCastView. Other than the receivers my processing PCs do not run 24/7.
An I make no attempt to process e.g. all GOES-16 data to make 4K animations.
I will use 4TB data HDDs in the future to get some more slack. I am convinced
that the two PC system is the future (Thorsten will object :-) route we should
take regarding the amount of data we already get and will soon get with MTGs.
After teething troubles with the latest Tellicast software update I’m now running smoothly with very little data loss. On the receiver I have a 1TB SSD for OS and basic service, and a second 1TB SSD for HVS-1. No RAMdrive.
I run TrimTree to remove anything over 48 hours old in the receiver data files.
The OS/Basic disk currently holds 68 GB, the HVS-1 drive, 4 GB. So disks are almost empty. But obviously a lot of writing and reading to them – I will be interested to see how long it takes to wear them out!
Hi David,toggle quoted messageShow quoted text
I take BAS, HVS-1 and HVS-2 with one PC. All data is flowing to a 9 GB RAM-Disk. The data from the RAM-Disk is processed by several progams and the results are stored at a external USB HDD with 4 TB. Sentinel 3 A/B have two 2 TB internal SATA HDD's for it's own.
I observe no data losses.
On Tue, Apr 9, 2019 at 12:08 PM, David J Taylor wrote:
David J Taylor
I had a question from someone about reducing packet loss when using Windows, but I felt that my advice might be out of date, so I would appreciate input here.
- in the past disk access speed has been the major problem for the Basic Service, which has been addressed by using a RAMdisk as detailed on my Web pages. As it works, I've left that on the systems here.
- for the High Volume Service HVS-1 I've had success using a separate hard disk drive for the data, and then either using that data directly on the receiving PC, or over the network on a second processing PC.
- I'm not really using HVS-2, but on one system I wrote the data to a RAMdisk, and on another to an HDD (not carrying BAS or system activity), and this seems to work well, too. The data is immediately deleted.
- I know that Arne van Belle has a "Write and Move" solution where he has a large RAMdisk and all three services use the same write to RAMdisk, then copy completed file to HDD. I tried something similar for HVS-1/2 using Robocopy and it also worked. However that PC now uses write HVS-1 to HDD, HVS-2 to RAMdisk.
With any modern PC CPU doesn't seem to be much of an issue for reception, although I notice that some recent Windows updates have caused CPU usage by the TelliCast process to increase. What is critical is minimising disk activity as mentioned in the EUMETSAT guides by stopping anti-virus, defrag, indexing etc. just for those disks (ideally all partitions on those physical disks). SSD I recommend just for the operating system, as taking the full data load onto a single SSD may wear it out prematurely.
OK, the group would like to hear from you on your successful solutions!
SatSignal Software - Quality software for you