Re: Gaps in AVHRR images yesterday.


Thorsten Miglus
 

Hi all,

now I am using a 128 GB external SSD with USB 3.0 for all temp data from BAS, HVS-1 and HVS-2.
All target directories are situated on the SSD too. There is no need for Robocopy, because David's Software delete the files from the target directory after processing.

In Front of the SSD I have installed PrimoCache. (PrimoCache download)
Using a cache of 3 GB and with Defer-Write enabled (Native Mode INFINITE) nothing is written to the SSD at all.
Native Mode INFINITE means that data is written to the SSD when the cache is filled up only.
Under normal conditions this never happens because the files are deleted immediately after processing.
In situations, when broken temp files arrive, they are written to the SSD when the cache filles up.
As there is 128 GB room on the SSD I have not to take care of broken temp files.

Cheers,
Thorsten












On Thu, Sep 24, 2020 at 05:02 PM, David J Taylor GM8ARV 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🇪🇺 wrote:
John,

I am not completely happy with SSDs for EUMETCast data because they have a limited number of write cycles, a limited total data throughput, resulting in a limited lifetime. Perhaps with today's SSDs these limits are such that it's an acceptable choice,and if the user is aware of that, fine. Remember that there's not only receiving the data, but there might be processing as well. For HVS-2 it's something like 450 GB per day from the source.

HDD - slow
SSD - fast
RAMdisk - much faster

But RAMdisk is limited in capacity and may need more management. Some of my systems are now using RAMdisk (1-2 GB for BAS, 10 GB for HVS-2 [receive-only]) with Robocopy moving the data from the RAMdisk onto HDD once a minute, and with TrimTree running regularly to remove any lingering "tmp" files. One HVS-1 system writes directly to a separate HD, One BAS system works well without any RAMdisk.

It's other things which can help too - preventing any unwanted disk access from e.g. indexing and anti-virus. Minimising the data you take.

One thing which I'm increasingly finding critical is to maximise the number of data paths through the system. Perhaps add a second HDD controller rather than just another SATA connection. I'm currently thinking about adding a second USB internal hub for the TBS5925 as the PC has a single master hub. There's an element of "try it and see" about this level of changes.

Cheers,
David

Join MSG-1@groups.io to automatically receive all group messages.