Topics

IMPORTANT: Photos are eating up our storage

 

Before we left Yahoo TekScopes was using about 1½GB of storage for our
posts. This meant our storage needs were too large to qualify for the free
version of Groups.io. That was fine because their Premium plan gives us 10GB
for a reasonable annual fee.



Quite by accident I discovered the other day that we are using it up rapidly
and more storage is out of the question.

This is a breakdown of what we are doing with our 10GB:

* TekScopes is currently using 5.2 GB total storage out of 10GB

* Attachments: 12 kB - 0% of 10GB

* Photos: 4.4 GB - 44% of 10GB

* Files: 733 MB - 7% of 10GB



The problem is this: To get more storage space the next plan Groups.io has
is the Enterprise plan. It gives us 1TB. BUT THIS PLAN COSTS $200 PER
MONTH!!! Our current plan costs $110 PER YEAR. So the Enterprise Plan is not
an option.



Photos are eating up our storage at an unsustainable rate. I estimate that
that are gobbling up 1½GB to 2GB per year. At that rate we will use up our
storage in 3 to 4 years. In all probability it will happen sooner since
photos are getting bigger not smaller and they are becoming easier to take
and post.



One of the things I notice about the photos is how many of them were posted
to deal with a specific problem that has since been solved. Do we need to
keep the photos forever in cases like this? If not how can we insure they
are removed at an appropriate time.



I would appreciate your suggestions particularly if they are about how other
Forums handle this problem.



Dennis Tillman W7PF

Ilya Tsemenko
 

Photo archival could be life and death question for some of older equipment, as finding know-how or older equipment troubleshooting or even teardown photos to find out stuff like cable harness photos is always a problem. I'm sure everyone here was upset at least once,  finding some obscure bit of repair blog post about beloved Tek/Fluke/Keithley/HP gear but with missing files/photos/PDFs..

I can offer unlimited space for free for the preservation and archive purpose, if there is easy way to pull/mirror the group contents. Have own server with plenty of storage space, that is accessible online.

BR, Illya Tsemenko
xDevs.com

On 5/17/2019 1:49 AM, Dennis Tillman W7PF wrote:
Before we left Yahoo TekScopes was using about 1½GB of storage for our
posts. This meant our storage needs were too large to qualify for the free
version of Groups.io. That was fine because their Premium plan gives us 10GB
for a reasonable annual fee.


Quite by accident I discovered the other day that we are using it up rapidly
and more storage is out of the question.

This is a breakdown of what we are doing with our 10GB:

* TekScopes is currently using 5.2 GB total storage out of 10GB

* Attachments: 12 kB - 0% of 10GB

* Photos: 4.4 GB - 44% of 10GB

* Files: 733 MB - 7% of 10GB


The problem is this: To get more storage space the next plan Groups.io has
is the Enterprise plan. It gives us 1TB. BUT THIS PLAN COSTS $200 PER
MONTH!!! Our current plan costs $110 PER YEAR. So the Enterprise Plan is not
an option.


Photos are eating up our storage at an unsustainable rate. I estimate that
that are gobbling up 1½GB to 2GB per year. At that rate we will use up our
storage in 3 to 4 years. In all probability it will happen sooner since
photos are getting bigger not smaller and they are becoming easier to take
and post.


One of the things I notice about the photos is how many of them were posted
to deal with a specific problem that has since been solved. Do we need to
keep the photos forever in cases like this? If not how can we insure they
are removed at an appropriate time.


I would appreciate your suggestions particularly if they are about how other
Forums handle this problem.


Dennis Tillman W7PF



Glenn Little
 

Could photos that have been posted for over a predetermined time be resized?
This should free up storage area and retain the photos for history.
If the photo author is posted with the photo, and someone needs a higher resolution photo, they could contact the author.

Just an idea.

Glenn
WB4UIV

On 5/16/2019 1:49 PM, Dennis Tillman W7PF wrote:
Before we left Yahoo TekScopes was using about 1½GB of storage for our
posts. This meant our storage needs were too large to qualify for the free
version of Groups.io. That was fine because their Premium plan gives us 10GB
for a reasonable annual fee.


Quite by accident I discovered the other day that we are using it up rapidly
and more storage is out of the question.

This is a breakdown of what we are doing with our 10GB:

* TekScopes is currently using 5.2 GB total storage out of 10GB

* Attachments: 12 kB - 0% of 10GB

* Photos: 4.4 GB - 44% of 10GB

* Files: 733 MB - 7% of 10GB


The problem is this: To get more storage space the next plan Groups.io has
is the Enterprise plan. It gives us 1TB. BUT THIS PLAN COSTS $200 PER
MONTH!!! Our current plan costs $110 PER YEAR. So the Enterprise Plan is not
an option.


Photos are eating up our storage at an unsustainable rate. I estimate that
that are gobbling up 1½GB to 2GB per year. At that rate we will use up our
storage in 3 to 4 years. In all probability it will happen sooner since
photos are getting bigger not smaller and they are becoming easier to take
and post.


One of the things I notice about the photos is how many of them were posted
to deal with a specific problem that has since been solved. Do we need to
keep the photos forever in cases like this? If not how can we insure they
are removed at an appropriate time.


I would appreciate your suggestions particularly if they are about how other
Forums handle this problem.


Dennis Tillman W7PF




--
-----------------------------------------------------------------------
Glenn Little ARRL Technical Specialist QCWA LM 28417
Amateur Callsign: WB4UIV wb4uiv@... AMSAT LM 2178
QTH: Goose Creek, SC USA (EM92xx) USSVI LM NRA LM SBE ARRL TAPR
"It is not the class of license that the Amateur holds but the class
of the Amateur that holds the license"

Tom Gardner
 

Roughly how many photos are there in the 4.4GB?

Most of the photos I post on other forums don't need to be any larger than 1000*800, and they take ~250kb of storage. I find heavy clipping and then resizing them to be effective and acceptable in most cases.

Might it be appropriate to reject larger photos?
For the few photos that need to be larger, perhaps a (the?) moderator might need to OK them.

On 16/05/19 18:49, Dennis Tillman W7PF wrote:
Before we left Yahoo TekScopes was using about 1½GB of storage for our
posts. This meant our storage needs were too large to qualify for the free
version of Groups.io. That was fine because their Premium plan gives us 10GB
for a reasonable annual fee.


Quite by accident I discovered the other day that we are using it up rapidly
and more storage is out of the question.

This is a breakdown of what we are doing with our 10GB:

* TekScopes is currently using 5.2 GB total storage out of 10GB

* Attachments: 12 kB - 0% of 10GB

* Photos: 4.4 GB - 44% of 10GB

* Files: 733 MB - 7% of 10GB


The problem is this: To get more storage space the next plan Groups.io has
is the Enterprise plan. It gives us 1TB. BUT THIS PLAN COSTS $200 PER
MONTH!!! Our current plan costs $110 PER YEAR. So the Enterprise Plan is not
an option.


Photos are eating up our storage at an unsustainable rate. I estimate that
that are gobbling up 1½GB to 2GB per year. At that rate we will use up our
storage in 3 to 4 years. In all probability it will happen sooner since
photos are getting bigger not smaller and they are becoming easier to take
and post.


One of the things I notice about the photos is how many of them were posted
to deal with a specific problem that has since been solved. Do we need to
keep the photos forever in cases like this? If not how can we insure they
are removed at an appropriate time.


I would appreciate your suggestions particularly if they are about how other
Forums handle this problem.

Carsten Bormann
 

Could photos that have been posted for over a predetermined time be resized?
This should free up storage area and retain the photos for history.
It will also destroy all the crucial detail that can make the photos so valuable.

If the photo author is posted with the photo, and someone needs a higher resolution photo, they could contact the author.
Who by then might be a silent key.
(Or simply not have them any more — who here has perfect archival discipline with those?)


Note that we are talking about reaching 10 GB, within a few years.
10 GB is about 20 cents worth of storage today.
We don’t even know what the groups.io tiers and limits will be in 3 or 4 years.
(It might be worth alerting them to the fact that their tiers will stop working for us in that time frame.)

So apart from a general request to maybe use a little bit of discretion with respect to file sizes when posting photos, there is no need for destructive action at this point.

Grüße, Carsten

 

Hi Carsten,
You make a good point about the loss of what might be crucial detail if existing photos are compressed.
If we can limit the size of future photos (I have to check if that is possible) then it means the author will have to focus in on the important information he/she is trying to convey and crop the irrelevant surroundings.

I often find the pictures posted to our photos section have the briefest of explanations which leaves me wondering where I am supposed to direct my attention. My personal preference when documenting a problem is to placing photos in a PDF document with running comments alongside and in-between photos to explain what the reader is looking at. PDFs are more compact and the accompanying text serves two purposes: 1) As the author describes his/her photos in the document it forces them to think about what is going on and sometimes that leads them to the solution; 2) It provides the reader with more information which speeds up finding the cause of the problem.

Dennis Tillman W7PF

-----Original Message-----
From: TekScopes@groups.io [mailto:TekScopes@groups.io] On Behalf Of Carsten Bormann
Sent: Thursday, May 16, 2019 11:57 AM
To: TekScopes@groups.io
Subject: Re: [TekScopes] IMPORTANT: Photos are eating up our storage


Could photos that have been posted for over a predetermined time be resized?
This should free up storage area and retain the photos for history.
It will also destroy all the crucial detail that can make the photos so valuable.

If the photo author is posted with the photo, and someone needs a higher resolution photo, they could contact the author.
Who by then might be a silent key.
(Or simply not have them any more — who here has perfect archival discipline with those?)


Note that we are talking about reaching 10 GB, within a few years.
10 GB is about 20 cents worth of storage today.
We don’t even know what the groups.io tiers and limits will be in 3 or 4 years.
(It might be worth alerting them to the fact that their tiers will stop working for us in that time frame.)

So apart from a general request to maybe use a little bit of discretion with respect to file sizes when posting photos, there is no need for destructive action at this point.

Grüße, Carsten




--
Dennis Tillman W7PF
TekScopes Moderator

Carsten Bormann
 

On May 16, 2019, at 22:07, Dennis Tillman W7PF <@Dennis_Tillman_W7PF> wrote:

Hi Carsten,
You make a good point about the loss of what might be crucial detail if existing photos are compressed.
If we can limit the size of future photos (I have to check if that is possible) then it means the author will have to focus in on the important information he/she is trying to convey and crop the irrelevant surroundings.
Wrong direction of movement. The direction should be making it easier to contribute, not harder. While I’d love it when people to spend effort making good contributions, not everybody knows how to do this. (Ceterum censeo: if the images are sent as attachments, you have the context of the mail message, which is usually much more useful. Can we fix this?)

I often find the pictures posted to our photos section have the briefest of explanations which leaves me wondering where I am supposed to direct my attention.
There we are. Put them in the message, and there is a natural place for that explanation.

My personal preference when documenting a problem is to placing photos in a PDF document with running comments alongside and in-between photos to explain what the reader is looking at.
My personal preference is to do the same, but in a mail message.
(Which doesn’t mean your preference is wrong, but you can’t extrapolate it to all of even a majority of the contributors.)

PDFs are more compact
Generally only if they employ heavy compression, and there we are with the loss of detail.
There are many tools for creating PDFs, most of which don’t handle natural images (photos) very well.

and the accompanying text serves two purposes: 1) As the author describes his/her photos in the document it forces them to think about what is going on and sometimes that leads them to the solution;
Yes. Put them in the message.

2) It provides the reader with more information which speeds up finding the cause of the problem.
Yes. Put them in the message.

I think the photos section is great when people want to document something with an explicit archival intent, and that’s where creating PDFs can (but still often won’t) be a good approach. The Tek wiki may be a better place for this archival intent, though.

Grüße, Carsten

@0culus
 

Hi Dennis,

I haven't sent that many photos to the list galleries, but the ones I did I ran through Photoshop's "save for web and devices" filter which does a pretty respectable job of preserving quality while massively reducing file sizes. I do think that detailed technical writeups in PDF format are good, but PDF sizes can also grow significantly unless you typeset using LaTeX, which is something I do regularly but may not be everyone's cup of tea.

Perhaps the suggestion above that important repair notes should be archived on tekwiki would be a good compromise.

Sean

On Thu, May 16, 2019 at 01:07 PM, Dennis Tillman W7PF wrote:


Hi Carsten,
You make a good point about the loss of what might be crucial detail if
existing photos are compressed.
If we can limit the size of future photos (I have to check if that is
possible) then it means the author will have to focus in on the important
information he/she is trying to convey and crop the irrelevant surroundings.

I often find the pictures posted to our photos section have the briefest of
explanations which leaves me wondering where I am supposed to direct my
attention. My personal preference when documenting a problem is to placing
photos in a PDF document with running comments alongside and in-between photos
to explain what the reader is looking at. PDFs are more compact and the
accompanying text serves two purposes: 1) As the author describes his/her
photos in the document it forces them to think about what is going on and
sometimes that leads them to the solution; 2) It provides the reader with more
information which speeds up finding the cause of the problem.

Dennis Tillman W7PF

John Griessen
 

On 5/16/19 6:30 PM, sdturne@q.com wrote:
Perhaps the suggestion above that important repair notes should be archived on tekwiki would be a good compromise.
Sean

I agree that cultivating curation is the best realistic way forward, unless an AI
gets good at figuring what old EEs like or think is important. That seems thin/wishfull.
We're odd, weird, sometimes even subtle.

EJP
 

Put them in the message
Absolutely not. Pictures or indeed any attachments that are put into the messages are permanently and irrevocably invisible to those of us who use the Web interface. I've been bitten by this several times in other groups.

EJP

Carsten Bormann
 

On May 17, 2019, at 01:52, EJP <esmond.pitt@...> wrote:

Put them in the message
Absolutely not. Pictures or indeed any attachments that are put into the messages are permanently and irrevocably invisible to those of us who use the Web interface. I’ve been bitten by this several times in other groups.
Huh? I don’t think so. Here are a couple of examples from a group that stopped disabling images in messages when they moved to groups.io:

https://groups.io/g/HP-Agilent-Keysight-equipment/topic/31623552
https://groups.io/g/HP-Agilent-Keysight-equipment/topic/31620801

Maybe you can explain in more detail what problem you have.

Grüße, Carsten

Mark Goldberg
 

Other forums I use limit photos to a maximum dimension. I haven't found it
to be too much trouble to carefully crop and shrink a photo and still
retain critical information. Phones these days take photos with large file
sizes, but are often fuzzy. Shrinking them by 1/4 conveys sufficient and
often the same information.

So, limiting photo size is my vote.

Regards,

Mark

 

Hi Mark,
Thanks for the suggestion. Several others have suggested it too. I think it will have the smallest impact on our photo archive going forward and it appears to dramatically slow down the rate we eat up storage.

I did additional research today to get a more accurate picture of the problem and the news is not good if we do nothing. We took 2.5GB of archives, files, and photos with us when we left Yahoo. Since we moved to Group.io 18 months ago we have used an additional 2.7GB of storage. That comes out to 2.7GB in 1.5years = 1.8GB per year. At that rate we will use up our remaining 4.8GB of storage in 2.6 years, or at the end of 2021.

Currently we allow photos to be as large as 4096x4096. If we reduce the maximum size /resolution to 2048x2048 it reduces the storage needed for a photograph by 1/4 which will postpone the inevitable by 4.8GB / 0.45GB per year= 10.6 years or the end of 2029, all things being the same. I can live with that.

In the meantime I will be monitoring our storage usage on a regular basis.

Dennis Tillman W7PF

-----Original Message-----
From: TekScopes@groups.io [mailto:TekScopes@groups.io] On Behalf Of Mark Goldberg
Sent: Thursday, May 16, 2019 6:38 PM
Subject: Re: [TekScopes] IMPORTANT: Photos are eating up our storage

Other forums I use limit photos to a maximum dimension. I haven't found it to be too much trouble to carefully crop and shrink a photo and still retain critical information. Phones these days take photos with large file sizes, but are often fuzzy. Shrinking them by 1/4 conveys sufficient and often the same information.

So, limiting photo size is my vote.

Regards,

Mark




--
Dennis Tillman W7PF
TekScopes Moderator

Tom Gardner
 

On 17/05/19 06:36, Dennis Tillman W7PF wrote:
Currently we allow photos to be as large as 4096x4096. If we reduce the maximum size /resolution to 2048x2048 it reduces the storage needed for a photograph by 1/4 which will postpone the inevitable by 4.8GB / 0.45GB per year= 10.6 years or the end of 2029, all things being the
I hope I will still be alive in 2029.

The price break at 10GB is a price break, not a cost break. I doubt that it will change in future.

If people are mostly posting photos smaller than 4k*4k and we are eating up the storage quickly, then your guesstimate of 2029 would be /optimistic/. It is easier to impose a limit now than have an even more severe limit in the future.

EEVBlog limits photos in postings to "25 per post, maximum total size 2000KB, maximum individual size 1000KB". I'm not aware anybody has found they can't work with that. If you want to see how well that works, have a look at these randomly chosen examples:

* http://www.eevblog.com/forum/testgear/bk-precision-4030-pulse-generator-impressions-and-teardown/msg2183957/#msg2183957
* http://www.eevblog.com/forum/testgear/lecroy-dda-5005-dso-teardown/msg2132716/#msg2132716
* http://www.eevblog.com/forum/testgear/tektronix-2465b-oscilloscope-teardown/msg213890/#msg213890

For example, there are may photos of screenshots, I'd be surprised if any benefit from more than 640*480 - but people merely leave their camera on the largest photo size.

I really don't believe that people on this /technical/ group can't find a way to crop and scale their photos.

Personally I'd opt for 1280*1280 or 1MB per photo.

 

One thought might be (if we can) to limit the file size of the pictures that
are posted - in almost all cases a 1024*768 (ish) will more than suffice -
you don't need a monster 5k*2.2k (e.g.) image.

I agree that purging pix of photos relating to solved problems is s good
idea,. I'll dive in there and delete a few myself.

David

-----Original Message-----
From: TekScopes@groups.io [mailto:TekScopes@groups.io] On Behalf Of Dennis
Tillman W7PF
Sent: 16 May 2019 18:49
To: TekScopes@groups.io
Subject: [TekScopes] IMPORTANT: Photos are eating up our storage

Before we left Yahoo TekScopes was using about 1½GB of storage for our
posts. This meant our storage needs were too large to qualify for the free
version of Groups.io. That was fine because their Premium plan gives us 10GB
for a reasonable annual fee.



Quite by accident I discovered the other day that we are using it up rapidly
and more storage is out of the question.

This is a breakdown of what we are doing with our 10GB:

* TekScopes is currently using 5.2 GB total storage out of 10GB

* Attachments: 12 kB - 0% of 10GB

* Photos: 4.4 GB - 44% of 10GB

* Files: 733 MB - 7% of 10GB



The problem is this: To get more storage space the next plan Groups.io has
is the Enterprise plan. It gives us 1TB. BUT THIS PLAN COSTS $200 PER
MONTH!!! Our current plan costs $110 PER YEAR. So the Enterprise Plan is not
an option.



Photos are eating up our storage at an unsustainable rate. I estimate that
that are gobbling up 1½GB to 2GB per year. At that rate we will use up our
storage in 3 to 4 years. In all probability it will happen sooner since
photos are getting bigger not smaller and they are becoming easier to take
and post.



One of the things I notice about the photos is how many of them were posted
to deal with a specific problem that has since been solved. Do we need to
keep the photos forever in cases like this? If not how can we insure they
are removed at an appropriate time.



I would appreciate your suggestions particularly if they are about how other
Forums handle this problem.



Dennis Tillman W7PF

Miguel Work
 

And or resize big photos.


-----Mensaje original-----
De: TekScopes@groups.io [mailto:TekScopes@groups.io] En nombre de David C. Partridge
Enviado el: viernes, 17 de mayo de 2019 10:32
Para: TekScopes@groups.io
Asunto: Re: [TekScopes] IMPORTANT: Photos are eating up our storage

One thought might be (if we can) to limit the file size of the pictures that are posted - in almost all cases a 1024*768 (ish) will more than suffice - you don't need a monster 5k*2.2k (e.g.) image.

I agree that purging pix of photos relating to solved problems is s good idea,. I'll dive in there and delete a few myself.

David

-----Original Message-----
From: TekScopes@groups.io [mailto:TekScopes@groups.io] On Behalf Of Dennis Tillman W7PF
Sent: 16 May 2019 18:49
To: TekScopes@groups.io
Subject: [TekScopes] IMPORTANT: Photos are eating up our storage

Before we left Yahoo TekScopes was using about 1½GB of storage for our posts. This meant our storage needs were too large to qualify for the free version of Groups.io. That was fine because their Premium plan gives us 10GB for a reasonable annual fee.



Quite by accident I discovered the other day that we are using it up rapidly and more storage is out of the question.

This is a breakdown of what we are doing with our 10GB:

* TekScopes is currently using 5.2 GB total storage out of 10GB

* Attachments: 12 kB - 0% of 10GB

* Photos: 4.4 GB - 44% of 10GB

* Files: 733 MB - 7% of 10GB



The problem is this: To get more storage space the next plan Groups.io has is the Enterprise plan. It gives us 1TB. BUT THIS PLAN COSTS $200 PER MONTH!!! Our current plan costs $110 PER YEAR. So the Enterprise Plan is not an option.



Photos are eating up our storage at an unsustainable rate. I estimate that that are gobbling up 1½GB to 2GB per year. At that rate we will use up our storage in 3 to 4 years. In all probability it will happen sooner since photos are getting bigger not smaller and they are becoming easier to take and post.



One of the things I notice about the photos is how many of them were posted to deal with a specific problem that has since been solved. Do we need to keep the photos forever in cases like this? If not how can we insure they are removed at an appropriate time.



I would appreciate your suggestions particularly if they are about how other Forums handle this problem.



Dennis Tillman W7PF

Craig Sawyers
 

If we can limit the size of future photos (I have to check if that is possible) then it means the
author
will have to focus in on the important information he/she is trying to convey and crop the
irrelevant
There is another forum I'm on that does not allow images at all - it relies on the poster to use an
image hosting service. That kind of works apart from when a poster allows their hosting to lapse or
purges the images when their own storage plan runs out of space.

And there are other forums that have no apparent limit at all. But that is when it has an IT admin
that hosts the forum; and since multi-Terabyte fully RAID drives are cheap(ish), they are a data black
hole. Every now and again there is a need for new hardware or an additional drive, and there is a whip
round of the list members.

Not suggesting those as appropriate for TekScopes - just mentioning them as other ways that forums
skin the data storage cat.

Craig

Dave Wise
 

Dedicated cameras have a menu setting for picture compression.

The Apple iPhone does not provide this on the camera function itself, but it does allow you to compress if you email the photo. I've done this, it's easy.
Or you can take a screenshot of the picture. There are online guides for both procedures.

There are also several free online picture compression services. I've used them, when I took the picture at high resolution. Some of them can operate in batch mode.

Dave Wise

Greg Muir
 

As part of my work I am frequently having to take photos of situations to send to clients who are located elsewhere. I use a “normal” camera and not a smart phone for the best high-resolution and often telephoto photos. This gives me the ability to then reduce photos to a suitable size for both reports and emailing. I usually end up with the 800x600 format which has always met with good acceptance when determining detail both near (equipment innards) and far (higher elevations of communication towers, etc.). If the client needs more resolution, I will send them the appropriate higher format but have never been asked to do so yet.

From this I always make a second set of photos in the 800x600 format, down from the multi-megs of the original. That way if I need to examine a photo in great detail, I still have the original to rely on. The advantage of reduction with regards to email is that it fits nicely into the message when embedded and is convenient to view on a standard computer monitor rather than having to scroll back and forth, up and down to see the entire photo.

I often get photos from others taken with their smart phones which are of hog-at-the-trough resolution usually stretching the capability of the somewhat primitive optics with related aberrations that these devices use and are of little use even at that file size when determining fine detail.

Greg

Dale H. Cook
 

On 5/17/2019 12:50 PM, Greg Muir wrote:

I use a “normal” camera and not a smart phone for the best high-resolution and often telephoto photos. This gives me the ability to then reduce photos to a suitable size for both reports and emailing. I usually end up with the 800x600 format which has always met with good acceptance when determining detail both near (equipment innards) and far (higher elevations of communication towers, etc.). If the client needs more resolution, I will send them the appropriate higher format but have never been asked to do so yet.
I do the same. My Pentax DSLR, in RAW format, takes 6000x4000 pixel images that run about 29mb, depending upon what metadata is stored in an image. An 800x600 pixel JPEG reduction is usually sufficient, but I archive all originals.
--
Dale H. Cook, Radio Contract Engineer, Roanoke/Lynchburg, VA
https://plymouthcolony.net/starcityeng/index.html