Updates to the Groups.io service.
February 21, 2020
- NEW: Member search now also searches moderator notes.
- BUGFIX: In some cases when we do a reverse DNS lookup on the IP address of a server sending us email, the Go DNS library would return a `server misbehaving` error, which we treated the same as not having a reverse DNS record, causing us to reject the incoming message. In those cases, we now repeat the lookup using an alternate DNS library. Specifically, China Mobile email servers seem to suffer from this.
- BUGFIX: For some queries, wiki search would return a blank second page.
- BUGFIX: For search queries with URL encoded characters, we would not properly decode them correctly on the subsequent search pages.
- API: Misc improvements to the documentation.
- API: Added "bad_request": "no space" errors to `/addphotos`, `/updatephoto` and `/addfiles` endpoints.
- CHANGE: Moderators now cannot change the role of Owners.
- CHANGE: Moderators now cannot change the subscription options of Owners.
- API: Added support for `extended=true` to the drafts API endpoints.
- API: `/updatemember` can now return: "bad_request": "moderators cannot change role of owners" is returned if a non-owner attempts to change the `mod_status` field of an owner.
- API: `/updatemember` can now return: "bad_request": "moderators cannot make members owners" is returned if a non-owner attempts to make a member an owner.
- API: `/updatemember` can now return: "bad_request": "moderators cannot change owner subscription settings" is returned if a non-owner attempts to change an owner's subscription settings.
February 14, 2020
- INTERNAL: Added additional monitoring of the search cluster.
- INTERNAL: Switched over to a new Elasticsearch 7 search cluster. All of this work around search has multiple goals:
- INTERNAL: Re-indexed all of our data in a new search cluster.
- BUGFIX: Found a file descriptor leak in some networking code.
- INTERNAL: Began deleting attachments in groups that are over their storage allotment.
- CHANGE: Changed more instances of `subscriber` to `member` on the website.
February 7, 2020
- CHANGE: Standardized on the term `member`; we used `subscriber` and `user` previously. Several tweaks to the group settings page. Changed NuM to NMM.
- INTERNAL: Much work to support moving from Elasticsearch 5 to Elasticsearch 7, including optimizations for full reindexing.
- BUGFIX: Introduced and then fixed a bug that caused errors when trying to join a subgroup.
- CHANGE: The text when creating a group said that new members would be NuM 2, but the group was created with NuM 1. All new groups are now created with NuM 2.
- API: Removed `sub_group_access` from `/creategroup` endpoint.
- CHANGE: Removed the Create Subgroups dropdown when creating a new group since subgroups are now a premium feature.
- INTERNAL: Switched from one SSL certificate for all enterprise domains, to individual certs for each domain.
- BUGFIX: Changed how we sort messages in the Messages/Expanded messages view to use message created date instead of object date, to account for groups that had messages imported that were not strictly date ordered (or that already had messages in the archives).
- NEW: Rate limiting joining groups and other things to help detect/prevent the `mass subscription` attacks we were seeing this week.
January 31, 2020
- INTERNAL: Better tracking of which mechanism a user is added to the system, to aid in spotting shenanigans.
- SYSADMIN: Added a lot of storage to increase the amount of historical logs we can keep.
- INTERNAL: Work on converting to elasticsearch 7.
- INTERNAL: Optimizations to the change data system that ingests data into the elasticsearch cluster. We were replaying a lot of old LSNs after each restart that we didn't need to.
- CHANGE: In the email announcing a pending subscription, change `rejected` to `deleted` to explain what happens after 14 days.
- SYSADMIN: Added an additional 6 nodes to the search cluster because searches were taking too long.
- BUGFIX: After searching/finding a photo, the Download button did nothing.
- BUGFIX: Viewing the Emailed Photos album using any sort order other than Posted would lead to the wrong order when viewing individual images.
- CHANGE: For Enterprise SSO, first redirect through the groups.io domain, then to the SSO provider, like we do with Facebook/Google.
- API: Added `start_time` parameter to `/downloadarchives` endpoint.
- API: Updated docs on `/updategroup` endpoint, adding some missing parameters.
- NEW: New premium feature, Send Invites On Join, sends an ICS file of all calendar invites when someone joins a group, and sends an ICS file of cancel notices when someone leaves a group.
- BUGFIX: Re-write of how we were generating calendar ICS files to clean up some technical debt but more importantly to generate more correct files in regards to repeating events. Previously the ICS files just contained single events for each instance of a repeating event instead of specifying the repeating event itself. This led to issues when cancelling events, both via ICS files as well as when someone tried to delete an event in their (third party) calendar.
- BUGFIX: Flash notices (like 'You are not allowed to create any more groups today') were not displayed in some instances when your home page was /groups.
- CHANGE: A group that is not restricted can no longer have `Members Visible` set to `Subscribers`, unless it is a subgroup and the parent group is restricted. This is prevent email address harvesting.
January 24, 2020
- INTERNAL: Cleaned up instances of the same hashtag appearing twice in the topic.
- BUGFIX: If sending to an announcement subgroup where you are not a moderator but you are a moderator of the parent group, the message would bounce.
- BUGFIX: When banning an existing member by entering their email address manually, we did not save their moderator notes, owner messages, etc. Now, it does the same thing as banning a member using the Actions dropdown when viewing the member list.
- NEW: Added another Yahoo hard bounce error code.
- INTERNAL: Rearchitected how we paginate through messages, to be more efficient with the database.
- SYSADMIN: Removed the extra block storage needed during the Yahoo Groups migration.
- MISC: Hired a technical writer to draft a Groups.io manual.
- CHANGE: Removed Canny.io integration.
- BUGFIX: The messages in the mbox file when exporting a group were not in ascending order, as required.
- CHANGE: Optimized drafts database queries.
- MISC: Hired a consulting firm to help with search issues, including member directory search and upgrading our existing cluster.
- CHANGE: Reorganized the Beta group hashtag list. Established a new group charter.
- NEW: In the Hashtags page, added an icon to indicate mods only hashtags.
- INTERNAL: Added some database indices to speed up topic deletion.
- INTERNAL: Optimized several search queries.
January 17, 2020
- BUGFIX: Fixed a deadlock issue in some cases during confirmation via email.
- INTERNAL: Addressed some technical debt around the calendar code, and in the process fixed two timezone conversion bugs in ICS feeds.
- API: The `message` object now has an `attachments` array.
- BUGFIX: The `your subscription is not yet approved` banner was appearing one page refresh too late.
- BUGFIX: Edit buttons were missing from the Hashtags page.
- API: The docs for `/createhashtag` incorrectly stated that `description` was required.
- BUGFIX: Fix crash when sorting a database by a column that no longer exists.
- BUGFIX: The user was not being displayed in `Message rejected from not confirmed member` activity log lines.
- API: `/downloadarchives` now supports `start_msg_num` parameter.
- NEW: When downloading archives either through the API or through the export group function, each message now has a `X-Groupsio-MsgNum` header with the message number of the message.
- API: `/newdraft` now supports `draft_type_dm`.
- NEW: Include descriptions in the list of hashtags on the New Topic page.
- APP: Created new App subgroup and got approval from Apple for the first test version of the app.
- BUGFIX: If a pending member was rejected on the group member screen, they were marked as Removed in the Past Members page, when they should be marked as Rejected.
- BUGFIX: The API call /directadd wasn't returning full `member_info` object information.
- CHANGE: Increased concurrency of the process responsible for handling email delivery status updates because it was bottlenecking email delivery due to our ever increasing email delivery numbers.
- BUGFIX: We were not handling emails with header lines that weren't properly capitalized correctly.
- BUGFIX: Editing a malformed message imported from Yahoo that was missing a body didn't work.
- CHANGE: Removed Facebook Integration from the group Integrations page.
- NEW: Added additional checks for our DNS servers to ensure that we catch any DNS issues.
- NEW: Added new `Messages that start new topics are moderated` group moderation option.
January 10, 2020
- BUGFIX: When a poll was created and moderated, if a group moderator edited the poll before approving it, the poll ownership was mistakenly changed to the moderator.
- CHANGE: All premium groups now have 20GB of storage.
- CHANGE: The 'Note Sent To Attendees' email sent to people reply yes to an event, as well as the email notifying someone they have been moved from the waitlist to the attending list, were sent from the +owner email address. They are now sent as coming from the creator of the event.
- INTERNAL: Wrote a new utility to do some consistency checking and cleanups of the various databases.
- API: We are now ratelimiting the `/downloadarchives` endpoint because of abuse.
- INTERNAL: Developed a new system to make it easier to grep through all the various logs to research problems in the system.
- INTERNAL: Switched from a redis-based rate limiting system to a custom system.
- BUGFIX: When changing your email address, we were sending the warning email to the new email address when we should have been sending it to the old email address.
- CHANGE: Display the updated date in the 'Uploaded' column in the files section.
- CHANGE: Previously, when a group had `separate footers` checked, we would not try to strip out any remnant footers from incoming replies. Because some mailers combine multiple text parts when replying, we still need to look for these footers in replies, otherwise unsubscribe links could be inadvertently sent out.
- BUGFIX: For some reason, a few people started receiving duplicate summaries the last couple of days. I refactored the code that generates the summaries and that fixed the problem even though that code hadn't been touched in months.
- NEW: Added new Event Summary Email interval: Bi-Weekly on Fridays.
- CHANGE: Changed Zendesk to send back an acknowledgement when a support email is received.
- BUGFIX: For enterprise groups without a custom home page, setting `disable sign up` to true would result in an endless redirect.
- CHANGE: Modified message import code to support mbox files from Yahoo Group's export system.
- API: Added `/updatewikipage`, `/deletewikipage` and `/deletewikirevision` endpoints.
- BUGFIX: For people on `special messages only`, the message footers wrongly said `You receive all messages sent to this group.`
- NEW: Special messages are now sent with the subject tag `[Special]`.
- API: Changed `moderated` to `moderation` in the Group object.
- NEW: Added additional information to the `Sent message requiring approval` activity log line, specifying exactly why a message was moderated. This duplicated the `Message from member required approval because of virus` line, which I have removed.
- SYSADMIN: Upgraded hot spare database instance to have the same resources as our main database.
- INTERNAL: Switched from a redis-based session tracking system to a custom system.
- BUGFIX: Fixed line breaks in the index in the plain text digest.
- API: Added `plan` field to `group` object.
- API: Added `nice_group_name`, `cover_photo_url`, `icon_url`, `subs_count`, `most_recent_message` fields to the `member_info` object.
- API: Changed the sort options for the `/getsubs` endpoint and added the `set_home_page` parameter.
- API: First pass at `/getdatabaserows` endpoint.
- NEW: Added 'Deleted Account' activity log entry for when people delete their accounts.
- API: Added `is_reply` field to `Message` object.
- INTERNAL: Lots of technical debt formatting issues cleaned up.
- API: Added `/getdatabases` endpoint.
- INTERNAL: Split smtpauthd from smtpd server to make things more clear.
December 13, 2019
- BUGFIX: Even if a group is configured to allow users to specify new hashtags (Messages can be tagged with new hashtags), the 'Create Hashtag' button was not displayed in the Hashtags page for members.
- NEW: Integrated with Canny.io to test feature request/tracking.
- API: Added `/newhashtag`, `/updatehashtag`, `/deletehashtag` endpoints.
- API: Added `/mutehashtag`, `/unmutehashtag`, `/followhashtag`, and `/unfollowhashtag` endpoints.
- CHANGE: Members can only donate to a donation request one time. This is to prevent accidental duplicate donations (which have happened, unfortunately).
December 7, 2019
- SYSADMIN: Upgraded the outbound email machine.
- CHANGE: We are no longer accepting new Yahoo Group transfer requests.
- API: Added `reply`, `bcc_me` and `from_owner` parameters to the `/postdraft` endpoint.
- API: Added `/authenticateloginlink` endpoint.
November 26, 2019
- API: Added `/sendloginlink`, `/resendconfirmationemail` and `/unbounceuser` endpoints.
- API: Added `/getsubgroupsbycategory` endpoint.
- NEW: When changing your email address, we now send a notification email to your old email address.
- API: Added `/getsubgroupcategories` endpoint.
November 15, 2019
- API: Added `/searchwikis` endpoint.
- API: Changed `search_result` object to `message_search_result`.
- API: Added `/getbounceinfo` and `/getdeliveryinfo' endpoints.
- API: Added `/getmessagehistory` endpoint.
- API: Added optional `type` parameter to the `getmembernotices` endpoint.
- API: Added `muted` and `followed` fields to `hashtag` object.
- API: Added `/getsinglefeed` endpoint.
- CHANGE: When switching from monthly to yearly billing, charge the card immediately instead of waiting until the next invoice date.
- BUGFIX: Updated the list of timezones with their current UDT offsets.
- BUGFIX: Fixed an issue where going to an open chat would just return you to the available chats page.
November 8, 2019
- INTERNAL: Several database optimizations to speed group imports.
- INTERNAL: Completed webhook server.
- API: Added `/getmemberdirectory` endpoint.
- API: Added `sticky_wiki_page_id` field to the `group` object.
- API: Added new `id` parameter to `/getwikiset` endpoint.
- API: Added new `Types` section to the docs. Currently only lists the `action` enum.
- API: Changed all view_* permissions to be *_visible, to be consistent.
- API: Added `msg_num`, `subject`, `extra`, `extra2`, `extra_id` fields to `webhook_event` objects.
November 1, 2019
- API: New endpoints: `newpoll`, `updatepoll`, `getpoll`, `vote`.
- API: New endpoint: `registeruser`.
- NEW: Testing prioritizing support emails from group owners of premium groups.
- API: New endpoints: `newwikipage`, `getwikipages`, `getwikipageset`.
October 18, 2019
- INTERNAL: Various software changes and additional machines to deal with the extreme massive influx of new groups.
- NEW: For premium/enterprise groups on annual billing, we now send out a courtesy reminder two weeks before billing them.
- API: Added `signature` to `member_info` object.
October 11, 2019
- INTERNAL: More work on the API webhooks system.
- CHANGE: When forwarding a message using the Mac email client (and maybe others), it takes the forwarded message, encloses it in a blockquote and adds at the top 'Begin forwarded message'. Because it was in a blockquote, we were collapsing it. This was not good behavior, so we now look for this special case and don't collapse that blockquote.
- API: Added `open_donations_visible` to `perms` object.
- API: Added `hashtags_visible` to `perms` object.
- API: Removed `view_archives` from `perms` object.
- API: Changed `view_guidelines` to `guidelines_visible` in `perms` object.
- API: Added `subgroups_visible` to `perms` object.
- API: Added `perms` field to `member_info` object.
- API: Added new Subs array to `user` object.
- API: The `member_info` object now only returns the fields that are set by the endpoint. Previously it returned all fields, regardless of whether the fields were set or not (some endpoints won't set all the fields based on permissions).
- BUGFIX: When dealing with a message from someone not a part of the group, who has DMARC p=reject set on their domain, use their name in the From line (ie Mark Fletcher via Groups.io). Previously this was blank.
- CHANGE: When banning a member of a group, we now generate a past member object for them so they properly appear in the past members list.
- CHANGE: Added autocomplete="off" to email fields when viewing a member and when viewing your account information to prevent the browser from changing the fields.
October 4, 2019
- INTERNAL: More work on the upcoming notifications system.
- INTERNAL: Much work to revamp the activity log system to support webhook API functionality.
- BUGFIX: The `/getmembers` API endpoint would return all members, including banned and pending members, if you didn't specify a `type`, instead of only returning normal members, which it should do.
- NEW: Added subject checks for 'ris:' and 'r: ' which can indicate a message is a reply.
- CHANGE: Changed the file upload notification so that only one notification is generated regardless of the number of files/directories that are uploaded.
- API: Added `/getphoto` endpoint.
- API: Added `display_name`, `user_name`, `profile_photo_url`, `profile_privacy`, `email` fields to the `photo` object.
- API: Added `/reportcontent` endpoint.
September 27, 2019
- NEW: Upload Folder button in the Files area to upload a directory of files.
- API: Added `/newmembernotice`, `/updatemembernotice`, `/deletemembernotice`, and `/getmembernotices` endpoints.
- NEW: Checkbox to notify the group when you modify a file.
- CHANGE: Rate limit how often someone can report a message, to make abuse more difficult.
- CHANGE: Changed how we present start/end dates/times in calendar invite emails, to match how we present them when clicking Download Event. Even though both should be equivalent, apparently Outlook likes the Download Event version better.
- API: Added `/searchfiles` endpoint.
- BUGFIX: We now consider email addresses with parentheses as invalid.
- API: Added `/searchphotos` endpoint.
- API: Added `/getdirectory` and `/getfile` endpoints.
- API: Changed `snippet` field to `summary` in the `search_result` object for naming consistency.
- INTERNAL: Upgraded Go compiler to 1.13.1
- API: Added `/testnotification` endpoint.
- API: Handle case where group is not in the same domain as the API request. Caused incorrect direct add emails.
September 20, 2019
- API: Added `sort_field`, `second_order`, `query`, `sort_dir` fields to the pagination object.
- BUGFIX: In some cases, when doing a photos search, clicking an individual photo would bring up a different photo.
- CHANGE: Sorting group calendar ICS feeds by start time, to maybe, hopefully, fix a weird Outlook bug.
- BUGFIX: Fix issue where the photo may not match the thumbnail if the album was sorted by Posted.
- API: Added `/registerdevice` endpoint for registering device tokens for notifications.
- NEW: Added `exclude_aliases` flag to `/gethashtags` endpoint.
- NEW: Include the IP address of the mail server we were attempting to delivery email to in bounce message lists on the website.
- API: Added `/searcharchives` endpoint and related `search_result` and `search_results_list` objects.
- API: Added `profile_photo_url`, `name` and `can_view_profile` fields to the `chat` object.
- API: Added `alias` field to `hashtag`.
- API: Added `announce` flag to `/newchat` endpoint.
- API: Setting the `chat_sub` field on the chat object whenever we can. Previously this was only set in a few instances.
- BUGFIX: You couldn't change the case of your email address; it thought you were merging with a different email address and would fail.
- API: Added `/addalbum`, `/updatealbum`, `/deletealbum`, `/addphotos`, `/updatephoto`, `/deletephoto` endpoints.
September 13, 2019
- NEW: More work on the new notifications system.
- API: Changed `gallery_url` field in the photo and attachment objects to `thumbnail_url` because it's more clear.
- API: Added `getalbums` and `getphotos` endpoints.
- API: Changed `profile_photo_id` field in `member_info` to `profile_photo_url`.
- BUGFIX: We had disabled signature parsing in plain text messages where we couldn't tell if the sig was part of the reply or the original message. That change wasn't also added to html messages.
- BUGFIX: Subscribing to a subgroup via email did not work.
- BUGFIX: Fixed a loophole that would sometimes allow emails from numbered qq.com accounts through.
- API: Added `/chatmembers` endpoint.
- API: Fixed bug in `/joinchat` endpoint that caused it to return a `no_group_specified` error.
- API: Added `updateprofile` endpoint.
- API: Added `num_drafts` field to `member_info` object.
- API: Added `updateprofilephoto` and `deleteprofilephoto` endpoints.
- CHANGE: No longer allowing the CSS height attribute because in some instances we were ending up with archives that had overlapping text at the bottom.
- BUGFIX: After creating a group, fix text for groups that are unlisted but with public archives.
September 6, 2019
- CHANGE: Changed a bunch of references to 'post' to 'message'.
- API: Changed `profile_photo_id` to `profile_photo_url` in the User object.