Escaping Workspace (pt. 1)

17 years ago I have moved from Gmail to my own domain with Google Apps (known as Google Workspace since 2020). Now, I am moving back to Gmail, trying to preserve the data stored in the Workspace account across all Google services. This part includes whining about the prices, describes my new Google account setup, and how to move the email messages.

History

Google Apps for Your Domain

In 2006, Google launched a new product: Google Apps for Your Domain. Among other things, it allowed to use Gmail and Gtalk for addresses on your own domain -- so you get a nicely looking email address and have full control over the domain, but also can use all features that Google offers with Gmail, like very precise spam filtering. And it's free! "Cool," thought I, registered a domain after my last name, signed up for Google Apps, and created five accounts for me and my family.

Back then creating a Google Apps email did not create a Google account -- you had to create one manually, using the same process as for creating a Google account for an email not managed by Google. That was different from Gmail, where registering for Gmail automatically created a Google account for this email. But registering an extra Google account manually wasn't a big deal.

Forced Google accounts for Google Apps Emails

In 2011, Google decided to follow the Gmail model for Google Apps accounts -- every email address on a custom domain in Google Apps automatically got a new Google account. All existing accounts for these addresses were renamed to something like name%[email protected]. This is called Conflicting accounts, and every time I log into Google the form asks me into which of two accounts I want to log in. These accounts are related in strange ways -- you can log into the @gtempaccount.com account using the email and the password of the main account, but two factor authentication and recovery phone settings are different for these accounts. The @gtempaccount.com account has separate quota for Drive, doesn't have email, and is probably borderline unusable. So I moved all my Drive files to the freshly created Google Apps account and abandoned the old one. There is still a form from 2011 which allows to create a Gmail address for this account:

Ten Years of Using Google Services

In 2012, Google removed the possibility to sign up for the free Google Apps option -- all new domains had to use a paid Google Apps for Business subscription. They didn't force the existing domains to pay, so my domain continued on the free option.

In 2016, Google Apps was renamed to G Suite. The free option was renamed to "G Suite legacy free edition". In 2020 the product was renamed again, now to Google Workspace.

Over these years, me and my family started using more and more Google services that consume storage. Google Pixel phones were coming with free Google Photos storage, and Google Photos was very good -- so we started storing the photos there. More and more files accumulated in Google Drive and in Gmail attachments. The free quota for the legacy free Workspace accounts was the same as for free Gmail accounts: 30GB. When necessary, you could buy extra storage with the same price as regular Gmail users.

Deprecation of the Free Edition

In 2022, 10 years after closing the free edition for new domains, Google decided that they don't want to support the free edition anymore, and announced that the free accounts must migrate to the paid plan. The paid plan was 5$ per month per account, and offered 30GB of storage per user, just like the free option did. After a lot of bad PR, Google reverted the decision, giving an option to self-identify as a personal non-commercial user and continue using the free edition. But they prohibited getting storage updates for Workspace accounts using consumer-facing Google One offering: now storage in the free edition Workspace accounts was hard capped to 30GB. These decisions triggered a lot of people to migrate (there was a lot of activity in r/gsuitelegacymigration two years ago), but I wasn't one of them.

In 2023, two users of my domain approached the 30GB quota limit. The cheapest non-free option had the same storage limit as the legacy one and didn't support quota sharing back then -- meaning that I still couldn't pay for extra space for the account by buying more accounts. The only to add storage was to migrate to the next tier, costing 12$ per month per account, and offering 2TB of storage per user, shared for the whole domain. I had to make the decision fast, because when the account runs out of quota, Gmail starts dropping email addressed to the account. I've decided to bite the bullet, received by 60$/month invoice, and got 10TB of storage quota, of which we barely used 1%. In 2024, they raised the prices, so I started paying 72$ (+taxes) per month.

Other Problems and Desire to Escape

In the meanwhile, consumer Google services were often delayed or not launched for Workspace accounts. You could not use the first Google LLM AI offering, Bard (but now you can use Gemini). You could not use Stadia before it was shut down. You cannot use FitBit, which requires you to migrate from their legacy account to a Google account. You cannot join a family group, so you cannot share YouTube Premium subscription with other people. I've heard about problems with Nest and Pay, but I don't use any of them.

Recently I understood that paying extra for getting less is not a good idea. I've decided to continue using my address on the custom domain, but move it out of Google Workspace management into a setup which combines Gmail, Cloudflare, and Amazon SES. In parallel, I started moving data to a regular consumer account, probably 2 years later than I should have.

Email and Account Setup

I've decided to continue using my old email address everywhere, to use Gmail for mail storage, as web mail client, as a mobile mail client, have the old email address as an alias to this Gmail address, and set up a system that forwards messages for my old email address to the Gmail address. I picked Cloudflare and Amazon to implement the system, and described the setup in the previous article: Custom email domain using Gmail, Cloudflare, and Amazon SES.

Gmail and Google accounts in general have good support for email aliases: Gmail can show which alias the email came to, and can respond from the same address using a custom non-Google SMTP server. Google accounts also support specifying aliases, and they are supported in most products: for example, sharing a Drive document with an alias automatically shares it with the main address. A notable exception is the calendar: inviting an alias to an event doesn't treat the alias as Google Calendar user.

A potential alternative involves two accounts: a new one for Gmail, but also a new non-Gmail Google account linked to my old email on the custom domain. I've decided against this option, but it was a hard decision:

• ➕ Some products work better when the email that you specify everywhere is the same as your primary Google account address.
• ➕ If I decide to switch from Gmail, I'll just keep this account and Google will deliver all correspondence related to the account the new email provider automatically. With Gmail account, I will have to set up forwarding.
• ➖ The migration process is harder: I can't create a consumer Google account in an existing Workspace domain, so I'd have to do either do the migration twice (once to a temporary Gmail account, and then to the new account on the custom domain), or make an irreversible step of renaming my Workspace domain and accounts very early in the process.
• ➖ Two accounts are generally harder to manage and are more confusing.

Migration Process

I had a few requirements to the migration process:

• Allow moving out quota-consuming things early, so that I can downgrade my Workspace subscription.
• Don't require all users of my domain to do some complex actions in a short time frame.
• Have no interruptions for the email:
  • Validate how the final setup works as early as possible.
  • Have the process well-understood before I start, so that I don't end up with broken email and some urgent decisions to make. That's how this blog article was born :-).
• Delay irreversible actions (like permanent account or domain deletion) as late as possible.

The Safe Part

I came up with the following prework steps that are easily reversible, not dangerous, and can be done user by user: they don't disrupt existing accounts or domains and don't remove any data. If something goes wrong or if I or one of my family members don't have time, we can easily pause and have no degraded experience.

1. Backup everything using Google Takeout in case something goes wrong.

   Takeout archives are supposed to have absolutely all data stored in the account, but it's not trivial to put the data back into an account in case something goes wrong: a lot of data in the archives is in some internal format, and the services can't import the data back in this format.

2. Create a new Gmail account, and start copying all old email into it. In parallel, start moving other things, such as Photos and Drive data.

3. Once old messages are copied, set up Gmail Email Forwarding to forward from my old Workspace address to the new Gmail address. Add the old address as an email alias to the new account. Configure the new account to send mail from the alias using Google Workspace's SMTP.

   When this is done, I should no longer use Workspace's Gmail UI and account for my email: all mail interactions should happen using my new Gmail account, but using the old address as the alias. If there are any unforeseen problems, I can still go back to my Workspace account and use mail directly from it.

4. Configure some other domain and address using Cloudflare for forwarding to my Gmail and using SES for outgoing mail.

   That allows me to validate how the final set up works and if there are some unforeseen problems for a low importance address without affecting the primary address.

5. Register some other temporary domain and add it as a secondary domain to Workspace. Change the primary domain of my Workspace subscription to this new domain. This step is slow: the propagation delay for the change is 48 hours.

   This allows me to remove my old domain from Workspace management during the next steps, while keeping the old Google accounts under a different name for some time.

The Unsafe Part

The following steps are risky: if something goes wrong, the data can be lost. With all safeguards in place, I managed to do the most problematic steps (2 and 3) in one evening without any problems.

1. Start deleting the "heavy" data from the old accounts to free the space: emails with attachments, videos, etc. Once this is done, my Workspace subscription can be downgraded.

   There is a risk of losing data, which should be mitigated by having the data in Takeout backups, prepared as the step 1 above. However, losing data is still not desired: ingesting data from Takeout backups is much harder than copying it between services directly.

2. Change the domain of the old Workspace account to the new temporary domain added in step 5 above.

   This step automatically adds addresses in the old domain as an alias, so email delivery is still not disrupted.

3. Change MX records of the old domain to point to Cloudflare email routing; change SMTP settings in the new Gmail account to use SES for outgoing email instead of Google Workspace's SMTP.

   Having both Workspace and Cloudflare configurations for the domain should allow losing no messages: even if some servers continue sending mail for the old domain to Google, it should still know how to handle it. There is a risk of misconfiguring something (like incorrect SPF records for SES), which should be mitigated by validating the setup on an unimportant domain in the step 4 above.

4. Confirm that all email delivered into the new Gmail mailbox for the old address are delivered by Cloudflare, not by Gmail. Confirm that outgoing emails work and get delivered to the recipients.

   Remove the old domain's alias from Workspace account on temporary domains; remove the old domain from Workspace subscription; add the old address as an alias to the new Gmail account.

   For some reason, when I was doing it for my account, Google started bouncing emails from [email protected] that Cloudflare attempted to forward to me with Transient error (421): 4.7.28 Gmail has detected an unusual rate of unsolicited mail (but not other emails). I had to temporarily redirect email to my @outlook.com address to validate the alias.

At this point, Google Workspace does not know anything about my domain anymore, and the setup for the domain matches the final setup described above. The old accounts are still present, but renamed to use the temporary domain. I will remove the Workspace subscription once I become fully confident that the old accounts are not needed anymore, but I have not done it yet.

Downgrading the Workspace Subscription

After deleting the heavy data from the accounts, I have downgraded my Workspace subscription. There are two caveats:

• Sometimes moving Google Drive files requires using shared drives, which are not available in the cheapest subscription. So before changing the subscription I had to fully complete the Google Drive migration.

• Quota usage calculation works in a strange way and lags behind the changes: for example, deleting photos from the account reflects in the usage displayed in Photos immediately, but Workspace Admin console still sees the old usage for a couple of days, and Gmail sees the old usage for even longer.

  I waited for the Workspace admin console to catch up, and downgraded the subscription while Gmail was still showing the old usage. Gmail showed for a couple of minutes that I am over the quota and that it is rejecting the messages, but then some reconciliation happened, and all three places started showing the same usage.

Copying Email

Fresh Google accounts have a nice button to import mail and contacts from other providers. When migrating from a another consumer Gmail account, this button offer a one-click integration with ShuttleCloud to import everything. This integration is not available when moving from a Workspace account (probably Google does not want people to move from Workspace accounts easily?), and ShuttleCloud also does not allow using it directly. There are two other alternatives.

POP3 Import

Gmail offers POP3-based email ingestion: POP3 credentials of the old account are put into Gmail settings of the new account, and then Gmail acts as a mail client, downloading all messages from the old account into the new one.

I used this approach for my biggest account, and found it generally okay. Here are the main problems:

• It is relatively slow: POP3 integration downloads 200 messages every 5 minutes.
• A lot of legitimate emails were marked as Spam, so I had to manually mark them as not spam (only 50 at a time)!
• All emails appear as unread in the mailbox, so I had to mark them as read (luckily this can be done in bulk).
• All Gmail-only meta information (labels, importance, classification, etc) is lost.
• Some things are marked as viruses and just get dropped from the new account.

Ironically, Google's interaction over POP3 uses plain text passwords instead of OAuth2, so it is classified as Less secure by Google themselves and will be deprecated for Workspace accounts by the end of 2024. Enabling less secure interactions for accounts without 2FA is account-wide and is done in security settings. Accounts with 2FA need to use a different approach: the account needs to define an app-specific password and use it instead of a real one for the POP3 connection. These steps need to be done also to authenticate with Google Workspace's SMTP servers for outgoing email. POP3 access also needs to be enabled in Gmail settings of the old account.

Also, for some reason, "Leave a copy of retrieved message on the server" tickbox needs to be not checked when doing POP3 imports between Gmail accounts. Checking it results in an unknown error, but not checking it still leaves the messages in the old account, if Gmail settings for POP3 of the old account specify that the messages should be kept.

Got Your Back Import

One of my family members didn't want to lose email labels, so I've also done one migration using Got Your Back (GYB) -- a tool which uses Gmail API to download or put emails into Gmail accounts. It has different tradeoffs compared to the POP3 import:

• Setting it up is much more convoluted, because it's hard to get Google API authorization tokens for personal use.
• It keeps some meta information: whether the message is read, unread, the labels, etc. But the restore still marks a lot of messages to spam! There is an open issue about it.
• Restores are extremely slow -- multiple seconds per message. Specifying the --batch flag helps a bit, but it is still slower than the POP3 import.

Overall, I've decided that configuring GYB is worth it: after everything is set up, GYB can incrementally back up all data from a Gmail account into a local directory, and restore this backup into any other Gmail account. I've decided to set it up for my new Gmail account after the migration.

50 Easy Steps to Get an OAuth Token For Personal Use

Authorization in Google APIs happens using OAuth2 tokens. Getting OAuth2 tokens requires setting up a Cloud project and integrating it with Google Sign In. GYB helps setting up some things, but other things still need to be done manually in the UI. At first, run gyb --action create_project, specifying an account which will be used for authorization and API calling purposes. This account doesn't have to be an account that that is being migrated from or to, but I used my new Gmail account for this. This step will authorize some central GYB infrastructure to create a Google Cloud project in the account. It's fine, but it's also safer to revoke the access once the project setup is finished.

The CLI then instructs which link to follow to configure OAuth consent screen and credentials. Unfortunately, the instructions are not very up to date (TODO: Send a PR to the GYB repository updating the documentation). The consent screen can be configured in the testing mode: when Google does not need to verify the app, but only allowlisted accounts can log into the app. It supports allowlisting not more than 100 accounts per project, what is more than enough for all my migrations and backups. Both new and old accounts need to be allowlisted. After configuring the consent screen, go to the Credentials section and create OAuth Client ID for the Desktop app application type. These credentials are the ones that the CLI expects to get.

GYB Backup and Restore

Now everything is ready: gyb --email [email protected] --action backup --local-folder me/ saves all data into a directory, and gyb --email [email protected] --action restore --local-folder me/ puts the data into a different account. You can run the commands (better in screen from a VPS) and watch how something is happening in the Cloud console:

Other Services

This article covered the most complex parts of the migration: Gmail and Google accounts. The next part, Escaping Workspace (pt. 2), is describing everything else: Photos, Drive, Android, Youtube, and other services.