Backup to Google Drive in FreeNAS with OAuth

oO.o · January 13, 2020, 8:21pm

UPDATE:

As of 11.3U1, this appears to be unnecessary as FreeNAS now has it’s own public-facing API that it uses. This is an added convenience, but they will have to do a lot to keep up with requesting API limit increases or performance and reliability will suffer greatly.

I’ll leave this for posterity. If you are using rclone manually, you’ll still want to follow the API OAuth procedure.

Most of the instructions I’ve seen on how to do this are not explicit enough, and I’ve found that it is easy to get lost in the Google API interface. Here is my step-by-step for creating OAuth credentials so FreeNAS can reliably sync to Google Drive.

Note that this is necessary for rclone (the underlying tool that syncs to cloud accounts) to sync reliably. Without it, rclone can quietly fail to copy all files.

This process will also work if you are using rclone manually.

Create a Project

Navigate to https://console.developers.google.com/

Click Select a Project (if you already have projects, this will be the name of one of your projects)

Click New Project

Type unique name under Project name

Click Create

Enable Google Drive API

Click + ENABLE APIS AND SERVICES

Type drive in Search for APIs & Services

Click Google Drive API

Click ENABLE

Generate OAuth Credentials

Navigate back to API home page by clicking Google APIs (top left)

Click Credentials (left)

Click CONFIGURE CONSENT SCREEN

Select Internal

Click Create

Type rclone under Application name (or whatever you want)

Set Support email to your address (or whatever you want)

Click Save

Click Create credentials

Click OAuth client ID

Click Other

Type rclone under Name (or whatever you want)

Save your client ID to your password manager

Save your client secret to your password manager

Add Credentials to FreeNAS

Navigate to your FreeNAS web UI

Navigate to System > Cloud Credentials

Type a descriptive name under Name

Select Google Drive under Provider

Paste your client ID under OAuth client ID

Paste your client secret under OAuth Client Secret

Click Authenticate

A popup window should appear

Click Proceed

Select your account (if prompted)

Click Allow

Access Token should populate automatically

Click Save

Next Steps

You can now configure cloud backups under Tasks > Cloud Sync Tasks

To monitor Traffic, Errors and Latency, navigate to https://console.developers.google.com

Select your project in the dropdown menu to the right of Google APIs (top left)

theprez · April 22, 2020, 1:17am

First…thank you for taking the time to post this level of detail. It’s what I needed.

Secondly, I was hoping I could use my own custom credentials as I’m getting rate limited already - either by placing them in the FreeNAS GUI (doesn’t appear to be possible with 11.3U2) or by SSH’ing into the server and manually editing a file.

Is that possible? If so, where would it be saving the configuration to and would it survive a reboot?

Thanks

oO.o · April 22, 2020, 2:51am

They took it away and now it only uses their API credentials. It’s definitely a bummer.

I don’t think you can manually edit the rclone config.

balrog · April 20, 2021, 8:40pm

Changes: I had some token expirity problems, but figured what the problem was - here is the workaround

I was able to use rclone with OAuth with the following instructions
(remove the * if there are in the Links):

1. Change project-access to internal

Sign-in to “Google Cloud Console”: *(https://console.cloud.google.com)

Select the project ID: RClone (id: <>)

Go to OAuth Consent Screen under APIs & Services

Go to User Type

Select Make Internal

Click Save

You also need to associate your project with your organization by following the steps below:

Create an Organization by following the “Quickstart Using Organizations” instructions:
(https://cloud.google*.com/resource-manager/docs/quickstart-organizations)

Migrate the project into the organization you created as shown in “Migrating Existing Projects into the Organization”:
(https://cloud.google*.com/resource-manager/docs/migrating-projects-billing).

Users in your organization can now use the app to directly access “OAuth scopes”:
(https://developers.google*.com/identity/protocols/googlescopes)
without any verification steps.

2. Login into Free-/TrueNAS

Klick on LOGIN TO PROVIDER and fill in your Credentials as normal

Now fill in the OAuth Client ID and OAuth Client Secret fields from your Project but let the token as it is!

3. Verify and Save

Click on VERIFY CREDENTIAL and confirm with SAVE.

(successfully tested on TrueNAS-12.0-U3)

oO.o · February 6, 2023, 5:41am

As of last week, I am getting rate limit errors across multiple TrueNAS Core systems using different API credentials on different Google Workspace accounts. I’ve tried generating new OAuth credentials and it still fails immediately.

Is the party over?

I haven’t tried rclone directly yet, still only fussing with TrueNAS. Doesn’t look good though.

xzpfzxds · February 6, 2023, 8:32am

Does https://console.cloud.google.com/apis/dashboard for the project you’re using show only 403/429 errors for Drive API, or other errors too? Are you authenticating as a user or a service account?

oO.o · February 6, 2023, 5:03pm

This is the new project I made yesterday with the OP instructions authenticating as a user account which is the owner of the Google Drive target (it’s an Enterprise Standard account). I tried running a backup that has successfully completed every day for years.

The rclone errors as captured by TrueNAS logs is:

error reading destination directory: couldn't list directory: googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' of service 'drive.googleapis.com' for consumer 'project_number:redacted'.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/redacted",
      "quota_limit": "defaultPerMinutePerProject",
      "quota_limit_value": "60000",
      "quota_location": "global",
      "quota_metric": "drive.googleapis.com/default",
      "service": "drive.googleapis.com"
    },
    "reason": "RATE_LIMIT_EXCEEDED"
  },
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Request a higher quota limit.",
        "url": "https://cloud.google.com/docs/quota#requesting_higher_quota"
      }
    ]
  }
]

I can try to request an increase, but I don’t understand how one backup would generate over 60000 API requests per minute, and why this would suddenly be an issue across multiple accounts/projects.

oO.o · February 6, 2023, 5:36pm

Ok, so I found the problem!

Google is now enforcing a 5TB/user pool even for Enterprise accounts. When I had signed up for the account, the storage was described as unlimited.

The Google Workspace support chatbot allowed me to request more storage, but I have been backing up production servers daily for years, so I am way over the limit. We’ll see, but I think that the party is indeed over… Time to look at Amazon Glacier Deep I guess.

Google Workspace Enterprise Standard

“Google Workspace Enterprise Standard” is an edition of Google Workspace comprised of all the Google Workspace Services except Client-Side Encryption, Google Cloud Search, and Workspace Add-Ons. Google Workspace Enterprise Standard also includes data loss prevention functionality for Gmail and Google Drive, and certain enhanced security and control features for Administrators (not including Google Workspace Security Center). Google Workspace Enterprise Standard will also allow for additional Gmail integration with other Google products, certain third-party archiving tools, and third-party OAuth applications. Customers that have 5 or more End Users will receive a total amount of Google Drive storage equal to 5TB times the number of End Users, with more storage available at Google’s discretion upon reasonable request to Google. Customers that have received a Google for Nonprofits discount or have 4 or fewer End Users will receive 1TB storage in total for Google Drive, Google Photos, and Gmail combined for each End User.
Google Workspace Enterprise Plus (prior edition: G Suite Enterprise)

“Google Workspace Enterprise Plus” is an edition of Google Workspace comprised of all the Google Workspace Services except Workspace Add-Ons. Google Workspace Enterprise Plus also includes data loss prevention functionality for Gmail and Google Drive, data region policy settings for primary data within Customer Data for certain Services, additional search and assist capabilities for content within third party data sources (which are only available to customers with at least 500 End User licenses), and enhanced security and control features for Administrators (including Google Workspace Security Center). If a customer wishes to implement a trial, proof of concept evaluation, or deployment of third party data indexing in Cloud Search, then that customer must do so via a Cloud Search certified partner. Google Workspace Enterprise Plus will also allow for additional Gmail integration with other Google products, certain third-party archiving tools, and third-party OAuth applications. Customers that have 5 or more End Users will receive a total amount of Google Drive storage equal to 5TB times the number of End Users, with more storage available at Google’s discretion upon reasonable request to Google. Customers that have received a Google for Nonprofits discount or have 4 or fewer End Users will receive 1TB storage in total for Google Drive, Google Photos, and Gmail combined for each End User.

Hmm, maybe not the whole story. I’m still getting rate limit errors. I think TrueNAS is using its own API credentials despite the client and secret values that I put into the config UI.

oO.o · February 6, 2023, 11:59pm

Ok, so in the TrueNAS GUI, if you edit a cloud credential and replace the client and secret fields, and then save it, it does not update the token. You also can’t leave the token blank. You have to go through the process of setting up an rclone config on the cli to get the token and fill it in manually. Kind of negates the point of having the GUI there but whatever.

As far as I can tell, the stock Google Drive credentials are unusable at this point. It will list the contents but I can’t push or pull a single file.

xzpfzxds · February 7, 2023, 8:16pm

Well it’s weird that the API metrics graph you shared shows zero usage, makes me think that it is not using the OAuth refresh token that you think it is. Try disabling the credentials in the Cloud cosole and if you don’t get an authentication error, then it’s not using that project.

Another way to check which project a refresh token is for, is to get an access token from the refresh token:

curl \
--request POST \
"${TOKEN_URL}" \
--data-urlencode "grant_type=refresh_token" \
--data-urlencode "client_id=${CLIENT_ID}" \
--data-urlencode "client_secret=${CLIENT_SECRET}" \
--data-urlencode "refresh_token=${REFRESH_TOKEN}" \
| tee access_token.json

Then get access token OAuth client info:

curl -v \
--header "Authorization: Bearer $(<access_token.json jq -r .access_token)" \
"https://oauth2.googleapis.com/tokeninfo"

The OAuth client ID, which contains the GCP project number as a prefix, is the contents of the aud field in the response. Is that the same as the project you created?

oO.o · February 7, 2023, 8:38pm

That is correct. The TrueNAS GUI is misleading. I had updated the client and secret values but it continued to use the existing token.