Repositories
A repository is a place where Kopia stores its snapshot data. Kopia currently supports the following storage backends:
- Google Cloud Storage
- Azure Blob Storage
- Amazon S3 (and compatible)
- Backblaze B2
- Google Drive
- SFTP
- WebDAV
- Rclone
- Local storage
In addition, Kopia can connect to a Kopia Repository Server that acts as a proxy for the storage backend.
Many storage providers offer products with a variety of “tiers” suitable for different use cases. Choosing appropriate storage tiers may result in lower costs depending on, for example, how frequently data needs to be retrieved. See Storage Tiers for further details.
Google Cloud Storage
Google Cloud Storage is a globally unified, scalable, and highly durable object storage for developers and enterprises.
Creating a repository
To create a repository in Google Cloud Storage you need to provision a storage bucket and install local credentials that can access that bucket. To do so, there are three methods (one that requires you to install Google Cloud SDK; the other method allows you to generate crendtials without Google Cloud SDK; the third method allows you to use Google Cloud Storage like it is AWS S3):
***Method #1: Installing Google Cloud SDK
- Create a storage bucket in Google Cloud Console
- Install Google Cloud SDK
- Log in with credentials that have permissions to the bucket.
$ gcloud auth application-default login
After these preparations we can create Kopia repository (assuming bucket named kopia-test-123
):
$ kopia repository create gcs --bucket kopia-test-123
At this point we should be able to confirm that Kopia has created the skeleton of the repository with 3 files in it:
$ gsutil ls gs://kopia-test-123
gs://kopia-test-123/kopia.repository
gs://kopia-test-123/n417ffc2adc8dbe93f1814eda3ba8a07c
gs://kopia-test-123/p78e034ac8b891168df97f9897d7ec316
***Method #2: Creating a Service Account and Using the JSON File
- Create a storage bucket in Google Cloud Console
- Create a Google Cloud Service Account that allows you to access your storage bucket. Directions are available on Google Cloud’s website. Make sure to download the JSON file for your service account and keep it safe.
After these preparations we can create Kopia repository (assuming bucket named kopia-test-123
):
$ kopia repository create gcs --credentials-file="/path/to/your/credentials/file.json" --bucket kopia-test-123
***Method #3: Enabling AWS S3 Interoperability in Google Cloud Storage
- Create a storage bucket in Google Cloud Console
- Go to Settings and then Interoperability in your Google Cloud Storage account
- Enable your project under “Default project for interoperable access” and generate access keys for this project – you will generate both access key and secret key, just like if you were using AWS S3
After these preparations we can create Kopia repository using the s3 function (assuming bucket named kopia-test-123
):
$ kopia repository create s3 --endpoint="storage.googleapis.com" --bucket="kopia-test-123" --access-key="access/key/here" --secret-access-key="secret/key/here"
Connecting To Repository
To connect to a repository that already exists, simply use kopia repository connect
instead of kopia repository create
. You can connect as many computers as you like to any repository, even simultaneously.
$ kopia repository connect gcs --bucket kopia-test-123
or
$ kopia repository connect gcs --credentials-file="/path/to/your/credentials/file.json" --bucket kopia-test-123
or
$ kopia repository connect s3 --endpoint="storage.googleapis.com" --bucket="kopia-test-123" --access-key="access/key/here" --secret-access-key="secret/key/here"
Detailed information and settings
Amazon S3
Kopia can connect to S3 compatible storage, such as Amazon S3, minio.io, Wasabi
Creating a repository
You will need your S3 bucket name, access key and secret access key.
$ kopia repository create s3 \
--bucket=... \
--access-key=... \
--secret-access-key=...
Connecting To Repository
$ kopia repository connect s3
Detailed information and settings
B2
B2 is a very cheap yet fully featured S3 compatible storage.
Creating a repository
You will need your B2 bucket name, key-id and key.
$ kopia repository create b2 \
--bucket=... \
--key-id=... \
--key=...
Connecting To Repository
$ kopia repository connect b2
Detailed information and settings
Google Drive
Google Drive is a file storage and synchronization service developed by Google, which you can set up as a storage backend for Kopia.
WARNING: Google Drive support is experimental, use at your own risk.
Kopia uses a Google Drive folder that you provide to store all the files in a repository. Kopia will only access files in this folder, and using Kopia does not impact your other Drive files. We recommend that you let Kopia manage this folder and do not upload any other content to this folder.
Creating a repository
Here’s a high-level rundown of what we will do:
-
Create or use an existing Google Drive folder for the new repository.
-
Create a Service Account for Kopia. A service account is a Google account for a robot user, and can be created and managed more easily than a real Gmail account.
-
Share the Google Drive folder with your new service account so that it can access the folder.
Ready? Here are the step-by-step instructions:
-
Create a Google Cloud project, or use an existing one.
-
Enable the Google Drive API using your project.
-
Create a service account. After enabling the API, you should be now prompted to create credentials. Choose “Service account” from the options, and give it a name. Note down the service account email.
-
Create a key for the service account. You can do this by viewing the service account, navigating to the “Keys” tab, and clicking “Add Key” -> “Create new key”. You should choose “JSON” for the key type. Save the file on your computer.
-
Create or pick an existing Google Drive folder. The browser URL should look something like
https://drive.google.com/drive/u/0/folders/z63ZZ1Npv3OFvDPwU3dX0w
. Note down the last part of the URL. That’s your folder ID. -
Share the folder with the service account. Open the share dialog for the folder, and put in the service account email. You should choose the “Editor” as the access role.
After these preparations we can create a Kopia repository (assuming the folder ID is z63ZZ1Npv3OFvDPwU3dX0w
):
$ kopia repository create gdrive \
--folder-id z63ZZ1Npv3OFvDPwU3dX0w \
--credentials-file=<where-you-have-stored-the-json-key-file>
If you view your folder on Google Drive, you should see that Kopia has created the skeleton of the repository with a kopia.repository
file and a couple of others.
Connecting To Repository
To connect to a repository that already exists, simply use kopia repository connect
instead of kopia repository create
.
You can connect as many computers as you like to any repository, even simultaneously. If you have multiple computers, we recommend that you create a new service account key for each computer for good security.
$ kopia repository connect gdrive \
--folder-id z63ZZ1Npv3OFvDPwU3dX0w \
--credentials-file=<where-you-have-stored-the-json-key-file>
Detailed information and settings
SFTP
The SFTP
provider can be used to connect to a file server over SFTP/SSH protocol.
You must first configure passwordless SFTP login by following these instructions. Choose an empty passphrase because Kopia does not allow password prompts for the backend.
If everything is configured correctly, you should be able to connect to your SFTP server without any password by using:
$ sftp [email protected]
Connected to my-server.
sftp>
Creating a repository
Once the passwordless connection works, then you can create a Kopia SFTP repository. Assuming you want the files to be stored under /remote/path
, run the command below. Adjust the username and paths to the key file and known hosts file as necessary.
$ kopia repository create sftp \
--host my-server \
--username some-user \
--keyfile ~/.ssh/id_rsa \
--known-hosts ~/.ssh/known_hosts \
--path /remote/path
When prompted, enter Kopia password to encrypt the repository contents.
If everything is done correctly, you should be able to verify that SFTP server indeed has Kopia files in the provided location, including special file named kopia.repository.f
:
$ sftp [email protected]
sftp> ls -al /remote/path
-rw-r--r-- 1 some-user some-user 661 Sep 18 16:12 kopia.repository.f
Connecting To Repository
To connect to an existing SFTP repository, simply use connect
instead of create
:
$ kopia repository connect sftp \
--host my-server \
--username some-user \
--keyfile ~/.ssh/id_rsa \
--known-hosts ~/.ssh/known_hosts \
--path /remote/path
If the connection to SFTP server does not work, try adding --external
which will launch external ssh
process, which supports more connectivity options which may be needed for some hosts.
Detailed information and settings
Rclone
Kopia can connect to certain backends supported by Rclone as long as they support server-side timestamps.
WARNING: Rclone support is experimental, use at your own risk.
Creating a repository
First you should follow rclone instructions for setting up a remote. This is provider specific, detailed instructions can be found at https://rclone.org/#providers.
Assuming you’ve configured a remote named my-remote
, you may create Kopia repository using:
$ kopia repository create rclone --remote-path my-remote:/some/path
Connecting to repository
$ kopia repository connect rclone --remote-path my-remote:/some/path
Detailed information and settings
Local storage
Local storage includes any directory mounted and accessible. You can mount any readable directory available on your storage, a directory on usb device, a directory mounted with smb, ntfs, sshfs or similar.
Creating a repository
$ kopia repository create filesystem --path /tmp/my-repository
Connecting to repository
$ kopia repository connect filesystem --path /tmp/my-repository
We can examine the directory to see which files were created. As you can see Kopia uses sharded directory structure to optimize performance.
$ find /tmp/my-repository -type f
/tmp/my-repository/n1b/d00/aa56a13c1140142b39befb654a2.f
/tmp/my-repository/pea/8b5/e01d92618653ffbf2bb9961448d.f
/tmp/my-repository/kopia.repository.f
Detailed information and settings