Upload from cloud storage

• Amazon AWS S3
• Microsoft Azure Blob
• Google Cloud Storage
• SFTP
• NFS

​

Amazon AWS S3

​

Prerequisites

• Creating an S3 bucket following the Amazon Simple Storage Service User Guide.
• Create an IAM user with Programmatic access. See Creating an IAM user in your AWS account in the AWS Identity and Access Management User Guide.
• Create an access key under Security credentials > Access keys. Save the access key ID and access key.
• Attach the following role policy to allow the user to access your S3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<your_bucket_name>",
                "arn:aws:s3:::<your_bucket_name>/*"
            ]
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<your_bucket_name>",
                "arn:aws:s3:::<your_bucket_name>/*"
            ]
        }
    ]
}

​

Set up connection

• Bucket Name
• Prefix (Optional)
• Region Name
• Access Key ID
• Access Key

​

Upload data from S3 buckets

​

Microsoft Azure Blob

​

Prerequisites

• Create an Azure storage account. See Create a storage account in the Microsoft Azure product documentation.
• Get the generated keys in the Access keys section in the Azure Portal.
• Create a storage container within your storage account by following the steps to Upload, download, and list blobs with the Azure portal in the Microsoft Azure product documentation.

​

Set up connection

• Container Name
• Folder (Optional)
• Account Name
• Account Key

​

Upload data from Azure Blob

​

Google Cloud Storage

​

Prerequisites

• Creating a bucket. See Creating storage buckets in the Google Cloud Storage guide.
• Choose uniform access control when choosing the access control method for the bucket.
• Create an IAM Service Account. See Creating and managing service accounts.
• Select the predefined Storage Object Admin IAM role to add to the service account so that the account can create, access, and delete objects in the bucket.
• Add a condition to the role that restricts the role to access only objects that belong to the bucket you created. You can add a condition in one of two ways:
  • Select Add Condition when setting up the service account IAM role, then use the Condition Builder to specify the following values:
    • Condition type: Name
    • Operator: Starts with
    • Value: projects/_/buckets/<your_bucket_name>
  • Or, use a Common Expression Language (CEL) to specify an IAM condition. For example, set the following: resource.name.startsWith('projects/_/buckets/<your_bucket_name>'). See CEL for Conditions in Overview of IAM Conditions in the Google Cloud Storage guide.
• Create a service account key and download the JSON. Follow the steps for Creating and managing service account keys in the Google Cloud Identity and Access Management guide.

​

Set up connection

• Bucket Name
• Folder (Optional)
• Project ID
• Credentials (service account key)

​

Upload data from GCS

​

SFTP

• Host: The domain name or IP address of the SFTP server.
• Port: The port on which the SFTP server is listening. The default SFTP port is 22.
• Username: The username for authenticating with the SFTP server.
• Password: The password for authenticating with the SFTP server.
• Private Key File: upload the private key used for authentication.
• Root Path: The root directory path within the SFTP server. This defines the base folder for storage operations.

​

NFS

• Folder: local directory path to an NFS share drive. The default root path for the NFS directory is under /raw-data