Transfer Files Between S3 Buckets in Different AWS Accounts

Getting Started#

Basic Concept#

Connect Buckets in Different AWS Environments Through a Single Transfer Path

AWS environments can be separated into different accounts for operations, development, and analysis, or different accounts can be used for different organizations and business functions.

For example, you can transfer source data generated in an S3 bucket in an operations account to a separate analytics account, or bring files created in an external collaboration environment into an internal AWS account.

In this case, the Source and Target exist in different AWS environments, but the two buckets can be connected within a single Flow for file transfer.

For example, you can configure the following structure.

Even when the accounts and regions differ, you can directly connect the bucket where files are generated to the bucket where they will actually be used and configure the required file transfer path.

Transfer Scope#

Select Only the Required Files and Paths Instead of the Entire Bucket

S3 buckets in different accounts can contain files used by multiple business functions.

Rather than transferring the entire bucket as-is, you can select only the paths and files actually required in the other environment and define the transfer scope accordingly.

For example, assume that a Source bucket contains the following data.

Of these, only the daily/ path required by the analytics environment can be transferred to the Target bucket.

When configuring the transfer scope, you can also set the following criteria.

This allows you to selectively transfer only the files actually required in the other AWS environment instead of copying all data across the account.

Transfer Structure#

Prepare the Required Access Scope for the Source and Target Accounts Separately

When connecting S3 buckets in different AWS accounts, the required access scope differs between the Source and Target.

The Source must be able to access and read the files to transfer, while the Target must be able to save files to the specified bucket and path.

Before starting the transfer, check the following environment settings.

This structure enables a transfer flow that retrieves files from one AWS account and directly reflects them in a specified bucket in another AWS account.

IT Engineer#

Environment Connection#

Connect S3 Storage in Different Accounts Separately

First, connect the S3 Storage to be used by the Source and Target separately.

For example, you can register a bucket in the operations AWS account as one Storage and a bucket in a separate analytics AWS account as another Storage.

Even when the two environments use different account information, they can each be selected as the Source and Target in a transfer Flow.

For example, you can configure the following structure.

Each connection is managed independently, so connection information or access scope for a specific account can be changed without affecting the management of other Storage settings.

Path Configuration#

Create a File Transfer Path by Connecting the Source and Target Prefixes

After the Storage connections are complete, configure the Source path containing the actual files and the storage location to use on the Target.

For example, you can select the following path on the Source.

s3://production-data/export/daily/

On the Target, you can configure files to be stored in the following location.

s3://analytics-data/incoming/production/

In this case, the file transfer is configured as follows.

If necessary, you can preserve the same folder structure on the Source and Target or reflect the files using a new Prefix structure suited to the Target environment.

Execution Criteria#

Start Transfers for New Files, Changed Files, or on a Defined Schedule

File transfers between different S3 buckets can be run under various conditions depending on the business workflow.

For example, if data files generated every day need to be transferred to another account, you can run the Flow on a schedule. If new files need to be transferred to the analytics environment as soon as they are created, you can start the transfer based on file status.

For example, when transferring reports generated daily in an operations account to an analytics account, you can configure the following flow.

File Creation → Check Scheduled Time → Check Source S3 Files → Transfer to Target S3 → Verify Results

Conversely, if files should be moved to the next account only after analytics processing is complete, the next transfer can be started based on the completion status of the previous Flow.

Cross-Account Transfer#

Directly Connect the Source and Target Buckets in a Single Flow

After configuring the transfer conditions and paths, connect the S3 buckets in different AWS accounts in the Flow.

The Source provides the files to transfer, while the Target saves them to the specified path.

Even when the accounts are different, you can configure a single file transfer job based on the connection information for each Storage.

The transfer environment does not have to be configured in only one direction.

For example, you can extend the flow by transferring source data from an operations account to an analytics account and then transferring the analytics results to another bucket.

This allows even separately managed S3 environments to connect file creation → cross-account transfer → analysis → result storage in a single file flow.

Result Verification#

Verify Transferred Files and Their Reflection in the Target Bucket

After the transfer is complete, check the overall job execution status in Runs and, when necessary, review the Source and Target processing results in the details.

Especially when moving files between different accounts and regions, it is important to verify not only whether the job completed but also whether the required files were actually reflected successfully in the target bucket.

Each Run provides the following information.

After checking the file count and transfer results, verify that the files were successfully reflected in the specified Prefix of the Target bucket.

Exception Handling#

Check Permissions, Paths, or File Status and Rerun the Required Job

During transfers between S3 buckets in different accounts, a job can fail because of access settings or storage path issues on either the Source or Target.

In this case, use the Run details to identify the stage where the problem occurred and adjust the required environment settings.

Applying this recipe lets you configure a single flow covering connecting S3 buckets in different AWS accounts or regions → specifying Source and Target paths → selecting required files → setting execution conditions → cross-account file transfer → verifying reflection in the target bucket → checking failed items and rerunning.

This enables a file transfer environment in which required files are automatically transferred to specified S3 buckets across account and region boundaries, with results verified, even in AWS environments separated for operations, analytics, development, backup, and other purposes.