使用 go 将文件流式上传到 AWS S3

Question

我想将 multipart/form-data（大）文件直接上传到 AWS S3，并且尽可能少地占用内存和文件磁盘空间。我怎样才能做到这一点？在线资源仅说明如何上传文件并将其本地存储在服务器上。

Answer 1

您可以使用上传管理器来流式传输文件并上传它，您可以阅读源代码中的评论， 您还可以配置参数来设置分段大小、并发和最大上传分段，以下是示例代码供参考。

package main

import (

    "fmt"

    "os"

    "github.com/aws/aws-sdk-go/aws/credentials"

    "github.com/aws/aws-sdk-go/aws"

    "github.com/aws/aws-sdk-go/aws/session"

    "github.com/aws/aws-sdk-go/service/s3/s3manager"

)

var filename = "file_name.zip"

var myBucket = "myBucket"

var myKey = "file_name.zip"

var accessKey = ""

var accessSecret = ""

func main() {

    var awsConfig *aws.Config

    if accessKey == "" || accessSecret == "" {

        //load default credentials

        awsConfig = &aws.Config{

            Region: aws.String("us-west-2"),

        }

    } else {

        awsConfig = &aws.Config{

            Region:      aws.String("us-west-2"),

            Credentials: credentials.NewStaticCredentials(accessKey, accessSecret, ""),

        }

    }

    // The session the S3 Uploader will use

    sess := session.Must(session.NewSession(awsConfig))

    // Create an uploader with the session and default options

    //uploader := s3manager.NewUploader(sess)

    // Create an uploader with the session and custom options

    uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {

        u.PartSize = 5 * 1024 * 1024 // The minimum/default allowed part size is 5MB

        u.Concurrency = 2            // default is 5

    })

    //open the file

    f, err := os.Open(filename)

    if err != nil {

        fmt.Printf("failed to open file %q, %v", filename, err)

        return

    }

    //defer f.Close()

    // Upload the file to S3.

    result, err := uploader.Upload(&s3manager.UploadInput{

        Bucket: aws.String(myBucket),

        Key:    aws.String(myKey),

        Body:   f,

    })

    //in case it fails to upload

    if err != nil {

        fmt.Printf("failed to upload file, %v", err)

        return

    }

    fmt.Printf("file uploaded to, %s\n", result.Location)

}

Answer 2

你可以使用minio-go做到这一点：

n, err := s3Client.PutObject("bucket-name", "objectName", object, size, "application/octet-stream")

PutObject() 自动在内部进行分段上传。例子

Answer 3

另一种选择是使用goofys挂载 S3 存储桶，然后将您的写入流式传输到挂载点。goofys 不会在本地缓冲内容，因此它可以很好地处理大文件。