濡備綍浣跨敤 Data Pipeline 灏?DynamoDB 琛ㄥ浠藉埌鍙︿竴涓处鎴蜂笅鐨?S3 瀛樺偍妗讹紵

涓婃鏇存柊鏃堕棿锛?020 骞?7 鏈?1 鏃?/i>

鎴戞兂浣跨敤 AWS Data Pipeline 灏?Amazon DynamoDB 琛ㄥ浠藉埌鍙︿竴涓?AWS 璐︽埛涓嬬殑 Amazon Simple Storage Service (Amazon S3) 瀛樺偍妗躲€?br />

绠€鐭弿杩?/h2>

1.鍦ㄦ簮璐︽埛涓紝闄勫姞涓€鏉?AWS Identity and Access Management (IAM) 绛栫暐锛屾巿浜?DataPipelineDefaultRole 鍜?DataPipelineDefaultResourceRole 瑙掕壊 Amazon S3 鏉冮檺銆?/p>

2.鍦ㄧ洰鏍囪处鎴蜂腑锛屽垱寤轰竴鏉″瓨鍌ㄦ《绛栫暐锛屽厑璁告簮璐︽埛涓殑 DataPipelineDefaultRole 鍜?DataPipelineDefaultResourceRole 瑙掕壊璁块棶 S3 瀛樺偍妗躲€?/p>

3.鍦ㄦ簮璐︽埛涓紝浣跨敤灏?DynamoDB 琛ㄥ鍑哄埌 S3 Data Pipeline 妯℃澘鍒涘缓绠￠亾銆?/p>

4.灏?BucketOwnerFullControl 鎴?AuthenticatedRead 棰勮璁块棶鎺у埗鍒楄〃 (ACL) 娣诲姞鍒扮閬撶殑 EmrActivity 瀵硅薄鐨?b>姝ラ瀛楁涓€?/p>

5.婵€娲荤閬撲互灏?DynamoDB 琛ㄥ浠藉埌鐩爣璐︽埛涓嬬殑 S3 瀛樺偍妗躲€?/p>

6. 鍦ㄧ洰鏍囪处鎴蜂腑鍒涘缓 DynamoDB 琛ㄣ€?/p>

7.瑕佸皢婧愯〃杩樺師鍒扮洰鏍囪〃锛岃浣跨敤浠?S3 瀵煎叆 DynamoDB 澶囦唤鏁版嵁 Data Pipeline 妯℃澘鍒涘缓绠￠亾銆?/p>

瑙ｅ喅鏂规硶

灏?IAM 绛栫暐闄勫姞鍒?Data Pipeline 榛樿瑙掕壊

1.鍦ㄦ簮璐︽埛涓紝鎵撳紑 IAM 鎺у埗鍙?/a>銆?/p>

2.閫夋嫨绛栫暐锛岀劧鍚庨€夋嫨鍒涘缓绛栫暐銆?/p>

3.閫夋嫨 JSON 閫夐」鍗★紝鐒跺悗杈撳叆涓庝互涓嬪唴瀹圭被浼肩殑 IAM 绛栫暐銆傚皢 awsdoc-example-bucket 鏇挎崲涓虹洰鏍囪处鎴蜂腑鐨?S3 瀛樺偍妗跺悕绉般€?/p> 

{
    "Version": "2012-10-17",
    "Statement": ,
            "Resource": 
        }
    ]
}

4.閫夋嫨鏌ョ湅绛栫暐銆?/p>

5.杈撳叆绛栫暐鐨?b>鍚嶇О锛岀劧鍚庨€夋嫨鍒涘缓绛栫暐銆?/p>

6.鍦ㄧ瓥鐣ュ垪琛ㄤ腑锛岄€変腑鍒氬垰鍒涘缓鐨勭瓥鐣ュ悕绉版梺杈圭殑澶嶉€夋銆傛偍鍙互浣跨敤绛涢€夋潯浠?/b>鑿滃崟鍜屾悳绱㈡鏉ヨ繃婊ょ瓥鐣ュ垪琛ㄣ€?/p>

7.閫夋嫨绛栫暐鎿嶄綔锛岀劧鍚庨€夋嫨闄勫姞銆?/p>

8.閫夋嫨 DataPipelineDefaultRole 鍜?b> DataPipelineDefaultResourceRole 锛岀劧鍚庨€夋嫨闄勫姞绛栫暐銆?/p>

灏嗗瓨鍌ㄦ《绛栫暐娣诲姞鍒?S3 瀛樺偍妗?/b>

鍦ㄧ洰鏍囪处鎴蜂腑锛屽垱寤轰笌浠ヤ笅鍐呭绫讳技鐨?a href="https://docs.aws.amazon.com/AmazonS3/latest/user-guide/add-bucket-policy.html" target="_blank">瀛樺偍妗剁瓥鐣?/a>銆傛浛鎹互涓嬬ず渚嬩腑鐨勭浉鍏冲€硷細

111122223333锛欴ata Pipeline 璐︽埛鐨?ID銆傛湁鍏虫洿澶氫俊鎭紝璇峰弬闃?a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html#FindingYourAWSId" target="_blank">鏌ユ壘 AWS 璐︽埛 ID銆?/li> awsdoc-example-bucket锛歋3 瀛樺偍妗剁殑鍚嶇О 
{
    "Version": "2012-10-17",
    "Id": "",
    "Statement": 
            },
            "Resource": 
        }
    ]
}

鍒涘缓骞舵縺娲荤閬?/b>

1.鍦ㄦ簮璐︽埛涓紝浣跨敤灏?DynamoDB 琛ㄥ鍑哄埌 S3 Data Pipeline 妯℃澘鍒涘缓绠￠亾锛?/p>

鍦?b>鍙傛暟閮ㄥ垎锛岃緭鍏?b>婧?DynamoDB 琛ㄥ悕绉?/b>鍜?b>杈撳嚭 S3 鏂囦欢澶?/b>銆備负瀛樺偍妗朵娇鐢ㄦ牸寮?s3://awsdoc-example-bucket/銆?br /> 鍦?b>瀹夊叏/璁块棶閮ㄥ垎锛岄€夋嫨 IAM 瑙掕壊鐨?b>榛樿璁剧疆銆?/p>

2.婵€娲?/b>绠￠亾涔嬪墠锛岃閫夋嫨鍦?Architect 涓紪杈?/b>銆?/p>

3.鎵撳紑 Activities 閮ㄥ垎锛岀劧鍚庢壘鍒?EmrActivity 瀵硅薄銆?/p>

4.鍦?b>姝ラ瀛楁涓紝娣诲姞 BucketOwnerFullControl 鎴?AuthenticatedRead 棰勮璁块棶鎺у埗鍒楄〃 (ACL)銆傝繖浜涢瑁?ACL 涓?Amazon EMR Apache Hadoop 浣滀笟鎻愪緵鍐欏叆鐩爣璐︽埛涓殑 S3 瀛樺偍妗剁殑鏉冮檺銆傝鍔″繀浣跨敤浠ヤ笅鏍煎紡锛?b>-Dfs.s3.canned.acl = BucketOwnerFullControl銆傚湪 org.apache.hadoop.dynamodb.tools.DynamoDbExport 涓?#{output.directoryPath} 涔嬮棿鎻掑叆璇彞銆備緥濡傦細

s3://dynamodb-dpl-#{myDDBRegion}/emr-ddb-storage-handler/4.11.0/emr-dynamodb-tools-4.11.0-SNAPSHOT-jar-with-dependencies.jar,org.apache.hadoop.dynamodb.tools.DynamoDBExport,-Dfs.s3.canned.acl=BucketOwnerFullControl,#{output.directoryPath},#{input.tableName},#{input.readThroughputPercent}

5.閫夋嫨淇濆瓨锛岀劧鍚庨€夋嫨婵€娲?/b>浠ユ縺娲荤閬擄紝骞跺皢 DynamoDB 琛ㄥ浠藉埌鐩爣璐︽埛涓殑 S3 瀛樺偍妗躲€?/p>

鍦ㄧ洰鏍囪处鎴蜂腑杩樺師澶囦唤

1.鍦ㄧ洰鏍囪处鎴蜂腑锛?a href="https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-importexport-ddb-prereq.html#dp-importexport-ddb-table-cli" target="_blank">鍒涘缓 DynamoDB 琛?/a>銆傝琛ㄤ笉蹇呬负绌恒€備絾鏄紝瀵煎叆杩囩▼灏嗘浛鎹笌瀵煎嚭鏂囦欢涓殑椤圭洰鍏锋湁鐩稿悓閿殑椤圭洰銆?/p>

2. 浣跨敤浠?S3 瀵煎叆 DynamoDB 澶囦唤鏁版嵁 Data Pipeline 妯℃澘鍒涘缓绠￠亾锛?/p>

鍦?b>鍙傛暟閮ㄥ垎鐨?b>杈撳叆 S3 鏂囦欢澶?/b>瀛楁锛岃緭鍏ュ瓨鍌?DynamoDB 澶囦唤鐨?S3 瀛樺偍妗躲€?br /> 鍦?b>瀹夊叏/璁块棶閮ㄥ垎锛岄€夋嫨 IAM 瑙掕壊鐨?b>榛樿璁剧疆銆?/p>

3. 婵€娲荤閬撲互灏嗗浠借繕鍘熷埌鐩爣琛ㄣ€?/p>