This Banner is For Sale !!
Get your ad here for a week in 20$ only and get upto 15k traffic Daily!!!

Paginate direct AWS SDK calls with AWS Step Functions


AWS Step Capabilities AWS SDK integration helps you to name large number of AWS providers immediately out of your Step Capabilities workflow.

For API calls that may return a big checklist of things, APIs are returning by default solely the primary set of outcomes. For instance, S3 checklist objects response returns by default max. 1000 objects. Remainder of the outcomes have to be requested by offering pagination token on the request.

For knowledge processing, pagination may be very helpful and necessary sample. Dividing a consequence set to fastened measurement pages makes it simpler to construct for instance significant retry logic for error dealing with. Additionally, executing partial consequence units in parallel can enhance a workflow working time considerably.

On this instance I’m exhibiting how itemizing objects(arn:aws:states:::aws-sdk:s3:listObjectsV2) on S3 bucket after which triggering a processing step with batch of S3 objects could be applied in Step Capabilities ASL(Amazon State Language).

Be aware: some AWS APIs use NextToken to paginate the outcomes. Workflow with pagination continues to be identical as I’m overlaying subsequent with ContinuationToken.



How one can implement pagination with ASL.

Cloudformation template

This instance exhibits quite simple stream of itemizing objects on S3 buckets after which triggering the processing step.

workflow

Under is the ASL definition. BatchSize parameter controls what number of S3 objects are included in every processing batch. We hold requesting new batches so long as the response is together with IsTruncated: true. Measurement of final object batch is under BatchSize with IsTruncated: false so we are able to end processing.

{
    "Remark": "Checklist S3 objects.",
    "StartAt": "list_s3",
    "States": {
        "list_s3": {
            "Remark": "Get first batch of objects.",
            "Sort": "Job",
            "Useful resource": "arn:aws:states:::aws-sdk:s3:listObjectsV2",
            "ResultPath": "$.s3_objects",
            "Parameters": {
                "Bucket": "${BucketName}",
                "MaxKeys": ${BatchSize}
            },
            "Subsequent": "process_s3_objects"
            },
        "process_s3_objects": {
            "Remark": "Processing logic. Now we simply wait.",
            "Sort": "Wait",
            "Seconds": 2,
            "Subsequent": "check_if_all_listed"
            },
        "check_if_all_listed": {
            "Sort": "Selection",
            "Decisions": [
                {
                "Variable": "$.s3_objects.IsTruncated",
                "BooleanEquals": false,
                "Next": "success_state"
                }
                ],
            "Default": "list_s3_with_continuation_token"
            },
        "list_s3_with_continuation_token": {
            "Remark": "Get subsequent batch of objects. Present ContinuationToken within the request.",
            "Sort": "Job",
            "Useful resource": "arn:aws:states:::aws-sdk:s3:listObjectsV2",
            "ResultPath": "$.s3_objects",
            "Parameters": {
                "Bucket": "${BucketName}",
                "MaxKeys": ${BatchSize},
                "ContinuationToken.$": "$.s3_objects.NextContinuationToken"
                },
            "Subsequent": "process_s3_objects"
            },
        "success_state": {
            "Sort": "Succeed"
            }
     }
}
Enter fullscreen mode

Exit fullscreen mode



Wrapping up

AWS Step Capabilities is an ideal match for coordinating workflows and orchestrating AWS providers. I strongly advocate constructing library of fine templates for getting a working begin for adapting it to your use circumstances.

The Article was Inspired from tech community site.
Contact us if this is inspired from your article and we will give you credit for it for serving the community.

This Banner is For Sale !!
Get your ad here for a week in 20$ only and get upto 10k Tech related traffic daily !!!

Leave a Reply

Your email address will not be published. Required fields are marked *

Want to Contribute to us or want to have 15k+ Audience read your Article ? Or Just want to make a strong Backlink?