How to List S3 Objects and Handle Pagination

When you list objects in an S3 bucket, the API might return only a limited number of objects in a single response. If your bucket has a lot of files, you may need to get more pages to see the full listing.

This guide demonstrates how to list objects in Cyberin S3 Object Storage, filter the results by prefix, retrieve additional pages, and troubleshoot listing errors using the S3 API and AWS CLI.

Requirements

Before you start, make sure you have:

Use the endpoint provided by your Cyberin. Don’t assume a DigitalOcean endpoint or region is correct for your account.

1. List Objects in an S3 Bucket

To list objects stored in a bucket, you use the ListObjectsV2 operation.

Open your terminal and execute:

aws s3api list-objects-v2 \
  --bucket YOUR_BUCKET_NAME \
  --endpoint-url "YOUR_CYBERIN_S3_ENDPOINT"

Replace YOUR_BUCKET_NAME with your bucket name and YOUR_CYBERIN_S3_ENDPOINT with the endpoint provided by Cyberin.

AWS CLI normally retrieves subsequent pages automatically when pagination is enabled.

A successful response may look like this:

{
  "Contents": [
    {
      "Key": "images/photo.jpg",
      "Size": 245760
    },
    {
      "Key": "documents/report.pdf",
      "Size": 512000
    }
  ],
  "KeyCount": 2,
  "IsTruncated": false
}

This is an example response. The actual results depend on the objects stored in your bucket.

2. List Objects Under a Specific Prefix

If you need to find files stored under a particular path, use the --prefix parameter.

For example, to list objects whose keys begin with images/, run:

aws s3api list-objects-v2 \
  --bucket YOUR_BUCKET_NAME \
  --prefix "images/" \
  --endpoint-url "YOUR_CYBERIN_S3_ENDPOINT"

This returns objects whose keys start with images/.

Note: S3 uses object keys and prefixes rather than traditional folders. For example, images/photo.jpg is an object key, and images/ is its prefix.

If you get no objects, try to list the bucket (without the prefix) to see if the objects exist in a different path.

3. Obtain Objects in Page

If you have a bucket with many objects, you can ask for a limited number of objects with each response of an API operation and use pagination to get the rest of the results.

Step 1: Request the first page

Use the following command to ask for up to 100 objects in one API response:

aws s3api list-objects-v2 \
  --bucket YOUR_BUCKET_NAME \
  --max-keys 100 \
  --no-paginate \
  --endpoint-url "YOUR_CYBERIN_S3_ENDPOINT"

The response may contain fewer than 100 objects.

Check these fields in the response:

Step 2: Check Whether More Results Are Available

If the response contains:

{
  "IsTruncated": true,
  "NextContinuationToken": "EXAMPLE_TOKEN"
}

additional results are available. The token shown here is illustrative; use the actual token returned by your request.

Step 3: Retrieve the Next Page

Pass the returned token using --continuation-token:

aws s3api list-objects-v2 \
  --bucket YOUR_BUCKET_NAME \
  --max-keys 100 \
  --no-paginate \
  --continuation-token "YOUR_RETURNED_TOKEN" \
  --endpoint-url "YOUR_CYBERIN_S3_ENDPOINT"

If the new response still shows IsTruncated: true, repeat the process using the latest NextContinuationToken.

Continue until IsTruncated is false. If it is false, the listing has reached the end.

Important: Copy the token from the preceding response exactly as returned. Do not create your own token or use a token from an unrelated listing request.

4. Troubleshoot Common Listing Errors

Access Denied (403)

Possible causes:

Solution:

  1. Verify the access key and secret key.

  2. Confirm that the credentials have the required bucket-listing permission.

  3. Check the endpoint and signing region.

  4. Retry the request. If it still fails, contact Cyberin support.

Never share your secret key in support messages.

No Such Bucket

Possible causes:

Solution:

  1. Copy the exact bucket name from your Cyberin storage account.

  2. Verify the endpoint supplied by Cyberin.

  3. Confirm that your credentials belong to the expected account.

  4. Run the command again.

The Listing Does Not Show All Objects

Possible causes:

Solution:

  1. Check the IsTruncated field.

  2. If it is true, retrieve the next page using NextContinuationToken.

  3. If you want AWS CLI to retrieve pages automatically, remove --no-paginate.

  4. If using an application or script, ensure it processes every page.

No Objects Are Returned

Possible causes:

Solution:

  1. Run the command without --prefix.

  2. Check the spelling and capitalization of the prefix.

  3. Verify the bucket name and endpoint.

  4. Confirm that the expected files exist in the selected bucket.

Invalid Continuation Token

Possible causes:

Solution:

  1. Start a new listing request.

  2. Copy the complete NextContinuationToken from the response.

  3. Pass it unchanged to the next request.

  4. Keep the bucket and listing parameters consistent.

Additional Resources

Note: These resources describe DigitalOcean Spaces. Use Cyberin's own endpoint and configuration details, and confirm compatibility if an API command behaves differently.