S3Proxy Setup with Azure and local filesystems

This is part of an idea on how to support AWS S3 within Azure, at present there are two possible ways:

Spin up an instance of an Object Store – such as StorageGRID – possible but expensive

S3Proxy – an implementation to proxy requests from S3 to Azure Blob – once again a working scable solution, but at present has limited support, excluding versioning, policies, tagging etc.

Reference here: https://github.com/gaul/s3proxy

Downloads here: https://github.com/gaul/s3proxy/releases

S3 Proxy implements the S3 API, and proxies’ requests to Azure, and other platforms including Backblaze, EMC Atmos, GCP and Openstack swift.

Usefully it can also act as a proxy to the local filesystem for testing.

There are limitations though:

S3Proxy has broad compatibility with the S3 API, however, it does not support:

• ACLs other than private and public-read
• BitTorrent hosting
• bucket logging
• bucket policies
• CORS bucket operations like getting or setting the CORS configuration for a bucket. S3Proxy only supports a static configuration.
• hosting static websites
• object server-side encryption
• object tagging
• object versioning
• POST upload policies
• requester pays buckets
• select object content

Configure s3proxy with a local file system.

Anyway, less of the problems, and onto the setup, firstly we will configure with a local filesystem as the target, and then secondily will move onto using Azure Blob as the target.

Download the s3proxy executable, and change permissions to execute the file.

Chmod +x s3proxy

Run ‘s3proxy –properties’, and you should get

Run an initial test on the local filesystem, to see how it works.

Create the following file, s3proxy.conf

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy

Create the required directory:

mkdir /tmp/s3proxy

Then run the command:

S3proxy –properties s3proxy.conf

Now create a bucket and write some files.

Create a Bucket
$ curl --request PUT http://localhost:8080/testbucket

Check
$ curl http://localhost:8080/

Copy a file
$ aws s3 cp s3proxy s3://testbucket/s3proxy --endpoint-url http://localhost:8080 --no-verify-ssl


List the buckets
$ aws s3 ls --endpoint-url http://localhost:8080 --no-verify-ssl


List the bucket ‘testbucket’
$ aws s3 ls testbucket --endpoint-url http://localhost:8080 --no-verify-ssl

Other commands that can be used:
Create Bucket (using aws s3api)
$ aws s3api create-bucket --bucket test2 --endpoint-url http://localhost:8080 --no-verify-ssl

List Buckets
$ aws s3api list-buckets --endpoint-url http://localhost:8080 --no-verify-ss

List Objects in bucket test
$ aws s3api list-objects --bucket test --endpoint-url http://localhost:8080 --no-verify-ssl 

Put Object
$ aws s3api put-object --bucket test --key azure.conf --endpoint-url http://localhost:8080 --no-verify-ssl

Delete Object
$ aws s3api delete-object --bucket test --key azure.conf --endpoint-url http://localhost:8080 --no-verify-ssl

Configure s3proxy with Azure.

As above but changing the configuration file to something similar to:

The last two comments lines, are what you pull from the Blob Access Keys section in the Azure Portal, so you can therefore work out what goes in the icloud.identity and iclouds.credentials sections….

Then of course rerun the profile command, remembering to stop you old one first of it will not bind the the same TCP port.