Developing java web application powered by Amazon S3.

More and more people are tilting towards cloud deployment and defacto go to guys for supporting their cloud app is Amazon AWS.  Apart for providing highly scalable computing capabilities,  AWS also throws out lot of cool services which we can directly leverage while developing our application. One such service is Amazon S3.

What is Amazon S3?

S3 is a highly scalable, durable and virtually limitless storage service for any kind of web-object like files, pictures, videos etc. S3 can store an object of upto 5 terabyte of size.

Buckets and keys – All the objects in S3 are stored within a logical container called Bucket. Bucket acts like a namespace so its name should be unique across all the buckets created across all the amazon accounts. Bucket name has certain restrictions to it. It can be between 3-63 character long and should be lowercase and alphanumeric. There are a bunch of restrictions out there, the complete listing can be found here  http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html. Any object stored in a bucket has a unique key within that bucket. For example if you store an object call mypic.jpg in a bucket called “demobucket” the mypic.jpg becomes a key. The key is sequence of Unicode character whose UTF-8 encoding are at-most 1024 bytes long.

Some examples of keys are

mypic.jpg
thumbnails/mypic.jpg
thumbnails/mypic1.jpg
medium/mypic.jpg
original/mypic.jpg

In nutshell bucket provides a folder like behavior for accessing and storing object. Objects can be accessed via key which are unique within the bucket. There is no limit to number of objects we can store in a bucket. But for each account we can create only 100 buckets.

So how can be use this awesome storage engine in our application? Awazon provides sdk for java which can be directly downloaded from here http://aws.amazon.com/sdkforjava/. In this article we will see how to download this sdk and use it to store photos in S3. Please note that amazon charges for storage space you are using, but for few MBs it is almost 0. So here are the steps.

1. Signup for your amazon account. You will need a credit card for this.

2. Now logon to your console “console.aws.amazon.com”. In your console home you will see lots of amazon services listed. Search for S3 and and click on the link.

3. On the s3 home click on create bucket. It is a button of top left of your screen. Select a unique name and region of your choice. Choose your region carefully as once your bucket is created in a region it will stay there for lifetime. There are nitty gritties here which I don’t want to go into.

4. You can provide different access controls to a bucket like read, write, all to different accounts, ip etc. Again not going into details.

5. Once you are done you need to download your security credentials. For this go to “My Account”/”security credentials”. What we are after is access credentials. You will see that there are 3 types of access credentials one of them is “Access Keys”. We need this to call all the rest webservice like S3. We will create a new set of access keys and store the access key and secret key is a safe location.

6. Now in your webapp download the  sdk for java from here. http://aws.amazon.com/sdkforjava. In lib folder of sdk copy the aws-java-sdk-1.3.33.jar to your webapp lib and make sure that it is in class path.

Maven user can add following dependency in their pom

<dependency>
	<groupId>com.amazonaws</groupId>
	<artifactId>aws-java-sdk</artifactId>
	<version>1.3.33</version>
</dependency>

7. Create a file called “AwsCredentials.properties” store this in WEB-INF/classes of your webapplication. This file will contain two main properties

accessKey =
secretKey =

The values for these properties are the one you got in step 5 of the tutorial

8. Now we can get our hand dirty with the code.  I am pasting the code directly here as it is simple. The AWS s3 client is thread safe and also it used heavy objects like HttpClient and HttpConnection so we will create it as singleton

package com.aranin.adconnect.util.aws;

import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;

import java.io.File;
import java.io.FileInputStream;
import java.util.List;
import java.util.Properties;

/**
 * Created by IntelliJ IDEA.
 * User: Niraj Singh
 * Date: 3/13/13
 * Time: 10:52 AM
 * To change this template use File | Settings | File Templates.
 */
public class AWSStorageUtil {
    private BasicAWSCredentials credentials;
    private AmazonS3 s3;
    private String bucketName;
    private static volatile AWSStorageUtil  awsstorageUtil = new  AWSStorageUtil();
    private   AWSStorageUtil(){
        try{
            Properties properties = new Properties();
            properties.load(new FileInputStream("D:/samayik/adkonnection/src/main/resources/AwsCredentials.properties"));
            this.credentials = new   BasicAWSCredentials(properties.getProperty("accessKey"),
                                                         properties.getProperty("secretKey"));
            this.bucketName = "aranin";
            this.s3 = new AmazonS3Client(this.credentials);

            /*
               You can use this in your web app where    AwsCredentials.properties is stored in web-inf/classes
             */
            //AmazonS3 s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());

        }catch(Exception e){
            System.out.println("exception while creating awss3client : " + e);
        }
    }

    public static AWSStorageUtil getInstance(){
        return awsstorageUtil;
    }

    public static AmazonS3 getAWSClient(){
         return awsstorageUtil.s3;
    }

    public static AmazonS3 getBucketName(){
         return awsstorageUtil.s3;
    }

    public void upload(File file){
        /**
         * key should be unique. an whatever key you set will be used to in url path to access the pic.
         */
        String key = "ads/" + file.getName();
        s3.putObject(this.bucketName, key, file);
    }

    public List<Bucket> listBuckets(){
        for (Bucket bucket : s3.listBuckets()) {
                System.out.println(" bucket name - " + bucket.getName());
        }

        return s3.listBuckets();
    }

    public void getObjectList(){
        System.out.println("Listing objects");
        ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()
                .withBucketName(bucketName)
                .withPrefix("ads"));
        for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
            System.out.println(" - " + objectSummary.getKey() + "  " +
                               "(size = " + objectSummary.getSize() + ")");
        }
    }

    public void getFileFromS3(String key){
        System.out.println("Downloading an object");
        S3Object object = s3.getObject(new GetObjectRequest(this.bucketName, key));
        System.out.println("Content-Type: "  + object.getObjectMetadata().getContentType());

    }

    public static void main(String args[]){
        File file = new File("D:/pics/cam/IMG_1201.jpg");
        AWSStorageUtil aWSStorageUtil = AWSStorageUtil.getInstance();
        //aWSStorageUtil.upload(file);
        aWSStorageUtil.getObjectList();

    }

}

The objects you upload can be retrieved easily using the key via the url. So lets say you have created the bucket in Singapore region and key is ads/IMG_1201.jpg then you can retrieve the object using url

https://s3-ap-southeast-1.amazonaws.com/<bucketName>/<keyname>

Picture uploaded in this example is

https://s3-ap-southeast-1.amazonaws.com/aranin/ads/IMG_1201.jpg

The guy wearing light blue TShirt is me :-) .

This is all I have to say, you guys are smart enough to extrapolate from here so I will leave it here. If you have questions feel free to drop a comment here or you can write me a mail at singh.niraj@aranin.com and I will love to answer it. (Although I am not an expert).

Bye and Take Care

Niraj

 

Print Friendly

About Niraj Singh

I am CEO and CoFounder of a startup "Aranin Software Private Limited, Bangalore. I completed my graduation in 2002 as an Aerospace Engineer from IIT Kharagpur. I love working on new ideas and projects and recently released my first open source project JaiomServer "http://jaiomserver.org". I have 9 years of experience in IT industries most of which I have spent in developing community applications for various clients using java. Some of the sites in which I have actively involved with are hgtv.com, food.com, foodnetwork.com, pickle.com, diynetwork.com etc.
This entry was posted in Amazon Web Services, Cloud Computing, Java, tutorial and tagged , . Bookmark the permalink.

17 Responses to Developing java web application powered by Amazon S3.

  1. Pingback: how to build

  2. markos says:

    I was really interested in this and glad I found the info here!

  3. Wow, amazing blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your web site is excellent, as well as the content!. Thanks For Your article about Developing java web application powered by Amazon S3. | Java Logs .

  4. Hi there, i read your blog from time to time and i own a similar one and i
    was just wondering if you get a lot of spam responses?
    If so how do you protect against it, any plugin or
    anything you can suggest? I get so much lately it’s driving me crazy so any assistance is very much appreciated.
    foreclosures Milford CT recently posted…foreclosures Milford CTMy Profile

  5. Hello would you mind sharing which blog platform you’re using? I’m
    looking to start my own blog soon but I’m having a hard time making a decision between BlogEngine/Wordpress/B2evolution and Drupal. The reason I ask is because your layout seems different then most blogs and I’m looking for
    something completely
    unique. P.S Sorry for getting off-topic but
    I had to ask!
    Milford CT apartments recently posted…Milford CT apartmentsMy Profile

  6. Hi there, I read your blog on a regular basis.
    Your humoristic style is witty, keep it up!
    http://www.blackpeoplemeet.Com recently posted…http://www.blackpeoplemeet.ComMy Profile

  7. Baju Bayi says:

    Wow, superb blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your site is excellent, let alone the content!. Thanks For Your article about Developing java web application powered by Amazon S3. | Java Logs .

  8. Smithe248 says:

    Very good blog post.Really thank you! Fantastic. ekedcdgfdd

Leave a Reply

Your email address will not be published. Required fields are marked *

Connect with Facebook

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

CommentLuv badge