Showing posts with label JAVA. Show all posts
Showing posts with label JAVA. Show all posts

Sunday, September 24, 2017

Cleaning orphan snapshots in AWS EC2 to save $




When we deregister an Amazon EBS-backed AMI, it doesn't affect the snapshots that were created during the AMI creation process. We'll continue to incur storage costs for these snapshots. Therefore, if we are finished with the snapshots, we should delete it.

In fact, they cant do the mistake of cleaning up snapshots, its a revenue for them!

So we will have to take care of cleaning up snapshots.
Here is how we can do it from AWS Java SDK.

Download AWS Java SDK from here
Add to java classpath/buildpath.

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.ec2.AmazonEC2Client;
import com.amazonaws.services.ec2.model.DeleteSnapshotRequest;
import com.amazonaws.services.ec2.model.DescribeImagesRequest;
import com.amazonaws.services.ec2.model.DescribeImagesResult;
import com.amazonaws.services.ec2.model.DescribeSnapshotsRequest;
import com.amazonaws.services.ec2.model.DescribeSnapshotsResult;
import com.amazonaws.services.ec2.model.Image;
import com.amazonaws.services.ec2.model.Snapshot;

public class FindSanpshots {

 public static void main(String[] args) throws IOException {

  BasicAWSCredentials basicAWSCredentials = new BasicAWSCredentials("xx", "yyy");
  AmazonEC2Client amazonEC2Client = new AmazonEC2Client(basicAWSCredentials);
  Region region = Region.getRegion(Regions.fromName("us-west-2"));
  amazonEC2Client.setEndpoint(region.getServiceEndpoint("ec2"));

  DescribeImagesRequest withOwners = new DescribeImagesRequest().withOwners("self");
  DescribeImagesResult images = amazonEC2Client.describeImages(withOwners);
  ArrayList < String > imageIdList = new ArrayList < String > ();
  List < Image > amiList = images.getImages();
  for (Image image: amiList) {
   imageIdList.add(image.getImageId());
  }

  DescribeSnapshotsRequest withOwnerIds = new DescribeSnapshotsRequest().
  withOwnerIds("self");
  DescribeSnapshotsResult describeSnapshots = amazonEC2Client.
  describeSnapshots(withOwnerIds);
  List < Snapshot > snapshots = describeSnapshots.getSnapshots();

  // ensure snapshot size and ami size in your region.
  System.out.println(snapshots.size());
  System.out.println(amiList.size());

  int count = 0;
  int size = 0;

  // find orphans and delete.

  for (Snapshot snapshot: snapshots) {

   String description = snapshot.getDescription();

   // get AMI id of snapshot using regex from its description.
   Pattern pattern = Pattern.compile("for(.*?)from");
   Matcher matcher = pattern.matcher(description);
   while (matcher.find()) {

    String amiId = matcher.group(1).trim();
    // ami id is currently 12 character long. 
    if (!imageIdList.contains(amiId) && amiId.length() <= 12) {
     String snapshotId = snapshot.getSnapshotId();
     DeleteSnapshotRequest r =
      new DeleteSnapshotRequest(snapshotId);
     amazonEC2Client.deleteSnapshot(r);
     System.out.println(amiId);

     size += snapshot.getVolumeSize();
     count++;
    }
   }
  }
  System.out.println("Orphan Snapshots Deleted : " + count);
  System.out.println("Orphan Snapshots Size : " + size);

 }

}

Saturday, November 26, 2016

How to profile JVM running on a remote server docker container.

On a Ubuntu desktop install JProfiler GUI from here.

Download script and run it.
http://download-keycdn.ej-technologies.com/jprofiler/jprofiler_linux_9_2.sh

Provide key if you have , else use evaluation.

Now on the Ubuntu server where docker container is running, you need to stop containers first and modify Dockerfile to add jprofiler

wget http://download-keycdn.ej-technologies.com/jprofiler/jprofiler_linux_9_2.tar.gz && \
tar -xzf jprofiler_linux_9_1_1.tar.gz -C /usr/local

Also expose the port 8849 from container to host so that you can connect from your desktop.

If you use docker-compose.yml, map the port in the service , which you want to connect.
For example

version: "2"
services:
  some_service:
    build: .
    ports:
      - "8849:8849"
    depends_on:
      - "db"
    entrypoint:


This will download jprofiler 9.2 and unpack it in /usr/local when you run docker container next time and map the port on host server.

Once docker container is up , you can ssh to it
docker exec -it /bin/bash
And just start the jprofiler using following command.
/usr/local/jprofiler9/bin/jpenable
You will be asked for 2 options , 1) GUI connect 2. Using config.xml

Use option 1 GUI connect
Now from Desktop you can open JProfiler UI and connect to server-ip:8849
You can profile the JVM you want.


Friday, February 22, 2013

Memory Profiling with JProfiler ( Basics )


Let’s see how we can use JProfiler to fix the memory leaks of a Java Application. JProfiler is a Java profiling tool targeted at Java EE and Java SE applications.

Profile means “drawing of the outline of anything” or summarize (  Read interesting story about the origin of this word  ) . In software engineering,  profiling is a form of dynamic program analysis that measures, for example, the space (memory) or time complexity of a program, the usage of particular instructions, or frequency and duration of function calls.

Let us take an example of a Java Application and try to solve a simple memory leak with JProfiler.

1.      Install JProfiler.
2.      Create a Simple Eclipse RCP Application with a View which has a memory leak.
3.      Run JProfiler and identify the leak.
4.      Fix the leak.
5.      Run JProfiler again to verify that leak is fixed.

Install JProfiler.


Download JProfiler evaluation version http://www.ej-technologies.com/download/jprofiler/files

During installation integrate JProfiler with you eclipse.



Create a Simple Eclipse RCP Application View with a memory leak.


1. Launch your eclipse IDE with a fresh workspace.
2. Go to File --> New --> Plugin Project.


3. Give a name to the Project and Press Next.

4. Select Yes for Rich Client Application and Press Next.

5. Select RCP Application with a view as shown below and press next.



In the next window, select Add Branding and say Finish.

This will take you to a Plugin development Perspective and your package explorer view should look something like below.


Now to introduce a memory leak we have to make some changes to our RCP. 

Open Perspective.java and  under createInitialLayout() method, delete the following line.

           layout.setFixed(true);

This is to make sure that we should be able to close the view. By default it is fixed and not closable.

Now open ApplicationActionBarAdvisor.java and override the following method.

@Override
protected void fillMenuBar(IMenuManager menuBar) {

IMenuManager mainMenu = getActionBarConfigurer().getMenuManager();
MenuManager windowMenu = new MenuManager("&Window",IWorkbenchActionConstants.M_WINDOW);
mainMenu.add(windowMenu);
windowMenu.add(ActionFactory.RESET_PERSPECTIVE.create(getActionBarConfigurer().getWindowConfigurer().getWindow()));

super.fillMenuBar(menuBar);

}

This is to make sure that after closing the view, we can bring it back by selecting the Reset Perspective Menu Action. This we need to test memory leak.

Now to introduce the memory leak, add a new singleton class and keep a HashMap in this singleton which stores our view as key by a strong reference.

package rcpwithmemoryleak;
import java.util.HashMap;
import org.eclipse.ui.part.ViewPart;

public class MemoryLeakerSingleton {

private static MemoryLeakerSingleton singleton;
HashMap<ViewPart, String> viewMap = new HashMap<ViewPart, String>();

private MemoryLeakerSingleton() {
}

public static MemoryLeakerSingleton getInstance() {
       if (singleton == null) {
              singleton = new MemoryLeakerSingleton();
       }
       return singleton;
}

public void add(ViewPart v) {
       viewMap.put(v, "Test");
}

}

The class should look something like above.  Now we add our view to this map whenever we create the view. This can be done by adding a line of code in createPartControl method of View.java

MemoryLeakerSingleton.getInstance().add(this);

Note that this kind of scenario we face in the real word when we use listeners or singletons which have the UI objects Hash Map.   Now run your RCP application. Right Click on the plugin and say RunAs à EclipseApplication.  You should see the application as below.


Run JProfiler and identify the leak.

Come back to your eclipse IDE.

You should have a toolbar item called Attach with JProfiler as shown below. Using this  tool you can attach Profiler with your running RCP application.


Click the tool and launch the Jprofiler. Open the Jprofiler in new window. Enter the evaluation license key if you have got one.

Select the Process Name you want to profile and press OK. In our case it is eclipse application.


By default JProfiler shows all the classes in memory. We can view according packages as well by changing the aggregation level.


OK. Now we are interested to see our RCP classes. It is very easy to do that. There is a View Filter below the window where you can type the fully qualified class name or package name to see all the classes in that package.  In our case package name is rcpwithmemoryleak. When you type this it shows all the classes.

Note that we have once instance of rcpwithmemoryleak.View.

Now you come to the RCP application and close the view by pressing X mark on the view and then come to JProfiler and press Run GC button on the tool bar. Now ideally after closing the view, the instance should be collected by Garbage Collector, but you still see once instance of View as it is being held by out Singleton Leaker.

You can reactivate the view by pressing Window à Reset Perspective. Close again and Reactivate again. Do it for multiple times and check in the JProfiler by Running the GC.

The second picture below shows the leak. I closed and open the view 6 times and old instances are not cleaned up.

 

Memory Leak for View Object

Now to check who is holding the view, you can take the heap dump and analyze.
Right click on the object of interest and say “Show Selection in Heap Walker”. This will take you to the Heap Walker. Select the Reference Tab from the below Tab Bar and select Incoming References from the combo box.


 

You can clearly see here which is holding your View in the memory.  I used Show Paths to GC Root option to see this tree. It directly takes you to the GC root.

Fix the leak.


Now you can go to the exact problematic point and fix your memory leak.
In our case we can fix this in two ways.

1.      Provide remove method in the MemoryLeakerSingleton like below

public void remove(ViewPart v) {
                     viewMap.remove(v);
       }

           Call this from the dispose() method of View as shown below.

@Override
              public void dispose() {
                     MemoryLeakerSingleton.getInstance().remove(this);
                     super.dispose();
       }


2.    You can make you HashMap as WeakHashMap , then GC will automatically collect these weak references.

Run JProfiler again to verify that leak is fixed.


Now you can run the RCP again after the code changes and rerun the profiler and do the steps described above. You should not see any instances if the view is closed. Your leak is fixed!
-------------------------------------------------------------------------------------------------------------------------------------------------------------------

One can use a word perfectly well under most circumstances without knowing its formal dictionary definition, but knowing that definition will certainly allow you to use it better. Knowing its etymology may bring out more nuances and connotations to the current semantics that you didn't realize are currently there. Here I've tried to give the etymology of word 'profile'.

Let’s now try to understand the etymology of word Profile.  The word Profile is made up of two words: Pro and File. Pro has been derived from Sanskrit Pra (प्र) means forth. 

What about file ? Here we have a big story about it. 

There is a root word (धातु) in Sanskrit called Shi (शि) which means Sharpen (निशाने). Based on this root, Sanskrit has derived a word - Shira (शिरा) which means a thin vein. In Sanskrit there is a rule which says “No difference between R and L” (रलयोरभेदः). Thus Shira became Shila (शिला). R got kicked by L.  

Now Sanskrit Shila (शिला) became žila in Slovak and Czech (žila and shila are very close in pronunciation) carrying same meaning (vein) and in Spanish – filón. Other few languages used this word for calling a thread or string (since thread/ string resembles a vein). Check the words used for thread/string in different languages below.

Croatian          žica
Latin                filum
Italian              fila, filo, filza
Catalan            fil
Galician           fío
French             fil

English used the word file. Earlier the word file was used for communicating “a string or wire on which documents are strung”. Now the word file means a document holder.  

In this way we can say that the meaning of word Profile is before getting into details drawing an outline of the subject. 

The journey of the words is really amazing! 

---------------------------------------------------------------------------------------------------------