AWS - How to encrypt instance launched from a community AMI ?

When we launch a instance from a public community AMI like ubuntu , centos etc, the volume will launch unencrypted. It is because Amazon EBS encryption uses AWS Key Management Service (AWS KMS) customer master keys (CMK) when creating encrypted volumes and any snapshots created from them. The first time we create an encrypted volume in a region, a default CMK is created for us automatically. This key is used for Amazon EBS encryption unless we select a CMK that we created separately using AWS KMS. Now this makes sense since every AWS customer needs to launch from this same public AMI, and we can't all share the same key.

However post launch we can encrypt this and then put our data on this.



1. Post launch , locate the volume to be encrypted. If you see , this volume will be unencrypted.

2. Create a snapshot of this volume.

3. Once snapshot is created and available, locate it and copy the snapshot , while copying there is an option to encrypt the volume.

4. copy it and locate new snapshot which is encrypted and create a AMI using this snapshot.

Service/program Start at startup Ubuntu 16.04

vi /etc/systemd/system/myscript.service

Create a file in /etc/systemd/system and add the following lines.
---------------------------------------------
[Unit]
Description=cifs start script

[Service]
ExecStart=/usr/bin/docker-volume-netshare cifs
Restart=always

[Install]
WantedBy=multi-user.target

---------------------------------------------
Then execute following commands.

sudo systemctl daemon-reload
sudo systemctl enable myscript.service
sudo systemctl start myscript.service

RAID Array of EBS Volumes - Ubuntu

RAID is an acronym for Redundant Array of Independent (or Inexpensive) Disks. In fact, RAID is the way of combining several independent and relatively small disks into a single storage of a large size. The disks included into the array are called array members. The disks can be combined into the array in different ways which are known as RAID levels. Each of RAID levels has its own characteristics of:

Fault-tolerance which is the ability to survive of one or several disk failures.
Performance which shows the change in the read and write speed of the entire array as compared to a single disk.
The capacity of the array which is determined by the amount of user data that can be written to the array. The array capacity depends on the RAID level and does not always match the sum of the sizes of the RAID member disks. To calculate the capacity of the particular RAID type and a set of the member disks you can use a free online RAID calculator.

How RAID is organized?
Two independent aspects are clearly distinguished in the RAID organization.

1.The organization of data in the array (RAID storage techniques: striping, mirroring, parity, combination of them).
2.Implementation of each particular RAID installation - hardware or software.

RAID storage techniques
The main methods of storing data in the array are:

Striping - splitting the flow of data into blocks of a certain size (called "block size") then writing of these blocks across the RAID one by one. This way of data storage affects on the performance.
Mirroring is a storage technique in which the identical copies of data are stored on the RAID members simultaneously. This type of data placement affects the fault tolerance as well as the performance.
Parity is a storage technique which is utilized striping and checksum methods. In parity technique, a certain parity function is calculated for the data blocks. If a drive fails, the missing block are recalculated from the checksum, providing the RAID fault tolerance.
All the existing RAID types are based on striping, mirroring, parity, or combination of these storage techniques.

RAID levels
RAID 0 - based on striping. This RAID level doesn't provide fault tolerance but increases the system performance (high read and write speed).
RAID 1 - utilizes mirroring technique, increases read speed in some cases, and provides fault tolerance in the loss of no more than one member disk.
RAID 0+1 - based on the combination of striping and mirroring techniques. This RAID level inherits RAID 0 performance and RAID 1 fault tolerance.
RAID1E - uses both striping and mirroring techniques, can survive a failure of one member disk or any number of nonadjacent disks. There are three subtypes of RAID 1E layout: near, interleaved, and far. More information and diagrams on the RAID 1E page.
RAID 5 - utilizes both striping and parity techniques. Provides the read speed improvement as in RAID 0 approximately, survives the loss of one RAID member disk.
RAID 5E - a variation of RAID 5 layout the only difference of which is an integrated spare space allowing to rebuild a failed array immediately in case of a disk failure. Read more on the RAID5E page.
RAID 5 with delayed parity - pretty similar to basic RAID 5 layout, but uses nonstandard scheme of striping. More information about RAID5 with delayed parity.
RAID 6 - similar to RAID 5 but uses two different parity functions. The read speed is the same as in RAID 5.

How to make RAID Array in Linux with AWS EBS volumes.

sudo mdadm --create --verbose /dev/md0 --level=0 --name=MY_RAID --raid-devices=4 /dev/xvdb /dev/xvdc /dev/xvdd /dev/xvde
sudo mkfs.ext4 -L MY_RAID /dev/md0
sudo mkdir /var/lib/docker
sudo mount LABEL=MY_RAID /var/lib/docker

To mount on startup, edit the file.

sudo vi /etc/fstab

Add following line and save.

LABEL=MY_RAID /var/lib/docker ext4 defaults 0 0

Now to mount this on system startup , we have to add an entry in fstab.

You could do it using sudo vi /etc/fstab.
For example, if you add
LABEL=MY_RAID /var/lib/docker ext4 defaults 0 0

It means that the device/partition located at MY_RAID will be mounted to /var/lib/docker using the file system ext4, with default mount options and no dumping and no error-checking enabled.

After we've added the new entry to /etc/fstab, we need to check that our entry works. Run the sudo mount -a command to mount all file systems in /etc/fstab.

sudo mount -a

If the previous command does not produce an error, then your /etc/fstab file is OK and your file system will mount automatically at the next boot. If the command does produce any errors, examine the errors and try to correct your /etc/fstab.

Static Code Analysis with Sonarqube

Sonar is a web based code quality analysis tool for Maven based Java projects. It covers a wide area of code quality check points which include: Architecture & Design, Complexity, Duplication, Coding Rules, Potential Bugs, Unit Test etc.

Make sure java is inatlled.
If not install uisng following command.

sudo apt-get install default-jre

1. Download and unzip the SonarQube distribution (let's say in "/etc/sonarqube")
2. Start the SonarQube server:
/etc/sonarqube/bin/[OS]/sonar.sh console

3. Download and unzip the SonarQube Scanner (let's say in "/etc/sonar-scanner")
4. Analyze a project:
Go to projects root directory and run follwing command.
/etc/sonar-scanner/bin/sonar-scanner -Dsonar.projectKey=DSP -Dsonar.sources=. -Dsonar.scm.disabled=true

6. Browse the results at http://localhost:9000 (default System administrator credentials are admin/admin)

SVN to GIT MIGRATION with Branches and Tags

git svn clone -r1:HEAD -s svnurl svnfolder

git branch
git branch -a


git remote add origin giturl.git

git checkout -b branchname origin/branchname
git checkout -f -b branchname origin/branchname

git push origin master branchname

git checkout origin/tag/tagname
git tag -a tagname -m "creating tag tagname"
git push origin tagname


You can use the following java code if there are too many branches and tags.
Keep all branches in file "branches.txt" and tags in file "taglist.txt"

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;

public class SVNGitMigrator {

 public static void main(String[] args) throws IOException {

  try {

   StringBuilder a = new StringBuilder("git push origin master ");
   File f = new File("branches.txt");
   BufferedReader b = new BufferedReader(new FileReader(f));
   String readLine = "";
   while ((readLine = b.readLine()) != null) {
    System.out.println("git checkout -f -b " + readLine + " origin/" + readLine);
    a.append(readLine+" ");
   }
   b.close();
   
   f = new File("taglist.txt");
   b = new BufferedReader(new FileReader(f));
   readLine = "";
   while ((readLine = b.readLine()) != null) {
    System.out.println("git checkout origin/tags/" + readLine);
    System.out.println("git tag -a " + readLine + " -m \"creating dsp tag " + readLine+"\"");
    a.append(readLine+" ");
   }
   System.out.println(a);
   b.close();
   
  } catch (IOException e) {
   e.printStackTrace();
  }

 }

}