Amazon EC2 – Build your own AMI

July 5th, 2012

So I like to know what I’m running. I don’t trust other people enough to run their AMI’s. So I’m going to show you how to build a CentOS 6 AMI.

First we need to launch a platform to use as our build enviroment. The Amazon Linux AMI works just fine for this since it has yum and supports ext4 nativley. Use a 32 or 64 bit to match the AMI you plan to build.

The default user for Amazon Linux is ec2-user instead of root. ec2-user has full sudo privliges. Once logged in you can just sudo su - to get to root.

Once you have your build instance up, create a EBS volume in the same Availability Zone and attach it to your instance. This will be the main disk of your image. I made mine 8 GB, but you can make it any size you want for your root partition.

Fist we need to install some tools.

# yum install MAKEDEV

Now lets create a filesystem on the EBS and mount it.

# mkdir /ami 
# mkfs.ext4 /dev/xvdf 
# mount /dev/xvdf /ami/ 
# e2label /dev/xvdf root

Next we need to set up some scaffolding directories.

# mkdir -p /ami/{dev,etc,proc,sys}

The /dev directory needs a handful of devices creating:

# MAKEDEV -d /ami/dev -x console 
# MAKEDEV -d /ami/dev -x null 
# MAKEDEV -d /ami/dev -x zero

Let set up a fstab with the bare minimum needed to boot this machine in /ami/etc/fstab

LABEL=root      /               ext4    defaults,noatime,nodiratime     1 1
tmpfs           /dev/shm        tmpfs   defaults,nosuid,nodev,noexec    0 0
devpts          /dev/pts        devpts  gid=5,mode=620                  0 0
sysfs           /sys            sysfs   defaults                        0 0
proc            /proc           proc    defaults                        0 0

We need a yum config file to use when installing our packages. I put mine just in /root/yum.conf

[main]
reposdir=/dev/null
exactarch=1

[base]
name=CentOS-6 - Base
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=os
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

#released updates
[updates]
name=CentOS-6 - Updates
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=updates
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

Setting the reposdir to something non-existant keep yum from reading other yum conf files.

Lets get the packages installed.

# yum -c /root/yum.conf --installroot=/ami/ groupinstall base core

I personally don’t want everything that comes default in base and core, so I’ve made my own list of packages to get a minimal install that is usable for me.

# yum -c /root/yum.conf --installroot=/ami/ install acl at attr audit basesystem \
bash bind-utils coreutils cpio cronie crontabs dhclient e2fsprogs file filesystem \
glibc grub initscripts iproute iptables iptables-ipv6 iputils kbd kernel logrotate \
lsof lynx man man-pages man-pages-overrides mtr ncurses ntp ntpdate ntsysv \
openssh-server parted passwd policycoreutils postfix procps psacct quota readahead \
rootfiles rpm rsync rsyslog screen selinux-policy-targeted setserial setup \
shadow-utils strace sudo sysstat system-config-firewall-tui \
system-config-network-tui tcpdump time tmpwatch traceroute util-linux-ng \
vim-enhanced vim-minimal wget which yum

Now just clean up to save some disk space.

# yum -c /root/yum.conf --installroot=/ami/ clean all

Networking! We need a network right? /ami/etc/sysconfig/network

NETWORKING=yes

Set up eth0 in /ami/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no

Disable selinux by changing /ami/etc/selinux/config to set SELINUX=disabled

Now lets set up SSH to only allow keyed logins. First edit /ami/etc/ssh/sshd_config and change these values.

# Only allow root to run commands over ssh, no shell
PermitRootLogin forced-commands-only

# EC2 uses keys for remote access
PasswordAuthentication no

Now lets chroot to the our AMI so we can add a user.

# chroot /ami

Now we can add a user to log in as, instead of root.

# useradd ec2-user -c "EC2 Default User"

Add ec2-user ALL = NOPASSWD: ALL to the end of /etc/sudoers

Now just type exit to get out of the chroot.

Next we need to set up grub and the boot loader.

Open up /ami/boot/grub/grub.conf and put this in it.

default=0
timeout=0
title CentOS 6 (KERN)
        root (hd0)
        kernel /boot/vmlinuz-KERN ro root=LABEL=root
        initrd /boot/initramfs-KERN.img

Now replace KERN with the installed kernel version.

# kern=`ls /ami/boot/vmlinuz-*|awk -F/ '{print $NF}'`
# kernver=${kern#vmlinuz-}
# sed -i "s/KERN/$kernver/" /ami/boot/grub/grub.conf

To match what you normally get on a regular CentOS host, a couple of symlinks should also be created:

# ln -s grub.conf /ami/boot/grub/menu.lst
# ln -s ../boot/grub/grub.conf /ami/etc/grub.conf

Now to set up /ami/etc/rc.local. This script runs on boot and will help us set up our SSH keys and a few other basics.

#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# Check/wait for the network is available and find out essential information...

INSTID=`curl -s -m 5 --fail http://169.254.169.254/latest/meta-data/instance-id|tr '[A-Z]' '[a-z]'`
while [[ "$INSTID" == "" ]]; do
        sleep 5
        INSTID=`curl -s -m 5 --fail http://169.254.169.254/latest/meta-data/instance-id|tr '[A-Z]' '[a-z]'`
done

# Create the .ssh folder if needed and set correct permissions

if [ ! -d /home/ec2-user/.ssh ] ; then
        mkdir -p /home/ec2-user/.ssh
        chmod 700 /home/ec2-user/.ssh
fi

# Get the ssh public key from instance data

/usr/bin/curl --silent --fail -o /home/ec2-user/.ssh/authorized_keys http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key

# Set proper permissions on keys file
chmod 600 /home/ec2-user/.ssh/authorized_keys

# Change ownership to ec2-user
chown -R ec2-user:ec2-user /home/ec2-user

PracticalClouds.com has a very fancy rc.local if you want more cool features.

Some final housekeeping to get our image as clean and small as possible.

# find /ami/var/log -type f -exec rm "{}" \;
# rm -rf /ami/root/.bash_history
# rm -rf /ami/var/cache/yum
# rm -rf /ami/var/lib/yum

Now lets package up our awesome new image. First unmount it.

# umount /ami

In the AWS Consol detach the EBS volume.

Detach EBS Volume

Then create a snapshot of the volume.

Create Snapshot

Lastly we need to create an image from the snapshot.

Create Image

Now we need to set the Kernel ID for our image. Go to the AWS Documentation listing the PV-GRUB Kernels. Find the one for your region and arch (x86_64) and with hd0.

Create Image Dialog

And we are done! Now you can launch an instance based on your image!

Amazon EC2 Basics

June 28th, 2012

So I’ve started playing with EC2 lately and I wanted to document some of the basics when you first get started.

Set up your Security Group

The first thing you should do is go set up your Security Groups. This is basically firewall rules for your instances. It’s best to leave the default group alone, it simply allows any instances using it to communicate with each other. An instance can be in many security groups so you can make individual ones per instance, or per application. For example you can make a Security Group to allow SSH from your IP and put all your instances in it. Just remember that once an instance is started you cannot change which goups it is in.

My SSH Security Group

Set up your Key Pair

Next you should set up your key pair. This will be used to connect to your instances without a password. Simply go to Key Pairs and generate a key pair. This will download the private key to your computer. Keep this safe as its your key into your systems. You can change your pem file to a Putty Private Key file for use in Putty.

X.509 Certificate

You should also go and create a X.509 Certificate under Security Credentials. This is used for command line access to the AWS API’s.

Security Credentials

This will give you a cert-XXX.pem and pk-XXX.pem file. Be sure to keep the pk-XX.pem file in a safe place with your other keys.

Launch an Instance

Now to launch an instance. For your first one, lets use the Amazon Linux AMI, since its pretty bullet proof. Its got yum and seems kinda like a Fedora flavor.

The wizard is pretty straight forward and you can pretty much just use the defaults. When you get to Firewall, just select your two security groups.

Once its launched you can SSH into it with the Public DNS name its given by Amazon.

Public DNS

And your done! Now go start playing with some of the advanced features!

 

 

 

 

Plesk Backup Extraction

October 19th, 2010

Ever wonder how to parse a plesk backup file?

It ain’t easy. The Parallels Knoledge Base explains how to try and read it by greping and tailing and junk. Or to use an email client to open the file.

I tried to open a backup in Thunderbird once. It was not a fan of my 500MB backup file.

So I whined at my co-worker Andy about the dreadful pain of extracting single files out of large backups. Being that he can acutaly code in a real language (C#) he whipped me up a little utillity.

PleskDump – A Plesk Backup Extractor GUI for Windows!

Now when some client who canceled their site 6 months ago comes back and says “I need this one jpg I had on my site” I can get them their file with relatively little effort.

Andy is also working on a RETS client for debugging RETS Servers and getting their metadata. Hope its done soon!

Plesk Mail Cleanup

October 7th, 2009

I recently discovered that Plesk does not delete the mail in a mailbox when it is disabled. This can leave large amounts of mail lying around when you switch from a mailbox to a forward. So I wrote a little bash script to clean up the folders for me.

#!/bin/bash
# Clean the maildir of all disabled mail accounts of mail
# Get the system hostname since the filename of all the emails contain it
HOSTNAME=`hostname -f`
# fetch a list of all the mailboxes that are turned off
# skip column names so we only get the mailbox paths
for d in `echo "SELECT CONCAT_WS('/',domains.name,mail.mail_name) AS 'path' FROM domains,mail WHERE domains.id = mail.dom_id AND mail.postbox = 'false' GROUP BY mail.id ORDER BY domains.name ASC , mail.mail_name ASC" | mysql psa --skip-column-names`;
do
# find all the files with the hostname in it and delete them.
# this makes sure we get the IMAP folders as well such as .sent-mail
find /var/qmail/mailnames/$d/Maildir/ -type f -name "*$HOSTNAME*" -exec rm -vf {} \;
done

This will delete all the mail, but leave the spamassassin and courier settings in place.

MySQL Too Many Unauthenticated connections

October 7th, 2009

A few days ago at work we had one of our MySQL servers start hitting the connection limit. This server gets about 20-40 new connections a second. The maxed out connections wouldn’t last long and after a few attempts I got a process list when the server was maxed out.

What I saw was a whole bunch of connections from our load balanced IP.

| 108160 | unauthenticated user | 192.168.XXX.XXX:3449 | NULL | Connect | NULL | login | NULL
| 108162 | unauthenticated user | 192.168.XXX.XXX:39610 | NULL | Connect | NULL | login | NULL |
| 108163 | unauthenticated user | 192.168.XXX.XXX:39611 | NULL | Connect | NULL | login | NULL |
| 108164 | unauthenticated user | 192.168.XXX.XXX:39612 | NULL | Connect | NULL | login | NULL |
| 108165 | unauthenticated user | 192.168.XXX.XXX:39613 | NULL | Connect | NULL | login | NULL |
| 108166 | unauthenticated user | 192.168.XXX.XXX:39614 | NULL | Connect | NULL | login | NULL |
| 108167 | unauthenticated user | 192.168.XXX.XXX:39615 | NULL | Connect | NULL | login | NULL |

After much Googling I found this post on the MySQL Forums with the answer! Because MySQL can use hostnames as well as IP addresses in user privileges it must look up all IP addresses to determine their hostname. The DNS servers were running a little slow that day and were taking a few seconds to do (and fail) the reverse lookups for our loadbalancer IP address.

I added the load balancer IP to the /etc/hosts file and have not had this issue since!

This bug post also details the problem.

Mircosoft Maps Tips and Tricks

December 17th, 2007

So I recently wrote a map search using Microsoft’s Virtual Earth mapping software. This is similar to what Google has available.

I wrote my map against the v5 API, but i’m sure not too much has changed in v6. The SDK can be found here http://dev.live.com/virtualearth/sdk/

The first problem I ran into with Virtual Earth maps is that unlike the Google counterparts, the MS VE maps do not fix transparent PNGs in IE 6.

TwinHelix has a great hack for getting transparent PNG support in IE 6. Once you download this zip you will need to add some code to your map page.
<style type="text/css">
img { behavior: url(/path/to/iepngfix.htc) }
</style>

You will also need to edit the iepngfix.htc to tell it where your blank.gif file is (provided in the zip).
// This must be a path to a blank image. That's all the configuration you need.
if (typeof blankImg == 'undefined') var blankImg = '/images/blank.gif';

The next problem I ran into was also an IE 6 issue. You would think that the Microsoft map would work with the Microsoft browser. Unlike the Google maps the info windows for the pins can escape the borders of the map div. This can be a problem if you have “ActiveX controls” near your map, Apparently in IE 6 a dropdown form element is an Activex Control and will show through a pins info window. I fixed this using the JQuery plugin BGIFrame from Brandon Aaron. To use you will need JQuery and just include the .js file. Then in the function that starts your VE Map bgiframe the ero box.
function generateMap()
{
/* Load the map */
map = new VEMap('mapContainer');
map.LoadMap(new VELatLong([map_lat], [map_lon]), [map_zoom]);
$('.ero-shadow').bgiframe();
}

The final problem I found was trying to manually open and info window. I wanted the info box to open when the user moused over the pin. The easy part is attaching the event. Just add this to the map generating function.
/* Attach the events */
map.AttachEvent('onmouseover', openWindow);

The hard part was getting the function to work. Here is my final openWindow function.
function openWindow(e)
{
if (e.elementID != null)
{
window.ero.setBoundingArea(
new Msn.VE.Geometry.Point(0,0),
new Msn.VE.Geometry.Point(document.body.clientWidth,document.body.clientHeight+600));
map.ShowInfoBox(e);
}
}

Don’t ask me why that works, it just does. But there were some people on the MSDN Forums talking about being able to get a reference to the map pin.

The one other thing that has been bugging me about the Virtual Earth info windows is there shape and size. They don’t seem to be all that resizeable compared to the Google map bubbles. But that will have to wait for another project.

I hope you have found this information useful and helped you build a better mapping application.

Get a Mac Ad: Meant for Work

March 18th, 2007

I am getting kind of tired of the Mac ads. They are starting to seem more and more like a bad politician’s smear campaign against his rival.

This ad talks about how a PC is built solely for the business environment while a Mac is perfect for the home.

The PC first complains about how he has to “make movies and blogs and listen to music”. Blogs are web based and are therefore platform independent. Music (except for the DRM’ed kind) are also platform independent. In fact a Linux computer could do both of these things. As for the movies bit, thats a bit of a different argument. I have used Windows Movie Maker and found it fairly simple to use, but also fairly limited in power. iMovie may be a bit better in this reguard. Anyone doing a serious amount of movie making will not be using either of these programs. They would be using something like FinalCut with costs about $1500. Since that is out of the price range of a 10 year old, it doest fit here. Any movie a 10 year old is making will most likely be shots from a web cam with some simple titles and transition effects. Both the PC and Mac movie makers can do this.

The PC claims that the Mac was “made to stimulate 10 year olds with your iLife jazz”. What exactly is in iLife?
iMovie HD : web cams don’t shoot in HD. If your 10 year old needs to edit HD movies you must be spoiling them.
iDVD : again, what 10 year old is making DVDs?
iWeb : a set of tools for creating your own website and posting it to a .Mac account. 10 year olds don’t have .Mac accounts. And why is it limited to just .Mac websites? Why cant I post anything I make to my own server?
iPhoto : kids don’t organize there photos on the desktop. They use the web and services like Flickr. Again, web services are platform independed/
GarageBand : OK, I’ll give you this one. GarageBand is cool and has no equivalent on the PC. And I can totally see a 10 year old using this.

What do kids fine most “engaging”? Video Games!!!! I can count on one hand the number of games for the Mac. In fact I think the iPod has more games. Here is a great game to engage young minds.

Math Blaster

And as Amazon clearly states, only on Windows. So how can a Mac be more engaging to a 10 year old with out any video games, even educational ones?

Now don’t get me wrong, I’m not a Windows fanboy or a Mac hater. I think all three platforms have there place and pros and cons. But I do hate decietflul lying ads that misrepresent facts in order to sell more stuff. I plan to do more posts on the problems with the Mac ads, but I will leave out some of the ads that are fairly accurate and not smear ads against their competitor.