Friday, February 27, 2009

Adding packages to zones

In Opensolaris 2008.11 you can create zones. But in contrast to zones in Solaris these Opensolaris zones do not contain packages from the global zone but a small set of dedicated packages, that are installed directly in the zone.
So there are no more sparse zones, that inherit the packages from the global zone and you have to install additional packages via pkg command. (No more pkgadd!)
If your local zone has no internet access you can install packages from the global zone as well. Although this is not documented. But the zoneadm -install does exactly this. Look at /usr/lib/brand/ipkg/pkgcreatezone
  1. Make sure the local zone is halted.
  2. From the global zone mount the local zone's root file system.
  3. Set the envrironment variable PKG_IMAGE=/wherever/the/rootfs/is/mounted
  4. Set the environment variable PKG_CACHE=/var/pkg/download
  5. Use the regular pkg command. It will install in the zoneroot: pkg install SUNWmypackage
  6. Unset at least PKG_IMAGE. Otherwise subsequent pkg install in the global zone will fail.
  7. Boot the local zone.

Sunday, February 22, 2009

5h of my life spoilt due to hacking

In my last post I suggested to clone zones and instead of running sys-unconfig I hacked the script, so that only /etc/nodename gets modified.
The script worked very well... It took 10 seconds to clone a zone. But after I installed MySQL and the monitoring agent, I failed monitoring the instances in the Enterprise Monitor. After "only" 5 hours of testing and re-installing I discovered that the Enterprise Monitor identifies the host via its ssh hostid, that was generated during install. Unfortunately all my clones had the same ssh hostid, so Enterprise Manager was a little confused.
What did I learn from these five hours?
  1. I will never again hack scripts that are written by people much smarter than me (Solaris engineering in this case).
  2. I can install zones regularly with zoneadm -z zonename install and everything is fine.
  3. I can clone a zone (with the standard script!) zoneadm -z zonename clone template but then MySQL Agent will fail and needs a TRUNCATE mysql.inventory on the local database. (Look at the agent's log file.) After that, the cloned zone will be monitored correctly.

Saturday, February 21, 2009

Cloning zones

THIS BLOG POST GIVES WRONG ADVICE. DON'T DO THIS!


As the zones use ZFS as root filesystem it's easy to do a zfs snapshot/clone to reproduce a zone. The different zones have only one difference: They differ in /etc/nodename, they have different rootpaths and they have different IP addresses.
I wrote a script to read from global's /etc/hosts the ip address. Zonerootpath is /zones/zonename and /etc/nodename must contain as well. The name of the zone is always identical to the hostname of the zone. That rule is not from OpenSolaris but from me.  ;-)

Here is my script: Beware: I hate to do a lot of checking. This is not for general use...

#!/bin/bash
echo "Creating zone $1 from zone TEMPLATE"
echo check if $1 does not already exist...
zoneadm list -cv | grep $1 && exit -1

echo -n "Retrieving TEMPLATE's IP: "
TEMPLATE_IP=$(grep TEMPLATE /etc/hosts | nawk '{print $1}')
echo $TEMPLATE_IP
echo -n "Retrieving $1's IP: "
NEW_IP=$(grep $1 /etc/hosts | nawk '{print $1}')
echo $NEW_IP

echo "Creating Command File"
rm /tmp/MBcommand 2> /dev/null
# We use the config of TEMPLATE zone and modify this
zonecfg -z TEMPLATE export | sed s/TEMPLATE/$1/ | sed s/$TEMPLATE_IP/$NEW_IP/ > /tmp/MBcommand

echo "generating new zone"
zonecfg -z $1 -f /tmp/MBcommand
echo "Installing new zone as snapshot"
zoneadm -z $1 clone TEMPLATE
echo done.

Very easy so far.
zoneadm clone will do a sys-unconfig to remove any network configuration from the clone. Acutally I don't like that. I don't want to run the whole sysidcfg configuration with every clone. So I did a hack:
AND HERE BEGINS MY PROBLEM. sys-unconfig IS NECESSARY FOR THE HOSTID. 
/usr/lib/brand/ipkg/clone is the shell script that will run, when you do a 'zoneadm clone'.
And here is the (malicious) call to /usr/sbin/sys-unconfig. I removed this and changed it to only write the new /etc/nodename file in the zone's root filesystem. Here are the last few lines of /usr/lib/brand/ipkg/clone:

/usr/sbin/mount -F zfs $zpds/$zpname/ROOT/$BENAME $zonepath/root || \
fail_incomplete "$f_zfs_mount"

#/usr/sbin/sys-unconfig -R $zonepath/root || fail_incomplete "$f_sysunconfig"
echo $zonename > $zonepath/root/etc/nodename
echo Run /usr/sbin/sys-unconfig manually. Command removed by Mario in $0
echo Instead edited only the /etc/nodename file with $zonename

/usr/sbin/umount $zonepath/root || fail_fatal "$f_umount"

exit $ZONE_SUBPROC_OK

Now I only have to populate /etc/hosts with my machines and run one command to have a completely installed and configured zone for my demo environment:

root@ap-mysql:~# ./create_zone_from_TEMPLATE berlin
Creating zone berlin from zone TEMPLATE
check if berlin does not already exist...
Retrieving TEMPLATE's IP:
Retrieving berlin's IP: 192.168.14.226
Creating Command File
generating new zone
Installing new zone as snapshot
Run /usr/sbin/sys-unconfig manually. Command removed by Mario in /usr/lib/brand/ipkg/clone
Instead only the /etc/nodename edited with berlin
done.


Creating a datacenter in one Solaris machine

There is a big event coming: Cebit!
And with that I took the task to implement a demo for MySQL Enterprise Monitor to show at the Sun booth. So I need a machine for the enterprise monitor and some machines to run MySQL databases. After all I need something to monitor.
So the setup will be an Ultra 24 desktop machine installed with OpenSolaris 2008.11 . For every machine I will install a zone to run a separate database and one zone to run the enterprise monitor.

So here is my first round of experiences: Installing a template container on OpenSolaris and cloning it to build four or five separate database machines. Let's go:



####################################################################################
# Creating ZFS filesystems for all zone root-fs and a zfs that's shared between global and all local zones for easy data exchange
####################################################################################

root@ap-mysql:/# zfs create -o mountpoint=/zones rpool/zones
root@ap-mysql:/# zfs create -o mountpoint=/zones/share rpool/zones/share


####################################################################################
# Create a template zone, that will be cloned for all demo zones
####################################################################################

root@ap-mysql:/# zonecfg -z TEMPLATE
TEMPLATE: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:TEMPLATE> create
zonecfg:TEMPLATE> set zonepath=/zones/TEMPLATE
zonecfg:TEMPLATE> add inherit-pkg-dir
zonecfg:TEMPLATE:inherit-pkg-dir> set dir=/zones/share
zonecfg:TEMPLATE:inherit-pkg-dir> end
zonecfg:TEMPLATE> add net
zonecfg:TEMPLATE:net> set physical=e1000g0
zonecfg:TEMPLATE:net> set address=192.168.14.225/24
zonecfg:TEMPLATE:net> end
zonecfg:TEMPLATE> verify
zonecfg:TEMPLATE> commit
zonecfg:TEMPLATE> exit
root@ap-mysql:/# zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / native shared
- TEMPLATE configured /zones/TEMPLATE ipkg shared

####################################################################################
# Now we have the TEMPLATE zone. Next step is to install the zone and boot it.
# CAVEAT: The machine needs internet access to install a zone. The software respository must be www.opensolaris.org in the moment... :-(
####################################################################################

root@ap-mysql:/# zoneadm -z TEMPLATE install
A ZFS file system has been created for this zone.
Authority: Using http://pkg.opensolaris.org/release/.
Image: Preparing at /zones/TEMPLATE/root ... done.
Installing: (output follows)
DOWNLOAD PKGS FILES XFER (MB)
Completed 52/52 7862/7862 72.41/72.41

PHASE ACTIONS
Install Phase 12939/12939
PHASE ITEMS
Reading Existing Index 9/9
Indexing Packages 52/52

Note: Man pages can be obtained by installing SUNWman
Postinstall: Copying SMF seed repository ... done.
Postinstall: Working around http://defect.opensolaris.org/bz/show_bug.cgi?id=741
Done: Installation completed in 518.993 seconds.

Next Steps: Boot the zone, then log into the zone console
(zlogin -C) to complete the configuration process


####################################################################################
# time to boot the zone
# prepare a second terminal window and start the zone's console with
# zlogin -C TEMPLATE
####################################################################################

root@ap-mysql:~# zoneadm -z TEMPLATE boot

####################################################################################
# In TEMPLATE's console, do the interactive sysidcfg procedure
# I like to install some other packages. The default set is too small for me
# Adding packages is easy: 'pkg install SUNWxyz'. Unfortunately you have to
# be in the internet to have access to www.opensolaris.org
# hopefully this will change in the future.
# Here is the list of packages I installed for the showcase:
# SUNWgnu-coreutils
# SUNWgcmn
# SUNWgtar    needed for installing MySQL tarball
# SUNWuiu8
# SUNWman    can't live without it
# SUNWless
# SUNWpkgcmds   to use pkgadd later with MySQL
####################################################################################

That's it. We have a zone called TEMPLATE. Next blog will be cloning to define 4 identical machines.

Saturday, February 14, 2009

Here we go...

After lots of motivation from colleagues now is the time: I will start my own blog.
Why? Is anybody interested in what I learn and see in my life? Maybe. 
So many times in the last years I learned things through blogs, recipes and private websites where people documented rather petty findings, or procedures that were useful for me, that I felt guilty for not sharing my own experiences.
Without other people's blogs I couldn't sync my phone, use most of the software or build funny electronic things like my barograph.

What will I blog about? Mostly it will be my work experience as a Systems Engineer with Sun. Technical stuff about MySQL and the tool ecosystem around it, Solaris, maybe a little hardware as well. I left the field of hardware some months ago after nearly ten years. But sometimes there are things with hardware that are so crazy, that I MUST blog about it... e.g. Brandon discovering that harddisks are emotionally sensitive. Do not shout at your disks!