IBM AIX - HACMP Made Easy - A complete Overview!!

HACMP : High Availability Cluster Multi-Processing

High Availability : Elimination of both planned and unplanned system and application downtime. This is achieved through elimination of H/W and S/W single points of failure.

Cluster Topology : The Nodes, networks, storage, clients, persistent node ip label/devices

Cluster resources: HACMP can move these components from one node to others Ex: Service labels, File systems and applications
RSCT Version: 2.4.2
SDD Version: 1.3.1.3

HA Configuration :

• Define the cluster and nodes
• Define the networks and disks
• Define the topology
• Verify and synchronize
• Define the resources and resource groups
• Verify and synchronize

After Installation changes : /etc/inittab,/etc/rc.net,/etc/services,/etc/snmpd.conf,/etc/snmpd.peers,/etc/syslog.conf,
/etc/trcfmt,/var/spool/cron/crontabs/root,/etc/host.

Software Components:
Application server
HACMP Layer
RSCT Layer
AIX Layer
LVM Layer
TCP/IP Layer

HACMP Services :
Cluster communication daemon(clcomdES)
Cluster Manager (clstrmgrES)
Cluster information daemon(clinfoES)
Cluster lock manager (cllockd)
Cluster SMUX peer daemon (clsmuxpd)

HACMP Deamons: clstrmgr, clinfo, clmuxpd, cllockd.
HA supports up to 32 nodes
HA supports up to 48 networks
HA supports up to 64 resource groups per cluster
HA supports up to 128 cluster resources

IP Label : The label that is associated with a particular IP address as defined by the DNS (/etc/hosts)

Base IP label : The default IP address. That is set on the interface by aix on startup.

Service IP label: a service is provided and it may be bound on a single/multiple nodes. These addresses that HACMP keep highly available.

IP alias: An IP alias is an IP address that is added to an interface. Rather than replacing its base IP address.
RSCT Monitors the state of the network interfaces and devices.

IPAT via replacement : The service IP label will replace the boot IP address on the interface.

IPAT via aliasing: The service IP label will be added as an alias on the interface.

Persistent IP address: this can be assigned to a network for a particular node.
In HACMP the NFS export : /use/es/sbin/cluster/etc/exports

Shared LVM:

• Shared volume group is a volume group that resides entirely on the external disks shared by cluster nodes
• Shared LVM can be made available on Non concurrent access mode, Concurrent Access mode, Enhanced concurrent access mode.

NON concurrent access mode: This environment typically uses journaled file systems to manage data.

Create a non concurrent shared volume group: smitty mkvgàGive VG name, No for automatically available after system restart, Yes for Activate VG after it is created, give VG major number.

Create a non concurrent shared file system: smitty crjfsàRename FS names, No to mount automatically system restart, test newly created FS by mounting and unmounting it.

Importing a volume group to a fallover node:

• · Varyoff the volume group
• · Run discover process
• · Import a volume group

Concurrent Acccess Mode: It’s not supported for file systems. Instead must use raw LV’s and Physical disks.
Creating concurrent access volume group:

• · Verify the disk status using lsdev –Cc disk
• · Smitty cl_convgàCreate a concurrent volume groupàenter
• · Import the volume group using importvg –C –y vg_name physical_volume_name
• · Varyonvg vgname

Create LV’s on the concurrent VG: smitty cl_conlv.

Enhanced concurrent mode VG’s: This can be used for both concurrent and non concurrent access. This VG is varied on all nodes in the cluster, The access for modifying the data is only granted to the node that has the resource group active.
Active or passive mode:
Active varyon: all high level operations permitted.
Passive varyon: Read only permissions on the VG.
Create an enhanced concurrent mode VG: mkvg –n –s 32 –C –y myvg hdisk11 hdisk12

Resource group behaviour:
Cascading: Fallover using dynamic node priority. Online on first available node
Rotating : Failover to next priority node in the list. Never fallback. Online using distribution policy.
Concurrent : Online on all available nodes . never fallback
RG dependencies:Clrgdependency –t
/etc/hosts : /etc/hosts for name resolution. All cluster node IP interfaces must be added on this file.
/etc/inittab : hacmp:2:once:/usr/es/sbin/cluster/etc/rc.init>/dev/console 2> &1 will strat the clcomdES and clstrmgrES.
/etc/rc.net file is called by cfgmgr. To configure and start TCP/IP during the boot process.
C-SPOC uses clcomdES to execute commands on remote nodes.
C-SPC commands located in /usr/es/sbin/cluster/cspoc
you should not stop a node with the forced option on more than one node at a time and also the RG in concurrent mode.
Cluster commands are in /usr/es/sbin/cluster
User Administration : cl_usergroup
Create a concurrent VG — > smitty cl_convg
To find the resource group information: clrginfo –P

HACMP Planning:
Maximum no.of nodes in a cluster is 32
In an HACMP Cluster, the heartbeat messages are exchanged via IP networks and Point-to-Point networks
IP Label represents the name associated with a specific IP address
Service IP label/address: The service IP address is an IP address used for client access.
2 types of service IP addresses:
Shared Service IP address: It can be active only on one node at a time.
Node bound service IP address: An IP address that can be configured only one node
Method of providing high availability service IP addresses:
IP address takeover via IP aliases
IPAT via IP replacement
IP alias is an IP address that is configured on a communication interface in addition to the base ip address. IP alias is an AIX function that is supported by HACMP. AIX supports multiple IP aliases on each communication interface. Each IP alias can be a different subnet.
Network Interface:
Service Interface: This interface used for providing access to the application running on that node. The service IP address is monitored by HACMP via RSCT heartbeat.
Boot Interface: This is a communication interface. With IPAT via aliasing, during failover the service IP label is aliased onto the boot interface
Persistent node IP label: Its useful for administrative purpose.
When an application is started or moved to another node together with its associated resource group, the service IP address can be configured in two ways.

• Replacing the base IP address of a communication interface. The service IP label and boot IP label must be on same subnet.
• Configuring one communication interface with an additional IP address on top of the existing one. This method is IP aliasing. All Ip addresses/labels must be on different subnet.

Default method is IP aliasing.
HACMP Security: Implemented directly by clcomdES, Uses HACMP ODM classes and the /usr/es/sbin/cluster/rhosts file to determine partners.
Resource Group Takeover relationship:
Resource Group: It’s a logical entity containing the resources to be made highly available by HACMP.
Resources: Filesystems, NFS, Raw logical volumes, Raw physical disks, Service IP addresses/Labels, Application servers, startup/stop scripts.
To made highly available by the HACMP each resource should be included in a Resource group.
Resource group takeover relationship:

1. Cascading
2. Rotating
3. Concurrent
4. Custom

Cascading:

• Cascading resource group is activated on its home node by default.
• Resource group can be activated on low priority node if the highest priority node is not available at cluster startup.
• If node failure resource group falls over to the available node with the next priority.
• Upon node reintegration into the cluster, a cascading resource group falls back to its home node by default.
• Attributes:

1. Inactive takeover(IT): Initial acquisition of a resource group in case the home node is not available.
2. Fallover priority can be configured in default node priority list.
3. cascading without fallback is an attribute that modifies the fall back behavior. If cwof flag is set to true, the resource group will not fall back to any node joining. When the flag is false the resource group falls back to the higher priority node.
Rotating:

• At cluster startup first available node in the node priority list will activate the resource group.
• If the resource group is on the takeover node. It will never fallback to a higher priority node if one becomes available.
• Rotating resource groups require the use of IP address takeover. The nodes in the resource chain must all share the same network connection to the resource group.

Concurrent:

• A concurrent RG can be active on multiple nodes at the same time.

Custom:

• Users have to explicitly specify the desired startup, fallover and fallback procedures.
• This support only IPAT – via aliasing service IP addresses.

Startup Options:

• Online on home node only
• Online on first available node
• Online on all available nodes
• Online using distribution policyàThe resource group will only be brought online if the node has no other resource group online. You can find this by lssrc –ls clstrmgrES

Fallover Options:

• Fallover to next priority node in list
• Fallover using dynamic node priorityàThe fallover node can be selected on the basis of either its available CPU, its available memory or the lowest disk usage. HACMP uses RSCT to gather all this information then the resource group will fallover to the node that best meets.
• Bring offlineàThe resource group will be brought offline in the event of an error occur. This option is designed for resource groups that are online on all available nodes.

Fallback Options:

• Fallback to higher priority node in the list
• Never fallback

Basic Steps to implement an HACMP cluster:

• Planning
• Install and connect the hardware
• Configure shared storage
• Installing and configuring application software
• Install HACMP software and reboot each node
• Define the cluster topology
• Synchronize the cluster topology
• Configure cluster resources
• Configure cluster resource group and shared storage
• Synchronize the cluster
• Test the cluster

HACMP installation and configuration:
HACMP release notes : /usr/es/lpp/cluster/doc
Smitty install_all à fast path for installation
Cluster.es and cluster.cspoc images must be installed on all servers
Start the cluster communication daemon à startsrc –s clcomdES
Upgrading the cluster options: node by node migration and snapshot conversion
Steps for migration:

• Stop cluster services on all nodes
• Upgrade the HACMP software on each node
• Start cluster services on one node at a time

Convert from supported version of HAS to hacmp

• Current s/w should be commited
• Save snapshot
• Remove the old version
• Install HA 5.1 and verify

Check previous version of cluster: lslpp –h “cluster”
To save your HACMP configuration, create a snapshot in HACMP
Remove old version of HACMP: smitty install_remove ( select software name cluster*)
Lppchk –v and lppchk –c cluster* both commands run clean if the installation is ok.
After you have installed HA on cluster nodes you need to convert and apply the snapshot. converting the snapshot must be performed before rebooting the cluster nodes
Clconvert_snapshot –C –v version –s à It converts HA old version snapshot to new version
After installation rebooting the cluster services is required because to activate the new cluster manager.
Verification and synchronization : smitty hacmpàextended configurationà extended verification and configuration à verify changes only
Perform Node-by-Node Migration:

• Save the current configuration in snapshot.
• Stop cluster services on one node using graceful with takeover
• Verify the cluster services
• Install hacmp latest version.
• Check the installed software using lppchk
• Reboot the node.
• Restart the HACMP software ( smitty hacmpàSystem ManagementàManage cluster servicesàstart cluster services
• Repeat above steps on all nodes
• Logs documenting on /tmp/hacmp.out /tmp/cm.log /tmp/clstrmgr.debug
• Config_too_long message appears when the cluster manager detects that an event has been processing for more than the specified time. To change the time interval ( smitty hacmpà extended configurationàextended event configurationàchange/show time until warning)

Cluster snapshots are saved in the /usr/es/sbin/cluster/snapshots.
Synchronization process will fail when migration is incomplete. To back out from the change you must restore the active ODM. (smitty hacmp à Problem determination toolsà Restore HACMP configuration database from active configuration.)
Upgrading HACMP new version involves converting the ODM from previous release to the current release. That is done by /usr/es/sbin/cluster/conversion/cl_convert –F –v 5.1
The log file for the conversion is /tmp/clconvert.log.
Clean up process once installation interrupted.( smitty installà software maintenance and installationà clean up after a interrupted installation)
Network Configuration:
Physical Networks: TCP/IP based, such as Ethernet and token ring Device based, RS 232 target mode SSA(tmssa)
Configuring cluster Topology:
Standard and Extended configuration
Smitty hacmpàInitialization and standard configuration
IP aliasing is used as the default mechanism for service IP label/address assignment to a network interface.

• Configure nodes : Smitty hacmpàInitialization and standard configurationàconfigure nodes to an hacmp clusterà (Give cluster name and node names)
• Configure resources: Use configure resources to make highly available ( configure IP address/label, Application server, Volume groups, Logical volumes, File systems
• Configure resource groups: Use configure HACMP resource groups . you can choose cascading, rotating, custom, concurrent
• Assign resources to each resource group: configure HACMP resource groupsà Change/show resources for a Resource group.
• Verify and synchronize the cluster configuration
• Display the cluster configuration

Steps for cluster configuration using extended path:

• Run discovery: Running discovery retrieves current AIX configuration information from all cluster nodes.
• Configuring an HA cluster: smitty hacmpàextended configurationàextended topology configurationàconfigure an HACMP clusteràAdd/change/show an HA cluster
• Defining a node: smitty hacmpàextended configurationàextended topology configurationàconfigure HACMP nodesàAdd a node to the HACMP cluster
• Defining sites: This is optional.
• Defining network: Run discover before network configuration.
1. IP based networks: smitty hacmpàextended configurationàextended topology configurationàconfigure HACMP networksàAdd a network to the HACMP clusteràselect the type of networkà(enter network name, type, netmask, enable IP takeover via IP aliases(default is true), IP address offset for heartbeating over IP aliases.

• Defining communication interfaces: smitty hacmpàextended configurationàextended topology configurationàHACMP cotmmunication interfaces/DevicesàSelect communication interfacesàadd node name, network name, network interface, IPlabel/address, network type
• Defining communication devices: smitty hacmpàextended configurationàextended topology configurationàconfigure HACMP communication interface/devicesàselect communication devices
• To see boot IP labels on a node use netstat –in
• Defining persistent IP labels: It always stays on the same node, does not require installing an additional physical interface, its not part of any resource group.smitty hacmpàextended topology configurationàconfigure persistent node IP label/addressesàadd persistent node IP label(enter node name, network name, node IP label/address)

Resource Group Configuration

• Smitty hacmpàinitialization and standard configurationàConfigure HACMP resource groupsà Add a standard resource groupà Select cascading/Rotating/Concurrent/Custom (enter resource group name, participating node names)
• Assigning resources to the RG. Smitty hacmpàinitialization and standard configurationà Configure HACMP resource groupsàchange/show resources for a standard resource group( add service IP label/address, VG, FS, Application servers.

Resource group and application management:

• Bring a resource group offline: smitty cl_adminàselect hacmp resource group and application managementàBring a resource group offline.
• Bring a resource group online: smitty hacmp àselect hacmp resource group and application managementàBring a resource group online.
• Move a resource group: smitty hacmp à select hacmp resource group and application managementà Move a resource group to another node

C-SPOC: Under smitty cl_admin

• Manage HACMP services
• HACMP Communication interface management
• HACMP resource group and application manipulation
• HACMP log viewing and management
• HACMP file collection management
• HACMP security and users management
• HACMP LVM
• HACMP concurrent LVM
• HACMP physical volume management

Post Implementation and administration:
C-Spoc commands are located in the /usr/es/sbin/cluster/cspoc directory.
HACMP for AIX ODM object classes are stored in /etc/es/objrepos.
User group administration in hacmp is smitty cl_usergroup
Problem Determination:
To verify the cluster configuration use smitty clverify.dialog
Log file to store output: /var/hacmp/clverify/clverify.log
HACMP Log Files:
/usr/es/adm/cluster.log: Generated by HACMP scripts and daemons.
/tmp/hacmp.out: This log file contains line – by – line record of every command executed by scripts.
/usr/es/sbin/cluster/history/cluster.mmddyyyy: System creates cluster history file everyday.
/tmp/clstrmgr.debug: This messages generated by clstrmgrES activity.
/tmp/cspoc.log: generated by hacmp c-spoc commands
/tmp/dms_loads.out: stores log messages every time hacmp triggers the deadman switch
/var/hacmp/clverify/clverify.log: cluster verification log.
/var/ha/log/grpsvcs, /var/ha/log/topsvcs, /var/ha/log/grpglsm: daemon logs.
Snapshots: The primary information saved in a cluster snapshot is the data stored in the HACMP ODM classes(HACMPcluster, HACMPnode, HACMPnetwork, HACMPdaemons).
The cluster snapshot utility stores the data it saves in two separate files:
ODM data file(.odm), Cluster state information file(.info)
To create a cluster snapshot: smitty hacmpàhacmp extended configurationàhacmp snapshot configurationàadd a cluster snapshot

Cluster Verification and testing:
High and Low water mark values are 33 and 24
The default value for syncd is 60.
Before starting the clu ster clcomd daemon is added to the /etc/inittab and started by init.
Verify the status of the cluster services: lssrc –g cluster ( cluster manager daemon(clstrmgrES), cluster SMUX peer daemon(clsmuxpd) and cluster topology services daemon(topsvcd) should be running.
Status of different cluster subsystems: lssrc –g topsvcs and lssrc –g emsvcs.
In /tmp/hacmp.out file look for the node_up and node_up_complete events.
To check the HACMP cluster status: /usr/sbin/cluster/clstat. To use this command you should have started the clinfo daemon.
To change the snmp version : /usr/sbin/snmpv3_ssw -1.
Stop the cluster services by using smitty clstop : graceful, takeover, forced. In the log file /tmp/hacmp.out search for node_down and node_down_complete.
Graceful: Node will be released, but will not be acquired by other nodes.
Graceful with takeover: Node will be released and acquired by other nodes.
Forced: Cluster services will be stopped but resource group will not be released.
Resource group states: online, offline, aquiring, releasing, error, temporary error, or unknown.
Find the resource group status: /usr/es/sbin/cluster/utilities/clfindres or clRGinfo.
Options: -t : If you want to display the settling time –p: display priority override locations
To review cluster topology: /usr/es/sbin/cluster/utilities/cltopinfo.
Different type of NFS mounts: hard and soft
Hard mount is default choice.
NFS export file: /usr/es/sbin/cluster/etc/exports.
If the adapter configured with a service IP address : verify in /tmp/hacmp.out event swap_adapter has occurred, Service IP address has been moved using the command netstat –in .
You can implement RS232 heartbeat network between any 2 nodes.
To test a serial connection lsdev –Cc tty, baud rate is set to 38400, parity to none, bits per character to 8
Test to see RSCT is functioning or not : lssrc –ls topsvcs
RSCT verification: lssrc –ls topsvcs. To check RSCT group services: lssrc –ls grpsvcs
Monitor heartbeat over all the defines networks: cllsif.log from /var/ha/run/topsvcs.clustername.
Prerequisites:
PowerHA Version 5.5 à AIX v5300-9 àRSCT levet 2.4.10
BOS components: bos.rte.*, bos.adt.*, bos.net.tcp.*,
Bos.clvm.enh ( when using the enhanced concurrent resource manager access)
Cluster.es.nfs fileset comes with the powerHA installation medium installs the NFSv4. From aix BOS bos.net.nfs.server 5.3.7.0 and bos.net.nfs.client 5.3.7.0 is required.
Check all the nodes must have same version of RSCT using lslpp –l rsct
Installing powerHA: release notes: /usr/es/sbin/cluster/release_notes
Enter smitty install_allàselect input deviceàPress f4 for a software listingàenter
Steps for increase the size of a shared lun:

• Stop the cluster on all nodes
• Run cfgmgr
• Varyonvg vgname
• Lsattr –El hdisk#
• Chvg –g vgname
• Lsvg vgname
• Varyoffvg vgname
• On subsequent cluster nodes that share the vg. (run cfgmgr, lsattr –El hdisk#, importvg –L vgname hdisk#)
• Synchronize

PowerHA creates a backup copy of the modified files during synchronization on all nodes. These backups are stored in /var/hacmp/filebackup directory.
The file collection logs are stored in /var/hacmp/log/clutils.log file.

User and group Administration:
Adding a user: smitty cl_usergroupàselect users in a HACMP clusteràAdd a user to the cluster.(list users, change/show characteristics of a user in cluster, Removing a user from the cluster
Adding a group: smitty cl_usergroupàselect groups in a HACMP clusteràAdd a group to the cluster.(list groups, change/show characteristics of a group in cluster, Removing a group from the cluster

Command is used to change password on all cluster nodes: /usr/es/sbin/cluster/utilities/clpasswd

Smitty cl_usergroupàusers in a HACMP cluster

• Add a user to the cluster
• List users in the cluster
• Change/show characteristics of a user in the cluster
• Remove a user from the cluster

Smitty cl_usergroupàGroups in a HACMP cluster

• Add a group to the cluster
• List groups to the cluster
• Change a group in the cluster
• Remove a group

Smitty cl_usergroupàPasswords in an HACMP cluster
Importing VG automatically: smitty hacmpàExtended configurationàHACMP extended resource configurationàChange/show resources and attributes for a resource group Automatically import volume groups to true.

C-SPOC LVM: smitty cl_admin à HACMP Logical Volume Management

• Shared Volume groups
• Shared Logical volumes
• Shared File systems
• Synchronize shared LVM mirrors (Synchronize by VG/Synchronize by LV)
• Synchronize a shared VG definition

C-SPOC concurrent LVM: smitty cl_admin à HACMP concurrent LVM

• Concurrent volume groups
• Concurrent Logical volumes
• Synchronize concurrent LVM mirrors

C-SPOC Physical volume management: smitty cl_adminàHACMP physical volume management

• Add a disk to the cluster
• Remove a disk from the cluster
• Cluster disk replacement
• Cluster datapath device management

Cluster Verification: smitty hacmpàExtended verificationàExtended verification and synchronization. Verification log files stored in /var/hacmp/clverify.
/var/hacmp/clverify/clverify.log à Verification log
/var/hacmp/clverify/pass/nodename à If verification succeeds
/var/hacmp/clverify/fail/nodename à If verification fails
Automatic cluster verification: Each time you start cluster services and every 24 hours.
Configure automatic cluster verification: smitty hacmpàproblem determination toolsàhacmp verification à Automatic cluster configuration monitoring.
Cluster status Monitoring: /usr/es/sbin/cluster/clstat –a and o.
/usr/es/sbin/cluster/utilities/cldumpàIt provides snapshot of the key cluster status components.
Clshowsrv: It displays the status

Disk Heartbeat:

• It’s a non-IP heartbeat
• It’s use dedicated disk/LUN
• It’s a point to point network
• If more than 2 nodes exist in your cluster, you will need a minimum of n number of non-IP heartbeat networks.
• Disk heartbeating will typically requires 4 seeks/second. That is each of two nodes will write to the disk and read from the disk once/second. Filemon tool monitors the seeks.

Configuring disk heartbeat:

• Vpaths are configured as member disks of an enhanced concurrent volume group. Smitty lvmàselect volume groupsàAdd a volume groupàGive VG name, PV names, VG major number, Set create VG concurrent capable to enhanced concurrent.
• Import the new VG on all nodes using smitty importvg or importvg –V 53 –y c23vg vpath5
• Create the diskhb networkàsmitty hacmpàextended configuration àextended topology configurationàconfigure hacmp networksàAdd a network to the HACMP clusteràchoose diskhb
• Add 2 communication devicesà smitty hacmpàextended configuration àextended topology configurationàConfigure HACMP communication Interfaces/DevicesàAdd communication interfaces/devicesàAdd pre-defined communication interfaces and devicesà communication devicesàchoose the diskhb
• Create one communication device for other node also

Testing Disk Heartbeat connectivity:/usr/sbin/rsct/dhb_read is used to test the validity of a diskhb connection.
Dhb_read –p vpath0 –r for receives data over diskhb network
Dhb_read –p vpath3 –t for transmits data over diskhb network.
Monitoring disk heartbeat: Monitor the activity of the disk heartbeats via lssrc –ls topsvcs. Monitor the Missed HBS field.
Configure HACMP Application Monitoring: smitty cm_cfg_appmonàAdd a process application monitoràgive process names, app startup/stop scripts
Application availability analysis tool: smitty hacmpàsystem managementàResource group and application managementàapplication availability analysis
Commands:
List the cluster topology : /usr/es/sbin/cluster/utilities/cllsif
/usr/es/sbin/cluster/clstat
Start cluster : smitty clstart .. Monitor with /tmp/hacmp.out and check for node_up_complete.
Stop the cluster : smitty cl_stop àMonitor with /tmp/hacmp.out and check fr node_down_complete.
Determine the state of cluster: /usr/es/sbin/cluster/utilities/clcheck_server
Display the status of HACMP subsystems: clshowsrv –v/-a
Display the topology information: cltopinfo –c/-n/-w/-i
Monitor the heartbeat activity: lssrc –ls topsvcs [ check for dropped, errors]
Display resource group attributes: clrginfo –v, -p, -t, -c, -a OR clfindres

IBM AIX - VIO Made Easy - A Complete Overview!!

PowerVM: It allows to increase the utilization of servers. Power VM includes Logical partitioning, Micro Partitioning, Systems Virtualization, VIO, hypervisor and so on.

Simultaneous Multi Threading : SMT is an IBM microprocessor technology that allows 2 separate H/W instruction streams to run concurrently on the same physical processor.

Virtual Ethernet : VLAN allows secure connection between logical partitions without the need for a physical IO adapter or cabling. The ability to securely share Ethernet bandwidth across multiple partitions increases H/W utilization.

Virtual SCSI: VSCSI provides secure communication between the partitions and VIO server.The combination of VSCSI and VIO capabilities allows you to share storage adapter bandwidth and to subdivide single large disks into smaller segments. The adapters and disks can shared across multiple partitions, increase utilization.

VIO server : Physical resources allows you to share the group of partitions.The VIO server can use both virtualized storage and network adapters, making use of VSCSI and virtual Ethernet.

Redundant VIO server: AIX or linux partitions can be a client of one or more VIO servers at the same time. A good strategy to improve availability for sets of client partitions is to connect them to 2 VIO servers. The reason for redundancy is ability to upgrade latest technologies without affecting production workloads.

Micro Partitioning: Sharing the processing capacity from one or more logical partitions. The benefit of Micro Partitioning is that it allows significantly increased overall utilization . n of processor resources. A micro partition must have 0.1 processing units. Maximum no.of partitions on any system P server is 254.

Uncapped Mode : The processing capacity can exceed the entitled capacity when resources are available in the shared processor pool and the micro partition is eligible to run.

Capped Mode : The processing capacity can never exceed the entitled capacity.

Virtual Processors :A virtual processor is a representation of a physical processor that is presented to the operating system running in a micro partition.

If a micro partition is having 1.60 processing units , and 2 virtual processors. Each virtual processor will have 0.80 processing units.

Dedicated processors : Dedicated processors are whole processors that are assigned to dedicated LPARs . The minimum processor allocation for an LPAR is one.

IVM(Integrated virtualization manager): IVM is a h/w management solution that performs a subset of the HMC features for a single server, avoiding the need of a dedicated HMC server.

Live partition Mobility: Allows you to move running AIX or Linux partitions from one physical Power6 server to another without disturb.

VIO

Version for VIO 1.5

For VIO command line interface is IOSCLI

The environment for VIO is oem_setup_env

The command for configuration through smit is cfgassist

Initial login to the VIO server is padmin

Help for vio commands ex: help errlog

Hardware requirements for creating VIO :

1. Power 5 or 6
2. HMC
3. At least one storage adapter
4. If you want to share Physical disk then one big Physical disk
5. Ethernet adapter
6. At least 512 MB memory

Latest version for vio is 2.1 fixpack 23

Copying the virtual IO server DVD media to a NIM server:

Mount /cdrom

Cd /cdrom

Cp /cdrom/bosinst.data /nim/resources

Execute the smitty installios command

Using smitty installios you can install the VIO S/w.

Topas –cecdisp flag shows the detailed disk statistics

Viostat –extdisk flag shows detailed disk statistics.

Wklmgr and wkldagent for handling workload manager. They can be used to record performance data and that can be viewed by wkldout.

Chtcpip command for changing tcpip parameters

Viosecure command for handling the secure settings

Mksp : to create a storage pool

Chsp: Adds or removes physical volumes from the storage pool

Lssp: lists information about storage pool

Mkbdsp: Attaches storage from storage pool to virtual SCSI adapter

Rmbdsp: removes storage from virtual scsi adapter and return it to storage pool

Default storage pool is rootvg

Creation of VIO server using HMC version 7 :

Select the managed system -> Configuration -> Create Logical Partition -> VIO server

Enter the partition name and ID.

Check the mover service box if the VIO server partition to be created will be supporting partition mobility.

Give a partition profile name ex:default

Processors : You can assign entire processors to your partition for dedicated use, or you can assign partial processors units from the shared processor pool. Select shared.

Specify the minimum, desired and maximum processing units.

Specify minimum, desired and maximum virtual processors. And select the uncapped weight is 191

The system will try to allocate the desired values

The partition will not start if the managed system cannot provide the minimum amount of processing units.

You cannot dynamically increase the amount of processing units to more than the maximum,

Assign the memory also min, desired and max.

The ratio between minimum and maximum amount of memory cannot be more than 1/64

IO selects the physical IO adapters for the partition. Required means the partition will not be able to start unless these are available in this partition. Desired means that the partition can start also without these adapters. A required adapter can not be moved in a dynamic LPAR operation.

VIO server partition requires a fiber channel adapter to attach SAN disks for the client partitions. It also requires an Ethernet adapter for shared Ethernet adapter bridging to external networks.

VIO requires minimum of 30GB of disk space.

Create Virtual Ethernet and SCSI adapters: increase the maximum no of virtual adapters to 100

The maximum no of adapters must not set more than 1024.

In actions -> select create -> Ethernet adapter give Adapter ID and VLAN id.

Select Access External Network Check Box to use this adapter as a gateway between internal and external network.

And also create SCSI adapter also.

VIO server S/W installation :

1. Place the CD/DVD in P5 Box
2. Activate the VIO server by clicking the activate. Select the default partition
3. Then check the Open terminal window or console section and click the advanced. And OK.
4. Under the boot mode drop down list select SMS.

After installation is complete login with padmin and press a(for s/w maintenance agreement terms)

License –accept for accepting the license.

Creating a shared Ethernet adapter

1. lsdev –virtual ( check the virtual Ethernet adapter)
2. lsdev –type adapter ( Check the physical Ethernet adapter)
3. you use the lsmap –all –net command to check the slot numbers of the virtual Ethernet adapter.
4. mkvdev –sea ent0 –vadapter ent2 –default ent2 –defaultid 1
5. lsmap –all –net
6. use the cfgassist or mktcpip command configure the tcp/ip or
7. mktcpip –hostname vio_server1 –inetaddr 9.3.5.196 –interface ent3 –netmask 255.255.244.0 –gateway 9.3.4.1

Defining virtual disks

Virtual disks can either be whole physical disks, logical volumes or files. The physical disks can be local or SAN disks.

Create the virtual disks

1. login to the padmin and run cfgdev command to rebuild the list of visible devices.
2. lsdev –virtual (make sure virtual scsi server adapters available ex:vhost0)
3. lsmap –all --> to check the slot numbers and vhost adapter numbers.
4. mkvg –f –vg rootvg_clients hdisk2 --> Creating rootvg_clients vg.
5. mklv –lv dbsrv_rvg rootvg_clients 10G

Creating virtual device mappings:

1. lsdev –vpd |grep vhost
2. mkvdev –vdev dbsrv_rvg -vadapter vhost2 –dev dbsrv_rvg
3. lsdev –virtual
4. lsmap –all

fget_config –Av command provided on the IBM DS4000 series for a listing of LUN names.

Virtual SCSI Optical devices:

A dvd or cd device can be virtualized and assigned to client partitions. Only one VIO client can access the device at a time.

Steps :

1. let the DVD drive assign to VIO server
2. Create a server SCSI adapter using the HMC.
3. Run the cfgdev command to get the new vhost adapter. Check using lsdev –virtual
4. Create the virtual device for the DVD drive.(mkvdev –vdev cd0 –vadapter vhost3 –dev vcd)
5. Create a client scsi adapter in each lpar using the HMC.
6. Run the cfgmgr

Moving the drive :

1. Find the vscsi adapter using lscfg |grep Cn(n is the slot number)
2. rmdev –Rl vscsin
3. run the cfgmgr in target LPAR

Through dsh command find which lpar is currently holding the drive.

Unconfiguring the dvd drive :

1. rmdev –dev vcd –ucfg
2. lsdev –slots
3. rmdev –dev pci5 –recursive –ucfg
4. cfgdev
5. lsdev –virtual

Mirroring the VIO rootvg:

1. chvg –factor 6 rootvg (rootvg can include upto 5 PVs with 6096 PPs)
2. extendvg –f rootvg hdisk2
3. lspv
4. mirrorios –f hdisk2
5. lsvg –lv rootvg
6. bootlist –mode –normal –ls

Creating Partitions :

1. Create new partition using HMC with AIX/linux
2. give partition ID and Partition name
3. Give proper memory settings(min/max/desired)
4. Skip the physical IO
5. give proper processing units (min/desired/max)
6. Create virtual ethernet adapter ( give adapter ID and VLAN id)
7. Create virtual SCSI adapter
8. In optional settings

• · Enable connection monitoring
• · Automatically start with managed system
• · Enable redundant error path reporting

9. bootmodes select normal

Advanced Virtualization:

Providing continuous availability of VIO servers : use multiple VIO servers for providing highly available virtual scsi and shared Ethernet services.

IVM supports a single VIO server.

Virtual scsi redundancy can be achieved by using MPIO and LVM mirroring at client partition and VIO server level.

Continuous availability for VIO

• Shared Ethernet adapter failover
• Network interface backup in the client
• MPIO in the client with SAN
• LVM Mirroring

Virtual Scsi Redundancy:

Virtual scsi redundancy can be achieved using MPIO and LVM mirroring.

Client is using MPIO to access a SAN disk, and LVM mirroring to access 2 scsi disks.

MPIO: MPIO for highly available virtual scsi configuration. The disks on the storage are assigned to both virtual IO servers. The MPIO for virtual scsi devices only supports failover mode.

Configuring MPIO:

• Create 2 virtual IO server partitions
• Install both VIO servers
• Change fc_err_recov( to fast_fail and dyntrk(AIX tolerate cabling changes) to yes. ( chdev –dev fscsi0 –attr fc_err_recov=fast_fail dyntrk=yes –perm
• Reboot the VIO servers
• Create the client partitions. Add virtual Ethernet adapters
• Use the fget_config(fget_config –vA) command to get the LUN to hdisk mappings.
• Use the lsdev –dev hdisk –vpd command to retrieve the information.
• The reserve_policy for each disk must be set to no_reserve.(chdev –dev hdisk2 –attr reserve_policy=no_reserve)
• Map the hdisks to vhost adapters.( mkvdev –vdev hdisk2 –vadapter vhost0 –dev app_server)
• Install the client partitions.
• Configure the client partitions
• Testing MPIO

Configure the client partitions:

• Check the MPIO configuration (lspv, lsdev –Cc disk)
• Run lspath
• Enable the health check mode (chdev –l hdisk0 –a hcheck_interval=50 –P
• Enable the vscsi client adapter path timeout ( chdev –l vscsi0 –a vscsi_path_to=30 –P)
• Changing the priority of a path( chpath –l hdisk0 –p vscsi0 –a priority=2)

Testing MPIO:

• Lspath
• Shutdown VIO2
• Lspath
• Start the vio2
• Lspath

LVM Mirroring: This is for setting up highly available virtual scsi configuration. The client partitions are configured with 2 virtual scsi adapters. Each of these virtual scsi adapters is connected to a different VIO server and provides one disk to the client partition.

Configuring LVM Mirroring:

• Create 2 virtual IO partitions, select one Ethernet adapter and one storage adapter
• Install both VIO servers
• Configure the virtual scsi adapters on both servers
• Create client partitions. Each client partition needs to be configured with 2 virtual scsi adapters.
• Add one or two virtual Ethernet adapters
• Create the volume group and logical volumes on VIO1 and VIO2
• A logical volume from the rootvg_clients VG should be mapped to each of the 4 vhost devices.( mkvdev –vdev nimsrv_rvg –vadapter vhost0 –dev vnimsrv_rvg)
• Lsmap –all
• When you bring up the client partitions you should have hdisk0 and hdisk1. Mirror the rootvg.
• Lspv
• Lsdev –Cc disk
• Extendvg rootvg hdisk1
• Mirrorvg –m rootvg hdisk1
• Test LVM mirroring

Testing LVM mirroring:

• Lsvg –l rootvg
• Shutdown VIO2
• Lspv hdisk1 (check the pvstate, stale partitions)
• Reactivate VIO and varyonvg rootvg
• Lspv hdisk1
• Lsvg –l rootvg

Shared Ethernet adapter: It can be used to connect a physical network to a virtual Ethernet network. Several client partitions to share one physical adapter.

Shared Ethernet Redundancy: This is for temporary failure of communication with external networks. Approaches to achieve continuous availability:

• Shared Ethernet adapter failover
• Network interface backup

Shared Ethernet adapter failover: It offers Ethernet redundancy. In a SEA failover configuration 2 VIO servers have the bridging functionality of the SEA. They use a control channel to determine which of them is supplying the Ethernet service to the client. The client partition gets one virtual Ethernet adapter bridged by 2 VIO servers.

Requirements for configuring SEA failover:

• One SEA on one VIOs acts as the primary adapter and the second SEA on the second VIOs acts as a backup adapter.
• Each SEA must have at least one virtual Ethernet adapter with the “access external network flag(trunk flag) checked. This enables the SEA to provide bridging functionality between the 2 VIO servers.
• This adapter on both the SEA’s has the same pvid
• Priority value defines which of the 2 SEA’s will be the primary and which is the secondary. An adapter with priority 1 will have the highest priority.

Procedure for configuring SEA failover:

• Configure a virtual Ethernet adapter via DLPAR. (ent2)
• Select the VIO-->Click task button-->choose DLPAR-->virtual adapters
• Click actions-->Create-->Ethernet adapter
• Enter Slot number for the virtual Ethernet adapter into adapter ID
• Enter the Port virtual Lan ID(PVID). The PVID allows the virtual Ethernet adapter to communicate with other virtual Ethernet adapters that have the same PVID.
• Select IEEE 802.1
• Check the box “access external network”
• Give the virtual adapter a low trunk priority
• Click OK.

• Create another virtual adapter to be used as a control channel on VIOS1.( give another VLAN ID, do not check the box “access external network” (ent3)
• Create SEA on VIO1 with failover attribute. ( mkvdev –sea ent0 –vadapter ent2 –default ent2 –defaultid 1 –attr ha_mode=auto ctl_chan=ent3. Ex: ent4
• Create VLAN Ethernet adapter on the SEA to communicate to the external VLAN tagged network ( mkvdev –vlan ent4 –tagid 222) Ex:ent5
• Assign an IP address to SEA VLAN adapter on VIOS1. using mktcpip
• Same steps to VIO2 also. ( give the higher trunk priority:2)

Client LPAR Procedure:

• Create client LPAR same as above.

Network interface backup : NIB can be used to provide redundant access to external networks when 2 VIO servers used.

Configuring NIB:

• Create 2 VIO server partitions
• Install both VIO servers
• Configure each VIO server with one virtual Ethernet adapter. Each VIO server needs to be a different VLAN.
• Define SEA with the correct VLAN ID
• Add virtual Scsi adapters
• Create client partitions
• Define the ether channel using smitty etherchannel

Configuring multiple shared processor pools:

Configuration --> Shared processor pool management --> Select the pool name

VIOs Security:

Enable basic firewall settings: viosecure –firewall on

view all open ports on firewall configuration: viosecure –firewall view

To view current security settings: viosecure –view nonint

Change system security settings to default: viosecure –level default

List all failed logins : lsfailedlogin

Dump the global command log: lsgcl

Backup:

Create a mksysb file of the system on a nfs mount: backupios –file /mnt/vios.mksysb –mksysb

Create a backup of all structures of VGs and/or storage pools: savevgstruct vdiskvg ( data will be stored to /home/ios/vgbackups)

List all backups made with savevgstruct: restorevgstruct –ls

Backup the system to a NFS mounted file system: backupios –file /mnt

Performance Monitoring:

Retrieve statistics for ent0: entstat –all ent0

Reset the statistics for ent0: entstat –reset ent0

View disk statistics: viostat 2

Show summary for the system in stats: viostat –sys 2

Show disk stats by adapter: viostat –adapter 2

Turn on disk performance counters: chdev –dev sys0 –attr iostat=true

Topas –cecdisp

Link aggregation on the VIO server:

Link aggregation means you can give one IP address to two network cards and connect to two different switches for redundancy purpose. One network card will be active on one time.

Devices --> communication --> Etherchannel/IEEE 802.3 ad Link Aggregation --> Add an etherchannel / Link aggregation

Select ent0 and mode 8023ad

Select backup adapter as redundancy ex: ent1

Automatically virtual adapter will be created named ent2.

Then put IP address : smitty tcpip --> Minimum configuration and startup --> select ent2 --> Put IP address