AIX | Linux | Virtualization | Made with Paper

*nix tips + random doodles

[rm-minus-rf]

A better way to sort through journalctl logs

if you’ve ever been kicked to emergency mode or if your system reboots into emergency mode, they usually will tell you to run “journelctl -xb” to view the logs, but this will show you all the logs and is a lot to sort through.  A better way would be to get get straight to the meat:

journalctl -xb -p3 

this will show you all the logs that are of type “err” or lower (emerg, alert, crit) so you can get to your root cause quicker

[rm-minus-rf]

Change console resolution in RHEL7

if you want to change the default console resolution on your RHEL7 vm to something smaller...i.e. 800x600x16, then do:

# vi /etc/sysconfig/grub find the GRUB_CMDLINE_LINUX line and append: nomodeset vga=788 you then need to generate a new grub.cfg file:

# grub2-mkconfig -o /boot/grub2/grub.cfg reboot and done! 

[rm-minus-rf]

Hard reset a RHEL vm

if you need to reboot a RHEL vm but don’t have vSphere access to do a hard reset, you can try this:

# echo 1 > /proc/sys/kernel/sysrq # echo b > /proc/sysrq-trigger

(if you:  # echo o > /proc/sysrq-trigger    it will do a shutdown instead)

[rm-minus-rf]

Reset a lost root password in RHEL7

you will need to interrupt your boot loader and edit the default boot loader entry:

 - append “console=tty1 rd.break” at the end of the “linux16″ line  - <ctrl+x> to boot with modified config

you should be getting the switch_root prompt and your /sysroot filesystem should be mounted ro, which you’ll have to remount rw:

 - mount -o remount,rw /sysroot  - chroot /sysroot

after you chroot it, you can change the password.  if you’re running selinux, you’ll have to force it to relabel the filesystem with the proper context or the new /etc/shadow file won’t be able to be read:

 - touch /.autorelabel

exit (twice) to continue with the boot process and you should be able to login with you newly recovered root password 

[rm-minus-rf]

Import existing SSL cert into a PKCS12 keystore then converting to JKS

openssl pkcs12 -export -in [ssl_cert.crt] -inkey [ssl_cert.key] -out keystore.p12 -name [alias]

if you then wish to export it to a JKS (for Java/Tomcat), then do:

keytool -importkeystore -srckeystore keystore.p12 -srcstoretype pkcs12 -srcalias [src_alias] -destkeystore tomcat.jks -deststoretype jks -deststorepass [keystore_password] -destalias [dest_alias]

[rm-minus-rf]

Setup an NFS server in RHEL6

on the NFS server, you will need to:

1) install the nfs-utils and rpcbind packages

2) edit the /etc/sysconfig/nfs file and comment out (and edit if you prefer to) the entries for RQUOTAD_PORT, LOCKD_TCPPORT, LOCKD_UDPPORT, STATD_PORT and MOUNTD_PORT and allow both tcp and udp version of those ports open along with 2049 (NFS) and 111 (rpcbind/sunrpc)

3) edit /etc/exports to set your permissions on what you're sharing and who can mount said share:

[directory_to_be_shared]        [host(options)]

common options are: rw,sync

4) start your NFS server

# /sbin/service nfs start

5) make sure you turn on nfs and rpcbind at startup:

# chkconfig nfs on # chkconfig rpcbind on

[rm-minus-rf]

View Glen Yu’s idea on Mix

Dooley not very pleased about getting woken up -_-“

[rm-minus-rf]

View Glen Yu’s idea on Mix

Bogart!

[rm-minus-rf]

mount: unknown filesystem type 'LVM2_member'

disk errors or file systems errors are what we sysadmins dread the most.  my most recent encounter of one taught me a thing or two:

# lvmdiskscan

# lvdisplay

if you tried to mount the disk using its VG name it might give you the following error:

"mount: special device /dev/[vg_name]/[lv_name] does not exist"

and if you did an:

# lvscan

it might show you that the disk is "inactive", so all you need to do now is make it active!

# modprobe dm-mod

# vgchange -ay

doing another lvscan now should label your disk now as "ACTIVE", so then go ahead now and mount it!

[rm-minus-rf]

Big Endian or Little Endian?

if you're ever wondering whether your linux is big or little endian, do this:

$ echo -n I | od -to2 | head -n1 | cut -f2 -d" " | cut -c6

it will return "0" if it's big, and "1" if it's little (typically, linux on x86 should be little) 

[rm-minus-rf]

interesting little bug that supposedly causes really large /var/log/lastlog files.  this could be a problem if you use/have some sort of AD integrations with large AD UIDs as your Linux UIDs

[rm-minus-rf]

came across this while i was looking for some sendmail command line examples (i find that quicker than reading man-page :p)

[rm-minus-rf]

Restarting an LPAR from HMC command line

to do so, you will need to be connected as hscroot, and then simply run the following command:

# chsysstate -r lpar -m [frame_name] -o shutdown --immed --restart -n [lpar_name]

[rm-minus-rf]

Storage performance tuning on AIX - part 2

ok, so you've done the tweaks mentioned in part 1 but job performance is still poor, what next?

you will have to increase the queue_depth of your disks.  queue_depth determines how many requests can be sent to your disks at any one time.  the larger the queue, and increasing it can increase disk I/O throughput

you will first need to figure out what your current queue_depth is set at:

# lsattr -El [pv_name]

you'll then have to figure out what you want to change it to.  i would say try doubling or tripling it at first, then monitor your job again and see what your %tm_act is at.  you do want it so that there's some activity on the %tm_act but not in such a way that it's consistently high

changing the queue_depth, however is a disruptive change and you will want to change it to all the pv/disks that are in the vg you're trying to tune:

# varyoffvg [vg_name]

# chdev -l [pv_name] -a queue_depth=[new_value]

# varyonvg [vg_name]

[rm-minus-rf]

Storage performance tuning on AIX - part 1

as a follow-up to my earlier post regarding network performance tuning on the AIX platform, this guide will further outline some other typical performance bottlenecks a server might face.

first of all, you will need to determine if a storage bottleneck exists.  to find that out, run iostat when running your job.  you will want to see if %tm_act is constantly high (90+%) and if it's all on a particular pv/disk.  if you have a mutil-disk vg setup, you may want try the following non-disruptive changes first and run your job again to see if you have performance gain:

# chlv -e'x' [lv_name]

# reorgvg [vg_name]

[rm-minus-rf]

Network performance tuning on AIX

if you're suspecting your network to be the bottleneck for some of your jobs, you can try to run your job and then on the source LPAR, do a:

$ netstat -an | grep [port]

you can see if your send queue (typically 3rd column) is filling up or how much it fills up.  compare this value with the global tcp_sendspace parameter or from your virtual network interface (if you're not using network options to define your parameters).  if you notice that it's filling the buffer everytime, maybe it's time to increase said buffer.

if you're using "no", then change the global network option settings with:

# no -F -p -o tcp_sendspace=[new_buffer_size]

test your job again and rinse/repeat until you've found something optimal.

NOTE #1: you can do something similar to the tcp_recvspace on the client side as well.

NOTE #2: your tcp_sendspace and tcp_recvspace values should not add up to more than "thewall":

no -a | grep thewall

"thewall" is memory reserved for network and is typically about 50% if your system memory (i know, it would be really hard to exceed this, but still..i have to mention it)

-----------------------------

here's how to change the TCP send and receive buffers on the virtual network interface (i.e. en0) ***WARNING*** this procedure is disruptive:

# chdev -l en0 -a tcp_sendspace=[new_buffer_size] -P

you will then have to unload and reload the devices and start it back up again via smit:

# rmdev -l en0 # rmdev -l ent0 # mkdev -l ent0 # mkdev -l en0

# smit tcpip --> Minimum Configuration & Startup --> en0 --> START TCP/IP daemons now --> yes

[rm-minus-rf]

Exporting NFS on AIX

on the NFS server, create or edit exisiting /etc/exports file and add entries based on following format:

[directory_to_be_shared] [-rw | -ro],access=[nfs_client_fqdn],root=[nfs_client_fqdn]

save the file and then made sure rpc.mountd, portmap and nfsd are all running.  if not, start them (and also, don't forget to export the fs):

# startsrc -s rpc.mountd # startsrc -s portmap # startsrc -g nfs # exportfs -a

on the client, mount the filesystem with:

# mount [nfs_server_hostname]:[directory_to_be_shared] [local_mount_point]