IT


IT and Geekery18 Dec 2006 05:52 pm

An Art Form

Speed does matter. Failure to perform a task with adequate speed can result in missing deadlines, no time to relax, and unimpressed friends. In this guide, I will share with you some of the practices I’ve adopted, how they’re useful, and how you can apply these same practices to increase your speed, efficiency, and effectiveness at daily tasks. All you will need is a computer with an internet connection.

The goals of perfecting this art are to never lose a piece of information you’ve been given, to retrieve all information quickly, and to implement a secure and consistent way of accessing this information, wherever you are, 24/7.

Version Control

The center of the information ninja’s arsenal is Subversion. With SVN and good habits, you can organize your information and continue to adapt and optimize its structure, tracking all of your changes as you go. SVN can be accessed securely over SSH, which uses high grade encryption (typically 128 bit AES in cipher block chaining mode).

A New Era

Let’s face it: we’re in the digital age. Paper is quickly becoming irrelevant. I highly suggest you invest in a flatbed scanner and a paper shredder, and perform the following when you receive a piece of paper in the mail:

  • 1. Is this information worth keeping in its original form? If not, skip step 2.
  • 2. Scan the paper in your scanner, organize the image file into your SVN repository (however you prefer) and commit.
  • 3. Throw the paper and envelope into the shredder.

Information

Information is a beautiful thing. How do you organize your information? What do you consider information worth organizing? To help you answer that question, here are some of the things I consider worth organizing.

  • Passwords
  • Bookmarks/favorite links
  • The numbers on plastic cards in my wallet (credit, ATM, debit)
  • Bank account numbers
  • Model and serial numbers for hardware
  • Serial license numbers for software
  • Important/upcoming dates
  • TODO list

UNIX Philosophy

Follow the UNIX philosophy. Store information in tab-delimited plaintext files when possible. In this format, you can take advantage of the score of UNIX utilities for text manipulation and searching. You need no special programs to view or modify your information, only a good editor (I prefer Vim), and cat.

Be disciplined. Store every single password you’ve ever needed for any website and every computer you log into. If you continue to update and organize your repository, you will excel in information retrieval. You will never lose another password or waste any time trying several passwords you might’ve used.

Structure of the Repository

This is largely up to you, but the way I have my repository organized is as follows:

Subdirectory Description
dot Shell init and config files for multi-system consistency
tools Small and highly useful C programs with a Makefile to build and deploy them to ~/bin
db Directory containing very sensitive information in plaintext, tab-delimited files
school Homework and projects for school
scans Contains scans of important mail

Dot

Your dot directory, or equivalent, is one of the most important aspects of this process. If you are going to be consistent in your habits, you will need a consistent interface to any computer from which you view or manipulate information. If you are going to be fast, you will want to learn all of the important shortcuts of the shell you use (I use bash), and make aliases to perform tasks, ranging from simple to highly complex, in as few keystrokes as possible.

I’ve provided a snapshot of my dot directory, and hope that you will find it useful, and build upon it. Untar it to your home directory, and run the install.sh script. Be careful not to clobber your existing ssh config file, if you have one. Read through the files to see what it does and modify it. rc.bash consists of some startup commands, aliases, and your prompt, and cmd.bash consists of some auxiliary functions that I find highly useful.

You should import the dot directory or have something like it in your SVN repository. The idea is to check it out using SVN on every machine you own.

Security Considerations

If you’re going to put your sensitive information in plaintext files like I do, it is a good idea to use disk encryption (I use FileVault on OS X) with a lengthy and secure password, limit your computer to one user (yourself), enable your screensaver to come on quickly, and password protect your computer on wake from sleep or the screensaver. If you follow these practices, then it is both sensible and convenient to store your information in plaintext files.

A True Ninja of the Information Age

Code of the True Ninja.

  • A True Ninja collects every piece of useful information he can get his hands on
  • A True Ninja organizes this information in a secure and consistent fashion
  • A True Ninja can access and update his wealth of information from anywhere in the world, 24/7
  • A True Ninja encrypts his information with secure passwords and large keys and stores up to date copies of it on computers all over the planet
  • A True Ninja utilizes the same command prompt and aliases on every UNIX based computer he has access to. He can update this from any machine, at his own discretion, since this too is information that he can organize
  • A True Ninja only leaves open port 22, and retrieves his information with SSH
  • A True Ninja makes contingency plans and continues to adapt, to ensure the integrity of his data
IT22 Jul 2006 08:34 pm

SNMP

In the first part, I covered building and installing Cacti from scratch on an OS X machine. Now I will guide you through setting up SNMP on your other OS X machines. I put together a package of some scripts I made to extract CPU usage, disk activity, and temperature (for OS X Server), which will be executed and read when Cacti polls the SNMP server.

Ruby

Ruby is awesome (It comes with OS X, by the way). The scripts I use to parse data from commands are written in Ruby. If you don’t know it, and if you’re looking for an all-purpose scripting and text processing language, I would highly suggest learning it. Other than the fact that it is a powerful and innovative language, it has clean syntax, that in my opinion, is easier to follow than Perl.

Scripts for OS X

We will now set up scripts to parse output from iostat and vm_stat. These commands will be executed to gather statistics on disk activity and CPU usage. If you’re running OS X Server, it will also parse output from servermgrd to get cpu temperatures.

I’ve bundled the necessary scripts and launchd jobs together into a single archive. Download here.

sudo tar -C / -xzvf snmp_scripts.tar.gz

The necessary files will be extracted to /usr/share/snmp, /opt/net-snmp, and /Library/LaunchDaemons. A configuration file is extracted to /usr/share/snmp/snmpd.conf (more about this later). The two launchd jobs will be responsible for keeping snmp running and for averaging statistics over a five minute interval every five minutes. The contents in /opt/net-snmp will contain scripts to collect data from commands and write the output to /opt/net-snmp/data.

Adding a User

Before we can continue, we need to add a user to query SNMP. You’ll need to remember this user and password when configuring Cacti. In this example, we’ll add a user with the name ’snmpstats’ and the password ’snmp10pass’.

sudo net-snmp-config --create-snmpv3-user -ro -a 'snmp10pass' snmpstats

SNMP Configuration

If you look at the file we extracted earlier, /usr/share/snmp/snmpd.conf, you’ll see several lines beginning with ‘exec’. This tells SNMP that when we’re querying a certain part of the MIB tree, to execute a command and use its output as a response to the query.

For executing commands in SNMP, we use the .1.3.6.1.4.1.2021 section of the MIB tree. The command will execute and stdout line #N will be accessible by accessing .101.N For example, if .1.3.6.1.4.1.2021.8.101 executes a command, the sixth line of output can be retrieved from .1.3.6.1.4.1.2021.8.101.101.6. If this seems confusing, read the snmpd.conf man page or look at the stock configuration file in /etc/snmpd.conf.

Firing it Up

We are now ready to start the launchd jobs, which will start the daemon and the data collection.

sudo launchctl load /Library/LaunchDaemons/org.machx.snmp-data.plist /Library/LaunchDaemons/org.machx.snmp.plist

Wait ten minutes and you should start seeing output in /opt/net-snmp/data. To test, run

snmpget -v3 -u snmpstats -l authNoPriv -a MD5 -A 'snmp10pass' localhost .1.3.6.1.4.1.2021.8.1.101.101.1

This should yield the CPU user usage.

Back to the Cacti Side

Log into Cacti and import templates from this XML file: cacti_host_template_mac_os_x.xml.

Add the SNMP host you just configured under the new ‘Mac OS X’ template and pick the graphs you want.

IT29 Jun 2006 12:03 am

y=x

Naturally, when you want awesome graphs, flexibility, awesome graphs, SNMP polling, and awesome graphs, you turn to Cacti. Cacti is a web application written in PHP that can graph statistics collected from hosts of your choosing. Hosts are polled with SNMP, the de-facto standard in network device monitoring and control.

Cacti CPU Temperature Graph Today, much to my dismay, I discovered that compiling Cacti’s dependencies correctly and getting it to just ‘work’ can be a pain in the ass on Mac OS X. I also discovered that there is no consolidated guide to setting up Cacti + RRDTool + SNMP on any platform, which would’ve saved me a few hours of work. Not to mention drastically reducing the number of Google searches I have to make and the tinkering I have to do to figure out how something works. Which is why I’ve decided to make one, and update it with useful information.

Disclaimer

There is lots of minimal configuration going on here. You may want more configuration for the applications built, or to use a preexisting installation, or to have better security. This is just a basic guide for someone who wants Cacti up and running quickly, from scratch. I highly suggest that you use a password for MySQL and password protect the Apache server running Cacti, or use SSL, or both, but I won’t go over that here.

Let’s Put Everything in /opt

Just for fun, we’re going to shove everything in /opt and call it a day. /opt is really just an easier-to-type /usr/local without a preexisting subdirectory structure, and we’re going to use it in a way that isolates each package, making multiple packages easier to maintain.

MySQL

./configure --prefix=/opt/mysql && make && sudo make install && sudo make init-db

Apache2

./configure --prefix=/opt/apache2 --enable-so && make && sudo make install

PHP

./configure --prefix=/opt/php --with-apxs2=/opt/apache2/bin/apxs --with-mysql=/opt/mysql && make && sudo make install

Configuration

Edit /opt/apache2/conf/httpd.conf and add the following line:

AddType application/x-httpd-php .php

Find the DirectoryIndex parameter and add index.php to the list.

DirectoryIndex index.php index.html ...

Next Up: Dependencies

We should have a working MySQL + Apache2 + PHP setup now. The next things we need are SNMP, which OS X graciously has installed by default, so we don’t have to worry about that, and RRDTool, which Cacti uses for graphing and log files.

RRDTool has several dependencies, one of which is very easy to miscompile (more about that in a moment). The dependencies are: libpng, libart, freetype2, and it doesn’t hurt to first build pkgconfig, which RRDTool can use to locate these dependencies.

pkgconfig

Just put pkgconfig in the default path, /usr/local/bin, which should be part of $PATH.
./configure && make && sudo make install

libpng

./configure --prefix=/opt/libpng && make && sudo make install

libart

./configure --prefix=/opt/libart && make && sudo make install

freetype2

Here’s the tricky one. I had to scour the internet to figure this one out: you’re going to need to set the following environment variables BEFORE you compile freetype2.

export LDFLAGS="-framework Carbon"; export CPPFLAGS=$LDFLAGS

If you don’t do this, it will compile fine, and RRDTool will bail when producing graphs, producing the following error message:

dyld: Symbol not found: _FSPathMakeRef

After setting the environment variables, we can build freetype:

./configure --prefix=/opt/freetype2 && make && sudo make install

Rolling it all together: RRDTool

First, let’s put those dependencies in our pkgconfig path.

export PKG_CONFIG_PATH=/opt/libpng/lib/pkgconfig:/opt/libart/lib/pkgconfig:/opt/freetype2/lib/pkgconfig

Also, check that pkgconfig is in your path and that it runs OK.

We’re also going to need to disable python support, since the python that ships with OS X breaks this. Now we can build:

./configure --prefix=/opt/rrdtool --disable-python && make && sudo make install

Configuring Cacti

We’re now ready for the fun part. Untar Cacti to /opt/apache2/cacti, and CD to it.

We’re going to need to turn on the MySQL server:

sudo /opt/mysql/bin/mysqld_safe >dev/null 2>&1 &

This should background the server. To test it:

/opt/mysql/bin/mysql -u root

This should yield a MySQL console logged in as root. Issue the following command in the MySQL shell:

create database cacti;

Quit out of MySQL and load the cacti.sql file in your cacti installation:

/opt/mysql/bin/mysql -u root cacti < cacti.sql

Your cacti database should now be populated. Now you can start apache2 and begin the web configuration:

sudo /opt/apache2/bin/apachectl start

Coming in Part 2

Graph/host/data templates, backend ruby scripts to parse CPU load, temperature, and disk usage from commandline utilities, and using SNMP to run custom commands.