[walkthroughs] archive category

Install secure Webmin 1.580 on Ubuntu 12.04 LTS Precise Pangolin

Webmin welcome screen welcomes

Installing Webmin on Ubuntu 12.04 LTS Precise Pangolin is quite simple. This article will walk you through the complete installation of Webmin 1.580 including the upgrading of the self-signed certificate to a 2048-bit key (a 512-bit key is the default).

This is my system:

$ uname -a
Linux brasenose 3.2.0-24-generic-pae #37-Ubuntu SMP Wed Apr 25 10:47:59 UTC 2012 i686 i686 i386 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 12.04 LTS
Release:	12.04
Codename:	precise
$ openssl version
OpenSSL 1.0.1 14 Mar 2012

That last check is pretty important. If you don’t have OpenSSL installed you are not going to be able to run Webmin over TLS so make sure it is installed.

My demonstration system is a minimal system with only a SSH Server installed and a static IP set-up.

Install Webmin

Things have come a long way in the Webmin world and some cranky old Perl dependencies have now been flushed from the code. Unfortunately, there is no specialized Ubuntu version, so aficionados need to install the Debian version and make manual changes. Fortunately, installing the Debian package is simple. First we need to add the official Webmin repository to our list of software packages:

$ sudo vi /etc/apt/sources.list

Add the following line to the bottom of the file:

64
deb http://download.webmin.com/download/repository sarge contrib

This adds the Webmin Debian repository to your package list. Wondering why the repo release code name is ‘Sarge’? My guess is that it simply never got changed once Debian moved on to Etch in 2007 because it works fine. Sarge was an ancient Debian release from the late pleistocene and it hasn’t been ’round these parts for many moons.

Now we need to add Webmin author Jamie Cameron’s public key to our keyring. Do this from your home directory:

$ cd ~
$ wget http://www.webmin.com/jcameron-key.asc
--2012-04-29 01:34:19--  http://www.webmin.com/jcameron-key.asc
Resolving www.webmin.com (www.webmin.com)... 216.34.181.97
Connecting to www.webmin.com (www.webmin.com)|216.34.181.97|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1320 (1.3K) [text/plain]
Saving to: `jcameron-key.asc'

100%[======================================>] 1,320       --.-K/s   in 0s      

2012-04-29 01:34:19 (41.4 MB/s) - `jcameron-key.asc' saved [1320/1320]
$ sudo apt-key add ~/jcameron-key.asc
[sudo] password for kelvin: 
OK

Now we can install Webmin from the repo we added:

$ sudo apt-get update
...
Fetched 12.6 MB in 37s (333 kB/s)                                              
Reading package lists... Done
$ sudo apt-get install webmin
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  apt-show-versions libapt-pkg-perl libauthen-pam-perl libio-pty-perl
  libnet-ssleay-perl
The following NEW packages will be installed:
  apt-show-versions libapt-pkg-perl libauthen-pam-perl libio-pty-perl
  libnet-ssleay-perl webmin
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 16.1 MB of archives.
After this operation, 100 MB of additional disk space will be used.
Do you want to continue [Y/n]? Y
Get:1 http://download.webmin.com/download/repository/ sarge/contrib webmin all 1.580 [15.8 MB]
Get:2 http://ca.archive.ubuntu.com/ubuntu/ precise/main libnet-ssleay-perl i386 1.42-1build1 [184 kB]
...
Setting up libnet-ssleay-perl (1.42-1build1) ...
Setting up libauthen-pam-perl (0.16-2build2) ...
Setting up libio-pty-perl (1:1.08-1build2) ...
Setting up libapt-pkg-perl (0.1.25build2) ...
Setting up apt-show-versions (0.17) ...
** initializing cache. This may take a while **
Setting up webmin (1.580) ...
Webmin install complete. You can now login to https://brasenose:10000/
as root with your root password, or as any user who can use sudo
to run commands as root.

Webmin TLS certificate warning

Webmin now is running on port 10000 but you can inspect the TLS properties and see that it is using a 512-bit key. Your browser may warn you of the weak default cryptographic key. That sort of thing is fine if you’re living in North Korea, but we need to upgrade it to use a 2048-bit key like all the cool kids.

The username and password for Webmin is the same as any user that has sudo rights on the system. My username is therefore ‘kelvin’ and my password is ‘PASSWORD’. LOL. No, I’m not going to tell you my password…

Upgrade the self-signed SSL Certificate

Webmin upgraded 2048-bit key warning

Upgrading the Webmin certificate reduces TLS warnings

OpenSSL will be used to generate the needed keys and certificates. We are going to make a self-signed certificate which means that it will raise warnings, scary red flags, a Cthulhu and whoknowswhatelse in most browsers. So if this system will be used by easily frightened system admins (most are) then you might want to get a properly signed certificate from a Certificate Authority instead. Having said that (and alienated most of my readership) let’s get on with it.

The self-signed certificate will be valid for 1825 days or 5 years which is also how long your OS will be maintained by Canonical. Simply change the value after the ‘days’ attribute in the command to meet your needs.

Use OpenSSL to make a private key and a self-signed certificate in one badass command:

$ cd /etc/webmin
$ sudo openssl req -newkey rsa:2048 -days 1825 -nodes -x509 -keyout server.key -out server.crt
[sudo] password for kelvin: 
Generating a 2048 bit RSA private key
.............................................................................................+++
.........+++
writing new private key to 'server.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:British Columbia
Locality Name (eg, city) []:Victoria
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Kelvin Wong Heavy Industries S.p.A.
Organizational Unit Name (eg, section) []:Network Operations
Common Name (e.g. server FQDN or YOUR name) []:brasenose.kelvinwong.ca
Email Address []:postmaster@kelvinwong.ca

Okay, so how cool was that? Now you have to make your artifacts usable and safe. First, concatenate the private key and the certificate into a single PEM file that Webmin can understand (tee used for piping because I’m cool and I can read Wikipedia). Second, set the correct permissions and file ownership.


$ pwd
/etc/webmin
$ cat server.crt server.key | sudo tee server.pem > /dev/null
$ sudo chmod 600 server.pem server.key server.crt
$ sudo chown root:bin server.pem server.key server.crt
$ ls -l server.*
-rw------- 1 root bin 1610 Apr 29 13:33 server.crt
-rw------- 1 root bin 1704 Apr 29 13:33 server.key
-rw------- 1 root bin 3314 Apr 29 13:45 server.pem

Now you need to tell Webmin to use your new upgraded certificate.

$ sudo vi /etc/webmin/miniserv.conf

Change the certificate name:

26
keyfile=/etc/webmin/server.pem

Then restart Webmin:

$ sudo invoke-rc.d webmin restart
Stopping Webmin server in /usr/share/webmin
Starting Webmin server in /usr/share/webmin
Pre-loaded WebminCore

Your Webmin installation is now totally badass like a Honey Badger.

Webmin 2048-bit key details

Success upgrading Webmin TLS to 2048-bit key

Question: What changes do you make to your Webmin configuration so that it runs well on Ubuntu?

Tags: , , , , , , , ,

Github says: Please audit your SSH keys

I got the following from Github after their benign hacker incident:

Please audit your SSH keys
On Sunday March 4, 2012 a security vulnerability related to SSH keys (public keys) was discovered. For your protection and to prevent unauthorized access we have disabled your public keys until you approve them.

They want me to audit my SSH keys (a simple process). First, find your public key that you use on GitHub (probably in your .ssh directory if you are using a Mac). Then get its fingerprint. Here’s how you do that on a Mac:

Trinity:~ kelvin$ ls -l .ssh/id_rsa*
-rw-------  1 kelvin  staff  1743 Sep 11  2009 .ssh/id_rsa
-rw-r--r--  1 kelvin  staff   400 Sep 11  2009 .ssh/id_rsa.pub
Trinity:~ kelvin$ ssh-keygen -lf .ssh/id_rsa
2048 XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX .ssh/id_rsa.pub (RSA)
Trinity:~ kelvin$

Using ssh-keygen you can get the fingerprint from your private key filename (it will look for your public key for you). That long list of “XX:XX” things will be a hexadecimal number that matches the key fingerprint at the bottom of the GitHub SSH audit page. If it doesn’t match then either Egor hacked you or you might have used a different key (keep looking!).

Tags: , , , , ,

Install webby Postgres 8.4 on CentOS 6.2

At the end of this walkthrough you will have the PostgreSQL 8.4 database installed on CentOS 6.2 ready for use with your web projects. Postgres 8.4 is not the latest version, but it is stable and good enough for web development purposes. This set-up is “webby” in the sense that the it should be familiar to web developers.

Prerequisites

You need to be familiar with basic Linux system administration including editing configuration files with text-editors like vi or emacs.

This is our system. It is a basic CentOS 6.2 installation with a static IP:

$ uname -a
Linux schettino.kelvinwong.ca 2.6.32-220.4.1.el6.i686 #1 SMP Mon Jan 23 22:37:12 GMT 2012 i686 i686 i386 GNU/Linux
$ cat /etc/redhat-release
CentOS release 6.2 (Final)

Install Postgres

Installation of Postgres with yum is simple:

[kelvin@schettino ~]$ sudo yum install postgresql-server
[sudo] password for kelvin: 
Loaded plugins: fastestmirror, presto
Loading mirror speeds from cached hostfile
 * base: mirror.its.sfu.ca
 * extras: mirror.its.sfu.ca
 * updates: mirror.its.sfu.ca
base                                                     | 3.7 kB     00:00     
extras                                                   | 3.5 kB     00:00     
updates                                                  | 3.5 kB     00:00     
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package postgresql-server.i686 0:8.4.9-1.el6_1.1 will be installed
--> Processing Dependency: postgresql-libs(x86-32) = 8.4.9-1.el6_1.1 for package: postgresql-server-8.4.9-1.el6_1.1.i686
--> Processing Dependency: postgresql(x86-32) = 8.4.9-1.el6_1.1 for package: postgresql-server-8.4.9-1.el6_1.1.i686
--> Processing Dependency: libpq.so.5 for package: postgresql-server-8.4.9-1.el6_1.1.i686
--> Running transaction check
---> Package postgresql.i686 0:8.4.9-1.el6_1.1 will be installed
---> Package postgresql-libs.i686 0:8.4.9-1.el6_1.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                  Arch        Version                 Repository   Size
================================================================================
Installing:
 postgresql-server        i686        8.4.9-1.el6_1.1         base        3.3 M
Installing for dependencies:
 postgresql               i686        8.4.9-1.el6_1.1         base        2.7 M
 postgresql-libs          i686        8.4.9-1.el6_1.1         base        201 k

Transaction Summary
================================================================================
Install       3 Package(s)

Total download size: 6.2 M
Installed size: 28 M
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
Processing delta metadata
Package(s) data still to download: 6.2 M
(1/3): postgresql-8.4.9-1.el6_1.1.i686.rpm               | 2.7 MB     00:01     
(2/3): postgresql-libs-8.4.9-1.el6_1.1.i686.rpm          | 201 kB     00:00     
(3/3): postgresql-server-8.4.9-1.el6_1.1.i686.rpm        | 3.3 MB     00:01     
--------------------------------------------------------------------------------
Total                                           1.5 MB/s | 6.2 MB     00:04     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : postgresql-libs-8.4.9-1.el6_1.1.i686                         1/3 
  Installing : postgresql-8.4.9-1.el6_1.1.i686                              2/3 
  Installing : postgresql-server-8.4.9-1.el6_1.1.i686                       3/3 

Installed:
  postgresql-server.i686 0:8.4.9-1.el6_1.1                                      

Dependency Installed:
  postgresql.i686 0:8.4.9-1.el6_1.1    postgresql-libs.i686 0:8.4.9-1.el6_1.1   

Complete!
[kelvin@schettino ~]$

The server is installed along with the required client programs.

Configure Postgres – Initialize and start service

After installing Postgres you will need to initialize the database (once only):

[kelvin@schettino ~]$ sudo service postgresql initdb
Initializing database:                                     [  OK  ]

Set the server to restart on reboots and start the postmaster service:

[kelvin@schettino ~]$ sudo chkconfig postgresql on
[sudo] password for kelvin: 
[kelvin@schettino ~]$ sudo service postgresql start
Starting postgresql service:                               [  OK  ]

Configure Postgres – Set superuser password

Now let’s set a password for the superuser (the postgres user) using the PostgreSQL interactive terminal. Jump into the postgres user by using su (with the dash to get a login shell):

[kelvin@schettino ~]$ su -
Password: 
[root@schettino ~]# su - postgres
-bash-4.1$ psql
psql (8.4.9)
Type "help" for help.

postgres=# \password postgres
Enter new password: 
Enter it again:
postgres=# \q
-bash-4.1$

Configure Postgres – Activate password authentication

By default, the server uses ident as defined in the “PostgreSQL Client Authentication Configuration File”. If you open up pg_hba.conf you can see this default configuration:

67
68
69
70
71
72
73
74
# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
 
# "local" is for Unix domain socket connections only
local   all         all                               ident
# IPv4 local connections:
host    all         all         127.0.0.1/32          ident
# IPv6 local connections:
host    all         all         ::1/128               ident

Ident is a mapping of local system users (see /etc/passwd for list of system users) to Postgres users. I have never found this authentication method useful for any of the web development work that I have done. I always change it to “md5” which allows you to create arbitrary users and passwords. Let’s change the server’s client configuration file (I assume you are still using the postgres user shell):

-bash-4.1$ whoami
postgres
-bash-4.1$ vim /var/lib/pgsql/data/pg_hba.conf

Change the “ident” methods to “md5” methods at the bottom of the pg_hba.conf file:

67
68
69
70
71
72
73
74
75
76
77
# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
 
# "local" is for Unix domain socket connections only
local   all         all                               md5
# IPv4 local connections:
host    all         all         127.0.0.1/32          md5
# IPv6 local connections:
host    all         all         ::1/128               md5
# If you don't want to open Postgres to the Internet
# don't enable this line
host    all         all         0.0.0.0/0             md5

By default, Postgres binds only to localhost and you will need to explicitly tell it to bind to your machine’s IP address. The setting is in postgres.conf. If you don’t need remote access you can skip this.

-bash-4.1$ vim /var/lib/pgsql/data/postgresql.conf

Change the listen_addresses setting to an asterisk to listen to all available IP addresses:

57
58
59
60
61
62
63
# - Connection Settings -
 
listen_addresses = '*'
#listen_addresses = 'localhost'         # what IP address(es) to listen on;
                                        # comma-separated list of addresses;
                                        # defaults to 'localhost', '*' = all
                                        # (change requires restart)

Restart your postgres server (exit postgres user into the root shell):

-bash-4.1$ exit
logout
[root@schettino ~]# service postgresql restart
Stopping postgresql service:                               [  OK  ]
Starting postgresql service:                               [  OK  ]
[root@schettino ~]#

Open Firewall (optional)

If you want remote access to the server on Postgres port 5432 you will have to open a port on the firewall. If you still are the root user, type the following:

[root@schettino ~]# whoami
root
[root@schettino ~]# vim /etc/sysconfig/iptables

You can just copy the SSH port rule in iptables and modify the port number from 22 to 5432. Add the following rule just below the SSH port rule and above the rejection rule for the INPUT chain:

10
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5432 -j ACCEPT

When changed, it should look like this:

2
3
4
5
6
7
8
9
10
11
12
13
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5432 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Reload the rules:

[root@schettino ~]# service iptables restart
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
iptables: Applying firewall rules:                         [  OK  ]
[root@schettino ~]# exit
logout
[kelvin@schettino ~]$

Try it out with pgbench (optional)

To demonstrate the basic use of your new Postgres server, you can try out pgbench which is in the postgresql-contrib RPM. Let’s install it, create a new user, create a new database and run pgbench against it:

[kelvin@schettino ~]$ sudo yum install postgresql-contrib
Loaded plugins: fastestmirror, presto
Loading mirror speeds from cached hostfile
 * base: mirror.its.sfu.ca
 * extras: mirror.its.sfu.ca
 * updates: mirror.its.sfu.ca
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package postgresql-contrib.i686 0:8.4.9-1.el6_1.1 will be installed
--> Processing Dependency: libxslt.so.1(LIBXML2_1.0.18) for package: postgresql-contrib-8.4.9-1.el6_1.1.i686
--> Processing Dependency: libxslt.so.1(LIBXML2_1.0.11) for package: postgresql-contrib-8.4.9-1.el6_1.1.i686
--> Processing Dependency: libxslt.so.1 for package: postgresql-contrib-8.4.9-1.el6_1.1.i686
--> Processing Dependency: libossp-uuid.so.16 for package: postgresql-contrib-8.4.9-1.el6_1.1.i686
--> Running transaction check
---> Package libxslt.i686 0:1.1.26-2.el6 will be installed
---> Package uuid.i686 0:1.6.1-10.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                   Arch        Version                Repository   Size
================================================================================
Installing:
 postgresql-contrib        i686        8.4.9-1.el6_1.1        base        346 k
Installing for dependencies:
 libxslt                   i686        1.1.26-2.el6           base        448 k
 uuid                      i686        1.6.1-10.el6           base         54 k

Transaction Summary
================================================================================
Install       3 Package(s)

Total download size: 848 k
Installed size: 3.3 M
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
Processing delta metadata
Package(s) data still to download: 848 k
(1/3): libxslt-1.1.26-2.el6.i686.rpm                     | 448 kB     00:00     
(2/3): postgresql-contrib-8.4.9-1.el6_1.1.i686.rpm       | 346 kB     00:00     
(3/3): uuid-1.6.1-10.el6.i686.rpm                        |  54 kB     00:00     
--------------------------------------------------------------------------------
Total                                           520 kB/s | 848 kB     00:01     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : uuid-1.6.1-10.el6.i686                                       1/3 
  Installing : libxslt-1.1.26-2.el6.i686                                    2/3 
  Installing : postgresql-contrib-8.4.9-1.el6_1.1.i686                      3/3 

Installed:
  postgresql-contrib.i686 0:8.4.9-1.el6_1.1                                     

Dependency Installed:
  libxslt.i686 0:1.1.26-2.el6              uuid.i686 0:1.6.1-10.el6             

Complete!
[kelvin@schettino ~]$ which pgbench
/usr/bin/pgbench

Create a new Postgres user by using the createuser wrapper (the P switch allows you to set a password for your new user):

[kelvin@schettino ~]$ su -
Password: 
[root@schettino ~]# su - postgres
-bash-4.1$ createuser -P francesco
Enter password for new role: [password for user francesco]
Enter it again: 
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
Password: [password for postgres]
-bash-4.1$

Make a new database named “winnings” and change the owner to “francesco”:

-bash-4.1$ createdb -O francesco winnings
Password: [password for postgres]
-bash-4.1$ 

Now we can fill it up with pgbench:

-bash-4.1$ pgbench -i -U francesco winnings
Password: [password for user francesco]
NOTICE:  table "pgbench_branches" does not exist, skipping
NOTICE:  table "pgbench_tellers" does not exist, skipping
NOTICE:  table "pgbench_accounts" does not exist, skipping
NOTICE:  table "pgbench_history" does not exist, skipping
creating tables...
10000 tuples done.
20000 tuples done.
30000 tuples done.
40000 tuples done.
50000 tuples done.
60000 tuples done.
70000 tuples done.
80000 tuples done.
90000 tuples done.
100000 tuples done.
set primary key...
NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index "pgbench_branches_pkey" for table "pgbench_branches"
NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index "pgbench_tellers_pkey" for table "pgbench_tellers"
NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index "pgbench_accounts_pkey" for table "pgbench_accounts"
vacuum...done.
-bash-4.1$ pgbench -c 4 -S -t 2000 -U francesco winnings
Password: [password for user francesco]
starting vacuum...end.
transaction type: SELECT only
scaling factor: 1
query mode: simple
number of clients: 4
number of transactions per client: 2000
number of transactions actually processed: 8000/8000
tps = 4836.016718 (including connections establishing)
tps = 5052.773057 (excluding connections establishing)
-bash-4.1$ pgbench -c 4 -t 2000 -U francesco winnings
Password: [password for user francesco]
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 4
number of transactions per client: 2000
number of transactions actually processed: 8000/8000
tps = 237.345234 (including connections establishing)
tps = 237.889294 (excluding connections establishing)
-bash-4.1$

You can clean up the database by dropping the “winnings” database and dropping “francesco”:

-bash-4.1$ dropdb winnings
Password: [password for postgres]
-bash-4.1$ dropuser francesco
Password: [password for postgres]
-bash-4.1$ 

Enjoy your webby Postgres!

Caveat! If you have an Apache/PHP5 server that wants to talk to your Postgres, you will have to set the appropriate SELinux boolean to allow the communication: setsebool -P httpd_can_network_connect_db 1

Tags: , , , , , , , , , ,

Multi-Site Solr for Drupal 6 Search on Tomcat 6 / CentOS 6

ApacheSolr for Drupal 6 improves on the out-of-the-box search experience for Drupal users. The easiest way to get Solr running on your Drupal web site is to use the hosted service provided by Acquia; it is way easier than running your own Solr. You simply point your queries to their Solr server and you’re done.

For various reasons, you might want to run your own Solr web service on your own machine. In this article, I will walk you through setting up a working Solr installation using Tomcat 6 on CentOS 6. The end result of this walkthrough will be two separate Solr indexes (via two separate Solr web apps) for two different web sites running on a single Tomcat. I will assume that you are using Acquia’s Drupal (which ships with SolrPHPClient).

Warning: This article assumes all services are on a single machine (suitable for a small organization). Running Solr on a separate machine is possible but raises security implications that are outside the scope of this article.

These are the tasks that we will work on:

  1. Set-up Solr
  2. Set-up Tomcat
  3. Tweak CentOS security thinger (SELinux)
  4. Configure Acquia Drupal

Prerequisites

The prerequisites are:

  • CentOS 6 Web Server w/ PHP 5.3, MySQL 5, Tomcat 6, Java 6 (all services running w/ no problemos)
  • Acquia Drupal 6 installed
  • Familiarity with Drupal (basic skills – enabling modules, setting permissions on nodes, etc)
  • Familiarity with Java & Tomcat (basic skills)
  • Familiarity working with Linux in a terminal and vi (intermediate skills)

This is my system (a web server set-up with Anaconda):

# uname -a
Linux templeton.localdomain 2.6.32-71.29.1.el6.i686 #1 SMP Mon Jun 27 18:07:00 BST 2011 i686 i686 i386 GNU/Linux
# cat /etc/redhat-release
CentOS Linux release 6.0 (Final)
# yum list installed | grep mysql-server
mysql-server.i686       5.1.52-1.el6_0.1  @updates
# yum list installed | grep php
php.i686                5.3.2-6.el6_0.1   @updates                              
php-cli.i686            5.3.2-6.el6_0.1   @updates                              
php-common.i686         5.3.2-6.el6_0.1   @updates                              
php-gd.i686             5.3.2-6.el6_0.1   @updates                              
php-mysql.i686          5.3.2-6.el6_0.1   @updates                              
php-pdo.i686            5.3.2-6.el6_0.1   @updates                              
php-pear.noarch         1:1.9.0-2.el6     @anaconda-centos-201106051823.i386/6.0
php-xml.i686            5.3.2-6.el6_0.1   @updates
# java -version
java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.5) (rhel-1.31.b17.el6_0-i386)
OpenJDK Client VM (build 14.0-b16, mixed mode)
# yum list installed | grep tomcat6
tomcat6.noarch          6.0.24-24.el6_0   @updates                              
tomcat6-el-2.1-api.noarch
tomcat6-jsp-2.1-api.noarch
tomcat6-lib.noarch      6.0.24-24.el6_0   @updates                              
tomcat6-servlet-2.5-api.noarch
# /sbin/service tomcat6 status
tomcat6 (pid 1790) is running...                           [  OK  ]
# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 24
Policy from config file:        targeted

Notice the hashmark (#) as my terminal prompt. It denotes that I am executing all these commands as root (use ‘su -‘). You can also prefix the following commands with ‘sudo’.

Download Solr

Obtain a copy of the Solr tarball from a nearby mirror:

http://www.apache.org/dyn/closer.cgi/lucene/solr/

Select Solr 1.4.1 or the latest recommended Solr:

ie. http://apache.sunsite.ualberta.ca//lucene/solr/1.4.1/

I’m using the 54M GZipped Tarball and downloading it using wget:

# wget http://apache.sunsite.ualberta.ca//lucene/solr/1.4.1/apache-solr-1.4.1.tgz
--2011-09-02 02:06:05--  http://apache.sunsite.ualberta.ca//lucene/solr/1.4.1/apache-solr-1.4.1.tgz
Resolving apache.sunsite.ualberta.ca... 129.128.5.190
Connecting to apache.sunsite.ualberta.ca|129.128.5.190|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 56374837 (54M) [application/x-tar]
Saving to: “apache-solr-1.4.1.tgz”

100%[=============================================>] 56,374,837   261K/s   in 6m 20s  

2011-09-02 02:12:42 (145 KB/s) - “apache-solr-1.4.1.tgz” saved [56374837/56374837]

# tar zxvf apache-solr-1.4.1.tgz
apache-solr-1.4.1/client/
...
# pwd
/root

Copy the Solr package somewhere reasonable like in the /opt folder:

# mkdir -p /opt/solr
# cp -r -p /root/apache-solr-1.4.1 /opt/solr

Link it (the Solr WAR file) to the Tomcat library directory:

# ln -s /opt/solr/apache-solr-1.4.1/dist/apache-solr-1.4.1.war /usr/share/tomcat6/lib/solr.war

In the future, when you upgrade your software, install the Solr upgrade and update the symlink.

Create Solr directories

You need to choose where your Solr indexes will be kept. I put them into the /var directory and that’s where I’m assuming that you will put yours:

# mkdir -p /var/solr
# cp -r -p /opt/solr/apache-solr-1.4.1/example/solr/ /var/solr/
# mv /var/solr/solr /var/solr/example.com
# ls -l /var/solr/example.com/
total 12
drwxr-xr-x. 2 root root 4096 Sep  2 02:44 bin
drwxr-xr-x. 3 root root 4096 Sep  2 02:44 conf
-rw-r--r--. 1 root root 2259 Sep  2 02:44 README.txt

Each domain has its own Solr indexes located in ‘data‘ and its own configuration files in ‘conf‘. There are two optional directories: ‘bin‘ (for replication scripts) and ‘lib‘ (for plugins). Unless your other apps use them, chances are they will be missing.

Install Drupal ApacheSolr plugin protwords, schema and solrconfig

You should already have Acquia Drupal 6 running or Drupal 6 with the ApacheSolr plugin installed. You can copy the ‘protwords.txt’, ‘schema.xml’, and ‘solrconfig.xml’ files from the plugin directory in your respective distribution rather than downloading it, but adjust the paths accordingly.

If you don’t already have the ApacheSolr plugin, get it from the Drupal web site.

http://drupal.org/project/apachesolr

Choose the latest Tarball and use wget to download it to your server, then copy the ApacheSolr configuration files (and backup originals using ‘b’ flag):

# wget http://ftp.drupal.org/files/projects/apachesolr-6.x-1.5.tar.gz
# tar zxvf apachesolr-6.x-1.5.tar.gz
...
# echo 'If ur root cp may give u a scary msg next cmd! Ignore it! Y to overwrite!'
If ur root cp may give u a scary msg next cmd! Ignore it! Y to overwrite!
#
# cp -b -p -f apachesolr/protwords.txt /var/solr/example.com/conf
# cp -b -p -f apachesolr/schema.xml /var/solr/example.com/conf
# cp -b -p -f apachesolr/solrconfig.xml /var/solr/example.com/conf
#
# echo 'Fix group so tomcat can use this!'
Fix group so tomcat can use this!
#
# chown -R root:tomcat /var/solr/example.com
# chmod -R 775 /var/solr/

Warning! If you are not using the Acquia distribution and instead installed the ApacheSolr plugin from the main Drupal web site then you should check that you have a copy of the SolrPhpClient (version r22 – see module README for the gory details). The Acquia distribution includes the correct SolrPhpClient (so you might want to use that instead?).

Make the two Solr instances for the two domains

This walkthrough will create two domains, but you can create more. Using the example.com folder as a prototype, just recursively copy it twice to make two domains (use ‘p’ switch to ‘preserve’ the file permissions and settings):

# cp -r -p /var/solr/example.com /var/solr/www1.kelvinwong.ca
# cp -r -p /var/solr/example.com /var/solr/www2.kelvinwong.ca

If the future, to add a new domain, copy the example.com folder you just made and customize it. This will also work for additional domains that you want to support.

Configure Tomcat 6

It’s All About Context: The Context element represents a web application run within a particular Tomcat virtual host. Each web application is based on a Web Application Archive (WAR) file or a corresponding unpacked directory. The web application used to process each web request is determined by matching the request to the path of each Context. You may define as many Context elements as you wish, but each Context MUST have a unique path. More on Context

Contexts are no longer put into Tomcat’s server.xml file since that file is read only at server start-up. Instead Contexts are placed into a folder hierarchy under CATALINA_BASE (on CentOS 6 it is /etc/tomcat6). Create and configure the following files:

# touch /etc/tomcat6/Catalina/localhost/www1.kelvinwong.ca.xml
# touch /etc/tomcat6/Catalina/localhost/www2.kelvinwong.ca.xml
# chown tomcat:root /etc/tomcat6/Catalina/localhost/{www1.kelvinwong.ca.xml,www2.kelvinwong.ca.xml}
# chmod 664 /etc/tomcat6/Catalina/localhost/{www1.kelvinwong.ca.xml,www2.kelvinwong.ca.xml}

Tomcat will use these files to find the WAR and deploy the application using the settings in the Context. Note: Contexts can be overridden (they often are) and there are more than a few in Tomcat. Review Tomcat’s documentation if they give you any trouble.

Make sure your Context fragments have .xml suffixes!

Place the following into /etc/tomcat6/Catalina/localhost/www1.kelvinwong.ca.xml

# vi /etc/tomcat6/Catalina/localhost/www1.kelvinwong.ca.xml
<?xml version="1.0" encoding="utf-8"?>
<Context docBase="/usr/share/tomcat6/lib/solr.war" debug="0" crossContext="true" >
   <Environment name="solr/home" type="java.lang.String" value="/var/solr/www1.kelvinwong.ca" override="true" />
</Context>

The Context fragment is simply telling Tomcat where to find the Context root (document base). It is an absolute path to its web app archive (WAR) file. CrossContext allows Solr to get a request dispatcher from ServletContext.getContext() for access to other web apps on the virtual host. The Environment tag defines the ‘solr/home‘ setting and allows it to be overridden. That’s all you need.

Change the other fragment:

# vi /etc/tomcat6/Catalina/localhost/www2.kelvinwong.ca.xml

Change the paths:

<?xml version="1.0" encoding="utf-8"?>
<Context docBase="/usr/share/tomcat6/lib/solr.war" debug="0" crossContext="true" >
   <Environment name="solr/home" type="java.lang.String" value="/var/solr/www2.kelvinwong.ca" override="true" />
</Context>

Bind Tomcat to Local Port

By default, Tomcat listens on port 8080. The default iptables ruleset in CentOS 6 does not allow remote connections to port 8080. For our purposes this is fine since we want our Drupal sites to connect locally on port 8080. Local good, remote bad.

You can also tell Tomcat to bind to localhost and not any of the other network adapters. Open Tomcat’s server.xml file:

# vi /etc/tomcat6/server.xml

Change Tomcat’s binding address to the localhost address (127.0.0.1) in the Connector tag:

69
70
71
72
73
74
    <Connector port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" 
               URIEncoding="UTF-8"
               maxHttpHeaderSize="65535"
               address="127.0.0.1" />

Solr is a web service that takes many requests from Drupal using the HTTP GET method, similar to you typing into your browser’s web address bar. These requests routinely get very long; you can increase the GET request character limit by increasing the maxHttpHeaderSize attribute (from 8k to 64k as shown). To handle non-English characters, you should also set the request encoding to UTF-8. The Connector as-shown does both.

Restart Tomcat to reload the server.xml file:

# /sbin/service tomcat6 restart
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]

View Solr Admin (optional)

You should now be able to view the Solr administration page if you open a local web browser on the server. If you don’t have a desktop on the server (as should be the case), you can use a text-browser like elinks.

View http://localhost:8080/www1.kelvinwong.ca/admin:

# elinks http://localhost:8080/www1.kelvinwong.ca/admin

You should see the Solr administration page in your browser.

SELinux

“Apache Solr: Your site was unable to contact the Apache Solr server,” reports Drupal; SELinux chuckles.

SELinux is enabled by default on CentOS 6, so you will likely have it running and it will not appreciate Apache trying to talk to Tomcat/Solr on port 8080 (check /var/log/audit/audit.log):

type=AVC msg=audit(1315100262.891:17629): avc:  denied  { name_connect } for  pid=2064 comm="httpd" dest=8080 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket

type=SYSCALL msg=audit(1315100262.891:17629): arch=40000003 syscall=102 success=no exit=-13 a0=3 a1=bfbe6590 a2=b70426f4 a3=11 items=0 ppid=2060 pid=2064 auid=500 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)

You can either turn off SELinux (not recommended) or fix the attributes so that SELinux allows Apache to talk to Tomcat. The handy tool sealert gives helpful advice:

# sealert -a /var/log/audit/audit.log | less

Summary:

SELinux is preventing the http daemon from connecting to itself or the relay ports

Detailed Description:

SELinux has denied the http daemon from connecting to itself or the relay ports. An httpd script is trying to make a network connection to an http/ftp port. If you did not setup httpd to make network connections, this could signal an intrusion attempt.

Allowing Access:

If you want httpd to connect to httpd/ftp ports you need to turn on the
httpd_can_network_relay boolean: "setsebool -P httpd_can_network_relay=1"

Fix Command:

setsebool -P httpd_can_network_relay=1

Additional Information:

Source Context                unconfined_u:system_r:httpd_t:s0
Target Context                system_u:object_r:http_cache_port_t:s0
Target Objects                None [ tcp_socket ]
Source                        httpd
Source Path                   /usr/sbin/httpd
Port                          8080
Host                          <Unknown>
Source RPM Packages           httpd-2.2.15-5.el6.centos
Target RPM Packages           
Policy RPM                    selinux-policy-3.7.19-54.el6_0.5
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Plugin Name                   httpd_can_network_relay
Host Name                     templeton.localdomainPlatform                      Linux templeton.localdomain                              2.6.32-71.29.1.el6.i686 #1 SMP Mon Jun 27 18:07:00
                              BST 2011 i686 i686
Alert Count                   14First Seen                    Sat Sep  3 18:25:40 2011Last Seen                     Sat Sep  3 18:37:42 2011Local ID                      4b66d238-ddf7-4b74-bbe5-3fb54be5b3e4Line Numbers                  178, 179, 180, 181, 182, 183, 184, 185, 192, 193,
                              194, 195, 196, 197, 198, 199, 200, 201, 202, 203,
                              204, 205, 206, 207, 208, 209, 210, 211

Once upon a time and a very good time it was there was a moocow coming down along the road and this moocow that was coming down along the road met a nicens little boy named baby tuckoo[1]

The quick fix is to set the network relay flag (‘P’ flag makes the change persistent across reboots):

# setsebool -P httpd_can_network_relay=1
# getsebool httpd_can_network_relay
httpd_can_network_relay --> on

You don’t need sealert to use setsebool but it is a useful utility to debug errors with SELinux. If you don’t have sealert installed, it is a simple thing to install it since it is part of the setroubleshoot package:

# yum install setroubleshoot

Configure Drupal to use Solr

Turning now to your Drupal installation…

Enable the Solr Search service module…

Configure the Apache Solr Search module by visiting http://www1.kelvinwong.ca/?q=admin/settings/apachesolr

Solr host name
localhost
Solr port
8080
Solr path
/www1.kelvinwong.ca

The Solr path is the name of your Context fragment minus the xml suffix (ie. /etc/tomcat6/Catalina/localhost/www1.kelvinwong.ca.xml)

The cron job indexes 50 nodes at a time by default. When indexed, you can then search for nodes by keyword.

Save the settings. You should see:

  • The configuration options have been saved.
  • Apache Solr: Your site has contacted the Apache Solr server.
  • Apache Solr PHP Client Library: Correct version “Revision: 22”.

Try a search

You can re-index the site by force or let cron do it gradually. Either way it take a while for Solr to process the data.

http://www1.kelvinwong.ca/?q=admin/settings/apachesolr/index

Once you have indexed your site and adjusted the permissions on the search form (so anonymous users can use the search form), visit it:

http://www1.kelvinwong.ca/?q=search/apachesolr_search

Intentionally misspell something and let Solr give you hints!

What about the other one??? www2?

Ah, yes…the other one is set-up in a similar manner, just use the following configuration in Drupal:

Solr host name
localhost
Solr port
8080
Solr path
/www2.kelvinwong.ca

Tags: , , , , , , , , , , , , , ,

Compile Python 2.5.6 for 64-bit CentOS/RHEL 5.6 (RedHat)

It is possible to build Python 2.5.6 as a 64-bit RPM for CentOS/RHEL(RedHat) 5.6:

[kelvin@campion ~]$ cat /etc/redhat-release 
CentOS release 5.6 (Final)
[kelvin@campion ~]$ python25
Python 2.5.6 (r256:88840, Jun 15 2011, 19:58:29) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-50)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

I’m going to follow the method detailed in a blog post by Bryan O’Sullivan and build an RPM using a source RPM from the Fedora Project. My system is a 64-bit virtual machine running CentOS 5.6. Except for a post-install update via yum, a static LAN IP and an Apache HTTPD, this machine is exactly what you would get if you installed the server from a netinstall:

[kelvin@campion ~]$ uname -a
Linux campion 2.6.18-238.el5 #1 SMP Thu Jan 13 15:51:15 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

Build tools

First step is to install the tools and packages that you will need to build your Python RPM (~70mb with dependencies):

$ sudo yum install autoconf bzip2-devel db4-devel \
  expat-devel findutils gcc-c++ gdbm-devel glibc-devel gmp-devel \
  libGL-devel libX11-devel libtermcap-devel ncurses-devel \
  openssl-devel pkgconfig readline-devel sqlite-devel tar \
  tix-devel tk-devel zlib-devel rpm-build

Find a Python 2.5 source RPM

The last Fedora that shipped with Python 2.5 was Fedora 10 so we need to get that source RPM. Visit your closest Fedora 10 mirror and download it to your working directory:

[kelvin@campion ~]$ cd
[kelvin@campion ~]$ wget http://mirrordenver.fdcservers.net/fedora/releases/10/Fedora/source/SRPMS/python-2.5.2-1.fc10.src.rpm

Now that you have the source RPM, extract it (into a temporary build directory) with the following:

[kelvin@campion ~]$ mkdir -p /tmp/py25/{BUILD,RPMS,SOURCES,SPECS}
[kelvin@campion ~]$ rpm --define '_topdir /tmp/py25' -ivh python-2.5.2-1.fc10.src.rpm
warning: python-2.5.2-1.fc10.src.rpm: Header V3 DSA signature: NOKEY, key ID 4ebfc273
   1:python                 warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
...

The ‘mock’ warnings refer to the Fedora build tool called ‘mock’ and they can be ignored. You have now extracted the source from the RPM and it resides in /tmp/py25.

Download the Python 2.5.6 source

At the time this post was written, Python 2.5 was due to be left unmaintained after Oct 2011. Alas, I have some unmigrated 2.5 apps so we need to get the latest Python 2.5 source and replace the BZipped tarball in the source RPM (ensure you download the BZipped source from the Python web site).

$ cd /tmp/py25/SOURCES/
$ wget http://www.python.org/ftp/python/2.5.6/Python-2.5.6.tar.bz2
$ ls -l Python-2.5.*
-rw-r--r-- 1 kelvin kelvin 9807597 Sep 24  2008 Python-2.5.2.tar.bz2
-rw-rw-r-- 1 kelvin kelvin 9821788 May 26 07:46 Python-2.5.6.tar.bz2

Edit the RPM spec and a patch file

You want to allow your RPM build to use the older 4.3 version of BerkeleyDB that ships with CentOS 5.6. You also want the RPM to use the source archive we just downloaded and not the one that came with the RPM. You need to make the following minor changes using an editor like vim or emacs or nano (yuk!):

$ cd /tmp/py25/SOURCES/
$ vim python-2.5-config.patch

Change line 251 (vim hint: in command mode ‘:251’ goes to line 251, ‘i’ enters insert mode, edit-edit-edit, ‘esc’ goes back to command mode, ‘ZZ’ saves and closes the file):

251
+DBLIBVER=4.3

Your RPM will now use the CentOS 5.6 standard BerkeleyDB version 4.3.

$ cd /tmp/py25/SPECS
$ vim python.spec

Edit these lines:

24
Version: 2.5.6
86
BuildPrereq: db4-devel >= 4.3
224
225
#%patch999 -p1 -b .cve2007-4965
#%patch998 -p0 -b .cve2008-2316

Your RPM build will now use the Python version 2.5.6 source archive in your SOURCES directory. The two CVE patches have already been applied in Python 2.5.6 so we must comment out those lines in the spec file so the included Fedora patches are not applied.

Build your Python 2.5.6 RPM

You are now ready to build the RPM. Go into the SPECS directory and build it:

$ cd /tmp/py25/SPECS
$ rpmbuild --define '_topdir /tmp/py25' --define '__python_ver 25' -bb python.spec

Once packaged, your RPMs can be found in the RPM directory (if you built an i386 version it will be in a different directory):

[kelvin@campion x86_64]$ cd /tmp/py25/RPMS/x86_64
[kelvin@campion x86_64]$ ls -l
total 13116
-rw-r--r-- 1 kelvin kelvin 6350252 Jun 15 20:12 python25-2.5.6-1.x86_64.rpm
-rw-r--r-- 1 kelvin kelvin  932782 Jun 15 20:12 python25-devel-2.5.6-1.x86_64.rpm
-rw-r--r-- 1 kelvin kelvin 1469432 Jun 15 20:12 python25-libs-2.5.6-1.x86_64.rpm
-rw-r--r-- 1 kelvin kelvin 3849692 Jun 15 20:13 python25-test-2.5.6-1.x86_64.rpm
-rw-r--r-- 1 kelvin kelvin  457052 Jun 15 20:12 python25-tools-2.5.6-1.x86_64.rpm
-rw-r--r-- 1 kelvin kelvin  329428 Jun 15 20:12 tkinter25-2.5.6-1.x86_64.rpm

Install your Python 2.5.6 RPM

Your 64-bit RPMs can be installed with one line:

[kelvin@campion x86_64]$ sudo rpm -ivh /tmp/py25/RPMS/x86_64/*.rpm
[sudo] password for kelvin: 
Preparing...                ########################################### [100%]
   1:python25               ########################################### [ 17%]
   2:python25-libs          ########################################### [ 33%]
   3:tkinter25              ########################################### [ 50%]
   4:python25-devel         ########################################### [ 67%]
   5:python25-test          ########################################### [ 83%]
   6:python25-tools         ########################################### [100%]
[kelvin@campion x86_64]$

Use your Python 2.5.6

Your new(-ish) Python 2.5.6 interpreter is invoked with python25 in order to preserve the system default Python 2.4 intrpreter used by yum and pretty much everything else on CentOS.

[kelvin@campion x86_64]$ python25
Python 2.5.6 (r256:88840, Jun 15 2011, 19:58:29) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-50)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>>

Tags: , , , , ,

Doctoring EXIF data for Sun Media (aka Ignatieff in Kuwait)

Not Iggy

A lot hinged on the veracity of the picture — the low-resolution image furnished to Teneycke lacked critical metadata that would have helped determine the time the picture was taken. However, the report that accompanied the picture referred to those metadata. (Pierre Karl Peladeau
President and CEO of Sun Media Corporation
)

Before sending your hoax photos to anybody at Sun Media, you better make sure that you doctor the EXIF metadata because they will check – trust me. You’re a busy lobbyist and you don’t have time to learn all about this nerdy stuff (metadata wazzat???). No problem, this is what you do.

First, get your doctored photo and open it with a metadata editor like ExifTool There are others available but you’re in a rush and there is an election afoot and you have a ton of disinformation that has to get out – today!

Well, as luck would have it, there is no metadata on your pic. Don’t panic! Your Iggy pic backstory is that he was in Kuwait so you need to copy legit metadata from a pic taken in Kuwait. Go ask Google Image search, type “Kuwait army” and restrict your results to large pictures. The reason that we are selecting only “large” images is that we want the original/unedited large size pics taken by some US military photographer – the exact ones downloaded off the camera. They are always huge files. I found one from Military Sealift Command from 2007 which is good enough for our demo. Save it to your current working directory as we are going to copy the EXIF metadata to make it look like our hoax pic was taken in Kuwait.

Using ExifTool, copy all the metadata from the authentic Kuwait photo to your fake Iggy pic (only one command – w00t):

$ exiftool -tagsFromFile metadata_source.jpg not_ignatieff_exif.jpg

Now, if you’re particularly lazy or busy you can stop there since the metadata is now copied. If you have more time on your hands you can actually edit the individual entries and geocode the photo. Check out what we did:

$ exiftool -list not_ignatieff_exif.jpg
ExifTool Version Number         : 8.56
File Name                       : not_ignatieff_exif.jpg
Directory                       : .
File Size                       : 78 kB
File Modification Date/Time     : 2011:04:27 13:18:53-07:00
File Permissions                : rw-r--r--
File Type                       : JPEG
MIME Type                       : image/jpeg
JFIF Version                    : 1.02
Exif Byte Order                 : Little-endian (Intel, II)
Make                            : NIKON CORPORATION
Camera Model Name               : NIKON D2X
Orientation                     : Horizontal (normal)
X Resolution                    : 300
Y Resolution                    : 300
Resolution Unit                 : inches
Software                        : Adobe Photoshop CS2 Windows
Modify Date                     : 2007:09:24 12:06:07
Y Cb Cr Positioning             : Centered
Exposure Time                   : 1/180
F Number                        : 13.0
Exposure Program                : Aperture-priority AE
ISO                             : 100
Exif Version                    : 0221
Date/Time Original              : 2007:08:29 10:47:40
Create Date                     : 2007:08:29 10:47:40
Components Configuration        : Y, Cb, Cr, -
Exposure Compensation           : -2/3
Max Aperture Value              : 4.0
Metering Mode                   : Multi-segment
Light Source                    : Unknown
Flash                           : No Flash
Focal Length                    : 20.0 mm
User Comment                    : 
Sub Sec Time                    : 00
Sub Sec Time Original           : 00
Sub Sec Time Digitized          : 00
Flashpix Version                : 0100
Color Space                     : sRGB
Exif Image Width                : 2100
Exif Image Height               : 1395
Sensing Method                  : One-chip color area
File Source                     : Digital Camera
Scene Type                      : Directly photographed
CFA Pattern                     : [Red,Green][Green,Blue]
Custom Rendered                 : Normal
Exposure Mode                   : Auto
White Balance                   : Auto
Digital Zoom Ratio              : 1
Focal Length In 35mm Format     : 30 mm
Scene Capture Type              : Standard
Gain Control                    : None
Contrast                        : Normal
Saturation                      : Normal
Sharpness                       : Normal
Subject Distance Range          : Unknown
GPS Version ID                  : 2.2.0.0
Compression                     : JPEG (old-style)
Thumbnail Offset                : 934
Thumbnail Length                : 5386
Current IPTC Digest             : 460cf28926b856dab09c01a1b0a79077
Application Record Version      : 2
Copyright Flag                  : False
Global Angle                    : 30
Global Altitude                 : 30
XMP Toolkit                     : Image::ExifTool 8.56
Format                          : image/jpeg
Compressed Bits Per Pixel       : 2
Date/Time Digitized             : 2007:08:29 10:47:40-04:00
Flash Fired                     : False
Flash Function                  : False
Flash Mode                      : Unknown
Flash Red Eye Mode              : False
Flash Return                    : No return detection
Color Mode                      : RGB
ICC Profile Name                : sRGB IEC61966-2.1
Creator Tool                    : Adobe Photoshop CS2 Windows
Metadata Date                   : 2007:09:24 12:06:07-04:00
Derived From Document ID        : adobe:docid:photoshop:0f10c753-6154-11dc-9f27-a9bb9c4b68e4
Derived From Instance ID        : adobe:docid:photoshop:0f10c753-6154-11dc-9f27-a9bb9c4b68e4
Document ID                     : uuid:FF3F520BB86ADC11827AB2BAB20EBAFA
Instance ID                     : uuid:319A5A0FB86ADC11827AB2BAB20EBAFA
History                         : 
Quality                         : 60%
DCT Encode Version              : 100
APP14 Flags 0                   : [14], Encoded with Blend=1 downsampling
APP14 Flags 1                   : (none)
Color Transform                 : YCbCr
Image Width                     : 640
Image Height                    : 480
Encoding Process                : Progressive DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:4:4 (1 1)
Aperture                        : 13.0
Shutter Speed                   : 1/180
Create Date                     : 2007:08:29 10:47:40.00
Date/Time Original              : 2007:08:29 10:47:40.00
Modify Date                     : 2007:09:24 12:06:07.00
Thumbnail Image                 : (Binary data 5386 bytes, use -b option to extract)
Image Size                      : 640x480
Light Value                     : 14.9
Scale Factor To 35 mm Equivalent: 1.5
Circle Of Confusion             : 0.020 mm
Field Of View                   : 61.9 deg
Focal Length                    : 20.0 mm (35 mm equivalent: 30.0 mm)
Hyperfocal Distance             : 1.54 m

Note: This is a parody entry. Don’t send any doctored pics to Sun Media. Also, don’t trust metadata as proof of anything.

Tags: , , , , , , ,

Manage multiple SSH private keys with IdentityFile

There are many guides that show you how to set-up your SSH client for password-less login using public-private key certificates. If you have different clients, you may have several different private keys. How can you manage them?

It was pointed out that ssh-agent and PuTTY’s Pagent can also be used to manage multiple private keys.

SSH has a per-user configuration file called ‘~/.ssh/config’ that it can use to select your private keys based on the remote user name and remote host by using wildcards. Let’s check out my ‘config’ file:

IdentityFile ~/.ssh/ids/%h/%r/id_rsa
IdentityFile ~/.ssh/ids/%h/%r/id_dsa
IdentityFile ~/.ssh/ids/%h/id_rsa
IdentityFile ~/.ssh/ids/%h/id_dsa
IdentityFile ~/.ssh/id_rsa
IdentityFile ~/.ssh/id_dsa

The percent-h and percent-r take the host and the remote user from your SSH user and hostname arguments. Consider this example command:

$ ssh remote_user@remote_hostname.example.com

From the example command, the SSH client would use the wildcards to seek the correct key to use:

~/.ssh/ids/remote_hostname.example.com/remote_user/

This means that if you had two private keys that you used to access two different servers, you would arrange them as follows. The first one is arranged as follows:

$ ls -l ~/.ssh/ids/remote.example.com/remote_user/
total 16
-rw-------  1 kelvin  staff  668 Mar 24 20:09 id_dsa
-rw-r--r--  1 kelvin  staff  610 Mar 24 20:09 id_dsa.pub
$ ssh remote_user@remote.example.com
[remote_user@remote ~]$

Our second example uses a simple hostname. If a remote user is not required, you can just use the hostname:

$ ls -l ~/.ssh/ids/webby.example.org/
total 16
-rw-------  1 kelvin  staff  668 Mar 24 20:09 id_rsa
-rw-r--r--  1 kelvin  staff  610 Mar 24 20:09 id_rsa.pub
$ ssh webby.example.org
[webby ~]$

For sure, these are totally contrived examples, but you can watch the cascade yourself by adding the verbosity flag(s) to your SSH client session (this one is my client’s WebFaction account):

Trinity:.ssh kelvin$ ssh -v user@user.webfactional.com
OpenSSH_5.2p1, OpenSSL 0.9.7l 28 Sep 2006
debug1: Reading configuration data /Users/kelvin/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to user.webfactional.com [192.168.0.254] port 22.
debug1: Connection established.
debug1: identity file /Users/kelvin/.ssh/ids/user.webfactional.com/user/id_rsa type -1
debug1: identity file /Users/kelvin/.ssh/ids/user.webfactional.com/user/id_dsa type 2
debug1: identity file /Users/kelvin/.ssh/ids/user.webfactional.com/id_rsa type -1
debug1: identity file /Users/kelvin/.ssh/ids/user.webfactional.com/id_dsa type -1
debug1: identity file /Users/kelvin/.ssh/id_rsa type 1
debug1: identity file /Users/kelvin/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.2
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'user.webfactional.com' is known and matches the RSA host key.
debug1: Found key in /Users/kelvin/.ssh/known_hosts:41
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/kelvin/.ssh/ids/user.webfactional.com/user/id_rsa
debug1: Offering public key: /Users/kelvin/.ssh/ids/user.webfactional.com/user/id_dsa
debug1: Server accepts key: pkalg ssh-dss blen 433
debug1: read PEM private key done: type DSA
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
Last login: Thu Mar 31 22:31:08 2015 from 192.168.0.200
[user@web ~]$

Tags: , , , ,

Speed up PHP with APC on Ubuntu 10.04LTS

Ubuntu 10.04 LTS makes it quite simple to set up a basic LAMP server using tasksel; however, the default PHP set up does not include APC, the Alternative PHP Cache, which speeds up many PHP applications like Drupal. In the past, setting up APC involved using PECL or installing from source, but with Ubuntu Lucid, the process has been simplified using apt-get.

First, let me identify my demo system. It is running Ubuntu 10.04 LTS Lucid and has been patched to the latest version:

$ uname -a
Linux demo 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:17:33 UTC 2010 i686 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 10.04.2 LTS
Release:	10.04
Codename:	lucid
$ sudo apache2ctl status | grep "Server Version"
Server Version: Apache/2.2.14 (Ubuntu) PHP/5.3.2-1ubuntu4.7 with Suhosin-Patch
$ apt-cache show php-apc | grep Version
Version: 3.1.3p1-2

Ubuntu has added a Debian package into universe that allows APC to be added to any system quite easily:

$ sudo apt-get install php-apc
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  php5-gd
The following NEW packages will be installed:
  php-apc
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0B/77.2kB of archives.
After this operation, 217kB of additional disk space will be used.
Selecting previously deselected package php-apc.
(Reading database ... 28911 files and directories currently installed.)
Unpacking php-apc (from .../php-apc_3.1.3p1-2_i386.deb) ...
Processing triggers for libapache2-mod-php5 ...
 * Reloading web server config apache2
   ...done.
Setting up php-apc (3.1.3p1-2) ...

Note: You must restart the web server to begin using APC

$ sudo apache2ctl graceful

Out of the box (er…package), APC has some sane defaults. If you are “a serious user,” you will want to change your configuration yourself. Seriously, that is what the documentation says:

serious users should consider tuning the following parameters…

To tweak your very serious APC installation, you can change the settings manually (using vim):

$ sudo vim /etc/php5/conf.d/apc.ini

The APC configuration file is seriously barren; it is little more than an extension include directive. You can add extra keys after reading the APC’s online documentation related to settings.

Finally, there is a small php script that provides more information on the operation of the APC module. Copy it to your web root and decompress it. You should change the default username and password used to protect the script by changing the username and password variables directly in the PHP code:

$ sudo cp /usr/share/doc/php-apc/apc.php.gz /var/www
$ sudo gzip -d /var/www/apc.php.gz
$ sudo vim /var/www/apc.php

Change credentials near line 41:

41
42
defaults('ADMIN_USERNAME','apc'); // Admin Username
defaults('ADMIN_PASSWORD','password'); // Admin Password - CHANGE THIS TO ENABLE!!!

Now, view your APC page (assuming your web server is at 192.168.0.6):

http://192.168.0.6/apc.php

Tags: , , , , , , ,

Install PHP 5.3.3 on Mac OS Leopard 10.5.8

My attempt at building and installing PHP 5.3.3 crashed and burned with the following error while building the MySQLi extension:

/bin/sh /Users/kelvin/phpsource/php-5.3.3/libtool --silent --preserve-dup-deps --mode=compile gcc  -Iext/mysqli/ -I/Users/kelvin/phpsource/php-5.3.3/ext/mysqli/ -DPHP_ATOM_INC -I/Users/kelvin/phpsource/php-5.3.3/include -I/Users/kelvin/phpsource/php-5.3.3/main -I/Users/kelvin/phpsource/php-5.3.3 -I/Users/kelvin/phpsource/php-5.3.3/ext/date/lib -I/Users/kelvin/phpsource/php-5.3.3/ext/ereg/regex -I/usr/local/php5/include/libxml2 -I/usr/local/php5/include -I/opt/local/include -I/usr/local/php5/include/freetype2 -I/Users/kelvin/phpsource/php-5.3.3/ext/mbstring/oniguruma -I/Users/kelvin/phpsource/php-5.3.3/ext/mbstring/libmbfl -I/Users/kelvin/phpsource/php-5.3.3/ext/mbstring/libmbfl/mbfl -I/usr/local/mysql/include/mysql -I/Users/kelvin/phpsource/php-5.3.3/ext/sqlite3/libsqlite -I/usr/local/pgsql/include -I/Users/kelvin/phpsource/php-5.3.3/TSRM -I/Users/kelvin/phpsource/php-5.3.3/Zend  -no-cpp-precomp  -I/usr/local/php5/include -g -O2 -fvisibility=hidden  -c /Users/kelvin/phpsource/php-5.3.3/ext/mysqli/mysqli.c -o ext/mysqli/mysqli.lo 
 In file included from /Users/kelvin/phpsource/php-5.3.3/ext/mysqli/php_mysqli_structs.h:57,
                  from /Users/kelvin/phpsource/php-5.3.3/ext/mysqli/mysqli.c:33:
 /usr/local/mysql/include/mysql/my_global.h:895: error: duplicate ‘unsigned’
 /usr/local/mysql/include/mysql/my_global.h:895: warning: useless type name in empty declaration
 make: *** [ext/mysqli/mysqli.lo] Error 1
Trinity:~ kelvin$

According to the PHP bug tracker, the problem is a bug that affects Snow Leopard and FreeBSD as well. The fix is pretty simple, either you can build PHP from the head of the trunk from Subversion, or you can replace php-5.3.3/ext/mysqli/php_mysqli_structs.h with this patched version or you can just open one file (php-5.3.3/ext/mysqli/php_mysqli_structs.h) in your text editor and fix it yourself:

Trinity:~ kelvin$ sudo vim /Users/kelvin/phpsource/php-5.3.3/ext/mysqli/php_mysqli_structs.h

Go to line 59 and insert the following:

59
60
61
#if defined(ulong) && !defined(HAVE_ULONG)
#define HAVE_ULONG
#endif

The patched section of the file will then read as follows:

46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
/*
  The libmysql headers (a PITA) also define it and there will be an warning.
  Undef it and later we might need to define it again.
*/
#ifdef HAVE_MBRLEN
#undef HAVE_MBRLEN
#define WE_HAD_MBRLEN
#endif
#ifdef HAVE_MBSTATE_T
#undef HAVE_MBSTATE_T
#define WE_HAD_MBSTATE_T
#endif
 
#if defined(ulong) && !defined(HAVE_ULONG)
#define HAVE_ULONG
#endif
 
#include <my_global.h>
 
#if !defined(HAVE_MBRLEN) && defined(WE_HAD_MBRLEN)
#define HAVE_MBRLEN 1
#endif
 
#if !defined(HAVE_MBSTATE_T) && defined(WE_HAD_MBSTATE_T)
#define HAVE_MBSTATE_T 1
#endif

The rest of the build and install was uneventful after patching php_mysqli_structs.h.

Tags: , ,

Python 2.7 on Dreamhost

Python 2.7 was released on 3 July 2010 and I wanted to use it on my Dreamhost account, but the usual installation method yields some warnings:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _tkinter           bsddb185
bz2                dl                 imageop         sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Three of these modules cannot be installed on Dreamhost’s 64-bit Debian servers anyway and one of them is an older version of a module I am going to install:

Now, if you don’t need any of those remaining modules, then you should be able to just complete the installation and be done with it. If you want all the modules that you can get, you are in for some extra building. This post does a good job of explaining the installation of Python 2.6; mine is based on it. Let’s put the files in the following directories:

Python 2.7
$HOME/local/Python-2.7
Berkeley DB 4.8
$HOME/local/BerkeleyDB.4.8
Other executables
$HOME/local/bin
Header files
$HOME/local/include
Libraries
$HOME/local/lib
Temporary artifacts
$HOME/temp

We’ll need to push these values into the UNIX environment by using the export tool under the default bash shell:

$ export LDFLAGS="-L$HOME/local/lib -L$HOME/local/BerkeleyDB.4.8/lib"
$ export CPPFLAGS="-I$HOME/local/include -I$HOME/local/BerkeleyDB.4.8/include"
$ export CXXFLAGS=$CPPFLAGS
$ export CFLAGS=$CPPFLAGS
$ export LD_LIBRARY_PATH=$HOME/local/lib:$HOME/local/BerkeleyDB.4.8/lib
$ export LD_RUN_PATH=$LD_LIBRARY_PATH

Next make the directories:

$ mkdir ~/temp ~/local

It’s also a good idea to check your machine (note the “x86_64” token). It should look similar to this:

$ uname -a
Linux machine 2.6.32.8-grsec-2.1.14-modsign-xeon-64 #2 SMP Sat Mar 13 00:42:43 PST 2010 x86_64 GNU/Linux
$ gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.3.2-1.1' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-cld --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.3.2 (Debian 4.3.2-1.1) 
$

Berkeley DB will be hooked into the installed TCL, so you should install TCL before you install Berkeley DB. Install Python last. Other than that, I don’t believe that order of installation should matter. Let’s do it!

TCL/TK

The warning “_tkinter” indicates that the _tkinter module was not built. You will need to build both TCL and TK:

$ cd ~/temp
$ pwd
/home/username/temp
$ wget http://prdownloads.sourceforge.net/tcl/tcl8.5.8-src.tar.gz
$ tar zxvf tcl8.5.8-src.tar.gz
$ cd tcl8.5.8/unix
$ ./configure --prefix=$HOME/local
$ make
$ make install
$ cd ../..
$ wget http://prdownloads.sourceforge.net/tcl/tk8.5.8-src.tar.gz
$ tar zxvf tk8.5.8-src.tar.gz
$ cd tk8.5.8/unix
$ ./configure --prefix=$HOME/local
$ make
$ make install
$ cd ../..

Berkeley DB 4.8

The warning “_bsddb” will go away when you install version 4.8 of the Oracle Berkeley DB:

$ cd ~/temp
$ wget http://download.oracle.com/berkeley-db/db-4.8.30.tar.gz
$ tar zxvf db-4.8.30.tar.gz
$ cd db-4.8.30/build_unix
$ ../dist/configure --prefix=$HOME/local/BerkeleyDB.4.8 --enable-tcl --with-tcl=$HOME/local/lib
$ make
$ make install
$ cd ../..

BZip2

Dreamhost has an earlier version of BZip2 (version 1.0.4) and no library (at least I couldn’t find one). To get the latest version:

$ cd ~/temp
$ wget http://www.bzip.org/1.0.5/bzip2-1.0.5.tar.gz
$ tar zxvf bzip2-1.0.5.tar.gz
$ cd bzip2-1.0.5
$ make -f Makefile-libbz2_so
$ make
$ make install PREFIX=$HOME/local
$ cp ./libbz2.so.1.0.4 $HOME/local/lib
$ ln -s $HOME/local/lib/libbz2.so.1.0.4 $HOME/local/lib/libbz2.so.1.0
$ cd ..

Python 2.7

You should be able to build and install Python 2.7 now, less the modules that either cannot be built on a 64-bit platform or cannot coexist with a contemporary version.

$ cd ~/temp
$ wget http://python.org/ftp/python/2.7/Python-2.7.tgz
$ tar zxvf Python-2.7.tgz
$ cd Python-2.7
$ ./configure --prefix=$HOME/local/Python-2.7
$ make
$ make install

At the end of the make process, you will see this:

Python build finished, but the necessary bits to build these modules were not found:
bsddb185           dl                 imageop         
sunaudiodev                                       
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

As mentioned earlier, these modules are either not buildable on Dreamhost’s 64-bit machines or not compatible with the newer version of Berkeley DB.

Hooking up the new Python

You want to put the new Python 2.7 on your PATH so that bash executes it before the systemwide Python 2.5. If you don’t want to append these export statements, you can also do it via a text editor (vim, emacs, etc).

$ cd ~
$ echo "export PATH=\"$HOME/local/bin:\$PATH\"" >> .bashrc
$ echo "export PATH=\"$HOME/local/Python-2.7/bin:\$PATH\"" >> .bashrc
$ source .bashrc
$ which python
/home/username/local/Python-2.7/bin/python
$ python -V
Python 2.7

Try it out!

Try out your new modules…at the shell prompt type “python”!

>>> import bsddb
>>> db = bsddb.btopen('/tmp/spam.db', 'c')
>>> for i in range(10): db['%d'%i] = '%d'% (i*i)
...
>>> db['3']
'9'
>>> db.keys()
['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
>>> db.sync()
>>> db.isOpen()
True
>>> import bz2
>>> print bz2.__author__
The bz2 python module was written by:
 
    Gustavo Niemeyer <niemeyer@conectiva.com>
 
>>> import Tkinter
>>> Tkinter.__version__
'$Revision: 81008 $'
>>>

Tags: ,