DBFS was introduced in Oracle 11.2. It enables the database to be used as a POSIX-compatible file system. It is based on the SecureFile technology which was introduced in Oracle 11.1 to support LOB functionality. Consequently DBFS is documented in the Oracle Database SecureFiles and Large Objects Developer’s Guide. At the time of writing DBFS only appears to be supported on Linux (both 32-bit and 64-bit). As a Solaris option is now available with Exadata, presumably DBFS will be extended to support this operating system at some point in the future.

Prerequisites

DBFS is based on the FUSE library infrastructure. FUSE (Filesystem in USErspace) is an open-source project that supports the implementation of a fully functional file-system in a user-space program. More information can be found about the FUSE library on its project page (http://fuse.sourceforge.net). FUSE is delivered as a set of source files which must be compiled and linked on the target libraries. FUSE runs as a dynamic kernel module; it is not necessary to recompile the kernel during installation.

Prior to compiling the FUSE library, the Linux kernel development RPM must be installed. This package and its dependencies may already be installed. To verify use:

$ rpm -q kernel-devel
kernel-devel-2.6.18-92.el5

If this package does not exist it can be installed from the distribution media or using yum.

FUSE can be downloaded from http://fuse.sourceforge.net and delivered as a zipped archive called fuse-2.7.3.tar.gz. I copied this to a new directory owned by the root user called /root/fuse. Make sure you download version 2.7.3; subsequent versions will not work with the Oracle client.

As the root user unzip the archive using

[root@server1]# tar -xzvf fuse-2.7.3.tar.gz

This will create a subdirectory, in this case called /root/fuse/fuse-2.7.3.

Before building the FUSE library determine the kernel directory for the current kernel. On Red Hat and Oracle Enterprise Linux this directory will have the format:

/usr/src/kernels/`uname -r`-`uname -p`

For example:

/usr/src/kernels/2.6.18-92.el5-i686

Change to the FUSE source directory and run the configure script specifying the kernel directory for the with-kernel parameter:

[root@server1]# cd /root/fuse/fuse-2.7.3
[root@server1]# ./configure --prefix=/usr --with-kernel=/usr/src/kernels/2.6.18-92.el5-i686

Build and install the library

[root@server1]# make
[root@server1]# make install

The documentation recommends performing the following steps to complete the installation. I am not entirely convinced that any of these steps are necessary when the previous steps have been performed by the root user. However they all appear to be harmless.

[root@server1]# /sbin/depmod
[root@server1]# /sbin/modprobe fuse
[root@server1]# chmod 666 /dev/fuse
[root@server1]# echo "/sbin/modprobe fuse" >> /etc/rc.modules

Creating a DBFS file system

It is necessary to create a tablespace for the DBFS file system. For a simple test the new tablespace does not need to be very large. For example:

CREATE TABLESPACE dbfs1
DATAFILE '/u01/oradata/TEST/dbfs01.dbf' SIZE 100M;

I initially attempted to create a DBFS file system using the SYS (AS SYSDBA) user. However, the create script failed when SYS attempted to GRANT privileges to itself so I switched to a non-default user called US01. In order to manage DBFS the user must be assigned the DBFS_ROLE. For example:

[oracle@server1]$ sqlplus / as sysdba
SQL> GRANT dbfs_role TO us01

To create a file system use the dbms_create_filesystem.sql script which is located in $ORACLE_HOME/rdbms/admin. This script calls dbms_create_filesystem_advanced.sql which is located in the same directory. In my version, SQL*Plus was unable to locate the latter script unless it was in the current working directory. Therefore I had to change directory to $ORACLE_HOME/rdbms/admin before executing SQL*Plus. For example:

[oracle@server1]$ cd $ORACLE_HOME/rdbms/admin
[oracle@server1]$ sqlplus us01/us01
SQL> @dbfs_create_filesystem.sql DBFS1 dbfs1

The dbfs_create_filesystem.sql script takes two parameters; the tablespace name (DBFS1) and the file system name (dbfs1). In retrospect I should have assigned different names to the tablespace and file system.

Mounting a DBFS file system

Now the new file system exists within the database. However, we cannot access it externally yet. A new utility called dbms_client which is shipped in $ORACLE_HOME/bin in Oracle 11.2.0.1 and above.allows the DBFS file system to be mounted and managed from the command line. The 11.2.0.1 version of this utility is quite basic and it is easy to imagine customer feedback will drive improvements in functionality in subsequent releases.

As the root user create a mount point. In this case the mount point will be called /mnt/dbfs1 and will be owned by the oracle user:

[root@server1]# mkdir /mnt/dbfs1
[root@server1]# chown oracle:dba /mnt/dbfs1

In Oracle 11.2.0.1, unlike almost all other tools and utilities in $ORACLE_HOME/bin the dbms_client utility does not have a wrapper script. On my server invoking dbfs_client failed with the following error:

[oracle@server1]$ cd $ORACLE_HOME/bin
[oracle@server1]$ ./dbfs_client
./dbfs_client: error while loading shared libraries: libclntsh.so.11.1: cannot open shared object file: No such file or directory

This error occurs because the LD_LIBRARY_PATH environment variable has not been set correctly. This is an oversight, but one that is likely to be increasingly common as this environment variable is normally set by the other wrapper scripts. To fix the error set LD_LIBRARY_PATH, preferably in .bash_profile. For example:

export LD_LIBRARY_PATH=$ORACLE_HOME/lib

The client should now start and display the following help message:

[oracle@server1]$ cd $ORACLE_HOME/bin
[oracle@server1]$ ./dbfs_client
usage: ./dbfs_client @ [options]
  db_user:              Name of Database user that owns DBFS content repository filesystem(s)
  db_server:            A valid connect string for Oracle database server
                        (for example, hrdb_host:1521/hrservice)
  mountpoint:           Path to mount Database File System(s)
                        All the file systems owned by the database user will be seen at the mountpoint.
DBFS options:
  -o direct_io          Bypass the Linux page cache. Gives much better performance for large files.
                        Programs in the file system cannot be executed with this option.
                        This option is recommended when DBFS is used as an ETL staging area.
  -o wallet             Run ./dbfs_client in background.
                        Wallet must be configured to get credentials.
  -o failover           ./dbfs_client fails over to surviving database instance with no data loss.
                        Some performance cost on writes, especially for small files.
  -o allow_root         Allows root access to the filesystem.
                        This option requires setting 'user_allow_other' parameter in '/etc/fuse.conf'.
  -o allow_other        Allows other users access to the file system.
                        This option requires setting 'user_allow_other' parameter in '/etc/fuse.conf'.
  -o rw                 Mount the filesystem read-write. [Default]
  -o ro                 Mount the filesystem read-only. Files cannot be modified.
  -o trace_file=STR     Tracing  | 'syslog'
  -o trace_level=N      Trace Level: 1->DEBUG, 2->INFO, 3->WARNING, 4->ERROR, 5->CRITICAL [Default: 4]
  -h                    help
  -V                    version

Note that in Oracle 11.2.0.1 the above usage message omits the –command option.

This version of the dbfs_client utility does not support the bequeath protocol, so on the server it is not possible to identify the target instance using the $ORACLE_SID environment variable. It is therefore necessary to ensure that a TNS address is specified in $TNS_ADMIN/tnsnames.ora.

TEST=
(DESCRIPTION=
  (ADDRESS=
    (PROTOCOL=TCP)(HOST=vm7)(PORT=1521)
  )
  (CONNECT_DATA=(SERVICE_NAME=TEST))
)

It is also possible to specify a database service and DBFS does support instance failover though I have not yet tested it in a RAC environment.

To mount a DBFS file system use the following syntax

dbfs_client @ [options]

For example:

[oracle@server1]$ cd $ORACLE_HOME/bin
[oracle@server1]$ ./dbfs_client us01@TEST /mnt/dbfs1

A password will be requested. In this case it is the Oracle database password for the US01 user.

By default dbfs_client will block until the file system is un-mounted.

In this version it is possible to workaround the blocking of dbfs_client using the nohup command to run the utility in the background. For example:

[oracle@server1]$ cd $ORACLE_HOME/bin
[oracle@server1]$ nohup ./dbfs_client us01@TEST /mnt/dbfs1 &

In another shell session verify that the file system has been mounted using the mount command. For example:

[oracle@server1]$ mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdb1 on /u01/oradata type ext3 (rw)
/dev/sdc1 on /home/oracle/stage type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
dbfs on /mnt/dbfs1 type fuse (rw,nosuid,nodev,max_read=1048576,default_permissions,user=oracle)

At this point it is possible to navigate into the new file system from the Linux shell using commands such as pwd, cd, mkdir, cp, rm etc.

For example:

[oracle@server1]$ cd /mnt/dbfs1/dbfs1
[oracle@server1]$ pwd
/mnt/dbfs1/dbfs1
[oracle@server1]$ mkdir dir1
[oracle@server1]$ cd dir1
[oracle@server1]$ pwd
/mnt/dbfs1/dbfs1/dir1
[oracle@server1]$ cp /tmp/gp/*.csv /mnt/dbfs1/dbfs1/dir1
[oracle@server1]$ ls -l
total 42
-rw-r--r-- 1 oracle oinstall  1129 Jul  7 19:33 circuit.csv
-rw-r--r-- 1 oracle oinstall   161 Jul  7 12:02 classification.csv
-rw-r--r-- 1 oracle oinstall   535 Jul  7 12:02 country.csv
-rw-r--r-- 1 oracle oinstall 12762 Jul  7 12:02 driver.csv
-rw-r--r-- 1 oracle oinstall   708 Jul  7 12:02 engine.csv
-rw-r--r-- 1 oracle oinstall   506 Jul  7 12:02 grandprix.csv
-rw-r--r-- 1 oracle oinstall 22385 Jul  7 12:02 race.csv
-rw-r--r-- 1 oracle oinstall   550 Jul  7 12:02 season.csv
-rw-r--r-- 1 oracle oinstall  1487 Jul  7 12:02 team.csv

It is possible to copy files in and out of the DBFS using standard operating system commands.

Unmounting a DBFS filesystem

To unmount the DBFS file system use the fusemount command. For example:

[oracle@server1]$ fusermount -u /mnt/dbfs1

Creating an Oracle Wallet

It is also possible to include the filesystem in /etc/fstab so that it can be mounted using the mount command. In order to ensure that the mount command is executed with the correct Oracle credentials, an Oracle wallet must be created. Note that use of Oracle Wallets may require licensing of the Oracle Advanced Security option. There are several steps to this process:

As the oracle user create a subdirectory for the wallet e.g.:

[oracle@server1]$ mkdir /home/oracle/wallet

As the oracle user create the wallet.

[oracle@server1]$ mkstore -wrl /home/oracle/wallet -create
Oracle Secret Store Tool : Version 11.2.0.1.0 - Production
Copyright (c) 2004, 2009, Oracle and/or its affiliates. All rights reserved.

Enter password:
Enter password again:

This creates the following files in /home/oracle/wallet:

[oracle@server1]$ ls -l /home/oracle/wallet
-rw------- 1 oracle oinstall 3589 Jul  7 11:11 cwallet.sso
-rw------- 1 oracle oinstall 3512 Jul  7 11:11 ewallet.p12

Edit $TNS_ADMIN/sqlnet.ora and add the following entries:

WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/home/oracle/wallet)))

SQLNET.WALLET_OVERRIDE=TRUE

Create the DBConnectString credential in the wallet specifying the Oracle username and password that should be associated with this credential. In this case both the username and password are “us01″.

[oracle@server1]$ mkstore -wrl /home/oracle/wallet -createCredential DBConnectString us01 us01
Oracle Secret Store Tool : Version 11.2.0.1.0 - Production
Copyright (c) 2004, 2009, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:  << Oracle11gR2 >> 

Create credential oracle.security.client.connect_string1

Following creation of this credential it should be possible to mount the new DBFS file system using the wallet. For example:

[oracle@server1]$ dbfs_client -o wallet /@DBConnectString /mnt/dbfs1

Note that the “/@” syntax specifies that the connection should obtain its credentials from the Oracle wallet.

In Oracle 11.2.0.1, the above command mounts the DBFS file system and returns to the command line without blocking.

Configuring the mount command

A few more steps are required to enable the mount command to use an entry in /etc/fstab.

The 11.2 documentation recommends that a symbolic link should be created between $ORACLE_HOME/bin/dbfs_client and /sbin/mount.dbfs. In my opinion the recommended syntax has been reversed. I believe that the following is correct:

[root@server1]# ln -s $ORACLE_HOME/bin/dbfs_client /sbin/mount.dbfs

However, I would not recommend using a symbolic link. Although I modified the bash profile for my root user to include $LD_LIBRARY_PATH environment variable, the mount command appear to unset this variable when it called the mount.dbfs helper. Therefore it may be more appropriate to create /sbin/mount.dbfs as a script containing commands similar to the following:

#!/bin/bash
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
nohup $ORACLE_HOME/bin/dbfs_client $* 2> /dev/null &

A new operating system is required for fuse. For example:

[root@server1]# /usr/sbin/groupadd fuse

The user that will be mounting the DBFS file system should be added to the fuse group. I used the oracle user which on my server already belonged to primary group oinstall (specified in /etc/passwd) and secondary group dba (specified in /etc/group). I used the following command:

[root@server1]# usermod -G dba,fuse oracle

The above command adds the oracle user to both the dba and fuse groups. Take care when specifying this command – it requires a list of all secondary groups so in this example if only the fuse group is specified (usermod -G fuse oracle) then the oracle user will be removed from the dba group. The changes should be written to /etc/group.

Finally the following line should be added to /etc/fstab:

/sbin/mount.dbfs#/@DBConnectString /mnt/dbfs1 fuse rw,user,noauto 0 0

Note the use of the Oracle wallet credential.

If the above configuration has been completed successfully it should be possible to mount the new DBFS file system using the mount command. For example:

[root@server1]# mount /mnt/dbfs1

The command will mount the file system and then block until the file system is unmounted. Again the nohup command can be used to ensure the mount command runs as a background process:

[root@server1]# nohup mount /mnt/dbfs1 &

I remain unconvinced that enabling use of the mount command for the DBFS file system justifies the amount of configuration required. It is not possible to configure DBFS to be auto-mounted during a system boot as this is not supported by the current version of FUSE. There is also a timing issue – the database must already be started before the DBFS file system can be mounted. In view of potential uses for DBFS (migration and data warehouse loading) it is probably sufficient to mount the DBFS file systems manually when required.

Executing Commands

In addition to the shell interface it is also possible to execute commands using dbfs_client. Whilst this is unnecessary when connected directly to the database server, it could be useful when the dbfs_client utility is being executed on another client.

Unfortunately in Oracle 11.2.0.1 dbfs_client does not appear to be sufficiently mature to accept options and to execute commands in the same invocation. This means that it is not possible to execute an operating system whilst using a wallet. Consequently it is necessary to manually enter a password for each execution of dbfs_client which restricts scripting options.

The following are examples of dbfs_client operating system commands:

$ dbfs_client us01@TEST --command mkdir dbfs:/dbfs1/dir1
$ dbfs_client us01@TEST --command cp /home/oracle/gp dbfs:/dbfs1/dir1
$ dbfs_client us01@TEST --command ls -l dbfs:/dbfs1/dir1
-rw-r--r--            oracle        oinstall          725859    Jul 07 19:33    dbfs:/dbfs1/dir1/car.csv
-rw-r--r--            oracle        oinstall            1129    Jul 07 19:33    dbfs:/dbfs1/dir1/circuit.csv
-rw-r--r--            oracle        oinstall             708    Jul 07 12:02    dbfs:/dbfs1/dir1/engine.csv
-rw-r--r--            oracle        oinstall             550    Jul 07 12:02    dbfs:/dbfs1/dir1/season.csv
-rw-r--r--            oracle        oinstall             161    Jul 07 12:02    dbfs:/dbfs1/dir1/classification.csv
-rw-r--r--            oracle        oinstall             506    Jul 07 12:02    dbfs:/dbfs1/dir1/grandprix.csv
-rw-r--r--            oracle        oinstall           22385    Jul 07 12:02    dbfs:/dbfs1/dir1/race.csv
-rw-r--r--            oracle        oinstall           12762    Jul 07 12:02    dbfs:/dbfs1/dir1/driver.csv
-rw-r--r--            oracle        oinstall             535    Jul 07 12:02    dbfs:/dbfs1/dir1/country.csv
-rw-r--r--            oracle        oinstall            1487    Jul 07 12:02    dbfs:/dbfs1/dir1/team.csv
$ dbfs_client us01@TEST --command rm dbfs:/dbfs1/dir1/car.csv

When copying large files to the DBFS file system, -o directio may be an attractive dbfs_client option as it bypasses the operating system page cache.

Conclusion

DBFS is relatively easy to set up and manage. In Oracle 11.2.0.1 there are still plenty of niggling design and usability issues. However I am confident that these issues will be rapidly resolved as DBFS appears to be a fundamental component of the Exadata strategy and should therefore experience accelerated development for a few releases.

It is important to understand the differences between DBFS and ACFS. Note that in Oracle 11.2.0.2 a separate licence is required for ACFS on some platforms.

This post was researched using Perl v5.10 with Oracle 11.2.0.2 Express Edition on Windows XP 32-bit. The examples are based on a table listing drivers in the 2011 Formula One World Championship.

The example table is defined as follows:

CREATE TABLE driver
(
  key NUMBER,
  name VARCHAR2(30),
  team VARCHAR2(30),
  points NUMBER
);

The following rows were inserted into the table (based on championship standings after 17 of the 19 races:

INSERT INTO driver VALUES (1,'Sebastian Vettel','Red Bull',374);
INSERT INTO driver VALUES (2,'Jenson Button','McLaren',240);
INSERT INTO driver VALUES (3,'Fernando Alonso','Ferrari',227);
INSERT INTO driver VALUES (4,'Mark Webber','Red Bull',221);
COMMIT;

Single Row Select

The first script counts the number of rows in the DRIVER table. Although trivial this script demonstrates the basic components of all Perl DBI scripts.

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql = "SELECT COUNT(*) FROM driver";

($count) = $dbh->selectrow_array ($sql);

printf ("Count = %d\n",$count);

$dbh->disconnect;

The use DBI pragma instructs Perl to load the Database Interface (DBI) module. I normally use the strict pragma, but have omitted it within this post to improve clarity.

A connection must be created before any statements are executed. A successful connection returns what is referred to as a database handle. Most Perl scripts seem to use $dbh for the database handle.

In this example the database SID is XE and therefore the connect string is DBI:Oracle:XE. In my highly unsecure database both the username and password are us01.

I like to define the SQL statement using a separate variable ($sql). This is not necessary, but in my opinion results in clearer and more supportable code as the length and complexity of the SQL statements increases.

In this example the query can only return one row. I have used the selectrow_array subroutine to execute the query and to fetch the result. The row is returned into the $count variable.

Finally the script disconnects from the database.

The script should return the following output:

Count = 4

Multi-Row Select

This example demonstrates how to select multiple rows from a cursor.

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql =
  "SELECT name,team,points ".
  "FROM driver ".
  "ORDER BY points DESC";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->execute;

while (@data = $sth->fetchrow_array())
{
  $name = $data[0];
  $team = $data[1];
  $points = $data[2];
  printf ("%-20s %-20s %3d\n",$name,$team,$points);
}

$sth->finish;

$dbh->disconnect;

In this example the SQL statement is parsed using the prepare subroutine. This returns a statement handle ($sth) which will be used to refer to the statement throughout the rest of the script.

This query does not have any bind variables, so it can be executed using the execute subroutine. Rows can then be fetched one at a time using the fetchrow_array subroutine which returns an array of results. In this case I have assigned each array element to a separate variable to demonstrate that the array is zero-based.
When all required rows have been fetched from the cursor the statement can be closed using the finish subroutine.

This script generates the following output:

Sebastian Vettel Red Bull 374
Jenson Button McLaren 240
Fernando Alonso Ferrari 227
Mark Webber Red Bull 221

The following script demonstrates a slightly different way of coding the fetch:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql =
  "SELECT name,team,points ".
  "FROM driver ".
  "ORDER BY points DESC";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->execute;

while (($name,$team,$points) = $sth->fetchrow_array())
{
  printf ("%-20s %-20s %3d\n",$name,$team,$points);
}

$sth->finish;

$dbh->disconnect;

In the above example each row is fetched directly into a list of variables.

The second script generates the same output:

Sebastian Vettel Red Bull 374
Jenson Button McLaren 240
Fernando Alonso Ferrari 227
Mark Webber Red Bull 221

Both the above methods are functionally equivalent and the choice of one or the other is simply a matter of personal taste.

However, there is a third way to achieve the same results. This method, however, is slightly different:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql =
  "SELECT name,team,points ".
  "FROM driver ".
  "ORDER BY points DESC";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->execute;

$array = $sth->fetchall_arrayref ();

foreach $row (@$array)
{
  ($name,$team,$points) = @$row;
  printf ("%-20s %-20s %3d\n",$name,$team,$points);
}

$sth->finish;

$dbh->disconnect;

In the above example, the fetchall_arrayref call fetches all rows into $array. This array can subsequently be processed without any further communication with the database. In some circumstances this technique could be very efficient – for example returning the top ten results. It could be less efficient in other circumstances particularly if a large number of rows may be returned or the later rows are likely to be discarded.

The third script generates the same output as its predecessors:

Sebastian Vettel Red Bull 374
Jenson Button McLaren 240
Fernando Alonso Ferrari 227
Mark Webber Red Bull 221

Bind Variables

We will normally want to use bind variables with SQL queries. The following example uses a bind variable to select the rows for a single team – in this case Red Bull:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql =
  "SELECT name,team,points ".
  "FROM driver ".
  "WHERE team = ?";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->bind_param (1,"Red Bull");

$sth->execute;

while (@data = $sth->fetchrow_array())
{
  $name = $data[0];
  $team = $data[1];
  $points = $data[2];
  printf ("%-20s %-20s %3d\n",$name,$team,$points);
}

$sth->finish;

$dbh->disconnect;

In Perl bind variables are represented by a “?” in the SQL statement text. Each bind variable should have an associated bind_param call. Bind parameters are numbered from 1 upwards.

You can avoid calls to bind_param by listing bind variables in the execute call:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql =
  "SELECT name,team,points ".
  "FROM driver ".
  "WHERE team = ?";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->execute ("Red Bull");

while (($name,$team,$points) = $sth->fetchrow_array())
{
  printf ("%-20s %-20s %3d\n",$name,$team,$points);
}

$sth->finish;

$dbh->disconnect;

In the above example the bind variable value (“Red Bull”) is specified as the first parameter in the execute call. Any number of bind variables can be specified in this call.

INSERT statements

The following script is an example of a simple INSERT statement:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql = "INSERT INTO driver VALUES (5,'Lewis Hamilton','McLaren',202)";

if (!($dbh->do ($sql)))
{
  die ("Failed to insert row: " . DBI->errstr);
};

$dbh->disconnect;

In the above script literal values have been specified for each column value.

The do subroutine allows us to execute a simple statement without the need to explicitly initialize a statement context.

Bind variables can also be specified. For example:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql = "INSERT INTO driver VALUES (?,?,?,?)";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->execute (6,"Felipe Massa","Ferrari",98);

$dbh->disconnect;

In the above example the bind variables are listed as parameters in the execute call.

It is also possible to set bind variables using the bind_param call. For example:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql = "INSERT INTO driver VALUES (?,?,?,?)";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->bind_param (1,7);
$sth->bind_param (2,'Nico Rosberg');
$sth->bind_param (3,'Mercedes');
$sth->bind_param (4,75);

$sth->execute;

$dbh->disconnect;

Note that bind parameters are numbered from 1 upwards.

In this example the primary key column could be generated automatically using a sequence. We can create the sequence as follows:

CREATE SEQUENCE driver_sequence START WITH 8;

The sequence starts at 8 as the table already contains 7 rows.

We can then create an INSERT trigger on the DRIVER table as follows:

CREATE OR REPLACE TRIGGER driver_trigger
BEFORE INSERT ON driver
FOR EACH ROW
DECLARE
  l_key NUMBER;
BEGIN
  SELECT driver_sequence.NEXTVAL INTO l_key FROM dual;
  :new.key := l_key;
END;
/

When we insert rows into the DRIVER table we no longer need to specify a key as this will be generated by the trigger.

However we might wish to know the value of the internally generated key. In this case we can use the RETURNING clause with the INSERT statement. For example:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql =
  "INSERT INTO driver (name,team,points) ".
  "VALUES (?,?,?) ".
  "RETURNING key INTO ?";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$key = 0;

$sth->bind_param (1,'Michael Schumacher');
$sth->bind_param (2,'Mercedes');
$sth->bind_param (3,70);
$sth->bind_param_inout (4,\$key,SQL_NUMERIC);

$sth->execute;

printf ("Key = %d\n",$key);

$dbh->disconnect;

Note that the RETURNING clause effectively declares a fourth bind variable. This variable differs from the others as it accepts a return value and is therefore declared as an INOUT parameter. Note that the $key parameter is passed by reference (not value) in the bind_param_inout call. and is therefore preceded by a backslash.

The above query returns the following output:

Key = 8

UPDATE Statements

As with INSERT statements the simplest form of an UPDATE statement uses literal values and the do subroutine.

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql = "UPDATE driver SET points = 399 WHERE key = 1";

if (!($dbh->do ($sql)))
{
  die ("Failed to update row: " . DBI->errstr);
};

$dbh->disconnect;

The above example adds 25 points to Sebastian Vettel’s current score of 374 points.

Alternatively we could use bind variables to achieve the same result. For example:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql = "UPDATE driver SET points = ? WHERE key = ?";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->execute (399,1);

$dbh->disconnect;

In this example the bind variable values are supplied as arguments to the execute call.

We can also use bind_param calls to achieve the same objective:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql =
  "UPDATE driver ".
  "SET points = points + ? ".
  "WHERE key = ?";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->bind_param (1,25);
$sth->bind_param (2,1);

$sth->execute;

$dbh->disconnect;

In the above example bind parameters have been used to specify the number of points and also the primary key of the row to be updated.

Assume we want to know how many points Vettel has now scored. To avoid selecting the row we can specify the RETURNING clause to the UPDATE statement

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql =
  "UPDATE driver ".
  "SET points = points + ? ".
  "WHERE key = ? ".
  "RETURNING points INTO ?";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$points = 0;

$sth->bind_param (1,25);
$sth->bind_param (2,1);
$sth->bind_param_inout (3,\$points,SQL_NUMERIC);

$sth->execute;

printf ("Value = %d\n",$points);

$dbh->disconnect;

In the above example a call to bind_param_inout has been added to return the total number of points following execution of the UPDATE statement. In this case the script generates the following output:

Value = 399

DELETE statements

DELETE statements can use literal values or bind variables. For example:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$sql =
  "DELETE FROM driver ".
  "WHERE key = ?";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->bind_param (1,8);

$sth->execute;

$dbh->disconnect;

In the above example the bind_param call is used to set the bind variable value to 8.

Transactions

The above examples do not include commit or rollback calls. This is because, by default, Perl DBI enables automatic commits. Few Oracle users will desire this behaviour. To disable automatic commits set the AutoCommit variable in the connection handle to 0. For example:

$dbh->{AutoCommit} = 0;

When automatic commits have been disabled, to commit a transaction use

$dbh->commit;

To rollback the transaction use:

$dbh->rollback;

The following script disables automatic commits and then explicitly specifies a commit at the end of the transaction:

use DBI;

if (!($dbh = DBI->connect ('DBI:Oracle:XE',"us01","us01")))
{
  die ("Failed to connect to database: " . DBI->errstr);
};

$dbh->{AutoCommit} = 0;

$sql = "INSERT INTO driver (name,team,points) VALUES (?,?,?)";

if (!($sth = $dbh->prepare ($sql)))
{
  die ("Failed to prepare statement: " . DBI->errstr);
};

$sth->bind_param (1,'Michael Schumacher');
$sth->bind_param (2,'Mercedes');
$sth->bind_param (3,70);

$sth->execute;

$dbh->commit;

$dbh->disconnect;

The AutoCommit parameter is set after the connection has been created and before the first DML statement.

The commit call is executed before the session is disconnected.

Therefore I decided to investigate the MAINTAIN_TABLES procedure which is also in the DBMS_STREAMS_ADM package. MAINTAIN_TABLE sets up Streams replication for one or more tables. I have used this procedure a few times in the past. I like it because it eliminates the need to code a series of fiddly steps including DataPump replication of the table from source to target database. However, Streams replication already existed for a number of tables and therefore any use of MAINTAIN_TABLES needed to complement this configuration.

Prerequisites

There are a number of pre-requisites that should be observed when using MAINTAIN_TABLES. Both source and target databases should have global names and database links using the global name should exist in both directions. I have been told in the past that the global names are not mandatory, just highly recommended. Also there should be a Streams administrative user (often STRMADMIN) in each database with appropriate privileges. Many customers create a dedicated tablespace for objects used by the Streams administrative user. If you are not using Automatic Memory Management then there are some minimum values for the STREAMS_POOL_SIZE parameter which should be configured.

Documentation

This site had an existing application running 10.2.0.4 and upgrading was not an option. The first thing I learned was that the MAINTAIN_TABLES API is not fully documented in Oracle 10.2. Although the procedure has more than 20 parameters, only five were mentioned at all in the documentation. In this situation I normally look at the most recent version of the manual only nine of the parameters are documented.

The poor documentation means a lot of guesswork is required when working with MAINTAIN_TABLES. The first question is whether MAINTAIN_TABLES can be used to add new tables incrementally to existing configurations or if it is just a single use procedure (like a wizard). Fortunately the answer is that it can be used incrementally without damaging the existing configuration. However extreme care needs to be taken if it is necessary to rollback the recoverable scripts as I noticed some steps were dropping existing captures, queues and propagations. Before using DBMS_RECOVERABLE_SCRIPT to rollback I therefore recommend you check each step carefully.

Table Names

Some care needs to be taken with the TABLE_NAMES parameter as the format is non-standard. The schema name must be included with the table name. For example:

'SCHEMA1.TABLE1'

My testing never extended to configuration of streams for a list of tables. I did read a couple of notes that suggested that MAINTAIN_TABLES was being used to configure Streams for tables in more than one schema, then it should be called individually for each schema – presumably a bug.

Capture, Propagation and Apply

If you already have a Streams configuration and you wish to use the existing capture, apply, propagation and underlying queues, you will need to do some research before invoking MAINTAIN_TABLES. If you do not explicitly specify any of these objects, MAINTAIN_TABLES will create new ones with system-defined names. MAINTAIN_TABLES is not great at working things out for itself; if you just specify a capture name it does try to figure out the capture queue table name etc, it just creates new ones again with system-defined names. So in Oracle 10.2 at least you need to tell it everything.

You can check the names of existing objects using:

SELECT capture_name FROM dba_capture;

SELECT propagation_name FROM dba_propagation;

SELECT apply_name FROM dba_apply;

SELECT owner, queue_table FROM dba_queue_tables;

SELECT owner,name FROM dba_queues;

If you make a mistake, you can drop any incorrectly created objects. However, this can be quite time-consuming as MAINTAIN_TABLES can create around 50 new objects even for a trivial table.

Capture, propagation and apply can be dropped using:

DBMS_CAPTURE_ADM.drop_capture
(
    capture_name => <capture_name>,
    drop_unused_rule_sets => TRUE
);

DBMS_PROPAGATION_ADM.drop_propagation
(
    propagation_name => <propagation_name>,
    drop_unused_rule_sets => TRUE
);

DBMS_APPLY_ADM.drop_apply
(
    apply_name => <apply_name>,
    drop_unused_rule_sets => TRUE
);

In each of the above examples I have set DROP_UNUSED_RULE_SETS to TRUE. MAINTAIN_TABLES creates a handful of rules for each object. Each rule is an individual data dictionary object and using the above parameter is the simplest way to clean up redundant ones.

Normally when a queue table is dropped, any queues supported by that table are also dropped. So the following can be used to drop a queue table and all of its queues:

DBMS_AQADM.DROP_QUEUE_TABLE
(
    queue_table => <queue_table_name>,
    force => TRUE
);

If you need to drop individual queues use:

DBMS_AQADM.DROP_QUEUE
(
    queue_name => <queue_name>
);

Instantiation

There are two ways to instantiate the table in the target database; using a Data Pump dump file and using Data Pump across the network. For large files you may want to use the dump file; for small or newly created tables the network option is easier.

The instantiation method is controlled by the INSTANTIATION parameter. To use the network set this parameter to DBMS_STREAMS_ADM.INSTANTIATION_TABLE_NETWORK. The default value is DBMS_STREAMS_ADM.INSTANTIATION_TABLE which performs a full Data Pump export on the source and a full Data Pump import on the target. Note that if the table already exists on the target then it will not be modified.

One issue that I hit with instantiation was with tablespace names. After MAINTAIN_TABLES failed to create a table in the target database, I discovered that a tablespace name had been explicitly specified in the CREATE TABLE statement. The only way around this was to create a new tablespace with the correct name in the target database.

Note that there is a PROPAGATION_NAME parameter, but there is no parameter for the propagation owner. In my example a propagation already existed for the Streams administrative user. However if I called MAINTAIN_TABLES for any other user it would create a new propagation together with the underlying objects that was owned by that user. From this I conclude that in most cases it probably only makes sense for the Streams administrative user to call MAINTAIN_TABLES.

Scripts

The PERFORM_ACTIONS parameter specifies whether or not MAINTAIN_TABLES should actually make changes to the database or just generate a script listing the actions it would perform. Beware the default value for this parameter is TRUE (implement changes).

If PERFORM_ACTIONS is set to FALSE then you should also specify a SCRIPT_NAME and a SCRIPT_DIRECTORY_OBJECT. The latter is just an Oracle directory object – I used EXPDP_DIR which already existed.

Experience now suggests that the best way of working with MAINTAIN_TABLES is to set PERFORM_ACTIONS=FALSE in order to generate a script which can be inspected manually. When you are happy with this script, run MAINTAIN_TABLES again with the same parameters and PERFORM_ACTIONS=TRUE. You could alternatively run the script manually. However you would not benefit from the recoverable scripts feature which does allow simple errors to be detected and resolved.

A typical call to MAINTAIN_TABLES to generate a script follows

BEGIN
  dbms_streams_adm.maintain_tables 
  (
    table_names => 'SCHEMA1.TABLE1',
    source_directory_object => NULL,
    destination_directory_object => NULL,
    source_database => 'EAST',
    destination_database => 'WEST',
    perform_actions => FALSE,
    script_name => 'table1.txt',
    script_directory_object => 'EXPDP_DIR',
    capture_name => 'CAPTURE1',
    capture_queue_table => 'CAPTURE_QUEUE_TABLE',
    capture_queue_name => 'CAPTURE_QUEUE',
    capture_queue_user => 'STRMADMIN',
    propagation_name => 'PROPAGATION1',
    apply_name => 'APPLY1',
    apply_queue_table => 'APPLY_QUEUE_TABLE',
    apply_queue_name => 'APPLY_QUEUE',
    apply_queue_user => 'STRMADMIN',
    include_ddl => TRUE,
    instantiation => DBMS_STREAMS_ADM.INSTANTIATION_TABLE_NETWORK
  );
END;
/

The script generated by MAINTAIN_TABLES is long, but highly instructive.

SET ECHO ON
SET VERIFY OFF
WHENEVER SQLERROR EXIT SQL.SQLCODE;
-------------------------------------------------------------------
-- get TNSNAME and streams admin user details for both the databases
--------------------------------------------------------------------
PROMPT
PROMPT 'Enter TNS Name of site 1 as parameter 1:'
DEFINE db1                 = &1
PROMPT
PROMPT 'Enter streams admin username for site 1 as parameter 2:'
DEFINE strm_adm_db1        = &2
PROMPT
PROMPT 'Enter streams admin password for site 1 as parameter 3:'
DEFINE strm_adm_pwd_db1    = &3
PROMPT
PROMPT 'Enter TNS Name of site 2 as parameter 4:'
DEFINE db2                 = &4
PROMPT
PROMPT 'Enter streams admin username for site 2 as parameter 5:'
DEFINE strm_adm_db2        = &5
PROMPT
PROMPT 'Enter streams admin password for site 2 as parameter 6:'
DEFINE strm_adm_pwd_db2    = &6
-- connect as streams administrator to site 1
PROMPT Connecting as streams administrator to site 1
CONNECT &strm_adm_db1/&strm_adm_pwd_db1@&db1
--
-- Add supplemental log group for table "SCHEMA1"."TABLE1"
--
BEGIN
  EXECUTE IMMEDIATE 'ALTER TABLE "SCHEMA1"."TABLE1" ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY, FOREIGN KEY, UNIQUE INDEX) COLUMNS';
EXCEPTION WHEN OTHERS THEN
  IF sqlcode = -32588 THEN NULL;  -- Logging attribute exists
  ELSE RAISE;
  END IF;
END;
/
--
-- Set up queue "STRMADMIN"."CAPTURE_QUEUE"
--
BEGIN
  dbms_streams_adm.set_up_queue(
    queue_table => '"STRMADMIN"."CAPTURE_QUEUE_TABLE"',
    storage_clause => NULL,
    queue_name => '"STRMADMIN"."CAPTURE_QUEUE"',
    queue_user => '"STRMADMIN"');
END;
/
--
-- PROPAGATE changes for table "SCHEMA1"."TABLE1"
--
DECLARE
  version_num            NUMBER := 0;
  release_num            NUMBER := 0;
  pos                    NUMBER;
  initpos                NUMBER;
  q2q                    BOOLEAN;
  stmt                   VARCHAR2(100);
  ver                    VARCHAR2(30);
  compat                 VARCHAR2(30); 
BEGIN
  BEGIN
    stmt := 'BEGIN dbms_utility.db_version@WEST(:ver , :compat); END;';
    EXECUTE IMMEDIATE stmt USING OUT ver, OUT compat;
    -- Extract version number
    initpos := 1;
    pos := INSTR(compat, '.', initpos, 1);
    IF pos > 0 THEN
      version_num := TO_NUMBER(SUBSTR(compat, initpos, pos - initpos));
      initpos := pos + 1;
      -- Extract release number
      pos := INSTR(compat, '.', initpos, 1);
      IF pos > 0 THEN
        release_num := TO_NUMBER(SUBSTR(compat, initpos,
                                   pos - initpos));
        initpos := pos + 1;
      ELSE
        release_num := TO_NUMBER(SUBSTR(compat, initpos));
      END IF;
    ELSE
      version_num := TO_NUMBER(SUBSTR(compat, initpos));
    END IF;
    -- use q2q propagation if compatibility >= 10.2
    IF version_num > 10 OR
       (version_num = 10 AND release_num >=2) THEN
      q2q := TRUE;
    ELSE
      q2q := FALSE;
    END IF;
  EXCEPTION WHEN OTHERS THEN
    q2q := FALSE;
  END;
  dbms_streams_adm.add_table_propagation_rules(
    table_name => '"SCHEMA1"."TABLE1"',
    streams_name => '"PROPAGATION1"',
    source_queue_name => '"STRMADMIN"."CAPTURE_QUEUE"',
    destination_queue_name => '"STRMADMIN"."APPLY_QUEUE"@WEST',
    include_dml => TRUE,
    include_ddl => TRUE,
    include_tagged_lcr => TRUE,
    source_database => 'EAST',
    inclusion_rule => TRUE,
    and_condition => NULL,
    queue_to_queue => q2q);
END;
/
--
-- Disable propagation. Enable after destination has been setup
--
DECLARE
  q2q       VARCHAR2(10);
  destn_q   VARCHAR2(65);
BEGIN
  SELECT queue_to_queue INTO q2q
  FROM dba_propagation
  WHERE source_queue_owner = 'STRMADMIN' AND
        source_queue_name = 'CAPTURE_QUEUE' AND
        destination_queue_owner = 'STRMADMIN' AND
        destination_queue_name = 'APPLY_QUEUE' AND
        destination_dblink = 'WEST';
  IF q2q = 'TRUE' THEN
    destn_q := '"STRMADMIN"."APPLY_QUEUE"';
  ELSE
    destn_q := NULL;
  END IF;
  dbms_aqadm.disable_propagation_schedule(
    queue_name => '"STRMADMIN"."CAPTURE_QUEUE"',
    destination => 'WEST',
    destination_queue => destn_q);
EXCEPTION WHEN OTHERS THEN
  IF sqlcode = -24065 THEN NULL;  -- propagation already disabled
  ELSE RAISE;
  END IF;
END;
/
--
-- CAPTURE changes for table "SCHEMA1"."TABLE1"
--
DECLARE
  compat         VARCHAR2(512);
  initpos        NUMBER;
  pos            NUMBER;
  version_num    NUMBER;
  release_num    NUMBER;
  compat_func    VARCHAR2(65);
  get_compatible VARCHAR2(4000);
BEGIN
  SELECT value INTO compat
  FROM v$parameter
  WHERE name = 'compatible';
  -- Extract version number
  initpos := 1;
  pos := INSTR(compat, '.', initpos, 1);
  IF pos > 0 THEN
    version_num := TO_NUMBER(SUBSTR(compat, initpos, pos - initpos));
    initpos := pos + 1;
    -- Extract release number
    pos := INSTR(compat, '.', initpos, 1);
    IF pos > 0 THEN
      release_num := TO_NUMBER(SUBSTR(compat, initpos, pos - initpos));
      initpos := pos + 1;
    ELSE
      release_num := TO_NUMBER(SUBSTR(compat, initpos));
    END IF;
  END IF;
  IF version_num < 10 THEN
    compat_func := 'dbms_streams.compatible_9_2';
  ELSIF version_num = 10 THEN
    IF release_num < 2 THEN
      compat_func := 'dbms_streams.compatible_10_1';
    ELSE
      compat_func := 'dbms_streams.compatible_10_2';
    END IF;
  ELSE
    compat_func := 'dbms_streams.compatible_10_2';
  END IF;
  get_compatible := ':lcr.get_compatible() <= '||compat_func;
  dbms_streams_adm.add_table_rules(
    table_name => '"SCHEMA1"."TABLE1"',
    streams_type => 'CAPTURE',
    streams_name => '"CAPTURE_STREAM"',
    queue_name => '"STRMADMIN"."CAPTURE_QUEUE"',
    include_dml => TRUE,
    include_ddl => TRUE,
    include_tagged_lcr => TRUE,
    source_database => 'EAST',
    inclusion_rule => TRUE,
    and_condition => get_compatible);
END;
/
--
-- Start capture process CAPTURE_STREAM
--
BEGIN
  dbms_capture_adm.start_capture(
    capture_name => '"CAPTURE_STREAM"');
EXCEPTION WHEN OTHERS THEN
  IF sqlcode = -26666 THEN NULL;  -- CAPTURE process already running
  ELSE RAISE;
  END IF;
END;
/
-- connect as streams administrator to site 2
PROMPT Connecting as streams administrator to site 2
CONNECT &strm_adm_db2/&strm_adm_pwd_db2@&db2
--
-- Datapump TABLE MODE IMPORT (NETWORK)
--
DECLARE
  h1                NUMBER;       -- data pump job handle
  name_expr_list    VARCHAR2(32767); -- for metadata_filter
  object_name       dbms_utility.uncl_array; -- object names
  cnt               NUMBER;
  object_owner      VARCHAR2(30); -- owner
  job_state         VARCHAR2(30); -- job state
  status            ku$_Status; -- data pump status
  job_not_exist     exception;
  pragma            exception_init(job_not_exist, -31626);
BEGIN
  object_name(1) := 'TABLE1';
  object_owner := 'SCHEMA1';
  FOR idx IN 1..1LOOP
    SELECT COUNT(1) INTO cnt FROM all_tables@WEST
      WHERE owner = object_owner AND table_name = object_name(idx);
    -- table does not exist locally, need instantiation
    IF cnt = 0 THEN
      IF name_expr_list IS NULL THEN
        name_expr_list := '(';
      ELSE
        name_expr_list := name_expr_list ||',';
      END IF;
      name_expr_list := name_expr_list||''''||object_name(idx)||'''';
    END IF;
  ENDLOOP;
  IF name_expr_list IS NOT NULL THEN
    name_expr_list := name_expr_list || ')';
  ELSE
    COMMIT;
    RETURN;
  END IF;
  h1 := dbms_datapump.open(operation=>'IMPORT',job_mode=>'TABLE',
    remote_link=>'EAST',
    job_name=>NULL, version=>'COMPATIBLE');
  dbms_datapump.metadata_filter(
    handle=>h1,
    name=>'NAME_EXPR',
    value=>'IN'||name_expr_list);
  dbms_datapump.metadata_filter(
    handle=>h1,
    name=>'SCHEMA_EXPR',
    value=>'IN'||'(''SCHEMA1'')');
  dbms_datapump.start_job(h1);
  job_state := 'UNDEFINED';
  BEGIN
    WHILE (job_state != 'COMPLETED') AND (job_state != 'STOPPED') LOOP
      status := dbms_datapump.get_status(
        handle => h1,
        mask => dbms_datapump.ku$_status_job_error +
                dbms_datapump.ku$_status_job_status +
                dbms_datapump.ku$_status_wip,
        timeout => -1);
      job_state := status.job_status.state;
      dbms_lock.sleep(10);
    ENDLOOP;
  EXCEPTION WHEN job_not_exist THEN
    dbms_output.put_line('job finished');
  END;
  COMMIT;
EXCEPTION WHEN OTHERS THEN
  ROLLBACK;
  RAISE;
END;
/
--
-- Set up queue "STRMADMIN"."APPLY_QUEUE"
--
BEGIN
  dbms_streams_adm.set_up_queue(
    queue_table => '"STRMADMIN"."APPLY_QUEUE_TABLE"',
    storage_clause => NULL,
    queue_name => '"STRMADMIN"."APPLY_QUEUE"',
    queue_user => '"STRMADMIN"');
END;
/
--
-- APPLY changes for table "SCHEMA1"."TABLE1"
--
DECLARE
  compat         VARCHAR2(512);
  initpos        NUMBER;
  pos            NUMBER;
  version_num    NUMBER;
  release_num    NUMBER;
  compat_func    VARCHAR2(65);
  get_compatible VARCHAR2(4000);
BEGIN
  SELECT value INTO compat
  FROM v$parameter
  WHERE name = 'compatible';
  -- Extract version number
  initpos := 1;
  pos := INSTR(compat, '.', initpos, 1);
  IF pos > 0 THEN
    version_num := TO_NUMBER(SUBSTR(compat, initpos, pos - initpos));
    initpos := pos + 1;
    -- Extract release number
    pos := INSTR(compat, '.', initpos, 1);
    IF pos > 0 THEN
      release_num := TO_NUMBER(SUBSTR(compat, initpos, pos - initpos));
      initpos := pos + 1;
    ELSE
      release_num := TO_NUMBER(SUBSTR(compat, initpos));
    END IF;
  END IF;
  IF version_num < 10 THEN
    compat_func := 'dbms_streams.compatible_9_2';
  ELSIF version_num = 10 THEN
    IF release_num < 2 THEN
      compat_func := 'dbms_streams.compatible_10_1';
    ELSE
      compat_func := 'dbms_streams.compatible_10_2';
    END IF;
  ELSE
    compat_func := 'dbms_streams.compatible_10_2';
  END IF;
  get_compatible := ':lcr.get_compatible() <= '||compat_func;
  dbms_streams_adm.add_table_rules(
    table_name => '"SCHEMA1"."TABLE1"',
    streams_type => 'APPLY',
    streams_name => '"APPLY_STREAM"',
    queue_name => '"STRMADMIN"."APPLY_QUEUE"',
    include_dml => TRUE,
    include_ddl => TRUE,
    include_tagged_lcr => TRUE,
    source_database => 'EAST',
    inclusion_rule => TRUE,
    and_condition => get_compatible);
END;
/
--
-- Get tag value to be used for Apply
--
DECLARE
  found   BINARY_INTEGER := 0;
  tag_num NUMBER;
BEGIN
  -- Use the apply object id as the tag
  SELECT o.object_id INTO tag_num
  FROM dba_objects o
  WHERE o.object_name= 'APPLY_STREAM' AND
        o.object_type='APPLY';
 LOOP
    BEGIN
      found := 0;
      SELECT 1 INTO found FROM dba_apply
      WHERE apply_name != 'APPLY_STREAM' AND
            apply_tag = hextoraw(tag_num);
    EXCEPTION WHEN no_data_found THEN
      EXIT;
    END;
    EXIT WHEN (found = 0);
    tag_num := tag_num + 1;
  ENDLOOP;
  -- alter apply
  dbms_apply_adm.alter_apply(
    apply_name => '"APPLY_STREAM"',
    apply_tag => hextoraw(tag_num));
END;
/
--
-- Start apply process APPLY_STREAM
--
BEGIN
  dbms_apply_adm.start_apply(
    apply_name => '"APPLY_STREAM"');
EXCEPTION WHEN OTHERS THEN
  IF sqlcode = -26666 THEN NULL;  -- APPLY process already running
  ELSE RAISE;
  END IF;
END;
/
-- connect as streams administrator to site 1
PROMPT Connecting as streams administrator to site 1
CONNECT &strm_adm_db1/&strm_adm_pwd_db1@&db1
--
-- Enable propagation schedule for "STRMADMIN"."CAPTURE_QUEUE"
-- to WEST
--
DECLARE
  q2q       VARCHAR2(10);
  destn_q   VARCHAR2(65);
BEGIN
  SELECT queue_to_queue INTO q2q
  FROM dba_propagation
  WHERE source_queue_owner = 'STRMADMIN' AND
        source_queue_name = 'CAPTURE_QUEUE' AND
        destination_queue_owner = 'STRMADMIN' AND
        destination_queue_name = 'APPLY_QUEUE' AND
        destination_dblink = 'WEST';
  IF q2q = 'TRUE' THEN
    destn_q := '"STRMADMIN"."APPLY_QUEUE"';
  ELSE
    destn_q := NULL;
  END IF;
  dbms_aqadm.enable_propagation_schedule(
    queue_name => '"STRMADMIN"."CAPTURE_QUEUE"',
    destination => 'WEST',
    destination_queue => destn_q);
EXCEPTION WHEN OTHERS THEN
  IF sqlcode = -24064 THEN NULL; -- propagation already enabled
  ELSE RAISE;
  END IF;
END;
/

When you are happy with the actions described in the script, you can use MAINTAIN_TABLES to update the objects using:

BEGIN
  dbms_streams_adm.maintain_tables 
  (
    table_names => 'SCHEMA1.TABLE1',
    source_directory_object => NULL,
    destination_directory_object => NULL,
    source_database => 'EAST',
    destination_database => 'WEST',
    perform_actions => TRUE,
    capture_name => 'CAPTURE1',
    capture_queue_table => 'CAPTURE_QUEUE_TABLE',
    capture_queue_name => 'CAPTURE_QUEUE',
    capture_queue_user => 'STRMADMIN',
    propagation_name => 'PROPAGATION1',
    apply_name => 'APPLY1',
    apply_queue_table => 'APPLY_QUEUE_TABLE',
    apply_queue_name => 'APPLY_QUEUE',
    apply_queue_user => 'STRMADMIN',
    include_ddl => TRUE,
    instantiation => DBMS_STREAMS_ADM.INSTANTIATION_TABLE_NETWORK
  );
END;
/

Calling MAINTAIN_TABLES from a PL/SQL procedure

Whilst I was initially testing MAINTAIN_TABLES I used anonymous PL/SQL blocks to invoke the procedure. However the customer requirement was to configure Streams from within a package. As a first step I therefore attempted to encapsulate the call to MAINTAIN_TABLES in a PL/SQL procedure. I spent several hours trying to get this simple change to work, but once the call was included in the procedure it failed repeatedly with

ORA-00942: table or view does not exist

I never solved this. Event 10046 level 4 trace suggested the API was encountering a parse error when attempting to select from DBA_OBJECTS. I tried copying all parameters to temporary variables, using IN OUT parameters and setting PLSQL_OPTIMIZER_LEVEL to 0, but none were successful.

My preference when building a standby environment is to keep the design as symmetrical as possible. This preference mainly affects naming conventions. 

The DB_NAME parameter specifies the name of the database and is the same at all locations. In this example the DB_NAME parameter is PROD. 

The DB_UNIQUE_NAME parameter allows a location specific alias to be created for a database. I always try to avoid using names indicative of the role such as PROD_PRIMARY and PROD_STANDBY. These names work well until a switchover is performed at which point the switch-back operation can become really confusing and difficult to document. Therefore I usually try to use geographical values for the DB_UNIQUE_NAME parameter such as PROD_EAST and PROD_WEST. The following discussion uses these values. 

The DB_UNIQUE_NAME parameter defaults to the value of the DB_NAME parameter. This is useful when you are creating a new database; fewer parameters to specify; but is not so efficient if that database will become part of a standby configuration. 

The DB_UNIQUE_NAME parameter is actually used by Oracle in a number of places. It is used to specify

• the database name in the OCR;
• the database directory name within ASM disk groups
• the diagnostic directory name. 

For example if I use DBCA to create a database using the DB_NAME of PROD, the DB_UNIQUE_NAME will default to PROD and will result in the following:

• The database will be called PROD in the OCR
• The database directory name within ASM will be PROD e.g. +DATA/PROD/ and +FRA/PROD/
• The diagnostic location will be $ORACLE_BASE/diag/rdbms/prod 

Under normal conditions this is probably what you want. However for a Data Guard standby database you may however, want the location specific value for the DB_UNIQUE_NAME in which case:

• The database will be called PROD_EAST in the OCR (the database name is still PROD)
• The database directory name within ASM will be PROD_EAST e.g. +DATA/PROD_EAST/ and +FRA/PROD_EAST/
• The diagnostic location will be $ORACLE_BASE/diag/rdbms/prod_east

It is easy enough to modify the DB_UNIQUE_NAME parameter after the database has been created, but then you are left with a few problems.

• The database will be called PROD in the OCR. The original data files etc will have been created in +DATA/PROD
• Any archived redo logs will be in +FRA/PROD
• The diagnostic location is $ORACLE_BASE/diag/rdbms/prod

So what happens when we rename DB_UNIQUE_NAME from PROD to PROD_EAST?

• SRVCTL does not allow databases to be renamed so the entries for the database and instances must be deleted and recreated using the new name of PROD_EAST.
• If we really care then we need to copy all the data files and redo logs from +DATA/PROD to +DATA/PROD_EAST and to update the control file
• I have never figured out how to relocate files in the FRA; it may not even be possible.
• The diagnostic location is created automatically as $ORACLE_BASE/diag/rdbms/prod_east,  but the previous location will still exist and contains some useful files notably the alert.log generated during database creation.

It is possible to modify the DB_UNIQUE_NAME parameter, but the result is a bit messy; not so good for standards-conscious organizations.

How can you work around this? Well the answer is really simple; set the DB_UNIQUE_NAME parameter during database creation. The parameter can be set in the Initialization Parameters page in the DBCA.

However, setting the DB_UNIQUE_NAME parameter in the DBCA will probably result in a failure during database creation. The cause of the failure is the CREATE SPFILE file which is one of the first statements executed during database creation.

If the DB_UNIQUE_NAME is PROD_EAST then by default the CREATE SPFILE statement will attempt to create an SPFILE called +DATA/PROD_EAST/spfilePROD_EAST.ora. However at this stage the +DATA/PROD_EAST directory will not exist within the +DATA disk group and the statement will fail. This appears to be an issue with the CREATE SPFILE statement; other statements such as CREATE DATABASE appear to create the directory if it does not already exist.

You can workaround this problem by creating the directory manually using SQL*Plus or ASMCMD. For example:

$ asmcmd mkdir +DATA/PROD_EAST

If the directory is created before DBCA is executed then the database should be created successfully with the correct names in the OCR, ASM and diagnostic areas.

However, this should really not be necessary; I don’t have access to the source code, but I’m guessing that it would be trivial for Oracle to fix the CREATE SPFILE to create intermediate ASM directories if they do not exist. So I am hoping Oracle product manager will read this and fix the problem in the next patch set….

To my knowledge, the catclust.sql script has existed since Oracle 9.0.1. I think it was originally developed to maintain backward compatibility between OPS and RAC so that the same tools could operate on both products. It creates new views that adhere to new feature names such as GCS and GES, based on older views based on OPS concepts such as DLM.

In order to prevent a repetition during the production upgrade, I wanted to know if there was an easy way to determine whether or not catclust.sql had been run. Obviously I could check if the views created by this script already existed, but perhaps there was a better way.

Recent versions of Oracle include a database registry. This structure records which components have been installed in the database. There is one database registry for each database; not to be confused with the Oracle Cluster Registry (OCR).

The contents of the database registry are externalized a dictionary view called DBA_REGISTRY.  This view describes the components currently installed in the database and contains the following columns:

SQL> DESC dba_registry
Name             Type
---------------  -------------
COMP_ID          VARCHAR2(30)
COMP_NAME        VARCHAR2(255)
VERSION          VARCHAR2(30)
STATUS           VARCHAR2(11)
MODIFIED         VARCHAR2(20)
NAMESPACE        VARCHAR2(30)
CONTROL          VARCHAR2(30)
SCHEMA           VARCHAR2(30)
PROCEDURE        VARCHAR2(61)
STARTUP          VARCHAR2(8)
PARENT_ID        VARCHAR2(30)
OTHER_SCHEMAS    VARCHAR2(4000)

In our example the database registry contained the following:

COLUMN comp_id FORMAT A10
COLUMN comp_name FORMAT A35
COLUMN version FORMAT A14
SELECT comp_id,comp_name,version FROM dba_registry;
COMP_ID    COMP_NAME                           VERSION
---------- ----------------------------------- -------------
APEX       Oracle Application Express          3.2.1.00.12
EM         Oracle Enterprise Manager           11.2.0.2.0
ORDIM      Oracle Multimedia                   11.2.0.2.0
XDB        Oracle XML Database                 11.2.0.2.0
CONTEXT    Oracle Text                         11.2.0.2.0
EXF        Oracle Expression Filter            11.2.0.2.0
RUL        Oracle Rules Manager                11.2.0.2.0
OWM        Oracle Workspace Manager            11.2.0.2.0
CATALOG    Oracle Database Catalog Views       11.2.0.2.0
CATPROC    Oracle Database Packages and Types  11.2.0.2.0
JAVAVM     JServer JAVA Virtual Machine        11.2.0.2.0
XML        Oracle XDK                          11.2.0.2.0
CATJAVA    Oracle Database Java Packages       11.2.0.2.0
13 rows selected.

After running $ORACLE_HOME/rdbms/admin/catclust.sql the registry now contained:

SELECT comp_id,comp_name,version FROM dba_registry;
COMP_ID    COMP_NAME                           VERSION
---------- ----------------------------------- -------------
APEX       Oracle Application Express          3.2.1.00.12
EM         Oracle Enterprise Manager           11.2.0.2.0
ORDIM      Oracle Multimedia                   11.2.0.2.0
XDB        Oracle XML Database                 11.2.0.2.0
CONTEXT    Oracle Text                         11.2.0.2.0
EXF        Oracle Expression Filter            11.2.0.2.0
RUL        Oracle Rules Manager                11.2.0.2.0
OWM        Oracle Workspace Manager            11.2.0.2.0
CATALOG    Oracle Database Catalog Views       11.2.0.2.0
CATPROC    Oracle Database Packages and Types  11.2.0.2.0
JAVAVM     JServer JAVA Virtual Machine        11.2.0.2.0
XML        Oracle XDK                          11.2.0.2.0
CATJAVA    Oracle Database Java Packages       11.2.0.2.0
RAC Oracle Real Application Clusters 11.2.0.2.0
14 rows selected.

So it is possible to verify whether or not components have been installed simply by querying the registry. Whilst useful for catclust.sql it is even more useful if you are auditing which components have been installed in each database. For example APEX has been installed by mistake in this database and the database upgrade might be a good opportunity to remove it.

 When a node is shutdown messages similar to the follow appear in /var/log/messages:

Jun 14 19:19:38 server20 multipathd: uevent trigger error
Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg3:
remove path (uevent)
Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg3:
spurious uevent, path not in pathvec

 When the node is restarted further messages are written to /var/log/messages:

Jun 14 19:19:38 server20 multipathd: uevent trigger error
Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg0:
  add path (uevent)
Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg0:
  failed to store path info 

Similar messages appear for a number of devices in our case including:

• ofsctl
• asm_ctl_vbg0 to asm_ctl_vbg8
• asm_ctl_vdbg
• asm_ctl_vmb
• asm_ctl_spec

We found the ofsctl device very easily:

[root@server20]# ls -l /dev/ofsctl
brw-rw-r-- 1 root oinstall 251, 0 Jun 14 19:19 /dev/ofsctl

The remaining devices were harder to locate. They did not appear to be in /dev/asm.

[root@server20]# ls -l /dev/asm
total 0

However, on further inspection we discovered they had a “dot” prefix:

[root@server20]# ls -la /dev/asm
total 0
drwxrwx---  2 root oinstall     280 Jun 14 19:19 .
drwxr-xr-x 17 root root        5680 Jun 14 19:19 ..
brwxrwx---  1 root oinstall 252,  0 Jun 14 19:19 .asm_ctl_spec
brwxrwx---  1 root oinstall 252, 10 Jun 14 19:19 .asm_ctl_vbg0
brwxrwx---  1 root oinstall 252, 11 Jun 14 19:19 .asm_ctl_vbg1
brwxrwx---  1 root oinstall 252, 12 Jun 14 19:19 .asm_ctl_vbg2
brwxrwx---  1 root oinstall 252, 13 Jun 14 19:19 .asm_ctl_vbg3
brwxrwx---  1 root oinstall 252, 14 Jun 14 19:19 .asm_ctl_vbg4
brwxrwx---  1 root oinstall 252, 15 Jun 14 19:19 .asm_ctl_vbg5
brwxrwx---  1 root oinstall 252, 16 Jun 14 19:19 .asm_ctl_vbg6
brwxrwx---  1 root oinstall 252, 17 Jun 14 19:19 .asm_ctl_vbg7
brwxrwx---  1 root oinstall 252, 18 Jun 14 19:19 .asm_ctl_vbg8
brwxrwx---  1 root oinstall 252,  1 Jun 14 19:19 .asm_ctl_vdbg
brwxrwx---  1 root oinstall 252,  2 Jun 14 19:19 .asm_ctl_vmb
brwxrwx---  1 root oinstall 252,  2 Jun 10 15:06 .asm_ctl_vmb

It appears these devices were added in Oracle 11.2.0.2 and the messages in /var/log/messages are fairly benign although still annoying.

MOS Note 1268895.1 “Multipathd: Asm!.Asm_ctl_spec: Failed To Store Path Info’ found In /var/log/messages” describes a similar error and recommends blacklisting the new devices in /etc/multipath.conf. For example:

 blacklist {
    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
    devnode "^(hd|xvd|vd)[a-z]*"
    devnode "ofsctl"           
    devnode "^asm/*"        
}

Adding the new blacklist entries (shown in bold) to /etc/multipath.conf eliminated the warnings in /var/log/messages  our environment.

We had configured /etc/multipath.conf to set the UID and GID for multipath devices provisioned to support ASM disk groups.

[oracle@server1]$ ls -l /dev/mapper/ora_data1
brw-rw---- 1 oracle dba 253, 3 Jun 14 09:00 ora_data1

I decided that I wanted to change the GID from the dba group to the oinstall group, so I updated the entry in /etc/multipath.conf.from

multipath {
  wwid 360060d40053211000000476200000a09
  alias ora_data1
  uid 500
  gid 201
}

to

multipath {
  wwid 360060d40053211000000476200000a09
  alias ora_data1
  uid 500
  gid 202
}

The server was subsequently rebooted, but the device group was still dba

[oracle@server1]$ ls -l /dev/mapper/ora_data1
brw-rw---- 1 oracle dba 253, 3 Jun 14 09:00 ora_data1

The most obvious possibility was that the group was being overwritten in /etc/rc.local, but that file only contained the default entry

touch /var/lock/subsys/local

Suspicious of the system administrators I searched through udev on the off-chance they had figured out how to change ownerships in there – but none of the files had changed since the operating system was installed.

Finally we discovered that the server was configured to boot from SAN. This means that some of the files required for the initial boot process must be copied into the boot partition. Updating them in /etc has no effect as the kernel has to read the files during the boot phase at which time /etc is unavailable.

Having discovered that /etc/multipath.conf was being ignored, my next concern was whether this affected any other configuration files.

Apparently this issue affects RHEL5U5 and above. To check which files are affected first determine the current kernel using uname -r. For example:

[root@server1]# uname -r 2.6.18-238.1.1.el5

Identify the current kernel image in the /boot directory. In this example the kernel image will be

/boot/initrd-2.6.18-238.1.1.el5.img

Change to another directory e.g. /tmp/initrd. This step is important to avoid inadvertently overwriting the /boot directory (I speak from experience).

[root@server1]# mkdir /tmp/initrd
[root@server1]# cd /tmp/initrd

The following command extracts the contents of the kernel image

zcat /boot/initrd-2.6.18-238.1.1.el5.img | cpio -id

This command will create a set of directories under /tmp/initrd and extract the configuration files into them. In our example there were four files in this directory:

• etc/multipath.conf
• etc/scsi_id.config
• etc/lvm/lvm.conf
• var/lib/multipath/bindings

It is also worth noting that both /etc/multipath.conf and /var/lib/multipath/bindings file are copied to the boot partition.

If you update /etc/multipath.conf it will be necessary to rebuild the initrd so that the changes are applied the next time the node is rebooted.

Before rebuilding the initrd, ensure you have backed up the previous image. For example:

cp /boot/initrd-2.6.18-238.1.1.el5.img /boot/initrd-2.6.18-238.1.1.el5.img.bak

Rebuild the initrd using the following command:

mkinitrd /boot/initrd-$(uname -r).img $(uname -r)

So far this is the only boot from SAN configuration I have encountered so it is too early to say if these are the only configuration files that can be overridden in the boot partition. But at least I now know how to check

During my presentation to the UKOUG RAC & HA SIG I observed that the fastest result seen to date on any platform was 8.92 seconds on an Intel Xeon X5670 six-core 2.93GHz processor. My friend and former colleague Steve Shaw from Intel Corporation was in the audience and suggested it may be possible to improve on these times using some relatively recent processor enhancements. We agreed to spend a day at the Intel lab in Winnersh testing the latest hardware.

The system we tested had two Intel Xeon X5680 3.33GHz six-core Westmere EP processors. The operating system was Oracle Enterprise Linux 5 Update 6. We tested with both the Red Hat Kernel (2.6-18.238.el5) and Oracle Unbreakable Linux kernel (2.6-32-100.26.2.el5).

The test script was as follows:

SET SERVEROUTPUT ON
SET TIMING ON

DECLARE
  n NUMBER := 0;
BEGIN
  FOR f IN 1..10000000
  LOOP
    n := MOD (n,999999) + SQRT (f);
  END LOOP;
  DBMS_OUTPUT.PUT_LINE ('Res = '||TO_CHAR (n,'999999.99'));
END;
/

When executed successfully this script should produce the following output:

Res = 873729.72

For each test we executed the script at least 10 times and took the time of the fastest execution.

We tested the impact of configuring Hyper Threading and Turbo Boost. For the Red Hat (2.6-18.238.el5) kernel the results (in seconds) were as follows:

|-------------|---------|---------------------|
|             |         |    Hyper Threading  |
|             |         |----------|----------|
|             |         |    Off   |    On    |
|-------------|---------|----------|----------|
|Turbo Boost  |   Off   |   8.98   |   8.89   |
|             |---------|----------|----------|
|             |   On    |   8.24   |   8.38   |
|-------------|---------|----------|----------|

So for the Red Hat kernel, performance was significantly improved by enabling Turbo Boost. Hyper Threading appeared to have less impact.

We had problems booting the Oracle Unbreakable Linux kernel (2.6-32-100.26.2.el5) when Turbo Boost was disabled. We think this may be a timing issue during the boot process as opposed to a specific problem with the kernel itself. However, we did perform some tests with Turbo Boost enabled and the results (in seconds) were as follows:

|-------------|---------|---------------------|
|             |         |    Hyper Threading  |
|             |         |----------|----------|
|             |         |    Off   |    On    |
|-------------|---------|----------|----------|
|Turbo Boost  |   On    |   8.33   |  8.21    |
|-------------|---------|----------|----------|

Unfortunately we do not have any results for the Oracle Unbreakable kernel with Turbo Boost disabled. However, the result of 8.21 seconds with both Hyper Threading and Turbo Boost enabled is the fastest this test has ever run.

Although the results for the Red Hat kernel contradict this, in our opinion Hyper Threading often has a positive impact on system throughput for more representative tests; it rarely has a negative impact and therefore on balance we would generally recommend enabling Hyper Threading on all Xeon processors.

However these results also suggest that the Turbo Boost feature should also be enabled. For the Red Hat kernel enabling Turbo Boost improved performance by 6% to 9%. As the Turbo Boost feature is rarely enabled in production environments, it may be worth investigating for CPU-bound systems.

Thanks to Steve Shaw of Intel Corporation for his assistance in the research for this post.

Add the following lines to /etc/security/limits.conf

oracle           soft    nproc           2047
oracle           hard    nproc           16384
oracle           soft    nofile          1024
oracle           hard    nofile          65536

Additional memlock limits are required if large pages are configured. The values of these limits is dependent on the number of huge pages allocated. One also occasionally sees other limits such as the stack limit increased.

Add the following line to /etc/pam.d/login

session    required     pam_limits.so

Add the following lines to /etc/profile

if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -u 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

I normally reboot the system after configuring the above settings.

During an Enterprise Manager Grid Control installation this week, we made the above changes. But when we tried to login to Oracle we hit the following error:

login as: oracle
oracle@vm7's password:
Last login: Wed Jan 19 05:39:00 2011 from 192.168.5.100
-bash: ulimit: open files: cannot modify limit: Operation not permitted

This error did not occurred if we logged in as root and then entered

[root@vm7]# su - oracle

After some investigation the cause of this error was identified. In /etc/ssh/sshd_config there is a parameter called UsePAM which enables PAM authentication. If this parameter is not set then it defaults to “no”. I think it is normally overridden during installation. In order to allow user limits to be set the value should be “yes”

UsePAM=yes

The sshd service must be restarted for the parameter change to take effect:

[root@vm7]# service sshd restart

If the change is successful the new shell limits should be reported by the ulimit command. For example:

[oracle@vm7]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 26560
max locked memory       (kbytes, -l) 1148576
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16384
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

In the above example the limits of interest are open files (-n) which should be 65536 and max user processes (-u) which should be 16384.

In many documents (including some Oracle 11gR2 installation guides) the following lines are recommended for /etc/profile

if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

Not many sites use the Korn shell with Linux but if you do the following error will occur during login:

login as: oracle
oracle@vm7's password:
Last login: Wed Oct 13 07:25:04 2010 from 192.168.5.100
/etc/profile[58]: ulimit: pipe: is read only

The error occurs because the ulimit -p parameter specifies the number of 512-byte blocks for pipe buffering. The Korn shell man page (man ksh) lists most ulimit parameters, but omits the one we need which is -u to specify the number of processes. So /etc/profile should contain:

if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -u 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

In the Korn shell we can confirm this using ulimit -a

[oracle@vm7]$ ulimit -a
address space limit (kbytes)   (-M)  unlimited
core file size (blocks)        (-c)  0
cpu time (seconds)             (-t)  unlimited
data size (kbytes)             (-d)  unlimited
file size (blocks)             (-f)  unlimited
locks                          (-L)  unlimited
locked address space (kbytes)  (-l)  1148576
nofile                         (-n)  65536
nproc                          (-u)  16384
pipe buffer size (bytes)       (-p)  4096
resident set size (kbytes)     (-m)  unlimited
socket buffer size (bytes)     (-b)  4096
stack size (kbytes)            (-s)  10240
threads                        (-T)  not supported
process size (kbytes)          (-v)  unlimited

In the above example nofile (-n) should be 65536 and nproc (-u) should be 16384

The Grid Infrastructure installation failed when running root.sh on the second node with the following error:

root@server21 # /data/app/oragrid/11.2.0/grid/root.sh
Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /data/app/oragrid/11.2.0/grid

Enter the full pathname of the local bin directory:
  [/usr/local/bin]:
The contents of "dbhome" have not changed.
  No need to overwrite.
The contents of "oraenv" have not changed.
  No need to overwrite.
The contents of "coraenv" have not changed.
  No need to overwrite.

Entries will be added to the /var/opt/oracle/oratab file as
needed by Database Configuration Assistant when a database
is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file:
  /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4402: The CSS daemon was started in exclusive mode but
found an active CSS daemon on node server21, number 1, and
is terminating
An active cluster was found during exclusive startup,
restarting to join the cluster
Failed to start Oracle Clusterware stack
Failed to start Cluster Synchorinisation Service in clustered
mode at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm
line 1017.
/u01/app/11.2.0/grid/perl/bin/perl
-I/u01/app/11.2.0/grid/perl/lib
-I/u01/app/11.2.0/grid/crs/install
/u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

We have seen this error before on Linux so we downloaded the mcasttest utility from MOS Note 1212703.1 “Grid Infrastructure install or upgrade may fail due to Multicasting”

A detailed explanation of the mcasttest utility is available in another post. The servers were called server21, server22 (renamed to protect the customer’s identity). The private network used device e1000g2 so we executed mcasttest as follows:

$ ./mcasttest.pl -n server21,server22 -i e1000g2
###########  Setup for node server21  ##########
Checking node access 'server21'
Checking node login 'server21'
Checking/Creating Directory /tmp/mcasttest for binary
  on node 'server21'
Distributing mcast2 binary to node 'server21'
###########  Setup for node server22  ##########
Checking node access 'server22'
Checking node login 'server22'
Checking/Creating Directory /tmp/mcasttest for binary
  on node 'server22'
Distributing mcast2 binary to node 'server22'
###########  testing Multicast on all nodes  ##########

Test for Multicast address 230.0.1.0

Multicast Failed for e1000g2
  using address 230.0.1.0:42000

Test for Multicast address 224.0.0.251

Multicast Failed for e1000g2
  using address 224.0.0.251:42001

In this example multicasting is failing for both the default address (230.0.1.0) and the alternative address (224.0.0.251). Therefore there is no point in installing the patch for bug 9974223 – “Grid Infrastructure needs multicast communication on 230.0.1.0 address working” as it will not solve the problem.

The customer was using CISCO switches on which multicasting had been deliberately disabled for business reasons. Therefore we were unable to explore the possibility of enabling multicasting at switch level.

An alternative approach was required, so we tried replacing the CISCO switches on the private network with a cheap 100Mb switch. If this test was successful the customer was planning to order two dedicated switches to provide resilience and to use IPMP to configure the private network.

After the new switch was installed the output of mcasttest was:

$ ./mcasttest.pl -n server21,server22 -i e1000g2
###########  Setup for node server21  ##########
Checking node access 'server21'
Checking node login 'server21'
Checking/Creating Directory /tmp/mcasttest for binary
  on node 'server21'
Distributing mcast2 binary to node 'server21'
###########  Setup for node server22  ##########
Checking node access 'server22'
Checking node login 'server22'
Checking/Creating Directory /tmp/mcasttest for binary
  on node 'server22'
Distributing mcast2 binary to node 'server22'
###########  testing Multicast on all nodes  ##########

Test for Multicast address 230.0.1.0

Multicast Succeeded for e1000g2
  using address 230.0.1.0:42000

Test for Multicast address 224.0.0.251

Multicast Succeeded for e1000g2
  using address 224.0.0.251:42001

So replacing the switch had solved the multicast issue.

We attempted to reinstall Grid Infrastructure, but again it failed when we ran root.sh on the second node with the following error:

Failed to start Oracle Clusterware stack
Failed to start Cluster Synchorinisation Service in clustered
mode at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm
line 1017.
/u01/app/11.2.0/grid/perl/bin/perl
-I/u01/app/11.2.0/grid/perl/lib
-I/u01/app/11.2.0/grid/crs/install
/u01/app/11.2.0/grid/crs/install/rootcrs.pl
  execution failed

We installed the patch for bug 9974223 – “Grid Infrastructure needs multicast communication on 230.0.1.0 address working”.

The README for the Solaris SPARC version of patch 9974223 was incorrect; it appeared to be a generic patch notice rather than a patch for Grid Infrastructure. We compared with the Linux x86-64 README and discovered this was completely different and appeared to be correct. So we followed the Linux x86-64 README using the Solaris patch.

However, following installation of this patch the error still occurred. We tried a couple more times, then resorted to digging around in the log files to try to identify exactly where root.sh was failing.

The root.sh script calls several other scripts and finally executes

$GRID_HOME/crs/install/rootcrs.pl.

The rootcrs.pl script creates a log file in

$GRID_HOME/cfgtoollogs/crsconfig/rootcrs_<server>.log

This script contains some useful output. The rootcrs.pl script attempts to start CSSD using a root agent. The log file for the root agent is:

$GRID_HOME/log/<server>/agent/ohasd/orarootagent_root/
  orarootagent_root.log

The root agent starts the CSSD daemon which logs its activities in:

$GRID_HOME/log/<server>/cssd/ocssd.log

Investigating these log files, we became convinced that the error for which we were looking was not caused by the multicasting configuration which appeared to be working correctly. However, we did notice a strange discrepancy in ifconfig for the interconnect on the first node after root.sh had been executed.

e1000g2:
   flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,
     IPv4,FIXEDMTU> mtu 9000 index 3
   inet 172.16.0.2 netmask ffffff00
   broadcast 172.16.0.255
   ether 0:c0:dd:14:c:58
e1000g2:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,
      IPv4> mtu 9216 index 3
   inet 169.254.35.5 netmask ffff0000
   broadcast 169.254.255.255

In Oracle 11.2.0.2 a new feature called the redundant interconnect has been introduced. Instead of configuring bonding at operating system level, it is theoretically possible to allow Oracle to manage multiple physical networks providing load balancing and failover capabilities. Within the Grid Infrastructure the new feature is known as HAIP. I will cover this in more detail in another post. I looked at the ifconfig from another customer where we had successfully installed Oracle 11.2.0.2. Even if the bonding has been configured for the private network, the Grid Infrastructure will automatically create a new VIP on the private network device for each node in the cluster. The IP address is automatically allocated from the 169.254.0.0 subnet, apparently using a random algorithm.

So in the above example the private network has been configured to use e1000g2. When Clusterware is started a new virtual IP is created called e1000g2:1. The latter address appears to be used by CSSD for inter-node communications.

However, take another look at the MTU sizes. For the e1000g2 the MTU size is 9000; for e1000g2:1 Oracle has used an MTU size of 9216. This anomaly had us scratching our heads. We took a look at the network device definition file

grid@server21 $ cat /etc/hostname.e1000g2
server21-priv mtu 9000

So we were hard coding the MTU size in /etc/hostname.e1000g2, but Oracle was ignoring this and using 9216. Which size was correct?

Unlike Linux, in Solaris the ping command does not have an option to specify the packet size, so I tried the traceroute command.

We used the following syntax for the traceroute command:

traceroute -s <source> -r -F <target> <MTU Size>

By trial and error we discovered that the traceroute command was successful at all MTU sizes up to 1518, but failed for MTU sizes of 1519 and above. We immediately suspected our cheap switch and further experimentation quickly confirmed that it did not support MTU sizes greater than 1518.

On the other interfaces, the customer had used the default MTU size of 1500, so we decided to revert to this. As it is the default we simply removed the MTU size from /etc/hostname.e1000g2. For example:

grid@server21 $ cat /etc/hostname.e1000g2
server21-priv

However, this did not have the desired effect. In fact the MTU size for e1000g2 increased from 9000 to 9216. On Solaris the MTU size is defined in two places; in the interface definition file e.g. /etc/hostname.e1000g2 and also in the kernel driver e.g. /kernel/drv/e1000g.conf.

Initially /kernel/drv/e1000g.conf contained the following

root@server21 # cat /kernel/drv/e1000g.conf
# Driver.conf file for Intel e1000g Gigabit Ethernet Adapter
MaxFrameSize=0,0,3,0,3,0,0,0,0,0,0,0,0,0,0,0;
        # 0 is for normal ethernet frames.
        # 1 is for upto 4k size frames.
        # 2 is for upto 8k size frames.
        # 3 is for upto 16k size frames.
        # These are maximum frame limits, not the actual
        # ethernet frame size. Your actual ethernet frame
        # size would be determined by protocol stack
        # configuration (please refer to ndd command man
        # pages)
        # For Jumbo Frame Support (9k ethernet packet)
        # use 3 (upto 16k size frames)

In the above file, the MaxFrameSize entry is defining the maximum frame size for each interface. The first entry is for e1000g0, the next for e1000g1 etc. In the above example the maximum frame size is set to the default (1500) for e1000g0, e1000g1 and e1000g3. It is set to 3 (up to 16k size frames) for e1000g2 and e1000g4. However as 9216 is the absolute limit for frame sizes these devices are restricted to this MTU size.

Therefore we updated MaxFrameSize so that all entries were set to zero and rebooted the node:

MaxFrameSize=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;

Following this change ifconfig reported the following:

e1000g2:
  flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,
     IPv4> mtu 1500 index 3
   inet 172.16.0.1 netmask ffffff00
   broadcast 172.16.0.255
   ether 0:c0:dd:14:7:80

So the MTU size was now set correctly to 1500

After reinstalling Grid Infrastructure on the first node (including root.sh) ifconfig reported the following:

e1000g2:
  flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,
    IPv4> mtu 1500 index 3
  inet 172.16.0.1 netmask ffffff00
  broadcast 172.16.0.255
  ether 0:c0:dd:14:7:80
e1000g2:1:
  flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,
    IPv4> mtu 1500 index 3
  inet 169.254.248.231 netmask ffff0000
  broadcast 169.254.255.255

So the new HAIP VIP was also now using the MTU size of 1500.

We tried running root.sh on the second node and this time it succeeded.