Table Utility Programs

(A Brief Documentation of the Programs dom / tmerge / tsplit / opc / xmat)

Contents

Introduction

I am sorry that there is no detailed documentation yet. Below you can find a brief explanation of how to grow a decision tree with the program dti, how to prune a decision tree with the program dtp, how to execute a decision tree with the program dtx, and how to extract rules from a decision tree with the program dtr. For a list of options, call the programs without any arguments.

Enjoy,
Christian Borgelt

As a simple example for the explanations below I use the dataset in the file table/ex/drug.tab, which lists 12 records of patient data (sex, age, and blood pressure) together with an effective drug (effective w.r.t. some unspecified disease). The contents of this file is:

   Sex    Age Blood_pressure Drug
   male   20  normal         A
   female 73  normal         B
   female 37  high           A
   male   33  low            B
   female 48  high           A
   male   29  normal         A
   female 52  normal         B
   male   42  low            B
   male   61  normal         B
   female 30  normal         A
   female 26  low            B
   male   54  high           A
back to the top top

Determining Attribute Domains

The domains of the columns of th table drug.tabcan be determined with the program dom

  dom -a drug.tab drug.dom

The program dom assumes that the first line of the table file contains the column names. (This is the case for the example file drug.tab.) If you have a table file without column names, you can let the program read the column names from another file (using the -h option) or you can let the program generate default names (using the -d option), which are simply the column numbers. The -a option tells the program to determine automatically the column data types. Thus the values of the Age column are automatically recognized as integer values.

After dom has finished, the contents of the file drug.dom should look like this:

  dom(Sex) = { male, female };
  dom(Age) = ZZ;
  dom(Blood_pressure) = { normal, high, low };
  dom(Drug) = { A, B };

The special domain ZZ represents the set of integer numbers, the special domain IR (not used here) the set of real numbers. (The double Z and the I in front of the R are intended to mimic the bold face or double stroke font used in mathematics to write the set of integer or the set of real numbers. All programs that need to read a domain description also recognize a single Z or a single R.)

back to the top top

Merge Tables

The program tmerge can be used to merge two tables and to project a table to a subset of its columns. The latter consists simply in merging a table to another which contains less columns. I only demonstrate the projection by merging the table in the file drug.tab to the empty table in the file drug.hdr:

  tmerge -a drug.hdr drug.tab drug.prj

This command removes the Age column from the table drug.tab (since this column is missing in the file drug.hdr) and writes the result to the file drug.prj. After the program tmerge has finished, the contents of the file drug.prj should be:

  Sex    Blood_pressure Drug
  male   normal         A
  female normal         B
  female high           A
  male   low            B
  female high           A
  male   normal         A
  female normal         B
  male   low            B
  male   normal         B
  female normal         A
  female low            B
  male   high           A

Since the option -a is given, the columns of the output file are aligned. If the file drug.hdr contained tuples, these tuples would precede the tuples from the file drug.tab in the output file drug.prj.

back to the top top

Split a Table

If you are interested in the sets of patients with low, normal, or high blood pressure, you can split the table into subtables, each of which contains only tuples with a specific value for the column Blood_pressure, with

  tsplit -a -c Blood_pressure drug.tab

This should result in three files - 0.tab, 1.tab and 2.tab - with the following contents:

  0.tab:   Sex    Age Blood_pressure Drug
           male   20  normal         A
           female 73  normal         B
           male   29  normal         A
           female 52  normal         B
           male   61  normal         B
           female 30  normal         A

  1.tab:   Sex    Age Blood_pressure Drug
           female 37  high           A
           female 48  high           A
           male   54  high           A

  2.tab:   Sex    Age Blood_pressure Drug
           male   33  low            B
           male   42  low            B
           female 26  low            B

That is, the file 0.tab contains all patients with normal blood pressure, the file 1.tab all patients with high blood pressure, and the file 2.tab all patients with low blood pressure. The tables are aligned since the option -a was given. With the -c option the column is specified on which the split is based. Similarly, the table can be split in such a way that the relative frequencies of the values are maintained (stratified split). For example, calling the program tsplit with

  tsplit -a -t3 -c Blood_pressure drug.tab

should result in three files (3 because of the -t3 option) - 0.tab, 1.tab and 2.tab - with the following contents:

  0.tab:   Sex    Age Blood_pressure Drug
           male   20  normal         A
           female 52  normal         B
           female 37  high           A
           male   33  low            B

  1.tab:   Sex    Age Blood_pressure Drug
           female 73  normal         B
           male   61  normal         B
           female 48  high           A
           male   42  low            B

  2.tab:   Sex    Age Blood_pressure Drug
           male   29  normal         A
           female 30  normal         A
           male   54  high           A
           female 26  low            B
back to the top top

Computing One Point Coverages

The program opc can be used to reduce a table. This does not change anything for the original table, but simplifies the table that resulted from the application of the program tmerge shown above. This table can be reduced by calling the program opc with

  opc -a drug.prj drug.red

After the program opc has finished, the contents of the file drug.red should read like this:

  Sex    Blood_pressure Drug #
  male   normal         A    2
  male   normal         B    1
  male   high           A    1
  male   low            B    2
  female normal         A    1
  female normal         B    2
  female high           A    2
  female low            B    1

The number in the last column indicates the number of occurences of the corresponding tuple (table row) in the original table.

The opc program can also be used to compute one point coverages, either in a fully expanded or in a compressed form. One point coverages are considered in possibility theory and computing them is important for inducing possibilistic network from data. However, explaining this in detail would lead too far.

back to the top top

Computing a Confusion Matrix

The program xmat can be used to evaluate a classification result. It reads a table file and computes a confusion matrix from two columns of this table. It uses the last two columns by default (the last column for the x- and the semi-last for the y-direction). Other columns can be selected via the options -x and -y followed by the name of the columns that are to be used for the x- or y-direction of the confusion matrix. To demonstrate this program we use the file drug.cls, which contains simply the data from the file drug.tab with an additional classification column:

  Sex    Age Blood_pressure Drug Class
  male   20  normal         A    B
  female 73  normal         B    B
  female 37  high           A    A
  male   33  low            B    B
  female 48  high           A    B
  male   29  normal         A    A
  female 52  normal         B    B
  male   42  low            B    A
  male   61  normal         B    B
  female 30  normal         A    A
  female 26  low            B    B
  male   54  high           A    A

To determine a confusion matrix for this table, simply call the program xmat with

  xmat drug.cls

The output, which by default is written to the terminal, should read like this:

  confusion matrix for Drug vs. Class:
   no | value  |      1      2 | errors
  ----+--------+---------------+-------
    1 | A      |      4      2 |      2
    2 | B      |      1      5 |      1
  ----+--------+---------------+-------
      | errors |      1      2 |      3

In this matrix the x-direction (columns) corresponds to the column Class and the y-direction (rows) to the column Drug. As you can see, for drug A the classification is wrong in two cases (first line, second column of the matrix), for drug B it is wrong in one case (second line, first column). Overall there are three errors.

back to the top top

License

(MIT license, or more precisely Expat License; to be found in the file mit-license.txt in the directory table/doc in the source package of the program, see also opensource.org and wikipedia.org)

© 1996-2016 Christian Borgelt

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

back to the top top

Download

Download page with most recent version.

back to the top top

Contact

E-mail: christian@borgelt.net
Website: www.borgelt.net
back to the top top

© 1999-2016 Christian Borgelt