Problem solve Get help with specific problems with your technologies, process and projects.

Trees in SQL

A set-oriented method for representing trees in SQL that runs orders of magnitude faster than the adjacency list method.

A set-oriented method for representing trees in SQL that runs orders of magnitude faster than the adjacency list...

The usual example of a tree structure in SQL books is called an adjacency list model and it looks like this:

``` CREATE TABLE Personnel (emp CHAR(10) NOT NULL PRIMARY KEY, boss CHAR(10) DEFAULT NULL REFERENCES Personnel(emp), salary DECIMAL(6,2) NOT NULL DEFAULT 100.00); Personnel emp boss salary =========================== 'Albert' 'NULL' 1000.00 'Bert' 'Albert' 900.00 'Chuck' 'Albert' 900.00 'Donna' 'Chuck' 800.00 'Eddie' 'Chuck' 700.00 'Fred' 'Chuck' 600.00
```

Another way of representing trees is to show them as nested sets. Since SQL is a set oriented language, this is a better model than the usual adjacency list approach you see in most text books. Let us define a simple Personnel table like this, ignoring the left (lft) and right (rgt) columns for now. This problem is always given with a column for the employee and one for his boss in the textbooks. This table without the lft and rgt columns is called the adjacency list model, after the graph theory technique of the same name; the pairs of nodes are adjacent to each other.

``` CREATE TABLE Personnel (emp CHAR(10) NOT NULL PRIMARY KEY, lft INTEGER NOT NULL UNIQUE CHECK (lft > 0), rgt INTEGER NOT NULL UNIQUE CHECK (rgt > 1), CONSTRAINT order_okay CHECK (lft < rgt) ); Personnel emp lft rgt ====================== 'Albert' 1 12 'Bert' 2 3 'Chuck' 4 11 'Donna' 5 6 'Eddie' 7 8 'Fred' 9 10
```

The organizational chart would look like this as a directed graph:

``` Albert (1,12) / \ / \ Bert (2,3) Chuck (4,11) / | \ / | \ / | \ / | \ Donna (5,6) Eddie (7,8) Fred (9,10)
```

The first table is denormalized in several ways. We are modeling both the personnel and the organizational chart in one table. But for the sake of saving space, pretend that the names are job titles and that we have another table which describes the personnel that hold those positions.

Another problem with the adjacency list model is that the boss and employee columns are the same kind of thing (i.e. names of personnel), and therefore should be shown in only one column in a normalized table. To prove that this is not normalized, assume that "Chuck" changes his name to "Charles"; you have to change his name in both columns and several places. The defining characteristic of a normalized table is that you have one fact, one place, one time.

More on programming in SQL

Learn to do SQL arithmetic with SELECT instead of arithmetic operations

See an expert's process for managing SQL parent table-child table relations

Find out how to sort numbers using SQL SELECT and SQL UNION ALL statements

The final problem is that the adjacency list model does not model subordination. Authority flows downhill in a hierarchy, but If I fire Chuck, I disconnect all of his subordinates from Albert. There are situations (i.e. water pipes) where this is true, but that is not the expected situation in this case.

To show a tree as nested sets, replace the nodes with ovals, then nest subordinate ovals inside each other. The root will be the largest oval and will contain every other node. The leaf nodes will be the innermost ovals with nothing else inside them and the nesting will show the hierarchical relationship. The rgt and lft columns (I cannot use the reserved words LEFT and RIGHT in SQL) are what shows the nesting.

If that mental model does not work, then imagine a little worm crawling anti-clockwise along the tree. Every time he gets to the left or right side of a node, he numbers it. The worm stops when he gets all the way around the tree and back to the top.

This is a natural way to model a parts explosion, since a final assembly is made of physically nested assemblies that final break down into separate parts.

At this point, the boss column is both redundant and denormalized, so it can be dropped. Also, note that the tree structure can be kept in one table and all the information about a node can be put in a second table and they can be joined on employee number for queries.

To convert the graph into a nested sets model think of a little worm crawling along the tree. The worm starts at the top, the root, makes a complete trip around the tree. When he comes to a node, he puts a number in the cell on the side that he is visiting and increments his counter. Each node will get two numbers, one of the right side and one for the left. Computer Science majors will recognize this as a modified preorder tree traversal algorithm. Finally, drop the unneeded Personnel.boss column which used to represent the edges of a graph.

This has some predictable results that we can use for building queries. The root is always (left = 1, right = 2 * (SELECT COUNT(*) FROM TreeTable)); leaf nodes always have (left + 1 = right); subtrees are defined by the BETWEEN predicate; etc. Here are two common queries which can be used to build others:

1. An employee and all their Supervisors, no matter how deep the tree.

``` SELECT P2.* FROM Personnel AS P1, Personnel AS P2 WHERE P1.lft BETWEEN P2.lft AND P2.rgt AND P1.emp = :myemployee;
```

2. The employee and all subordinates. There is a nice symmetry here.

``` SELECT P1.* FROM Personnel AS P1, Personnel AS P2 WHERE P1.lft BETWEEN P2.lft AND P2.rgt AND P2.emp = :myemployee;
```

3. Add a GROUP BY and aggregate functions to these basic queries and you have hierarchical reports. For example, the total salaries which each employee controls:

``` SELECT P2.emp, SUM(S1.salary) FROM Personnel AS P1, Personnel AS P2, Salaries AS S1 WHERE P1.lft BETWEEN P2.lft AND P2.rgt AND P1.emp = S1.emp GROUP BY P2.emp;
```

4. To find the level of each node, so you can print the tree as an indented listing.

``` SELECT COUNT(P2.emp) AS indentation, P1.emp FROM Personnel AS P1, Personnel AS P2 WHERE P1.lft BETWEEN P2.lft AND P2.rgt GROUP BY P1.emp ORDER BY P1.lft;
```

5. The nested set model has an implied ordering of siblings which the adjacency list model does not. To insert a new node as the rightmost sibling.

``` BEGIN DECLARE right_most_sibling INTEGER; SET right_most_sibling = (SELECT rgt FROM Personnel WHERE emp = :your_boss); UPDATE Personnel SET lft = CASE WHEN lft > right_most_sibling THEN lft + 2 ELSE lft END, rgt = CASE WHEN rgt >= right_most_sibling THEN rgt + 2 ELSE rgt END WHERE rgt >= right_most_sibling; INSERT INTO Personnel (emp, lft, rgt) VALUES ('New Guy', right_most_sibling, (right_most_sibling + 1)) END;
```

6. To convert an adjacency list model into a nested set model, use a push down stack algorithm. Assume that we have these tables:

``` -- Tree holds the adjacency model CREATE TABLE Tree (emp CHAR(10) NOT NULL, boss CHAR(10)); INSERT INTO Tree SELECT emp, boss FROM Personnel; -- Stack starts empty, will holds the nested set model CREATE TABLE Stack (stack_top INTEGER NOT NULL, emp CHAR(10) NOT NULL, lft INTEGER, rgt INTEGER); BEGIN ATOMIC DECLARE counter INTEGER; DECLARE max_counter INTEGER; DECLARE current_top INTEGER; SET counter = 2; SET max_counter = 2 * (SELECT COUNT(*) FROM Tree); SET current_top = 1; INSERT INTO Stack SELECT 1, emp, 1, NULL FROM Tree WHERE boss IS NULL; DELETE FROM Tree WHERE boss IS NULL; WHILE counter <= (max_counter - 2) LOOP IF EXISTS (SELECT * FROM Stack AS S1, Tree AS T1 WHERE S1.emp = T1.boss AND S1.stack_top = current_top) THEN BEGIN -- push when top has subordinates and set lft value INSERT INTO Stack SELECT (current_top + 1), MIN(T1.emp), counter, NULL FROM Stack AS S1, Tree AS T1 WHERE S1.emp = T1.boss AND S1.stack_top = current_top; DELETE FROM Tree WHERE emp = (SELECT emp FROM Stack WHERE stack_top = current_top + 1); SET counter = counter + 1; SET current_top = current_top + 1; END ELSE BEGIN -- pop the stack and set rgt value UPDATE Stack SET rgt = counter, stack_top = -stack_top -- pops the stack WHERE stack_top = current_top SET counter = counter + 1; SET current_top = current_top - 1; END IF; END LOOP; END;
```

This approach will be two to three orders of magnitude faster than the adjacency list model for subtree and aggregate operations.

Michael S. had some questions (in italics), to which Joe responded. Here's the exchange:

I was interested to read this article about implementing tree structures in SQL. I work for Streamload, an online file storage website where I helped developed the database technology that powers the site. We use a SQL database to store information about hundreds of millions of files and folders, basically a "tree in SQL" per registered user. We use the adjacency list method, and we are forced to use recursive execution to get all files under a folder, so this nested sets solution is of great interest to us.

I have had other people with similar problems. They added another column to the table for each tree within the forest:

``` CREATE TABLE Forest (tree_id CHAR(10) NOT NULL, node_id CHAR(10) NOT NULL, lft INTEGER NOT NULL UNIQUE CHECK (lft > 0), rgt INTEGER NOT NULL UNIQUE CHECK (rgt > 1), CONSTRAINT order_okay CHECK (lft < rgt), PRIMARY KEY (tree_id, node_id));
```

Upon looking at the ramifications of the nested sets implementation, however, it would seem to require too much database thrashing for tree modifying operations. The code to do a tree insert was provided, and it seemed at initial glance that a) the state of the entire tree needed to remain fixed (locked) for the duration of the operation in order to not risk leaving the tree in an invalid state if another insert operations happened concurrently, and b) it seemed possible that much if not most of the tree could be modified by a simple insertion, not just the ancestor nodes as one might expect/hope, but possibly everything to the left and/or right of a node. These seem to be major impediments to implementing the nested sets solution in practice. I'd be interested in getting some response from the author and/or others as to the validity of these complications and any possible workarounds.

These limits are not a problem if your tree is relatively fixed -- most companies do not do re-organizations on a weekly basis. Well, that might not have been true for dot-coms...

But in your case, a rapidly changing tree structure, has been implemented successful for a newsgroup project. The forest was the newsgroup itself, each tree was a message thread, and each node was a posted message. There was almost an order of magnitude improvement in performance (handling and displaying text in and out of SQL is expensive, so most of the execution time was spent there).

The gimmick? The nested set model has two properties which I call:

1) The Algebra Property: For each node (n), (rgt-lft+1)/2 = size of subtree rooted at (n). This requires that the union of the rgt and lft numbers be an unbroken sequence, no gaps.

2) The Between-ness Property that you can find superiors and subordinates with a BETWEEN predicate:

``` SELECT O2.* FROM OrgChart AS O1, OrgChart AS O2 WHERE O1.lft BETWEEN O2.lft AND O2.rgt AND O1.emp = :my_employee;
```

The employee and all subordinates. There is a nice symmetry here.

``` SELECT O1.* FROM OrgChart AS O1, OrgChart AS O2 WHERE O1.lft BETWEEN O2.lft AND O2.rgt AND O2.emp = :my_employee;
```

This does NOT require that the union of the rgt and lft numbers be an unbroken sequence. All you need is the condition that (lft < rgt), uniqueness of lft and rgt, and that subordination is represented by containment of one (lft, rgt) pair within the range another (lft, rgt) pair.

Can you tell that I was a math major? In English: the trick is to use a bigger step than 1, if you don't need the Algebra property. For example:

``` CREATE TABLE OrgChart (emp CHAR(10) NOT NULL PRIMARY KEY, lft INTEGER NOT NULL UNIQUE CHECK (lft > 0), rgt INTEGER NOT NULL UNIQUE CHECK (rgt > 1), CONSTRAINT order_okay CHECK (lft < rgt) ); OrgChart emp lft rgt ====================== 'Albert' 100 1200 'Bert' 200 300 'Chuck' 400 1100 'Donna' 500 600 'Eddie' 700 800 'Fred' 900 1000
```

The organizational chart would look like this as a directed graph:

``` Albert (100,1200) / \ / \ Bert (200,300) Chuck (400,1100) / | \ / | \ / | \ / | \ / | \ Donna (500,600) Eddie (700,800) Fred (900,1000)
```

To insert someone under Bert, say Betty, you look at the size of Bert's range and pack from the left:

``` INSERT INTO OrgChart VALUES ('Betty', 210, 220);
```

To insert someone under Betty, you look at the size of Betty's range and pack from the left:

``` INSERT INTO OrgChart VALUES ('Bobby', 211, 212);
```

Given a clustered index on (lft,rgt), a reasonable fill factor per data page and reasonably smart concurrency control, only the datapage involved will be locked for each insertion. Assuming you have a 32 bit integer, you can have a depth of nine or ten levels before you have to re-organize this particular tree within the forest. If you don't mind negstive numbers, you can use the full range of the integers:

``` INSERT INTO Tree VALUES ('root', -4,294,967,296, 4294967296);
```

I am obviously skipping some of the algebra for computing the gap size, but you get the basic idea. There are some other tricks that involve powers of two and binary trees, but that is another topic. A consultant does not give away all his tricks without getting paid!

Joe Celko is author of Joe Celko's SQL for Smarties: Advanced SQL Programming (Morgan-Kaufmann, 1999, second edition).

This was last published in March 2001

Content

Find more PRO+ content and other member only offers, here.

Start the conversation

Send me notifications when other members comment.

SearchDataManagement

• Hyperledger Fabric offers path to enterprise blockchain future

Blockchain arose from bitcoin, but it's looking to find a place in the enterprise. Frameworks like Hyperledger Fabric could ...

• MongoDB 4.0 takes ACID transactions to multi-document level

MongoDB is taking a deeper step into SQL-style processing waters with a 4.0 update that brings increased support for ...

• Data lake concept needs firm hand to pay big data dividends

Data lakes pose technology deployment and data management challenges that can leave analytics users high and dry if the ...

• AI functionality limited today but could be a game-changer

Limited AI capabilities could soon give way to technology that is truly transformative for enterprises, surpassing the overhyped ...

SearchSAP

• SAP's Barry Padgett on future of SAP Ariba Network

In this Q&A, new SAP Ariba President Barry Padgett discusses the future of procurement and the experience he will bring to Ariba ...

• Avoiding SAP indirect access woes requires good faith

Some customers are concerned that SAP will hit them for indirect access licensing fees, but they can avoid trouble if they act in...

• ControlPanelGRC app eases Steelcase's compliance pain

When Steelcase's SAP environment grew in size and complexity, it turned to Symmetry ControlPanelGRC to save time, have more ...

SearchSQLServer

SQL Operations Studio simplifies routine administration of SQL Server and Azure SQL databases, making database development and ...

• Meltdown and Spectre fixes eyed for SQL Server performance issues

Microsoft has responded to the Spectre and Meltdown chip vulnerabilities with patches and other fixes. But IT teams need to sort ...

• Five SQL Server maintenance steps you should take -- ASAP

Putting off SQL Server administration tasks can lead to database problems. Enact these often-neglected maintenance items to help ...

TheServerSide.com

• How DevOps concepts eluted from cloud computing and service platforms

The popularity of DevOps can be traced back to the emergence of cloud computing. As programmers began scripting their ...

• Pluralsight IQ, Stack Overflow boost developer street cred

Tying the Pluralsight IQ skills test to the Stack Overflow Developer Story helps developers measure their technical skills and ...

• Why this quantum computing breakthrough is a security risk

Quantum computing will void pretty much all security encryption techniques and open the door to hackers. Here's how to protect ...

SearchDataCenter

• IBM Power9 servers seek market inroads to AI, cloud

IBM follows up its first Power9 server with a raft of systems designed to appeal to a wider array of markets -- most notably, AI ...

• Evaluate read-intensive and write-intensive SSD use cases

Consider write wear, performance and other factors when choosing between read-intensive, write-intensive and mixed-use ...

• Some hyper-converged infrastructure use cases pose pitfalls

Hyper-converged infrastructure adoption is skyrocketing, but that doesn't mean that the technology is the best choice for every ...

SearchContentManagement

• Content management in the cloud a main theme in 2018

The future of content management resides in the cloud and with AI, as several 2018 conferences will assure you.

• Six things to know about today's SharePoint implementations

As companies migrate their on-premises Microsoft SharePoint sites to the cloud, here are some things they should know about the ...

• Upgrades for the SharePoint Online portal

As more organizations migrate SharePoint sites to the cloud, Microsoft has increased at-a-glance dashboard data and analytics to ...

SearchHRSoftware

• Social media key to future of learning and development

Employees aren't just wasting time on social media; some are learning the career skills that will take them to the next level. ...

• Limeade releases new employee engagement software

HR tech news roundup: Employee engagement software vendor Limeade expands platform and releases inclusion module for large ...

• Don't overlook the many benefits of Microsoft Excel for HR

The maligned spreadsheet tool is no substitute for enterprise apps like HRMS and people analytics, but it will do in a pinch and ...

Close