Compressing files and directory structures with limited disk space

This tip helps anyone on Unix who needs to compress files or directory structures with limited disk space.

This is a general Unix tip -- you can use this method to compress all standard Unix files and also Oracle 7.x and above .dbf (datafiles) and .dmp (export dumpfiles) files. This tip works solely on Unix platforms -- I have tried it on Solaris 2.x, HP-UX 10.x, DEC Tru-64 4.2g+ and Redhat Linux 7.3. As such, it is not intended to work with MS SQL Server. I have not played with DB2 and cannot verify if the exported files from DB2 will support compression through tar or compress.

Although storage capacity is growing day-by-day, there are still occasions when there still just isn't enough. I have found this to be true especially when I need to send or archive a complete directory structure and haven't enough disk space to perform a tar and then a compress. This is due to the fact that if you perform a tar, you use as much space as you already used for the files and then to compress, you use a percentage of that space again for the temporary compressed file.

This simple method of combining the two commands eliminates the need for so much temporary disk usage.

As with many Unix commands, you are able to stream the output of a command to standard output (utilising the /tmp partition or swap). In this case, we send the output of tar to standard output and pipe this through to compress to create our final compressed tar file.

tar cvf - /root1/files/* | compress - > /root1/archive/outfile.tar.Z
Similarly, when restoring these files, we can do this in reverse, bearing in mind that we would now use the zcat command instead of uncompress as zcat streams the contents of the compressed file. We then come out with
zcat outfile.tar.Z | tar xvf -

It is important to note that when compressing the files, the output file should NOT be placed in that directory tree. If it is, then the tar command includes a zero byte size file of that compressed file and on restore, that file is then extracted, thereby losing your real compressed tar file contents. For example,

root> ls -l 
-rw--r--r--   1   root other  768  Aug 6  13:50    file1
-rw--r--r--   1   root other  712  Aug 6  13:50    file2
-rw--r--r--   1   root other  256  Aug 6  13:50    file3
root> tar cvf - ./* | compress ./files.tar.Z
root> rm file1 file2 file3
root> ls -l 
-rw--r--r--   1   root other 1064  Aug 6  13:51    files.tar.Z
root> zcat ./files.tar.Z | tar xvf -
root> ls -l 
-rw--r--r--   1   root other  768  Aug 6  13:50    file1
-rw--r--r--   1   root other  712  Aug 6  13:50    file2
-rw--r--r--   1   root other  256  Aug 6  13:50    file3
-rw--r--r--   1   root other    0  Aug 6  13:50    files.tar.Z
So, it's best to put the tar.Z file in another directory than the one you are archiving.

For More Information

  • Feedback: E-mail the editor with your thoughts about this tip.
  • More tips: Hundreds of free Oracle tips and scripts.
  • Tip contest: Have an Oracle tip to offer your fellow DBAs and developers? The best tips submitted will receive a cool prize -- submit your tip today!
  • Ask the Experts: Our SQL, database design, Oracle, SQL Server, DB2, metadata, and data warehousing gurus are waiting to answer your toughest questions.
  • Forums: Ask your technical Oracle questions--or help out your peers by answering them--in our active forums.
  • Best Web Links: Oracle tips, tutorials, and scripts from around the Web.

Dig Deeper on Oracle database backup and recovery

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.