Compressing files and directory structures with limited disk space

This is a general Unix tip -- you can use this method to compress all standard Unix files and also Oracle 7.x and above .dbf (datafiles) and .dmp (export dumpfiles) files. This tip works solely on Unix platforms -- I have tried it on Solaris 2.x, HP-UX 10.x, DEC Tru-64 4.2g+ and Redhat Linux 7.3. As such, it is not intended to work with MS SQL Server. I have not played with DB2 and cannot verify if the exported files from DB2 will support compression through tar or compress.

Although storage capacity is growing day-by-day, there are still occasions when there still just isn't enough. I have found this to be true especially when I need to send or archive a complete directory structure and haven't enough disk space to perform a tar and then a compress. This is due to the fact that if you perform a tar, you use as much space as you already used for the files and then to compress, you use a percentage of that space again for the temporary compressed file.

This simple method of combining the two commands eliminates the need for so much temporary disk usage.

As with many Unix commands, you are able to stream the output of a command to standard output (utilising the /tmp partition or swap). In this case, we send the output of tar to standard output and pipe this through to compress to create our final compressed tar file.

tar cvf - /root1/files/* | compress - > /root1/archive/outfile.tar.Z
Similarly, when restoring these files, we can do this in

    Requires Free Membership to View

reverse, bearing in mind that we would now use the zcat command instead of uncompress as zcat streams the contents of the compressed file. We then come out with
zcat outfile.tar.Z | tar xvf -

It is important to note that when compressing the files, the output file should NOT be placed in that directory tree. If it is, then the tar command includes a zero byte size file of that compressed file and on restore, that file is then extracted, thereby losing your real compressed tar file contents. For example,

root> ls -l 
-rw--r--r--   1   root other  768  Aug 6  13:50    file1
-rw--r--r--   1   root other  712  Aug 6  13:50    file2
-rw--r--r--   1   root other  256  Aug 6  13:50    file3
root> tar cvf - ./* | compress ./files.tar.Z
root> rm file1 file2 file3
root> ls -l 
-rw--r--r--   1   root other 1064  Aug 6  13:51    files.tar.Z
root> zcat ./files.tar.Z | tar xvf -
root> ls -l 
-rw--r--r--   1   root other  768  Aug 6  13:50    file1
-rw--r--r--   1   root other  712  Aug 6  13:50    file2
-rw--r--r--   1   root other  256  Aug 6  13:50    file3
-rw--r--r--   1   root other    0  Aug 6  13:50    files.tar.Z
So, it's best to put the tar.Z file in another directory than the one you are archiving.

For More Information

  • Feedback: E-mail the editor with your thoughts about this tip.
  • More tips: Hundreds of free Oracle tips and scripts.
  • Tip contest: Have an Oracle tip to offer your fellow DBAs and developers? The best tips submitted will receive a cool prize -- submit your tip today!
  • Ask the Experts: Our SQL, database design, Oracle, SQL Server, DB2, metadata, and data warehousing gurus are waiting to answer your toughest questions.
  • Forums: Ask your technical Oracle questions--or help out your peers by answering them--in our active forums.
  • Best Web Links: Oracle tips, tutorials, and scripts from around the Web.

This was first published in September 2002

There are Comments. Add yours.

TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.