Author Topic: My Backup Script Fails  (Read 1745 times)

0 Members and 1 Guest are viewing this topic.

Offline Eniliad

  • Sword And Shield Of The Innocent
  • The Beast
  • *****
  • Posts: 1265
  • Gender: Male
  • Perpetually horny cock-slave
My Backup Script Fails
« on: March 05, 2012, 06:03:38 pm »
I'm learning Linux, and part of that, for me, includes bash scripting. If you're not familiar, it's basically executing a number of Terminal (command prompt) commands in sequence. There's also some basic flow control that lets you write simple programs. Anyway, while learning it, I decided to try writing some marginally useful things as exercises. One of these was a simple backup script. It would create a folder on my external hard drive based on the date, and copy the contents of my Pictures, Music, etc... folders to identically named directories inside the dated folder. Well, that worked, but the total size of the directory created is about 67 GB. I had an idea: What if I compress the folder? Like with 7zip?

And this is where things start to fall apart.

I tried using 7zip on the directory in question, but it didn't like working on an external hdd (it spat back a "no room on disk" error- there's 700 GB of free space left) So, I did this: I had the script first copy the files over to my Windows hard drive, then 7Zip it, then move the .7z file onto the external hard drive, then delete all the backup files from the WinBox, as I call it.

Well, surprisingly, the problem wasn't with the code that I wrote; it was written perfectly. However... there were other issues. The original script takes about 20 minutes to copy all my shit onto the external hdd. This script took well over five hours. 99% of that time was spent running 7zip. In fact, it took so long that my external hard drive unmounted, or shut down to conserve power, whatever - so when the script tried to locate the final backup directory, it utterly failed to do so, and I was left with a massive .7z on my WinBox.

To make matters even funnier, the final result of my ultra-compression? The .7z was only 6 GB smaller than just moving the folders as they were. 6 GB may seem like a lot, but the difference between 61 and 67 isn't much compared to the fact that I was unable to use my computer for hours (CPU was at 99% usage).

I know, I know, there are much better backup solutions that just writing some bash script. I just wanted to test my own writing skills. THAT, I guess, was a success, but the degree to which my little experiment failed was hilarious.
<Miles> "If dildoes are outlawed then only outlaws will have dildoes."
Quote from: Mlle Antéchrist
Yeah, gays cause hurricanes, tits cause earthquakes, and lack of prayer causes tornadoes. Learn to science, people.
Quote from: Mlle Antéchrist
Porn peddlers peddling pedal porn? My life is complete.

Offline Dan

  • Apprentice
  • **
  • Posts: 99
  • Gender: Male
Re: My Backup Script Fails
« Reply #1 on: March 07, 2012, 02:47:11 pm »
I tried using 7zip on the directory in question, but it didn't like working on an external hdd (it spat back a "no room on disk" error- there's 700 GB of free space left)
Your problem here is that your external disk is almost certainly formatted as FAT32, which has a limit of 4GB on file sizes. The error message is of course incorrect, but it's the usual message you see when the file size limit rears its head.

You can solve the problem in two ways: reformat to NTFS or reformat to one of Linux's native filesystems such as ext4. But Linux support for writing to NTFS is dubious at best, whereas Windows support for ext4 is non-existent save for third party tools.

If reformatting is not an option, you could try compressing on a folder by folder basis to give smaller files. That might also help narrow down why your compressed archive isn't compressing very much and where the speed bottleneck is.


If you're more concerned by backup time than the required space, look into rsync. It will do incremental backups similar to Windows' xcopy or robocopy.

Offline Captain Jack Harkness

  • Petter, Brony, and All-Around Cartoon Addict
  • The Beast
  • *****
  • Posts: 2868
  • Gender: Male
  • Or as a friend calls him, Captain Jack Hotness!
Re: My Backup Script Fails
« Reply #2 on: March 07, 2012, 03:31:31 pm »
Eniliad, you are a magnificent bastard.  Thank you for writing this post.  My day was truly brightened. ^_^
My friend's blog.  Check it out!

I blame/credit The Doctor with inspiring my name change.