Download a list of files

Posted on August 27, 2015  •  1 minutes  • 204 words  •  Suggest Changes

Downloading ~150 links in a text file, sounds like a horror show, right ? Not really ! The wget application in any linux distro will help you out.

wget -i file_with_downloads

Since my links where coming from one source, wget told me it was  “reusing the connection” (keep-alive ?) After some time however the server on the other side decided I had downloaded more then enough and killed some of the connections, so some files where not downloaded. Now the problem even got worse, since I had to check if wget had downloaded them or not. There might be an easier solution, but I took my best bash from the closet and came up with :

while read p; do
  FILE=`echo $p | cut -c44-100`
  if [ -f $FILE ];
  then
    echo "echo OK"
  else
    echo "downloading $FILE"
    wget $p &
  fi;
done <download_list

This started some 10 wgets simultaneously, which is not all that great, but it worked. Probably some better ways exists to tackle this problem, but if it works it works! 🙂

happy downloading.

notice : the cut -c44-100 actually cuts off the base url, this will be different from what you download. (unless your downloading the same files, highly unlikely)