Download a list of files
Posted on August 27, 2015 • 1 minutes • 204 words • Suggest Changes
Downloading ~150 links in a text file, sounds like a horror show, right ? Not really ! The wget application in any linux distro will help you out.
wget -i file_with_downloads
Since my links where coming from one source, wget told me it was “reusing the connection” (keep-alive ?) After some time however the server on the other side decided I had downloaded more then enough and killed some of the connections, so some files where not downloaded. Now the problem even got worse, since I had to check if wget had downloaded them or not. There might be an easier solution, but I took my best bash from the closet and came up with :
while read p; do FILE=`echo $p | cut -c44-100` if [ -f $FILE ]; then echo "echo OK" else echo "downloading $FILE" wget $p & fi; done <download_list
This started some 10 wgets simultaneously, which is not all that great, but it worked. Probably some better ways exists to tackle this problem, but if it works it works! 🙂
notice : the cut -c44-100 actually cuts off the base url, this will be different from what you download. (unless your downloading the same files, highly unlikely)