Tag: wget

Quick Shell Script to Download Files

Posted on September 11th, 2011 by tom

Share

Here is a shell script that I ran from the command line to do some file downloads from S3. Note the sh Parameter Expansion to parse the url.

for url in 'https://example-bucket.s3.amazonaws.com:443/example.tomswebhosting.com.zip?Expires=1234567890&AWSAccessKeyId=12345678901234567890&Signature=123456789012345678901234567%3D' 'https://example-bucket.s3.amazonaws.com:443/example.tomswebhosting.com.zip?Expires=1234567890&AWSAccessKeyId=12345678901234567890&Signature=123456789012345678901234567%3D';

do

    file=${url##*443/}; # get everything after "433/" in the url
    file=${file%%\?*}; # get everything before "?" in the url

    wget --no-check-certificate "$url" -O $file;

    done

Comments (0)       
Tags: parameter expansion, remember, script, sh, wget

Using Wget To Archive A Website

Posted on November 30th, 2009 by tom

Share

I needed to grab a simple archive of a website and found Wget does an adequate job. I wanted the website and any media files that it pointed to but not any other external websites.

wget -Hkmp -Dyour.website.here,external.website.1.here,external.website.2.here http://your.website.here

Wget is a complex tool so you'll definitely want to read up on it before you use it.

Comments (0)       
Tags: archive website, code, software, wget