A programmer mentioned a useful utility he once had that would display only the items in your current directory that you've modified that day. I jacked his idea and came up with this one-liner (and alias) and figured I'd share:
Might be a bit of a hack and could probably be prettier. But here is a wget command to download all pdfs within one layer from a URL and put them in a directory.
-r --level=1 = recurse through directories, 1 link deep
-H = host spanning
-nd = don't create directories
-N = new copy
-np = stay in specified directory, no parents
-A = filetype filter
-e robots=off = ignore all rules
--wait 0.25 = rate limiting to reduce server load
You're a horrible person ignoring the robots.txt :p ;)
But you reminded me of: Lynx -listonly -dump $URL > $FILE To get all 'Links' of a website and place them in a file. Also Lynx is a good browser for cheap bash scripts.
You're right, now that I look at it again that looks pretty bad. I can't remember why I put that in. I think that I was basically looking around on different forums for examples until I found things that did what I wanted and patched them together.
I won't use that bit any more. I remember putting the wait command in to not be a complete asshole.
it was a joke :p but for admin's love its just best to respect the robots.txt unless you really want the stuff then go ahead and the wait command is also for you're own best interest. You don't want an warning that you need to resolve an anti-robot puzzle.
makes a copy of /home/user/sync_src/ folder to /home/user/sync_dst/ with rsync and produces a log on /home/user/logs/ of the operation with date and time on the filename
Makes an archive of home/user/sync_dst/ folder with 7z for good compression (or use tar a -pcz when persevering file permission is required) with date and time on filename archive
Keep the 6 most recent archives on the /home/user/sync_archives/ folder and deletes the old ones
I think this one is actually from Ubuntu... I have it in an alias (alert) and I use it whenever I run long commands so I get notified when its finished. Then I don't have to keep my eye on it for the most part: