hey guys
basically i am looking for something that can scan a ftp server for new files at a particular time and then retrieve them.
its for a seedbox so i would like it to scan my downloads drive and fetch any new files on the box.
I do this a lot using kermit with a script like this one: http://www.columbia.edu/kermit/ftp/scripts/ckermit/ftpsyncdown
1 Like
awesome.
had a quick look can you spare some time to elaborate on how you do it?
Sure, let's say I have a download I need to run several times per day. I need to recursively download all available files on the FTP server, then delete them after they have been successfully downloaded (if you don't need to delete, it's just an option on one line, you can remove it).
Here's a generi-fied version of my script:
#!/usr/bin/kermit
.host = 0.0.0.0 ; Change to desired host
.rdirectory = / ; Change to desired host directory
.ldirectory = /my/dir/ ; Change to desired local directory
.logfile := \v(home)ftpsync.log ; Change to desired logfile name
; End of parameter defintions.
set transaction-log brief ; Choose brief transaction-log format
set exit warning off ; No "OK to exit?" prompts
set quiet on ; Suppress progress messages
lcd \m(ldirectory) ; CD to desired local directory
if fail exit 1 "LCD failed - \m(ldirectory)" ; Make sure we did
ftp open \m(host) /USER:myusername /PASSWORD:mypassword ; Open and check FTP connection
if fail exit 1 Login failed
if not \v(ftp_loggedin) exit 1 "Login failed"
log transactions \m(logfile) ; Start log
ftp cd \m(rdirectory) ; CD to desired server directory
if fail exit 1 "FTP CD failed - \m(rdirectory)"
ftp mget /recursive /delete /collision:overwrite * ; Get all server files in update mode, delete after downloading
if fail exit 1 "Mget failed" ; Check for errors
bye ; Disconnect from server
close transaction-log ; Close log
exit 0I need to run the script from cron, but I want to make sure I don't spin up multiple instances of the script in case it runs long on a big download, so I have a bash script wrapper that checks to see if the kermit script is already running, and only runs the kermit script is it is not already running:
#!/bin/bash
FIND_PROC=`ps -ef |grep my_script_name | awk '{if ($8 !~ /grep/) print $2}'`
# if FIND_PROC is empty, the process is not currently running; start it
if [ -z "${FIND_PROC}" ]; then
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd ${DIR}
./my_script_name
fi
exit 0This wrapper expects the kermit script to be in it's working directory, so we need to put them both in the same place.
I have my cron job set up like this:
0 3,9,15,21 * * * /path/to/script/kermit_cron_runner.shThat's pretty much it, it just hums along doing it's thing. My actual setup has the kermit script as the first step in a process that then goes on to unpack tar files that are downloaded, and do stuff with the contents therein.
Holy shit reading that I realise I still have so much to learn lol.
Thanks heaps will have a fiddle and see how I go.
No problem, it's not as burly as it looks at first glance. You'll only need to change a few lines to make it work for you. If you get stuck on anything, give me a shout.
cheers for that it looks like i got the first step working. now just have to automate it with cron.
is there a way i can get it to use lftp instead of generic ftp? i really need multi segmenting to make it quicker.
Lftp is a newer alternative to kermit with a bunch of additional features, so you could definitely use it in this way, but I haven't used lftp, I'm not sure what the differences in syntax would be. I imagine you could translate this kermit script over to lftp pretty easily.
1 Like
cheers
will see how this goes. i have it set to run during the night so hopefully all goes well
That's pretty handy. Bookmarked.
Yeah I have your script set up and will see how it goes.
Just hope it all is finished before I wake up. I have very limited bandwidth here :(
You should be able to use wget to do this.
Some entries from the manual.
-m
--mirror
Turn on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. It is currently equivalent to -r -N -l
inf --no-remove-listing.
-np
--no-parent
Do not ever ascend to the parent directory when retrieving recursively. This is a useful option, since it guarantees that only the files below a certain hierarchy will be downloaded.
-c
--continue
Continue getting a partially-downloaded file. This is useful when you want to finish up a download started by a previous instance of Wget, or by another program. For instance:
wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
--user=user
--password=password
Specify the username user and password password for both FTP and HTTP file retrieval. These parameters can be overridden using the --ftp-user and --ftp-password options for FTP connections and
the --http-user and --http-password options for HTTP connections.
--ask-password
Prompt for a password for each connection established. Cannot be specified when --password is being used, because they are mutually exclusive.
-P prefix
--directory-prefix=prefix
Set directory prefix to prefix. The directory prefix is the directory where all other files and subdirectories will be saved to, i.e. the top of the retrieval tree. The default is . (the current directory).
-q
--quiet
Turn off Wget's output.
--ask-password
Prompt for a password for each connection established. Cannot be specified when --password is being used, because they are mutually exclusive.
1 Like