#1  
Old 10th September 2007, 21:27
bschultz bschultz is offline
Senior Member
 
Join Date: Jul 2006
Posts: 221
Thanks: 11
Thanked 10 Times in 10 Posts
Default wget cron job

Hi. I have a bit of a problem. I need to download a bunch of mp3 on a weekly basis for my job. At work, we use a Mac OS X server. I can't get wget to work on it during a cron job. If I type it in on the Mac terminal, everything works fine.

I can however, make my Linux machine at home download the files without problem.

Here's my question. Is it possible to then take those downloaded files on my server, and send them off to another (off-site) server (that is behind a firewall)?

These can't be sent as a tar archive either...they need to be to full files.

Thanks!

Brian
Reply With Quote
Sponsored Links
  #2  
Old 11th September 2007, 08:49
Ben Ben is offline
Moderator
 
Join Date: Jul 2006
Posts: 1,029
Thanks: 7
Thanked 62 Times in 56 Posts
Default

Sure you can, but you have to configure the firewall to allow your incoming traffic on the service's port you want to send the data.
Also the firewall must forward this incoming traffic on that port to the machine you want to reach.
Reply With Quote
  #3  
Old 11th September 2007, 12:40
bschultz bschultz is offline
Senior Member
 
Join Date: Jul 2006
Posts: 221
Thanks: 11
Thanked 10 Times in 10 Posts
Default

I control the firewall, so that's not a problem...can you point me in the direction of the code I will need to use?

Thanks for the help.
Reply With Quote
  #4  
Old 11th September 2007, 15:21
Ben Ben is offline
Moderator
 
Join Date: Jul 2006
Posts: 1,029
Thanks: 7
Thanked 62 Times in 56 Posts
Default

unfortunately I can't give you "code", just some hints

Things are quite easy, at first you could check why wget is not working inside the cron. Maybe you could source your users profile,
e.g. * * * * * * . /root/.bash_profile && wget ...
This should set environment vars etc. But this file(s) differ from distro to distro, and actually I have no clue what it is in MacOS. Then you could write stdout & stderr to a file in the cron, maybe wget is telling you what its problem is:
* * * * * wget ... 2>&1 >> /temp/wget.log

If you want to upload files from @home, you have to check with which method you want to upload the files, e.g. ftp(s) (ftps != sftp, sftp is based on sshd) (needs ftp server in the company site), scp (needs sshd on the company site). Based on this you have to define a firewall rule allowing these ports from outside, if you have a static IP only from that IP, if you have a dynamic one, from all IPs or a possible range. Then forward these packets to the local IP of the machine where the accepting daemon (ftp/ssh, etc) is running.
Reply With Quote
The Following User Says Thank You to Ben For This Useful Post:
bschultz (11th September 2007)
  #5  
Old 11th September 2007, 15:41
bschultz bschultz is offline
Senior Member
 
Join Date: Jul 2006
Posts: 221
Thanks: 11
Thanked 10 Times in 10 Posts
Default

I found some stuff on Google today...just had to figure out how to word my search!... Here's what I camp up with.

Code:
cat /localdirectory/file.mp3  |ssh username:password@domain.com "cd /remotedirectory; cat > file.mp3"
That works...except it asks me for a password (which needs to be fixed if this is run as a cron) and I can't seem to specify the whole directory...just individual files (which will change names on a weekly basis).

Why the cron doesn't work on the mac? I don't know for sure other than to say, it's a M-A-C!

Last edited by bschultz; 11th September 2007 at 15:43.
Reply With Quote
  #6  
Old 11th September 2007, 18:48
Ben Ben is offline
Moderator
 
Join Date: Jul 2006
Posts: 1,029
Thanks: 7
Thanked 62 Times in 56 Posts
Default

well you could write a shellscript with a for loop, or you do it with scp:

scp ./sourcedir/* user@your.server.com:/destination/dir
Reply With Quote
  #7  
Old 11th September 2007, 20:48
bschultz bschultz is offline
Senior Member
 
Join Date: Jul 2006
Posts: 221
Thanks: 11
Thanked 10 Times in 10 Posts
Default

That worked, but still asked for a password. Is there a way to pass the password in the command string? No, I don't have keys enabled on the Mac, so I can't use that.
Reply With Quote
  #8  
Old 12th September 2007, 08:45
Ben Ben is offline
Moderator
 
Join Date: Jul 2006
Posts: 1,029
Thanks: 7
Thanked 62 Times in 56 Posts
Default

Well either you use certificate based authentication (without a pw for the cert), but be aware than, that if sb. steals the cert from your home pc could easily authenticate (except your firewall allows only incoming transfer from the static IP of your homepc, than an attacker would additionaly have to fake this).

Another, but kind of ugly way is, to catch the question it i think it was expect and than paste it anyhow. but I would prefer certs.
Reply With Quote
  #9  
Old 12th September 2007, 18:25
bschultz bschultz is offline
Senior Member
 
Join Date: Jul 2006
Posts: 221
Thanks: 11
Thanked 10 Times in 10 Posts
Default

I've tried to delay the password being echoed...without any success.

Can you run public/private keys AND a username and password on the same system? Maybe that's my best bet. I just didn't want to have to give everyone here in the building the keys...changing the password is easier than getting the new keys to everyone. Maybe I can just do the keys for my home server.

Thanks again for all the help on this Ben!
Reply With Quote
  #10  
Old 12th September 2007, 18:53
bschultz bschultz is offline
Senior Member
 
Join Date: Jul 2006
Posts: 221
Thanks: 11
Thanked 10 Times in 10 Posts
 
Default

Nevermind...I got it figured out here:

http://www.astro.caltech.edu/~mbonat...sword_ssh.html

Thanks again for all the help!
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
mail_logs.php cron job causing high server loads? jameso General 15 3rd April 2007 11:18
Dynamic IP and cron job satimis Server Operation 11 19th December 2006 14:34
cron job modified after upgrading V 2.2.5 jfruano Installation/Configuration 1 21st July 2006 12:31
Scheduling a cron job Rocky Installation/Configuration 1 5th March 2006 01:06
How to disable the shell cron jobs? raymov General 3 1st November 2005 04:43


All times are GMT +2. The time now is 05:33.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.