wget cron job

[bschultz]

Hi. I have a bit of a problem. I need to download a bunch of mp3 on a weekly basis for my job. At work, we use a Mac OS X server. I can't get wget to work on it during a cron job. If I type it in on the Mac terminal, everything works fine.

I can however, make my Linux machine at home download the files without problem.

Here's my question. Is it possible to then take those downloaded files on my server, and send them off to another (off-site) server (that is behind a firewall)?

These can't be sent as a tar archive either...they need to be to full files.

Thanks!

Brian

[Ben]

Sure you can, but you have to configure the firewall to allow your incoming traffic on the service's port you want to send the data.
Also the firewall must forward this incoming traffic on that port to the machine you want to reach.

[bschultz]

I control the firewall, so that's not a problem...can you point me in the direction of the code I will need to use?

Thanks for the help.

[Ben]

unfortunately I can't give you "code", just some hints

Things are quite easy, at first you could check why wget is not working inside the cron. Maybe you could source your users profile,
e.g. * * * * * * . /root/.bash_profile && wget ...
This should set environment vars etc. But this file(s) differ from distro to distro, and actually I have no clue what it is in MacOS. Then you could write stdout & stderr to a file in the cron, maybe wget is telling you what its problem is:
* * * * * wget ... 2>&1 >> /temp/wget.log

If you want to upload files from @home, you have to check with which method you want to upload the files, e.g. ftp(s) (ftps != sftp, sftp is based on sshd) (needs ftp server in the company site), scp (needs sshd on the company site). Based on this you have to define a firewall rule allowing these ports from outside, if you have a static IP only from that IP, if you have a dynamic one, from all IPs or a possible range. Then forward these packets to the local IP of the machine where the accepting daemon (ftp/ssh, etc) is running.

[bschultz]

I found some stuff on Google today...just had to figure out how to word my search!... Here's what I camp up with.

cat /localdirectory/file.mp3  |ssh username:password@domain.com "cd /remotedirectory; cat > file.mp3"

That works...except it asks me for a password (which needs to be fixed if this is run as a cron) and I can't seem to specify the whole directory...just individual files (which will change names on a weekly basis).

Why the cron doesn't work on the mac? I don't know for sure other than to say, it's a M-A-C!

[Ben]

well you could write a shellscript with a for loop, or you do it with scp:

scp ./sourcedir/* user@your.server.com:/destination/dir

[bschultz]

That worked, but still asked for a password. Is there a way to pass the password in the command string? No, I don't have keys enabled on the Mac, so I can't use that.

[Ben]

Well either you use certificate based authentication (without a pw for the cert), but be aware than, that if sb. steals the cert from your home pc could easily authenticate (except your firewall allows only incoming transfer from the static IP of your homepc, than an attacker would additionaly have to fake this).

Another, but kind of ugly way is, to catch the question it i think it was expect and than paste it anyhow. but I would prefer certs.

[bschultz]

I've tried to delay the password being echoed...without any success.

Can you run public/private keys AND a username and password on the same system? Maybe that's my best bet. I just didn't want to have to give everyone here in the building the keys...changing the password is easier than getting the new keys to everyone. Maybe I can just do the keys for my home server.

Thanks again for all the help on this Ben!

[bschultz]

Nevermind...I got it figured out here:

http://www.astro.caltech.edu/~mbonat...sword_ssh.html

Thanks again for all the help!