Actually it is not absurd to use ftp to get your files from your server.
I am assuming that you have a router the connects both of your computers to the net. In that case, when you want to access your files, just type in your internal ip address of the server. This is probably in the form of 192.168.1.x. Use ifconfig on your server to find out your ip address.
I believe that some routers are also intelligent enough to figure out when a request is being made from within the network, and they treat it as local. This means that even if you access your server though a domain name, such as yourdomain.com, and you're on your home network, the router will figure this out, and connect you with your server directly, bypassing your service provider.
As for samba, you can configure it for authentication, and as long as you don't open up the samba port to the outside world, it can be relatively safe.