[FIXED] When uploading new files to FTP server, how to prevent reupload of files that were deleted on the server meanwhile

Issue

I need to automate the upload of some files from client PCs to a central server. We're building central statistics for an online gaming community, processing game replay files.

• target is my own small VPS server running ubuntu
• upload file size 2-3MB
• 20-40 different clients running windows spread around the globe
• I expect ~6GB of wanted data to be uploaded over the course of 7 weeks (a season in our game) and 5-10x that amount of "unwanted" data.

The files are processed on the server, and then they're not required anymore, and ought to be deleted to not run out of disk space eventually. I also only need some of the files, but due to the files requiring very complex processing including decryption, so i can only determine that after the server processed it.

My initial idea was to use a scriptable client such as WinSCP, and use some Windows scheduler entry to automate it. WinSCP documentation looks very nice. I am a bit hesitant because I see the following problems:

• after deletion on the server, how to prevent re-upload ?
• ease of setup to technical novices
• reliability of the solution

I was thinking maybe someone has done the same before and can give some advice.

Solution

There's article on WinSCP site that deals with all this:
How do I transfer new/modified files only?

For advanced logic, like yours, it uses PowerShell script with use of WinSCP .NET assembly.

• Particularly, there is a section that you will be interested in: Remembering the last timestamp – It shows how to remember the timestamp of the last uploaded file, so that the next time you will transfer only newer files, even if the previously uploaded files are not on the server anymore.

  The example is for downloads with Session.GetFiles, but it will with small changes work for uploads with Session.PutFiles too.

• It also points to another article: Remember already downloaded files so they are not downloaded again, which shows another method – To store names of already transferrer file to a file and use it the next time to decide, which files are new.

Answered By - Martin Prikryl

Answer Checked By - Katrina (PHPFixing Volunteer)