uscheduler/examples/Native-Sync/README.md

Native PowerShell Sync Script

Version: 1.0.0 Last Updated: 2026-01-26

Overview

Production-ready file synchronization solution using pure PowerShell with no external dependencies. Supports Mirror, Update, and TwoWay sync modes with filtering, progress reporting, and secure credential management.

Features

• ✅ No External Dependencies - Pure PowerShell implementation
• ✅ Multiple Sync Modes - Mirror, Update, and TwoWay synchronization
• ✅ Flexible Comparison - Compare by modification time and file size
• ✅ Deletion Policies - Recycle Bin, Permanent delete, or Versioning
• ✅ Filter Support - Include/exclude patterns for files and folders
• ✅ Remote Storage Support - Sync to UNC shares with secure credential management
• ✅ Progress Reporting - File-by-file progress output
• ✅ Dry Run Mode - Test sync without making changes
• ✅ Detailed Logging - Comprehensive logging with timestamps and severity levels
• ✅ Lock Files - Prevents concurrent execution
• ✅ Flexible Scheduling - Schedule sync by month, weekday, and time

Requirements

System Requirements

• Windows with PowerShell 5.1 or later
• Administrator privileges (for network share authentication)
• Network access to target share (if using UNC paths)

Dependencies

• SchedulerTemplate.psm1 module (located in parent directory)
• scriptsettings.json configuration file

File Structure

Native-Sync/
├── native-sync.bat              # Batch launcher with admin check
├── native-sync.ps1              # Main PowerShell script
├── scriptsettings.json          # Configuration file
└── README.md                    # This file

Installation

1. Copy Files

   # Copy the entire Native-Sync folder to your desired location
   # Ensure SchedulerTemplate.psm1 is in the parent directory
   

2. Configure Settings

   Edit scriptsettings.json with your environment settings:

   {
     "syncMode": "Mirror",
     "folderPairs": [
       {
         "left": "C:\\Source",
         "right": "D:\\Backup"
       }
     ]
   }
   

3. Setup Credentials (for UNC paths)

   If syncing to a network share, create a Machine-level environment variable:

   # Create Base64-encoded credentials
   $username = "DOMAIN\user"
   $password = "your-password"
   $creds = "$username:$password"
   $encoded = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($creds))
   
   # Set Machine-level environment variable
   [System.Environment]::SetEnvironmentVariable("YOUR_ENV_VAR_NAME", $encoded, "Machine")
   

4. Test Manual Execution

   # Run as Administrator
   .\native-sync.bat
   # or
   .\native-sync.ps1
   
   # Test with dry run first (set dryRun: true in scriptsettings.json)
   

Configuration Reference

Sync Modes

Mode	Description	LeftOnly	LeftNewer	RightNewer	RightOnly
Mirror	Make right identical to left	Copy to right	Update right	Update right	Delete from right
Update	Copy new/updated to right only	Copy to right	Update right	Skip	Skip
TwoWay	Propagate changes both ways	Copy to right	Update right	Update left	Copy to left

Schedule Settings

Property	Type	Description	Example
runMonth	array	Month names to run. Empty = every month	["January", "June"] or []
runWeekday	array	Weekday names to run. Empty = every day	["Monday", "Friday"]
runTime	array	UTC times to run (HH:mm format)	["00:00", "12:00"]
minIntervalMinutes	number	Minimum minutes between runs	10

Sync Settings

Property	Type	Required	Description
syncMode	string	Yes	Sync mode: Mirror, Update, or TwoWay
compareMethod	string	No	Comparison method: TimeAndSize (default)
deletionPolicy	string	No	Deletion handling: RecycleBin (default), Permanent, Versioning
versioningFolder	string	No	Path for versioning when deletionPolicy is Versioning
folderPairs	array	Yes	Array of {left, right} folder pairs to sync

Filter Settings

Property	Type	Description
filters.include	array	Patterns to include (use ["*"] for all)
filters.exclude	array	Patterns to exclude

Pattern Syntax:

• Directory patterns end with \ (e.g., \System Volume Information\)
• File patterns use wildcards (e.g., *.tmp, thumbs.db)
• Prefix with *\ to match in any subdirectory (e.g., *\thumbs.db)

Default Excludes:

• \System Volume Information\
• \$Recycle.Bin\
• \RECYCLE?\
• \Recovery\
• *\thumbs.db

Note: Filter matching uses PowerShell's -like operator. For complex filtering needs, test with dryRun: true first to verify expected behavior.

Options

Property	Type	Default	Description
excludeSymlinks	bool	true	Skip symbolic links
ignoreTimeShift	bool	false	Ignore 1-hour DST time differences
showProgress	bool	true	Display file-by-file progress
dryRun	bool	false	Simulate sync without changes

Network Settings

Property	Type	Required	Description
nasRootShare	string	No	UNC path for authentication
credentialEnvVar	string	No*	Environment variable name (*Required for UNC paths)

Usage

Manual Execution

Using Batch File (Recommended):

REM Right-click and select "Run as administrator"
native-sync.bat

Using PowerShell:

# Run as Administrator
.\native-sync.ps1

# With verbose output
.\native-sync.ps1 -Verbose

Automated Execution

The script supports automated execution through the UScheduler service:

# Called by scheduler with -Automated flag
.\native-sync.ps1 -Automated -CurrentDateTimeUtc "2026-01-26 00:00:00"

When -Automated is specified:

• Schedule is enforced (month, weekday, time)
• Lock files prevent concurrent execution
• Interval checking prevents duplicate runs
• Logs are formatted for service logger (no timestamps)

How It Works

Sync Process Flow

1. Initialization

   • Load SchedulerTemplate.psm1 module
   • Load and validate scriptsettings.json
   • Parse sync mode, filters, and options

2. Pre-flight Checks

   • Authenticate to NAS share (if UNC)
   • Verify source paths exist
   • Create destination directories if needed

3. Scan Phase

   • Recursively scan source directory
   • Recursively scan destination directory
   • Apply include/exclude filters
   • Skip symlinks if configured

4. Comparison Phase

   • Compare files by modification time and size
   • Determine file status: Same, LeftOnly, LeftNewer, RightNewer, RightOnly
   • Build list of sync actions based on sync mode

5. Execution Phase

   • Sort actions: copies → updates → file deletes → directory deletes
   • Execute each action with progress reporting
   • Handle deletion policy (RecycleBin, Permanent, Versioning)

6. Summary

   • Display sync statistics
   • Report errors and warnings

Deletion Policies

Policy	Description
RecycleBin	Move deleted files to Windows Recycle Bin (recoverable)
Permanent	Delete files permanently (not recoverable)
Versioning	Move deleted files to versioning folder

Progress Output

[INFO] Processing Folder Pair 1
[INFO]   Left:  E:\Users\maksym\source
[INFO]   Right: \\server\share\source
[INFO] Scanning source directory...
[INFO]   Found 1234 files, 56 directories
[INFO] Scanning destination directory...
[INFO]   Found 1200 files, 54 directories
[INFO] Calculating sync actions...
[INFO]   45 actions to perform
[INFO] Executing sync actions...
[INFO] [COPY] docs/newfile.txt (15.2 KB)
[INFO] [UPDATE] src/main.cs (8.5 KB)
[WARNING] [DELETE] old/removed.txt (1.2 KB)

Sync Summary

========================================
SYNC SUMMARY
========================================
Start Time    : 2026-01-26 00:00:00
End Time      : 2026-01-26 00:05:30
Duration      : 0h 5m 30s
Status        : SUCCESS

Files Scanned : 1,234
Files Copied  : 45
Files Updated : 12
Files Deleted : 3
Files Skipped : 1,174
Bytes Copied  : 156.7 MB

Errors        : 0
Warnings      : 3
========================================

Logging

Log Levels

Level	Description	Color (Manual)
Info	Informational messages	White
Success	Successful operations	Green
Warning	Non-critical issues (deletions)	Yellow
Error	Critical errors	Red

Log Format

Manual Execution:

[2026-01-26 00:00:00] [Info] Native PowerShell Sync Started
[2026-01-26 00:00:01] [Success] All files synchronized

Automated Execution:

[Info] Native PowerShell Sync Started
[Success] All files synchronized

Exit Codes

Code	Description
0	Success (no errors)
1	Error occurred (config, paths, sync errors)

Troubleshooting

Common Issues

1. Module Not Found

Error: Failed to load SchedulerTemplate.psm1

Solution: Ensure SchedulerTemplate.psm1 is in the parent directory (../SchedulerTemplate.psm1)

2. Source Path Not Found

Error: Source path does not exist

Solution: Verify the left path in folderPairs exists and is accessible

3. UNC Path Authentication Failed

Error: Failed to connect to \\server\share

Solution:

• Verify credentialEnvVar is set in scriptsettings.json
• Verify environment variable exists at Machine level
• Verify credentials are Base64-encoded in format: username:password
• Test with: net use \\server\share manually

4. Lock File Exists

Guard: Lock file exists. Skipping.

Solution:

• Another instance is running, or previous run didn't complete
• Manually delete .lock file if stuck
• Check for hung PowerShell processes

5. Permission Denied

Error: Access to the path is denied

Solution:

• Run as Administrator
• Verify file/folder permissions
• Check if files are locked by other processes

Debug Mode

Run with verbose output:

.\native-sync.ps1 -Verbose

Test with dry run (set in scriptsettings.json):

{
  "options": {
    "dryRun": true
  }
}

Best Practices

1. Test First - Always test sync with dryRun: true in settings before actual execution
2. Backup First - Ensure you have backups before first sync
3. Verify Paths - Double-check source and destination paths
4. Monitor Logs - Check sync summaries regularly
5. Secure Credentials - Use Machine-level environment variables
6. Schedule Wisely - Run syncs during low-usage periods
7. Review Settings - Understand your sync mode implications
8. Test Restores - Periodically verify you can restore from synced data

Security Considerations

• Credentials are stored Base64-encoded in Machine-level environment variables
• Script requires Administrator privileges for network authentication
• Network credentials are passed to net use command
• Consider using dedicated sync account with minimal required permissions
• Sync data should be stored on secured network shares with appropriate ACLs

Performance Considerations

• Sync time depends on file count, size, and network speed
• Network speed is typically the bottleneck for UNC shares
• File scanning can take time for large directories
• Use filter rules to exclude unnecessary files
• Memory usage increases with file count during scanning
• Run during off-peak hours to minimize network impact

Scalability Limitations

This script performs full directory scans into in-memory hashtables before comparing files. This design means:

• High memory usage on very large directory trees (millions of files)
• No streaming or batching - all file metadata is loaded before sync begins
• Recommended limit - works well for typical backup scenarios (tens of thousands of files)

Future versions may introduce streaming/batching to improve scalability for very large datasets.

Version History

1.0.0 (2026-01-26)

• Initial release
• Pure PowerShell implementation
• Mirror, Update, TwoWay sync modes
• TimeAndSize comparison method
• RecycleBin, Permanent, Versioning deletion policies
• Include/exclude filter patterns
• Symlink exclusion option
• DST time shift handling
• Progress reporting with file-by-file output
• Dry run mode
• UNC share support with credential management
• Integration with SchedulerTemplate.psm1

Support

For issues or questions:

1. Check the Troubleshooting section
2. Review script logs for error details
3. Verify all Requirements are met

License

See LICENSE in the root directory.

Related Files

• ../SchedulerTemplate.psm1 - Shared scheduling and logging module
• scriptsettings.json - Configuration file
• native-sync.bat - Batch launcher
• native-sync.ps1 - Main script