diff --git a/README.md b/README.md index 1e33d17..35a56f4 100644 --- a/README.md +++ b/README.md @@ -30,9 +30,10 @@ Designed for system administrators — and also for those who *feel like* system - [License](#license) ## Scripts Examples -- [Scheduler Template Module](./examples/SchedulerTemplate.psm1) - [Hyper-V Backup](./examples/HyperV-Backup/README.md) - Production-ready Hyper-V VM backup solution with scheduling and retention management +- [Native-Sync](./examples/Native-Sync/README.md) - Production-ready file synchronization solution using pure PowerShell with no external dependencies - [File-Sync](./examples/File-Sync/README.md) - [FreeFileSync](https://freefilesync.org/) batch job execution +- [Scheduler Template Module](./examples/SchedulerTemplate.psm1) --- diff --git a/examples/Native-Sync/README.md b/examples/Native-Sync/README.md new file mode 100644 index 0000000..d722d83 --- /dev/null +++ b/examples/Native-Sync/README.md @@ -0,0 +1,437 @@ +# Native PowerShell Sync Script + +**Version:** 1.0.0 +**Last Updated:** 2026-01-26 + +## Overview + +Production-ready file synchronization solution using pure PowerShell with no external dependencies. Supports Mirror, Update, and TwoWay sync modes with filtering, progress reporting, and secure credential management. + +## Features + +- ✅ **No External Dependencies** - Pure PowerShell implementation +- ✅ **Multiple Sync Modes** - Mirror, Update, and TwoWay synchronization +- ✅ **Flexible Comparison** - Compare by modification time and file size +- ✅ **Deletion Policies** - Recycle Bin, Permanent delete, or Versioning +- ✅ **Filter Support** - Include/exclude patterns for files and folders +- ✅ **Remote Storage Support** - Sync to UNC shares with secure credential management +- ✅ **Progress Reporting** - File-by-file progress output +- ✅ **Dry Run Mode** - Test sync without making changes +- ✅ **Detailed Logging** - Comprehensive logging with timestamps and severity levels +- ✅ **Lock Files** - Prevents concurrent execution +- ✅ **Flexible Scheduling** - Schedule sync by month, weekday, and time + +## Requirements + +### System Requirements +- Windows with PowerShell 5.1 or later +- Administrator privileges (for network share authentication) +- Network access to target share (if using UNC paths) + +### Dependencies +- `SchedulerTemplate.psm1` module (located in parent directory) +- `scriptsettings.json` configuration file + +## File Structure + +``` +Native-Sync/ +├── native-sync.bat # Batch launcher with admin check +├── native-sync.ps1 # Main PowerShell script +├── scriptsettings.json # Configuration file +└── README.md # This file +``` + +## Installation + +1. **Copy Files** + ```powershell + # Copy the entire Native-Sync folder to your desired location + # Ensure SchedulerTemplate.psm1 is in the parent directory + ``` + +2. **Configure Settings** + + Edit `scriptsettings.json` with your environment settings: + ```json + { + "syncMode": "Mirror", + "folderPairs": [ + { + "left": "C:\\Source", + "right": "D:\\Backup" + } + ] + } + ``` + +3. **Setup Credentials (for UNC paths)** + + If syncing to a network share, create a Machine-level environment variable: + ```powershell + # Create Base64-encoded credentials + $username = "DOMAIN\user" + $password = "your-password" + $creds = "$username:$password" + $encoded = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($creds)) + + # Set Machine-level environment variable + [System.Environment]::SetEnvironmentVariable("YOUR_ENV_VAR_NAME", $encoded, "Machine") + ``` + +4. **Test Manual Execution** + ```powershell + # Run as Administrator + .\native-sync.bat + # or + .\native-sync.ps1 + + # Test with dry run first (set dryRun: true in scriptsettings.json) + ``` + +## Configuration Reference + +### Sync Modes + +| Mode | Description | LeftOnly | LeftNewer | RightNewer | RightOnly | +|------|-------------|----------|-----------|------------|-----------| +| `Mirror` | Make right identical to left | Copy to right | Update right | Update right | Delete from right | +| `Update` | Copy new/updated to right only | Copy to right | Update right | Skip | Skip | +| `TwoWay` | Propagate changes both ways | Copy to right | Update right | Update left | Copy to left | + +### Schedule Settings + +| Property | Type | Description | Example | +|----------|------|-------------|---------| +| `runMonth` | array | Month names to run. Empty = every month | `["January", "June"]` or `[]` | +| `runWeekday` | array | Weekday names to run. Empty = every day | `["Monday", "Friday"]` | +| `runTime` | array | UTC times to run (HH:mm format) | `["00:00", "12:00"]` | +| `minIntervalMinutes` | number | Minimum minutes between runs | `10` | + +### Sync Settings + +| Property | Type | Required | Description | +|----------|------|----------|-------------| +| `syncMode` | string | Yes | Sync mode: `Mirror`, `Update`, or `TwoWay` | +| `compareMethod` | string | No | Comparison method: `TimeAndSize` (default) | +| `deletionPolicy` | string | No | Deletion handling: `RecycleBin` (default), `Permanent`, `Versioning` | +| `versioningFolder` | string | No | Path for versioning when deletionPolicy is `Versioning` | +| `folderPairs` | array | Yes | Array of `{left, right}` folder pairs to sync | + +### Filter Settings + +| Property | Type | Description | +|----------|------|-------------| +| `filters.include` | array | Patterns to include (use `["*"]` for all) | +| `filters.exclude` | array | Patterns to exclude | + +**Pattern Syntax:** +- Directory patterns end with `\` (e.g., `\System Volume Information\`) +- File patterns use wildcards (e.g., `*.tmp`, `thumbs.db`) +- Prefix with `*\` to match in any subdirectory (e.g., `*\thumbs.db`) + +**Default Excludes:** +- `\System Volume Information\` +- `\$Recycle.Bin\` +- `\RECYCLE?\` +- `\Recovery\` +- `*\thumbs.db` + +> **Note:** Filter matching uses PowerShell's `-like` operator. For complex filtering needs, test with `dryRun: true` first to verify expected behavior. + +### Options + +| Property | Type | Default | Description | +|----------|------|---------|-------------| +| `excludeSymlinks` | bool | `true` | Skip symbolic links | +| `ignoreTimeShift` | bool | `false` | Ignore 1-hour DST time differences | +| `showProgress` | bool | `true` | Display file-by-file progress | +| `dryRun` | bool | `false` | Simulate sync without changes | + +### Network Settings + +| Property | Type | Required | Description | +|----------|------|----------|-------------| +| `nasRootShare` | string | No | UNC path for authentication | +| `credentialEnvVar` | string | No* | Environment variable name (*Required for UNC paths) | + +## Usage + +### Manual Execution + +**Using Batch File (Recommended):** +```batch +REM Right-click and select "Run as administrator" +native-sync.bat +``` + +**Using PowerShell:** +```powershell +# Run as Administrator +.\native-sync.ps1 + +# With verbose output +.\native-sync.ps1 -Verbose +``` + +### Automated Execution + +The script supports automated execution through the UScheduler service: + +```powershell +# Called by scheduler with -Automated flag +.\native-sync.ps1 -Automated -CurrentDateTimeUtc "2026-01-26 00:00:00" +``` + +When `-Automated` is specified: +- Schedule is enforced (month, weekday, time) +- Lock files prevent concurrent execution +- Interval checking prevents duplicate runs +- Logs are formatted for service logger (no timestamps) + +## How It Works + +### Sync Process Flow + +1. **Initialization** + - Load SchedulerTemplate.psm1 module + - Load and validate scriptsettings.json + - Parse sync mode, filters, and options + +2. **Pre-flight Checks** + - Authenticate to NAS share (if UNC) + - Verify source paths exist + - Create destination directories if needed + +3. **Scan Phase** + - Recursively scan source directory + - Recursively scan destination directory + - Apply include/exclude filters + - Skip symlinks if configured + +4. **Comparison Phase** + - Compare files by modification time and size + - Determine file status: Same, LeftOnly, LeftNewer, RightNewer, RightOnly + - Build list of sync actions based on sync mode + +5. **Execution Phase** + - Sort actions: copies → updates → file deletes → directory deletes + - Execute each action with progress reporting + - Handle deletion policy (RecycleBin, Permanent, Versioning) + +6. **Summary** + - Display sync statistics + - Report errors and warnings + +### Deletion Policies + +| Policy | Description | +|--------|-------------| +| `RecycleBin` | Move deleted files to Windows Recycle Bin (recoverable) | +| `Permanent` | Delete files permanently (not recoverable) | +| `Versioning` | Move deleted files to versioning folder | + +### Progress Output + +``` +[INFO] Processing Folder Pair 1 +[INFO] Left: E:\Users\maksym\source +[INFO] Right: \\server\share\source +[INFO] Scanning source directory... +[INFO] Found 1234 files, 56 directories +[INFO] Scanning destination directory... +[INFO] Found 1200 files, 54 directories +[INFO] Calculating sync actions... +[INFO] 45 actions to perform +[INFO] Executing sync actions... +[INFO] [COPY] docs/newfile.txt (15.2 KB) +[INFO] [UPDATE] src/main.cs (8.5 KB) +[WARNING] [DELETE] old/removed.txt (1.2 KB) +``` + +### Sync Summary + +``` +======================================== +SYNC SUMMARY +======================================== +Start Time : 2026-01-26 00:00:00 +End Time : 2026-01-26 00:05:30 +Duration : 0h 5m 30s +Status : SUCCESS + +Files Scanned : 1,234 +Files Copied : 45 +Files Updated : 12 +Files Deleted : 3 +Files Skipped : 1,174 +Bytes Copied : 156.7 MB + +Errors : 0 +Warnings : 3 +======================================== +``` + +## Logging + +### Log Levels + +| Level | Description | Color (Manual) | +|-------|-------------|----------------| +| `Info` | Informational messages | White | +| `Success` | Successful operations | Green | +| `Warning` | Non-critical issues (deletions) | Yellow | +| `Error` | Critical errors | Red | + +### Log Format + +**Manual Execution:** +``` +[2026-01-26 00:00:00] [Info] Native PowerShell Sync Started +[2026-01-26 00:00:01] [Success] All files synchronized +``` + +**Automated Execution:** +``` +[Info] Native PowerShell Sync Started +[Success] All files synchronized +``` + +## Exit Codes + +| Code | Description | +|------|-------------| +| `0` | Success (no errors) | +| `1` | Error occurred (config, paths, sync errors) | + +## Troubleshooting + +### Common Issues + +**1. Module Not Found** +``` +Error: Failed to load SchedulerTemplate.psm1 +``` +**Solution:** Ensure SchedulerTemplate.psm1 is in the parent directory (`../SchedulerTemplate.psm1`) + +**2. Source Path Not Found** +``` +Error: Source path does not exist +``` +**Solution:** Verify the `left` path in folderPairs exists and is accessible + +**3. UNC Path Authentication Failed** +``` +Error: Failed to connect to \\server\share +``` +**Solution:** +- Verify `credentialEnvVar` is set in scriptsettings.json +- Verify environment variable exists at Machine level +- Verify credentials are Base64-encoded in format: `username:password` +- Test with: `net use \\server\share` manually + +**4. Lock File Exists** +``` +Guard: Lock file exists. Skipping. +``` +**Solution:** +- Another instance is running, or previous run didn't complete +- Manually delete `.lock` file if stuck +- Check for hung PowerShell processes + +**5. Permission Denied** +``` +Error: Access to the path is denied +``` +**Solution:** +- Run as Administrator +- Verify file/folder permissions +- Check if files are locked by other processes + +### Debug Mode + +Run with verbose output: +```powershell +.\native-sync.ps1 -Verbose +``` + +Test with dry run (set in scriptsettings.json): +```json +{ + "options": { + "dryRun": true + } +} +``` + +## Best Practices + +1. **Test First** - Always test sync with `dryRun: true` in settings before actual execution +2. **Backup First** - Ensure you have backups before first sync +3. **Verify Paths** - Double-check source and destination paths +4. **Monitor Logs** - Check sync summaries regularly +5. **Secure Credentials** - Use Machine-level environment variables +6. **Schedule Wisely** - Run syncs during low-usage periods +7. **Review Settings** - Understand your sync mode implications +8. **Test Restores** - Periodically verify you can restore from synced data + +## Security Considerations + +- **Credentials** are stored Base64-encoded in Machine-level environment variables +- Script requires **Administrator privileges** for network authentication +- **Network credentials** are passed to `net use` command +- Consider using **dedicated sync account** with minimal required permissions +- **Sync data** should be stored on secured network shares with appropriate ACLs + +## Performance Considerations + +- **Sync time** depends on file count, size, and network speed +- **Network speed** is typically the bottleneck for UNC shares +- **File scanning** can take time for large directories +- Use **filter rules** to exclude unnecessary files +- **Memory usage** increases with file count during scanning +- Run during **off-peak hours** to minimize network impact + +### Scalability Limitations + +This script performs full directory scans into in-memory hashtables before comparing files. This design means: + +- **High memory usage** on very large directory trees (millions of files) +- **No streaming or batching** - all file metadata is loaded before sync begins +- **Recommended limit** - works well for typical backup scenarios (tens of thousands of files) + +Future versions may introduce streaming/batching to improve scalability for very large datasets. + +## Version History + +### 1.0.0 (2026-01-26) +- Initial release +- Pure PowerShell implementation +- Mirror, Update, TwoWay sync modes +- TimeAndSize comparison method +- RecycleBin, Permanent, Versioning deletion policies +- Include/exclude filter patterns +- Symlink exclusion option +- DST time shift handling +- Progress reporting with file-by-file output +- Dry run mode +- UNC share support with credential management +- Integration with SchedulerTemplate.psm1 + +## Support + +For issues or questions: +1. Check the [Troubleshooting](#troubleshooting) section +2. Review script logs for error details +3. Verify all [Requirements](#requirements) are met + +## License + +See [LICENSE](../../LICENSE.md) in the root directory. + +## Related Files + +- `../SchedulerTemplate.psm1` - Shared scheduling and logging module +- `scriptsettings.json` - Configuration file +- `native-sync.bat` - Batch launcher +- `native-sync.ps1` - Main script diff --git a/examples/Native-Sync/native-sync.bat b/examples/Native-Sync/native-sync.bat new file mode 100644 index 0000000..344ce04 --- /dev/null +++ b/examples/Native-Sync/native-sync.bat @@ -0,0 +1,74 @@ +@echo off +setlocal EnableDelayedExpansion + +REM ============================================================================ +REM Native Sync Launcher +REM VERSION: 1.0.0 +REM DATE: 2026-01-26 +REM DESCRIPTION: Batch file launcher for native-sync.ps1 with admin check +REM ============================================================================ + +echo. +echo ============================================ +echo Native PowerShell Sync Launcher +echo ============================================ +echo. + +REM Check for Administrator privileges +net session >nul 2>&1 +if %errorLevel% NEQ 0 ( + echo [ERROR] This script must be run as Administrator! + echo. + echo Please right-click and select "Run as administrator" + echo. + pause + exit /b 1 +) + +echo [OK] Running with Administrator privileges +echo. + +REM Get script directory +set "SCRIPT_DIR=%~dp0" +set "PS_SCRIPT=%SCRIPT_DIR%native-sync.ps1" + +REM Check if PowerShell script exists +if not exist "%PS_SCRIPT%" ( + echo [ERROR] PowerShell script not found: %PS_SCRIPT% + echo. + pause + exit /b 1 +) + +echo [OK] Found PowerShell script: %PS_SCRIPT% +echo. +echo ============================================ +echo Starting sync process... +echo ============================================ +echo. + +REM Execute PowerShell script +REM Note: Logging is handled by UScheduler service +powershell.exe -NoProfile -ExecutionPolicy Bypass -File "%PS_SCRIPT%" + +REM Capture exit code +set "EXIT_CODE=%ERRORLEVEL%" + +echo. +echo ============================================ +echo Sync process completed +echo Exit Code: %EXIT_CODE% +echo ============================================ +echo. + +if %EXIT_CODE% EQU 0 ( + echo [SUCCESS] Sync completed successfully +) else ( + echo [ERROR] Sync completed with errors +) + +echo. +pause + +endlocal +exit /b %EXIT_CODE% diff --git a/examples/Native-Sync/native-sync.ps1 b/examples/Native-Sync/native-sync.ps1 new file mode 100644 index 0000000..f3ff260 --- /dev/null +++ b/examples/Native-Sync/native-sync.ps1 @@ -0,0 +1,888 @@ +[CmdletBinding()] +param ( + [switch]$Automated, + [string]$CurrentDateTimeUtc +) + +#Requires -RunAsAdministrator + +<# +.SYNOPSIS + Native PowerShell File Synchronization Script +.DESCRIPTION + Production-ready file synchronization solution using pure PowerShell. + Supports Mirror, Update, and TwoWay sync modes with filtering, + progress reporting, and secure credential management. +.VERSION + 1.0.0 +.DATE + 2026-01-26 +.NOTES + - No external dependencies (pure PowerShell) + - Requires SchedulerTemplate.psm1 module + - Supports local and UNC paths +#> + +# Script Version +$ScriptVersion = "1.0.0" +$ScriptDate = "2026-01-26" + +try { + Import-Module "$PSScriptRoot\..\SchedulerTemplate.psm1" -Force -ErrorAction Stop +} +catch { + Write-Error "Failed to load SchedulerTemplate.psm1: $_" + exit 1 +} + +# Load Settings ============================================================ + +$settingsFile = Join-Path $PSScriptRoot "scriptsettings.json" + +if (-not (Test-Path $settingsFile)) { + Write-Error "Settings file not found: $settingsFile" + exit 1 +} + +try { + $settings = Get-Content $settingsFile -Raw | ConvertFrom-Json + Write-Verbose "Loaded settings from $settingsFile" +} +catch { + Write-Error "Failed to load settings from $settingsFile : $_" + exit 1 +} + +# Process Settings ========================================================= + +# Validate required settings +$requiredSettings = @('syncMode', 'folderPairs') +foreach ($setting in $requiredSettings) { + if (-not $settings.$setting) { + Write-Error "Required setting '$setting' is missing or empty in $settingsFile" + exit 1 + } +} + +# Extract settings +$SyncMode = $settings.syncMode +$CompareMethod = if ($settings.compareMethod) { $settings.compareMethod } else { "TimeAndSize" } +$DeletionPolicy = if ($settings.deletionPolicy) { $settings.deletionPolicy } else { "RecycleBin" } +$VersioningFolder = $settings.versioningFolder +$FolderPairs = $settings.folderPairs +$Filters = $settings.filters +$Options = $settings.options +$NasRootShare = $settings.nasRootShare +$CredentialEnvVar = $settings.credentialEnvVar + +# Get DryRun from settings +$DryRun = $Options.dryRun + +# Schedule Configuration +$Config = @{ + RunMonth = $settings.schedule.runMonth + RunWeekday = $settings.schedule.runWeekday + RunTime = $settings.schedule.runTime + MinIntervalMinutes = $settings.schedule.minIntervalMinutes +} + +# End Settings ============================================================= + +# Global variables +$script:SyncStats = @{ + StartTime = Get-Date + EndTime = $null + Success = $false + FilesScanned = 0 + FilesCopied = 0 + FilesUpdated = 0 + FilesDeleted = 0 + FilesSkipped = 0 + BytesCopied = 0 + Errors = 0 + Warnings = 0 + ErrorMessages = @() +} + +# Helper Functions ========================================================= + +function Test-FilterMatch { + param( + [string]$RelativePath, + [array]$IncludePatterns, + [array]$ExcludePatterns + ) + + # Check exclude patterns first + foreach ($pattern in $ExcludePatterns) { + # Handle directory patterns (ending with \) + if ($pattern.EndsWith('\')) { + $dirPattern = $pattern.TrimEnd('\') + if ($RelativePath -like "*$dirPattern*") { + return $false + } + } + # Handle file patterns + elseif ($RelativePath -like $pattern) { + return $false + } + # Handle wildcard patterns like *\thumbs.db + elseif ($pattern.StartsWith('*\')) { + $filePattern = $pattern.Substring(2) + if ($RelativePath -like "*\$filePattern" -or $RelativePath -eq $filePattern) { + return $false + } + } + } + + # Check include patterns + if ($IncludePatterns.Count -eq 0 -or ($IncludePatterns.Count -eq 1 -and $IncludePatterns[0] -eq '*')) { + return $true + } + + foreach ($pattern in $IncludePatterns) { + if ($RelativePath -like $pattern) { + return $true + } + } + + return $false +} + +function Get-RelativePath { + param( + [string]$FullPath, + [string]$BasePath + ) + + $BasePath = $BasePath.TrimEnd('\') + if ($FullPath.StartsWith($BasePath, [StringComparison]::OrdinalIgnoreCase)) { + return $FullPath.Substring($BasePath.Length).TrimStart('\') + } + return $FullPath +} + +function Get-LongPath { + param( + [string]$Path + ) + + # Add \\?\ prefix for long path support (>260 chars) on Windows + # Skip if already prefixed or if it's a relative path + if ($Path.StartsWith('\\?\') -or $Path.StartsWith('\\?\UNC\')) { + return $Path + } + + # Handle UNC paths: \\server\share -> \\?\UNC\server\share + if ($Path.StartsWith('\\')) { + return '\\?\UNC\' + $Path.Substring(2) + } + + # Handle local paths: C:\path -> \\?\C:\path + if ($Path.Length -ge 2 -and $Path[1] -eq ':') { + return '\\?\' + $Path + } + + return $Path +} + +function Compare-FileByTimeAndSize { + param( + [System.IO.FileInfo]$SourceFile, + [System.IO.FileInfo]$DestFile, + [switch]$IgnoreTimeShift + ) + + if (-not $DestFile -or -not $DestFile.Exists) { + return "LeftOnly" + } + + if (-not $SourceFile -or -not $SourceFile.Exists) { + return "RightOnly" + } + + # Compare sizes first + if ($SourceFile.Length -ne $DestFile.Length) { + if ($SourceFile.LastWriteTimeUtc -gt $DestFile.LastWriteTimeUtc) { + return "LeftNewer" + } + else { + return "RightNewer" + } + } + + # Compare times + $timeDiff = ($SourceFile.LastWriteTimeUtc - $DestFile.LastWriteTimeUtc).TotalSeconds + + # Handle DST time shift (1 hour = 3600 seconds) + if ($IgnoreTimeShift) { + $timeDiff = $timeDiff % 3600 + } + + if ([Math]::Abs($timeDiff) -lt 2) { + return "Same" + } + elseif ($timeDiff -gt 0) { + return "LeftNewer" + } + else { + return "RightNewer" + } +} + +function Get-AllItems { + param( + [string]$Path, + [switch]$ExcludeSymlinks, + [switch]$Automated + ) + + $items = @{ + Files = @{} + Directories = @{} + } + + $longPath = Get-LongPath -Path $Path + + if (-not (Test-Path -LiteralPath $longPath)) { + Write-Log "Path does not exist: $Path" -Level Warning -Automated:$Automated + return $items + } + + try { + $allItems = Get-ChildItem -LiteralPath $longPath -Recurse -Force -ErrorAction SilentlyContinue + + foreach ($item in $allItems) { + # Skip symlinks if configured + if ($ExcludeSymlinks -and $item.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { + continue + } + + # Strip the long path prefix for relative path calculation + $itemPath = $item.FullName + if ($itemPath.StartsWith('\\?\UNC\')) { + $itemPath = '\\' + $itemPath.Substring(8) + } + elseif ($itemPath.StartsWith('\\?\')) { + $itemPath = $itemPath.Substring(4) + } + + $relativePath = Get-RelativePath -FullPath $itemPath -BasePath $Path + + if ($item.PSIsContainer) { + $items.Directories[$relativePath] = $item + } + else { + $items.Files[$relativePath] = $item + } + } + } + catch { + Write-Log "Error scanning path $Path : $_" -Level Error -Automated:$Automated + $script:SyncStats.Errors++ + } + + return $items +} + +function Get-SyncActions { + param( + [hashtable]$SourceItems, + [hashtable]$DestItems, + [string]$SyncMode, + [string]$SourcePath, + [string]$DestPath, + [array]$IncludePatterns, + [array]$ExcludePatterns, + [switch]$IgnoreTimeShift, + [switch]$Automated + ) + + $actions = @() + + # Process source files + foreach ($relativePath in $SourceItems.Files.Keys) { + $script:SyncStats.FilesScanned++ + + # Check filters + if (-not (Test-FilterMatch -RelativePath $relativePath -IncludePatterns $IncludePatterns -ExcludePatterns $ExcludePatterns)) { + $script:SyncStats.FilesSkipped++ + continue + } + + $sourceFile = $SourceItems.Files[$relativePath] + $destFile = $DestItems.Files[$relativePath] + + $comparison = Compare-FileByTimeAndSize -SourceFile $sourceFile -DestFile $destFile -IgnoreTimeShift:$IgnoreTimeShift + + $action = $null + + switch ($comparison) { + "LeftOnly" { + $action = @{ + Type = "Copy" + Source = $sourceFile.FullName + Destination = Join-Path $DestPath $relativePath + RelativePath = $relativePath + Size = $sourceFile.Length + } + } + "LeftNewer" { + $action = @{ + Type = "Update" + Source = $sourceFile.FullName + Destination = Join-Path $DestPath $relativePath + RelativePath = $relativePath + Size = $sourceFile.Length + } + } + "RightNewer" { + switch ($SyncMode) { + "Mirror" { + # Mirror overwrites right with left + $action = @{ + Type = "Update" + Source = $sourceFile.FullName + Destination = Join-Path $DestPath $relativePath + RelativePath = $relativePath + Size = $sourceFile.Length + } + } + "TwoWay" { + # TwoWay updates left from right + $action = @{ + Type = "Update" + Source = $destFile.FullName + Destination = $sourceFile.FullName + RelativePath = $relativePath + Size = $destFile.Length + Direction = "ToLeft" + } + } + # Update mode: skip + } + } + "Same" { + $script:SyncStats.FilesSkipped++ + } + } + + if ($action) { + $actions += $action + } + } + + # Process destination-only files + foreach ($relativePath in $DestItems.Files.Keys) { + if ($SourceItems.Files.ContainsKey($relativePath)) { + continue # Already processed + } + + $script:SyncStats.FilesScanned++ + + # Check filters + if (-not (Test-FilterMatch -RelativePath $relativePath -IncludePatterns $IncludePatterns -ExcludePatterns $ExcludePatterns)) { + $script:SyncStats.FilesSkipped++ + continue + } + + $destFile = $DestItems.Files[$relativePath] + + switch ($SyncMode) { + "Mirror" { + $actions += @{ + Type = "Delete" + Source = $null + Destination = $destFile.FullName + RelativePath = $relativePath + Size = $destFile.Length + } + } + "TwoWay" { + $actions += @{ + Type = "Copy" + Source = $destFile.FullName + Destination = Join-Path $SourcePath $relativePath + RelativePath = $relativePath + Size = $destFile.Length + Direction = "ToLeft" + } + } + # Update mode: skip destination-only files + default { + $script:SyncStats.FilesSkipped++ + } + } + } + + # Process directories (for deletion in Mirror mode) + if ($SyncMode -eq "Mirror") { + foreach ($relativePath in $DestItems.Directories.Keys) { + if (-not $SourceItems.Directories.ContainsKey($relativePath)) { + # Check if directory should be excluded + if (-not (Test-FilterMatch -RelativePath $relativePath -IncludePatterns $IncludePatterns -ExcludePatterns $ExcludePatterns)) { + continue + } + + $actions += @{ + Type = "DeleteDir" + Source = $null + Destination = $DestItems.Directories[$relativePath].FullName + RelativePath = $relativePath + Size = 0 + } + } + } + } + + return $actions +} + +function Remove-ToRecycleBin { + param( + [string]$Path + ) + + Add-Type -AssemblyName Microsoft.VisualBasic + if (Test-Path -LiteralPath $Path) { + $item = Get-Item -LiteralPath $Path + if ($item.PSIsContainer) { + [Microsoft.VisualBasic.FileIO.FileSystem]::DeleteDirectory( + $Path, + [Microsoft.VisualBasic.FileIO.UIOption]::OnlyErrorDialogs, + [Microsoft.VisualBasic.FileIO.RecycleOption]::SendToRecycleBin + ) + } + else { + [Microsoft.VisualBasic.FileIO.FileSystem]::DeleteFile( + $Path, + [Microsoft.VisualBasic.FileIO.UIOption]::OnlyErrorDialogs, + [Microsoft.VisualBasic.FileIO.RecycleOption]::SendToRecycleBin + ) + } + } +} + +function Invoke-SyncAction { + param( + [hashtable]$Action, + [string]$DeletionPolicy, + [string]$VersioningFolder, + [switch]$DryRun, + [switch]$ShowProgress, + [switch]$Automated + ) + + $actionType = $Action.Type + $source = $Action.Source + $destination = $Action.Destination + $relativePath = $Action.RelativePath + $size = $Action.Size + $direction = if ($Action.Direction) { " ($($Action.Direction))" } else { "" } + + # Format size for display + $sizeDisplay = if ($size -gt 1MB) { + "{0:N2} MB" -f ($size / 1MB) + } + elseif ($size -gt 1KB) { + "{0:N2} KB" -f ($size / 1KB) + } + else { + "$size bytes" + } + + $prefix = if ($DryRun) { "[DRY-RUN] " } else { "" } + + try { + switch ($actionType) { + "Copy" { + if ($ShowProgress) { + Write-Log "$prefix[COPY]$direction $relativePath ($sizeDisplay)" -Level Info -Automated:$Automated + } + + if (-not $DryRun) { + $destDir = Split-Path $destination -Parent + $longDestDir = Get-LongPath -Path $destDir + $longSource = Get-LongPath -Path $source + $longDest = Get-LongPath -Path $destination + + if (-not (Test-Path -LiteralPath $longDestDir)) { + New-Item -Path $longDestDir -ItemType Directory -Force | Out-Null + } + Copy-Item -LiteralPath $longSource -Destination $longDest -Force + $script:SyncStats.FilesCopied++ + $script:SyncStats.BytesCopied += $size + } + else { + $script:SyncStats.FilesCopied++ + } + } + "Update" { + if ($ShowProgress) { + Write-Log "$prefix[UPDATE]$direction $relativePath ($sizeDisplay)" -Level Info -Automated:$Automated + } + + if (-not $DryRun) { + $longSource = Get-LongPath -Path $source + $longDest = Get-LongPath -Path $destination + Copy-Item -LiteralPath $longSource -Destination $longDest -Force + $script:SyncStats.FilesUpdated++ + $script:SyncStats.BytesCopied += $size + } + else { + $script:SyncStats.FilesUpdated++ + } + } + "Delete" { + if ($ShowProgress) { + Write-Log "$prefix[DELETE] $relativePath ($sizeDisplay)" -Level Warning -Automated:$Automated + } + + if (-not $DryRun) { + $longDest = Get-LongPath -Path $destination + switch ($DeletionPolicy) { + "RecycleBin" { + Remove-ToRecycleBin -Path $destination + } + "Versioning" { + if ($VersioningFolder) { + $versionPath = Join-Path $VersioningFolder $relativePath + $versionDir = Split-Path $versionPath -Parent + $longVersionDir = Get-LongPath -Path $versionDir + $longVersionPath = Get-LongPath -Path $versionPath + if (-not (Test-Path -LiteralPath $longVersionDir)) { + New-Item -Path $longVersionDir -ItemType Directory -Force | Out-Null + } + Move-Item -LiteralPath $longDest -Destination $longVersionPath -Force + } + else { + Remove-Item -LiteralPath $longDest -Force + } + } + default { + # Permanent + Remove-Item -LiteralPath $longDest -Force + } + } + $script:SyncStats.FilesDeleted++ + } + else { + $script:SyncStats.FilesDeleted++ + } + } + "DeleteDir" { + if ($ShowProgress) { + Write-Log "$prefix[DELETE DIR] $relativePath" -Level Warning -Automated:$Automated + } + + if (-not $DryRun) { + $longDest = Get-LongPath -Path $destination + switch ($DeletionPolicy) { + "RecycleBin" { + Remove-ToRecycleBin -Path $destination + } + "Versioning" { + if ($VersioningFolder) { + $versionPath = Join-Path $VersioningFolder $relativePath + $versionParent = Split-Path $versionPath -Parent + $longVersionParent = Get-LongPath -Path $versionParent + $longVersionPath = Get-LongPath -Path $versionPath + if (-not (Test-Path -LiteralPath $longVersionParent)) { + New-Item -Path $longVersionParent -ItemType Directory -Force | Out-Null + } + Move-Item -LiteralPath $longDest -Destination $longVersionPath -Force + } + else { + Remove-Item -LiteralPath $longDest -Recurse -Force + } + } + default { + Remove-Item -LiteralPath $longDest -Recurse -Force + } + } + } + } + } + return $true + } + catch { + Write-Log "Error executing $actionType on $relativePath : $_" -Level Error -Automated:$Automated + $script:SyncStats.Errors++ + $script:SyncStats.ErrorMessages += "[$actionType] $relativePath : $_" + return $false + } +} + +function Connect-NasShare { + param( + [string]$SharePath, + [switch]$Automated + ) + + # Check if path is UNC + if (-not $SharePath.StartsWith("\\")) { + Write-Log "NAS path is local, no authentication needed" -Level Info -Automated:$Automated + return $true + } + + # Validate UNC path format + if (-not (Test-UNCPath -Path $SharePath)) { + Write-Log "Invalid UNC path format: $SharePath (expected \\server\share)" -Level Error -Automated:$Automated + return $false + } + + Write-Log "Authenticating to NAS share: $SharePath" -Level Info -Automated:$Automated + + # Validate credential environment variable name is configured + if (-not $CredentialEnvVar) { + Write-Log "credentialEnvVar is not configured in settings for UNC path authentication" -Level Error -Automated:$Automated + return $false + } + + try { + # Retrieve credentials from environment variable + $creds = Get-CredentialFromEnvVar -EnvVarName $CredentialEnvVar -Automated:$Automated + + if (-not $creds) { + return $false + } + + # Check if already connected + $existingConnection = net use | Select-String $SharePath + if ($existingConnection) { + Write-Log "Already connected to $SharePath" -Level Info -Automated:$Automated + return $true + } + + # Connect to share + $netUseResult = cmd /c "net use $SharePath $($creds.Password) /user:$($creds.Username) 2>&1" + + if ($LASTEXITCODE -ne 0) { + Write-Log "Failed to connect to $SharePath : $netUseResult" -Level Error -Automated:$Automated + return $false + } + + Write-Log "Successfully authenticated to $SharePath" -Level Success -Automated:$Automated + return $true + } + catch { + Write-Log "Error connecting to share: $_" -Level Error -Automated:$Automated + return $false + } +} + +function Write-SyncSummary { + param([switch]$Automated) + + $script:SyncStats.EndTime = Get-Date + $duration = $script:SyncStats.EndTime - $script:SyncStats.StartTime + + # Format bytes + $bytesDisplay = if ($script:SyncStats.BytesCopied -gt 1GB) { + "{0:N2} GB" -f ($script:SyncStats.BytesCopied / 1GB) + } + elseif ($script:SyncStats.BytesCopied -gt 1MB) { + "{0:N2} MB" -f ($script:SyncStats.BytesCopied / 1MB) + } + elseif ($script:SyncStats.BytesCopied -gt 1KB) { + "{0:N2} KB" -f ($script:SyncStats.BytesCopied / 1KB) + } + else { + "$($script:SyncStats.BytesCopied) bytes" + } + + Write-Log "" -Level Info -Automated:$Automated + Write-Log "========================================" -Level Info -Automated:$Automated + Write-Log "SYNC SUMMARY" -Level Info -Automated:$Automated + Write-Log "========================================" -Level Info -Automated:$Automated + Write-Log "Start Time : $($script:SyncStats.StartTime.ToString('yyyy-MM-dd HH:mm:ss'))" -Level Info -Automated:$Automated + Write-Log "End Time : $($script:SyncStats.EndTime.ToString('yyyy-MM-dd HH:mm:ss'))" -Level Info -Automated:$Automated + Write-Log "Duration : $($duration.Hours)h $($duration.Minutes)m $($duration.Seconds)s" -Level Info -Automated:$Automated + Write-Log "Status : $(if ($script:SyncStats.Errors -eq 0) { 'SUCCESS' } else { 'COMPLETED WITH ERRORS' })" -Level $(if ($script:SyncStats.Errors -eq 0) { 'Success' } else { 'Warning' }) -Automated:$Automated + Write-Log "" -Level Info -Automated:$Automated + Write-Log "Files Scanned : $($script:SyncStats.FilesScanned)" -Level Info -Automated:$Automated + Write-Log "Files Copied : $($script:SyncStats.FilesCopied)" -Level Info -Automated:$Automated + Write-Log "Files Updated : $($script:SyncStats.FilesUpdated)" -Level Info -Automated:$Automated + Write-Log "Files Deleted : $($script:SyncStats.FilesDeleted)" -Level Info -Automated:$Automated + Write-Log "Files Skipped : $($script:SyncStats.FilesSkipped)" -Level Info -Automated:$Automated + Write-Log "Bytes Copied : $bytesDisplay" -Level Info -Automated:$Automated + Write-Log "" -Level Info -Automated:$Automated + Write-Log "Errors : $($script:SyncStats.Errors)" -Level $(if ($script:SyncStats.Errors -eq 0) { 'Info' } else { 'Error' }) -Automated:$Automated + Write-Log "Warnings : $($script:SyncStats.Warnings)" -Level Info -Automated:$Automated + + if ($script:SyncStats.ErrorMessages.Count -gt 0) { + Write-Log "" -Level Info -Automated:$Automated + Write-Log "Error Details:" -Level Error -Automated:$Automated + foreach ($msg in $script:SyncStats.ErrorMessages) { + Write-Log " - $msg" -Level Error -Automated:$Automated + } + } + + Write-Log "========================================" -Level Info -Automated:$Automated + + $script:SyncStats.Success = ($script:SyncStats.Errors -eq 0) +} + +# Main Business Logic ====================================================== + +function Start-BusinessLogic { + param( + [switch]$Automated, + [switch]$DryRun + ) + + Write-Log "========================================" -Level Info -Automated:$Automated + Write-Log "Native PowerShell Sync Started" -Level Info -Automated:$Automated + Write-Log "Script Version: $ScriptVersion ($ScriptDate)" -Level Info -Automated:$Automated + Write-Log "Sync Mode: $SyncMode" -Level Info -Automated:$Automated + Write-Log "Compare Method: $CompareMethod" -Level Info -Automated:$Automated + Write-Log "Deletion Policy: $DeletionPolicy" -Level Info -Automated:$Automated + if ($DryRun) { + Write-Log "DRY RUN MODE - No changes will be made" -Level Warning -Automated:$Automated + } + Write-Log "========================================" -Level Info -Automated:$Automated + + # Connect to NAS share if needed + if ($NasRootShare) { + if (-not (Connect-NasShare -SharePath $NasRootShare -Automated:$Automated)) { + Write-Log "Failed to connect to NAS share. Aborting sync." -Level Error -Automated:$Automated + exit 1 + } + } + + # Process each folder pair + $pairIndex = 0 + foreach ($pair in $FolderPairs) { + $pairIndex++ + + $leftPath = $pair.left + $rightPath = $pair.right + + # Skip empty pairs + if ([string]::IsNullOrWhiteSpace($leftPath) -or [string]::IsNullOrWhiteSpace($rightPath)) { + continue + } + + Write-Log "" -Level Info -Automated:$Automated + Write-Log "Processing Folder Pair $pairIndex" -Level Info -Automated:$Automated + Write-Log " Left: $leftPath" -Level Info -Automated:$Automated + Write-Log " Right: $rightPath" -Level Info -Automated:$Automated + + # Verify paths exist (using long path support) + $longLeftPath = Get-LongPath -Path $leftPath + $longRightPath = Get-LongPath -Path $rightPath + + if (-not (Test-Path -LiteralPath $longLeftPath)) { + Write-Log "Source path does not exist: $leftPath" -Level Error -Automated:$Automated + $script:SyncStats.Errors++ + continue + } + + # Create destination if it doesn't exist + if (-not (Test-Path -LiteralPath $longRightPath)) { + if (-not $DryRun) { + try { + New-Item -Path $longRightPath -ItemType Directory -Force | Out-Null + Write-Log "Created destination directory: $rightPath" -Level Info -Automated:$Automated + } + catch { + Write-Log "Failed to create destination: $rightPath - $_" -Level Error -Automated:$Automated + $script:SyncStats.Errors++ + continue + } + } + else { + Write-Log "[DRY-RUN] Would create destination directory: $rightPath" -Level Info -Automated:$Automated + } + } + + # Scan directories + Write-Log "Scanning source directory..." -Level Info -Automated:$Automated + $sourceItems = Get-AllItems -Path $leftPath -ExcludeSymlinks:$Options.excludeSymlinks -Automated:$Automated + Write-Log " Found $($sourceItems.Files.Count) files, $($sourceItems.Directories.Count) directories" -Level Info -Automated:$Automated + + Write-Log "Scanning destination directory..." -Level Info -Automated:$Automated + $destItems = Get-AllItems -Path $rightPath -ExcludeSymlinks:$Options.excludeSymlinks -Automated:$Automated + Write-Log " Found $($destItems.Files.Count) files, $($destItems.Directories.Count) directories" -Level Info -Automated:$Automated + + # Get include/exclude patterns + $includePatterns = if ($Filters -and $Filters.include) { $Filters.include } else { @('*') } + $excludePatterns = if ($Filters -and $Filters.exclude) { $Filters.exclude } else { @() } + + # Calculate sync actions + Write-Log "Calculating sync actions..." -Level Info -Automated:$Automated + $actions = Get-SyncActions ` + -SourceItems $sourceItems ` + -DestItems $destItems ` + -SyncMode $SyncMode ` + -SourcePath $leftPath ` + -DestPath $rightPath ` + -IncludePatterns $includePatterns ` + -ExcludePatterns $excludePatterns ` + -IgnoreTimeShift:$Options.ignoreTimeShift ` + -Automated:$Automated + + Write-Log " $($actions.Count) actions to perform" -Level Info -Automated:$Automated + + # Execute actions + if ($actions.Count -gt 0) { + Write-Log "" -Level Info -Automated:$Automated + Write-Log "Executing sync actions..." -Level Info -Automated:$Automated + + # Sort actions: copies first, then updates, then deletes (files before directories) + $sortedActions = $actions | Sort-Object { + switch ($_.Type) { + "Copy" { 1 } + "Update" { 2 } + "Delete" { 3 } + "DeleteDir" { 4 } + } + } + + # For directory deletions, sort by path length descending (delete deepest first) + $dirDeletes = $sortedActions | Where-Object { $_.Type -eq "DeleteDir" } | Sort-Object { $_.RelativePath.Length } -Descending + $otherActions = $sortedActions | Where-Object { $_.Type -ne "DeleteDir" } + $sortedActions = @($otherActions) + @($dirDeletes) + + foreach ($action in $sortedActions) { + Invoke-SyncAction ` + -Action $action ` + -DeletionPolicy $DeletionPolicy ` + -VersioningFolder $VersioningFolder ` + -DryRun:$DryRun ` + -ShowProgress:$Options.showProgress ` + -Automated:$Automated | Out-Null + } + } + } + + # Print summary + Write-SyncSummary -Automated:$Automated + + # Exit with appropriate code + if ($script:SyncStats.Errors -gt 0) { + exit 1 + } +} + +# Entry Point ============================================================== + +if ($Automated) { + if (Get-Command Invoke-ScheduledExecution -ErrorAction SilentlyContinue) { + $params = @{ + Config = $Config + Automated = $Automated + CurrentDateTimeUtc = $CurrentDateTimeUtc + ScriptBlock = { Start-BusinessLogic -Automated:$Automated -DryRun:$DryRun } + } + Invoke-ScheduledExecution @params + } + else { + Write-Log "Invoke-ScheduledExecution not available. Execution aborted." -Level Error -Automated:$Automated + exit 1 + } +} +else { + Write-Log "Manual execution started" -Level Info -Automated:$Automated + Start-BusinessLogic -Automated:$Automated -DryRun:$DryRun +} diff --git a/examples/Native-Sync/scriptsettings.json b/examples/Native-Sync/scriptsettings.json new file mode 100644 index 0000000..1c245d7 --- /dev/null +++ b/examples/Native-Sync/scriptsettings.json @@ -0,0 +1,68 @@ +{ + "$schema": "https://json-schema.org/draft-07/schema", + "title": "Native Sync Script Settings", + "description": "Configuration file for native-sync.ps1 script (pure PowerShell file synchronization)", + "version": "1.0.0", + "lastModified": "2026-01-26", + "schedule": { + "runMonth": [], + "runWeekday": ["Monday"], + "runTime": ["00:00"], + "minIntervalMinutes": 10 + }, + "syncMode": "Mirror", + "compareMethod": "TimeAndSize", + "deletionPolicy": "RecycleBin", + "versioningFolder": "", + "folderPairs": [ + { + "left": "E:\\Users\\maksym\\source", + "right": "\\\\nassrv0001.corp.maks-it.com\\data-1\\Users\\maksym\\source" + } + ], + "filters": { + "include": ["*"], + "exclude": [ + "\\System Volume Information\\", + "\\$Recycle.Bin\\", + "\\RECYCLE?\\", + "\\Recovery\\", + "*\\thumbs.db" + ] + }, + "options": { + "excludeSymlinks": true, + "ignoreTimeShift": false, + "showProgress": true, + "dryRun": false + }, + "nasRootShare": "\\\\nassrv0001.corp.maks-it.com\\data-1", + "credentialEnvVar": "nassrv0001", + "_comments": { + "version": "Configuration schema version", + "lastModified": "Last modification date (YYYY-MM-DD)", + "schedule": { + "runMonth": "Array of month names (e.g. 'January', 'June', 'December') to run sync. Empty array = every month.", + "runWeekday": "Array of weekday names (e.g. 'Monday', 'Friday') to run sync. Empty array = every day.", + "runTime": "Array of UTC times in HH:mm format when sync should run.", + "minIntervalMinutes": "Minimum minutes between sync runs to prevent duplicate executions." + }, + "syncMode": "Synchronization mode: 'Mirror' (make right identical to left), 'Update' (copy new/updated to right only), 'TwoWay' (propagate changes both ways)", + "compareMethod": "File comparison method: 'TimeAndSize' (compare by modification time and file size)", + "deletionPolicy": "How to handle deleted files: 'RecycleBin' (move to recycle bin), 'Permanent' (delete permanently), 'Versioning' (move to versioning folder)", + "versioningFolder": "Path for versioning folder when deletionPolicy is 'Versioning'. Leave empty if not using versioning.", + "folderPairs": "Array of folder pairs to synchronize. Each pair has 'left' (source) and 'right' (destination) paths.", + "filters": { + "include": "Array of patterns to include. Use '*' to include all files.", + "exclude": "Array of patterns to exclude. Supports wildcards and path patterns." + }, + "options": { + "excludeSymlinks": "Skip symbolic links during synchronization", + "ignoreTimeShift": "Ignore 1-hour time differences (DST changes)", + "showProgress": "Display file-by-file progress during sync", + "dryRun": "Simulate sync without making changes (can also use -DryRun parameter)" + }, + "nasRootShare": "UNC path to NAS root share for authentication. Only used for connecting to the share.", + "credentialEnvVar": "Name of Machine-level environment variable containing Base64-encoded 'username:password' for NAS authentication" + } +}