(feature): powershell native file sync example

This commit is contained in:
Maksym Sadovnychyy 2026-01-26 21:43:42 +01:00
parent de1add2172
commit 5839f55999
5 changed files with 1469 additions and 1 deletions

View File

@ -30,9 +30,10 @@ Designed for system administrators — and also for those who *feel like* system
- [License](#license) - [License](#license)
## Scripts Examples ## Scripts Examples
- [Scheduler Template Module](./examples/SchedulerTemplate.psm1)
- [Hyper-V Backup](./examples/HyperV-Backup/README.md) - Production-ready Hyper-V VM backup solution with scheduling and retention management - [Hyper-V Backup](./examples/HyperV-Backup/README.md) - Production-ready Hyper-V VM backup solution with scheduling and retention management
- [Native-Sync](./examples/Native-Sync/README.md) - Production-ready file synchronization solution using pure PowerShell with no external dependencies
- [File-Sync](./examples/File-Sync/README.md) - [FreeFileSync](https://freefilesync.org/) batch job execution - [File-Sync](./examples/File-Sync/README.md) - [FreeFileSync](https://freefilesync.org/) batch job execution
- [Scheduler Template Module](./examples/SchedulerTemplate.psm1)
--- ---

View File

@ -0,0 +1,437 @@
# Native PowerShell Sync Script
**Version:** 1.0.0
**Last Updated:** 2026-01-26
## Overview
Production-ready file synchronization solution using pure PowerShell with no external dependencies. Supports Mirror, Update, and TwoWay sync modes with filtering, progress reporting, and secure credential management.
## Features
- ✅ **No External Dependencies** - Pure PowerShell implementation
- ✅ **Multiple Sync Modes** - Mirror, Update, and TwoWay synchronization
- ✅ **Flexible Comparison** - Compare by modification time and file size
- ✅ **Deletion Policies** - Recycle Bin, Permanent delete, or Versioning
- ✅ **Filter Support** - Include/exclude patterns for files and folders
- ✅ **Remote Storage Support** - Sync to UNC shares with secure credential management
- ✅ **Progress Reporting** - File-by-file progress output
- ✅ **Dry Run Mode** - Test sync without making changes
- ✅ **Detailed Logging** - Comprehensive logging with timestamps and severity levels
- ✅ **Lock Files** - Prevents concurrent execution
- ✅ **Flexible Scheduling** - Schedule sync by month, weekday, and time
## Requirements
### System Requirements
- Windows with PowerShell 5.1 or later
- Administrator privileges (for network share authentication)
- Network access to target share (if using UNC paths)
### Dependencies
- `SchedulerTemplate.psm1` module (located in parent directory)
- `scriptsettings.json` configuration file
## File Structure
```
Native-Sync/
├── native-sync.bat # Batch launcher with admin check
├── native-sync.ps1 # Main PowerShell script
├── scriptsettings.json # Configuration file
└── README.md # This file
```
## Installation
1. **Copy Files**
```powershell
# Copy the entire Native-Sync folder to your desired location
# Ensure SchedulerTemplate.psm1 is in the parent directory
```
2. **Configure Settings**
Edit `scriptsettings.json` with your environment settings:
```json
{
"syncMode": "Mirror",
"folderPairs": [
{
"left": "C:\\Source",
"right": "D:\\Backup"
}
]
}
```
3. **Setup Credentials (for UNC paths)**
If syncing to a network share, create a Machine-level environment variable:
```powershell
# Create Base64-encoded credentials
$username = "DOMAIN\user"
$password = "your-password"
$creds = "$username:$password"
$encoded = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($creds))
# Set Machine-level environment variable
[System.Environment]::SetEnvironmentVariable("YOUR_ENV_VAR_NAME", $encoded, "Machine")
```
4. **Test Manual Execution**
```powershell
# Run as Administrator
.\native-sync.bat
# or
.\native-sync.ps1
# Test with dry run first (set dryRun: true in scriptsettings.json)
```
## Configuration Reference
### Sync Modes
| Mode | Description | LeftOnly | LeftNewer | RightNewer | RightOnly |
|------|-------------|----------|-----------|------------|-----------|
| `Mirror` | Make right identical to left | Copy to right | Update right | Update right | Delete from right |
| `Update` | Copy new/updated to right only | Copy to right | Update right | Skip | Skip |
| `TwoWay` | Propagate changes both ways | Copy to right | Update right | Update left | Copy to left |
### Schedule Settings
| Property | Type | Description | Example |
|----------|------|-------------|---------|
| `runMonth` | array | Month names to run. Empty = every month | `["January", "June"]` or `[]` |
| `runWeekday` | array | Weekday names to run. Empty = every day | `["Monday", "Friday"]` |
| `runTime` | array | UTC times to run (HH:mm format) | `["00:00", "12:00"]` |
| `minIntervalMinutes` | number | Minimum minutes between runs | `10` |
### Sync Settings
| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `syncMode` | string | Yes | Sync mode: `Mirror`, `Update`, or `TwoWay` |
| `compareMethod` | string | No | Comparison method: `TimeAndSize` (default) |
| `deletionPolicy` | string | No | Deletion handling: `RecycleBin` (default), `Permanent`, `Versioning` |
| `versioningFolder` | string | No | Path for versioning when deletionPolicy is `Versioning` |
| `folderPairs` | array | Yes | Array of `{left, right}` folder pairs to sync |
### Filter Settings
| Property | Type | Description |
|----------|------|-------------|
| `filters.include` | array | Patterns to include (use `["*"]` for all) |
| `filters.exclude` | array | Patterns to exclude |
**Pattern Syntax:**
- Directory patterns end with `\` (e.g., `\System Volume Information\`)
- File patterns use wildcards (e.g., `*.tmp`, `thumbs.db`)
- Prefix with `*\` to match in any subdirectory (e.g., `*\thumbs.db`)
**Default Excludes:**
- `\System Volume Information\`
- `\$Recycle.Bin\`
- `\RECYCLE?\`
- `\Recovery\`
- `*\thumbs.db`
> **Note:** Filter matching uses PowerShell's `-like` operator. For complex filtering needs, test with `dryRun: true` first to verify expected behavior.
### Options
| Property | Type | Default | Description |
|----------|------|---------|-------------|
| `excludeSymlinks` | bool | `true` | Skip symbolic links |
| `ignoreTimeShift` | bool | `false` | Ignore 1-hour DST time differences |
| `showProgress` | bool | `true` | Display file-by-file progress |
| `dryRun` | bool | `false` | Simulate sync without changes |
### Network Settings
| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `nasRootShare` | string | No | UNC path for authentication |
| `credentialEnvVar` | string | No* | Environment variable name (*Required for UNC paths) |
## Usage
### Manual Execution
**Using Batch File (Recommended):**
```batch
REM Right-click and select "Run as administrator"
native-sync.bat
```
**Using PowerShell:**
```powershell
# Run as Administrator
.\native-sync.ps1
# With verbose output
.\native-sync.ps1 -Verbose
```
### Automated Execution
The script supports automated execution through the UScheduler service:
```powershell
# Called by scheduler with -Automated flag
.\native-sync.ps1 -Automated -CurrentDateTimeUtc "2026-01-26 00:00:00"
```
When `-Automated` is specified:
- Schedule is enforced (month, weekday, time)
- Lock files prevent concurrent execution
- Interval checking prevents duplicate runs
- Logs are formatted for service logger (no timestamps)
## How It Works
### Sync Process Flow
1. **Initialization**
- Load SchedulerTemplate.psm1 module
- Load and validate scriptsettings.json
- Parse sync mode, filters, and options
2. **Pre-flight Checks**
- Authenticate to NAS share (if UNC)
- Verify source paths exist
- Create destination directories if needed
3. **Scan Phase**
- Recursively scan source directory
- Recursively scan destination directory
- Apply include/exclude filters
- Skip symlinks if configured
4. **Comparison Phase**
- Compare files by modification time and size
- Determine file status: Same, LeftOnly, LeftNewer, RightNewer, RightOnly
- Build list of sync actions based on sync mode
5. **Execution Phase**
- Sort actions: copies → updates → file deletes → directory deletes
- Execute each action with progress reporting
- Handle deletion policy (RecycleBin, Permanent, Versioning)
6. **Summary**
- Display sync statistics
- Report errors and warnings
### Deletion Policies
| Policy | Description |
|--------|-------------|
| `RecycleBin` | Move deleted files to Windows Recycle Bin (recoverable) |
| `Permanent` | Delete files permanently (not recoverable) |
| `Versioning` | Move deleted files to versioning folder |
### Progress Output
```
[INFO] Processing Folder Pair 1
[INFO] Left: E:\Users\maksym\source
[INFO] Right: \\server\share\source
[INFO] Scanning source directory...
[INFO] Found 1234 files, 56 directories
[INFO] Scanning destination directory...
[INFO] Found 1200 files, 54 directories
[INFO] Calculating sync actions...
[INFO] 45 actions to perform
[INFO] Executing sync actions...
[INFO] [COPY] docs/newfile.txt (15.2 KB)
[INFO] [UPDATE] src/main.cs (8.5 KB)
[WARNING] [DELETE] old/removed.txt (1.2 KB)
```
### Sync Summary
```
========================================
SYNC SUMMARY
========================================
Start Time : 2026-01-26 00:00:00
End Time : 2026-01-26 00:05:30
Duration : 0h 5m 30s
Status : SUCCESS
Files Scanned : 1,234
Files Copied : 45
Files Updated : 12
Files Deleted : 3
Files Skipped : 1,174
Bytes Copied : 156.7 MB
Errors : 0
Warnings : 3
========================================
```
## Logging
### Log Levels
| Level | Description | Color (Manual) |
|-------|-------------|----------------|
| `Info` | Informational messages | White |
| `Success` | Successful operations | Green |
| `Warning` | Non-critical issues (deletions) | Yellow |
| `Error` | Critical errors | Red |
### Log Format
**Manual Execution:**
```
[2026-01-26 00:00:00] [Info] Native PowerShell Sync Started
[2026-01-26 00:00:01] [Success] All files synchronized
```
**Automated Execution:**
```
[Info] Native PowerShell Sync Started
[Success] All files synchronized
```
## Exit Codes
| Code | Description |
|------|-------------|
| `0` | Success (no errors) |
| `1` | Error occurred (config, paths, sync errors) |
## Troubleshooting
### Common Issues
**1. Module Not Found**
```
Error: Failed to load SchedulerTemplate.psm1
```
**Solution:** Ensure SchedulerTemplate.psm1 is in the parent directory (`../SchedulerTemplate.psm1`)
**2. Source Path Not Found**
```
Error: Source path does not exist
```
**Solution:** Verify the `left` path in folderPairs exists and is accessible
**3. UNC Path Authentication Failed**
```
Error: Failed to connect to \\server\share
```
**Solution:**
- Verify `credentialEnvVar` is set in scriptsettings.json
- Verify environment variable exists at Machine level
- Verify credentials are Base64-encoded in format: `username:password`
- Test with: `net use \\server\share` manually
**4. Lock File Exists**
```
Guard: Lock file exists. Skipping.
```
**Solution:**
- Another instance is running, or previous run didn't complete
- Manually delete `.lock` file if stuck
- Check for hung PowerShell processes
**5. Permission Denied**
```
Error: Access to the path is denied
```
**Solution:**
- Run as Administrator
- Verify file/folder permissions
- Check if files are locked by other processes
### Debug Mode
Run with verbose output:
```powershell
.\native-sync.ps1 -Verbose
```
Test with dry run (set in scriptsettings.json):
```json
{
"options": {
"dryRun": true
}
}
```
## Best Practices
1. **Test First** - Always test sync with `dryRun: true` in settings before actual execution
2. **Backup First** - Ensure you have backups before first sync
3. **Verify Paths** - Double-check source and destination paths
4. **Monitor Logs** - Check sync summaries regularly
5. **Secure Credentials** - Use Machine-level environment variables
6. **Schedule Wisely** - Run syncs during low-usage periods
7. **Review Settings** - Understand your sync mode implications
8. **Test Restores** - Periodically verify you can restore from synced data
## Security Considerations
- **Credentials** are stored Base64-encoded in Machine-level environment variables
- Script requires **Administrator privileges** for network authentication
- **Network credentials** are passed to `net use` command
- Consider using **dedicated sync account** with minimal required permissions
- **Sync data** should be stored on secured network shares with appropriate ACLs
## Performance Considerations
- **Sync time** depends on file count, size, and network speed
- **Network speed** is typically the bottleneck for UNC shares
- **File scanning** can take time for large directories
- Use **filter rules** to exclude unnecessary files
- **Memory usage** increases with file count during scanning
- Run during **off-peak hours** to minimize network impact
### Scalability Limitations
This script performs full directory scans into in-memory hashtables before comparing files. This design means:
- **High memory usage** on very large directory trees (millions of files)
- **No streaming or batching** - all file metadata is loaded before sync begins
- **Recommended limit** - works well for typical backup scenarios (tens of thousands of files)
Future versions may introduce streaming/batching to improve scalability for very large datasets.
## Version History
### 1.0.0 (2026-01-26)
- Initial release
- Pure PowerShell implementation
- Mirror, Update, TwoWay sync modes
- TimeAndSize comparison method
- RecycleBin, Permanent, Versioning deletion policies
- Include/exclude filter patterns
- Symlink exclusion option
- DST time shift handling
- Progress reporting with file-by-file output
- Dry run mode
- UNC share support with credential management
- Integration with SchedulerTemplate.psm1
## Support
For issues or questions:
1. Check the [Troubleshooting](#troubleshooting) section
2. Review script logs for error details
3. Verify all [Requirements](#requirements) are met
## License
See [LICENSE](../../LICENSE.md) in the root directory.
## Related Files
- `../SchedulerTemplate.psm1` - Shared scheduling and logging module
- `scriptsettings.json` - Configuration file
- `native-sync.bat` - Batch launcher
- `native-sync.ps1` - Main script

View File

@ -0,0 +1,74 @@
@echo off
setlocal EnableDelayedExpansion
REM ============================================================================
REM Native Sync Launcher
REM VERSION: 1.0.0
REM DATE: 2026-01-26
REM DESCRIPTION: Batch file launcher for native-sync.ps1 with admin check
REM ============================================================================
echo.
echo ============================================
echo Native PowerShell Sync Launcher
echo ============================================
echo.
REM Check for Administrator privileges
net session >nul 2>&1
if %errorLevel% NEQ 0 (
echo [ERROR] This script must be run as Administrator!
echo.
echo Please right-click and select "Run as administrator"
echo.
pause
exit /b 1
)
echo [OK] Running with Administrator privileges
echo.
REM Get script directory
set "SCRIPT_DIR=%~dp0"
set "PS_SCRIPT=%SCRIPT_DIR%native-sync.ps1"
REM Check if PowerShell script exists
if not exist "%PS_SCRIPT%" (
echo [ERROR] PowerShell script not found: %PS_SCRIPT%
echo.
pause
exit /b 1
)
echo [OK] Found PowerShell script: %PS_SCRIPT%
echo.
echo ============================================
echo Starting sync process...
echo ============================================
echo.
REM Execute PowerShell script
REM Note: Logging is handled by UScheduler service
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "%PS_SCRIPT%"
REM Capture exit code
set "EXIT_CODE=%ERRORLEVEL%"
echo.
echo ============================================
echo Sync process completed
echo Exit Code: %EXIT_CODE%
echo ============================================
echo.
if %EXIT_CODE% EQU 0 (
echo [SUCCESS] Sync completed successfully
) else (
echo [ERROR] Sync completed with errors
)
echo.
pause
endlocal
exit /b %EXIT_CODE%

View File

@ -0,0 +1,888 @@
[CmdletBinding()]
param (
[switch]$Automated,
[string]$CurrentDateTimeUtc
)
#Requires -RunAsAdministrator
<#
.SYNOPSIS
Native PowerShell File Synchronization Script
.DESCRIPTION
Production-ready file synchronization solution using pure PowerShell.
Supports Mirror, Update, and TwoWay sync modes with filtering,
progress reporting, and secure credential management.
.VERSION
1.0.0
.DATE
2026-01-26
.NOTES
- No external dependencies (pure PowerShell)
- Requires SchedulerTemplate.psm1 module
- Supports local and UNC paths
#>
# Script Version
$ScriptVersion = "1.0.0"
$ScriptDate = "2026-01-26"
try {
Import-Module "$PSScriptRoot\..\SchedulerTemplate.psm1" -Force -ErrorAction Stop
}
catch {
Write-Error "Failed to load SchedulerTemplate.psm1: $_"
exit 1
}
# Load Settings ============================================================
$settingsFile = Join-Path $PSScriptRoot "scriptsettings.json"
if (-not (Test-Path $settingsFile)) {
Write-Error "Settings file not found: $settingsFile"
exit 1
}
try {
$settings = Get-Content $settingsFile -Raw | ConvertFrom-Json
Write-Verbose "Loaded settings from $settingsFile"
}
catch {
Write-Error "Failed to load settings from $settingsFile : $_"
exit 1
}
# Process Settings =========================================================
# Validate required settings
$requiredSettings = @('syncMode', 'folderPairs')
foreach ($setting in $requiredSettings) {
if (-not $settings.$setting) {
Write-Error "Required setting '$setting' is missing or empty in $settingsFile"
exit 1
}
}
# Extract settings
$SyncMode = $settings.syncMode
$CompareMethod = if ($settings.compareMethod) { $settings.compareMethod } else { "TimeAndSize" }
$DeletionPolicy = if ($settings.deletionPolicy) { $settings.deletionPolicy } else { "RecycleBin" }
$VersioningFolder = $settings.versioningFolder
$FolderPairs = $settings.folderPairs
$Filters = $settings.filters
$Options = $settings.options
$NasRootShare = $settings.nasRootShare
$CredentialEnvVar = $settings.credentialEnvVar
# Get DryRun from settings
$DryRun = $Options.dryRun
# Schedule Configuration
$Config = @{
RunMonth = $settings.schedule.runMonth
RunWeekday = $settings.schedule.runWeekday
RunTime = $settings.schedule.runTime
MinIntervalMinutes = $settings.schedule.minIntervalMinutes
}
# End Settings =============================================================
# Global variables
$script:SyncStats = @{
StartTime = Get-Date
EndTime = $null
Success = $false
FilesScanned = 0
FilesCopied = 0
FilesUpdated = 0
FilesDeleted = 0
FilesSkipped = 0
BytesCopied = 0
Errors = 0
Warnings = 0
ErrorMessages = @()
}
# Helper Functions =========================================================
function Test-FilterMatch {
param(
[string]$RelativePath,
[array]$IncludePatterns,
[array]$ExcludePatterns
)
# Check exclude patterns first
foreach ($pattern in $ExcludePatterns) {
# Handle directory patterns (ending with \)
if ($pattern.EndsWith('\')) {
$dirPattern = $pattern.TrimEnd('\')
if ($RelativePath -like "*$dirPattern*") {
return $false
}
}
# Handle file patterns
elseif ($RelativePath -like $pattern) {
return $false
}
# Handle wildcard patterns like *\thumbs.db
elseif ($pattern.StartsWith('*\')) {
$filePattern = $pattern.Substring(2)
if ($RelativePath -like "*\$filePattern" -or $RelativePath -eq $filePattern) {
return $false
}
}
}
# Check include patterns
if ($IncludePatterns.Count -eq 0 -or ($IncludePatterns.Count -eq 1 -and $IncludePatterns[0] -eq '*')) {
return $true
}
foreach ($pattern in $IncludePatterns) {
if ($RelativePath -like $pattern) {
return $true
}
}
return $false
}
function Get-RelativePath {
param(
[string]$FullPath,
[string]$BasePath
)
$BasePath = $BasePath.TrimEnd('\')
if ($FullPath.StartsWith($BasePath, [StringComparison]::OrdinalIgnoreCase)) {
return $FullPath.Substring($BasePath.Length).TrimStart('\')
}
return $FullPath
}
function Get-LongPath {
param(
[string]$Path
)
# Add \\?\ prefix for long path support (>260 chars) on Windows
# Skip if already prefixed or if it's a relative path
if ($Path.StartsWith('\\?\') -or $Path.StartsWith('\\?\UNC\')) {
return $Path
}
# Handle UNC paths: \\server\share -> \\?\UNC\server\share
if ($Path.StartsWith('\\')) {
return '\\?\UNC\' + $Path.Substring(2)
}
# Handle local paths: C:\path -> \\?\C:\path
if ($Path.Length -ge 2 -and $Path[1] -eq ':') {
return '\\?\' + $Path
}
return $Path
}
function Compare-FileByTimeAndSize {
param(
[System.IO.FileInfo]$SourceFile,
[System.IO.FileInfo]$DestFile,
[switch]$IgnoreTimeShift
)
if (-not $DestFile -or -not $DestFile.Exists) {
return "LeftOnly"
}
if (-not $SourceFile -or -not $SourceFile.Exists) {
return "RightOnly"
}
# Compare sizes first
if ($SourceFile.Length -ne $DestFile.Length) {
if ($SourceFile.LastWriteTimeUtc -gt $DestFile.LastWriteTimeUtc) {
return "LeftNewer"
}
else {
return "RightNewer"
}
}
# Compare times
$timeDiff = ($SourceFile.LastWriteTimeUtc - $DestFile.LastWriteTimeUtc).TotalSeconds
# Handle DST time shift (1 hour = 3600 seconds)
if ($IgnoreTimeShift) {
$timeDiff = $timeDiff % 3600
}
if ([Math]::Abs($timeDiff) -lt 2) {
return "Same"
}
elseif ($timeDiff -gt 0) {
return "LeftNewer"
}
else {
return "RightNewer"
}
}
function Get-AllItems {
param(
[string]$Path,
[switch]$ExcludeSymlinks,
[switch]$Automated
)
$items = @{
Files = @{}
Directories = @{}
}
$longPath = Get-LongPath -Path $Path
if (-not (Test-Path -LiteralPath $longPath)) {
Write-Log "Path does not exist: $Path" -Level Warning -Automated:$Automated
return $items
}
try {
$allItems = Get-ChildItem -LiteralPath $longPath -Recurse -Force -ErrorAction SilentlyContinue
foreach ($item in $allItems) {
# Skip symlinks if configured
if ($ExcludeSymlinks -and $item.Attributes -band [System.IO.FileAttributes]::ReparsePoint) {
continue
}
# Strip the long path prefix for relative path calculation
$itemPath = $item.FullName
if ($itemPath.StartsWith('\\?\UNC\')) {
$itemPath = '\\' + $itemPath.Substring(8)
}
elseif ($itemPath.StartsWith('\\?\')) {
$itemPath = $itemPath.Substring(4)
}
$relativePath = Get-RelativePath -FullPath $itemPath -BasePath $Path
if ($item.PSIsContainer) {
$items.Directories[$relativePath] = $item
}
else {
$items.Files[$relativePath] = $item
}
}
}
catch {
Write-Log "Error scanning path $Path : $_" -Level Error -Automated:$Automated
$script:SyncStats.Errors++
}
return $items
}
function Get-SyncActions {
param(
[hashtable]$SourceItems,
[hashtable]$DestItems,
[string]$SyncMode,
[string]$SourcePath,
[string]$DestPath,
[array]$IncludePatterns,
[array]$ExcludePatterns,
[switch]$IgnoreTimeShift,
[switch]$Automated
)
$actions = @()
# Process source files
foreach ($relativePath in $SourceItems.Files.Keys) {
$script:SyncStats.FilesScanned++
# Check filters
if (-not (Test-FilterMatch -RelativePath $relativePath -IncludePatterns $IncludePatterns -ExcludePatterns $ExcludePatterns)) {
$script:SyncStats.FilesSkipped++
continue
}
$sourceFile = $SourceItems.Files[$relativePath]
$destFile = $DestItems.Files[$relativePath]
$comparison = Compare-FileByTimeAndSize -SourceFile $sourceFile -DestFile $destFile -IgnoreTimeShift:$IgnoreTimeShift
$action = $null
switch ($comparison) {
"LeftOnly" {
$action = @{
Type = "Copy"
Source = $sourceFile.FullName
Destination = Join-Path $DestPath $relativePath
RelativePath = $relativePath
Size = $sourceFile.Length
}
}
"LeftNewer" {
$action = @{
Type = "Update"
Source = $sourceFile.FullName
Destination = Join-Path $DestPath $relativePath
RelativePath = $relativePath
Size = $sourceFile.Length
}
}
"RightNewer" {
switch ($SyncMode) {
"Mirror" {
# Mirror overwrites right with left
$action = @{
Type = "Update"
Source = $sourceFile.FullName
Destination = Join-Path $DestPath $relativePath
RelativePath = $relativePath
Size = $sourceFile.Length
}
}
"TwoWay" {
# TwoWay updates left from right
$action = @{
Type = "Update"
Source = $destFile.FullName
Destination = $sourceFile.FullName
RelativePath = $relativePath
Size = $destFile.Length
Direction = "ToLeft"
}
}
# Update mode: skip
}
}
"Same" {
$script:SyncStats.FilesSkipped++
}
}
if ($action) {
$actions += $action
}
}
# Process destination-only files
foreach ($relativePath in $DestItems.Files.Keys) {
if ($SourceItems.Files.ContainsKey($relativePath)) {
continue # Already processed
}
$script:SyncStats.FilesScanned++
# Check filters
if (-not (Test-FilterMatch -RelativePath $relativePath -IncludePatterns $IncludePatterns -ExcludePatterns $ExcludePatterns)) {
$script:SyncStats.FilesSkipped++
continue
}
$destFile = $DestItems.Files[$relativePath]
switch ($SyncMode) {
"Mirror" {
$actions += @{
Type = "Delete"
Source = $null
Destination = $destFile.FullName
RelativePath = $relativePath
Size = $destFile.Length
}
}
"TwoWay" {
$actions += @{
Type = "Copy"
Source = $destFile.FullName
Destination = Join-Path $SourcePath $relativePath
RelativePath = $relativePath
Size = $destFile.Length
Direction = "ToLeft"
}
}
# Update mode: skip destination-only files
default {
$script:SyncStats.FilesSkipped++
}
}
}
# Process directories (for deletion in Mirror mode)
if ($SyncMode -eq "Mirror") {
foreach ($relativePath in $DestItems.Directories.Keys) {
if (-not $SourceItems.Directories.ContainsKey($relativePath)) {
# Check if directory should be excluded
if (-not (Test-FilterMatch -RelativePath $relativePath -IncludePatterns $IncludePatterns -ExcludePatterns $ExcludePatterns)) {
continue
}
$actions += @{
Type = "DeleteDir"
Source = $null
Destination = $DestItems.Directories[$relativePath].FullName
RelativePath = $relativePath
Size = 0
}
}
}
}
return $actions
}
function Remove-ToRecycleBin {
param(
[string]$Path
)
Add-Type -AssemblyName Microsoft.VisualBasic
if (Test-Path -LiteralPath $Path) {
$item = Get-Item -LiteralPath $Path
if ($item.PSIsContainer) {
[Microsoft.VisualBasic.FileIO.FileSystem]::DeleteDirectory(
$Path,
[Microsoft.VisualBasic.FileIO.UIOption]::OnlyErrorDialogs,
[Microsoft.VisualBasic.FileIO.RecycleOption]::SendToRecycleBin
)
}
else {
[Microsoft.VisualBasic.FileIO.FileSystem]::DeleteFile(
$Path,
[Microsoft.VisualBasic.FileIO.UIOption]::OnlyErrorDialogs,
[Microsoft.VisualBasic.FileIO.RecycleOption]::SendToRecycleBin
)
}
}
}
function Invoke-SyncAction {
param(
[hashtable]$Action,
[string]$DeletionPolicy,
[string]$VersioningFolder,
[switch]$DryRun,
[switch]$ShowProgress,
[switch]$Automated
)
$actionType = $Action.Type
$source = $Action.Source
$destination = $Action.Destination
$relativePath = $Action.RelativePath
$size = $Action.Size
$direction = if ($Action.Direction) { " ($($Action.Direction))" } else { "" }
# Format size for display
$sizeDisplay = if ($size -gt 1MB) {
"{0:N2} MB" -f ($size / 1MB)
}
elseif ($size -gt 1KB) {
"{0:N2} KB" -f ($size / 1KB)
}
else {
"$size bytes"
}
$prefix = if ($DryRun) { "[DRY-RUN] " } else { "" }
try {
switch ($actionType) {
"Copy" {
if ($ShowProgress) {
Write-Log "$prefix[COPY]$direction $relativePath ($sizeDisplay)" -Level Info -Automated:$Automated
}
if (-not $DryRun) {
$destDir = Split-Path $destination -Parent
$longDestDir = Get-LongPath -Path $destDir
$longSource = Get-LongPath -Path $source
$longDest = Get-LongPath -Path $destination
if (-not (Test-Path -LiteralPath $longDestDir)) {
New-Item -Path $longDestDir -ItemType Directory -Force | Out-Null
}
Copy-Item -LiteralPath $longSource -Destination $longDest -Force
$script:SyncStats.FilesCopied++
$script:SyncStats.BytesCopied += $size
}
else {
$script:SyncStats.FilesCopied++
}
}
"Update" {
if ($ShowProgress) {
Write-Log "$prefix[UPDATE]$direction $relativePath ($sizeDisplay)" -Level Info -Automated:$Automated
}
if (-not $DryRun) {
$longSource = Get-LongPath -Path $source
$longDest = Get-LongPath -Path $destination
Copy-Item -LiteralPath $longSource -Destination $longDest -Force
$script:SyncStats.FilesUpdated++
$script:SyncStats.BytesCopied += $size
}
else {
$script:SyncStats.FilesUpdated++
}
}
"Delete" {
if ($ShowProgress) {
Write-Log "$prefix[DELETE] $relativePath ($sizeDisplay)" -Level Warning -Automated:$Automated
}
if (-not $DryRun) {
$longDest = Get-LongPath -Path $destination
switch ($DeletionPolicy) {
"RecycleBin" {
Remove-ToRecycleBin -Path $destination
}
"Versioning" {
if ($VersioningFolder) {
$versionPath = Join-Path $VersioningFolder $relativePath
$versionDir = Split-Path $versionPath -Parent
$longVersionDir = Get-LongPath -Path $versionDir
$longVersionPath = Get-LongPath -Path $versionPath
if (-not (Test-Path -LiteralPath $longVersionDir)) {
New-Item -Path $longVersionDir -ItemType Directory -Force | Out-Null
}
Move-Item -LiteralPath $longDest -Destination $longVersionPath -Force
}
else {
Remove-Item -LiteralPath $longDest -Force
}
}
default {
# Permanent
Remove-Item -LiteralPath $longDest -Force
}
}
$script:SyncStats.FilesDeleted++
}
else {
$script:SyncStats.FilesDeleted++
}
}
"DeleteDir" {
if ($ShowProgress) {
Write-Log "$prefix[DELETE DIR] $relativePath" -Level Warning -Automated:$Automated
}
if (-not $DryRun) {
$longDest = Get-LongPath -Path $destination
switch ($DeletionPolicy) {
"RecycleBin" {
Remove-ToRecycleBin -Path $destination
}
"Versioning" {
if ($VersioningFolder) {
$versionPath = Join-Path $VersioningFolder $relativePath
$versionParent = Split-Path $versionPath -Parent
$longVersionParent = Get-LongPath -Path $versionParent
$longVersionPath = Get-LongPath -Path $versionPath
if (-not (Test-Path -LiteralPath $longVersionParent)) {
New-Item -Path $longVersionParent -ItemType Directory -Force | Out-Null
}
Move-Item -LiteralPath $longDest -Destination $longVersionPath -Force
}
else {
Remove-Item -LiteralPath $longDest -Recurse -Force
}
}
default {
Remove-Item -LiteralPath $longDest -Recurse -Force
}
}
}
}
}
return $true
}
catch {
Write-Log "Error executing $actionType on $relativePath : $_" -Level Error -Automated:$Automated
$script:SyncStats.Errors++
$script:SyncStats.ErrorMessages += "[$actionType] $relativePath : $_"
return $false
}
}
function Connect-NasShare {
param(
[string]$SharePath,
[switch]$Automated
)
# Check if path is UNC
if (-not $SharePath.StartsWith("\\")) {
Write-Log "NAS path is local, no authentication needed" -Level Info -Automated:$Automated
return $true
}
# Validate UNC path format
if (-not (Test-UNCPath -Path $SharePath)) {
Write-Log "Invalid UNC path format: $SharePath (expected \\server\share)" -Level Error -Automated:$Automated
return $false
}
Write-Log "Authenticating to NAS share: $SharePath" -Level Info -Automated:$Automated
# Validate credential environment variable name is configured
if (-not $CredentialEnvVar) {
Write-Log "credentialEnvVar is not configured in settings for UNC path authentication" -Level Error -Automated:$Automated
return $false
}
try {
# Retrieve credentials from environment variable
$creds = Get-CredentialFromEnvVar -EnvVarName $CredentialEnvVar -Automated:$Automated
if (-not $creds) {
return $false
}
# Check if already connected
$existingConnection = net use | Select-String $SharePath
if ($existingConnection) {
Write-Log "Already connected to $SharePath" -Level Info -Automated:$Automated
return $true
}
# Connect to share
$netUseResult = cmd /c "net use $SharePath $($creds.Password) /user:$($creds.Username) 2>&1"
if ($LASTEXITCODE -ne 0) {
Write-Log "Failed to connect to $SharePath : $netUseResult" -Level Error -Automated:$Automated
return $false
}
Write-Log "Successfully authenticated to $SharePath" -Level Success -Automated:$Automated
return $true
}
catch {
Write-Log "Error connecting to share: $_" -Level Error -Automated:$Automated
return $false
}
}
function Write-SyncSummary {
param([switch]$Automated)
$script:SyncStats.EndTime = Get-Date
$duration = $script:SyncStats.EndTime - $script:SyncStats.StartTime
# Format bytes
$bytesDisplay = if ($script:SyncStats.BytesCopied -gt 1GB) {
"{0:N2} GB" -f ($script:SyncStats.BytesCopied / 1GB)
}
elseif ($script:SyncStats.BytesCopied -gt 1MB) {
"{0:N2} MB" -f ($script:SyncStats.BytesCopied / 1MB)
}
elseif ($script:SyncStats.BytesCopied -gt 1KB) {
"{0:N2} KB" -f ($script:SyncStats.BytesCopied / 1KB)
}
else {
"$($script:SyncStats.BytesCopied) bytes"
}
Write-Log "" -Level Info -Automated:$Automated
Write-Log "========================================" -Level Info -Automated:$Automated
Write-Log "SYNC SUMMARY" -Level Info -Automated:$Automated
Write-Log "========================================" -Level Info -Automated:$Automated
Write-Log "Start Time : $($script:SyncStats.StartTime.ToString('yyyy-MM-dd HH:mm:ss'))" -Level Info -Automated:$Automated
Write-Log "End Time : $($script:SyncStats.EndTime.ToString('yyyy-MM-dd HH:mm:ss'))" -Level Info -Automated:$Automated
Write-Log "Duration : $($duration.Hours)h $($duration.Minutes)m $($duration.Seconds)s" -Level Info -Automated:$Automated
Write-Log "Status : $(if ($script:SyncStats.Errors -eq 0) { 'SUCCESS' } else { 'COMPLETED WITH ERRORS' })" -Level $(if ($script:SyncStats.Errors -eq 0) { 'Success' } else { 'Warning' }) -Automated:$Automated
Write-Log "" -Level Info -Automated:$Automated
Write-Log "Files Scanned : $($script:SyncStats.FilesScanned)" -Level Info -Automated:$Automated
Write-Log "Files Copied : $($script:SyncStats.FilesCopied)" -Level Info -Automated:$Automated
Write-Log "Files Updated : $($script:SyncStats.FilesUpdated)" -Level Info -Automated:$Automated
Write-Log "Files Deleted : $($script:SyncStats.FilesDeleted)" -Level Info -Automated:$Automated
Write-Log "Files Skipped : $($script:SyncStats.FilesSkipped)" -Level Info -Automated:$Automated
Write-Log "Bytes Copied : $bytesDisplay" -Level Info -Automated:$Automated
Write-Log "" -Level Info -Automated:$Automated
Write-Log "Errors : $($script:SyncStats.Errors)" -Level $(if ($script:SyncStats.Errors -eq 0) { 'Info' } else { 'Error' }) -Automated:$Automated
Write-Log "Warnings : $($script:SyncStats.Warnings)" -Level Info -Automated:$Automated
if ($script:SyncStats.ErrorMessages.Count -gt 0) {
Write-Log "" -Level Info -Automated:$Automated
Write-Log "Error Details:" -Level Error -Automated:$Automated
foreach ($msg in $script:SyncStats.ErrorMessages) {
Write-Log " - $msg" -Level Error -Automated:$Automated
}
}
Write-Log "========================================" -Level Info -Automated:$Automated
$script:SyncStats.Success = ($script:SyncStats.Errors -eq 0)
}
# Main Business Logic ======================================================
function Start-BusinessLogic {
param(
[switch]$Automated,
[switch]$DryRun
)
Write-Log "========================================" -Level Info -Automated:$Automated
Write-Log "Native PowerShell Sync Started" -Level Info -Automated:$Automated
Write-Log "Script Version: $ScriptVersion ($ScriptDate)" -Level Info -Automated:$Automated
Write-Log "Sync Mode: $SyncMode" -Level Info -Automated:$Automated
Write-Log "Compare Method: $CompareMethod" -Level Info -Automated:$Automated
Write-Log "Deletion Policy: $DeletionPolicy" -Level Info -Automated:$Automated
if ($DryRun) {
Write-Log "DRY RUN MODE - No changes will be made" -Level Warning -Automated:$Automated
}
Write-Log "========================================" -Level Info -Automated:$Automated
# Connect to NAS share if needed
if ($NasRootShare) {
if (-not (Connect-NasShare -SharePath $NasRootShare -Automated:$Automated)) {
Write-Log "Failed to connect to NAS share. Aborting sync." -Level Error -Automated:$Automated
exit 1
}
}
# Process each folder pair
$pairIndex = 0
foreach ($pair in $FolderPairs) {
$pairIndex++
$leftPath = $pair.left
$rightPath = $pair.right
# Skip empty pairs
if ([string]::IsNullOrWhiteSpace($leftPath) -or [string]::IsNullOrWhiteSpace($rightPath)) {
continue
}
Write-Log "" -Level Info -Automated:$Automated
Write-Log "Processing Folder Pair $pairIndex" -Level Info -Automated:$Automated
Write-Log " Left: $leftPath" -Level Info -Automated:$Automated
Write-Log " Right: $rightPath" -Level Info -Automated:$Automated
# Verify paths exist (using long path support)
$longLeftPath = Get-LongPath -Path $leftPath
$longRightPath = Get-LongPath -Path $rightPath
if (-not (Test-Path -LiteralPath $longLeftPath)) {
Write-Log "Source path does not exist: $leftPath" -Level Error -Automated:$Automated
$script:SyncStats.Errors++
continue
}
# Create destination if it doesn't exist
if (-not (Test-Path -LiteralPath $longRightPath)) {
if (-not $DryRun) {
try {
New-Item -Path $longRightPath -ItemType Directory -Force | Out-Null
Write-Log "Created destination directory: $rightPath" -Level Info -Automated:$Automated
}
catch {
Write-Log "Failed to create destination: $rightPath - $_" -Level Error -Automated:$Automated
$script:SyncStats.Errors++
continue
}
}
else {
Write-Log "[DRY-RUN] Would create destination directory: $rightPath" -Level Info -Automated:$Automated
}
}
# Scan directories
Write-Log "Scanning source directory..." -Level Info -Automated:$Automated
$sourceItems = Get-AllItems -Path $leftPath -ExcludeSymlinks:$Options.excludeSymlinks -Automated:$Automated
Write-Log " Found $($sourceItems.Files.Count) files, $($sourceItems.Directories.Count) directories" -Level Info -Automated:$Automated
Write-Log "Scanning destination directory..." -Level Info -Automated:$Automated
$destItems = Get-AllItems -Path $rightPath -ExcludeSymlinks:$Options.excludeSymlinks -Automated:$Automated
Write-Log " Found $($destItems.Files.Count) files, $($destItems.Directories.Count) directories" -Level Info -Automated:$Automated
# Get include/exclude patterns
$includePatterns = if ($Filters -and $Filters.include) { $Filters.include } else { @('*') }
$excludePatterns = if ($Filters -and $Filters.exclude) { $Filters.exclude } else { @() }
# Calculate sync actions
Write-Log "Calculating sync actions..." -Level Info -Automated:$Automated
$actions = Get-SyncActions `
-SourceItems $sourceItems `
-DestItems $destItems `
-SyncMode $SyncMode `
-SourcePath $leftPath `
-DestPath $rightPath `
-IncludePatterns $includePatterns `
-ExcludePatterns $excludePatterns `
-IgnoreTimeShift:$Options.ignoreTimeShift `
-Automated:$Automated
Write-Log " $($actions.Count) actions to perform" -Level Info -Automated:$Automated
# Execute actions
if ($actions.Count -gt 0) {
Write-Log "" -Level Info -Automated:$Automated
Write-Log "Executing sync actions..." -Level Info -Automated:$Automated
# Sort actions: copies first, then updates, then deletes (files before directories)
$sortedActions = $actions | Sort-Object {
switch ($_.Type) {
"Copy" { 1 }
"Update" { 2 }
"Delete" { 3 }
"DeleteDir" { 4 }
}
}
# For directory deletions, sort by path length descending (delete deepest first)
$dirDeletes = $sortedActions | Where-Object { $_.Type -eq "DeleteDir" } | Sort-Object { $_.RelativePath.Length } -Descending
$otherActions = $sortedActions | Where-Object { $_.Type -ne "DeleteDir" }
$sortedActions = @($otherActions) + @($dirDeletes)
foreach ($action in $sortedActions) {
Invoke-SyncAction `
-Action $action `
-DeletionPolicy $DeletionPolicy `
-VersioningFolder $VersioningFolder `
-DryRun:$DryRun `
-ShowProgress:$Options.showProgress `
-Automated:$Automated | Out-Null
}
}
}
# Print summary
Write-SyncSummary -Automated:$Automated
# Exit with appropriate code
if ($script:SyncStats.Errors -gt 0) {
exit 1
}
}
# Entry Point ==============================================================
if ($Automated) {
if (Get-Command Invoke-ScheduledExecution -ErrorAction SilentlyContinue) {
$params = @{
Config = $Config
Automated = $Automated
CurrentDateTimeUtc = $CurrentDateTimeUtc
ScriptBlock = { Start-BusinessLogic -Automated:$Automated -DryRun:$DryRun }
}
Invoke-ScheduledExecution @params
}
else {
Write-Log "Invoke-ScheduledExecution not available. Execution aborted." -Level Error -Automated:$Automated
exit 1
}
}
else {
Write-Log "Manual execution started" -Level Info -Automated:$Automated
Start-BusinessLogic -Automated:$Automated -DryRun:$DryRun
}

View File

@ -0,0 +1,68 @@
{
"$schema": "https://json-schema.org/draft-07/schema",
"title": "Native Sync Script Settings",
"description": "Configuration file for native-sync.ps1 script (pure PowerShell file synchronization)",
"version": "1.0.0",
"lastModified": "2026-01-26",
"schedule": {
"runMonth": [],
"runWeekday": ["Monday"],
"runTime": ["00:00"],
"minIntervalMinutes": 10
},
"syncMode": "Mirror",
"compareMethod": "TimeAndSize",
"deletionPolicy": "RecycleBin",
"versioningFolder": "",
"folderPairs": [
{
"left": "E:\\Users\\maksym\\source",
"right": "\\\\nassrv0001.corp.maks-it.com\\data-1\\Users\\maksym\\source"
}
],
"filters": {
"include": ["*"],
"exclude": [
"\\System Volume Information\\",
"\\$Recycle.Bin\\",
"\\RECYCLE?\\",
"\\Recovery\\",
"*\\thumbs.db"
]
},
"options": {
"excludeSymlinks": true,
"ignoreTimeShift": false,
"showProgress": true,
"dryRun": false
},
"nasRootShare": "\\\\nassrv0001.corp.maks-it.com\\data-1",
"credentialEnvVar": "nassrv0001",
"_comments": {
"version": "Configuration schema version",
"lastModified": "Last modification date (YYYY-MM-DD)",
"schedule": {
"runMonth": "Array of month names (e.g. 'January', 'June', 'December') to run sync. Empty array = every month.",
"runWeekday": "Array of weekday names (e.g. 'Monday', 'Friday') to run sync. Empty array = every day.",
"runTime": "Array of UTC times in HH:mm format when sync should run.",
"minIntervalMinutes": "Minimum minutes between sync runs to prevent duplicate executions."
},
"syncMode": "Synchronization mode: 'Mirror' (make right identical to left), 'Update' (copy new/updated to right only), 'TwoWay' (propagate changes both ways)",
"compareMethod": "File comparison method: 'TimeAndSize' (compare by modification time and file size)",
"deletionPolicy": "How to handle deleted files: 'RecycleBin' (move to recycle bin), 'Permanent' (delete permanently), 'Versioning' (move to versioning folder)",
"versioningFolder": "Path for versioning folder when deletionPolicy is 'Versioning'. Leave empty if not using versioning.",
"folderPairs": "Array of folder pairs to synchronize. Each pair has 'left' (source) and 'right' (destination) paths.",
"filters": {
"include": "Array of patterns to include. Use '*' to include all files.",
"exclude": "Array of patterns to exclude. Supports wildcards and path patterns."
},
"options": {
"excludeSymlinks": "Skip symbolic links during synchronization",
"ignoreTimeShift": "Ignore 1-hour time differences (DST changes)",
"showProgress": "Display file-by-file progress during sync",
"dryRun": "Simulate sync without making changes (can also use -DryRun parameter)"
},
"nasRootShare": "UNC path to NAS root share for authentication. Only used for connecting to the share.",
"credentialEnvVar": "Name of Machine-level environment variable containing Base64-encoded 'username:password' for NAS authentication"
}
}