feat: Implement provisioning pipelines for subscription management

- Add ReactivatePipeline to handle subscription reactivation, including scaling Docker services, health verification, status updates, audit logging, and broadcasting status changes.
- Introduce RotateCredentialsPipeline for OAuth2 credential rotation, managing the deletion of old apps, creation of new ones, credential storage, access verification, and audit logging.
- Create StepRunner to manage job step execution, including lifecycle management and progress broadcasting via SignalR.
- Implement SuspendPipeline for subscription suspension, scaling down services, updating statuses, logging audits, and broadcasting changes.
- Add UpdateScreenLimitPipeline to update Xibo CMS screen limits and record snapshots.
- Introduce XiboFeatureManifests for hardcoded feature ACLs per role.
- Add docker-compose.dev.yml for local development with PostgreSQL setup.

.env.example

@@ -0,0 +1,11 @@
+# OTSSignsOrchestrator.Server — required environment variables
+# Copy to .env and fill in real values.
+
+ConnectionStrings__OrchestratorDb=Host=localhost;Port=5432;Database=orchestrator_dev;Username=ots;Password=devpassword
+Stripe__WebhookSecret=whsec_...
+Stripe__SecretKey=sk_test_...
+Jwt__Key=change-me-to-a-random-256-bit-key
+Authentik__BaseUrl=https://auth.example.com
+Authentik__ApiToken=
+SendGrid__ApiKey=SG....
+OTS_SIGNS_SERVER_URL=http://localhost:5000

.github/copilot-instructions.md

@@ -15,9 +15,30 @@ Layered MVVM desktop app for provisioning and managing Xibo CMS instances on Doc
 - Bitwarden Secrets Manager is the source of truth for all sensitive config. `SettingsService` caches in-memory.
 - Local SQLite DB (`otssigns-desktop.db`) stores SSH hosts + operation logs. Credentials encrypted via Data Protection API.
 
+### Scope & file discipline
+**The Server project is net-new — keep concerns separated.**
+- Never modify `OTSSignsOrchestrator.Core` or `OTSSignsOrchestrator.Desktop` unless the prompt explicitly says to.
+- When in doubt, add new code to `OTSSignsOrchestrator.Server`.
+- Never modify `XiboContext.cs` without explicit instruction.
+
 ### External integrations
 Xibo CMS REST API (OAuth2), Authentik SAML IdP, Bitwarden Secrets SDK, Docker Swarm (via SSH), Git (LibGit2Sharp), MySQL 8.4, NFS volumes, Pangolin/Newt VPN.
 
+#### Xibo API rules — non-negotiable
+- `GET /api/application` is **BLOCKED**. Only POST and DELETE exist.
+- All group endpoints are `/api/group`, never `/api/usergroup`.
+- Feature assignment is `POST /api/group/{id}/acl`, NOT `/features`.
+- Xibo paginates at 10 items by default. **Always pass `length=200`** and use `GetAllPagesAsync` for every list call. Missing this causes silent data truncation.
+- OAuth2 client secret is returned **ONCE** in the `POST /api/application` response. Capture it immediately — **it cannot be retrieved again**.
+
+#### Stripe webhooks — idempotency is mandatory
+- Every Stripe webhook handler must check `OrchestratorDbContext.StripeEvents` for the `stripe_event_id` before processing anything.
+- Insert the `StripeEvent` row first, then process the webhook. This is not optional — duplicate webhook delivery is guaranteed by Stripe.
+
+#### No AI autonomy in infrastructure actions
+- Never generate any endpoint or method that sends a message, makes an external call, or takes infrastructure action without an explicit operator-initiated `Job` record being created first.
+- All automated actions flow through the `ProvisioningWorker` job queue.
+
 ## Build and Test
 
 ```bash
@@ -34,6 +55,16 @@ dotnet run --project OTSSignsOrchestrator.Desktop/OTSSignsOrchestrator.Desktop.c
 - Runtime identifiers: `linux-x64`, `win-x64`, `osx-x64`, `osx-arm64`
 - EF Core migrations in `OTSSignsOrchestrator.Core/Migrations/`
 
+### Test coverage non-negotiables
+Unit tests are **required** for:
+- Evidence hashing and tamper detection
+- AI context assembly
+- Pattern detection ruleset engine
+- `AbbreviationService` uniqueness logic
+- Stripe webhook idempotency
+
+Integration tests **require** Testcontainers with a real PostgreSQL 16 instance — **no SQLite substitutions**.
+
 ## Conventions
 
 ### ViewModels
@@ -42,6 +73,13 @@ dotnet run --project OTSSignsOrchestrator.Desktop/OTSSignsOrchestrator.Desktop.c
 - Resolve services from `IServiceProvider` in constructors. Navigation via `MainWindowViewModel.CurrentView`.
 - Confirmation dialogs use `Func<string, string, Task<bool>> ConfirmAsync` property — wired by the View.
 
+### Avalonia threading — critical for stability
+All SignalR message handlers and background thread continuations that touch `ObservableProperty` or `ObservableCollection` **MUST** be wrapped in:
+```csharp
+Avalonia.Threading.Dispatcher.UIThread.InvokeAsync(() => { ... });
+```
+**Failure to do this causes silent cross-thread exceptions in Avalonia.** Never suggest direct property assignment from a non-UI thread.
+
 ### Views (Avalonia XAML)
 - Compiled bindings enabled (`x:CompileBindings="True"`). DataTemplates in `MainWindow.axaml` map ViewModel types to View UserControls.
 - Layout: DockPanel with status bar (bottom), sidebar nav (left), dynamic ContentControl (center).
@@ -58,11 +96,23 @@ dotnet run --project OTSSignsOrchestrator.Desktop/OTSSignsOrchestrator.Desktop.c
 - Secret names built via `AppConstants` helpers (e.g., `CustomerMysqlPasswordSecretName(abbrev)`).
 - `AppConstants.SanitizeName()` filters to `[a-z0-9_-]`.
 
+### Credential handling
+Never store OAuth2 client secrets, Stripe keys, or SSH passwords in the database. Secrets go to the Bitwarden CLI wrapper only. `OauthAppRegistry` stores `clientId` only — never the secret. Log credentials to `JobStep` output **ONLY** as a last-resort break-glass fallback, and mark it explicitly as emergency recovery data in the log.
+
+### Code generation verification
+After generating any class that implements an interface, **verify all interface members are implemented.** After generating any pipeline, **verify all steps are implemented as `JobStep` entities with progress broadcast via `IHubContext<FleetHub>`.** Do not stub steps as TODO — implement them fully or flag explicitly that the step requires external infrastructure access that cannot be completed in this context.
+
 ### Data layer
 - Entities in `Core/Models/Entities/`, DTOs in `Core/Models/DTOs/`.
 - `XiboContext` applies unique index on `SshHost.Label` and encrypts credential fields.
 - Add new migrations via: `dotnet ef migrations add <Name> --project OTSSignsOrchestrator.Core --startup-project OTSSignsOrchestrator.Desktop`
 
+### Immutability enforcement
+**AuditLog, Message, and Evidence are append-only by design.** Never generate `Update()` or `Delete()` methods on these repositories. Add an explicit comment on each repository class:
+```csharp
+// IMMUTABLE — no update or delete operations permitted.
+```
+
 ## Pitfalls
 
 - **SSH singleton state**: `SshDockerCliService.SetHost()` must be called before each host operation — stale host causes silent failures.

OTSSignsOrchestrator.Desktop/App.axaml.cs

@@ -5,8 +5,12 @@ using Microsoft.AspNetCore.DataProtection;
 using Microsoft.EntityFrameworkCore;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Http;
 using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Options;
+using Polly;
+using Polly.Extensions.Http;
+using Refit;
 using Serilog;
 using OTSSignsOrchestrator.Core.Configuration;
 using OTSSignsOrchestrator.Core.Data;
@@ -81,10 +85,26 @@ public class App : Application
             window.Activate();
             Log.Information("MainWindow Show() + Activate() called");
 
+            // Start the SignalR connection (fire-and-forget, reconnect handles failures)
+            _ = Task.Run(async () =>
+            {
+                try
+                {
+                    var signalR = Services.GetRequiredService<ServerSignalRService>();
+                    await signalR.StartAsync();
+                }
+                catch (Exception ex)
+                {
+                    Log.Warning(ex, "Failed to start SignalR connection on startup");
+                }
+            });
+
             desktop.ShutdownRequested += (_, _) =>
             {
                 var ssh = Services.GetService<SshConnectionService>();
                 ssh?.Dispose();
+                var signalR = Services.GetService<ServerSignalRService>();
+                signalR?.DisposeAsync().AsTask().GetAwaiter().GetResult();
             };
         }
         else
@@ -140,6 +160,20 @@ public class App : Application
         services.AddHttpClient("XiboHealth");
         services.AddHttpClient("AuthentikApi");
 
+        // ── Server API integration ──────────────────────────────────────────
+        services.AddSingleton<TokenStoreService>();
+        services.AddTransient<AuthHeaderHandler>();
+
+        var serverBaseUrl = config["ServerBaseUrl"] ?? "https://localhost:5001";
+        services.AddRefitClient<IServerApiClient>()
+            .ConfigureHttpClient(c => c.BaseAddress = new Uri(serverBaseUrl))
+            .AddHttpMessageHandler<AuthHeaderHandler>()
+            .AddPolicyHandler(HttpPolicyExtensions
+                .HandleTransientHttpError()
+                .WaitAndRetryAsync(3, attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt))));
+
+        services.AddSingleton<ServerSignalRService>();
+
         // SSH services (singletons — maintain connections)
         services.AddSingleton<SshConnectionService>();
 

OTSSignsOrchestrator.Desktop/Models/LiveStackItem.cs

@@ -22,4 +22,10 @@ public class LiveStackItem
 
     /// <summary>Label of the host — convenience property for data-binding.</summary>
     public string HostLabel => Host?.Label ?? string.Empty;
+
+    /// <summary>
+    /// Server-side customer ID. Populated when fleet data is loaded from the server API.
+    /// Null when loaded only from local Docker discovery.
+    /// </summary>
+    public Guid? CustomerId { get; set; }
 }

OTSSignsOrchestrator.Desktop/OTSSignsOrchestrator.Desktop.csproj

@@ -32,6 +32,9 @@
     <PackageReference Include="Serilog.Extensions.Logging" Version="9.0.0" />
     <PackageReference Include="Serilog.Sinks.Console" Version="6.0.0" />
     <PackageReference Include="Serilog.Sinks.File" Version="6.0.0" />
+    <PackageReference Include="Microsoft.AspNetCore.SignalR.Client" Version="9.0.2" />
+    <PackageReference Include="Microsoft.Extensions.Http.Polly" Version="9.0.2" />
+    <PackageReference Include="Refit.HttpClientFactory" Version="8.0.0" />
     <PackageReference Include="SSH.NET" Version="2024.2.0" />
   </ItemGroup>
 

OTSSignsOrchestrator.Desktop/Services/IServerApiClient.cs

@@ -0,0 +1,152 @@
+using Refit;
+
+namespace OTSSignsOrchestrator.Desktop.Services;
+
+// ── DTOs matching server REST API responses ─────────────────────────────────
+
+public record FleetSummaryDto
+{
+    public Guid CustomerId { get; init; }
+    public string Abbreviation { get; init; } = string.Empty;
+    public string CompanyName { get; init; } = string.Empty;
+    public string Plan { get; init; } = string.Empty;
+    public int ScreenCount { get; init; }
+    public string HealthStatus { get; init; } = "Unknown";
+    public DateTime? LastHealthCheck { get; init; }
+    public bool HasRunningJob { get; init; }
+}
+
+public record CustomerDetailDto
+{
+    public Guid Id { get; init; }
+    public string Abbreviation { get; init; } = string.Empty;
+    public string CompanyName { get; init; } = string.Empty;
+    public string? AdminEmail { get; init; }
+    public string Plan { get; init; } = string.Empty;
+    public int ScreenCount { get; init; }
+    public string Status { get; init; } = string.Empty;
+    public DateTime CreatedAt { get; init; }
+    public List<CustomerInstanceDto> Instances { get; init; } = [];
+    public List<CustomerJobDto> ActiveJobs { get; init; } = [];
+}
+
+public record CustomerInstanceDto
+{
+    public Guid Id { get; init; }
+    public string? XiboUrl { get; init; }
+    public string? DockerStackName { get; init; }
+    public string HealthStatus { get; init; } = "Unknown";
+    public DateTime? LastHealthCheck { get; init; }
+}
+
+public record CustomerJobDto
+{
+    public Guid Id { get; init; }
+    public string JobType { get; init; } = string.Empty;
+    public string Status { get; init; } = string.Empty;
+    public DateTime CreatedAt { get; init; }
+    public DateTime? StartedAt { get; init; }
+}
+
+public record CreateJobRequest(Guid CustomerId, string JobType, string? Parameters);
+
+public record CreateJobResponse
+{
+    public Guid Id { get; init; }
+    public string JobType { get; init; } = string.Empty;
+    public string Status { get; init; } = string.Empty;
+}
+
+public record JobDetailDto
+{
+    public Guid Id { get; init; }
+    public Guid CustomerId { get; init; }
+    public string JobType { get; init; } = string.Empty;
+    public string Status { get; init; } = string.Empty;
+    public string? TriggeredBy { get; init; }
+    public string? Parameters { get; init; }
+    public DateTime CreatedAt { get; init; }
+    public DateTime? StartedAt { get; init; }
+    public DateTime? CompletedAt { get; init; }
+    public string? ErrorMessage { get; init; }
+    public List<JobStepDto> Steps { get; init; } = [];
+}
+
+public record JobStepDto
+{
+    public Guid Id { get; init; }
+    public string StepName { get; init; } = string.Empty;
+    public string Status { get; init; } = string.Empty;
+    public string? LogOutput { get; init; }
+    public DateTime? StartedAt { get; init; }
+    public DateTime? CompletedAt { get; init; }
+}
+
+public record LoginRequest(string Email, string Password);
+public record RefreshRequest(string RefreshToken);
+
+public record AuthResponse
+{
+    public string Token { get; init; } = string.Empty;
+    public string RefreshToken { get; init; } = string.Empty;
+}
+
+public record RefreshResponse
+{
+    public string Token { get; init; } = string.Empty;
+}
+
+// ── Refit interface ─────────────────────────────────────────────────────────
+
+[Headers("Accept: application/json")]
+public interface IServerApiClient
+{
+    [Get("/api/fleet")]
+    Task<List<FleetSummaryDto>> GetFleetAsync();
+
+    [Get("/api/fleet/{id}")]
+    Task<CustomerDetailDto> GetCustomerDetailAsync(Guid id);
+
+    [Post("/api/jobs")]
+    Task<CreateJobResponse> CreateJobAsync([Body] CreateJobRequest body);
+
+    [Get("/api/jobs/{id}")]
+    Task<JobDetailDto> GetJobAsync(Guid id);
+
+    [Post("/api/auth/login")]
+    Task<AuthResponse> LoginAsync([Body] LoginRequest body);
+
+    [Post("/api/auth/refresh")]
+    Task<RefreshResponse> RefreshAsync([Body] RefreshRequest body);
+
+    [Get("/api/reports/billing")]
+    Task<HttpResponseMessage> GetBillingCsvAsync([Query] DateOnly from, [Query] DateOnly to);
+
+    [Get("/api/reports/fleet-health")]
+    Task<HttpResponseMessage> GetFleetHealthPdfAsync();
+
+    [Post("/api/fleet/bulk/{action}")]
+    Task<HttpResponseMessage> BulkActionAsync(string action);
+}
+
+// ── DelegatingHandler for Bearer token injection ────────────────────────────
+
+public class AuthHeaderHandler : DelegatingHandler
+{
+    private readonly TokenStoreService _tokenStore;
+
+    public AuthHeaderHandler(TokenStoreService tokenStore)
+    {
+        _tokenStore = tokenStore;
+    }
+
+    protected override async Task<HttpResponseMessage> SendAsync(
+        HttpRequestMessage request, CancellationToken cancellationToken)
+    {
+        var jwt = _tokenStore.GetJwt();
+        if (!string.IsNullOrEmpty(jwt))
+            request.Headers.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", jwt);
+
+        return await base.SendAsync(request, cancellationToken);
+    }
+}

OTSSignsOrchestrator.Desktop/Services/ServerSignalRService.cs

@@ -0,0 +1,112 @@
+using Avalonia.Threading;
+using CommunityToolkit.Mvvm.Messaging;
+using Microsoft.AspNetCore.SignalR.Client;
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.Logging;
+
+namespace OTSSignsOrchestrator.Desktop.Services;
+
+/// <summary>
+/// Singleton service managing the persistent SignalR connection to the server's FleetHub.
+/// All handlers dispatch to the UI thread and republish via <see cref="WeakReferenceMessenger"/>.
+/// </summary>
+public sealed class ServerSignalRService : IAsyncDisposable
+{
+    private readonly HubConnection _connection;
+    private readonly ILogger<ServerSignalRService> _logger;
+
+    public ServerSignalRService(
+        TokenStoreService tokenStore,
+        IConfiguration config,
+        ILogger<ServerSignalRService> logger)
+    {
+        _logger = logger;
+        var baseUrl = config["ServerBaseUrl"] ?? "https://localhost:5001";
+
+        _connection = new HubConnectionBuilder()
+            .WithUrl($"{baseUrl}/hubs/fleet", options =>
+            {
+                options.AccessTokenProvider = () => Task.FromResult(tokenStore.GetJwt());
+            })
+            .WithAutomaticReconnect(new[] { TimeSpan.Zero, TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(30) })
+            .Build();
+
+        RegisterHandlers();
+
+        _connection.Reconnecting += ex =>
+        {
+            _logger.LogWarning(ex, "SignalR reconnecting...");
+            return Task.CompletedTask;
+        };
+
+        _connection.Reconnected += connectionId =>
+        {
+            _logger.LogInformation("SignalR reconnected (connId={ConnectionId})", connectionId);
+            return Task.CompletedTask;
+        };
+
+        _connection.Closed += ex =>
+        {
+            _logger.LogWarning(ex, "SignalR connection closed");
+            return Task.CompletedTask;
+        };
+    }
+
+    /// <summary>
+    /// Starts the SignalR connection. Call from <c>App.OnFrameworkInitializationCompleted</c>.
+    /// Failures are logged but do not throw — automatic reconnect will retry.
+    /// </summary>
+    public async Task StartAsync()
+    {
+        try
+        {
+            await _connection.StartAsync();
+            _logger.LogInformation("SignalR connected to FleetHub");
+        }
+        catch (Exception ex)
+        {
+            _logger.LogWarning(ex, "SignalR initial connection failed — will retry via automatic reconnect");
+        }
+    }
+
+    public async Task StopAsync()
+    {
+        try { await _connection.StopAsync(); }
+        catch (Exception ex) { _logger.LogWarning(ex, "Error stopping SignalR connection"); }
+    }
+
+    public HubConnectionState State => _connection.State;
+
+    private void RegisterHandlers()
+    {
+        _connection.On<string, string, string>("SendJobCreated", (jobId, abbrev, jobType) =>
+            Dispatcher.UIThread.InvokeAsync(() =>
+                WeakReferenceMessenger.Default.Send(
+                    new JobCreatedMessage(new(jobId, abbrev, jobType)))));
+
+        _connection.On<string, string, int, string>("SendJobProgressUpdate", (jobId, stepName, pct, logLine) =>
+            Dispatcher.UIThread.InvokeAsync(() =>
+                WeakReferenceMessenger.Default.Send(
+                    new JobProgressUpdateMessage(new(jobId, stepName, pct, logLine)))));
+
+        _connection.On<string, bool, string>("SendJobCompleted", (jobId, success, summary) =>
+            Dispatcher.UIThread.InvokeAsync(() =>
+                WeakReferenceMessenger.Default.Send(
+                    new JobCompletedMessage(new(jobId, success, summary)))));
+
+        _connection.On<string, string>("SendInstanceStatusChanged", (customerId, status) =>
+            Dispatcher.UIThread.InvokeAsync(() =>
+                WeakReferenceMessenger.Default.Send(
+                    new InstanceStatusChangedMessage(new(customerId, status)))));
+
+        _connection.On<string, string>("SendAlertRaised", (severity, message) =>
+            Dispatcher.UIThread.InvokeAsync(() =>
+                WeakReferenceMessenger.Default.Send(
+                    new AlertRaisedMessage(new(severity, message)))));
+    }
+
+    public async ValueTask DisposeAsync()
+    {
+        await _connection.DisposeAsync();
+    }
+}

OTSSignsOrchestrator.Desktop/Services/SignalRMessages.cs

@@ -0,0 +1,35 @@
+using CommunityToolkit.Mvvm.Messaging.Messages;
+
+namespace OTSSignsOrchestrator.Desktop.Services;
+
+/// <summary>SignalR push messages republished via WeakReferenceMessenger for ViewModel consumption.</summary>
+
+public sealed class JobCreatedMessage : ValueChangedMessage<JobCreatedMessage.Payload>
+{
+    public JobCreatedMessage(Payload value) : base(value) { }
+    public record Payload(string JobId, string Abbrev, string JobType);
+}
+
+public sealed class JobProgressUpdateMessage : ValueChangedMessage<JobProgressUpdateMessage.Payload>
+{
+    public JobProgressUpdateMessage(Payload value) : base(value) { }
+    public record Payload(string JobId, string StepName, int Pct, string LogLine);
+}
+
+public sealed class JobCompletedMessage : ValueChangedMessage<JobCompletedMessage.Payload>
+{
+    public JobCompletedMessage(Payload value) : base(value) { }
+    public record Payload(string JobId, bool Success, string Summary);
+}
+
+public sealed class InstanceStatusChangedMessage : ValueChangedMessage<InstanceStatusChangedMessage.Payload>
+{
+    public InstanceStatusChangedMessage(Payload value) : base(value) { }
+    public record Payload(string CustomerId, string Status);
+}
+
+public sealed class AlertRaisedMessage : ValueChangedMessage<AlertRaisedMessage.Payload>
+{
+    public AlertRaisedMessage(Payload value) : base(value) { }
+    public record Payload(string Severity, string Message);
+}

OTSSignsOrchestrator.Desktop/Services/TokenStoreService.cs

@@ -0,0 +1,268 @@
+using System.Runtime.InteropServices;
+using System.Security.Cryptography;
+using System.Text;
+using Microsoft.Extensions.Logging;
+
+namespace OTSSignsOrchestrator.Desktop.Services;
+
+/// <summary>
+/// Stores and retrieves operator JWT and refresh tokens using the OS credential store.
+/// Windows: advapi32 Credential Manager; macOS: Security.framework Keychain;
+/// Linux: AES-encrypted file fallback in AppData.
+/// </summary>
+public sealed class TokenStoreService
+{
+    private const string ServiceName = "OTSSignsOrchestrator";
+    private const string JwtAccount = "operator-jwt";
+    private const string RefreshAccount = "operator-refresh";
+
+    private readonly ILogger<TokenStoreService> _logger;
+
+    public TokenStoreService(ILogger<TokenStoreService> logger)
+    {
+        _logger = logger;
+    }
+
+    public void StoreTokens(string jwt, string refreshToken)
+    {
+        WriteCredential(JwtAccount, jwt);
+        WriteCredential(RefreshAccount, refreshToken);
+        _logger.LogDebug("Tokens stored in OS credential store");
+    }
+
+    public string? GetJwt() => ReadCredential(JwtAccount);
+
+    public string? GetRefreshToken() => ReadCredential(RefreshAccount);
+
+    public void ClearTokens()
+    {
+        DeleteCredential(JwtAccount);
+        DeleteCredential(RefreshAccount);
+        _logger.LogDebug("Tokens cleared from OS credential store");
+    }
+
+    // ── Platform dispatch ────────────────────────────────────────────────────
+
+    private void WriteCredential(string account, string secret)
+    {
+        if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
+            WindowsCredentialManager.Write(ServiceName, account, secret);
+        else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
+            MacKeychain.Write(ServiceName, account, secret);
+        else
+            LinuxEncryptedFile.Write(ServiceName, account, secret);
+    }
+
+    private string? ReadCredential(string account)
+    {
+        if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
+            return WindowsCredentialManager.Read(ServiceName, account);
+        if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
+            return MacKeychain.Read(ServiceName, account);
+        return LinuxEncryptedFile.Read(ServiceName, account);
+    }
+
+    private void DeleteCredential(string account)
+    {
+        if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
+            WindowsCredentialManager.Delete(ServiceName, account);
+        else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
+            MacKeychain.Delete(ServiceName, account);
+        else
+            LinuxEncryptedFile.Delete(ServiceName, account);
+    }
+
+    // ═══════════════════════════════════════════════════════════════════════
+    // Windows — advapi32.dll Credential Manager
+    // ═══════════════════════════════════════════════════════════════════════
+    private static class WindowsCredentialManager
+    {
+        private const int CredTypeGeneric = 1;
+        private const int CredPersistLocalMachine = 2;
+
+        [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
+        private struct CREDENTIAL
+        {
+            public uint Flags;
+            public uint Type;
+            public string TargetName;
+            public string Comment;
+            public long LastWritten;
+            public uint CredentialBlobSize;
+            public IntPtr CredentialBlob;
+            public uint Persist;
+            public uint AttributeCount;
+            public IntPtr Attributes;
+            public string TargetAlias;
+            public string UserName;
+        }
+
+        [DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
+        private static extern bool CredWriteW(ref CREDENTIAL credential, uint flags);
+
+        [DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
+        private static extern bool CredReadW(string target, uint type, uint flags, out IntPtr credential);
+
+        [DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
+        private static extern bool CredDeleteW(string target, uint type, uint flags);
+
+        [DllImport("advapi32.dll")]
+        private static extern void CredFree(IntPtr buffer);
+
+        private static string TargetName(string service, string account) => $"{service}/{account}";
+
+        public static void Write(string service, string account, string secret)
+        {
+            var bytes = Encoding.UTF8.GetBytes(secret);
+            var handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
+            try
+            {
+                var cred = new CREDENTIAL
+                {
+                    Type = CredTypeGeneric,
+                    TargetName = TargetName(service, account),
+                    UserName = account,
+                    CredentialBlob = handle.AddrOfPinnedObject(),
+                    CredentialBlobSize = (uint)bytes.Length,
+                    Persist = CredPersistLocalMachine,
+                };
+                CredWriteW(ref cred, 0);
+            }
+            finally { handle.Free(); }
+        }
+
+        public static string? Read(string service, string account)
+        {
+            if (!CredReadW(TargetName(service, account), CredTypeGeneric, 0, out var credPtr))
+                return null;
+            try
+            {
+                var cred = Marshal.PtrToStructure<CREDENTIAL>(credPtr);
+                if (cred.CredentialBlobSize == 0 || cred.CredentialBlob == IntPtr.Zero) return null;
+                var bytes = new byte[cred.CredentialBlobSize];
+                Marshal.Copy(cred.CredentialBlob, bytes, 0, bytes.Length);
+                return Encoding.UTF8.GetString(bytes);
+            }
+            finally { CredFree(credPtr); }
+        }
+
+        public static void Delete(string service, string account)
+            => CredDeleteW(TargetName(service, account), CredTypeGeneric, 0);
+    }
+
+    // ═══════════════════════════════════════════════════════════════════════
+    // macOS — Security.framework Keychain via /usr/bin/security CLI
+    // ═══════════════════════════════════════════════════════════════════════
+    private static class MacKeychain
+    {
+        public static void Write(string service, string account, string secret)
+        {
+            // Delete first to avoid "duplicate" errors on update
+            Delete(service, account);
+            RunSecurity($"add-generic-password -s \"{service}\" -a \"{account}\" -w \"{EscapeShell(secret)}\" -U");
+        }
+
+        public static string? Read(string service, string account)
+        {
+            var (exitCode, stdout) = RunSecurity($"find-generic-password -s \"{service}\" -a \"{account}\" -w");
+            return exitCode == 0 ? stdout.Trim() : null;
+        }
+
+        public static void Delete(string service, string account)
+            => RunSecurity($"delete-generic-password -s \"{service}\" -a \"{account}\"");
+
+        private static (int exitCode, string stdout) RunSecurity(string args)
+        {
+            using var proc = new System.Diagnostics.Process();
+            proc.StartInfo = new System.Diagnostics.ProcessStartInfo
+            {
+                FileName = "/usr/bin/security",
+                Arguments = args,
+                RedirectStandardOutput = true,
+                RedirectStandardError = true,
+                UseShellExecute = false,
+                CreateNoWindow = true,
+            };
+            proc.Start();
+            var stdout = proc.StandardOutput.ReadToEnd();
+            proc.WaitForExit(5000);
+            return (proc.ExitCode, stdout);
+        }
+
+        private static string EscapeShell(string s) => s.Replace("\\", "\\\\").Replace("\"", "\\\"");
+    }
+
+    // ═══════════════════════════════════════════════════════════════════════
+    // Linux — AES-256-GCM encrypted file in ~/.local/share
+    // ═══════════════════════════════════════════════════════════════════════
+    private static class LinuxEncryptedFile
+    {
+        // Machine-specific key derived from machine-id + user name
+        private static byte[] DeriveKey()
+        {
+            var machineId = "linux-default";
+            try
+            {
+                if (File.Exists("/etc/machine-id"))
+                    machineId = File.ReadAllText("/etc/machine-id").Trim();
+            }
+            catch { /* fallback */ }
+
+            var material = $"{machineId}:{Environment.UserName}:{ServiceName}";
+            return SHA256.HashData(Encoding.UTF8.GetBytes(material));
+        }
+
+        private static string FilePath(string service, string account)
+        {
+            var dir = Path.Combine(
+                Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),
+                service, "credentials");
+            Directory.CreateDirectory(dir);
+            return Path.Combine(dir, $"{account}.enc");
+        }
+
+        public static void Write(string service, string account, string secret)
+        {
+            var key = DeriveKey();
+            var plaintext = Encoding.UTF8.GetBytes(secret);
+            var nonce = new byte[12];
+            RandomNumberGenerator.Fill(nonce);
+            var ciphertext = new byte[plaintext.Length];
+            var tag = new byte[16];
+
+            using var aes = new AesGcm(key, 16);
+            aes.Encrypt(nonce, plaintext, ciphertext, tag);
+
+            // File format: [12 nonce][16 tag][ciphertext]
+            var output = new byte[12 + 16 + ciphertext.Length];
+            nonce.CopyTo(output, 0);
+            tag.CopyTo(output, 12);
+            ciphertext.CopyTo(output, 28);
+            File.WriteAllBytes(FilePath(service, account), output);
+        }
+
+        public static string? Read(string service, string account)
+        {
+            var path = FilePath(service, account);
+            if (!File.Exists(path)) return null;
+
+            var data = File.ReadAllBytes(path);
+            if (data.Length < 28) return null;
+
+            var nonce = data[..12];
+            var tag = data[12..28];
+            var ciphertext = data[28..];
+            var plaintext = new byte[ciphertext.Length];
+
+            using var aes = new AesGcm(DeriveKey(), 16);
+            aes.Decrypt(nonce, ciphertext, tag, plaintext);
+            return Encoding.UTF8.GetString(plaintext);
+        }
+
+        public static void Delete(string service, string account)
+        {
+            var path = FilePath(service, account);
+            if (File.Exists(path)) File.Delete(path);
+        }
+    }
+}

OTSSignsOrchestrator.Desktop/ViewModels/InstancesViewModel.cs

@@ -2,8 +2,10 @@ using System.Collections.ObjectModel;
 using Avalonia.Threading;
 using CommunityToolkit.Mvvm.ComponentModel;
 using CommunityToolkit.Mvvm.Input;
+using CommunityToolkit.Mvvm.Messaging;
 using Microsoft.EntityFrameworkCore;
 using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Logging;
 using OTSSignsOrchestrator.Core.Data;
 using OTSSignsOrchestrator.Core.Models.DTOs;
 using OTSSignsOrchestrator.Core.Models.Entities;
@@ -16,10 +18,18 @@ namespace OTSSignsOrchestrator.Desktop.ViewModels;
 /// <summary>
 /// ViewModel for listing, viewing, and managing CMS instances.
 /// All data is fetched live from Docker Swarm hosts — nothing stored locally.
+/// Server operations (decommission, suspend, reactivate) go through the REST API.
+/// Real-time updates arrive via SignalR → WeakReferenceMessenger.
 /// </summary>
-public partial class InstancesViewModel : ObservableObject
+public partial class InstancesViewModel : ObservableObject,
+    IRecipient<AlertRaisedMessage>,
+    IRecipient<InstanceStatusChangedMessage>,
+    IRecipient<JobCreatedMessage>,
+    IRecipient<JobCompletedMessage>
 {
     private readonly IServiceProvider _services;
+    private readonly ILogger<InstancesViewModel> _logger;
+    private readonly IServerApiClient? _serverApi;
 
     [ObservableProperty] private ObservableCollection<LiveStackItem> _instances = new();
     [ObservableProperty] private LiveStackItem? _selectedInstance;
@@ -32,6 +42,10 @@ public partial class InstancesViewModel : ObservableObject
     [ObservableProperty] private ObservableCollection<SshHost> _availableHosts = new();
     [ObservableProperty] private SshHost? _selectedSshHost;
 
+    // ── P1 Authentik Banner ──────────────────────────────────────────────────
+    [ObservableProperty] private bool _isAuthentikP1BannerVisible;
+    [ObservableProperty] private string _authentikP1Message = string.Empty;
+
     // ── Container Logs ──────────────────────────────────────────────────────
     [ObservableProperty] private ObservableCollection<ServiceLogEntry> _logEntries = new();
     [ObservableProperty] private ObservableCollection<string> _logServiceFilter = new();
@@ -54,11 +68,26 @@ public partial class InstancesViewModel : ObservableObject
     /// </summary>
     public Func<string, string, Task<bool>>? ConfirmAsync { get; set; }
 
+    /// <summary>
+    /// Callback the View wires up to show a multi-step confirmation dialog for decommission.
+    /// Parameters: (abbreviation) → returns true if confirmed through all steps.
+    /// </summary>
+    public Func<string, Task<bool>>? ConfirmDecommissionAsync { get; set; }
+
     private string? _pendingSelectAbbrev;
 
     public InstancesViewModel(IServiceProvider services)
     {
         _services = services;
+        _logger = services.GetRequiredService<ILogger<InstancesViewModel>>();
+        _serverApi = services.GetService<IServerApiClient>();
+
+        // Register for SignalR messages via WeakReferenceMessenger
+        WeakReferenceMessenger.Default.Register<AlertRaisedMessage>(this);
+        WeakReferenceMessenger.Default.Register<InstanceStatusChangedMessage>(this);
+        WeakReferenceMessenger.Default.Register<JobCreatedMessage>(this);
+        WeakReferenceMessenger.Default.Register<JobCompletedMessage>(this);
+
         _ = RefreshAllAsync();
     }
 
@@ -69,6 +98,49 @@ public partial class InstancesViewModel : ObservableObject
     public void SetPendingSelection(string abbrev)
         => _pendingSelectAbbrev = abbrev;
 
+    // ── SignalR Message Handlers ─────────────────────────────────────────────
+    // These are called on the UI thread (SignalR handlers dispatch via Dispatcher.UIThread).
+
+    void IRecipient<AlertRaisedMessage>.Receive(AlertRaisedMessage message)
+    {
+        var (severity, msg) = message.Value;
+        if (string.Equals(severity, "Critical", StringComparison.OrdinalIgnoreCase) &&
+            msg.Contains("Authentik", StringComparison.OrdinalIgnoreCase))
+        {
+            AuthentikP1Message = msg;
+            IsAuthentikP1BannerVisible = true;
+        }
+    }
+
+    void IRecipient<InstanceStatusChangedMessage>.Receive(InstanceStatusChangedMessage message)
+    {
+        var (customerId, status) = message.Value;
+        _logger.LogInformation("Instance status changed: customer={CustomerId} status={Status}", customerId, status);
+        StatusMessage = $"Instance {customerId} status → {status}";
+        // Refresh the list to pick up the new status
+        _ = RefreshAllAsync();
+    }
+
+    void IRecipient<JobCreatedMessage>.Receive(JobCreatedMessage message)
+    {
+        var (jobId, abbrev, jobType) = message.Value;
+        _logger.LogInformation("Job created: {JobId} type={JobType} abbrev={Abbrev}", jobId, jobType, abbrev);
+        StatusMessage = $"Job '{jobType}' created for {abbrev} (id: {jobId[..8]}…)";
+    }
+
+    void IRecipient<JobCompletedMessage>.Receive(JobCompletedMessage message)
+    {
+        var (jobId, success, summary) = message.Value;
+        _logger.LogInformation("Job completed: {JobId} success={Success} summary={Summary}", jobId, success, summary);
+        StatusMessage = success
+            ? $"Job {jobId[..8]}… completed: {summary}"
+            : $"Job {jobId[..8]}… failed: {summary}";
+        // Refresh the instance list to reflect changes from the completed job
+        _ = RefreshAllAsync();
+    }
+
+    // ── Load / Refresh ──────────────────────────────────────────────────────
+
     /// <summary>
     /// Enumerates all SSH hosts, then calls docker stack ls on each to build the
     /// live instance list. Only stacks matching *-cms-stack are shown.
@@ -112,6 +184,25 @@ public partial class InstancesViewModel : ObservableObject
                 catch (Exception ex) { errors.Add($"{host.Label}: {ex.Message}"); }
             }
 
+            // Enrich with server-side customer IDs if the server API is available
+            if (_serverApi is not null)
+            {
+                try
+                {
+                    var fleet = await _serverApi.GetFleetAsync();
+                    var lookup = fleet.ToDictionary(f => f.Abbreviation, f => f.CustomerId, StringComparer.OrdinalIgnoreCase);
+                    foreach (var item in all)
+                    {
+                        if (lookup.TryGetValue(item.CustomerAbbrev, out var customerId))
+                            item.CustomerId = customerId;
+                    }
+                }
+                catch (Exception ex)
+                {
+                    _logger.LogWarning(ex, "Could not enrich instances with server fleet data");
+                }
+            }
+
             if (!string.IsNullOrWhiteSpace(FilterText))
                 all = all.Where(i =>
                     i.StackName.Contains(FilterText, StringComparison.OrdinalIgnoreCase) ||
@@ -371,6 +462,144 @@ public partial class InstancesViewModel : ObservableObject
         finally { IsBusy = false; }
     }
 
+    // ── P1 Banner Commands ────────────────────────────────────────────────
+
+    [RelayCommand]
+    private void DismissP1Banner()
+    {
+        IsAuthentikP1BannerVisible = false;
+        AuthentikP1Message = string.Empty;
+    }
+
+    /// <summary>
+    /// Called from a SignalR <c>AlertRaised</c> handler (runs on a background thread).
+    /// CRITICAL: wraps all property updates with <see cref="Dispatcher.UIThread"/> to
+    /// avoid silent cross-thread exceptions in Avalonia.
+    /// </summary>
+    public void HandleAlertRaised(string severity, string message)
+    {
+        if (string.Equals(severity, "Critical", StringComparison.OrdinalIgnoreCase) &&
+            message.Contains("Authentik", StringComparison.OrdinalIgnoreCase))
+        {
+            Dispatcher.UIThread.InvokeAsync(() =>
+            {
+                AuthentikP1Message = message;
+                IsAuthentikP1BannerVisible = true;
+            });
+        }
+    }
+
+    // ── Server-side Job Commands (decommission, suspend, reactivate) ────────
+    // Desktop has NO direct infrastructure access — all operations go through the server REST API.
+
+    [RelayCommand]
+    private async Task DecommissionAsync(LiveStackItem? instance)
+    {
+        instance ??= SelectedInstance;
+        if (instance is null) return;
+
+        if (instance.CustomerId is null)
+        {
+            StatusMessage = "Cannot decommission: no server-side customer ID available for this instance.";
+            return;
+        }
+
+        // Multi-step confirmation: user must type the abbreviation to confirm
+        if (ConfirmDecommissionAsync is not null)
+        {
+            var confirmed = await ConfirmDecommissionAsync(instance.CustomerAbbrev);
+            if (!confirmed) return;
+        }
+        else if (ConfirmAsync is not null)
+        {
+            var confirmed = await ConfirmAsync(
+                "Decommission Instance",
+                $"Are you sure you want to decommission '{instance.CustomerAbbrev}'?\n\n" +
+                "This will:\n" +
+                "  • Remove all Docker services and stack\n" +
+                "  • Delete Docker secrets\n" +
+                "  • Remove NFS volumes and data\n" +
+                "  • Revoke Authentik provider\n" +
+                "  • Mark the customer as decommissioned\n\n" +
+                "This action is IRREVERSIBLE.");
+            if (!confirmed) return;
+        }
+
+        await CreateServerJobAsync(instance, "decommission");
+    }
+
+    [RelayCommand]
+    private async Task SuspendInstanceAsync(LiveStackItem? instance)
+    {
+        instance ??= SelectedInstance;
+        if (instance is null) return;
+
+        if (instance.CustomerId is null)
+        {
+            StatusMessage = "Cannot suspend: no server-side customer ID available for this instance.";
+            return;
+        }
+
+        if (ConfirmAsync is not null)
+        {
+            var confirmed = await ConfirmAsync(
+                "Suspend Instance",
+                $"Are you sure you want to suspend '{instance.CustomerAbbrev}'?\n\n" +
+                "The instance will be scaled to zero replicas. Data will be preserved.");
+            if (!confirmed) return;
+        }
+
+        await CreateServerJobAsync(instance, "suspend");
+    }
+
+    [RelayCommand]
+    private async Task ReactivateInstanceAsync(LiveStackItem? instance)
+    {
+        instance ??= SelectedInstance;
+        if (instance is null) return;
+
+        if (instance.CustomerId is null)
+        {
+            StatusMessage = "Cannot reactivate: no server-side customer ID available for this instance.";
+            return;
+        }
+
+        await CreateServerJobAsync(instance, "reactivate");
+    }
+
+    private async Task CreateServerJobAsync(LiveStackItem instance, string jobType)
+    {
+        if (_serverApi is null)
+        {
+            StatusMessage = "Server API client is not configured.";
+            return;
+        }
+
+        IsBusy = true;
+        StatusMessage = $"Requesting '{jobType}' for {instance.CustomerAbbrev}...";
+        try
+        {
+            var response = await _serverApi.CreateJobAsync(
+                new CreateJobRequest(instance.CustomerId!.Value, jobType, null));
+            StatusMessage = $"Job '{jobType}' created (id: {response.Id.ToString()[..8]}…). Status: {response.Status}";
+            _logger.LogInformation("Server job created: {JobId} type={JobType} customer={CustomerId}",
+                response.Id, jobType, instance.CustomerId);
+        }
+        catch (Refit.ApiException ex)
+        {
+            StatusMessage = $"Server error creating '{jobType}' job: {ex.StatusCode} — {ex.Content}";
+            _logger.LogError(ex, "Failed to create {JobType} job for customer {CustomerId}", jobType, instance.CustomerId);
+        }
+        catch (Exception ex)
+        {
+            StatusMessage = $"Error creating '{jobType}' job: {ex.Message}";
+            _logger.LogError(ex, "Failed to create {JobType} job for customer {CustomerId}", jobType, instance.CustomerId);
+        }
+        finally { IsBusy = false; }
+    }
+
+    // ── Details ─────────────────────────────────────────────────────────────
+
     [RelayCommand]
     private async Task OpenDetailsAsync()
     {

OTSSignsOrchestrator.Server.Tests/ByoiCertExpiryThresholdTests.cs

@@ -0,0 +1,96 @@
+using OTSSignsOrchestrator.Server.Jobs;
+
+namespace OTSSignsOrchestrator.Server.Tests;
+
+public class ByoiCertExpiryThresholdTests
+{
+    // ── ShouldAlert ─────────────────────────────────────────────────────────
+
+    [Theory]
+    [InlineData(61, false)]   // 61 days: above all thresholds → no alert
+    [InlineData(60, true)]    // 60 days: at first threshold → alert
+    [InlineData(59, true)]    // 59 days: below 60 → alert
+    [InlineData(31, true)]    // 31 days: between 60 and 30 → alert
+    [InlineData(30, true)]    // 30 days: at second threshold → alert
+    [InlineData(8, true)]     // 8 days: between 30 and 7 → alert
+    [InlineData(7, true)]     // 7 days: at critical threshold → alert
+    [InlineData(1, true)]     // 1 day: below critical → alert
+    [InlineData(0, true)]     // 0 days: expiry day → alert
+    [InlineData(-1, true)]    // -1 day: already expired → alert
+    public void ShouldAlert_ReturnsCorrectValue(double daysRemaining, bool expected)
+    {
+        Assert.Equal(expected, ByoiCertExpiryJob.ShouldAlert(daysRemaining));
+    }
+
+    [Fact]
+    public void ShouldAlert_LargeValue_NoAlert()
+    {
+        Assert.False(ByoiCertExpiryJob.ShouldAlert(365));
+    }
+
+    // ── GetSeverity ─────────────────────────────────────────────────────────
+
+    [Theory]
+    [InlineData(60, "Warning")]
+    [InlineData(30, "Warning")]
+    [InlineData(8, "Warning")]
+    [InlineData(7.01, "Warning")]
+    [InlineData(7, "Critical")]    // Exactly at critical boundary
+    [InlineData(6, "Critical")]
+    [InlineData(1, "Critical")]
+    [InlineData(0, "Critical")]
+    [InlineData(-1, "Critical")]   // Already expired
+    public void GetSeverity_ReturnsCorrectLevel(double daysRemaining, string expected)
+    {
+        Assert.Equal(expected, ByoiCertExpiryJob.GetSeverity(daysRemaining));
+    }
+
+    // ── Threshold constants ─────────────────────────────────────────────────
+
+    [Fact]
+    public void AlertThresholds_AreDescending()
+    {
+        var thresholds = ByoiCertExpiryJob.AlertThresholdDays;
+        for (int i = 1; i < thresholds.Length; i++)
+        {
+            Assert.True(thresholds[i - 1] > thresholds[i],
+                $"Thresholds must be in descending order: {thresholds[i - 1]} should be > {thresholds[i]}");
+        }
+    }
+
+    [Fact]
+    public void CriticalThreshold_IsSmallestAlertThreshold()
+    {
+        Assert.Equal(
+            ByoiCertExpiryJob.CriticalThresholdDays,
+            ByoiCertExpiryJob.AlertThresholdDays[^1]);
+    }
+
+    // ── Boundary precision ──────────────────────────────────────────────────
+
+    [Fact]
+    public void ShouldAlert_JustAboveThreshold_NoAlert()
+    {
+        // 60.001 days — just above 60-day threshold
+        Assert.False(ByoiCertExpiryJob.ShouldAlert(60.001));
+    }
+
+    [Fact]
+    public void ShouldAlert_JustBelowThreshold_Alerts()
+    {
+        // 59.999 days — just below 60-day threshold
+        Assert.True(ByoiCertExpiryJob.ShouldAlert(59.999));
+    }
+
+    [Fact]
+    public void GetSeverity_JustAboveCritical_IsWarning()
+    {
+        Assert.Equal("Warning", ByoiCertExpiryJob.GetSeverity(7.001));
+    }
+
+    [Fact]
+    public void GetSeverity_ExactlyCritical_IsCritical()
+    {
+        Assert.Equal("Critical", ByoiCertExpiryJob.GetSeverity(7.0));
+    }
+}

OTSSignsOrchestrator.Server.Tests/OTSSignsOrchestrator.Server.Tests.csproj

@@ -0,0 +1,25 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>net9.0</TargetFramework>
+    <ImplicitUsings>enable</ImplicitUsings>
+    <Nullable>enable</Nullable>
+    <IsPackable>false</IsPackable>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="coverlet.collector" Version="6.0.2" />
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
+    <PackageReference Include="xunit" Version="2.9.2" />
+    <PackageReference Include="xunit.runner.visualstudio" Version="2.8.2" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <Using Include="Xunit" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <ProjectReference Include="..\OTSSignsOrchestrator.Server\OTSSignsOrchestrator.Server.csproj" />
+  </ItemGroup>
+
+</Project>

OTSSignsOrchestrator.Server/Api/CustomerPortalApi.cs

@@ -0,0 +1,248 @@
+using System.Security.Cryptography.X509Certificates;
+using System.Text.Json;
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+using OTSSignsOrchestrator.Server.Workers;
+
+namespace OTSSignsOrchestrator.Server.Api;
+
+public static class CustomerPortalApi
+{
+    private const int MinCertDaysRemaining = 30;
+
+    public static void MapCustomerPortalEndpoints(this WebApplication app)
+    {
+        var portal = app.MapGroup("/api/portal/byoi")
+            .RequireAuthorization("CustomerPortal");
+
+        portal.MapPost("/configure", HandleConfigureByoi);
+        portal.MapGet("/sp-metadata", HandleGetSpMetadata);
+        portal.MapPost("/rotate-cert", HandleRotateCert);
+    }
+
+    // ── POST /api/portal/byoi/configure ─────────────────────────────────────
+    private static async Task<IResult> HandleConfigureByoi(
+        ConfigureByoiRequest req,
+        OrchestratorDbContext db,
+        IHubContext<FleetHub, IFleetClient> hub,
+        HttpContext httpContext,
+        ILogger<ConfigureByoiRequest> logger)
+    {
+        // Resolve customer from the authenticated JWT
+        var customer = await ResolveCustomerAsync(httpContext, db);
+        if (customer is null)
+            return Results.Forbid();
+
+        if (customer.Plan != CustomerPlan.Pro)
+            return Results.Problem("BYOI SAML is only available for Pro tier customers.", statusCode: 403);
+
+        // Validate cert PEM
+        var certValidation = ValidateCertPem(req.CertPem);
+        if (certValidation is not null)
+            return Results.ValidationProblem(
+                new Dictionary<string, string[]> { ["certPem"] = [certValidation] });
+
+        // Validate required fields
+        var errors = new Dictionary<string, string[]>();
+        if (string.IsNullOrWhiteSpace(req.SsoUrl))
+            errors["ssoUrl"] = ["ssoUrl is required."];
+        if (string.IsNullOrWhiteSpace(req.IdpEntityId))
+            errors["idpEntityId"] = ["idpEntityId is required."];
+        if (errors.Count > 0)
+            return Results.ValidationProblem(errors);
+
+        // Create a provision-byoi Job
+        var parametersJson = JsonSerializer.Serialize(new ByoiParameters
+        {
+            CustomerCertPem = req.CertPem!,
+            CustomerSsoUrl = req.SsoUrl!,
+            CustomerIdpEntityId = req.IdpEntityId!,
+            CustomerSloUrl = req.SloUrl,
+        });
+
+        var job = new Job
+        {
+            Id = Guid.NewGuid(),
+            CustomerId = customer.Id,
+            JobType = "provision-byoi",
+            Status = JobStatus.Queued,
+            TriggeredBy = $"customer-portal:{customer.AdminEmail}",
+            Parameters = parametersJson,
+            CreatedAt = DateTime.UtcNow,
+        };
+
+        db.Jobs.Add(job);
+        await db.SaveChangesAsync();
+
+        logger.LogInformation("BYOI configure job {JobId} created for customer {CustomerId}",
+            job.Id, customer.Id);
+
+        await hub.Clients.All.SendJobCreated(
+            job.Id.ToString(), customer.Abbreviation, job.JobType);
+
+        return Results.Created($"/api/jobs/{job.Id}", new { jobId = job.Id });
+    }
+
+    // ── GET /api/portal/byoi/sp-metadata ────────────────────────────────────
+    private static async Task<IResult> HandleGetSpMetadata(
+        OrchestratorDbContext db,
+        IAuthentikClient authentikClient,
+        HttpContext httpContext,
+        ILogger<ConfigureByoiRequest> logger)
+    {
+        var customer = await ResolveCustomerAsync(httpContext, db);
+        if (customer is null)
+            return Results.Forbid();
+
+        var instance = customer.Instances.FirstOrDefault();
+        if (instance is null)
+            return Results.NotFound("No instance found for this customer.");
+
+        var byoiConfig = await db.ByoiConfigs
+            .AsNoTracking()
+            .FirstOrDefaultAsync(b => b.InstanceId == instance.Id && b.Enabled);
+
+        if (byoiConfig is null)
+            return Results.NotFound("No BYOI configuration found for this instance.");
+
+        var metadataResponse = await authentikClient.GetSamlSourceMetadataAsync(byoiConfig.Slug);
+        if (!metadataResponse.IsSuccessStatusCode || metadataResponse.Content is null)
+        {
+            logger.LogError("Failed to fetch SP metadata for slug {Slug}: {Error}",
+                byoiConfig.Slug, metadataResponse.Error?.Content ?? metadataResponse.ReasonPhrase);
+            return Results.Problem("Failed to retrieve SP metadata from Authentik.", statusCode: 502);
+        }
+
+        return Results.Content(metadataResponse.Content, "application/xml");
+    }
+
+    // ── POST /api/portal/byoi/rotate-cert ───────────────────────────────────
+    private static async Task<IResult> HandleRotateCert(
+        RotateCertRequest req,
+        OrchestratorDbContext db,
+        IHubContext<FleetHub, IFleetClient> hub,
+        HttpContext httpContext,
+        ILogger<RotateCertRequest> logger)
+    {
+        var customer = await ResolveCustomerAsync(httpContext, db);
+        if (customer is null)
+            return Results.Forbid();
+
+        if (customer.Plan != CustomerPlan.Pro)
+            return Results.Problem("BYOI SAML is only available for Pro tier customers.", statusCode: 403);
+
+        // Validate cert PEM
+        var certValidation = ValidateCertPem(req.CertPem);
+        if (certValidation is not null)
+            return Results.ValidationProblem(
+                new Dictionary<string, string[]> { ["certPem"] = [certValidation] });
+
+        var instance = customer.Instances.FirstOrDefault();
+        if (instance is null)
+            return Results.NotFound("No instance found for this customer.");
+
+        var existingConfig = await db.ByoiConfigs
+            .FirstOrDefaultAsync(b => b.InstanceId == instance.Id && b.Enabled);
+
+        if (existingConfig is null)
+            return Results.NotFound("No active BYOI configuration found to rotate.");
+
+        // Create a re-provisioning job with the new cert
+        var parametersJson = JsonSerializer.Serialize(new ByoiParameters
+        {
+            CustomerCertPem = req.CertPem!,
+            CustomerSsoUrl = existingConfig.SsoUrl,
+            CustomerIdpEntityId = existingConfig.EntityId,
+            CustomerSloUrl = null,
+        });
+
+        var job = new Job
+        {
+            Id = Guid.NewGuid(),
+            CustomerId = customer.Id,
+            JobType = "provision-byoi",
+            Status = JobStatus.Queued,
+            TriggeredBy = $"customer-portal:cert-rotate:{customer.AdminEmail}",
+            Parameters = parametersJson,
+            CreatedAt = DateTime.UtcNow,
+        };
+
+        db.Jobs.Add(job);
+        await db.SaveChangesAsync();
+
+        logger.LogInformation("BYOI cert rotate job {JobId} created for customer {CustomerId}",
+            job.Id, customer.Id);
+
+        await hub.Clients.All.SendJobCreated(
+            job.Id.ToString(), customer.Abbreviation, job.JobType);
+
+        return Results.Ok(new { jobId = job.Id });
+    }
+
+    // ── Helpers ─────────────────────────────────────────────────────────────
+
+    /// <summary>
+    /// Validates a PEM certificate string. Returns an error message on failure, or null if valid.
+    /// Rejects self-signed, expired, and certs expiring in &lt; 30 days.
+    /// </summary>
+    private static string? ValidateCertPem(string? certPem)
+    {
+        if (string.IsNullOrWhiteSpace(certPem))
+            return "certPem is required.";
+
+        X509Certificate2 cert;
+        try
+        {
+            var base64 = certPem
+                .Replace("-----BEGIN CERTIFICATE-----", "")
+                .Replace("-----END CERTIFICATE-----", "")
+                .Replace("\r", "")
+                .Replace("\n", "")
+                .Trim();
+
+            cert = X509CertificateLoader.LoadCertificate(Convert.FromBase64String(base64));
+        }
+        catch (Exception)
+        {
+            return "Invalid certificate PEM format.";
+        }
+
+        using (cert)
+        {
+            if (cert.NotAfter.ToUniversalTime() < DateTime.UtcNow)
+                return "Certificate has already expired.";
+
+            if ((cert.NotAfter.ToUniversalTime() - DateTime.UtcNow).TotalDays < MinCertDaysRemaining)
+                return $"Certificate expires in less than {MinCertDaysRemaining} days. Provide a certificate with a longer validity period.";
+
+            // Reject self-signed: issuer == subject
+            if (string.Equals(cert.Issuer, cert.Subject, StringComparison.OrdinalIgnoreCase))
+                return "Self-signed certificates are not accepted. Provide a CA-signed certificate.";
+        }
+
+        return null;
+    }
+
+    /// <summary>
+    /// Resolves the current customer from the authenticated JWT claims.
+    /// Expects a "customer_id" claim in the token.
+    /// </summary>
+    private static async Task<Customer?> ResolveCustomerAsync(HttpContext httpContext, OrchestratorDbContext db)
+    {
+        var customerIdClaim = httpContext.User.FindFirst("customer_id")?.Value;
+        if (customerIdClaim is null || !Guid.TryParse(customerIdClaim, out var customerId))
+            return null;
+
+        return await db.Customers
+            .Include(c => c.Instances)
+            .FirstOrDefaultAsync(c => c.Id == customerId);
+    }
+}
+
+// ── Request DTOs ────────────────────────────────────────────────────────────
+public record ConfigureByoiRequest(string? CertPem, string? SsoUrl, string? IdpEntityId, string? SloUrl);
+public record RotateCertRequest(string? CertPem);

OTSSignsOrchestrator.Server/Api/FleetApi.cs

@@ -0,0 +1,245 @@
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+using OTSSignsOrchestrator.Server.Reports;
+
+namespace OTSSignsOrchestrator.Server.Api;
+
+public static class FleetApi
+{
+    public static void MapFleetEndpoints(this WebApplication app)
+    {
+        var fleet = app.MapGroup("/api/fleet").RequireAuthorization();
+        fleet.MapGet("/", GetFleetSummary);
+        fleet.MapGet("/{id:guid}", GetFleetDetail);
+
+        var jobs = app.MapGroup("/api/jobs").RequireAuthorization();
+        jobs.MapPost("/", CreateJob);
+        jobs.MapGet("/{id:guid}", GetJob);
+
+        app.MapGet("/api/health", () => Results.Ok(new { status = "healthy" }));
+
+        // ── Report endpoints (admin only) ────────────────────────────────────
+        var reports = app.MapGroup("/api/reports").RequireAuthorization()
+            .RequireAuthorization(policy => policy.RequireRole("admin"));
+
+        reports.MapGet("/billing", GetBillingCsv);
+        reports.MapGet("/version-drift", GetVersionDriftCsv);
+        reports.MapGet("/fleet-health", GetFleetHealthPdf);
+        reports.MapGet("/customer/{id:guid}/usage", GetCustomerUsagePdf);
+
+        fleet.MapPost("/bulk/export-fleet-report", ExportFleetReport)
+            .RequireAuthorization(policy => policy.RequireRole("admin"));
+    }
+
+    // ── GET /api/fleet ──────────────────────────────────────────────────────
+    private static async Task<IResult> GetFleetSummary(OrchestratorDbContext db)
+    {
+        var customers = await db.Customers
+            .AsNoTracking()
+            .Include(c => c.Instances)
+            .Include(c => c.Jobs)
+            .ToListAsync();
+
+        // Get latest health event per instance in one query
+        var latestHealth = await db.HealthEvents
+            .AsNoTracking()
+            .GroupBy(h => h.InstanceId)
+            .Select(g => g.OrderByDescending(h => h.OccurredAt).First())
+            .ToDictionaryAsync(h => h.InstanceId);
+
+        var result = customers.Select(c =>
+        {
+            var primaryInstance = c.Instances.FirstOrDefault();
+            HealthEvent? health = null;
+            if (primaryInstance is not null)
+                latestHealth.TryGetValue(primaryInstance.Id, out health);
+
+            return new FleetSummaryDto
+            {
+                CustomerId = c.Id,
+                Abbreviation = c.Abbreviation,
+                CompanyName = c.CompanyName,
+                Plan = c.Plan.ToString(),
+                ScreenCount = c.ScreenCount,
+                HealthStatus = health?.Status.ToString() ?? primaryInstance?.HealthStatus.ToString() ?? "Unknown",
+                LastHealthCheck = health?.OccurredAt ?? primaryInstance?.LastHealthCheck,
+                HasRunningJob = c.Jobs.Any(j => j.Status == JobStatus.Running || j.Status == JobStatus.Queued),
+            };
+        }).ToArray();
+
+        return Results.Ok(result);
+    }
+
+    // ── GET /api/fleet/{id} ─────────────────────────────────────────────────
+    private static async Task<IResult> GetFleetDetail(Guid id, OrchestratorDbContext db)
+    {
+        var customer = await db.Customers
+            .AsNoTracking()
+            .Include(c => c.Instances)
+            .Include(c => c.Jobs.Where(j => j.Status == JobStatus.Running || j.Status == JobStatus.Queued))
+            .FirstOrDefaultAsync(c => c.Id == id);
+
+        if (customer is null)
+            return Results.NotFound();
+
+        return Results.Ok(new
+        {
+            customer.Id,
+            customer.Abbreviation,
+            customer.CompanyName,
+            customer.AdminEmail,
+            Plan = customer.Plan.ToString(),
+            customer.ScreenCount,
+            Status = customer.Status.ToString(),
+            customer.CreatedAt,
+            Instances = customer.Instances.Select(i => new
+            {
+                i.Id,
+                i.XiboUrl,
+                i.DockerStackName,
+                HealthStatus = i.HealthStatus.ToString(),
+                i.LastHealthCheck,
+            }),
+            ActiveJobs = customer.Jobs.Select(j => new
+            {
+                j.Id,
+                j.JobType,
+                Status = j.Status.ToString(),
+                j.CreatedAt,
+                j.StartedAt,
+            }),
+        });
+    }
+
+    // ── POST /api/jobs ──────────────────────────────────────────────────────
+    private static async Task<IResult> CreateJob(
+        CreateJobRequest req,
+        OrchestratorDbContext db,
+        IHubContext<FleetHub, IFleetClient> hub,
+        ILogger<CreateJobRequest> logger)
+    {
+        var customer = await db.Customers.FindAsync(req.CustomerId);
+        if (customer is null)
+            return Results.NotFound("Customer not found.");
+
+        var job = new Job
+        {
+            Id = Guid.NewGuid(),
+            CustomerId = req.CustomerId,
+            JobType = req.JobType,
+            Status = JobStatus.Queued,
+            TriggeredBy = "operator",
+            Parameters = req.Parameters,
+            CreatedAt = DateTime.UtcNow,
+        };
+
+        db.Jobs.Add(job);
+        await db.SaveChangesAsync();
+
+        logger.LogInformation("Job created: {JobId} type={JobType} customer={CustomerId}",
+            job.Id, job.JobType, job.CustomerId);
+
+        await hub.Clients.All.SendJobCreated(job.Id.ToString(), customer.Abbreviation, job.JobType);
+
+        return Results.Created($"/api/jobs/{job.Id}", new { job.Id, job.JobType, Status = job.Status.ToString() });
+    }
+
+    // ── GET /api/jobs/{id} ──────────────────────────────────────────────────
+    private static async Task<IResult> GetJob(Guid id, OrchestratorDbContext db)
+    {
+        var job = await db.Jobs
+            .AsNoTracking()
+            .Include(j => j.Steps.OrderBy(s => s.StartedAt))
+            .FirstOrDefaultAsync(j => j.Id == id);
+
+        if (job is null)
+            return Results.NotFound();
+
+        return Results.Ok(new
+        {
+            job.Id,
+            job.CustomerId,
+            job.JobType,
+            Status = job.Status.ToString(),
+            job.TriggeredBy,
+            job.Parameters,
+            job.CreatedAt,
+            job.StartedAt,
+            job.CompletedAt,
+            job.ErrorMessage,
+            Steps = job.Steps.Select(s => new
+            {
+                s.Id,
+                s.StepName,
+                Status = s.Status.ToString(),
+                s.LogOutput,
+                s.StartedAt,
+                s.CompletedAt,
+            }),
+        });
+    }
+
+    // ── GET /api/reports/billing?from=&to= ──────────────────────────────────
+    private static async Task<IResult> GetBillingCsv(
+        DateOnly from, DateOnly to, BillingReportService billing)
+    {
+        var csv = await billing.GenerateBillingCsvAsync(from, to);
+        return Results.File(csv, "text/csv", $"billing-{from:yyyy-MM-dd}-{to:yyyy-MM-dd}.csv");
+    }
+
+    // ── GET /api/reports/version-drift ──────────────────────────────────────
+    private static async Task<IResult> GetVersionDriftCsv(BillingReportService billing)
+    {
+        var csv = await billing.GenerateVersionDriftCsvAsync();
+        return Results.File(csv, "text/csv", $"version-drift-{DateTime.UtcNow:yyyy-MM-dd}.csv");
+    }
+
+    // ── GET /api/reports/fleet-health?from=&to= ─────────────────────────────
+    private static async Task<IResult> GetFleetHealthPdf(
+        DateOnly from, DateOnly to, FleetHealthPdfService pdfService)
+    {
+        var pdf = await pdfService.GenerateFleetHealthPdfAsync(from, to);
+        return Results.File(pdf, "application/pdf", $"fleet-health-{from:yyyy-MM-dd}-{to:yyyy-MM-dd}.pdf");
+    }
+
+    // ── GET /api/reports/customer/{id}/usage?from=&to= ──────────────────────
+    private static async Task<IResult> GetCustomerUsagePdf(
+        Guid id, DateOnly from, DateOnly to, FleetHealthPdfService pdfService)
+    {
+        try
+        {
+            var pdf = await pdfService.GenerateCustomerUsagePdfAsync(id, from, to);
+            return Results.File(pdf, "application/pdf", $"customer-usage-{id}-{from:yyyy-MM-dd}-{to:yyyy-MM-dd}.pdf");
+        }
+        catch (InvalidOperationException ex)
+        {
+            return Results.NotFound(ex.Message);
+        }
+    }
+
+    // ── POST /api/fleet/bulk/export-fleet-report ────────────────────────────
+    private static async Task<IResult> ExportFleetReport(FleetHealthPdfService pdfService)
+    {
+        var to = DateOnly.FromDateTime(DateTime.UtcNow);
+        var from = to.AddDays(-7);
+        var pdf = await pdfService.GenerateFleetHealthPdfAsync(from, to);
+        return Results.File(pdf, "application/pdf", $"fleet-health-{from:yyyy-MM-dd}-{to:yyyy-MM-dd}.pdf");
+    }
+}
+
+public record FleetSummaryDto
+{
+    public Guid CustomerId { get; init; }
+    public string Abbreviation { get; init; } = string.Empty;
+    public string CompanyName { get; init; } = string.Empty;
+    public string Plan { get; init; } = string.Empty;
+    public int ScreenCount { get; init; }
+    public string HealthStatus { get; init; } = "Unknown";
+    public DateTime? LastHealthCheck { get; init; }
+    public bool HasRunningJob { get; init; }
+}
+
+public record CreateJobRequest(Guid CustomerId, string JobType, string? Parameters);

OTSSignsOrchestrator.Server/Api/SignupApi.cs

@@ -0,0 +1,173 @@
+using System.ComponentModel.DataAnnotations;
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+using Stripe.Checkout;
+
+namespace OTSSignsOrchestrator.Server.Api;
+
+public static class SignupApi
+{
+    public static void MapSignupEndpoints(this WebApplication app)
+    {
+        app.MapPost("/api/signup/initiate", HandleInitiate)
+            .RequireRateLimiting("signup");
+
+        app.MapGet("/api/signup/status/{token:guid}", HandleStatus);
+    }
+
+    private static async Task<IResult> HandleInitiate(
+        SignupRequest req,
+        OrchestratorDbContext db,
+        IConfiguration config,
+        ILogger<SignupRequest> logger)
+    {
+        // ── Validation ──────────────────────────────────────────────────────
+        var errors = new List<string>();
+        if (string.IsNullOrWhiteSpace(req.CompanyName))
+            errors.Add("companyName is required.");
+        if (string.IsNullOrWhiteSpace(req.AdminEmail) || !new EmailAddressAttribute().IsValid(req.AdminEmail))
+            errors.Add("A valid adminEmail is required.");
+        if (string.IsNullOrWhiteSpace(req.Plan) ||
+            !req.Plan.Equals("Essentials", StringComparison.OrdinalIgnoreCase) &&
+            !req.Plan.Equals("Pro", StringComparison.OrdinalIgnoreCase))
+            errors.Add("plan must be 'Essentials' or 'Pro'.");
+        if (req.ScreenCount < 1)
+            errors.Add("screenCount must be at least 1.");
+        if (req.Plan?.Equals("Essentials", StringComparison.OrdinalIgnoreCase) == true && req.ScreenCount > 50)
+            errors.Add("Essentials plan supports a maximum of 50 screens.");
+        if (string.IsNullOrWhiteSpace(req.BillingFrequency) ||
+            !req.BillingFrequency.Equals("monthly", StringComparison.OrdinalIgnoreCase) &&
+            !req.BillingFrequency.Equals("annual", StringComparison.OrdinalIgnoreCase))
+            errors.Add("billingFrequency must be 'monthly' or 'annual'.");
+
+        if (errors.Count > 0)
+            return Results.ValidationProblem(
+                errors.ToDictionary(e => e, _ => new[] { "Validation failed." }));
+
+        // ── Create pending customer ─────────────────────────────────────────
+        var plan = Enum.Parse<CustomerPlan>(req.Plan!, true);
+        var customer = new Customer
+        {
+            Id = Guid.NewGuid(),
+            CompanyName = req.CompanyName!.Trim(),
+            AdminEmail = req.AdminEmail!.Trim().ToLowerInvariant(),
+            AdminFirstName = req.AdminFirstName?.Trim() ?? string.Empty,
+            AdminLastName = req.AdminLastName?.Trim() ?? string.Empty,
+            Plan = plan,
+            ScreenCount = req.ScreenCount,
+            Status = CustomerStatus.PendingPayment,
+            CreatedAt = DateTime.UtcNow,
+        };
+
+        db.Customers.Add(customer);
+        await db.SaveChangesAsync();
+
+        // ── Stripe Checkout Session ─────────────────────────────────────────
+        var priceKey = $"Stripe:Prices:{req.Plan}:{req.BillingFrequency}".ToLowerInvariant();
+        var priceId = config[priceKey];
+        if (string.IsNullOrWhiteSpace(priceId))
+        {
+            logger.LogError("Stripe price ID not configured for key {PriceKey}", priceKey);
+            return Results.Problem("Billing configuration error. Contact support.", statusCode: 500);
+        }
+
+        var sessionOptions = new SessionCreateOptions
+        {
+            Mode = "subscription",
+            CustomerEmail = customer.AdminEmail,
+            LineItems = new List<SessionLineItemOptions>
+            {
+                new()
+                {
+                    Price = priceId,
+                    Quantity = req.ScreenCount,
+                },
+            },
+            SubscriptionData = new SessionSubscriptionDataOptions
+            {
+                TrialPeriodDays = 14,
+            },
+            Metadata = new Dictionary<string, string>
+            {
+                ["ots_customer_id"] = customer.Id.ToString(),
+                ["company_name"] = customer.CompanyName,
+                ["admin_email"] = customer.AdminEmail,
+                ["admin_first_name"] = customer.AdminFirstName,
+                ["admin_last_name"] = customer.AdminLastName,
+                ["plan"] = req.Plan!,
+                ["screen_count"] = req.ScreenCount.ToString(),
+                ["billing_frequency"] = req.BillingFrequency!,
+            },
+            SuccessUrl = config["Stripe:SuccessUrl"] ?? "https://app.ots-signs.com/signup/success?session_id={CHECKOUT_SESSION_ID}",
+            CancelUrl = config["Stripe:CancelUrl"] ?? "https://app.ots-signs.com/signup/cancel",
+        };
+
+        var sessionService = new SessionService();
+        var session = await sessionService.CreateAsync(sessionOptions);
+
+        customer.StripeCheckoutSessionId = session.Id;
+        await db.SaveChangesAsync();
+
+        logger.LogInformation(
+            "Signup initiated: customer={CustomerId}, company={Company}, plan={Plan}, screens={Screens}",
+            customer.Id, customer.CompanyName, req.Plan, req.ScreenCount);
+
+        return Results.Ok(new { checkoutUrl = session.Url, statusToken = customer.Id });
+    }
+
+    private static async Task<IResult> HandleStatus(
+        Guid token,
+        OrchestratorDbContext db)
+    {
+        var customer = await db.Customers
+            .AsNoTracking()
+            .FirstOrDefaultAsync(c => c.Id == token);
+
+        if (customer is null)
+            return Results.NotFound();
+
+        // Find latest provisioning job if any
+        var job = await db.Jobs
+            .AsNoTracking()
+            .Where(j => j.CustomerId == customer.Id && j.JobType == "provision")
+            .OrderByDescending(j => j.CreatedAt)
+            .FirstOrDefaultAsync();
+
+        int pctComplete = customer.Status switch
+        {
+            CustomerStatus.PendingPayment => 0,
+            CustomerStatus.Provisioning => job?.Status switch
+            {
+                JobStatus.Running => 50,
+                JobStatus.Completed => 100,
+                _ => 10,
+            },
+            CustomerStatus.Active => 100,
+            _ => 0,
+        };
+
+        return Results.Ok(new
+        {
+            status = customer.Status.ToString(),
+            provisioningStep = job?.Steps
+                .Where(s => s.Status == JobStepStatus.Running)
+                .Select(s => s.StepName)
+                .FirstOrDefault() ?? (customer.Status == CustomerStatus.Active ? "complete" : "waiting"),
+            pctComplete,
+        });
+    }
+}
+
+public record SignupRequest(
+    string? CompanyName,
+    string? AdminFirstName,
+    string? AdminLastName,
+    string? AdminEmail,
+    string? Phone,
+    string? Plan,
+    int ScreenCount,
+    string? BillingFrequency,
+    string? PromoCode);

OTSSignsOrchestrator.Server/Auth/JwtOptions.cs

@@ -0,0 +1,10 @@
+namespace OTSSignsOrchestrator.Server.Auth;
+
+public class JwtOptions
+{
+    public const string Section = "Jwt";
+
+    public string Key { get; set; } = string.Empty;
+    public string Issuer { get; set; } = "OTSSignsOrchestrator";
+    public string Audience { get; set; } = "OTSSignsOrchestrator";
+}

OTSSignsOrchestrator.Server/Auth/OperatorAuthService.cs

@@ -0,0 +1,102 @@
+using System.IdentityModel.Tokens.Jwt;
+using System.Security.Claims;
+using System.Security.Cryptography;
+using System.Text;
+using Microsoft.EntityFrameworkCore;
+using Microsoft.Extensions.Options;
+using Microsoft.IdentityModel.Tokens;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Auth;
+
+public class OperatorAuthService
+{
+    private readonly OrchestratorDbContext _db;
+    private readonly JwtOptions _jwt;
+    private readonly ILogger<OperatorAuthService> _logger;
+
+    public OperatorAuthService(
+        OrchestratorDbContext db,
+        IOptions<JwtOptions> jwt,
+        ILogger<OperatorAuthService> logger)
+    {
+        _db = db;
+        _jwt = jwt.Value;
+        _logger = logger;
+    }
+
+    public async Task<(string Jwt, string RefreshToken)> LoginAsync(string email, string password)
+    {
+        var op = await _db.Operators.FirstOrDefaultAsync(
+            o => o.Email == email.Trim().ToLowerInvariant());
+
+        if (op is null || !BCrypt.Net.BCrypt.Verify(password, op.PasswordHash))
+        {
+            _logger.LogWarning("Login failed for {Email}", email);
+            throw new UnauthorizedAccessException("Invalid email or password.");
+        }
+
+        _logger.LogInformation("Operator {Email} logged in", op.Email);
+        var jwt = GenerateJwt(op);
+        var refresh = await CreateRefreshTokenAsync(op.Id);
+        return (jwt, refresh);
+    }
+
+    public async Task<string> RefreshAsync(string refreshToken)
+    {
+        var token = await _db.RefreshTokens
+            .Include(r => r.Operator)
+            .FirstOrDefaultAsync(r => r.Token == refreshToken);
+
+        if (token is null || token.RevokedAt is not null || token.ExpiresAt < DateTime.UtcNow)
+            throw new UnauthorizedAccessException("Invalid or expired refresh token.");
+
+        // Revoke the used token (single-use rotation)
+        token.RevokedAt = DateTime.UtcNow;
+        await _db.SaveChangesAsync();
+
+        _logger.LogInformation("Refresh token used for operator {Email}", token.Operator.Email);
+        return GenerateJwt(token.Operator);
+    }
+
+    private string GenerateJwt(Operator op)
+    {
+        var key = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(_jwt.Key));
+        var creds = new SigningCredentials(key, SecurityAlgorithms.HmacSha256);
+
+        var claims = new[]
+        {
+            new Claim(JwtRegisteredClaimNames.Sub, op.Id.ToString()),
+            new Claim(JwtRegisteredClaimNames.Email, op.Email),
+            new Claim(ClaimTypes.Name, op.Email),
+            new Claim(ClaimTypes.Role, op.Role.ToString()),
+            new Claim(JwtRegisteredClaimNames.Jti, Guid.NewGuid().ToString()),
+        };
+
+        var token = new JwtSecurityToken(
+            issuer: _jwt.Issuer,
+            audience: _jwt.Audience,
+            claims: claims,
+            expires: DateTime.UtcNow.AddMinutes(15),
+            signingCredentials: creds);
+
+        return new JwtSecurityTokenHandler().WriteToken(token);
+    }
+
+    private async Task<string> CreateRefreshTokenAsync(Guid operatorId)
+    {
+        var tokenValue = Convert.ToBase64String(RandomNumberGenerator.GetBytes(64));
+
+        _db.RefreshTokens.Add(new RefreshToken
+        {
+            Id = Guid.NewGuid(),
+            OperatorId = operatorId,
+            Token = tokenValue,
+            ExpiresAt = DateTime.UtcNow.AddDays(7),
+        });
+
+        await _db.SaveChangesAsync();
+        return tokenValue;
+    }
+}

OTSSignsOrchestrator.Server/Clients/IAuthentikClient.cs

@@ -0,0 +1,146 @@
+using Refit;
+
+namespace OTSSignsOrchestrator.Server.Clients;
+
+// ── Configuration ───────────────────────────────────────────────────────────
+public sealed class AuthentikOptions
+{
+    public const string Section = "Authentik";
+
+    public string BaseUrl { get; set; } = string.Empty;
+    public string ApiToken { get; set; } = string.Empty;
+
+    /// <summary>UUID of the OTS signing certificate-key pair used for all SAML sources.</summary>
+    public string OtsSigningKpId { get; set; } = string.Empty;
+
+    /// <summary>Authentik pre-authentication flow slug for SAML sources (e.g. "default-source-pre-authentication").</summary>
+    public string SourcePreAuthFlowSlug { get; set; } = "default-source-pre-authentication";
+
+    /// <summary>Authentik authentication flow slug for SAML sources (e.g. "default-source-authentication").</summary>
+    public string SourceAuthFlowSlug { get; set; } = "default-source-authentication";
+}
+
+// ── Request DTOs ────────────────────────────────────────────────────────────
+public record CreateSamlProviderRequest(
+    string Name,
+    string AuthorizationFlow,
+    string AcsUrl,
+    string Issuer,
+    string SpBinding,
+    string Audience,
+    string? SigningKp);
+
+public record CreateAuthentikApplicationRequest(
+    string Name,
+    string Slug,
+    string Provider,
+    string? MetaLaunchUrl);
+
+public record CreateAuthentikGroupRequest(
+    string Name,
+    bool? IsSuperuser,
+    string? Parent);
+
+public record CreateFlowRequest(
+    string Name,
+    bool? SingleUse,
+    DateTimeOffset? Expires);
+
+public record CreateAuthentikUserRequest(
+    string Username,
+    string Name,
+    string Email,
+    string[] Groups);
+
+public record ImportCertRequest(
+    string Name,
+    string CertificateData,
+    string? KeyData);
+
+public record CreateSamlSourceRequest(
+    string Name,
+    string Slug,
+    string SsoUrl,
+    string? SloUrl,
+    string Issuer,
+    string? SigningKp,
+    string? VerificationKp,
+    string BindingType,
+    string NameIdPolicy,
+    string PreAuthenticationFlow,
+    string AuthenticationFlow,
+    bool AllowIdpInitiated);
+
+// ── Response DTOs ───────────────────────────────────────────────────────────
+/// <summary>Authentik paginated list response. Results contain dictionaries with entity fields.</summary>
+public record AuthentikPagedResult(
+    List<Dictionary<string, object>> Results);
+
+// ── Authentik Refit Interface ───────────────────────────────────────────────
+// One global Authentik instance serves all tenants.
+[Headers("Authorization: Bearer")]
+public interface IAuthentikClient
+{
+    // ── SAML Providers ──────────────────────────────────────────────────────
+    [Post("/api/v3/providers/saml/")]
+    Task<ApiResponse<Dictionary<string, object>>> CreateSamlProviderAsync(
+        [Body] CreateSamlProviderRequest body);
+
+    [Get("/api/v3/providers/saml/{id}/")]
+    Task<ApiResponse<Dictionary<string, object>>> GetSamlProviderAsync(int id);
+
+    [Delete("/api/v3/providers/saml/{id}/")]
+    Task DeleteSamlProviderAsync(int id);
+
+    // ── Applications ────────────────────────────────────────────────────────
+    [Post("/api/v3/core/applications/")]
+    Task<ApiResponse<Dictionary<string, object>>> CreateApplicationAsync(
+        [Body] CreateAuthentikApplicationRequest body);
+
+    [Delete("/api/v3/core/applications/{slug}/")]
+    Task DeleteApplicationAsync(string slug);
+
+    // ── Groups ──────────────────────────────────────────────────────────────
+    [Get("/api/v3/core/groups/")]
+    Task<ApiResponse<AuthentikPagedResult>> ListGroupsAsync([AliasAs("search")] string? search = null);
+
+    [Post("/api/v3/core/groups/")]
+    Task<ApiResponse<Dictionary<string, object>>> CreateGroupAsync(
+        [Body] CreateAuthentikGroupRequest body);
+
+    [Delete("/api/v3/core/groups/{id}/")]
+    Task DeleteGroupAsync(string id);
+
+    // ── Invitations ─────────────────────────────────────────────────────────
+    [Post("/api/v3/stages/invitation/invitations/")]
+    Task<ApiResponse<Dictionary<string, object>>> CreateInvitationAsync(
+        [Body] CreateFlowRequest body);
+
+    // ── Users ───────────────────────────────────────────────────────────────
+    [Post("/api/v3/core/users/")]
+    Task<ApiResponse<Dictionary<string, object>>> CreateUserAsync(
+        [Body] CreateAuthentikUserRequest body);
+
+    // ── Health ──────────────────────────────────────────────────────────────
+    [Get("/api/v3/-/health/ready/")]
+    Task<ApiResponse<object>> CheckHealthAsync();
+
+    // ── Certificates ────────────────────────────────────────────────────────
+    [Get("/api/v3/crypto/certificatekeypairs/{kpId}/")]
+    Task<ApiResponse<Dictionary<string, object>>> GetCertificateKeyPairAsync(string kpId);
+
+    [Post("/api/v3/crypto/certificatekeypairs/")]
+    Task<ApiResponse<Dictionary<string, object>>> ImportCertificateAsync(
+        [Body] ImportCertRequest body);
+
+    // ── SAML Sources ────────────────────────────────────────────────────────
+    [Post("/api/v3/sources/saml/")]
+    Task<ApiResponse<Dictionary<string, object>>> CreateSamlSourceAsync(
+        [Body] CreateSamlSourceRequest body);
+
+    [Get("/api/v3/sources/saml/{slug}/metadata/")]
+    Task<ApiResponse<string>> GetSamlSourceMetadataAsync(string slug);
+
+    [Delete("/api/v3/sources/saml/{slug}/")]
+    Task DeleteSamlSourceAsync(string slug);
+}

OTSSignsOrchestrator.Server/Clients/IXiboApiClient.cs

@@ -0,0 +1,138 @@
+using Refit;
+
+namespace OTSSignsOrchestrator.Server.Clients;
+
+// ── Request DTOs ────────────────────────────────────────────────────────────
+public record CreateUserRequest(
+    string UserName,
+    string Email,
+    string Password,
+    int UserTypeId,
+    int HomePageId);
+
+public record UpdateUserRequest(
+    string? UserName,
+    string? Email,
+    string? Password,
+    int? UserTypeId,
+    int? HomePageId,
+    int? Retired);
+
+public record CreateGroupRequest(string Group, string? Description);
+
+public record AssignMemberRequest(int[] UserId);
+
+public record SetAclRequest(string[] ObjectId, string[] PermissionsId);
+
+public record CreateApplicationRequest(string Name);
+
+public record UpdateSettingsRequest(Dictionary<string, string> Settings);
+
+public record CreateDisplayRequest(string Display, string? Description);
+
+// ── Xibo CMS Refit Interface ────────────────────────────────────────────────
+// CRITICAL: GET /api/application is BLOCKED — only POST and DELETE exist.
+// All group endpoints use /api/group, NOT /api/usergroup.
+// Feature assignment is POST /api/group/{id}/acl, NOT /features.
+// Xibo paginates at 10 items by default — always pass start + length params.
+[Headers("Authorization: Bearer")]
+public interface IXiboApiClient
+{
+    // ── About ───────────────────────────────────────────────────────────────
+    [Get("/about")]
+    Task<ApiResponse<object>> GetAboutAsync();
+
+    // ── Users ───────────────────────────────────────────────────────────────
+    [Get("/user")]
+    Task<List<Dictionary<string, object>>> GetUsersAsync(
+        [AliasAs("start")] int? start = 0,
+        [AliasAs("length")] int? length = 200);
+
+    [Post("/user")]
+    Task<ApiResponse<Dictionary<string, object>>> CreateUserAsync(
+        [Body(BodySerializationMethod.UrlEncoded)] CreateUserRequest body);
+
+    [Put("/user/{userId}")]
+    Task<ApiResponse<Dictionary<string, object>>> UpdateUserAsync(
+        int userId,
+        [Body(BodySerializationMethod.UrlEncoded)] UpdateUserRequest body);
+
+    [Delete("/user/{userId}")]
+    Task DeleteUserAsync(int userId);
+
+    // ── Groups (NOT /usergroup) ─────────────────────────────────────────────
+    [Get("/group")]
+    Task<List<Dictionary<string, object>>> GetGroupsAsync(
+        [AliasAs("start")] int? start = 0,
+        [AliasAs("length")] int? length = 200);
+
+    [Post("/group")]
+    Task<ApiResponse<Dictionary<string, object>>> CreateGroupAsync(
+        [Body(BodySerializationMethod.UrlEncoded)] CreateGroupRequest body);
+
+    [Delete("/group/{groupId}")]
+    Task DeleteGroupAsync(int groupId);
+
+    [Post("/group/members/assign/{groupId}")]
+    Task<ApiResponse<object>> AssignUserToGroupAsync(
+        int groupId,
+        [Body(BodySerializationMethod.UrlEncoded)] AssignMemberRequest body);
+
+    // ACL — NOT /features
+    [Post("/group/{groupId}/acl")]
+    Task<ApiResponse<object>> SetGroupAclAsync(
+        int groupId,
+        [Body(BodySerializationMethod.UrlEncoded)] SetAclRequest body);
+
+    // ── Displays ────────────────────────────────────────────────────────────
+    [Get("/display")]
+    Task<List<Dictionary<string, object>>> GetDisplaysAsync(
+        [AliasAs("start")] int? start = 0,
+        [AliasAs("length")] int? length = 200,
+        [AliasAs("authorised")] int? authorised = null);
+
+    // ── Applications (POST + DELETE only — GET is BLOCKED) ──────────────────
+    [Post("/application")]
+    Task<ApiResponse<Dictionary<string, object>>> CreateApplicationAsync(
+        [Body(BodySerializationMethod.UrlEncoded)] CreateApplicationRequest body);
+
+    [Delete("/application/{key}")]
+    Task DeleteApplicationAsync(string key);
+
+    // ── Settings ────────────────────────────────────────────────────────────
+    [Get("/settings")]
+    Task<ApiResponse<object>> GetSettingsAsync();
+
+    [Put("/settings")]
+    Task<ApiResponse<object>> UpdateSettingsAsync(
+        [Body(BodySerializationMethod.UrlEncoded)] UpdateSettingsRequest body);
+}
+
+// ── Pagination helper ───────────────────────────────────────────────────────
+public static class XiboApiClientExtensions
+{
+    /// <summary>
+    /// Pages through a Xibo list endpoint until a page returns fewer items than pageSize.
+    /// </summary>
+    public static async Task<List<T>> GetAllPagesAsync<T>(
+        this IXiboApiClient client,
+        Func<int, int, Task<List<T>>> listMethod,
+        int pageSize = 200)
+    {
+        var all = new List<T>();
+        var start = 0;
+
+        while (true)
+        {
+            var page = await listMethod(start, pageSize);
+            all.AddRange(page);
+
+            if (page.Count < pageSize)
+                break;
+
+            start += pageSize;
+        }
+
+        return all;
+    }
+}

OTSSignsOrchestrator.Server/Clients/XiboClientFactory.cs

@@ -0,0 +1,211 @@
+using System.Collections.Concurrent;
+using System.Net;
+using System.Net.Http.Headers;
+using System.Text.Json;
+using Polly;
+using Polly.Retry;
+using Refit;
+
+namespace OTSSignsOrchestrator.Server.Clients;
+
+/// <summary>
+/// Creates per-instance <see cref="IXiboApiClient"/> Refit proxies with
+/// OAuth2 bearer-token caching, auto-refresh on 401, and Polly retry.
+/// Registered as a singleton.
+/// </summary>
+public sealed class XiboClientFactory
+{
+    private readonly IHttpClientFactory _httpClientFactory;
+    private readonly ConcurrentDictionary<string, TokenEntry> _tokenCache = new();
+    private readonly SemaphoreSlim _tokenLock = new(1, 1);
+
+    private static readonly TimeSpan TokenCacheTtl = TimeSpan.FromMinutes(5);
+
+    public XiboClientFactory(IHttpClientFactory httpClientFactory)
+    {
+        _httpClientFactory = httpClientFactory;
+    }
+
+    /// <summary>
+    /// Build a Refit client targeting <paramref name="instanceBaseUrl"/>/api.
+    /// Tokens are cached per base URL for 5 minutes and auto-refreshed on 401.
+    /// </summary>
+    public async Task<IXiboApiClient> CreateAsync(
+        string instanceBaseUrl,
+        string clientId,
+        string clientSecret)
+    {
+        // Ensure we have a valid token up-front
+        var token = await GetOrRefreshTokenAsync(instanceBaseUrl, clientId, clientSecret);
+
+        var retryPipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
+            .AddRetry(new RetryStrategyOptions<HttpResponseMessage>
+            {
+                MaxRetryAttempts = 3,
+                BackoffType = DelayBackoffType.Exponential,
+                Delay = TimeSpan.FromSeconds(1),
+                ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
+                    .HandleResult(r =>
+                        r.StatusCode is HttpStatusCode.RequestTimeout
+                            or HttpStatusCode.TooManyRequests
+                            or >= HttpStatusCode.InternalServerError),
+            })
+            .Build();
+
+        var handler = new XiboDelegatingHandler(
+            this, instanceBaseUrl, clientId, clientSecret, token, retryPipeline)
+        {
+            InnerHandler = new HttpClientHandler(),
+        };
+
+        var httpClient = new HttpClient(handler)
+        {
+            BaseAddress = new Uri(instanceBaseUrl.TrimEnd('/') + "/api"),
+        };
+
+        return RestService.For<IXiboApiClient>(httpClient);
+    }
+
+    // ── Token management ────────────────────────────────────────────────────
+
+    internal async Task<string> GetOrRefreshTokenAsync(
+        string instanceBaseUrl,
+        string clientId,
+        string clientSecret,
+        bool forceRefresh = false)
+    {
+        var key = instanceBaseUrl.TrimEnd('/').ToLowerInvariant();
+
+        if (!forceRefresh
+            && _tokenCache.TryGetValue(key, out var cached)
+            && cached.ExpiresAt > DateTimeOffset.UtcNow)
+        {
+            return cached.AccessToken;
+        }
+
+        await _tokenLock.WaitAsync();
+        try
+        {
+            // Double-check after acquiring lock
+            if (!forceRefresh
+                && _tokenCache.TryGetValue(key, out cached)
+                && cached.ExpiresAt > DateTimeOffset.UtcNow)
+            {
+                return cached.AccessToken;
+            }
+
+            var token = await RequestTokenAsync(instanceBaseUrl, clientId, clientSecret);
+            _tokenCache[key] = new TokenEntry(token, DateTimeOffset.UtcNow.Add(TokenCacheTtl));
+            return token;
+        }
+        finally
+        {
+            _tokenLock.Release();
+        }
+    }
+
+    private static async Task<string> RequestTokenAsync(
+        string instanceBaseUrl,
+        string clientId,
+        string clientSecret)
+    {
+        using var http = new HttpClient();
+
+        var content = new FormUrlEncodedContent(new Dictionary<string, string>
+        {
+            ["grant_type"] = "client_credentials",
+            ["client_id"] = clientId,
+            ["client_secret"] = clientSecret,
+        });
+
+        var response = await http.PostAsync(
+            $"{instanceBaseUrl.TrimEnd('/')}/api/authorize/access_token",
+            content);
+
+        response.EnsureSuccessStatusCode();
+
+        var json = await response.Content.ReadAsStringAsync();
+        using var doc = JsonDocument.Parse(json);
+        return doc.RootElement.GetProperty("access_token").GetString()
+            ?? throw new InvalidOperationException("Token response missing access_token.");
+    }
+
+    private sealed record TokenEntry(string AccessToken, DateTimeOffset ExpiresAt);
+
+    // ── Delegating handler ──────────────────────────────────────────────────
+
+    private sealed class XiboDelegatingHandler : DelegatingHandler
+    {
+        private readonly XiboClientFactory _factory;
+        private readonly string _instanceBaseUrl;
+        private readonly string _clientId;
+        private readonly string _clientSecret;
+        private readonly ResiliencePipeline<HttpResponseMessage> _retryPipeline;
+        private string _accessToken;
+
+        public XiboDelegatingHandler(
+            XiboClientFactory factory,
+            string instanceBaseUrl,
+            string clientId,
+            string clientSecret,
+            string accessToken,
+            ResiliencePipeline<HttpResponseMessage> retryPipeline)
+        {
+            _factory = factory;
+            _instanceBaseUrl = instanceBaseUrl;
+            _clientId = clientId;
+            _clientSecret = clientSecret;
+            _accessToken = accessToken;
+            _retryPipeline = retryPipeline;
+        }
+
+        protected override async Task<HttpResponseMessage> SendAsync(
+            HttpRequestMessage request,
+            CancellationToken cancellationToken)
+        {
+            return await _retryPipeline.ExecuteAsync(async ct =>
+            {
+                // Clone the request for retries (original may already be disposed)
+                using var clone = await CloneRequestAsync(request);
+                clone.Headers.Authorization =
+                    new AuthenticationHeaderValue("Bearer", _accessToken);
+
+                var response = await base.SendAsync(clone, ct);
+
+                if (response.StatusCode == HttpStatusCode.Unauthorized)
+                {
+                    // Force-refresh the token and retry once
+                    _accessToken = await _factory.GetOrRefreshTokenAsync(
+                        _instanceBaseUrl, _clientId, _clientSecret, forceRefresh: true);
+
+                    using var retry = await CloneRequestAsync(request);
+                    retry.Headers.Authorization =
+                        new AuthenticationHeaderValue("Bearer", _accessToken);
+
+                    response = await base.SendAsync(retry, ct);
+                }
+
+                return response;
+            }, cancellationToken);
+        }
+
+        private static async Task<HttpRequestMessage> CloneRequestAsync(
+            HttpRequestMessage original)
+        {
+            var clone = new HttpRequestMessage(original.Method, original.RequestUri);
+
+            if (original.Content != null)
+            {
+                var body = await original.Content.ReadAsByteArrayAsync();
+                clone.Content = new ByteArrayContent(body);
+                foreach (var header in original.Content.Headers)
+                    clone.Content.Headers.TryAddWithoutValidation(header.Key, header.Value);
+            }
+
+            foreach (var header in original.Headers)
+                clone.Headers.TryAddWithoutValidation(header.Key, header.Value);
+
+            return clone;
+        }
+    }
+}

OTSSignsOrchestrator.Server/Data/Entities/AuditLog.cs

@@ -0,0 +1,13 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public class AuditLog
+{
+    public Guid Id { get; set; }
+    public Guid? InstanceId { get; set; }
+    public string Actor { get; set; } = string.Empty;
+    public string Action { get; set; } = string.Empty;
+    public string Target { get; set; } = string.Empty;
+    public string? Outcome { get; set; }
+    public string? Detail { get; set; }
+    public DateTime OccurredAt { get; set; }
+}

OTSSignsOrchestrator.Server/Data/Entities/AuthentikMetrics.cs

@@ -0,0 +1,17 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public enum AuthentikMetricsStatus
+{
+    Healthy,
+    Degraded,
+    Critical
+}
+
+public class AuthentikMetrics
+{
+    public Guid Id { get; set; }
+    public DateTime CheckedAt { get; set; }
+    public AuthentikMetricsStatus Status { get; set; }
+    public int LatencyMs { get; set; }
+    public string? ErrorMessage { get; set; }
+}

OTSSignsOrchestrator.Server/Data/Entities/ByoiConfig.cs

@@ -0,0 +1,16 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public class ByoiConfig
+{
+    public Guid Id { get; set; }
+    public Guid InstanceId { get; set; }
+    public string Slug { get; set; } = string.Empty;
+    public string EntityId { get; set; } = string.Empty;
+    public string SsoUrl { get; set; } = string.Empty;
+    public string CertPem { get; set; } = string.Empty;
+    public DateTime CertExpiry { get; set; }
+    public bool Enabled { get; set; }
+    public DateTime CreatedAt { get; set; }
+
+    public Instance Instance { get; set; } = null!;
+}

OTSSignsOrchestrator.Server/Data/Entities/Customer.cs

@@ -0,0 +1,38 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public enum CustomerPlan
+{
+    Essentials,
+    Pro
+}
+
+public enum CustomerStatus
+{
+    PendingPayment,
+    Provisioning,
+    Active,
+    Suspended,
+    Decommissioned
+}
+
+public class Customer
+{
+    public Guid Id { get; set; }
+    public string Abbreviation { get; set; } = string.Empty;
+    public string CompanyName { get; set; } = string.Empty;
+    public string AdminEmail { get; set; } = string.Empty;
+    public string AdminFirstName { get; set; } = string.Empty;
+    public string AdminLastName { get; set; } = string.Empty;
+    public CustomerPlan Plan { get; set; }
+    public int ScreenCount { get; set; }
+    public string? StripeCustomerId { get; set; }
+    public string? StripeSubscriptionId { get; set; }
+    public string? StripeCheckoutSessionId { get; set; }
+    public CustomerStatus Status { get; set; }
+    public int FailedPaymentCount { get; set; }
+    public DateTime? FirstPaymentFailedAt { get; set; }
+    public DateTime CreatedAt { get; set; }
+
+    public ICollection<Instance> Instances { get; set; } = [];
+    public ICollection<Job> Jobs { get; set; } = [];
+}

OTSSignsOrchestrator.Server/Data/Entities/HealthEvent.cs

@@ -0,0 +1,21 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public enum HealthEventStatus
+{
+    Healthy,
+    Degraded,
+    Critical
+}
+
+public class HealthEvent
+{
+    public Guid Id { get; set; }
+    public Guid InstanceId { get; set; }
+    public string CheckName { get; set; } = string.Empty;
+    public HealthEventStatus Status { get; set; }
+    public string? Message { get; set; }
+    public bool Remediated { get; set; }
+    public DateTime OccurredAt { get; set; }
+
+    public Instance Instance { get; set; } = null!;
+}

OTSSignsOrchestrator.Server/Data/Entities/Instance.cs

@@ -0,0 +1,30 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public enum HealthStatus
+{
+    Unknown,
+    Healthy,
+    Degraded,
+    Critical
+}
+
+public class Instance
+{
+    public Guid Id { get; set; }
+    public Guid CustomerId { get; set; }
+    public string XiboUrl { get; set; } = string.Empty;
+    public string DockerStackName { get; set; } = string.Empty;
+    public string MysqlDatabase { get; set; } = string.Empty;
+    public string NfsPath { get; set; } = string.Empty;
+    public string? CmsAdminPassRef { get; set; }
+    public string? AuthentikProviderId { get; set; }
+    public HealthStatus HealthStatus { get; set; }
+    public DateTime? LastHealthCheck { get; set; }
+    public DateTime CreatedAt { get; set; }
+
+    public Customer Customer { get; set; } = null!;
+    public ICollection<HealthEvent> HealthEvents { get; set; } = [];
+    public ICollection<ScreenSnapshot> ScreenSnapshots { get; set; } = [];
+    public ICollection<OauthAppRegistry> OauthAppRegistries { get; set; } = [];
+    public ICollection<ByoiConfig> ByoiConfigs { get; set; } = [];
+}

OTSSignsOrchestrator.Server/Data/Entities/Job.cs

@@ -0,0 +1,26 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public enum JobStatus
+{
+    Queued,
+    Running,
+    Completed,
+    Failed
+}
+
+public class Job
+{
+    public Guid Id { get; set; }
+    public Guid CustomerId { get; set; }
+    public string JobType { get; set; } = string.Empty;
+    public JobStatus Status { get; set; }
+    public string? TriggeredBy { get; set; }
+    public string? Parameters { get; set; }
+    public DateTime CreatedAt { get; set; }
+    public DateTime? StartedAt { get; set; }
+    public DateTime? CompletedAt { get; set; }
+    public string? ErrorMessage { get; set; }
+
+    public Customer Customer { get; set; } = null!;
+    public ICollection<JobStep> Steps { get; set; } = [];
+}

OTSSignsOrchestrator.Server/Data/Entities/JobStep.cs

@@ -0,0 +1,22 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public enum JobStepStatus
+{
+    Queued,
+    Running,
+    Completed,
+    Failed
+}
+
+public class JobStep
+{
+    public Guid Id { get; set; }
+    public Guid JobId { get; set; }
+    public string StepName { get; set; } = string.Empty;
+    public JobStepStatus Status { get; set; }
+    public string? LogOutput { get; set; }
+    public DateTime? StartedAt { get; set; }
+    public DateTime? CompletedAt { get; set; }
+
+    public Job Job { get; set; } = null!;
+}

OTSSignsOrchestrator.Server/Data/Entities/OauthAppRegistry.cs

@@ -0,0 +1,11 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public class OauthAppRegistry
+{
+    public Guid Id { get; set; }
+    public Guid InstanceId { get; set; }
+    public string ClientId { get; set; } = string.Empty;
+    public DateTime CreatedAt { get; set; }
+
+    public Instance Instance { get; set; } = null!;
+}

OTSSignsOrchestrator.Server/Data/Entities/Operator.cs

@@ -0,0 +1,18 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public enum OperatorRole
+{
+    Admin,
+    Viewer
+}
+
+public class Operator
+{
+    public Guid Id { get; set; }
+    public string Email { get; set; } = string.Empty;
+    public string PasswordHash { get; set; } = string.Empty;
+    public OperatorRole Role { get; set; }
+    public DateTime CreatedAt { get; set; }
+
+    public ICollection<RefreshToken> RefreshTokens { get; set; } = [];
+}

OTSSignsOrchestrator.Server/Data/Entities/RefreshToken.cs

@@ -0,0 +1,12 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public class RefreshToken
+{
+    public Guid Id { get; set; }
+    public Guid OperatorId { get; set; }
+    public string Token { get; set; } = string.Empty;
+    public DateTime ExpiresAt { get; set; }
+    public DateTime? RevokedAt { get; set; }
+
+    public Operator Operator { get; set; } = null!;
+}

OTSSignsOrchestrator.Server/Data/Entities/ScreenSnapshot.cs

@@ -0,0 +1,12 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public class ScreenSnapshot
+{
+    public Guid Id { get; set; }
+    public Guid InstanceId { get; set; }
+    public DateOnly SnapshotDate { get; set; }
+    public int ScreenCount { get; set; }
+    public DateTime CreatedAt { get; set; }
+
+    public Instance Instance { get; set; } = null!;
+}

OTSSignsOrchestrator.Server/Data/Entities/StripeEvent.cs

@@ -0,0 +1,9 @@
+namespace OTSSignsOrchestrator.Server.Data.Entities;
+
+public class StripeEvent
+{
+    public string StripeEventId { get; set; } = string.Empty;
+    public string EventType { get; set; } = string.Empty;
+    public DateTime ProcessedAt { get; set; }
+    public string? Payload { get; set; }
+}

OTSSignsOrchestrator.Server/Data/OrchestratorDbContext.cs

@@ -0,0 +1,190 @@
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Data;
+
+public class OrchestratorDbContext : DbContext
+{
+    public OrchestratorDbContext(DbContextOptions<OrchestratorDbContext> options)
+        : base(options) { }
+
+    public DbSet<Customer> Customers => Set<Customer>();
+    public DbSet<Instance> Instances => Set<Instance>();
+    public DbSet<Job> Jobs => Set<Job>();
+    public DbSet<JobStep> JobSteps => Set<JobStep>();
+    public DbSet<HealthEvent> HealthEvents => Set<HealthEvent>();
+    public DbSet<AuditLog> AuditLogs => Set<AuditLog>();
+    public DbSet<StripeEvent> StripeEvents => Set<StripeEvent>();
+    public DbSet<ScreenSnapshot> ScreenSnapshots => Set<ScreenSnapshot>();
+    public DbSet<OauthAppRegistry> OauthAppRegistries => Set<OauthAppRegistry>();
+    public DbSet<AuthentikMetrics> AuthentikMetrics => Set<AuthentikMetrics>();
+    public DbSet<Operator> Operators => Set<Operator>();
+    public DbSet<RefreshToken> RefreshTokens => Set<RefreshToken>();
+    public DbSet<ByoiConfig> ByoiConfigs => Set<ByoiConfig>();
+
+    protected override void OnModelCreating(ModelBuilder modelBuilder)
+    {
+        base.OnModelCreating(modelBuilder);
+
+        // ── Snake-case naming convention ─────────────────────────────────
+        foreach (var entity in modelBuilder.Model.GetEntityTypes())
+        {
+            entity.SetTableName(ToSnakeCase(entity.GetTableName()!));
+
+            foreach (var property in entity.GetProperties())
+                property.SetColumnName(ToSnakeCase(property.GetColumnName()));
+
+            foreach (var key in entity.GetKeys())
+                key.SetName(ToSnakeCase(key.GetName()!));
+
+            foreach (var fk in entity.GetForeignKeys())
+                fk.SetConstraintName(ToSnakeCase(fk.GetConstraintName()!));
+
+            foreach (var index in entity.GetIndexes())
+                index.SetDatabaseName(ToSnakeCase(index.GetDatabaseName()!));
+        }
+
+        // ── Customer ────────────────────────────────────────────────────
+        modelBuilder.Entity<Customer>(e =>
+        {
+            e.HasKey(c => c.Id);
+            e.Property(c => c.Abbreviation).HasMaxLength(8);
+            e.Property(c => c.Plan).HasConversion<string>();
+            e.Property(c => c.Status).HasConversion<string>();
+            e.Property(c => c.FailedPaymentCount).HasDefaultValue(0);
+            e.HasIndex(c => c.Abbreviation).IsUnique();
+            e.HasIndex(c => c.StripeCustomerId).IsUnique();
+        });
+
+        // ── Instance ────────────────────────────────────────────────────
+        modelBuilder.Entity<Instance>(e =>
+        {
+            e.HasKey(i => i.Id);
+            e.Property(i => i.HealthStatus).HasConversion<string>();
+            e.HasIndex(i => i.CustomerId);
+            e.HasIndex(i => i.DockerStackName).IsUnique();
+            e.HasOne(i => i.Customer)
+                .WithMany(c => c.Instances)
+                .HasForeignKey(i => i.CustomerId);
+        });
+
+        // ── Job ─────────────────────────────────────────────────────────
+        modelBuilder.Entity<Job>(e =>
+        {
+            e.HasKey(j => j.Id);
+            e.Property(j => j.Status).HasConversion<string>();
+            e.Property(j => j.Parameters).HasColumnType("text");
+            e.HasIndex(j => j.CustomerId);
+            e.HasOne(j => j.Customer)
+                .WithMany(c => c.Jobs)
+                .HasForeignKey(j => j.CustomerId);
+        });
+
+        // ── JobStep ─────────────────────────────────────────────────────
+        modelBuilder.Entity<JobStep>(e =>
+        {
+            e.HasKey(s => s.Id);
+            e.Property(s => s.Status).HasConversion<string>();
+            e.Property(s => s.LogOutput).HasColumnType("text");
+            e.HasIndex(s => s.JobId);
+            e.HasOne(s => s.Job)
+                .WithMany(j => j.Steps)
+                .HasForeignKey(s => s.JobId);
+        });
+
+        // ── HealthEvent ─────────────────────────────────────────────────
+        modelBuilder.Entity<HealthEvent>(e =>
+        {
+            e.HasKey(h => h.Id);
+            e.Property(h => h.Status).HasConversion<string>();
+            e.HasIndex(h => h.InstanceId);
+            e.HasOne(h => h.Instance)
+                .WithMany(i => i.HealthEvents)
+                .HasForeignKey(h => h.InstanceId);
+        });
+
+        // ── AuditLog ────────────────────────────────────────────────────
+        modelBuilder.Entity<AuditLog>(e =>
+        {
+            e.HasKey(a => a.Id);
+            e.Property(a => a.Detail).HasColumnType("text");
+            e.HasIndex(a => a.InstanceId);
+            e.HasIndex(a => a.OccurredAt);
+        });
+
+        // ── StripeEvent ─────────────────────────────────────────────────
+        modelBuilder.Entity<StripeEvent>(e =>
+        {
+            e.HasKey(s => s.StripeEventId);
+            e.Property(s => s.Payload).HasColumnType("text");
+        });
+
+        // ── ScreenSnapshot ──────────────────────────────────────────────
+        modelBuilder.Entity<ScreenSnapshot>(e =>
+        {
+            e.HasKey(s => s.Id);
+            e.HasIndex(s => s.InstanceId);
+            e.HasOne(s => s.Instance)
+                .WithMany(i => i.ScreenSnapshots)
+                .HasForeignKey(s => s.InstanceId);
+        });
+
+        // ── OauthAppRegistry ────────────────────────────────────────────
+        modelBuilder.Entity<OauthAppRegistry>(e =>
+        {
+            e.HasKey(o => o.Id);
+            e.HasIndex(o => o.InstanceId);
+            e.HasIndex(o => o.ClientId).IsUnique();
+            e.HasOne(o => o.Instance)
+                .WithMany(i => i.OauthAppRegistries)
+                .HasForeignKey(o => o.InstanceId);
+        });
+
+        // ── AuthentikMetrics ────────────────────────────────────────────
+        modelBuilder.Entity<AuthentikMetrics>(e =>
+        {
+            e.HasKey(a => a.Id);
+            e.Property(a => a.Status).HasConversion<string>();
+        });
+
+        // ── Operator ────────────────────────────────────────────────────
+        modelBuilder.Entity<Operator>(e =>
+        {
+            e.HasKey(o => o.Id);
+            e.Property(o => o.Role).HasConversion<string>();
+            e.HasIndex(o => o.Email).IsUnique();
+        });
+
+        // ── RefreshToken ────────────────────────────────────────────────
+        modelBuilder.Entity<RefreshToken>(e =>
+        {
+            e.HasKey(r => r.Id);
+            e.HasIndex(r => r.Token).IsUnique();
+            e.HasIndex(r => r.OperatorId);
+            e.HasOne(r => r.Operator)
+                .WithMany(o => o.RefreshTokens)
+                .HasForeignKey(r => r.OperatorId);
+        });
+
+        // ── ByoiConfig ──────────────────────────────────────────────────
+        modelBuilder.Entity<ByoiConfig>(e =>
+        {
+            e.HasKey(b => b.Id);
+            e.Property(b => b.CertPem).HasColumnType("text");
+            e.HasIndex(b => b.InstanceId);
+            e.HasIndex(b => b.Slug).IsUnique();
+            e.HasOne(b => b.Instance)
+                .WithMany(i => i.ByoiConfigs)
+                .HasForeignKey(b => b.InstanceId);
+        });
+    }
+
+    private static string ToSnakeCase(string name)
+    {
+        return string.Concat(
+            name.Select((c, i) =>
+                i > 0 && char.IsUpper(c) && !char.IsUpper(name[i - 1])
+                    ? "_" + char.ToLowerInvariant(c)
+                    : char.ToLowerInvariant(c).ToString()));
+    }
+}

OTSSignsOrchestrator.Server/Health/AuthentikGlobalHealthJob.cs

@@ -0,0 +1,37 @@
+using Quartz;
+using OTSSignsOrchestrator.Server.Health.Checks;
+
+namespace OTSSignsOrchestrator.Server.Health;
+
+/// <summary>
+/// Quartz job that runs the <see cref="AuthentikGlobalHealthCheck"/> every 2 minutes
+/// on a separate schedule from the per-instance health checks.
+/// </summary>
+[DisallowConcurrentExecution]
+public sealed class AuthentikGlobalHealthJob : IJob
+{
+    private readonly AuthentikGlobalHealthCheck _check;
+    private readonly ILogger<AuthentikGlobalHealthJob> _logger;
+
+    public AuthentikGlobalHealthJob(
+        AuthentikGlobalHealthCheck check,
+        ILogger<AuthentikGlobalHealthJob> logger)
+    {
+        _check = check;
+        _logger = logger;
+    }
+
+    public async Task Execute(IJobExecutionContext context)
+    {
+        try
+        {
+            var result = await _check.RunGlobalAsync(context.CancellationToken);
+            _logger.LogInformation("Authentik global health: {Status} — {Message}",
+                result.Status, result.Message);
+        }
+        catch (Exception ex)
+        {
+            _logger.LogError(ex, "Authentik global health job failed");
+        }
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/AdminIntegrityHealthCheck.cs

@@ -0,0 +1,183 @@
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies that both <c>ots-admin-{abbrev}</c> and <c>ots-svc-{abbrev}</c> exist
+/// with <c>userTypeId == 1</c> (SuperAdmin). MUST use <see cref="XiboApiClientExtensions.GetAllPagesAsync{T}"/>
+/// because Xibo paginates at 10 items by default.
+///
+/// <c>saml-usertypeid</c> is JIT-only and does NOT maintain SuperAdmin on existing users —
+/// this check IS the ongoing enforcement mechanism.
+/// </summary>
+public sealed class AdminIntegrityHealthCheck : IHealthCheck
+{
+    private readonly XiboClientFactory _clientFactory;
+    private readonly IServiceProvider _services;
+    private readonly ILogger<AdminIntegrityHealthCheck> _logger;
+
+    public string CheckName => "AdminIntegrity";
+    public bool AutoRemediate => true;
+
+    public AdminIntegrityHealthCheck(
+        XiboClientFactory clientFactory,
+        IServiceProvider services,
+        ILogger<AdminIntegrityHealthCheck> logger)
+    {
+        _clientFactory = clientFactory;
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var (client, abbrev) = await ResolveAsync(instance);
+        if (client is null)
+            return new HealthCheckResult(HealthStatus.Critical, "No OAuth app — cannot verify admin accounts");
+
+        var users = await client.GetAllPagesAsync(
+            (start, length) => client.GetUsersAsync(start, length));
+
+        var adminName = $"ots-admin-{abbrev}";
+        var svcName = $"ots-svc-{abbrev}";
+        var problems = new List<string>();
+
+        foreach (var expected in new[] { adminName, svcName })
+        {
+            var user = users.FirstOrDefault(u =>
+                u.TryGetValue("userName", out var n) &&
+                string.Equals(n?.ToString(), expected, StringComparison.OrdinalIgnoreCase));
+
+            if (user is null)
+            {
+                problems.Add($"{expected} is MISSING");
+                continue;
+            }
+
+            if (user.TryGetValue("userTypeId", out var typeObj) &&
+                typeObj?.ToString() != "1")
+            {
+                problems.Add($"{expected} has userTypeId={typeObj} (expected 1)");
+            }
+        }
+
+        if (problems.Count == 0)
+            return new HealthCheckResult(HealthStatus.Healthy, "Admin accounts intact");
+
+        return new HealthCheckResult(
+            HealthStatus.Critical,
+            $"Admin integrity issues: {string.Join("; ", problems)}",
+            string.Join("\n", problems));
+    }
+
+    public async Task<bool> RemediateAsync(Instance instance, CancellationToken ct)
+    {
+        var (client, abbrev) = await ResolveAsync(instance);
+        if (client is null) return false;
+
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+
+        var users = await client.GetAllPagesAsync(
+            (start, length) => client.GetUsersAsync(start, length));
+
+        var adminName = $"ots-admin-{abbrev}";
+        var svcName = $"ots-svc-{abbrev}";
+        var allFixed = true;
+
+        foreach (var expected in new[] { adminName, svcName })
+        {
+            var user = users.FirstOrDefault(u =>
+                u.TryGetValue("userName", out var n) &&
+                string.Equals(n?.ToString(), expected, StringComparison.OrdinalIgnoreCase));
+
+            if (user is null)
+            {
+                // Recreate missing account
+                var email = $"{expected}@otssigns.internal";
+                var password = GenerateRandomPassword(32);
+                var createResp = await client.CreateUserAsync(new CreateUserRequest(
+                    expected, email, password, UserTypeId: 1, HomePageId: 1));
+
+                if (!createResp.IsSuccessStatusCode)
+                {
+                    _logger.LogError("Failed to recreate {User}: {Err}", expected, createResp.Error?.Content);
+                    allFixed = false;
+                    continue;
+                }
+
+                // Audit
+                db.AuditLogs.Add(new AuditLog
+                {
+                    Id = Guid.NewGuid(),
+                    InstanceId = instance.Id,
+                    Actor = "HealthCheckEngine:AdminIntegrity",
+                    Action = "RecreateUser",
+                    Target = expected,
+                    Outcome = "Success",
+                    Detail = "User was missing — recreated as SuperAdmin",
+                    OccurredAt = DateTime.UtcNow,
+                });
+            }
+            else
+            {
+                // Fix userTypeId if wrong
+                if (user.TryGetValue("userTypeId", out var typeObj) && typeObj?.ToString() != "1")
+                {
+                    var userId = int.Parse(user["userId"]?.ToString() ?? "0");
+                    if (userId == 0) { allFixed = false; continue; }
+
+                    var updateResp = await client.UpdateUserAsync(userId, new UpdateUserRequest(
+                        UserName: null, Email: null, Password: null, UserTypeId: 1,
+                        HomePageId: null, Retired: null));
+
+                    if (!updateResp.IsSuccessStatusCode)
+                    {
+                        _logger.LogError("Failed to fix userTypeId for {User}: {Err}",
+                            expected, updateResp.Error?.Content);
+                        allFixed = false;
+                        continue;
+                    }
+
+                    db.AuditLogs.Add(new AuditLog
+                    {
+                        Id = Guid.NewGuid(),
+                        InstanceId = instance.Id,
+                        Actor = "HealthCheckEngine:AdminIntegrity",
+                        Action = "FixUserType",
+                        Target = expected,
+                        Outcome = "Success",
+                        Detail = $"Changed userTypeId from {typeObj} to 1 (SuperAdmin)",
+                        OccurredAt = DateTime.UtcNow,
+                    });
+                }
+            }
+        }
+
+        await db.SaveChangesAsync(ct);
+        return allFixed;
+    }
+
+    private async Task<(IXiboApiClient? Client, string Abbrev)> ResolveAsync(Instance instance)
+    {
+        var abbrev = instance.Customer.Abbreviation;
+        var oauthApp = instance.OauthAppRegistries.FirstOrDefault();
+        if (oauthApp is null) return (null, abbrev);
+
+        var settings = _services.GetRequiredService<Core.Services.SettingsService>();
+        var secret = await settings.GetAsync(Core.Services.SettingsService.InstanceOAuthSecretId(abbrev));
+        if (string.IsNullOrEmpty(secret)) return (null, abbrev);
+
+        var client = await _clientFactory.CreateAsync(instance.XiboUrl, oauthApp.ClientId, secret);
+        return (client, abbrev);
+    }
+
+    private static string GenerateRandomPassword(int length)
+    {
+        const string chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*";
+        return System.Security.Cryptography.RandomNumberGenerator.GetString(chars, length);
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/AuthentikGlobalHealthCheck.cs

@@ -0,0 +1,107 @@
+using System.Diagnostics;
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Probes the central Authentik instance at <c>GET /api/v3/-/health/ready/</c>.
+/// Measures latency and writes an <see cref="AuthentikMetrics"/> row.
+/// If down: <c>Severity = Critical</c>, message "Central Authentik is DOWN — all customer web UI logins failing".
+/// This is a fleet-wide P1 alert. Runs every 2 minutes on a separate schedule.
+///
+/// This check is NOT per-instance — it runs once globally. The engine skips it for
+/// per-instance checks. Instead it is scheduled independently as a Quartz job.
+/// </summary>
+public sealed class AuthentikGlobalHealthCheck : IHealthCheck
+{
+    private readonly IAuthentikClient _authentikClient;
+    private readonly IServiceProvider _services;
+    private readonly ILogger<AuthentikGlobalHealthCheck> _logger;
+
+    public string CheckName => "AuthentikGlobal";
+    public bool AutoRemediate => false;
+
+    public AuthentikGlobalHealthCheck(
+        IAuthentikClient authentikClient,
+        IServiceProvider services,
+        ILogger<AuthentikGlobalHealthCheck> logger)
+    {
+        _authentikClient = authentikClient;
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        // This check doesn't use the instance parameter — it checks global Authentik health.
+        return await RunGlobalAsync(ct);
+    }
+
+    /// <summary>
+    /// Core logic — callable from the Quartz job without an instance context.
+    /// </summary>
+    public async Task<HealthCheckResult> RunGlobalAsync(CancellationToken ct)
+    {
+        var sw = Stopwatch.StartNew();
+        AuthentikMetricsStatus metricsStatus;
+        string? errorMessage = null;
+        HealthCheckResult result;
+
+        try
+        {
+            var response = await _authentikClient.CheckHealthAsync();
+            sw.Stop();
+
+            if (response.IsSuccessStatusCode)
+            {
+                metricsStatus = AuthentikMetricsStatus.Healthy;
+                result = new HealthCheckResult(HealthStatus.Healthy,
+                    $"Authentik healthy (latency: {sw.ElapsedMilliseconds}ms)");
+            }
+            else
+            {
+                metricsStatus = AuthentikMetricsStatus.Critical;
+                errorMessage = $"HTTP {response.StatusCode}";
+                result = new HealthCheckResult(HealthStatus.Critical,
+                    "Central Authentik is DOWN — all customer web UI logins failing",
+                    $"Health endpoint returned {response.StatusCode}");
+            }
+        }
+        catch (Exception ex)
+        {
+            sw.Stop();
+            metricsStatus = AuthentikMetricsStatus.Critical;
+            errorMessage = ex.Message;
+            result = new HealthCheckResult(HealthStatus.Critical,
+                "Central Authentik is DOWN — all customer web UI logins failing",
+                ex.Message);
+        }
+
+        // Write metrics row
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        db.AuthentikMetrics.Add(new AuthentikMetrics
+        {
+            Id = Guid.NewGuid(),
+            CheckedAt = DateTime.UtcNow,
+            Status = metricsStatus,
+            LatencyMs = (int)sw.ElapsedMilliseconds,
+            ErrorMessage = errorMessage,
+        });
+        await db.SaveChangesAsync(ct);
+
+        // Broadcast alert if critical
+        if (result.Status == HealthStatus.Critical)
+        {
+            var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+            await hub.Clients.All.SendAlertRaised("Critical", result.Message);
+        }
+
+        return result;
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/AuthentikSamlProviderHealthCheck.cs

@@ -0,0 +1,60 @@
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies the per-instance SAML provider in Authentik is active by checking
+/// the provider exists using the stored <see cref="Instance.AuthentikProviderId"/>.
+/// </summary>
+public sealed class AuthentikSamlProviderHealthCheck : IHealthCheck
+{
+    private readonly IAuthentikClient _authentikClient;
+    private readonly ILogger<AuthentikSamlProviderHealthCheck> _logger;
+
+    public string CheckName => "AuthentikSamlProvider";
+    public bool AutoRemediate => false;
+
+    public AuthentikSamlProviderHealthCheck(
+        IAuthentikClient authentikClient,
+        ILogger<AuthentikSamlProviderHealthCheck> logger)
+    {
+        _authentikClient = authentikClient;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        if (string.IsNullOrEmpty(instance.AuthentikProviderId))
+        {
+            return new HealthCheckResult(HealthStatus.Degraded,
+                "No Authentik provider ID stored — SAML not provisioned");
+        }
+
+        if (!int.TryParse(instance.AuthentikProviderId, out var providerId))
+        {
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"Invalid Authentik provider ID: {instance.AuthentikProviderId}");
+        }
+
+        try
+        {
+            var response = await _authentikClient.GetSamlProviderAsync(providerId);
+
+            if (response.IsSuccessStatusCode && response.Content is not null)
+            {
+                return new HealthCheckResult(HealthStatus.Healthy,
+                    $"SAML provider {providerId} is active in Authentik");
+            }
+
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"SAML provider {providerId} not found or inaccessible",
+                response.Error?.Content);
+        }
+        catch (Exception ex)
+        {
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"Failed to check SAML provider: {ex.Message}");
+        }
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/ByoiCertExpiryHealthCheck.cs

@@ -0,0 +1,69 @@
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// For Pro plan BYOI customers: checks certificate expiry from <see cref="ByoiConfig"/>.
+/// Alerts at 60-day (Warning), 30-day (Warning), 7-day (Critical) thresholds.
+/// AutoRemediate=false — customer must rotate their IdP certificate via the portal.
+/// </summary>
+public sealed class ByoiCertExpiryHealthCheck : IHealthCheck
+{
+    /// <summary>Alert thresholds in days (descending).</summary>
+    internal static readonly int[] AlertThresholdDays = [60, 30, 7];
+
+    /// <summary>Days at or below which severity escalates to Critical.</summary>
+    internal const int CriticalThresholdDays = 7;
+
+    public string CheckName => "ByoiCertExpiry";
+    public bool AutoRemediate => false;
+
+    public Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        // Only applies to instances with an enabled BYOI config
+        var byoiConfig = instance.ByoiConfigs.FirstOrDefault(b => b.Enabled);
+        if (byoiConfig is null)
+        {
+            return Task.FromResult(new HealthCheckResult(HealthStatus.Healthy,
+                "No BYOI config — check not applicable"));
+        }
+
+        // Only Pro customers have BYOI
+        if (instance.Customer.Plan != CustomerPlan.Pro)
+        {
+            return Task.FromResult(new HealthCheckResult(HealthStatus.Healthy,
+                "Non-Pro plan — BYOI check not applicable"));
+        }
+
+        var daysRemaining = (byoiConfig.CertExpiry - DateTime.UtcNow).TotalDays;
+
+        if (daysRemaining <= 0)
+        {
+            return Task.FromResult(new HealthCheckResult(HealthStatus.Critical,
+                $"BYOI certificate has EXPIRED (expired {Math.Abs((int)daysRemaining)} days ago)",
+                "Customer must rotate their IdP certificate via the portal immediately"));
+        }
+
+        if (daysRemaining <= CriticalThresholdDays)
+        {
+            return Task.FromResult(new HealthCheckResult(HealthStatus.Critical,
+                $"BYOI certificate expires in {(int)daysRemaining} days",
+                "Urgent: customer must rotate their IdP certificate"));
+        }
+
+        // Check warning thresholds (60 and 30 days)
+        foreach (var threshold in AlertThresholdDays)
+        {
+            if (threshold <= CriticalThresholdDays) continue;
+            if (daysRemaining <= threshold)
+            {
+                return Task.FromResult(new HealthCheckResult(HealthStatus.Degraded,
+                    $"BYOI certificate expires in {(int)daysRemaining} days (threshold: {threshold}d)",
+                    "Customer should plan certificate rotation"));
+            }
+        }
+
+        return Task.FromResult(new HealthCheckResult(HealthStatus.Healthy,
+            $"BYOI certificate valid for {(int)daysRemaining} more days"));
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/DisplayAuthorisedHealthCheck.cs

@@ -0,0 +1,74 @@
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies the count of authorised displays does not exceed the customer's licensed
+/// <see cref="Customer.ScreenCount"/>. Uses <see cref="XiboApiClientExtensions.GetAllPagesAsync{T}"/>
+/// with <c>authorised=1</c> filter to get all authorised displays.
+/// </summary>
+public sealed class DisplayAuthorisedHealthCheck : IHealthCheck
+{
+    private readonly XiboClientFactory _clientFactory;
+    private readonly IServiceProvider _services;
+    private readonly ILogger<DisplayAuthorisedHealthCheck> _logger;
+
+    public string CheckName => "DisplayAuthorised";
+    public bool AutoRemediate => false;
+
+    public DisplayAuthorisedHealthCheck(
+        XiboClientFactory clientFactory,
+        IServiceProvider services,
+        ILogger<DisplayAuthorisedHealthCheck> logger)
+    {
+        _clientFactory = clientFactory;
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var (client, _) = await ResolveAsync(instance);
+        if (client is null)
+            return new HealthCheckResult(HealthStatus.Critical, "No OAuth app — cannot check displays");
+
+        try
+        {
+            var displays = await client.GetAllPagesAsync(
+                (start, length) => client.GetDisplaysAsync(start, length, authorised: 1));
+
+            var authorisedCount = displays.Count;
+            var licensed = instance.Customer.ScreenCount;
+
+            if (authorisedCount <= licensed)
+            {
+                return new HealthCheckResult(HealthStatus.Healthy,
+                    $"Authorised displays: {authorisedCount}/{licensed}");
+            }
+
+            return new HealthCheckResult(HealthStatus.Degraded,
+                $"Authorised displays ({authorisedCount}) exceeds license ({licensed})",
+                $"Over-provisioned by {authorisedCount - licensed} display(s)");
+        }
+        catch (Exception ex)
+        {
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"Failed to check displays: {ex.Message}");
+        }
+    }
+
+    private async Task<(IXiboApiClient? Client, string Abbrev)> ResolveAsync(Instance instance)
+    {
+        var abbrev = instance.Customer.Abbreviation;
+        var oauthApp = instance.OauthAppRegistries.FirstOrDefault();
+        if (oauthApp is null) return (null, abbrev);
+
+        var settings = _services.GetRequiredService<Core.Services.SettingsService>();
+        var secret = await settings.GetAsync(Core.Services.SettingsService.InstanceOAuthSecretId(abbrev));
+        if (string.IsNullOrEmpty(secret)) return (null, abbrev);
+
+        var client = await _clientFactory.CreateAsync(instance.XiboUrl, oauthApp.ClientId, secret);
+        return (client, abbrev);
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/GroupStructureHealthCheck.cs

@@ -0,0 +1,124 @@
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies all 4 expected Xibo groups exist for the instance:
+/// <c>{abbrev}-viewer</c>, <c>{abbrev}-editor</c>, <c>{abbrev}-admin</c>, <c>ots-it-{abbrev}</c>.
+/// Uses <see cref="XiboApiClientExtensions.GetAllPagesAsync{T}"/> to avoid pagination truncation.
+/// </summary>
+public sealed class GroupStructureHealthCheck : IHealthCheck
+{
+    private readonly XiboClientFactory _clientFactory;
+    private readonly IServiceProvider _services;
+    private readonly ILogger<GroupStructureHealthCheck> _logger;
+
+    public string CheckName => "GroupStructure";
+    public bool AutoRemediate => true;
+
+    public GroupStructureHealthCheck(
+        XiboClientFactory clientFactory,
+        IServiceProvider services,
+        ILogger<GroupStructureHealthCheck> logger)
+    {
+        _clientFactory = clientFactory;
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var (client, abbrev) = await ResolveAsync(instance);
+        if (client is null)
+            return new HealthCheckResult(HealthStatus.Critical, "No OAuth app — cannot verify groups");
+
+        var expected = ExpectedGroups(abbrev);
+        var groups = await client.GetAllPagesAsync(
+            (start, length) => client.GetGroupsAsync(start, length));
+
+        var existing = groups
+            .Select(g => g.TryGetValue("group", out var n) ? n?.ToString() : null)
+            .Where(n => n is not null)
+            .ToHashSet(StringComparer.OrdinalIgnoreCase);
+
+        var missing = expected.Where(e => !existing.Contains(e)).ToList();
+
+        if (missing.Count == 0)
+            return new HealthCheckResult(HealthStatus.Healthy, "All 4 expected groups present");
+
+        return new HealthCheckResult(
+            HealthStatus.Critical,
+            $"Missing groups: {string.Join(", ", missing)}",
+            $"Expected: {string.Join(", ", expected)}");
+    }
+
+    public async Task<bool> RemediateAsync(Instance instance, CancellationToken ct)
+    {
+        var (client, abbrev) = await ResolveAsync(instance);
+        if (client is null) return false;
+
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+
+        var expected = ExpectedGroups(abbrev);
+        var groups = await client.GetAllPagesAsync(
+            (start, length) => client.GetGroupsAsync(start, length));
+
+        var existing = groups
+            .Select(g => g.TryGetValue("group", out var n) ? n?.ToString() : null)
+            .Where(n => n is not null)
+            .ToHashSet(StringComparer.OrdinalIgnoreCase);
+
+        var allFixed = true;
+        foreach (var name in expected.Where(e => !existing.Contains(e)))
+        {
+            var resp = await client.CreateGroupAsync(new CreateGroupRequest(name, $"Auto-created by health check for {abbrev}"));
+            if (resp.IsSuccessStatusCode)
+            {
+                db.AuditLogs.Add(new AuditLog
+                {
+                    Id = Guid.NewGuid(),
+                    InstanceId = instance.Id,
+                    Actor = "HealthCheckEngine:GroupStructure",
+                    Action = "CreateGroup",
+                    Target = name,
+                    Outcome = "Success",
+                    Detail = $"Recreated missing group {name}",
+                    OccurredAt = DateTime.UtcNow,
+                });
+            }
+            else
+            {
+                _logger.LogError("Failed to create group {Group}: {Err}", name, resp.Error?.Content);
+                allFixed = false;
+            }
+        }
+
+        await db.SaveChangesAsync(ct);
+        return allFixed;
+    }
+
+    private static string[] ExpectedGroups(string abbrev) =>
+    [
+        $"{abbrev}-viewer",
+        $"{abbrev}-editor",
+        $"{abbrev}-admin",
+        $"ots-it-{abbrev}",
+    ];
+
+    private async Task<(IXiboApiClient? Client, string Abbrev)> ResolveAsync(Instance instance)
+    {
+        var abbrev = instance.Customer.Abbreviation;
+        var oauthApp = instance.OauthAppRegistries.FirstOrDefault();
+        if (oauthApp is null) return (null, abbrev);
+
+        var settings = _services.GetRequiredService<Core.Services.SettingsService>();
+        var secret = await settings.GetAsync(Core.Services.SettingsService.InstanceOAuthSecretId(abbrev));
+        if (string.IsNullOrEmpty(secret)) return (null, abbrev);
+
+        var client = await _clientFactory.CreateAsync(instance.XiboUrl, oauthApp.ClientId, secret);
+        return (client, abbrev);
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/InvitationFlowHealthCheck.cs

@@ -0,0 +1,63 @@
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies the <c>invite-{abbrev}</c> flow exists in Authentik by searching for it
+/// in the invitation stages list.
+/// </summary>
+public sealed class InvitationFlowHealthCheck : IHealthCheck
+{
+    private readonly IAuthentikClient _authentikClient;
+    private readonly ILogger<InvitationFlowHealthCheck> _logger;
+
+    public string CheckName => "InvitationFlow";
+    public bool AutoRemediate => false;
+
+    public InvitationFlowHealthCheck(
+        IAuthentikClient authentikClient,
+        ILogger<InvitationFlowHealthCheck> logger)
+    {
+        _authentikClient = authentikClient;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var abbrev = instance.Customer.Abbreviation;
+        var expectedName = $"invite-{abbrev}";
+
+        try
+        {
+            // Search Authentik groups for evidence of the invitation flow
+            // The invitation is created as a stage invitation; we verify via the
+            // Authentik API by searching for it by name.
+            var groupResponse = await _authentikClient.ListGroupsAsync(expectedName);
+
+            if (groupResponse.IsSuccessStatusCode && groupResponse.Content?.Results is { Count: > 0 })
+            {
+                var found = groupResponse.Content.Results.Any(g =>
+                    g.TryGetValue("name", out var n) &&
+                    string.Equals(n?.ToString(), expectedName, StringComparison.OrdinalIgnoreCase));
+
+                if (found)
+                {
+                    return new HealthCheckResult(HealthStatus.Healthy,
+                        $"Invitation flow '{expectedName}' exists in Authentik");
+                }
+            }
+
+            // If groups don't show it, it's still possible the invitation was created
+            // as a separate stage object. Log as degraded since we can't fully confirm.
+            return new HealthCheckResult(HealthStatus.Degraded,
+                $"Invitation flow '{expectedName}' not found in Authentik",
+                "The invitation may exist but could not be verified via group search");
+        }
+        catch (Exception ex)
+        {
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"Failed to check invitation flow: {ex.Message}");
+        }
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/MySqlConnectHealthCheck.cs

@@ -0,0 +1,106 @@
+using Renci.SshNet;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies connectivity to the instance's MySQL database by running a simple query
+/// via SSH against the Docker Swarm host.
+/// </summary>
+public sealed class MySqlConnectHealthCheck : IHealthCheck
+{
+    private readonly IServiceProvider _services;
+    private readonly ILogger<MySqlConnectHealthCheck> _logger;
+
+    public string CheckName => "MySqlConnect";
+    public bool AutoRemediate => false;
+
+    public MySqlConnectHealthCheck(
+        IServiceProvider services,
+        ILogger<MySqlConnectHealthCheck> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var dbName = instance.MysqlDatabase;
+        if (string.IsNullOrEmpty(dbName))
+            return new HealthCheckResult(HealthStatus.Critical, "No MySQL database configured");
+
+        try
+        {
+            var settings = _services.GetRequiredService<Core.Services.SettingsService>();
+            var sshInfo = await GetSwarmSshHostAsync(settings);
+            var mysqlHost = await settings.GetAsync(Core.Services.SettingsService.MySqlHost, "localhost");
+            var mysqlPort = await settings.GetAsync(Core.Services.SettingsService.MySqlPort, "3306");
+            var mysqlUser = await settings.GetAsync(Core.Services.SettingsService.MySqlAdminUser, "root");
+            var mysqlPass = await settings.GetAsync(Core.Services.SettingsService.MySqlAdminPassword, "");
+
+            using var sshClient = CreateSshClient(sshInfo);
+            sshClient.Connect();
+            try
+            {
+                // Simple connectivity test — SELECT 1 against the instance database
+                var cmd = $"mysql -h {mysqlHost} -P {mysqlPort} -u {mysqlUser} " +
+                          $"-p'{mysqlPass}' -e 'SELECT 1' {dbName} 2>&1";
+                var output = RunSshCommand(sshClient, cmd);
+
+                return new HealthCheckResult(HealthStatus.Healthy,
+                    $"MySQL connection to {dbName} successful");
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }
+        catch (Exception ex)
+        {
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"MySQL connection failed for {dbName}: {ex.Message}");
+        }
+    }
+
+    private static async Task<SshConnectionInfo> GetSwarmSshHostAsync(Core.Services.SettingsService settings)
+    {
+        var host = await settings.GetAsync("Ssh.SwarmHost")
+            ?? throw new InvalidOperationException("SSH Swarm host not configured.");
+        var portStr = await settings.GetAsync("Ssh.SwarmPort", "22");
+        var user = await settings.GetAsync("Ssh.SwarmUser", "root");
+        var keyPath = await settings.GetAsync("Ssh.SwarmKeyPath");
+        var password = await settings.GetAsync("Ssh.SwarmPassword");
+        if (!int.TryParse(portStr, out var port)) port = 22;
+        return new SshConnectionInfo(host, port, user, keyPath, password);
+    }
+
+    private static SshClient CreateSshClient(SshConnectionInfo info)
+    {
+        var authMethods = new List<AuthenticationMethod>();
+        if (!string.IsNullOrEmpty(info.KeyPath))
+            authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(info.KeyPath)));
+        if (!string.IsNullOrEmpty(info.Password))
+            authMethods.Add(new PasswordAuthenticationMethod(info.Username, info.Password));
+        if (authMethods.Count == 0)
+        {
+            var defaultKeyPath = Path.Combine(
+                Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".ssh", "id_rsa");
+            if (File.Exists(defaultKeyPath))
+                authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(defaultKeyPath)));
+            else
+                throw new InvalidOperationException($"No SSH auth method for {info.Host}:{info.Port}.");
+        }
+        var connInfo = new Renci.SshNet.ConnectionInfo(info.Host, info.Port, info.Username, authMethods.ToArray());
+        return new SshClient(connInfo);
+    }
+
+    private static string RunSshCommand(SshClient client, string command)
+    {
+        using var cmd = client.RunCommand(command);
+        if (cmd.ExitStatus != 0)
+            throw new InvalidOperationException($"SSH command failed (exit {cmd.ExitStatus}): {cmd.Error}");
+        return cmd.Result;
+    }
+
+    internal sealed record SshConnectionInfo(string Host, int Port, string Username, string? KeyPath, string? Password);
+}

OTSSignsOrchestrator.Server/Health/Checks/NfsAccessHealthCheck.cs

@@ -0,0 +1,121 @@
+using Renci.SshNet;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies NFS paths for the instance are accessible by running <c>ls</c> via SSH.
+/// </summary>
+public sealed class NfsAccessHealthCheck : IHealthCheck
+{
+    private readonly IServiceProvider _services;
+    private readonly ILogger<NfsAccessHealthCheck> _logger;
+
+    public string CheckName => "NfsAccess";
+    public bool AutoRemediate => false;
+
+    public NfsAccessHealthCheck(
+        IServiceProvider services,
+        ILogger<NfsAccessHealthCheck> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var nfsPath = instance.NfsPath;
+        if (string.IsNullOrEmpty(nfsPath))
+            return new HealthCheckResult(HealthStatus.Critical, "No NFS path configured");
+
+        try
+        {
+            var settings = _services.GetRequiredService<Core.Services.SettingsService>();
+            var sshInfo = await GetSwarmSshHostAsync(settings);
+            var nfsServer = await settings.GetAsync(Core.Services.SettingsService.NfsServer);
+            var nfsExport = await settings.GetAsync(Core.Services.SettingsService.NfsExport);
+
+            if (string.IsNullOrEmpty(nfsServer) || string.IsNullOrEmpty(nfsExport))
+                return new HealthCheckResult(HealthStatus.Critical, "NFS server/export not configured");
+
+            using var sshClient = CreateSshClient(sshInfo);
+            sshClient.Connect();
+            try
+            {
+                // Mount temporarily and check the path is listable
+                var mountPoint = $"/tmp/healthcheck-nfs-{Guid.NewGuid():N}";
+                RunSshCommand(sshClient, $"sudo mkdir -p {mountPoint}");
+                try
+                {
+                    RunSshCommand(sshClient, $"sudo mount -t nfs4 {nfsServer}:{nfsExport} {mountPoint}");
+                    var output = RunSshCommand(sshClient, $"ls {mountPoint}/{nfsPath} 2>&1");
+
+                    return new HealthCheckResult(HealthStatus.Healthy,
+                        $"NFS path accessible: {nfsPath}");
+                }
+                finally
+                {
+                    RunSshCommandAllowFailure(sshClient, $"sudo umount {mountPoint}");
+                    RunSshCommandAllowFailure(sshClient, $"sudo rmdir {mountPoint}");
+                }
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }
+        catch (Exception ex)
+        {
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"NFS access check failed for {nfsPath}: {ex.Message}");
+        }
+    }
+
+    private static async Task<SshConnectionInfo> GetSwarmSshHostAsync(Core.Services.SettingsService settings)
+    {
+        var host = await settings.GetAsync("Ssh.SwarmHost")
+            ?? throw new InvalidOperationException("SSH Swarm host not configured.");
+        var portStr = await settings.GetAsync("Ssh.SwarmPort", "22");
+        var user = await settings.GetAsync("Ssh.SwarmUser", "root");
+        var keyPath = await settings.GetAsync("Ssh.SwarmKeyPath");
+        var password = await settings.GetAsync("Ssh.SwarmPassword");
+        if (!int.TryParse(portStr, out var port)) port = 22;
+        return new SshConnectionInfo(host, port, user, keyPath, password);
+    }
+
+    private static SshClient CreateSshClient(SshConnectionInfo info)
+    {
+        var authMethods = new List<AuthenticationMethod>();
+        if (!string.IsNullOrEmpty(info.KeyPath))
+            authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(info.KeyPath)));
+        if (!string.IsNullOrEmpty(info.Password))
+            authMethods.Add(new PasswordAuthenticationMethod(info.Username, info.Password));
+        if (authMethods.Count == 0)
+        {
+            var defaultKeyPath = Path.Combine(
+                Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".ssh", "id_rsa");
+            if (File.Exists(defaultKeyPath))
+                authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(defaultKeyPath)));
+            else
+                throw new InvalidOperationException($"No SSH auth method for {info.Host}:{info.Port}.");
+        }
+        var connInfo = new Renci.SshNet.ConnectionInfo(info.Host, info.Port, info.Username, authMethods.ToArray());
+        return new SshClient(connInfo);
+    }
+
+    private static string RunSshCommand(SshClient client, string command)
+    {
+        using var cmd = client.RunCommand(command);
+        if (cmd.ExitStatus != 0)
+            throw new InvalidOperationException($"SSH command failed (exit {cmd.ExitStatus}): {cmd.Error}");
+        return cmd.Result;
+    }
+
+    private static void RunSshCommandAllowFailure(SshClient client, string command)
+    {
+        using var cmd = client.RunCommand(command);
+        // Intentionally ignore exit code — cleanup operations
+    }
+
+    internal sealed record SshConnectionInfo(string Host, int Port, string Username, string? KeyPath, string? Password);
+}

OTSSignsOrchestrator.Server/Health/Checks/OauthAppAgeHealthCheck.cs

@@ -0,0 +1,50 @@
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Checks the age of the OAuth2 application credentials from <see cref="OauthAppRegistry.CreatedAt"/>.
+/// Alerts Warning at 180 days, Critical at 365 days. AutoRemediate=false — suggests
+/// a "rotate-oauth2" job instead.
+/// </summary>
+public sealed class OauthAppAgeHealthCheck : IHealthCheck
+{
+    /// <summary>Days at which severity escalates to Warning.</summary>
+    internal const int WarningThresholdDays = 180;
+
+    /// <summary>Days at which severity escalates to Critical.</summary>
+    internal const int CriticalThresholdDays = 365;
+
+    public string CheckName => "OauthAppAge";
+    public bool AutoRemediate => false;
+
+    public Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var oauthApp = instance.OauthAppRegistries
+            .OrderByDescending(o => o.CreatedAt)
+            .FirstOrDefault();
+
+        if (oauthApp is null)
+            return Task.FromResult(new HealthCheckResult(HealthStatus.Degraded,
+                "No OAuth app registered"));
+
+        var ageDays = (DateTime.UtcNow - oauthApp.CreatedAt).TotalDays;
+
+        if (ageDays >= CriticalThresholdDays)
+        {
+            return Task.FromResult(new HealthCheckResult(HealthStatus.Critical,
+                $"OAuth2 credentials are {(int)ageDays} days old (critical threshold: {CriticalThresholdDays}d)",
+                "Create a 'rotate-credentials' job to rotate the OAuth2 application"));
+        }
+
+        if (ageDays >= WarningThresholdDays)
+        {
+            return Task.FromResult(new HealthCheckResult(HealthStatus.Degraded,
+                $"OAuth2 credentials are {(int)ageDays} days old (warning threshold: {WarningThresholdDays}d)",
+                "Schedule credential rotation before they reach 365 days"));
+        }
+
+        return Task.FromResult(new HealthCheckResult(HealthStatus.Healthy,
+            $"OAuth2 credentials are {(int)ageDays} days old"));
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/OauthAppHealthCheck.cs

@@ -0,0 +1,59 @@
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies the OAuth2 app in <see cref="OauthAppRegistry"/> can still authenticate
+/// by testing a <c>client_credentials</c> flow against the Xibo CMS instance.
+/// AutoRemediate=false — credential rotation requires a separate job.
+/// </summary>
+public sealed class OauthAppHealthCheck : IHealthCheck
+{
+    private readonly XiboClientFactory _clientFactory;
+    private readonly IServiceProvider _services;
+    private readonly ILogger<OauthAppHealthCheck> _logger;
+
+    public string CheckName => "OauthApp";
+    public bool AutoRemediate => false;
+
+    public OauthAppHealthCheck(
+        XiboClientFactory clientFactory,
+        IServiceProvider services,
+        ILogger<OauthAppHealthCheck> logger)
+    {
+        _clientFactory = clientFactory;
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var oauthApp = instance.OauthAppRegistries.FirstOrDefault();
+        if (oauthApp is null)
+            return new HealthCheckResult(HealthStatus.Critical, "No OAuth app registered");
+
+        var abbrev = instance.Customer.Abbreviation;
+        var settings = _services.GetRequiredService<Core.Services.SettingsService>();
+        var secret = await settings.GetAsync(Core.Services.SettingsService.InstanceOAuthSecretId(abbrev));
+
+        if (string.IsNullOrEmpty(secret))
+            return new HealthCheckResult(HealthStatus.Critical,
+                "OAuth client secret not found in Bitwarden — cannot authenticate");
+
+        try
+        {
+            // Attempt to create a client (which fetches a token via client_credentials)
+            var client = await _clientFactory.CreateAsync(instance.XiboUrl, oauthApp.ClientId, secret);
+
+            // If we got here, the token was obtained successfully
+            return new HealthCheckResult(HealthStatus.Healthy, "OAuth2 client_credentials flow successful");
+        }
+        catch (Exception ex)
+        {
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"OAuth2 authentication failed: {ex.Message}",
+                "Credential rotation job may be required");
+        }
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/StackHealthCheck.cs

@@ -0,0 +1,127 @@
+using Renci.SshNet;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies the Docker stack is healthy by running <c>docker stack ps {stackName}</c>
+/// via SSH and checking that all services report Running state.
+/// </summary>
+public sealed class StackHealthCheck : IHealthCheck
+{
+    private readonly IServiceProvider _services;
+    private readonly ILogger<StackHealthCheck> _logger;
+
+    public string CheckName => "StackHealth";
+    public bool AutoRemediate => false;
+
+    public StackHealthCheck(
+        IServiceProvider services,
+        ILogger<StackHealthCheck> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var stackName = instance.DockerStackName;
+        if (string.IsNullOrEmpty(stackName))
+            return new HealthCheckResult(HealthStatus.Critical, "No Docker stack name configured");
+
+        try
+        {
+            var settings = _services.GetRequiredService<Core.Services.SettingsService>();
+            var sshInfo = await GetSwarmSshHostAsync(settings);
+
+            using var sshClient = CreateSshClient(sshInfo);
+            sshClient.Connect();
+
+            try
+            {
+                // Get task status for all services in the stack
+                var output = RunSshCommand(sshClient,
+                    $"docker stack ps {stackName} --format '{{{{.Name}}}}|{{{{.CurrentState}}}}|{{{{.DesiredState}}}}'");
+
+                var lines = output.Split('\n', StringSplitOptions.RemoveEmptyEntries);
+                var notRunning = new List<string>();
+
+                foreach (var line in lines)
+                {
+                    var parts = line.Split('|');
+                    if (parts.Length < 3) continue;
+
+                    var name = parts[0].Trim();
+                    var currentState = parts[1].Trim();
+                    var desiredState = parts[2].Trim();
+
+                    // Only check tasks whose desired state is Running
+                    if (desiredState.Equals("Running", StringComparison.OrdinalIgnoreCase) &&
+                        !currentState.StartsWith("Running", StringComparison.OrdinalIgnoreCase))
+                    {
+                        notRunning.Add($"{name}: {currentState}");
+                    }
+                }
+
+                if (notRunning.Count == 0)
+                    return new HealthCheckResult(HealthStatus.Healthy,
+                        $"All services in {stackName} are Running");
+
+                return new HealthCheckResult(HealthStatus.Critical,
+                    $"{notRunning.Count} service(s) not running in {stackName}",
+                    string.Join("\n", notRunning));
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }
+        catch (Exception ex)
+        {
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"SSH check failed for {stackName}: {ex.Message}");
+        }
+    }
+
+    private static async Task<SshConnectionInfo> GetSwarmSshHostAsync(Core.Services.SettingsService settings)
+    {
+        var host = await settings.GetAsync("Ssh.SwarmHost")
+            ?? throw new InvalidOperationException("SSH Swarm host not configured.");
+        var portStr = await settings.GetAsync("Ssh.SwarmPort", "22");
+        var user = await settings.GetAsync("Ssh.SwarmUser", "root");
+        var keyPath = await settings.GetAsync("Ssh.SwarmKeyPath");
+        var password = await settings.GetAsync("Ssh.SwarmPassword");
+        if (!int.TryParse(portStr, out var port)) port = 22;
+        return new SshConnectionInfo(host, port, user, keyPath, password);
+    }
+
+    private static SshClient CreateSshClient(SshConnectionInfo info)
+    {
+        var authMethods = new List<AuthenticationMethod>();
+        if (!string.IsNullOrEmpty(info.KeyPath))
+            authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(info.KeyPath)));
+        if (!string.IsNullOrEmpty(info.Password))
+            authMethods.Add(new PasswordAuthenticationMethod(info.Username, info.Password));
+        if (authMethods.Count == 0)
+        {
+            var defaultKeyPath = Path.Combine(
+                Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".ssh", "id_rsa");
+            if (File.Exists(defaultKeyPath))
+                authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(defaultKeyPath)));
+            else
+                throw new InvalidOperationException($"No SSH auth method for {info.Host}:{info.Port}.");
+        }
+        var connInfo = new Renci.SshNet.ConnectionInfo(info.Host, info.Port, info.Username, authMethods.ToArray());
+        return new SshClient(connInfo);
+    }
+
+    private static string RunSshCommand(SshClient client, string command)
+    {
+        using var cmd = client.RunCommand(command);
+        if (cmd.ExitStatus != 0)
+            throw new InvalidOperationException($"SSH command failed (exit {cmd.ExitStatus}): {cmd.Error}");
+        return cmd.Result;
+    }
+
+    internal sealed record SshConnectionInfo(string Host, int Port, string Username, string? KeyPath, string? Password);
+}

OTSSignsOrchestrator.Server/Health/Checks/ThemeHealthCheck.cs

@@ -0,0 +1,145 @@
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies the Xibo CMS theme is set to <c>otssigns</c> by calling <c>GET /api/settings</c>.
+/// Auto-remediates by calling <c>PUT /api/settings</c> if the theme is incorrect.
+/// </summary>
+public sealed class ThemeHealthCheck : IHealthCheck
+{
+    private readonly XiboClientFactory _clientFactory;
+    private readonly IServiceProvider _services;
+    private readonly ILogger<ThemeHealthCheck> _logger;
+
+    private const string ExpectedTheme = "otssigns";
+
+    public string CheckName => "Theme";
+    public bool AutoRemediate => true;
+
+    public ThemeHealthCheck(
+        XiboClientFactory clientFactory,
+        IServiceProvider services,
+        ILogger<ThemeHealthCheck> logger)
+    {
+        _clientFactory = clientFactory;
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var (client, _) = await ResolveAsync(instance);
+        if (client is null)
+            return new HealthCheckResult(HealthStatus.Critical, "No OAuth app — cannot check theme");
+
+        try
+        {
+            var settingsResp = await client.GetSettingsAsync();
+            if (!settingsResp.IsSuccessStatusCode)
+                return new HealthCheckResult(HealthStatus.Critical,
+                    $"GET /settings returned {settingsResp.StatusCode}");
+
+            var settings = settingsResp.Content;
+            if (settings is null)
+                return new HealthCheckResult(HealthStatus.Critical, "Settings response was null");
+
+            // Xibo returns settings as a list of { setting, value } objects or a dictionary
+            var themeName = ExtractSetting(settings, "THEME_NAME");
+            if (string.Equals(themeName, ExpectedTheme, StringComparison.OrdinalIgnoreCase))
+                return new HealthCheckResult(HealthStatus.Healthy, $"Theme is {ExpectedTheme}");
+
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"Theme is '{themeName}', expected '{ExpectedTheme}'");
+        }
+        catch (Exception ex)
+        {
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"Theme check failed: {ex.Message}");
+        }
+    }
+
+    public async Task<bool> RemediateAsync(Instance instance, CancellationToken ct)
+    {
+        var (client, _) = await ResolveAsync(instance);
+        if (client is null) return false;
+
+        try
+        {
+            var resp = await client.UpdateSettingsAsync(
+                new UpdateSettingsRequest(new Dictionary<string, string>
+                {
+                    ["THEME_NAME"] = ExpectedTheme,
+                }));
+
+            if (resp.IsSuccessStatusCode)
+            {
+                await using var scope = _services.CreateAsyncScope();
+                var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+                db.AuditLogs.Add(new AuditLog
+                {
+                    Id = Guid.NewGuid(),
+                    InstanceId = instance.Id,
+                    Actor = "HealthCheckEngine:Theme",
+                    Action = "FixTheme",
+                    Target = instance.Customer.Abbreviation,
+                    Outcome = "Success",
+                    Detail = $"Reset THEME_NAME to {ExpectedTheme}",
+                    OccurredAt = DateTime.UtcNow,
+                });
+                await db.SaveChangesAsync(ct);
+                return true;
+            }
+
+            _logger.LogError("Failed to fix theme: {Err}", resp.Error?.Content);
+            return false;
+        }
+        catch (Exception ex)
+        {
+            _logger.LogError(ex, "Theme remediation failed");
+            return false;
+        }
+    }
+
+    private static string? ExtractSetting(object settingsObj, string key)
+    {
+        // Settings may come back as a dictionary or a list of objects
+        if (settingsObj is System.Text.Json.JsonElement je)
+        {
+            if (je.ValueKind == System.Text.Json.JsonValueKind.Object &&
+                je.TryGetProperty(key, out var val))
+                return val.GetString();
+
+            if (je.ValueKind == System.Text.Json.JsonValueKind.Array)
+            {
+                foreach (var item in je.EnumerateArray())
+                {
+                    if (item.TryGetProperty("setting", out var settingProp) &&
+                        string.Equals(settingProp.GetString(), key, StringComparison.OrdinalIgnoreCase) &&
+                        item.TryGetProperty("value", out var valueProp))
+                    {
+                        return valueProp.GetString();
+                    }
+                }
+            }
+        }
+
+        return null;
+    }
+
+    private async Task<(IXiboApiClient? Client, string Abbrev)> ResolveAsync(Instance instance)
+    {
+        var abbrev = instance.Customer.Abbreviation;
+        var oauthApp = instance.OauthAppRegistries.FirstOrDefault();
+        if (oauthApp is null) return (null, abbrev);
+
+        var settings = _services.GetRequiredService<Core.Services.SettingsService>();
+        var secret = await settings.GetAsync(Core.Services.SettingsService.InstanceOAuthSecretId(abbrev));
+        if (string.IsNullOrEmpty(secret)) return (null, abbrev);
+
+        var client = await _clientFactory.CreateAsync(instance.XiboUrl, oauthApp.ClientId, secret);
+        return (client, abbrev);
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/XiboApiHealthCheck.cs

@@ -0,0 +1,61 @@
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Verifies the Xibo CMS API is reachable by calling GET /about and expecting a 200 response.
+/// </summary>
+public sealed class XiboApiHealthCheck : IHealthCheck
+{
+    private readonly XiboClientFactory _clientFactory;
+    private readonly IServiceProvider _services;
+    private readonly ILogger<XiboApiHealthCheck> _logger;
+
+    public string CheckName => "XiboApi";
+    public bool AutoRemediate => false;
+
+    public XiboApiHealthCheck(
+        XiboClientFactory clientFactory,
+        IServiceProvider services,
+        ILogger<XiboApiHealthCheck> logger)
+    {
+        _clientFactory = clientFactory;
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var client = await ResolveClientAsync(instance);
+        if (client is null)
+            return new HealthCheckResult(HealthStatus.Critical, "No OAuth app registered — cannot reach API");
+
+        try
+        {
+            var response = await client.GetAboutAsync();
+            return response.IsSuccessStatusCode
+                ? new HealthCheckResult(HealthStatus.Healthy, "Xibo API reachable")
+                : new HealthCheckResult(HealthStatus.Critical,
+                    $"Xibo API returned {response.StatusCode}",
+                    response.Error?.Content);
+        }
+        catch (Exception ex)
+        {
+            return new HealthCheckResult(HealthStatus.Critical, $"Xibo API unreachable: {ex.Message}");
+        }
+    }
+
+    private async Task<IXiboApiClient?> ResolveClientAsync(Instance instance)
+    {
+        var oauthApp = instance.OauthAppRegistries.FirstOrDefault();
+        if (oauthApp is null) return null;
+
+        var settings = _services.GetRequiredService<OTSSignsOrchestrator.Core.Services.SettingsService>();
+        var abbrev = instance.Customer.Abbreviation;
+        var secret = await settings.GetAsync(Core.Services.SettingsService.InstanceOAuthSecretId(abbrev));
+        if (string.IsNullOrEmpty(secret)) return null;
+
+        return await _clientFactory.CreateAsync(instance.XiboUrl, oauthApp.ClientId, secret);
+    }
+}

OTSSignsOrchestrator.Server/Health/Checks/XiboVersionHealthCheck.cs

@@ -0,0 +1,87 @@
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using Microsoft.Extensions.Configuration;
+
+namespace OTSSignsOrchestrator.Server.Health.Checks;
+
+/// <summary>
+/// Compares the installed Xibo CMS version (from GET /about) against the latest known
+/// release configured in <c>HealthChecks:LatestXiboVersion</c>. Reports Degraded if behind.
+/// </summary>
+public sealed class XiboVersionHealthCheck : IHealthCheck
+{
+    private readonly XiboClientFactory _clientFactory;
+    private readonly IServiceProvider _services;
+    private readonly IConfiguration _configuration;
+    private readonly ILogger<XiboVersionHealthCheck> _logger;
+
+    public string CheckName => "XiboVersion";
+    public bool AutoRemediate => false;
+
+    public XiboVersionHealthCheck(
+        XiboClientFactory clientFactory,
+        IServiceProvider services,
+        IConfiguration configuration,
+        ILogger<XiboVersionHealthCheck> logger)
+    {
+        _clientFactory = clientFactory;
+        _services = services;
+        _configuration = configuration;
+        _logger = logger;
+    }
+
+    public async Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct)
+    {
+        var latestVersion = _configuration["HealthChecks:LatestXiboVersion"];
+        if (string.IsNullOrEmpty(latestVersion))
+            return new HealthCheckResult(HealthStatus.Healthy, "LatestXiboVersion not configured — skipping");
+
+        var (client, _) = await ResolveAsync(instance);
+        if (client is null)
+            return new HealthCheckResult(HealthStatus.Critical, "No OAuth app — cannot check version");
+
+        try
+        {
+            var response = await client.GetAboutAsync();
+            if (!response.IsSuccessStatusCode || response.Content is null)
+                return new HealthCheckResult(HealthStatus.Critical, "GET /about failed");
+
+            string? installedVersion = null;
+            if (response.Content is System.Text.Json.JsonElement je &&
+                je.TryGetProperty("version", out var verProp))
+            {
+                installedVersion = verProp.GetString();
+            }
+
+            if (string.IsNullOrEmpty(installedVersion))
+                return new HealthCheckResult(HealthStatus.Degraded, "Could not determine installed version");
+
+            if (string.Equals(installedVersion, latestVersion, StringComparison.OrdinalIgnoreCase))
+                return new HealthCheckResult(HealthStatus.Healthy,
+                    $"Xibo version {installedVersion} is current");
+
+            return new HealthCheckResult(HealthStatus.Degraded,
+                $"Xibo version {installedVersion}, latest is {latestVersion}",
+                "Consider scheduling an upgrade");
+        }
+        catch (Exception ex)
+        {
+            return new HealthCheckResult(HealthStatus.Critical,
+                $"Version check failed: {ex.Message}");
+        }
+    }
+
+    private async Task<(IXiboApiClient? Client, string Abbrev)> ResolveAsync(Instance instance)
+    {
+        var abbrev = instance.Customer.Abbreviation;
+        var oauthApp = instance.OauthAppRegistries.FirstOrDefault();
+        if (oauthApp is null) return (null, abbrev);
+
+        var settings = _services.GetRequiredService<Core.Services.SettingsService>();
+        var secret = await settings.GetAsync(Core.Services.SettingsService.InstanceOAuthSecretId(abbrev));
+        if (string.IsNullOrEmpty(secret)) return (null, abbrev);
+
+        var client = await _clientFactory.CreateAsync(instance.XiboUrl, oauthApp.ClientId, secret);
+        return (client, abbrev);
+    }
+}

OTSSignsOrchestrator.Server/Health/HealthCheckEngine.cs

@@ -0,0 +1,289 @@
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using Quartz;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+
+namespace OTSSignsOrchestrator.Server.Health;
+
+/// <summary>
+/// Background service that schedules and runs all <see cref="IHealthCheck"/> implementations
+/// against every active <see cref="Instance"/>. Persists <see cref="HealthEvent"/> rows,
+/// aggregates worst-severity to update <see cref="Instance.HealthStatus"/>,
+/// broadcasts changes via <see cref="FleetHub"/>, and triggers auto-remediation when applicable.
+///
+/// Uses Quartz to stagger per-instance jobs across the check interval (avoids thundering herd).
+/// Concurrency is capped at 4 simultaneous check runs via <see cref="SemaphoreSlim"/>.
+/// </summary>
+public sealed class HealthCheckEngine : BackgroundService
+{
+    private readonly IServiceProvider _services;
+    private readonly ISchedulerFactory _schedulerFactory;
+    private readonly ILogger<HealthCheckEngine> _logger;
+
+    /// <summary>Default interval between full health-check sweeps.</summary>
+    internal static readonly TimeSpan DefaultCheckInterval = TimeSpan.FromMinutes(5);
+
+    public HealthCheckEngine(
+        IServiceProvider services,
+        ISchedulerFactory schedulerFactory,
+        ILogger<HealthCheckEngine> logger)
+    {
+        _services = services;
+        _schedulerFactory = schedulerFactory;
+        _logger = logger;
+    }
+
+    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
+    {
+        // Wait briefly for the rest of the app to start
+        await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
+
+        var scheduler = await _schedulerFactory.GetScheduler(stoppingToken);
+
+        while (!stoppingToken.IsCancellationRequested)
+        {
+            try
+            {
+                await ScheduleInstanceChecks(scheduler, stoppingToken);
+            }
+            catch (Exception ex) when (ex is not OperationCanceledException)
+            {
+                _logger.LogError(ex, "Error scheduling health check sweep");
+            }
+
+            await Task.Delay(DefaultCheckInterval, stoppingToken);
+        }
+    }
+
+    /// <summary>
+    /// Load all active instances and schedule staggered Quartz jobs so that
+    /// check start times are spread across the interval.
+    /// </summary>
+    private async Task ScheduleInstanceChecks(IScheduler scheduler, CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+
+        var instances = await db.Instances
+            .AsNoTracking()
+            .Include(i => i.Customer)
+            .Where(i => i.Customer.Status == CustomerStatus.Active)
+            .ToListAsync(ct);
+
+        if (instances.Count == 0)
+            return;
+
+        // Spread jobs across 80 % of the check interval to leave a buffer
+        var spreadMs = (int)(DefaultCheckInterval.TotalMilliseconds * 0.8);
+        var stepMs = instances.Count > 1 ? spreadMs / (instances.Count - 1) : 0;
+
+        for (var i = 0; i < instances.Count; i++)
+        {
+            var instance = instances[i];
+            var delay = TimeSpan.FromMilliseconds(stepMs * i);
+
+            var jobKey = new JobKey($"health-{instance.Id}", "health-checks");
+
+            // Remove previous trigger if it still exists (idempotent reschedule)
+            if (await scheduler.CheckExists(jobKey, ct))
+                await scheduler.DeleteJob(jobKey, ct);
+
+            var job = JobBuilder.Create<InstanceHealthCheckJob>()
+                .WithIdentity(jobKey)
+                .UsingJobData("instanceId", instance.Id.ToString())
+                .Build();
+
+            var trigger = TriggerBuilder.Create()
+                .WithIdentity($"health-{instance.Id}-trigger", "health-checks")
+                .StartAt(DateTimeOffset.UtcNow.Add(delay))
+                .Build();
+
+            await scheduler.ScheduleJob(job, trigger, ct);
+        }
+
+        _logger.LogInformation(
+            "Scheduled health checks for {Count} active instance(s)", instances.Count);
+    }
+}
+
+/// <summary>
+/// Quartz job that executes all <see cref="IHealthCheck"/> implementations for a single instance.
+/// </summary>
+[DisallowConcurrentExecution]
+public sealed class InstanceHealthCheckJob : IJob
+{
+    /// <summary>Global concurrency limiter — max 4 parallel health check runs.</summary>
+    private static readonly SemaphoreSlim s_concurrency = new(4);
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<InstanceHealthCheckJob> _logger;
+
+    public InstanceHealthCheckJob(
+        IServiceProvider services,
+        ILogger<InstanceHealthCheckJob> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task Execute(IJobExecutionContext context)
+    {
+        var instanceIdStr = context.MergedJobDataMap.GetString("instanceId");
+        if (!Guid.TryParse(instanceIdStr, out var instanceId))
+        {
+            _logger.LogWarning("InstanceHealthCheckJob: invalid instanceId {Id}", instanceIdStr);
+            return;
+        }
+
+        await s_concurrency.WaitAsync(context.CancellationToken);
+        try
+        {
+            await RunChecksForInstanceAsync(instanceId, context.CancellationToken);
+        }
+        finally
+        {
+            s_concurrency.Release();
+        }
+    }
+
+    private async Task RunChecksForInstanceAsync(Guid instanceId, CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+        var checks = scope.ServiceProvider.GetServices<IHealthCheck>();
+
+        var instance = await db.Instances
+            .Include(i => i.Customer)
+            .Include(i => i.OauthAppRegistries)
+            .Include(i => i.ByoiConfigs)
+            .FirstOrDefaultAsync(i => i.Id == instanceId, ct);
+
+        if (instance is null)
+        {
+            _logger.LogWarning("InstanceHealthCheckJob: instance {Id} not found", instanceId);
+            return;
+        }
+
+        var abbrev = instance.Customer.Abbreviation;
+        var worstStatus = HealthStatus.Healthy;
+
+        foreach (var check in checks)
+        {
+            // Skip the AuthentikGlobalHealthCheck — it runs on its own schedule
+            if (check.CheckName == "AuthentikGlobal")
+                continue;
+
+            HealthCheckResult result;
+            try
+            {
+                result = await check.RunAsync(instance, ct);
+            }
+            catch (Exception ex)
+            {
+                _logger.LogError(ex, "Health check {Check} failed for {Abbrev}", check.CheckName, abbrev);
+                result = new HealthCheckResult(HealthStatus.Critical, $"Check threw exception: {ex.Message}");
+            }
+
+            // Persist HealthEvent
+            var healthEvent = new HealthEvent
+            {
+                Id = Guid.NewGuid(),
+                InstanceId = instanceId,
+                CheckName = check.CheckName,
+                Status = ToEventStatus(result.Status),
+                Message = result.Message,
+                Remediated = false,
+                OccurredAt = DateTime.UtcNow,
+            };
+
+            // Auto-remediation
+            if (check.AutoRemediate && result.Status == HealthStatus.Critical)
+            {
+                try
+                {
+                    var fixed_ = await check.RemediateAsync(instance, ct);
+                    healthEvent.Remediated = fixed_;
+
+                    // Append-only audit log
+                    db.AuditLogs.Add(new AuditLog
+                    {
+                        Id = Guid.NewGuid(),
+                        InstanceId = instanceId,
+                        Actor = $"HealthCheckEngine:{check.CheckName}",
+                        Action = "AutoRemediate",
+                        Target = abbrev,
+                        Outcome = fixed_ ? "Success" : "Failed",
+                        Detail = result.Detail,
+                        OccurredAt = DateTime.UtcNow,
+                    });
+
+                    if (fixed_)
+                    {
+                        _logger.LogInformation(
+                            "Auto-remediated {Check} for {Abbrev}", check.CheckName, abbrev);
+                        // Downgrade severity since we fixed it
+                        healthEvent.Status = HealthEventStatus.Healthy;
+                    }
+                }
+                catch (Exception ex)
+                {
+                    _logger.LogError(ex, "Remediation for {Check} failed on {Abbrev}", check.CheckName, abbrev);
+                    db.AuditLogs.Add(new AuditLog
+                    {
+                        Id = Guid.NewGuid(),
+                        InstanceId = instanceId,
+                        Actor = $"HealthCheckEngine:{check.CheckName}",
+                        Action = "AutoRemediate",
+                        Target = abbrev,
+                        Outcome = "Error",
+                        Detail = ex.Message,
+                        OccurredAt = DateTime.UtcNow,
+                    });
+                }
+            }
+
+            db.HealthEvents.Add(healthEvent);
+
+            // Track worst severity (only from non-remediated results)
+            if (!healthEvent.Remediated)
+            {
+                var status = FromEventStatus(healthEvent.Status);
+                if (status > worstStatus)
+                    worstStatus = status;
+            }
+        }
+
+        // Update instance health status
+        var previousStatus = instance.HealthStatus;
+        instance.HealthStatus = worstStatus;
+        instance.LastHealthCheck = DateTime.UtcNow;
+
+        await db.SaveChangesAsync(ct);
+
+        // Broadcast status change
+        if (previousStatus != worstStatus)
+        {
+            await hub.Clients.All.SendInstanceStatusChanged(
+                instance.CustomerId.ToString(), worstStatus.ToString());
+        }
+    }
+
+    private static HealthEventStatus ToEventStatus(HealthStatus status) => status switch
+    {
+        HealthStatus.Healthy => HealthEventStatus.Healthy,
+        HealthStatus.Degraded => HealthEventStatus.Degraded,
+        HealthStatus.Critical => HealthEventStatus.Critical,
+        _ => HealthEventStatus.Critical,
+    };
+
+    private static HealthStatus FromEventStatus(HealthEventStatus status) => status switch
+    {
+        HealthEventStatus.Healthy => HealthStatus.Healthy,
+        HealthEventStatus.Degraded => HealthStatus.Degraded,
+        HealthEventStatus.Critical => HealthStatus.Critical,
+        _ => HealthStatus.Critical,
+    };
+}

OTSSignsOrchestrator.Server/Health/IHealthCheck.cs

@@ -0,0 +1,32 @@
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Health;
+
+/// <summary>
+/// Result of a single health check execution.
+/// </summary>
+public record HealthCheckResult(HealthStatus Status, string Message, string? Detail = null);
+
+/// <summary>
+/// Contract for an individual health check that runs against a specific <see cref="Instance"/>.
+/// </summary>
+public interface IHealthCheck
+{
+    /// <summary>Human-readable name written to <see cref="HealthEvent.CheckName"/>.</summary>
+    string CheckName { get; }
+
+    /// <summary>
+    /// When true the engine will automatically call <see cref="RemediateAsync"/>
+    /// if the check returns <see cref="HealthStatus.Critical"/>.
+    /// </summary>
+    bool AutoRemediate { get; }
+
+    /// <summary>Execute the check for <paramref name="instance"/>.</summary>
+    Task<HealthCheckResult> RunAsync(Instance instance, CancellationToken ct);
+
+    /// <summary>
+    /// Attempt automatic remediation. Return true if the issue was fixed.
+    /// The default implementation does nothing and returns false.
+    /// </summary>
+    Task<bool> RemediateAsync(Instance instance, CancellationToken ct) => Task.FromResult(false);
+}

OTSSignsOrchestrator.Server/Hubs/FleetHub.cs

@@ -0,0 +1,49 @@
+using System.Security.Claims;
+using Microsoft.AspNetCore.Authorization;
+using Microsoft.AspNetCore.SignalR;
+
+namespace OTSSignsOrchestrator.Server.Hubs;
+
+/// <summary>
+/// Server→client push-only hub for real-time fleet notifications.
+/// Desktop clients never send messages via SignalR — they use REST for commands.
+/// </summary>
+[Authorize]
+public class FleetHub : Hub<IFleetClient>
+{
+    private readonly ILogger<FleetHub> _logger;
+
+    public FleetHub(ILogger<FleetHub> logger)
+    {
+        _logger = logger;
+    }
+
+    public override Task OnConnectedAsync()
+    {
+        var name = Context.User?.FindFirst(ClaimTypes.Name)?.Value ?? "unknown";
+        _logger.LogInformation("FleetHub: operator {Name} connected (connId={ConnectionId})",
+            name, Context.ConnectionId);
+        return base.OnConnectedAsync();
+    }
+
+    public override Task OnDisconnectedAsync(Exception? exception)
+    {
+        var name = Context.User?.FindFirst(ClaimTypes.Name)?.Value ?? "unknown";
+        _logger.LogInformation("FleetHub: operator {Name} disconnected (connId={ConnectionId})",
+            name, Context.ConnectionId);
+        return base.OnDisconnectedAsync(exception);
+    }
+}
+
+/// <summary>
+/// Strongly-typed client interface for FleetHub push messages.
+/// Inject IHubContext&lt;FleetHub, IFleetClient&gt; to call these from services.
+/// </summary>
+public interface IFleetClient
+{
+    Task SendJobCreated(string jobId, string abbrev, string jobType);
+    Task SendJobProgressUpdate(string jobId, string stepName, int pct, string logLine);
+    Task SendJobCompleted(string jobId, bool success, string summary);
+    Task SendInstanceStatusChanged(string customerId, string status);
+    Task SendAlertRaised(string severity, string message);
+}

OTSSignsOrchestrator.Server/Jobs/ByoiCertExpiryJob.cs

@@ -0,0 +1,106 @@
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using Quartz;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+
+namespace OTSSignsOrchestrator.Server.Jobs;
+
+/// <summary>
+/// Quartz job that runs daily to check BYOI certificate expiry dates across all enabled
+/// ByoiConfig entries. Alerts at 60, 30, and 7 day thresholds via FleetHub and logs to AuditLog.
+///
+/// Severity escalation:
+///   - &gt; 7 days remaining  → "Warning"
+///   - ≤ 7 days remaining  → "Critical"
+/// </summary>
+// IMMUTABLE AuditLog — this job only appends, never updates or deletes audit records.
+[DisallowConcurrentExecution]
+public sealed class ByoiCertExpiryJob : IJob
+{
+    /// <summary>Alert thresholds in days. Alerts fire when remaining days ≤ threshold.</summary>
+    internal static readonly int[] AlertThresholdDays = [60, 30, 7];
+
+    /// <summary>Days at or below which severity escalates to "Critical".</summary>
+    internal const int CriticalThresholdDays = 7;
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<ByoiCertExpiryJob> _logger;
+
+    public ByoiCertExpiryJob(
+        IServiceProvider services,
+        ILogger<ByoiCertExpiryJob> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task Execute(IJobExecutionContext context)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+
+        var configs = await db.ByoiConfigs
+            .AsNoTracking()
+            .Include(b => b.Instance)
+                .ThenInclude(i => i.Customer)
+            .Where(b => b.Enabled)
+            .ToListAsync(context.CancellationToken);
+
+        foreach (var config in configs)
+        {
+            var daysRemaining = (config.CertExpiry - DateTime.UtcNow).TotalDays;
+            var abbrev = config.Instance.Customer.Abbreviation;
+
+            if (!ShouldAlert(daysRemaining))
+                continue;
+
+            var severity = GetSeverity(daysRemaining);
+            var daysInt = (int)Math.Floor(daysRemaining);
+            var message = daysRemaining <= 0
+                ? $"BYOI cert for {abbrev} has EXPIRED."
+                : $"BYOI cert for {abbrev} expires in {daysInt} days.";
+
+            _logger.LogWarning("BYOI cert expiry alert: {Severity} — {Message}", severity, message);
+
+            await hub.Clients.All.SendAlertRaised(severity, message);
+
+            // Append-only audit log
+            db.AuditLogs.Add(new AuditLog
+            {
+                Id = Guid.NewGuid(),
+                InstanceId = config.InstanceId,
+                Actor = "ByoiCertExpiryJob",
+                Action = "CertExpiryAlert",
+                Target = config.Slug,
+                Outcome = severity,
+                Detail = message,
+                OccurredAt = DateTime.UtcNow,
+            });
+        }
+
+        await db.SaveChangesAsync(context.CancellationToken);
+    }
+
+    /// <summary>
+    /// Determines whether an alert should fire based on remaining days.
+    /// Alerts at ≤ 60, ≤ 30, ≤ 7 days (or already expired).
+    /// </summary>
+    internal static bool ShouldAlert(double daysRemaining)
+    {
+        foreach (var threshold in AlertThresholdDays)
+        {
+            if (daysRemaining <= threshold)
+                return true;
+        }
+        return false;
+    }
+
+    /// <summary>
+    /// Returns "Critical" when ≤ 7 days remain, otherwise "Warning".
+    /// </summary>
+    internal static string GetSeverity(double daysRemaining) =>
+        daysRemaining <= CriticalThresholdDays ? "Critical" : "Warning";
+}

OTSSignsOrchestrator.Server/Jobs/DailySnapshotJob.cs

@@ -0,0 +1,118 @@
+using Microsoft.EntityFrameworkCore;
+using Quartz;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Jobs;
+
+/// <summary>
+/// Quartz job scheduled at 2 AM UTC daily (<c>0 0 2 * * ?</c>).
+/// For each active <see cref="Instance"/>: calls <see cref="XiboApiClientExtensions.GetAllPagesAsync{T}"/>
+/// with <c>authorised=1</c> to count authorised displays; inserts a <see cref="ScreenSnapshot"/> row.
+///
+/// Uses ON CONFLICT DO NOTHING semantics to protect against double-runs.
+/// THIS DATA CANNOT BE RECOVERED — if the job misses a day, that data is permanently lost.
+/// </summary>
+[DisallowConcurrentExecution]
+public sealed class DailySnapshotJob : IJob
+{
+    private readonly IServiceProvider _services;
+    private readonly ILogger<DailySnapshotJob> _logger;
+
+    public DailySnapshotJob(
+        IServiceProvider services,
+        ILogger<DailySnapshotJob> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task Execute(IJobExecutionContext context)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var clientFactory = scope.ServiceProvider.GetRequiredService<XiboClientFactory>();
+        var settings = scope.ServiceProvider.GetRequiredService<Core.Services.SettingsService>();
+
+        var today = DateOnly.FromDateTime(DateTime.UtcNow);
+
+        var instances = await db.Instances
+            .Include(i => i.Customer)
+            .Include(i => i.OauthAppRegistries)
+            .Where(i => i.Customer.Status == CustomerStatus.Active)
+            .ToListAsync(context.CancellationToken);
+
+        _logger.LogInformation("DailySnapshotJob: processing {Count} active instance(s) for {Date}",
+            instances.Count, today);
+
+        foreach (var instance in instances)
+        {
+            var abbrev = instance.Customer.Abbreviation;
+            try
+            {
+                var oauthApp = instance.OauthAppRegistries.FirstOrDefault();
+                if (oauthApp is null)
+                {
+                    _logger.LogWarning(
+                        "DailySnapshotJob: skipping {Abbrev} — no OAuth app registered", abbrev);
+                    continue;
+                }
+
+                var secret = await settings.GetAsync(
+                    Core.Services.SettingsService.InstanceOAuthSecretId(abbrev));
+                if (string.IsNullOrEmpty(secret))
+                {
+                    _logger.LogWarning(
+                        "DailySnapshotJob: skipping {Abbrev} — OAuth secret not found", abbrev);
+                    continue;
+                }
+
+                var client = await clientFactory.CreateAsync(
+                    instance.XiboUrl, oauthApp.ClientId, secret);
+
+                var displays = await client.GetAllPagesAsync(
+                    (start, length) => client.GetDisplaysAsync(start, length, authorised: 1));
+
+                var screenCount = displays.Count;
+
+                // ON CONFLICT DO NOTHING — protect against double-runs.
+                // Check if a snapshot already exists for this instance + date.
+                var exists = await db.ScreenSnapshots.AnyAsync(
+                    s => s.InstanceId == instance.Id && s.SnapshotDate == today,
+                    context.CancellationToken);
+
+                if (!exists)
+                {
+                    db.ScreenSnapshots.Add(new ScreenSnapshot
+                    {
+                        Id = Guid.NewGuid(),
+                        InstanceId = instance.Id,
+                        SnapshotDate = today,
+                        ScreenCount = screenCount,
+                        CreatedAt = DateTime.UtcNow,
+                    });
+
+                    _logger.LogInformation(
+                        "DailySnapshotJob: {Abbrev} — {Count} authorised display(s)",
+                        abbrev, screenCount);
+                }
+                else
+                {
+                    _logger.LogInformation(
+                        "DailySnapshotJob: {Abbrev} — snapshot already exists for {Date}, skipping",
+                        abbrev, today);
+                }
+            }
+            catch (Exception ex)
+            {
+                // THIS DATA CANNOT BE RECOVERED — log prominently
+                _logger.LogWarning(ex,
+                    "DailySnapshotJob: FAILED to capture snapshot for {Abbrev} on {Date}. " +
+                    "This data is permanently lost.", abbrev, today);
+            }
+        }
+
+        await db.SaveChangesAsync(context.CancellationToken);
+    }
+}

OTSSignsOrchestrator.Server/Jobs/ScheduledReportJob.cs

@@ -0,0 +1,175 @@
+using Microsoft.EntityFrameworkCore;
+using Quartz;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Reports;
+using OTSSignsOrchestrator.Server.Services;
+
+namespace OTSSignsOrchestrator.Server.Jobs;
+
+/// <summary>
+/// Quartz job with two triggers:
+///   - Weekly (Monday 08:00 UTC): fleet health PDF → operator email list
+///   - Monthly (1st of month 08:00 UTC): billing CSV + fleet health PDF → operators;
+///     per-customer usage PDF → each active customer's admin email
+/// </summary>
+[DisallowConcurrentExecution]
+public sealed class ScheduledReportJob : IJob
+{
+    /// <summary>Quartz job data key indicating whether this is a monthly trigger.</summary>
+    public const string IsMonthlyKey = "IsMonthly";
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<ScheduledReportJob> _logger;
+
+    public ScheduledReportJob(
+        IServiceProvider services,
+        ILogger<ScheduledReportJob> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task Execute(IJobExecutionContext context)
+    {
+        var isMonthly = context.MergedJobDataMap.GetBoolean(IsMonthlyKey);
+
+        _logger.LogInformation("ScheduledReportJob fired — isMonthly={IsMonthly}", isMonthly);
+
+        await using var scope = _services.CreateAsyncScope();
+        var billing = scope.ServiceProvider.GetRequiredService<BillingReportService>();
+        var pdfService = scope.ServiceProvider.GetRequiredService<FleetHealthPdfService>();
+        var emailService = scope.ServiceProvider.GetRequiredService<EmailService>();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+
+        // Get operator email list (admin operators)
+        var operatorEmails = await db.Operators
+            .AsNoTracking()
+            .Where(o => o.Role == OperatorRole.Admin)
+            .Select(o => o.Email)
+            .ToListAsync(context.CancellationToken);
+
+        if (operatorEmails.Count == 0)
+        {
+            _logger.LogWarning("No admin operators found — skipping report email dispatch");
+            return;
+        }
+
+        // ── Weekly: fleet health PDF for past 7 days ───────────────────────
+        var to = DateOnly.FromDateTime(DateTime.UtcNow);
+        var weekFrom = to.AddDays(-7);
+
+        byte[] fleetPdf;
+        try
+        {
+            fleetPdf = await pdfService.GenerateFleetHealthPdfAsync(weekFrom, to);
+        }
+        catch (Exception ex)
+        {
+            _logger.LogError(ex, "Failed to generate weekly fleet health PDF");
+            return;
+        }
+
+        foreach (var email in operatorEmails)
+        {
+            var success = await emailService.SendReportEmailAsync(
+                email,
+                $"fleet-health-{weekFrom:yyyy-MM-dd}-{to:yyyy-MM-dd}.pdf",
+                fleetPdf,
+                "application/pdf");
+
+            if (!success)
+                _logger.LogError("Failed to send weekly fleet health PDF to {Email}", email);
+        }
+
+        // ── Monthly: billing CSV + per-customer usage PDFs ─────────────────
+        if (!isMonthly) return;
+
+        var monthStart = new DateOnly(to.Year, to.Month, 1).AddMonths(-1);
+        var monthEnd = monthStart.AddMonths(1).AddDays(-1);
+
+        // Billing CSV
+        byte[]? billingCsv = null;
+        try
+        {
+            billingCsv = await billing.GenerateBillingCsvAsync(monthStart, monthEnd);
+        }
+        catch (Exception ex)
+        {
+            _logger.LogError(ex, "Failed to generate monthly billing CSV");
+        }
+
+        // Monthly fleet health PDF
+        byte[]? monthlyFleetPdf = null;
+        try
+        {
+            monthlyFleetPdf = await pdfService.GenerateFleetHealthPdfAsync(monthStart, monthEnd);
+        }
+        catch (Exception ex)
+        {
+            _logger.LogError(ex, "Failed to generate monthly fleet health PDF");
+        }
+
+        // Send monthly reports to operators
+        foreach (var email in operatorEmails)
+        {
+            if (billingCsv is not null)
+            {
+                var ok = await emailService.SendReportEmailAsync(
+                    email,
+                    $"billing-{monthStart:yyyy-MM-dd}-{monthEnd:yyyy-MM-dd}.csv",
+                    billingCsv,
+                    "text/csv");
+                if (!ok)
+                    _logger.LogError("Failed to send monthly billing CSV to {Email}", email);
+            }
+
+            if (monthlyFleetPdf is not null)
+            {
+                var ok = await emailService.SendReportEmailAsync(
+                    email,
+                    $"fleet-health-{monthStart:yyyy-MM-dd}-{monthEnd:yyyy-MM-dd}.pdf",
+                    monthlyFleetPdf,
+                    "application/pdf");
+                if (!ok)
+                    _logger.LogError("Failed to send monthly fleet health PDF to {Email}", email);
+            }
+        }
+
+        // Per-customer usage PDFs → customer admin emails
+        var activeCustomers = await db.Customers
+            .AsNoTracking()
+            .Where(c => c.Status == CustomerStatus.Active)
+            .ToListAsync(context.CancellationToken);
+
+        foreach (var customer in activeCustomers)
+        {
+            try
+            {
+                var usagePdf = await pdfService.GenerateCustomerUsagePdfAsync(
+                    customer.Id, monthStart, monthEnd);
+
+                var ok = await emailService.SendReportEmailAsync(
+                    customer.AdminEmail,
+                    $"usage-{customer.Abbreviation}-{monthStart:yyyy-MM-dd}-{monthEnd:yyyy-MM-dd}.pdf",
+                    usagePdf,
+                    "application/pdf");
+
+                if (!ok)
+                    _logger.LogError("Failed to send usage PDF to {Email} for customer {Abbrev}",
+                        customer.AdminEmail, customer.Abbreviation);
+                else
+                    _logger.LogInformation("Sent usage PDF to {Email} for customer {Abbrev}",
+                        customer.AdminEmail, customer.Abbreviation);
+            }
+            catch (Exception ex)
+            {
+                _logger.LogError(ex, "Failed to generate/send usage PDF for customer {Abbrev}",
+                    customer.Abbreviation);
+            }
+        }
+
+        _logger.LogInformation("Monthly report dispatch complete — {CustomerCount} customer reports processed",
+            activeCustomers.Count);
+    }
+}

OTSSignsOrchestrator.Server/OTSSignsOrchestrator.Server.csproj

@@ -0,0 +1,38 @@
+<Project Sdk="Microsoft.NET.Sdk.Web">
+
+  <PropertyGroup>
+    <TargetFramework>net9.0</TargetFramework>
+    <Nullable>enable</Nullable>
+    <ImplicitUsings>enable</ImplicitUsings>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <InternalsVisibleTo Include="OTSSignsOrchestrator.Server.Tests" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <PackageReference Include="BCrypt.Net-Next" Version="4.0.3" />
+    <PackageReference Include="CsvHelper" Version="33.1.0" />
+    <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="9.0.2" />
+    <PackageReference Include="Microsoft.AspNetCore.SignalR" Version="1.1.0" />
+    <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="9.0.2">
+      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
+      <PrivateAssets>all</PrivateAssets>
+    </PackageReference>
+    <PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="9.0.4" />
+    <PackageReference Include="PdfSharpCore" Version="1.3.67" />
+    <PackageReference Include="SendGrid" Version="9.29.3" />
+    <PackageReference Include="SSH.NET" Version="2025.1.0" />
+    <PackageReference Include="Stripe.net" Version="47.4.0" />
+    <PackageReference Include="Serilog.AspNetCore" Version="9.0.0" />
+    <PackageReference Include="Quartz.AspNetCore" Version="3.13.1" />
+    <PackageReference Include="Refit" Version="8.0.0" />
+    <PackageReference Include="Refit.HttpClientFactory" Version="8.0.0" />
+    <PackageReference Include="Polly" Version="8.5.2" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <ProjectReference Include="..\OTSSignsOrchestrator.Core\OTSSignsOrchestrator.Core.csproj" />
+  </ItemGroup>
+
+</Project>

OTSSignsOrchestrator.Server/Program.cs

@@ -0,0 +1,257 @@
+using System.Text;
+using System.Threading.RateLimiting;
+using Microsoft.AspNetCore.Authentication.JwtBearer;
+using Microsoft.AspNetCore.RateLimiting;
+using Microsoft.EntityFrameworkCore;
+using Microsoft.IdentityModel.Tokens;
+using Microsoft.Extensions.Options;
+using OTSSignsOrchestrator.Server.Api;
+using OTSSignsOrchestrator.Server.Auth;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Hubs;
+using OTSSignsOrchestrator.Server.Reports;
+using OTSSignsOrchestrator.Server.Services;
+using OTSSignsOrchestrator.Server.Webhooks;
+using OTSSignsOrchestrator.Server.Workers;
+using Refit;
+using Quartz;
+using OTSSignsOrchestrator.Server.Jobs;
+using OTSSignsOrchestrator.Server.Health;
+using OTSSignsOrchestrator.Server.Health.Checks;
+using Serilog;
+using Stripe;
+
+var builder = WebApplication.CreateBuilder(args);
+
+// ── Serilog ──────────────────────────────────────────────────────────────────
+builder.Host.UseSerilog((context, config) =>
+    config.ReadFrom.Configuration(context.Configuration));
+
+// ── EF Core — PostgreSQL ─────────────────────────────────────────────────────
+builder.Services.AddDbContext<OrchestratorDbContext>(options =>
+    options.UseNpgsql(builder.Configuration.GetConnectionString("OrchestratorDb")));
+
+// ── JWT Authentication ──────────────────────────────────────────────────────
+builder.Services.Configure<JwtOptions>(builder.Configuration.GetSection(JwtOptions.Section));
+var jwtKey = builder.Configuration[$"{JwtOptions.Section}:Key"]
+    ?? throw new InvalidOperationException("Jwt:Key must be configured.");
+
+builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
+    .AddJwtBearer(options =>
+    {
+        options.TokenValidationParameters = new TokenValidationParameters
+        {
+            ValidateIssuer = true,
+            ValidateAudience = true,
+            ValidateLifetime = true,
+            ValidateIssuerSigningKey = true,
+            ValidIssuer = builder.Configuration[$"{JwtOptions.Section}:Issuer"] ?? "OTSSignsOrchestrator",
+            ValidAudience = builder.Configuration[$"{JwtOptions.Section}:Audience"] ?? "OTSSignsOrchestrator",
+            IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(jwtKey)),
+            ClockSkew = TimeSpan.FromSeconds(30),
+        };
+
+        // Allow SignalR to receive the JWT via query string
+        options.Events = new JwtBearerEvents
+        {
+            OnMessageReceived = context =>
+            {
+                var accessToken = context.Request.Query["access_token"];
+                var path = context.HttpContext.Request.Path;
+                if (!string.IsNullOrEmpty(accessToken) && path.StartsWithSegments("/hubs"))
+                    context.Token = accessToken;
+                return Task.CompletedTask;
+            },
+        };
+    });
+
+builder.Services.AddAuthorization(options =>
+{
+    options.AddPolicy("CustomerPortal", policy =>
+        policy.RequireClaim("customer_id"));
+});
+
+// ── Application services ────────────────────────────────────────────────────
+builder.Services.AddScoped<OperatorAuthService>();
+builder.Services.AddScoped<AbbreviationService>();
+builder.Services.Configure<EmailOptions>(builder.Configuration.GetSection(EmailOptions.Section));
+builder.Services.AddSingleton<EmailService>();
+
+// ── Report services ─────────────────────────────────────────────────────────
+builder.Services.AddScoped<BillingReportService>();
+builder.Services.AddScoped<FleetHealthPdfService>();
+
+// ── Provisioning pipelines + worker ─────────────────────────────────────────
+builder.Services.AddScoped<IProvisioningPipeline, Phase1Pipeline>();
+builder.Services.AddScoped<IProvisioningPipeline, Phase2Pipeline>();
+builder.Services.AddScoped<IProvisioningPipeline, ByoiSamlPipeline>();
+builder.Services.AddScoped<IProvisioningPipeline, SuspendPipeline>();
+builder.Services.AddScoped<IProvisioningPipeline, ReactivatePipeline>();
+builder.Services.AddScoped<IProvisioningPipeline, UpdateScreenLimitPipeline>();
+builder.Services.AddScoped<IProvisioningPipeline, DecommissionPipeline>();
+builder.Services.AddScoped<IProvisioningPipeline, RotateCredentialsPipeline>();
+builder.Services.AddHostedService<ProvisioningWorker>();
+
+// ── External API clients ────────────────────────────────────────────────────
+builder.Services.AddHttpClient();
+builder.Services.AddSingleton<XiboClientFactory>();
+
+builder.Services.Configure<AuthentikOptions>(
+    builder.Configuration.GetSection(AuthentikOptions.Section));
+
+builder.Services.AddRefitClient<IAuthentikClient>()
+    .ConfigureHttpClient((sp, client) =>
+    {
+        var opts = sp.GetRequiredService<IOptions<AuthentikOptions>>().Value;
+        client.BaseAddress = new Uri(opts.BaseUrl.TrimEnd('/'));
+        client.DefaultRequestHeaders.Authorization =
+            new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", opts.ApiToken);
+    });
+
+// ── Stripe ──────────────────────────────────────────────────────────────────
+StripeConfiguration.ApiKey = builder.Configuration["Stripe:SecretKey"];
+
+// ── Health check engine + individual checks ─────────────────────────────────
+builder.Services.AddHostedService<HealthCheckEngine>();
+builder.Services.AddScoped<IHealthCheck, XiboApiHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, AdminIntegrityHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, GroupStructureHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, OauthAppHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, DisplayAuthorisedHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, StackHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, MySqlConnectHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, NfsAccessHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, ThemeHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, XiboVersionHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, OauthAppAgeHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, ByoiCertExpiryHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, AuthentikGlobalHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, AuthentikSamlProviderHealthCheck>();
+builder.Services.AddScoped<IHealthCheck, InvitationFlowHealthCheck>();
+// AuthentikGlobalHealthCheck also registered as concrete type for the Quartz job
+builder.Services.AddScoped<AuthentikGlobalHealthCheck>();
+
+// ── Quartz scheduler ─────────────────────────────────────────────────────────
+builder.Services.AddQuartz(q =>
+{
+    var certExpiryKey = new JobKey("byoi-cert-expiry-global", "byoi-cert-expiry");
+    q.AddJob<ByoiCertExpiryJob>(opts => opts.WithIdentity(certExpiryKey).StoreDurably());
+    q.AddTrigger(opts => opts
+        .ForJob(certExpiryKey)
+        .WithIdentity("byoi-cert-expiry-global-trigger", "byoi-cert-expiry")
+        .WithSimpleSchedule(s => s
+            .WithIntervalInHours(24)
+            .RepeatForever())
+        .StartNow());
+
+    // Authentik global health check — every 2 minutes
+    var authentikHealthKey = new JobKey("authentik-global-health", "health-checks");
+    q.AddJob<AuthentikGlobalHealthJob>(opts => opts.WithIdentity(authentikHealthKey).StoreDurably());
+    q.AddTrigger(opts => opts
+        .ForJob(authentikHealthKey)
+        .WithIdentity("authentik-global-health-trigger", "health-checks")
+        .WithSimpleSchedule(s => s
+            .WithIntervalInMinutes(2)
+            .RepeatForever())
+        .StartNow());
+
+    // Daily screen snapshot — 2 AM UTC
+    var dailySnapshotKey = new JobKey("daily-snapshot", "snapshots");
+    q.AddJob<OTSSignsOrchestrator.Server.Jobs.DailySnapshotJob>(opts => opts.WithIdentity(dailySnapshotKey).StoreDurably());
+    q.AddTrigger(opts => opts
+        .ForJob(dailySnapshotKey)
+        .WithIdentity("daily-snapshot-trigger", "snapshots")
+        .WithCronSchedule("0 0 2 * * ?"));
+
+    // Scheduled reports — weekly (Monday 08:00 UTC) + monthly (1st 08:00 UTC)
+    var reportJobKey = new JobKey("scheduled-report", "reports");
+    q.AddJob<ScheduledReportJob>(opts => opts.WithIdentity(reportJobKey).StoreDurably());
+    q.AddTrigger(opts => opts
+        .ForJob(reportJobKey)
+        .WithIdentity("weekly-report-trigger", "reports")
+        .UsingJobData(ScheduledReportJob.IsMonthlyKey, false)
+        .WithCronSchedule("0 0 8 ? * MON *"));
+    q.AddTrigger(opts => opts
+        .ForJob(reportJobKey)
+        .WithIdentity("monthly-report-trigger", "reports")
+        .UsingJobData(ScheduledReportJob.IsMonthlyKey, true)
+        .WithCronSchedule("0 0 8 1 * ? *"));
+});
+builder.Services.AddQuartzHostedService(q => q.WaitForJobsToComplete = true);
+
+// ── SignalR ──────────────────────────────────────────────────────────────────
+builder.Services.AddSignalR();
+
+// ── Rate limiting ────────────────────────────────────────────────────────────
+builder.Services.AddRateLimiter(options =>
+{
+    options.RejectionStatusCode = 429;
+
+    options.AddFixedWindowLimiter("fixed", limiter =>
+    {
+        limiter.PermitLimit = 60;
+        limiter.Window = TimeSpan.FromMinutes(1);
+    });
+
+    options.AddSlidingWindowLimiter("signup", limiter =>
+    {
+        limiter.PermitLimit = 3;
+        limiter.Window = TimeSpan.FromMinutes(10);
+        limiter.SegmentsPerWindow = 2;
+    });
+});
+
+var app = builder.Build();
+
+// ── Middleware ────────────────────────────────────────────────────────────────
+app.UseSerilogRequestLogging();
+app.UseRateLimiter();
+app.UseAuthentication();
+app.UseAuthorization();
+
+// ── Auth endpoints (no auth required) ────────────────────────────────────────
+app.MapPost("/api/auth/login", async (LoginRequest req, OperatorAuthService auth) =>
+{
+    try
+    {
+        var (jwt, refresh) = await auth.LoginAsync(req.Email, req.Password);
+        return Results.Ok(new { token = jwt, refreshToken = refresh });
+    }
+    catch (UnauthorizedAccessException)
+    {
+        return Results.Unauthorized();
+    }
+});
+
+app.MapPost("/api/auth/refresh", async (RefreshRequest req, OperatorAuthService auth) =>
+{
+    try
+    {
+        var jwt = await auth.RefreshAsync(req.RefreshToken);
+        return Results.Ok(new { token = jwt });
+    }
+    catch (UnauthorizedAccessException)
+    {
+        return Results.Unauthorized();
+    }
+});
+
+// ── Signup endpoints (no auth) ──────────────────────────────────────────────
+app.MapSignupEndpoints();
+
+// ── Stripe webhook (no auth, no rate limit) ─────────────────────────────────
+app.MapStripeWebhook();
+
+// ── Fleet + Jobs REST endpoints (auth required) ─────────────────────────────
+app.MapFleetEndpoints();
+// ── Customer Portal BYOI endpoints (customer JWT required) ──────────────
+app.MapCustomerPortalEndpoints();
+// ── SignalR hub ─────────────────────────────────────────────────────────────
+app.MapHub<FleetHub>("/hubs/fleet");
+
+app.Run();
+
+// ── Request DTOs for auth endpoints ─────────────────────────────────────────
+public record LoginRequest(string Email, string Password);
+public record RefreshRequest(string RefreshToken);

OTSSignsOrchestrator.Server/Reports/BillingReportService.cs

@@ -0,0 +1,179 @@
+using System.Globalization;
+using System.Text;
+using CsvHelper;
+using CsvHelper.Configuration;
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Reports;
+
+public class BillingReportService
+{
+    private readonly IServiceProvider _services;
+    private readonly ILogger<BillingReportService> _logger;
+
+    public BillingReportService(IServiceProvider services, ILogger<BillingReportService> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    /// <summary>
+    /// Generates a billing CSV for the given date range.
+    /// Columns: customer_abbrev, company_name, date, screen_count, plan
+    /// </summary>
+    public async Task<byte[]> GenerateBillingCsvAsync(DateOnly from, DateOnly to)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+
+        var rows = await db.ScreenSnapshots
+            .AsNoTracking()
+            .Include(s => s.Instance)
+                .ThenInclude(i => i.Customer)
+            .Where(s => s.SnapshotDate >= from && s.SnapshotDate <= to)
+            .Select(s => new
+            {
+                Abbrev = s.Instance.Customer.Abbreviation,
+                CompanyName = s.Instance.Customer.CompanyName,
+                Plan = s.Instance.Customer.Plan,
+                s.SnapshotDate,
+                s.ScreenCount,
+            })
+            .ToListAsync();
+
+        // Group by customer abbreviation + date, sum screen counts across instances
+        var grouped = rows
+            .GroupBy(r => new { r.Abbrev, r.SnapshotDate })
+            .Select(g => new BillingCsvRow
+            {
+                CustomerAbbrev = g.Key.Abbrev,
+                CompanyName = g.First().CompanyName,
+                Date = g.Key.SnapshotDate,
+                ScreenCount = g.Sum(r => r.ScreenCount),
+                Plan = g.First().Plan.ToString(),
+            })
+            .OrderBy(r => r.CustomerAbbrev)
+            .ThenBy(r => r.Date)
+            .ToList();
+
+        _logger.LogInformation("Generated billing CSV: {RowCount} rows for {From} to {To}",
+            grouped.Count, from, to);
+
+        return WriteCsv(grouped);
+    }
+
+    /// <summary>
+    /// Generates a version drift CSV showing current vs latest Xibo version and OAuth credential age.
+    /// Columns: abbrev, current_version, latest_version, credential_age_days
+    /// </summary>
+    public async Task<byte[]> GenerateVersionDriftCsvAsync()
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+
+        // Latest health event per instance where check_name = "xibo-version"
+        var versionEvents = await db.HealthEvents
+            .AsNoTracking()
+            .Include(h => h.Instance)
+                .ThenInclude(i => i.Customer)
+            .Where(h => h.CheckName == "xibo-version")
+            .GroupBy(h => h.InstanceId)
+            .Select(g => g.OrderByDescending(h => h.OccurredAt).First())
+            .ToListAsync();
+
+        // Latest OauthAppRegistry per instance
+        var latestOauth = await db.OauthAppRegistries
+            .AsNoTracking()
+            .GroupBy(o => o.InstanceId)
+            .Select(g => g.OrderByDescending(o => o.CreatedAt).First())
+            .ToDictionaryAsync(o => o.InstanceId);
+
+        var rows = versionEvents.Select(h =>
+        {
+            var parts = ParseVersionMessage(h.Message);
+            latestOauth.TryGetValue(h.InstanceId, out var oauth);
+            var credAgeDays = oauth is not null
+                ? (int)(DateTime.UtcNow - oauth.CreatedAt).TotalDays
+                : -1;
+
+            return new VersionDriftCsvRow
+            {
+                Abbrev = h.Instance.Customer.Abbreviation,
+                CurrentVersion = parts.Current,
+                LatestVersion = parts.Latest,
+                CredentialAgeDays = credAgeDays,
+            };
+        })
+        .OrderBy(r => r.Abbrev)
+        .ToList();
+
+        _logger.LogInformation("Generated version drift CSV: {RowCount} rows", rows.Count);
+
+        return WriteCsv(rows);
+    }
+
+    private static byte[] WriteCsv<T>(IEnumerable<T> records)
+    {
+        using var stream = new MemoryStream();
+        using var writer = new StreamWriter(stream, Encoding.UTF8);
+        using var csv = new CsvWriter(writer, new CsvConfiguration(CultureInfo.InvariantCulture)
+        {
+            HasHeaderRecord = true,
+        });
+        csv.WriteRecords(records);
+        writer.Flush();
+        return stream.ToArray();
+    }
+
+    /// <summary>
+    /// Parses the health event message to extract current/latest version strings.
+    /// Expected format from XiboVersionHealthCheck: "Current: X.Y.Z, Latest: A.B.C" or similar.
+    /// Returns ("unknown", "unknown") if parsing fails.
+    /// </summary>
+    private static (string Current, string Latest) ParseVersionMessage(string? message)
+    {
+        if (string.IsNullOrWhiteSpace(message))
+            return ("unknown", "unknown");
+
+        var current = "unknown";
+        var latest = "unknown";
+
+        // Try to extract "Current: X.Y.Z" pattern
+        var currentIdx = message.IndexOf("Current:", StringComparison.OrdinalIgnoreCase);
+        if (currentIdx >= 0)
+        {
+            var afterCurrent = message[(currentIdx + 8)..].Trim();
+            var end = afterCurrent.IndexOfAny([',', ' ', '\n']);
+            current = end > 0 ? afterCurrent[..end].Trim() : afterCurrent.Trim();
+        }
+
+        var latestIdx = message.IndexOf("Latest:", StringComparison.OrdinalIgnoreCase);
+        if (latestIdx >= 0)
+        {
+            var afterLatest = message[(latestIdx + 7)..].Trim();
+            var end = afterLatest.IndexOfAny([',', ' ', '\n']);
+            latest = end > 0 ? afterLatest[..end].Trim() : afterLatest.Trim();
+        }
+
+        return (current, latest);
+    }
+}
+
+internal sealed class BillingCsvRow
+{
+    public string CustomerAbbrev { get; set; } = string.Empty;
+    public string CompanyName { get; set; } = string.Empty;
+    public DateOnly Date { get; set; }
+    public int ScreenCount { get; set; }
+    public string Plan { get; set; } = string.Empty;
+}
+
+internal sealed class VersionDriftCsvRow
+{
+    public string Abbrev { get; set; } = string.Empty;
+    public string CurrentVersion { get; set; } = string.Empty;
+    public string LatestVersion { get; set; } = string.Empty;
+    public int CredentialAgeDays { get; set; }
+}

OTSSignsOrchestrator.Server/Reports/FleetHealthPdfService.cs

@@ -0,0 +1,401 @@
+using Microsoft.EntityFrameworkCore;
+using PdfSharpCore.Drawing;
+using PdfSharpCore.Pdf;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Reports;
+
+public class FleetHealthPdfService
+{
+    private readonly IServiceProvider _services;
+    private readonly ILogger<FleetHealthPdfService> _logger;
+
+    // OTS Signs brand colours
+    private static readonly XColor BrandAccent = XColor.FromArgb(0x3B, 0x82, 0xF6);
+    private static readonly XColor HeaderBg = XColor.FromArgb(0x1E, 0x29, 0x3B);
+    private static readonly XColor TextDark = XColor.FromArgb(0x1F, 0x28, 0x37);
+    private static readonly XColor TextMuted = XColor.FromArgb(0x6B, 0x72, 0x80);
+    private static readonly XColor TableBorder = XColor.FromArgb(0xD1, 0xD5, 0xDB);
+    private static readonly XColor RowAlt = XColor.FromArgb(0xF9, 0xFA, 0xFB);
+
+    private const double PageWidth = 595;  // A4 points
+    private const double PageHeight = 842;
+    private const double MarginX = 50;
+    private const double ContentWidth = PageWidth - 2 * MarginX;
+
+    public FleetHealthPdfService(IServiceProvider services, ILogger<FleetHealthPdfService> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    /// <summary>
+    /// Generates a fleet-wide health PDF report for the given date range.
+    /// Includes: branded header, fleet summary stats, per-instance health table,
+    /// Authentik uptime percentage.
+    /// </summary>
+    public async Task<byte[]> GenerateFleetHealthPdfAsync(DateOnly from, DateOnly to)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+
+        var fromDt = from.ToDateTime(TimeOnly.MinValue, DateTimeKind.Utc);
+        var toDt = to.ToDateTime(TimeOnly.MaxValue, DateTimeKind.Utc);
+
+        var instances = await db.Instances
+            .AsNoTracking()
+            .Include(i => i.Customer)
+            .ToListAsync();
+
+        var healthEvents = await db.HealthEvents
+            .AsNoTracking()
+            .Where(h => h.OccurredAt >= fromDt && h.OccurredAt <= toDt)
+            .ToListAsync();
+
+        var screenSnapshots = await db.ScreenSnapshots
+            .AsNoTracking()
+            .Where(s => s.SnapshotDate >= from && s.SnapshotDate <= to)
+            .ToListAsync();
+
+        var authentikMetrics = await db.AuthentikMetrics
+            .AsNoTracking()
+            .Where(m => m.CheckedAt >= fromDt && m.CheckedAt <= toDt)
+            .ToListAsync();
+
+        // Compute stats
+        var totalInstances = instances.Count;
+        var totalScreens = screenSnapshots
+            .GroupBy(s => s.InstanceId)
+            .Sum(g => g.OrderByDescending(s => s.SnapshotDate).First().ScreenCount);
+
+        var healthBreakdown = instances.GroupBy(i => i.HealthStatus)
+            .ToDictionary(g => g.Key, g => g.Count());
+
+        var authentikUptime = ComputeAuthentikUptime(authentikMetrics);
+
+        var latestHealthByInstance = healthEvents
+            .GroupBy(h => h.InstanceId)
+            .ToDictionary(g => g.Key, g => g.OrderByDescending(h => h.OccurredAt).First());
+
+        var p1CountByInstance = healthEvents
+            .Where(h => h.Status == HealthEventStatus.Critical)
+            .GroupBy(h => h.InstanceId)
+            .ToDictionary(g => g.Key, g => g.Count());
+
+        // Build PDF
+        using var doc = new PdfDocument();
+        doc.Info.Title = "OTS Signs — Fleet Health Report";
+        doc.Info.Author = "OTS Signs Orchestrator";
+        var page = doc.AddPage();
+        page.Width = PageWidth;
+        page.Height = PageHeight;
+        var gfx = XGraphics.FromPdfPage(page);
+
+        var y = DrawBrandedHeader(gfx, "Fleet Health Report", from, to);
+
+        // Summary stats
+        var fontBold = new XFont("Helvetica", 12, XFontStyle.Bold);
+        var fontNormal = new XFont("Helvetica", 10);
+
+        y = DrawSectionTitle(gfx, "Fleet Summary", y);
+
+        var summaryData = new (string Label, string Value)[]
+        {
+            ("Total Instances", totalInstances.ToString()),
+            ("Total Screens", totalScreens.ToString()),
+            ("Healthy", healthBreakdown.GetValueOrDefault(HealthStatus.Healthy).ToString()),
+            ("Degraded", healthBreakdown.GetValueOrDefault(HealthStatus.Degraded).ToString()),
+            ("Critical", healthBreakdown.GetValueOrDefault(HealthStatus.Critical).ToString()),
+            ("Unknown", healthBreakdown.GetValueOrDefault(HealthStatus.Unknown).ToString()),
+            ("Authentik Uptime", $"{authentikUptime:F1}%"),
+        };
+
+        foreach (var (label, value) in summaryData)
+        {
+            gfx.DrawString(label + ":", fontBold, new XSolidBrush(TextDark),
+                new XRect(MarginX, y, 150, 16), XStringFormats.TopLeft);
+            gfx.DrawString(value, fontNormal, new XSolidBrush(TextDark),
+                new XRect(MarginX + 160, y, 150, 16), XStringFormats.TopLeft);
+            y += 18;
+        }
+
+        y += 10;
+
+        // Per-instance health table
+        y = DrawSectionTitle(gfx, "Per-Instance Health", y);
+
+        var columns = new (string Header, double Width)[]
+        {
+            ("Abbrev", 60), ("Status", 70), ("Last Check", 110), ("P1 Inc.", 50), ("URL", ContentWidth - 290),
+        };
+
+        y = DrawTableHeader(gfx, columns, y);
+
+        var rowIndex = 0;
+        foreach (var instance in instances.OrderBy(i => i.Customer.Abbreviation))
+        {
+            if (y > PageHeight - 60)
+            {
+                page = doc.AddPage();
+                page.Width = PageWidth;
+                page.Height = PageHeight;
+                gfx = XGraphics.FromPdfPage(page);
+                y = 50;
+                y = DrawTableHeader(gfx, columns, y);
+                rowIndex = 0;
+            }
+
+            latestHealthByInstance.TryGetValue(instance.Id, out var latestHealth);
+            p1CountByInstance.TryGetValue(instance.Id, out var p1Count);
+
+            var cellValues = new[]
+            {
+                instance.Customer.Abbreviation,
+                instance.HealthStatus.ToString(),
+                latestHealth?.OccurredAt.ToString("yyyy-MM-dd HH:mm") ?? "—",
+                p1Count.ToString(),
+                instance.XiboUrl,
+            };
+
+            y = DrawTableRow(gfx, columns, cellValues, y, rowIndex % 2 == 1);
+            rowIndex++;
+        }
+
+        _logger.LogInformation("Generated fleet health PDF: {Instances} instances, period {From} to {To}",
+            totalInstances, from, to);
+
+        return SavePdf(doc);
+    }
+
+    /// <summary>
+    /// Generates a per-customer usage PDF for the given date range.
+    /// </summary>
+    public async Task<byte[]> GenerateCustomerUsagePdfAsync(Guid customerId, DateOnly from, DateOnly to)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+
+        var customer = await db.Customers
+            .AsNoTracking()
+            .Include(c => c.Instances)
+            .FirstOrDefaultAsync(c => c.Id == customerId)
+            ?? throw new InvalidOperationException($"Customer {customerId} not found.");
+
+        var snapshots = await db.ScreenSnapshots
+            .AsNoTracking()
+            .Where(s => s.Instance.CustomerId == customerId
+                && s.SnapshotDate >= from
+                && s.SnapshotDate <= to)
+            .OrderBy(s => s.SnapshotDate)
+            .ToListAsync();
+
+        using var doc = new PdfDocument();
+        doc.Info.Title = $"OTS Signs — Usage Report — {customer.CompanyName}";
+        doc.Info.Author = "OTS Signs Orchestrator";
+        var page = doc.AddPage();
+        page.Width = PageWidth;
+        page.Height = PageHeight;
+        var gfx = XGraphics.FromPdfPage(page);
+
+        var y = DrawBrandedHeader(gfx, $"Customer Usage Report — {customer.CompanyName}", from, to);
+
+        // Customer details
+        y = DrawSectionTitle(gfx, "Customer Details", y);
+        var fontBold = new XFont("Helvetica", 10, XFontStyle.Bold);
+        var fontNormal = new XFont("Helvetica", 10);
+
+        var details = new (string Label, string Value)[]
+        {
+            ("Company", customer.CompanyName),
+            ("Abbreviation", customer.Abbreviation),
+            ("Plan", customer.Plan.ToString()),
+            ("Screen Quota", customer.ScreenCount.ToString()),
+            ("Status", customer.Status.ToString()),
+        };
+
+        var primaryInstance = customer.Instances.FirstOrDefault();
+        if (primaryInstance is not null)
+            details = [.. details, ("Instance URL", primaryInstance.XiboUrl)];
+
+        foreach (var (label, value) in details)
+        {
+            gfx.DrawString(label + ":", fontBold, new XSolidBrush(TextDark),
+                new XRect(MarginX, y, 120, 16), XStringFormats.TopLeft);
+            gfx.DrawString(value, fontNormal, new XSolidBrush(TextDark),
+                new XRect(MarginX + 130, y, 350, 16), XStringFormats.TopLeft);
+            y += 18;
+        }
+
+        y += 10;
+
+        // Screen count history
+        y = DrawSectionTitle(gfx, "Screen Count History", y);
+
+        if (snapshots.Count == 0)
+        {
+            var fontItalic = new XFont("Helvetica", 10, XFontStyle.Italic);
+            gfx.DrawString("No screen snapshot data available for this period.",
+                fontItalic, new XSolidBrush(TextMuted),
+                new XRect(MarginX, y, ContentWidth, 16), XStringFormats.TopLeft);
+            y += 20;
+        }
+        else
+        {
+            var byDate = snapshots
+                .GroupBy(s => s.SnapshotDate)
+                .OrderBy(g => g.Key)
+                .Select(g => (Date: g.Key, Count: g.Sum(s => s.ScreenCount)))
+                .ToList();
+
+            // Table
+            var cols = new (string Header, double Width)[] { ("Date", 150), ("Screen Count", 150) };
+            y = DrawTableHeader(gfx, cols, y);
+            var rowIdx = 0;
+            foreach (var (date, count) in byDate)
+            {
+                if (y > PageHeight - 60)
+                {
+                    page = doc.AddPage();
+                    page.Width = PageWidth;
+                    page.Height = PageHeight;
+                    gfx = XGraphics.FromPdfPage(page);
+                    y = 50;
+                    y = DrawTableHeader(gfx, cols, y);
+                    rowIdx = 0;
+                }
+                y = DrawTableRow(gfx, cols, [date.ToString("yyyy-MM-dd"), count.ToString()], y, rowIdx % 2 == 1);
+                rowIdx++;
+            }
+
+            y += 20;
+
+            // Simple bar chart
+            if (y + byDate.Count * 14 + 30 > PageHeight - 40)
+            {
+                page = doc.AddPage();
+                page.Width = PageWidth;
+                page.Height = PageHeight;
+                gfx = XGraphics.FromPdfPage(page);
+                y = 50;
+            }
+
+            y = DrawSectionTitle(gfx, "Daily Screen Count", y);
+            var maxCount = byDate.Max(d => d.Count);
+            if (maxCount > 0)
+            {
+                var barMaxWidth = ContentWidth - 120;
+                var fontSmall = new XFont("Helvetica", 8);
+                var barBrush = new XSolidBrush(BrandAccent);
+
+                foreach (var (date, count) in byDate)
+                {
+                    var barWidth = Math.Max(2, (double)count / maxCount * barMaxWidth);
+                    gfx.DrawString(date.ToString("MM-dd"), fontSmall, new XSolidBrush(TextDark),
+                        new XRect(MarginX, y, 50, 12), XStringFormats.TopLeft);
+                    gfx.DrawRectangle(barBrush, MarginX + 55, y + 1, barWidth, 10);
+                    gfx.DrawString(count.ToString(), fontSmall, new XSolidBrush(TextDark),
+                        new XRect(MarginX + 60 + barWidth, y, 50, 12), XStringFormats.TopLeft);
+                    y += 14;
+                }
+            }
+        }
+
+        _logger.LogInformation("Generated customer usage PDF for {CustomerId} ({Abbrev}), period {From} to {To}",
+            customerId, customer.Abbreviation, from, to);
+
+        return SavePdf(doc);
+    }
+
+    // ── Drawing helpers ─────────────────────────────────────────────────────
+
+    private static double DrawBrandedHeader(XGraphics gfx, string title, DateOnly from, DateOnly to)
+    {
+        var y = 40.0;
+
+        var brandFont = new XFont("Helvetica", 22, XFontStyle.Bold);
+        gfx.DrawString("OTS Signs", brandFont, new XSolidBrush(BrandAccent),
+            new XRect(MarginX, y, ContentWidth, 28), XStringFormats.TopLeft);
+        y += 30;
+
+        var titleFont = new XFont("Helvetica", 14, XFontStyle.Bold);
+        gfx.DrawString(title, titleFont, new XSolidBrush(TextDark),
+            new XRect(MarginX, y, ContentWidth, 20), XStringFormats.TopLeft);
+        y += 22;
+
+        var dateFont = new XFont("Helvetica", 9);
+        gfx.DrawString($"Period: {from:yyyy-MM-dd} to {to:yyyy-MM-dd}  |  Generated: {DateTime.UtcNow:yyyy-MM-dd HH:mm} UTC",
+            dateFont, new XSolidBrush(TextMuted),
+            new XRect(MarginX, y, ContentWidth, 14), XStringFormats.TopLeft);
+        y += 18;
+
+        // Separator
+        gfx.DrawLine(new XPen(BrandAccent, 1.5), MarginX, y, PageWidth - MarginX, y);
+        y += 15;
+
+        return y;
+    }
+
+    private static double DrawSectionTitle(XGraphics gfx, string title, double y)
+    {
+        var font = new XFont("Helvetica", 13, XFontStyle.Bold);
+        gfx.DrawString(title, font, new XSolidBrush(TextDark),
+            new XRect(MarginX, y, ContentWidth, 18), XStringFormats.TopLeft);
+        return y + 22;
+    }
+
+    private static double DrawTableHeader(XGraphics gfx, (string Header, double Width)[] columns, double y)
+    {
+        var font = new XFont("Helvetica", 9, XFontStyle.Bold);
+        var brush = new XSolidBrush(XColors.White);
+        var bgBrush = new XSolidBrush(HeaderBg);
+        var x = MarginX;
+
+        var totalWidth = columns.Sum(c => c.Width);
+        gfx.DrawRectangle(bgBrush, x, y, totalWidth, 18);
+
+        foreach (var (header, width) in columns)
+        {
+            gfx.DrawString(header, font, brush, new XRect(x + 4, y + 3, width - 8, 14), XStringFormats.TopLeft);
+            x += width;
+        }
+
+        return y + 18;
+    }
+
+    private static double DrawTableRow(XGraphics gfx, (string Header, double Width)[] columns, string[] values, double y, bool alternate)
+    {
+        var font = new XFont("Helvetica", 8);
+        var textBrush = new XSolidBrush(TextDark);
+        var x = MarginX;
+        var totalWidth = columns.Sum(c => c.Width);
+
+        if (alternate)
+            gfx.DrawRectangle(new XSolidBrush(RowAlt), x, y, totalWidth, 16);
+
+        gfx.DrawLine(new XPen(TableBorder, 0.5), x, y + 16, x + totalWidth, y + 16);
+
+        for (int i = 0; i < columns.Length && i < values.Length; i++)
+        {
+            gfx.DrawString(values[i], font, textBrush,
+                new XRect(x + 4, y + 2, columns[i].Width - 8, 14), XStringFormats.TopLeft);
+            x += columns[i].Width;
+        }
+
+        return y + 16;
+    }
+
+    private static double ComputeAuthentikUptime(List<AuthentikMetrics> metrics)
+    {
+        if (metrics.Count == 0) return 100.0;
+        var healthy = metrics.Count(m => m.Status == AuthentikMetricsStatus.Healthy);
+        return (double)healthy / metrics.Count * 100;
+    }
+
+    private static byte[] SavePdf(PdfDocument doc)
+    {
+        using var stream = new MemoryStream();
+        doc.Save(stream, false);
+        return stream.ToArray();
+    }
+}

OTSSignsOrchestrator.Server/Services/AbbreviationService.cs

@@ -0,0 +1,78 @@
+using System.Text.RegularExpressions;
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Data;
+
+namespace OTSSignsOrchestrator.Server.Services;
+
+public class AbbreviationService
+{
+    private static readonly HashSet<string> StopWords = new(StringComparer.OrdinalIgnoreCase)
+    {
+        "inc", "llc", "ltd", "co", "corp", "group", "signs", "digital",
+        "media", "the", "and", "of", "a"
+    };
+
+    private readonly OrchestratorDbContext _db;
+    private readonly ILogger<AbbreviationService> _logger;
+
+    public AbbreviationService(OrchestratorDbContext db, ILogger<AbbreviationService> logger)
+    {
+        _db = db;
+        _logger = logger;
+    }
+
+    public async Task<string> GenerateAsync(string companyName)
+    {
+        var words = Regex.Split(companyName.Trim(), @"\s+")
+            .Select(w => Regex.Replace(w, @"[^a-zA-Z0-9]", ""))
+            .Where(w => w.Length > 0 && !StopWords.Contains(w))
+            .ToList();
+
+        if (words.Count == 0)
+            throw new InvalidOperationException(
+                $"Cannot generate abbreviation from company name '{companyName}' — no usable words after filtering.");
+
+        string abbrev;
+        if (words.Count >= 3)
+        {
+            // Take first letter of first 3 words
+            abbrev = string.Concat(words.Take(3).Select(w => w[0]));
+        }
+        else if (words.Count == 2)
+        {
+            // First letter of each word + second char of last word
+            abbrev = $"{words[0][0]}{words[1][0]}{(words[1].Length > 1 ? words[1][1] : words[0][1])}";
+        }
+        else
+        {
+            // Single word — take up to 3 chars
+            abbrev = words[0].Length >= 3 ? words[0][..3] : words[0].PadRight(3, 'X');
+        }
+
+        abbrev = Regex.Replace(abbrev.ToUpperInvariant(), @"[^A-Z0-9]", "");
+        if (abbrev.Length > 3) abbrev = abbrev[..3];
+
+        // Check uniqueness
+        if (!await _db.Customers.AnyAsync(c => c.Abbreviation == abbrev))
+        {
+            _logger.LogInformation("Generated abbreviation {Abbrev} for '{CompanyName}'", abbrev, companyName);
+            return abbrev;
+        }
+
+        // Collision — try suffix 2–9
+        var prefix = abbrev[..2];
+        for (var suffix = 2; suffix <= 9; suffix++)
+        {
+            var candidate = $"{prefix}{suffix}";
+            if (!await _db.Customers.AnyAsync(c => c.Abbreviation == candidate))
+            {
+                _logger.LogInformation("Generated abbreviation {Abbrev} (collision resolved) for '{CompanyName}'",
+                    candidate, companyName);
+                return candidate;
+            }
+        }
+
+        throw new InvalidOperationException(
+            $"All abbreviation variants for '{companyName}' are taken ({prefix}2–{prefix}9).");
+    }
+}

OTSSignsOrchestrator.Server/Services/EmailService.cs

@@ -0,0 +1,173 @@
+using Microsoft.Extensions.Options;
+using SendGrid;
+using SendGrid.Helpers.Mail;
+
+namespace OTSSignsOrchestrator.Server.Services;
+
+public class EmailOptions
+{
+    public const string Section = "Email";
+    public string SendGridApiKey { get; set; } = string.Empty;
+    public string SenderEmail { get; set; } = "noreply@otssigns.com";
+    public string SenderName { get; set; } = "OTS Signs";
+}
+
+/// <summary>
+/// Email service backed by SendGrid. All methods catch exceptions, log as Error,
+/// and return bool success — they never throw.
+/// </summary>
+public class EmailService
+{
+    private readonly ILogger<EmailService> _logger;
+    private readonly EmailOptions _options;
+    private readonly SendGridClient? _client;
+
+    public EmailService(ILogger<EmailService> logger, IOptions<EmailOptions> options)
+    {
+        _logger = logger;
+        _options = options.Value;
+
+        if (!string.IsNullOrWhiteSpace(_options.SendGridApiKey))
+        {
+            _client = new SendGridClient(_options.SendGridApiKey);
+        }
+        else
+        {
+            _logger.LogWarning("SendGrid API key not configured — emails will be logged only");
+        }
+    }
+
+    public async Task<bool> SendWelcomeEmailAsync(
+        string toEmail, string firstName, string instanceUrl, string invitationLink)
+    {
+        var subject = "Welcome to OTS Signs — Your CMS is Ready!";
+        var html = $"""
+            <h2>Welcome to OTS Signs, {HtmlEncode(firstName)}!</h2>
+            <p>Your Xibo CMS instance has been provisioned and is ready to use.</p>
+            <p><strong>Instance URL:</strong> <a href="{HtmlEncode(instanceUrl)}">{HtmlEncode(instanceUrl)}</a></p>
+            <p><strong>Accept your invitation to get started:</strong></p>
+            <p><a href="{HtmlEncode(invitationLink)}" style="display:inline-block;padding:12px 24px;background-color:#3B82F6;color:#fff;text-decoration:none;border-radius:6px;">Accept Invitation</a></p>
+            <p>If you have any questions, please contact our support team.</p>
+            <p>— The OTS Signs Team</p>
+            """;
+
+        return await SendEmailAsync(toEmail, subject, html);
+    }
+
+    public async Task<bool> SendPaymentFailedEmailAsync(
+        string toEmail, string companyName, int failedCount)
+    {
+        var daysSinceFirst = failedCount * 3; // approximate
+        string subject;
+        string urgency;
+
+        if (daysSinceFirst >= 21)
+        {
+            subject = $"FINAL NOTICE — Payment Required for {companyName}";
+            urgency = "<p style=\"color:#DC2626;font-weight:bold;\">⚠️ FINAL NOTICE: Your service will be suspended if payment is not received immediately.</p>";
+        }
+        else if (daysSinceFirst >= 7)
+        {
+            subject = $"URGENT — Payment Failed for {companyName}";
+            urgency = "<p style=\"color:#F59E0B;font-weight:bold;\">⚠️ URGENT: Please update your payment method to avoid service interruption.</p>";
+        }
+        else
+        {
+            subject = $"Payment Failed for {companyName}";
+            urgency = "<p>Please update your payment method at your earliest convenience.</p>";
+        }
+
+        var html = $"""
+            <h2>Payment Failed — {HtmlEncode(companyName)}</h2>
+            {urgency}
+            <p>We were unable to process your payment (attempt #{failedCount}).</p>
+            <p>Please update your payment method to keep your OTS Signs service active.</p>
+            <p>— The OTS Signs Team</p>
+            """;
+
+        return await SendEmailAsync(toEmail, subject, html);
+    }
+
+    public async Task<bool> SendTrialEndingEmailAsync(
+        string toEmail, string firstName, DateTime trialEndDate)
+    {
+        var daysLeft = (int)Math.Ceiling((trialEndDate - DateTime.UtcNow).TotalDays);
+        var subject = $"Your OTS Signs Trial Ends in {Math.Max(0, daysLeft)} Day{(daysLeft != 1 ? "s" : "")}";
+        var html = $"""
+            <h2>Hi {HtmlEncode(firstName)},</h2>
+            <p>Your OTS Signs trial ends on <strong>{trialEndDate:MMMM dd, yyyy}</strong>.</p>
+            <p>To continue using your CMS without interruption, please subscribe before your trial expires.</p>
+            <p>— The OTS Signs Team</p>
+            """;
+
+        return await SendEmailAsync(toEmail, subject, html);
+    }
+
+    public async Task<bool> SendReportEmailAsync(
+        string toEmail, string attachmentName, byte[] attachment, string mimeType)
+    {
+        var subject = $"OTS Signs Report — {attachmentName}";
+        var html = $"""
+            <h2>OTS Signs Report</h2>
+            <p>Please find the attached report: <strong>{HtmlEncode(attachmentName)}</strong></p>
+            <p>This report was generated automatically by the OTS Signs Orchestrator.</p>
+            <p>— The OTS Signs Team</p>
+            """;
+
+        return await SendEmailAsync(toEmail, subject, html, attachmentName, attachment, mimeType);
+    }
+
+    private async Task<bool> SendEmailAsync(
+        string toEmail,
+        string subject,
+        string htmlContent,
+        string? attachmentName = null,
+        byte[]? attachmentData = null,
+        string? attachmentMimeType = null)
+    {
+        try
+        {
+            if (_client is null)
+            {
+                _logger.LogWarning(
+                    "SendGrid not configured — would send email to {Email}: {Subject}",
+                    toEmail, subject);
+                return true; // Not a failure — just not configured
+            }
+
+            var from = new EmailAddress(_options.SenderEmail, _options.SenderName);
+            var to = new EmailAddress(toEmail);
+            var msg = MailHelper.CreateSingleEmail(from, to, subject, null, htmlContent);
+
+            if (attachmentData is not null && attachmentName is not null)
+            {
+                msg.AddAttachment(
+                    attachmentName,
+                    Convert.ToBase64String(attachmentData),
+                    attachmentMimeType ?? "application/octet-stream");
+            }
+
+            var response = await _client.SendEmailAsync(msg);
+
+            if (response.IsSuccessStatusCode)
+            {
+                _logger.LogInformation("Email sent to {Email}: {Subject}", toEmail, subject);
+                return true;
+            }
+
+            var body = await response.Body.ReadAsStringAsync();
+            _logger.LogError(
+                "SendGrid returned {StatusCode} for email to {Email}: {Body}",
+                response.StatusCode, toEmail, body);
+            return false;
+        }
+        catch (Exception ex)
+        {
+            _logger.LogError(ex, "Failed to send email to {Email}: {Subject}", toEmail, subject);
+            return false;
+        }
+    }
+
+    private static string HtmlEncode(string value) =>
+        System.Net.WebUtility.HtmlEncode(value);
+}

OTSSignsOrchestrator.Server/Webhooks/StripeWebhookHandler.cs

@@ -0,0 +1,327 @@
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+using OTSSignsOrchestrator.Server.Services;
+using Stripe;
+
+namespace OTSSignsOrchestrator.Server.Webhooks;
+
+public static class StripeWebhookHandler
+{
+    public static void MapStripeWebhook(this WebApplication app)
+    {
+        app.MapPost("/api/webhooks/stripe", HandleWebhook)
+            .DisableRateLimiting();
+    }
+
+    private static async Task<IResult> HandleWebhook(
+        HttpContext httpContext,
+        IConfiguration config,
+        OrchestratorDbContext db,
+        AbbreviationService abbreviationService,
+        EmailService emailService,
+        IHubContext<FleetHub, IFleetClient> hub,
+        ILogger<StripeEvent> logger)
+    {
+        // ── Verify signature ────────────────────────────────────────────────
+        var json = await new StreamReader(httpContext.Request.Body).ReadToEndAsync();
+        var webhookSecret = config["Stripe:WebhookSecret"]
+            ?? throw new InvalidOperationException("Stripe:WebhookSecret not configured.");
+
+        Event stripeEvent;
+        try
+        {
+            stripeEvent = EventUtility.ConstructEvent(
+                json,
+                httpContext.Request.Headers["Stripe-Signature"],
+                webhookSecret);
+        }
+        catch (StripeException ex)
+        {
+            logger.LogWarning("Stripe signature verification failed: {Error}", ex.Message);
+            return Results.BadRequest("Invalid signature.");
+        }
+
+        // ── Idempotency check ───────────────────────────────────────────────
+        if (await db.StripeEvents.AnyAsync(e => e.StripeEventId == stripeEvent.Id))
+        {
+            logger.LogInformation("Stripe event {EventId} already processed — skipping", stripeEvent.Id);
+            return Results.Ok();
+        }
+
+        // ── Record event before processing ──────────────────────────────────
+        db.StripeEvents.Add(new Data.Entities.StripeEvent
+        {
+            StripeEventId = stripeEvent.Id,
+            EventType = stripeEvent.Type,
+            ProcessedAt = DateTime.UtcNow,
+            Payload = json,
+        });
+        await db.SaveChangesAsync();
+
+        // ── Dispatch ────────────────────────────────────────────────────────
+        logger.LogInformation("Processing Stripe event {EventId} type={EventType}", stripeEvent.Id, stripeEvent.Type);
+
+        switch (stripeEvent.Type)
+        {
+            case EventTypes.CheckoutSessionCompleted:
+                await HandleCheckoutSessionCompleted(stripeEvent, db, abbreviationService, hub, logger);
+                break;
+
+            case EventTypes.InvoicePaid:
+                logger.LogInformation("Invoice paid: {EventId}", stripeEvent.Id);
+                break;
+
+            case EventTypes.InvoicePaymentFailed:
+                await HandleInvoicePaymentFailed(stripeEvent, db, emailService, hub, logger);
+                break;
+
+            case EventTypes.CustomerSubscriptionUpdated:
+                await HandleSubscriptionUpdated(stripeEvent, db, hub, logger);
+                break;
+
+            case EventTypes.CustomerSubscriptionDeleted:
+                await HandleSubscriptionDeleted(stripeEvent, db, hub, logger);
+                break;
+
+            case "customer.subscription.trial_will_end":
+                logger.LogInformation("Trial ending soon for event {EventId}", stripeEvent.Id);
+                break;
+
+            default:
+                logger.LogInformation("Unhandled Stripe event type: {EventType}", stripeEvent.Type);
+                break;
+        }
+
+        return Results.Ok();
+    }
+
+    // ── checkout.session.completed ──────────────────────────────────────────
+    private static async Task HandleCheckoutSessionCompleted(
+        Event stripeEvent,
+        OrchestratorDbContext db,
+        AbbreviationService abbreviationService,
+        IHubContext<FleetHub, IFleetClient> hub,
+        ILogger<StripeEvent> logger)
+    {
+        var session = stripeEvent.Data.Object as Stripe.Checkout.Session;
+        if (session is null) return;
+
+        var meta = session.Metadata;
+        if (!meta.TryGetValue("ots_customer_id", out var customerIdStr) ||
+            !Guid.TryParse(customerIdStr, out var customerId))
+        {
+            logger.LogWarning("checkout.session.completed missing ots_customer_id metadata");
+            return;
+        }
+
+        var customer = await db.Customers.FindAsync(customerId);
+        if (customer is null)
+        {
+            logger.LogWarning("Customer {CustomerId} not found for checkout session", customerId);
+            return;
+        }
+
+        // Update customer from Stripe metadata
+        customer.StripeCustomerId = session.CustomerId;
+        customer.StripeSubscriptionId = session.SubscriptionId;
+        customer.Status = CustomerStatus.Provisioning;
+
+        if (meta.TryGetValue("company_name", out var cn)) customer.CompanyName = cn;
+        if (meta.TryGetValue("admin_email", out var ae)) customer.AdminEmail = ae;
+        if (meta.TryGetValue("admin_first_name", out var fn)) customer.AdminFirstName = fn;
+        if (meta.TryGetValue("admin_last_name", out var ln)) customer.AdminLastName = ln;
+        if (meta.TryGetValue("plan", out var planStr) && Enum.TryParse<CustomerPlan>(planStr, true, out var plan))
+            customer.Plan = plan;
+        if (meta.TryGetValue("screen_count", out var scStr) && int.TryParse(scStr, out var sc))
+            customer.ScreenCount = sc;
+
+        // Generate abbreviation
+        var abbrev = await abbreviationService.GenerateAsync(customer.CompanyName);
+        customer.Abbreviation = abbrev;
+
+        // Create provisioning job
+        var job = new Job
+        {
+            Id = Guid.NewGuid(),
+            CustomerId = customer.Id,
+            JobType = "provision",
+            Status = JobStatus.Queued,
+            TriggeredBy = "stripe-webhook",
+            CreatedAt = DateTime.UtcNow,
+        };
+        db.Jobs.Add(job);
+        await db.SaveChangesAsync();
+
+        logger.LogInformation(
+            "Checkout completed: customer={CustomerId}, abbrev={Abbrev}, job={JobId}",
+            customer.Id, abbrev, job.Id);
+
+        await hub.Clients.All.SendJobCreated(job.Id.ToString(), abbrev, "provision");
+    }
+
+    // ── invoice.payment_failed ──────────────────────────────────────────────
+    private static async Task HandleInvoicePaymentFailed(
+        Event stripeEvent,
+        OrchestratorDbContext db,
+        EmailService emailService,
+        IHubContext<FleetHub, IFleetClient> hub,
+        ILogger<StripeEvent> logger)
+    {
+        var invoice = stripeEvent.Data.Object as Stripe.Invoice;
+        if (invoice is null) return;
+
+        var customer = await db.Customers.FirstOrDefaultAsync(
+            c => c.StripeCustomerId == invoice.CustomerId);
+        if (customer is null)
+        {
+            logger.LogWarning("No customer found for Stripe customer {StripeCustomerId}", invoice.CustomerId);
+            return;
+        }
+
+        customer.FailedPaymentCount++;
+        customer.FirstPaymentFailedAt ??= DateTime.UtcNow;
+        await db.SaveChangesAsync();
+
+        logger.LogInformation(
+            "Payment failed for customer {CustomerId} ({Company}) — count={Count}",
+            customer.Id, customer.CompanyName, customer.FailedPaymentCount);
+
+        await emailService.SendPaymentFailedEmailAsync(
+            customer.AdminEmail, customer.CompanyName, customer.FailedPaymentCount);
+
+        // At day 14 with 3+ failures → suspend
+        var daysSinceFirst = (DateTime.UtcNow - customer.FirstPaymentFailedAt.Value).TotalDays;
+        if (customer.FailedPaymentCount >= 3 && daysSinceFirst >= 14)
+        {
+            logger.LogWarning(
+                "Customer {CustomerId} ({Company}) has {Count} failed payments over {Days} days — creating suspend job",
+                customer.Id, customer.CompanyName, customer.FailedPaymentCount, (int)daysSinceFirst);
+
+            var job = new Job
+            {
+                Id = Guid.NewGuid(),
+                CustomerId = customer.Id,
+                JobType = "suspend",
+                Status = JobStatus.Queued,
+                TriggeredBy = "stripe-webhook",
+                CreatedAt = DateTime.UtcNow,
+            };
+            db.Jobs.Add(job);
+            await db.SaveChangesAsync();
+
+            await hub.Clients.All.SendJobCreated(job.Id.ToString(), customer.Abbreviation, "suspend");
+        }
+    }
+
+    // ── customer.subscription.updated ───────────────────────────────────────
+    private static async Task HandleSubscriptionUpdated(
+        Event stripeEvent,
+        OrchestratorDbContext db,
+        IHubContext<FleetHub, IFleetClient> hub,
+        ILogger<StripeEvent> logger)
+    {
+        var subscription = stripeEvent.Data.Object as Stripe.Subscription;
+        if (subscription is null) return;
+
+        var customer = await db.Customers.FirstOrDefaultAsync(
+            c => c.StripeSubscriptionId == subscription.Id);
+        if (customer is null)
+        {
+            logger.LogWarning("No customer found for subscription {SubscriptionId}", subscription.Id);
+            return;
+        }
+
+        // Sync screen count from subscription quantity
+        var quantity = subscription.Items?.Data?.FirstOrDefault()?.Quantity;
+        if (quantity.HasValue && (int)quantity.Value != customer.ScreenCount)
+        {
+            var oldCount = customer.ScreenCount;
+            customer.ScreenCount = (int)quantity.Value;
+
+            logger.LogInformation(
+                "Screen count changed for customer {CustomerId}: {Old} → {New}",
+                customer.Id, oldCount, customer.ScreenCount);
+
+            var job = new Job
+            {
+                Id = Guid.NewGuid(),
+                CustomerId = customer.Id,
+                JobType = "update-screen-limit",
+                Status = JobStatus.Queued,
+                TriggeredBy = "stripe-webhook",
+                Parameters = $"{{\"oldCount\":{oldCount},\"newCount\":{customer.ScreenCount}}}",
+                CreatedAt = DateTime.UtcNow,
+            };
+            db.Jobs.Add(job);
+            await db.SaveChangesAsync();
+
+            await hub.Clients.All.SendJobCreated(job.Id.ToString(), customer.Abbreviation, "update-screen-limit");
+        }
+
+        // Reactivate if subscription is active and customer was suspended
+        if (subscription.Status == "active" && customer.Status == CustomerStatus.Suspended)
+        {
+            logger.LogInformation("Reactivating customer {CustomerId} — subscription now active", customer.Id);
+
+            customer.FailedPaymentCount = 0;
+            customer.FirstPaymentFailedAt = null;
+
+            var job = new Job
+            {
+                Id = Guid.NewGuid(),
+                CustomerId = customer.Id,
+                JobType = "reactivate",
+                Status = JobStatus.Queued,
+                TriggeredBy = "stripe-webhook",
+                CreatedAt = DateTime.UtcNow,
+            };
+            db.Jobs.Add(job);
+            await db.SaveChangesAsync();
+
+            await hub.Clients.All.SendJobCreated(job.Id.ToString(), customer.Abbreviation, "reactivate");
+        }
+
+        await db.SaveChangesAsync();
+        await hub.Clients.All.SendInstanceStatusChanged(customer.Id.ToString(), customer.Status.ToString());
+    }
+
+    // ── customer.subscription.deleted ───────────────────────────────────────
+    private static async Task HandleSubscriptionDeleted(
+        Event stripeEvent,
+        OrchestratorDbContext db,
+        IHubContext<FleetHub, IFleetClient> hub,
+        ILogger<StripeEvent> logger)
+    {
+        var subscription = stripeEvent.Data.Object as Stripe.Subscription;
+        if (subscription is null) return;
+
+        var customer = await db.Customers.FirstOrDefaultAsync(
+            c => c.StripeSubscriptionId == subscription.Id);
+        if (customer is null)
+        {
+            logger.LogWarning("No customer found for deleted subscription {SubscriptionId}", subscription.Id);
+            return;
+        }
+
+        logger.LogInformation(
+            "Subscription deleted for customer {CustomerId} ({Company}) — creating decommission job",
+            customer.Id, customer.CompanyName);
+
+        var job = new Job
+        {
+            Id = Guid.NewGuid(),
+            CustomerId = customer.Id,
+            JobType = "decommission",
+            Status = JobStatus.Queued,
+            TriggeredBy = "stripe-webhook",
+            CreatedAt = DateTime.UtcNow,
+        };
+        db.Jobs.Add(job);
+        await db.SaveChangesAsync();
+
+        await hub.Clients.All.SendJobCreated(job.Id.ToString(), customer.Abbreviation, "decommission");
+    }
+}

OTSSignsOrchestrator.Server/Workers/ByoiSamlPipeline.cs

@@ -0,0 +1,207 @@
+using System.Security.Cryptography.X509Certificates;
+using System.Text.Json;
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using Microsoft.Extensions.Options;
+using Quartz;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+using OTSSignsOrchestrator.Server.Jobs;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// BYOI SAML provisioning pipeline — configures an upstream SAML source in Authentik
+/// so a Pro-tier customer can federate their own Identity Provider.
+///
+/// Handles <c>JobType = "provision-byoi"</c>.
+///
+/// Steps:
+///   1. import-cert          — Import the customer's public cert PEM into Authentik
+///   2. create-saml-source   — Create the upstream SAML source in Authentik
+///   3. store-metadata       — Persist slug, entity_id, sso_url, cert info to ByoiConfig
+///   4. schedule-cert-monitor — Register a Quartz ByoiCertExpiryJob for daily cert expiry checks
+///
+/// IMPORTANT: Xibo's settings-custom.php is IDENTICAL for Pro BYOI and Essentials.
+/// Xibo always authenticates through Authentik. BYOI is entirely handled within
+/// Authentik's upstream SAML Source config — no Xibo changes are needed.
+/// </summary>
+public sealed class ByoiSamlPipeline : IProvisioningPipeline
+{
+    public string HandlesJobType => "provision-byoi";
+
+    private const int TotalSteps = 4;
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<ByoiSamlPipeline> _logger;
+
+    public ByoiSamlPipeline(
+        IServiceProvider services,
+        ILogger<ByoiSamlPipeline> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task ExecuteAsync(Job job, CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+        var authentikClient = scope.ServiceProvider.GetRequiredService<IAuthentikClient>();
+        var authentikOpts = scope.ServiceProvider.GetRequiredService<IOptions<AuthentikOptions>>().Value;
+        var schedulerFactory = scope.ServiceProvider.GetRequiredService<ISchedulerFactory>();
+
+        // Load customer + instance
+        var customer = await db.Customers
+            .Include(c => c.Instances)
+            .FirstOrDefaultAsync(c => c.Id == job.CustomerId, ct)
+            ?? throw new InvalidOperationException($"Customer {job.CustomerId} not found.");
+
+        if (customer.Plan != CustomerPlan.Pro)
+            throw new InvalidOperationException("BYOI SAML is only available for Pro tier customers.");
+
+        var instance = customer.Instances.FirstOrDefault()
+            ?? throw new InvalidOperationException($"No instance found for customer {job.CustomerId}.");
+
+        var abbrev = customer.Abbreviation;
+        var companyName = customer.CompanyName;
+
+        // Parse BYOI parameters from Job.Parameters JSON
+        var parms = JsonSerializer.Deserialize<ByoiParameters>(
+            job.Parameters ?? throw new InvalidOperationException("Job.Parameters is null."),
+            new JsonSerializerOptions { PropertyNameCaseInsensitive = true })
+            ?? throw new InvalidOperationException("Failed to deserialize BYOI parameters.");
+
+        var runner = new StepRunner(db, hub, _logger, job.Id, TotalSteps);
+
+        // Mutable state accumulated across steps
+        string? verificationKpId = null;
+        string slug = $"byoi-{abbrev}";
+        DateTime certExpiry;
+
+        // Parse the cert up front to get expiry and validate
+        using var cert = X509CertificateLoader.LoadCertificate(
+            Convert.FromBase64String(ExtractBase64FromPem(parms.CustomerCertPem)));
+        certExpiry = cert.NotAfter.ToUniversalTime();
+
+        // ── Step 1: import-cert ─────────────────────────────────────────────
+        await runner.RunAsync("import-cert", async () =>
+        {
+            var response = await authentikClient.ImportCertificateAsync(new ImportCertRequest(
+                Name: $"byoi-{abbrev}-cert",
+                CertificateData: parms.CustomerCertPem,
+                KeyData: null)); // PUBLIC cert only — no private key
+
+            if (!response.IsSuccessStatusCode || response.Content is null)
+                throw new InvalidOperationException(
+                    $"Failed to import certificate: {response.Error?.Content ?? response.ReasonPhrase}");
+
+            verificationKpId = response.Content["pk"]?.ToString()
+                ?? throw new InvalidOperationException("Authentik cert import returned no pk.");
+
+            return $"Imported customer cert as keypair '{verificationKpId}'.";
+        }, ct);
+
+        // ── Step 2: create-saml-source ──────────────────────────────────────
+        await runner.RunAsync("create-saml-source", async () =>
+        {
+            var response = await authentikClient.CreateSamlSourceAsync(new CreateSamlSourceRequest(
+                Name: $"{companyName} IdP",
+                Slug: slug,
+                SsoUrl: parms.CustomerSsoUrl,
+                SloUrl: parms.CustomerSloUrl,
+                Issuer: parms.CustomerIdpEntityId,
+                SigningKp: string.IsNullOrWhiteSpace(authentikOpts.OtsSigningKpId) ? null : authentikOpts.OtsSigningKpId,
+                VerificationKp: verificationKpId,
+                BindingType: "redirect",
+                NameIdPolicy: "urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress",
+                PreAuthenticationFlow: authentikOpts.SourcePreAuthFlowSlug,
+                AuthenticationFlow: authentikOpts.SourceAuthFlowSlug,
+                AllowIdpInitiated: false)); // REQUIRED: IdP-initiated SSO is a CSRF risk
+
+            if (!response.IsSuccessStatusCode || response.Content is null)
+                throw new InvalidOperationException(
+                    $"Failed to create SAML source: {response.Error?.Content ?? response.ReasonPhrase}");
+
+            return $"SAML source '{slug}' created with AllowIdpInitiated=false.";
+        }, ct);
+
+        // ── Step 3: store-metadata ──────────────────────────────────────────
+        await runner.RunAsync("store-metadata", async () =>
+        {
+            var byoiConfig = new ByoiConfig
+            {
+                Id = Guid.NewGuid(),
+                InstanceId = instance.Id,
+                Slug = slug,
+                EntityId = parms.CustomerIdpEntityId,
+                SsoUrl = parms.CustomerSsoUrl,
+                CertPem = parms.CustomerCertPem,
+                CertExpiry = certExpiry,
+                Enabled = true,
+                CreatedAt = DateTime.UtcNow,
+            };
+
+            db.ByoiConfigs.Add(byoiConfig);
+            await db.SaveChangesAsync(ct);
+
+            return $"ByoiConfig stored: slug={slug}, certExpiry={certExpiry:O}.";
+        }, ct);
+
+        // ── Step 4: schedule-cert-monitor ───────────────────────────────────
+        await runner.RunAsync("schedule-cert-monitor", async () =>
+        {
+            var scheduler = await schedulerFactory.GetScheduler(ct);
+            var jobKey = new JobKey($"byoi-cert-expiry-{abbrev}", "byoi-cert-expiry");
+
+            // Only schedule if not already present (idempotent)
+            if (!await scheduler.CheckExists(jobKey, ct))
+            {
+                var quartzJob = JobBuilder.Create<ByoiCertExpiryJob>()
+                    .WithIdentity(jobKey)
+                    .UsingJobData("instanceId", instance.Id.ToString())
+                    .Build();
+
+                var trigger = TriggerBuilder.Create()
+                    .WithIdentity($"byoi-cert-expiry-{abbrev}-trigger", "byoi-cert-expiry")
+                    .WithDailyTimeIntervalSchedule(s => s
+                        .OnEveryDay()
+                        .StartingDailyAt(TimeOfDay.HourAndMinuteOfDay(6, 0))
+                        .WithRepeatCount(1))
+                    .StartNow()
+                    .Build();
+
+                await scheduler.ScheduleJob(quartzJob, trigger, ct);
+            }
+
+            return $"Quartz ByoiCertExpiryJob scheduled daily for instance {instance.Id}.";
+        }, ct);
+    }
+
+    /// <summary>
+    /// Extracts the Base64 payload from a PEM string, stripping headers/footers.
+    /// </summary>
+    private static string ExtractBase64FromPem(string pem)
+    {
+        return pem
+            .Replace("-----BEGIN CERTIFICATE-----", "")
+            .Replace("-----END CERTIFICATE-----", "")
+            .Replace("\r", "")
+            .Replace("\n", "")
+            .Trim();
+    }
+}
+
+/// <summary>
+/// Deserialized from Job.Parameters JSON for provision-byoi jobs.
+/// </summary>
+public sealed record ByoiParameters
+{
+    public string CustomerCertPem { get; init; } = string.Empty;
+    public string CustomerSsoUrl { get; init; } = string.Empty;
+    public string CustomerIdpEntityId { get; init; } = string.Empty;
+    public string? CustomerSloUrl { get; init; }
+}

OTSSignsOrchestrator.Server/Workers/DecommissionPipeline.cs

@@ -0,0 +1,430 @@
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using Renci.SshNet;
+using OTSSignsOrchestrator.Core.Services;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// Full decommission pipeline — removes all infrastructure for a cancelled subscription.
+/// Handles <c>JobType = "decommission"</c>.
+///
+/// Steps:
+///   1. stack-remove      — docker stack rm xibo-{abbrev}
+///   2. authentik-cleanup — Delete SAML provider, application, 4 tenant groups (+BYOI source)
+///   3. oauth2-cleanup    — Delete Xibo OAuth2 application via API
+///   4. mysql-cleanup     — DROP DATABASE + DROP USER via SSH
+///   5. nfs-archive       — mv /nfs/{abbrev} → /nfs/archived/{abbrev}-{timestamp} (retain 30d min)
+///   6. registry-update   — Customer.Status = Decommissioned, Instance health = Critical,
+///                          final AuditLog, broadcast InstanceStatusChanged
+/// </summary>
+public sealed class DecommissionPipeline : IProvisioningPipeline
+{
+    public string HandlesJobType => "decommission";
+
+    private const int TotalSteps = 6;
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<DecommissionPipeline> _logger;
+
+    public DecommissionPipeline(
+        IServiceProvider services,
+        ILogger<DecommissionPipeline> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task ExecuteAsync(Job job, CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+        var authentikClient = scope.ServiceProvider.GetRequiredService<IAuthentikClient>();
+        var xiboFactory = scope.ServiceProvider.GetRequiredService<XiboClientFactory>();
+        var settings = scope.ServiceProvider.GetRequiredService<SettingsService>();
+
+        var ctx = await BuildContextAsync(job, db, ct);
+        var runner = new StepRunner(db, hub, _logger, job.Id, TotalSteps);
+
+        var abbrev = ctx.Abbreviation;
+        var stackName = ctx.DockerStackName;
+
+        // ── Step 1: stack-remove ────────────────────────────────────────────
+        await runner.RunAsync("stack-remove", async () =>
+        {
+            var sshHost = await GetSwarmSshHostAsync(settings);
+            using var sshClient = CreateSshClient(sshHost);
+            sshClient.Connect();
+
+            try
+            {
+                var result = RunSshCommand(sshClient, $"docker stack rm {stackName}");
+
+                db.AuditLogs.Add(new AuditLog
+                {
+                    Id = Guid.NewGuid(),
+                    InstanceId = ctx.InstanceId,
+                    Actor = "system/decommission",
+                    Action = "stack-remove",
+                    Target = stackName,
+                    Outcome = "success",
+                    Detail = $"Docker stack '{stackName}' removed. Job {job.Id}.",
+                    OccurredAt = DateTime.UtcNow,
+                });
+                await db.SaveChangesAsync(ct);
+
+                return $"Docker stack '{stackName}' removed. Output: {result}";
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }, ct);
+
+        // ── Step 2: authentik-cleanup ───────────────────────────────────────
+        await runner.RunAsync("authentik-cleanup", async () =>
+        {
+            var instance = await db.Instances
+                .Include(i => i.ByoiConfigs)
+                .FirstOrDefaultAsync(i => i.Id == ctx.InstanceId, ct)
+                ?? throw new InvalidOperationException($"Instance {ctx.InstanceId} not found.");
+
+            var cleaned = new List<string>();
+
+            // Delete SAML provider (stored on Instance.AuthentikProviderId)
+            if (!string.IsNullOrEmpty(instance.AuthentikProviderId)
+                && int.TryParse(instance.AuthentikProviderId, out var providerId))
+            {
+                await authentikClient.DeleteSamlProviderAsync(providerId);
+                cleaned.Add($"SAML provider {providerId}");
+            }
+
+            // Delete Authentik application
+            await authentikClient.DeleteApplicationAsync($"xibo-{abbrev}");
+            cleaned.Add($"application xibo-{abbrev}");
+
+            // Delete 4 tenant groups — search by name prefix, match by exact name
+            var groupNames = new[]
+            {
+                $"customer-{abbrev}",
+                $"customer-{abbrev}-viewer",
+                $"customer-{abbrev}-editor",
+                $"customer-{abbrev}-admin",
+            };
+
+            var searchResp = await authentikClient.ListGroupsAsync(search: $"customer-{abbrev}");
+            if (searchResp.IsSuccessStatusCode && searchResp.Content?.Results is { } groups)
+            {
+                foreach (var groupName in groupNames)
+                {
+                    var match = groups.FirstOrDefault(g =>
+                        g.TryGetValue("name", out var n) && n?.ToString() == groupName);
+
+                    if (match is not null && match.TryGetValue("pk", out var pk))
+                    {
+                        await authentikClient.DeleteGroupAsync(pk.ToString()!);
+                        cleaned.Add($"group {groupName}");
+                    }
+                }
+            }
+
+            // If BYOI was enabled, delete the SAML source
+            var byoiConfig = instance.ByoiConfigs.FirstOrDefault(b => b.Enabled);
+            if (byoiConfig is not null)
+            {
+                await authentikClient.DeleteSamlSourceAsync($"byoi-{abbrev}");
+                cleaned.Add($"BYOI SAML source byoi-{abbrev}");
+            }
+
+            db.AuditLogs.Add(new AuditLog
+            {
+                Id = Guid.NewGuid(),
+                InstanceId = ctx.InstanceId,
+                Actor = "system/decommission",
+                Action = "authentik-cleanup",
+                Target = $"xibo-{abbrev}",
+                Outcome = "success",
+                Detail = $"Cleaned up: {string.Join(", ", cleaned)}. Job {job.Id}.",
+                OccurredAt = DateTime.UtcNow,
+            });
+            await db.SaveChangesAsync(ct);
+
+            return $"Authentik cleanup completed: {string.Join(", ", cleaned)}.";
+        }, ct);
+
+        // ── Step 3: oauth2-cleanup ──────────────────────────────────────────
+        await runner.RunAsync("oauth2-cleanup", async () =>
+        {
+            var oauthReg = await db.OauthAppRegistries
+                .Where(r => r.InstanceId == ctx.InstanceId)
+                .OrderByDescending(r => r.CreatedAt)
+                .FirstOrDefaultAsync(ct);
+
+            if (oauthReg is null)
+                return "No OauthAppRegistry found — skipping OAuth2 cleanup.";
+
+            // Get Xibo client to delete the application
+            var oauthSecret = await settings.GetAsync(SettingsService.InstanceOAuthSecretId(abbrev));
+            if (string.IsNullOrEmpty(oauthSecret))
+                return $"OAuth secret not found for '{abbrev}' — cannot authenticate to delete app. Manual cleanup required.";
+
+            var xiboClient = await xiboFactory.CreateAsync(ctx.InstanceUrl, oauthReg.ClientId, oauthSecret);
+            await xiboClient.DeleteApplicationAsync(oauthReg.ClientId);
+
+            db.AuditLogs.Add(new AuditLog
+            {
+                Id = Guid.NewGuid(),
+                InstanceId = ctx.InstanceId,
+                Actor = "system/decommission",
+                Action = "oauth2-cleanup",
+                Target = oauthReg.ClientId,
+                Outcome = "success",
+                Detail = $"OAuth2 application '{oauthReg.ClientId}' deleted from Xibo. Job {job.Id}.",
+                OccurredAt = DateTime.UtcNow,
+            });
+            await db.SaveChangesAsync(ct);
+
+            return $"OAuth2 application '{oauthReg.ClientId}' deleted.";
+        }, ct);
+
+        // ── Step 4: mysql-cleanup ───────────────────────────────────────────
+        await runner.RunAsync("mysql-cleanup", async () =>
+        {
+            var mysqlHost = await settings.GetAsync(SettingsService.MySqlHost, "localhost");
+            var mysqlPort = await settings.GetAsync(SettingsService.MySqlPort, "3306");
+            var mysqlAdminUser = await settings.GetAsync(SettingsService.MySqlAdminUser, "root");
+            var mysqlAdminPassword = await settings.GetAsync(SettingsService.MySqlAdminPassword, string.Empty);
+
+            if (!int.TryParse(mysqlPort, out var port)) port = 3306;
+
+            var dbName = $"xibo_{abbrev}";
+            var userName = $"xibo_{abbrev}";
+
+            var sshHost = await GetSwarmSshHostAsync(settings);
+            using var sshClient = CreateSshClient(sshHost);
+            sshClient.Connect();
+
+            try
+            {
+                // DROP DATABASE
+                var dropDbCmd = $"mysql -h {mysqlHost} -P {port} -u {mysqlAdminUser} " +
+                    $"-p'{mysqlAdminPassword}' -e " +
+                    $"\"DROP DATABASE IF EXISTS \\`{dbName}\\`\"";
+                RunSshCommand(sshClient, dropDbCmd);
+
+                // DROP USER
+                var dropUserCmd = $"mysql -h {mysqlHost} -P {port} -u {mysqlAdminUser} " +
+                    $"-p'{mysqlAdminPassword}' -e " +
+                    $"\"DROP USER IF EXISTS '{userName}'@'%'\"";
+                RunSshCommand(sshClient, dropUserCmd);
+
+                db.AuditLogs.Add(new AuditLog
+                {
+                    Id = Guid.NewGuid(),
+                    InstanceId = ctx.InstanceId,
+                    Actor = "system/decommission",
+                    Action = "mysql-cleanup",
+                    Target = dbName,
+                    Outcome = "success",
+                    Detail = $"Database '{dbName}' and user '{userName}' dropped. Job {job.Id}.",
+                    OccurredAt = DateTime.UtcNow,
+                });
+                await db.SaveChangesAsync(ct);
+
+                return $"Database '{dbName}' and user '{userName}' dropped on {mysqlHost}:{port}.";
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }, ct);
+
+        // ── Step 5: nfs-archive ─────────────────────────────────────────────
+        await runner.RunAsync("nfs-archive", async () =>
+        {
+            var nfsServer = await settings.GetAsync(SettingsService.NfsServer);
+            var nfsExport = await settings.GetAsync(SettingsService.NfsExport);
+            var nfsExportFolder = await settings.GetAsync(SettingsService.NfsExportFolder);
+
+            if (string.IsNullOrWhiteSpace(nfsServer))
+                return "NFS server not configured — skipping archive.";
+
+            var export = (nfsExport ?? string.Empty).TrimEnd('/');
+            var folder = (nfsExportFolder ?? string.Empty).Trim('/');
+            var basePath = string.IsNullOrEmpty(folder) ? export : $"{export}/{folder}";
+
+            var timestamp = DateTime.UtcNow.ToString("yyyyMMddHHmmss");
+            var sourcePath = $"{basePath}/{abbrev}";
+            var archivePath = $"{basePath}/archived/{abbrev}-{timestamp}";
+
+            var sshHost = await GetSwarmSshHostAsync(settings);
+            using var sshClient = CreateSshClient(sshHost);
+            sshClient.Connect();
+
+            try
+            {
+                // Temporarily mount NFS to move directories
+                var mountPoint = $"/tmp/nfs-decommission-{abbrev}";
+                RunSshCommand(sshClient, $"sudo mkdir -p {mountPoint}");
+                RunSshCommand(sshClient, $"sudo mount -t nfs4 {nfsServer}:{basePath} {mountPoint}");
+
+                try
+                {
+                    // Ensure archive directory exists
+                    RunSshCommand(sshClient, $"sudo mkdir -p {mountPoint}/archived");
+                    // Move — DO NOT delete (retain for 30 days minimum)
+                    RunSshCommand(sshClient, $"sudo mv {mountPoint}/{abbrev} {mountPoint}/archived/{abbrev}-{timestamp}");
+                }
+                finally
+                {
+                    RunSshCommandAllowFailure(sshClient, $"sudo umount {mountPoint}");
+                    RunSshCommandAllowFailure(sshClient, $"sudo rmdir {mountPoint}");
+                }
+
+                db.AuditLogs.Add(new AuditLog
+                {
+                    Id = Guid.NewGuid(),
+                    InstanceId = ctx.InstanceId,
+                    Actor = "system/decommission",
+                    Action = "nfs-archive",
+                    Target = sourcePath,
+                    Outcome = "success",
+                    Detail = $"NFS data archived to {archivePath}. Retained for minimum 30 days. Job {job.Id}.",
+                    OccurredAt = DateTime.UtcNow,
+                });
+                await db.SaveChangesAsync(ct);
+
+                return $"NFS data moved from {sourcePath} to {archivePath}. Retained for 30+ days.";
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }, ct);
+
+        // ── Step 6: registry-update ─────────────────────────────────────────
+        await runner.RunAsync("registry-update", async () =>
+        {
+            var customer = await db.Customers.FirstOrDefaultAsync(c => c.Id == ctx.CustomerId, ct)
+                ?? throw new InvalidOperationException($"Customer {ctx.CustomerId} not found.");
+            var instance = await db.Instances.FirstOrDefaultAsync(i => i.Id == ctx.InstanceId, ct)
+                ?? throw new InvalidOperationException($"Instance {ctx.InstanceId} not found.");
+
+            customer.Status = CustomerStatus.Decommissioned;
+            instance.HealthStatus = HealthStatus.Critical;
+
+            db.AuditLogs.Add(new AuditLog
+            {
+                Id = Guid.NewGuid(),
+                InstanceId = ctx.InstanceId,
+                Actor = "system/decommission",
+                Action = "decommission-complete",
+                Target = $"xibo-{abbrev}",
+                Outcome = "success",
+                Detail = $"Instance fully decommissioned. Customer status → Decommissioned. Job {job.Id}.",
+                OccurredAt = DateTime.UtcNow,
+            });
+            await db.SaveChangesAsync(ct);
+
+            await hub.Clients.All.SendInstanceStatusChanged(
+                ctx.CustomerId.ToString(), CustomerStatus.Decommissioned.ToString());
+
+            return $"Customer '{abbrev}' → Decommissioned. Instance health → Critical. Broadcast sent.";
+        }, ct);
+
+        _logger.LogInformation("DecommissionPipeline completed for job {JobId} (abbrev={Abbrev})", job.Id, abbrev);
+    }
+
+    // ─────────────────────────────────────────────────────────────────────────
+    // Helpers (shared pattern from Phase1Pipeline)
+    // ─────────────────────────────────────────────────────────────────────────
+
+    private static async Task<PipelineContext> BuildContextAsync(Job job, OrchestratorDbContext db, CancellationToken ct)
+    {
+        var customer = await db.Customers
+            .Include(c => c.Instances)
+            .FirstOrDefaultAsync(c => c.Id == job.CustomerId, ct)
+            ?? throw new InvalidOperationException($"Customer {job.CustomerId} not found for job {job.Id}.");
+
+        var instance = customer.Instances.FirstOrDefault()
+            ?? throw new InvalidOperationException($"No instance found for customer {job.CustomerId}.");
+
+        var abbrev = customer.Abbreviation.ToLowerInvariant();
+
+        return new PipelineContext
+        {
+            JobId = job.Id,
+            CustomerId = customer.Id,
+            InstanceId = instance.Id,
+            Abbreviation = abbrev,
+            CompanyName = customer.CompanyName,
+            AdminEmail = customer.AdminEmail,
+            AdminFirstName = customer.AdminFirstName,
+            InstanceUrl = instance.XiboUrl,
+            DockerStackName = instance.DockerStackName,
+            ParametersJson = job.Parameters,
+        };
+    }
+
+    private static async Task<SshConnectionInfo> GetSwarmSshHostAsync(SettingsService settings)
+    {
+        var host = await settings.GetAsync("Ssh.SwarmHost")
+            ?? throw new InvalidOperationException("SSH Swarm host not configured (Ssh.SwarmHost).");
+        var portStr = await settings.GetAsync("Ssh.SwarmPort", "22");
+        var user = await settings.GetAsync("Ssh.SwarmUser", "root");
+        var keyPath = await settings.GetAsync("Ssh.SwarmKeyPath");
+        var password = await settings.GetAsync("Ssh.SwarmPassword");
+
+        if (!int.TryParse(portStr, out var port)) port = 22;
+
+        return new SshConnectionInfo(host, port, user, keyPath, password);
+    }
+
+    private static SshClient CreateSshClient(SshConnectionInfo info)
+    {
+        var authMethods = new List<AuthenticationMethod>();
+
+        if (!string.IsNullOrEmpty(info.KeyPath))
+            authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(info.KeyPath)));
+
+        if (!string.IsNullOrEmpty(info.Password))
+            authMethods.Add(new PasswordAuthenticationMethod(info.Username, info.Password));
+
+        if (authMethods.Count == 0)
+        {
+            var defaultKeyPath = Path.Combine(
+                Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".ssh", "id_rsa");
+            if (File.Exists(defaultKeyPath))
+                authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(defaultKeyPath)));
+            else
+                throw new InvalidOperationException(
+                    $"No SSH authentication method available for {info.Host}:{info.Port}.");
+        }
+
+        var connInfo = new Renci.SshNet.ConnectionInfo(info.Host, info.Port, info.Username, authMethods.ToArray());
+        return new SshClient(connInfo);
+    }
+
+    private static string RunSshCommand(SshClient client, string command)
+    {
+        using var cmd = client.RunCommand(command);
+        if (cmd.ExitStatus != 0)
+            throw new InvalidOperationException(
+                $"SSH command failed (exit {cmd.ExitStatus}): {cmd.Error}");
+        return cmd.Result;
+    }
+
+    private static void RunSshCommandAllowFailure(SshClient client, string command)
+    {
+        using var cmd = client.RunCommand(command);
+        // Intentionally ignore exit code — used for idempotent cleanup operations
+    }
+
+    internal sealed record SshConnectionInfo(
+        string Host, int Port, string Username, string? KeyPath, string? Password);
+}

OTSSignsOrchestrator.Server/Workers/IProvisioningPipeline.cs

@@ -0,0 +1,37 @@
+using OTSSignsOrchestrator.Server.Data.Entities;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// Common interface for all provisioning pipelines (Phase1, Phase2, etc.).
+/// Resolved from DI via <see cref="IEnumerable{IProvisioningPipeline}"/> and matched by
+/// <see cref="HandlesJobType"/>.
+/// </summary>
+public interface IProvisioningPipeline
+{
+    /// <summary>The <see cref="Job.JobType"/> this pipeline handles (e.g. "provision", "bootstrap").</summary>
+    string HandlesJobType { get; }
+
+    /// <summary>Execute the pipeline steps for the given job.</summary>
+    Task ExecuteAsync(Job job, CancellationToken ct);
+}
+
+/// <summary>
+/// Shared context extracted from <see cref="Job.Parameters"/> JSON and the associated
+/// <see cref="Customer"/> / <see cref="Instance"/> entities. Passed between pipeline steps.
+/// </summary>
+public sealed record PipelineContext
+{
+    public required Guid JobId { get; init; }
+    public required Guid CustomerId { get; init; }
+    public required Guid InstanceId { get; init; }
+    public required string Abbreviation { get; init; }
+    public required string CompanyName { get; init; }
+    public required string AdminEmail { get; init; }
+    public required string AdminFirstName { get; init; }
+    public required string InstanceUrl { get; init; }
+    public required string DockerStackName { get; init; }
+
+    /// <summary>Raw parameters JSON from the Job entity for step-specific overrides.</summary>
+    public string? ParametersJson { get; init; }
+}

OTSSignsOrchestrator.Server/Workers/Phase1Pipeline.cs

@@ -0,0 +1,494 @@
+using System.Security.Cryptography;
+using System.Text.Json;
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using Renci.SshNet;
+using OTSSignsOrchestrator.Core.Configuration;
+using OTSSignsOrchestrator.Core.Services;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+using static OTSSignsOrchestrator.Core.Configuration.AppConstants;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// Phase 1 provisioning pipeline — infrastructure setup. Handles <c>JobType = "provision"</c>.
+///
+/// Steps:
+///   1. mysql-setup        — Create DB + user on external MySQL via SSH tunnel
+///   2. docker-secrets     — Create Docker Swarm secrets via SSH
+///   3. nfs-dirs           — Create NFS sub-directories via SSH
+///   4. authentik-provision — SAML provider, application, groups, invitation flow in Authentik
+///   5. stack-deploy       — Render compose YAML and deploy via <c>docker stack deploy</c>
+///   6. credential-store   — Store generated credentials in Bitwarden Secrets Manager
+/// </summary>
+public sealed class Phase1Pipeline : IProvisioningPipeline
+{
+    public string HandlesJobType => "provision";
+
+    private const int TotalSteps = 6;
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<Phase1Pipeline> _logger;
+
+    public Phase1Pipeline(
+        IServiceProvider services,
+        ILogger<Phase1Pipeline> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task ExecuteAsync(Job job, CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+        var authentikClient = scope.ServiceProvider.GetRequiredService<IAuthentikClient>();
+        var composeRenderer = scope.ServiceProvider.GetRequiredService<ComposeRenderService>();
+        var gitService = scope.ServiceProvider.GetRequiredService<GitTemplateService>();
+        var bws = scope.ServiceProvider.GetRequiredService<IBitwardenSecretService>();
+        var settings = scope.ServiceProvider.GetRequiredService<SettingsService>();
+
+        var ctx = await BuildContextAsync(job, db, ct);
+        var runner = new StepRunner(db, hub, _logger, job.Id, TotalSteps);
+
+        var abbrev = ctx.Abbreviation;
+        var stackName = ctx.DockerStackName;
+
+        // Mutable state accumulated across steps
+        var mysqlPassword = GenerateRandomPassword(32);
+        var mysqlDbName = $"xibo_{abbrev}";
+        var mysqlUserName = $"xibo_{abbrev}";
+        string? authentikProviderId = null;
+
+        // ── Step 1: mysql-setup ─────────────────────────────────────────────
+        await runner.RunAsync("mysql-setup", async () =>
+        {
+            var mysqlHost = await settings.GetAsync(SettingsService.MySqlHost, "localhost");
+            var mysqlPort = await settings.GetAsync(SettingsService.MySqlPort, "3306");
+            var mysqlAdminUser = await settings.GetAsync(SettingsService.MySqlAdminUser, "root");
+            var mysqlAdminPassword = await settings.GetAsync(SettingsService.MySqlAdminPassword, string.Empty);
+
+            if (!int.TryParse(mysqlPort, out var port)) port = 3306;
+
+            // SSH to MySQL host, create database and user
+            // Reuse pattern from InstanceService.CreateMySqlDatabaseAsync — runs SQL
+            // via SSH tunnel to the MySQL server.
+            var sshHost = await GetSwarmSshHostAsync(settings);
+            using var sshClient = CreateSshClient(sshHost);
+            sshClient.Connect();
+
+            try
+            {
+                // Create database
+                var createDbCmd = $"mysql -h {mysqlHost} -P {port} -u {mysqlAdminUser} " +
+                    $"-p'{mysqlAdminPassword}' -e " +
+                    $"\"CREATE DATABASE IF NOT EXISTS \\`{mysqlDbName}\\`\"";
+                RunSshCommand(sshClient, createDbCmd);
+
+                // Create user
+                var createUserCmd = $"mysql -h {mysqlHost} -P {port} -u {mysqlAdminUser} " +
+                    $"-p'{mysqlAdminPassword}' -e " +
+                    $"\"CREATE USER IF NOT EXISTS '{mysqlUserName}'@'%' IDENTIFIED BY '{mysqlPassword}'\"";
+                RunSshCommand(sshClient, createUserCmd);
+
+                // Grant privileges
+                var grantCmd = $"mysql -h {mysqlHost} -P {port} -u {mysqlAdminUser} " +
+                    $"-p'{mysqlAdminPassword}' -e " +
+                    $"\"GRANT ALL PRIVILEGES ON \\`{mysqlDbName}\\`.* TO '{mysqlUserName}'@'%'; FLUSH PRIVILEGES;\"";
+                RunSshCommand(sshClient, grantCmd);
+
+                return $"Database '{mysqlDbName}' and user '{mysqlUserName}' created on {mysqlHost}:{port}.";
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }, ct);
+
+        // ── Step 2: docker-secrets ──────────────────────────────────────────
+        await runner.RunAsync("docker-secrets", async () =>
+        {
+            // Reuse IDockerSecretsService pattern — create secrets via SSH docker CLI
+            var sshHost = await GetSwarmSshHostAsync(settings);
+            using var sshClient = CreateSshClient(sshHost);
+            sshClient.Connect();
+
+            try
+            {
+                var secrets = new Dictionary<string, string>
+                {
+                    [CustomerMysqlPasswordSecretName(abbrev)] = mysqlPassword,
+                    [CustomerMysqlUserSecretName(abbrev)] = mysqlUserName,
+                    [GlobalMysqlHostSecretName] = await settings.GetAsync(SettingsService.MySqlHost, "localhost"),
+                    [GlobalMysqlPortSecretName] = await settings.GetAsync(SettingsService.MySqlPort, "3306"),
+                };
+
+                var created = new List<string>();
+                foreach (var (name, value) in secrets)
+                {
+                    // Remove existing secret if present (idempotent rotate)
+                    RunSshCommandAllowFailure(sshClient, $"docker secret rm {name}");
+
+                    var safeValue = value.Replace("'", "'\\''");
+                    var cmd = $"printf '%s' '{safeValue}' | docker secret create {name} -";
+                    RunSshCommand(sshClient, cmd);
+                    created.Add(name);
+                }
+
+                return $"Docker secrets created: {string.Join(", ", created)}.";
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }, ct);
+
+        // ── Step 3: nfs-dirs ────────────────────────────────────────────────
+        await runner.RunAsync("nfs-dirs", async () =>
+        {
+            var nfsServer = await settings.GetAsync(SettingsService.NfsServer);
+            var nfsExport = await settings.GetAsync(SettingsService.NfsExport);
+            var nfsExportFolder = await settings.GetAsync(SettingsService.NfsExportFolder);
+
+            if (string.IsNullOrWhiteSpace(nfsServer))
+                return "NFS server not configured — skipping directory creation.";
+
+            var sshHost = await GetSwarmSshHostAsync(settings);
+            using var sshClient = CreateSshClient(sshHost);
+            sshClient.Connect();
+
+            try
+            {
+                // Build the base path for NFS dirs
+                var export = (nfsExport ?? string.Empty).TrimEnd('/');
+                var folder = (nfsExportFolder ?? string.Empty).Trim('/');
+                var basePath = string.IsNullOrEmpty(folder) ? export : $"{export}/{folder}";
+
+                var subdirs = new[]
+                {
+                    $"{abbrev}/cms-custom",
+                    $"{abbrev}/cms-backup",
+                    $"{abbrev}/cms-library",
+                    $"{abbrev}/cms-userscripts",
+                    $"{abbrev}/cms-ca-certs",
+                };
+
+                // Create directories via sudo on the NFS host
+                // The swarm node mounts NFS, so we can create directories by
+                // temporarily mounting, creating, then unmounting.
+                var mountPoint = $"/tmp/nfs-provision-{abbrev}";
+                RunSshCommand(sshClient, $"sudo mkdir -p {mountPoint}");
+                RunSshCommand(sshClient,
+                    $"sudo mount -t nfs4 {nfsServer}:{basePath} {mountPoint}");
+
+                try
+                {
+                    foreach (var subdir in subdirs)
+                    {
+                        RunSshCommand(sshClient, $"sudo mkdir -p {mountPoint}/{subdir}");
+                    }
+                }
+                finally
+                {
+                    RunSshCommandAllowFailure(sshClient, $"sudo umount {mountPoint}");
+                    RunSshCommandAllowFailure(sshClient, $"sudo rmdir {mountPoint}");
+                }
+
+                return $"NFS directories created under {basePath}: {string.Join(", ", subdirs)}.";
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }, ct);
+
+        // ── Step 4: authentik-provision ─────────────────────────────────────
+        await runner.RunAsync("authentik-provision", async () =>
+        {
+            var instanceUrl = ctx.InstanceUrl;
+            var authFlowSlug = await settings.GetAsync(SettingsService.AuthentikAuthorizationFlowSlug, "default-provider-authorization-implicit-consent");
+            var signingKpId = await settings.GetAsync(SettingsService.AuthentikSigningKeypairId);
+
+            // 1. Create SAML provider
+            var providerResponse = await authentikClient.CreateSamlProviderAsync(new CreateSamlProviderRequest(
+                Name: $"xibo-{abbrev}",
+                AuthorizationFlow: authFlowSlug,
+                AcsUrl: $"{instanceUrl}/saml/acs",
+                Issuer: $"xibo-{abbrev}",
+                SpBinding: "post",
+                Audience: instanceUrl,
+                SigningKp: signingKpId));
+
+            EnsureSuccess(providerResponse);
+            var providerData = providerResponse.Content
+                ?? throw new InvalidOperationException("Authentik SAML provider creation returned empty response.");
+            authentikProviderId = providerData["pk"]?.ToString();
+
+            // 2. Create Authentik application linked to provider
+            var appResponse = await authentikClient.CreateApplicationAsync(new CreateAuthentikApplicationRequest(
+                Name: $"xibo-{abbrev}",
+                Slug: $"xibo-{abbrev}",
+                Provider: authentikProviderId!,
+                MetaLaunchUrl: instanceUrl));
+            EnsureSuccess(appResponse);
+
+            // 3. Create 4 tenant groups
+            var groupNames = new[]
+            {
+                $"customer-{abbrev}",
+                $"customer-{abbrev}-viewer",
+                $"customer-{abbrev}-editor",
+                $"customer-{abbrev}-admin",
+            };
+
+            foreach (var groupName in groupNames)
+            {
+                var groupResp = await authentikClient.CreateGroupAsync(new CreateAuthentikGroupRequest(
+                    Name: groupName,
+                    IsSuperuser: false,
+                    Parent: null));
+                EnsureSuccess(groupResp);
+            }
+
+            // 4. Create invitation flow
+            var inviteResponse = await authentikClient.CreateInvitationAsync(new CreateFlowRequest(
+                Name: $"invite-{abbrev}",
+                SingleUse: true,
+                Expires: DateTimeOffset.UtcNow.AddDays(30)));
+            EnsureSuccess(inviteResponse);
+
+            // Store Authentik provider ID on the Instance entity
+            var instance = await db.Instances.FirstOrDefaultAsync(i => i.Id == ctx.InstanceId, ct);
+            if (instance is not null)
+            {
+                instance.AuthentikProviderId = authentikProviderId;
+                await db.SaveChangesAsync(ct);
+            }
+
+            return $"Authentik SAML provider '{authentikProviderId}', application 'xibo-{abbrev}', " +
+                   $"4 groups, and invitation flow created.";
+        }, ct);
+
+        // ── Step 5: stack-deploy ────────────────────────────────────────────
+        await runner.RunAsync("stack-deploy", async () =>
+        {
+            // Fetch template
+            var repoUrl = await settings.GetAsync(SettingsService.GitRepoUrl);
+            var repoPat = await settings.GetAsync(SettingsService.GitRepoPat);
+
+            if (string.IsNullOrWhiteSpace(repoUrl))
+                throw new InvalidOperationException("Git template repository URL is not configured.");
+
+            var templateConfig = await gitService.FetchAsync(repoUrl, repoPat);
+
+            // Build render context
+            var renderCtx = new RenderContext
+            {
+                CustomerName = ctx.CompanyName,
+                CustomerAbbrev = abbrev,
+                StackName = stackName,
+                CmsServerName = await settings.GetAsync(SettingsService.DefaultCmsServerNameTemplate, "app.ots-signs.com"),
+                HostHttpPort = 80,
+                CmsImage = await settings.GetAsync(SettingsService.DefaultCmsImage, "ghcr.io/xibosignage/xibo-cms:release-4.2.3"),
+                MemcachedImage = await settings.GetAsync(SettingsService.DefaultMemcachedImage, "memcached:alpine"),
+                QuickChartImage = await settings.GetAsync(SettingsService.DefaultQuickChartImage, "ianw/quickchart"),
+                NewtImage = await settings.GetAsync(SettingsService.DefaultNewtImage, "fosrl/newt"),
+                ThemeHostPath = await settings.GetAsync(SettingsService.DefaultThemeHostPath, "/cms/ots-theme"),
+                MySqlHost = await settings.GetAsync(SettingsService.MySqlHost, "localhost"),
+                MySqlPort = await settings.GetAsync(SettingsService.MySqlPort, "3306"),
+                MySqlDatabase = mysqlDbName,
+                MySqlUser = mysqlUserName,
+                MySqlPassword = mysqlPassword,
+                SmtpServer = await settings.GetAsync(SettingsService.SmtpServer, string.Empty),
+                SmtpUsername = await settings.GetAsync(SettingsService.SmtpUsername, string.Empty),
+                SmtpPassword = await settings.GetAsync(SettingsService.SmtpPassword, string.Empty),
+                SmtpUseTls = await settings.GetAsync(SettingsService.SmtpUseTls, "YES"),
+                SmtpUseStartTls = await settings.GetAsync(SettingsService.SmtpUseStartTls, "YES"),
+                SmtpRewriteDomain = await settings.GetAsync(SettingsService.SmtpRewriteDomain, string.Empty),
+                SmtpHostname = await settings.GetAsync(SettingsService.SmtpHostname, string.Empty),
+                SmtpFromLineOverride = await settings.GetAsync(SettingsService.SmtpFromLineOverride, "NO"),
+                PhpPostMaxSize = await settings.GetAsync(SettingsService.DefaultPhpPostMaxSize, "10G"),
+                PhpUploadMaxFilesize = await settings.GetAsync(SettingsService.DefaultPhpUploadMaxFilesize, "10G"),
+                PhpMaxExecutionTime = await settings.GetAsync(SettingsService.DefaultPhpMaxExecutionTime, "600"),
+                PangolinEndpoint = await settings.GetAsync(SettingsService.PangolinEndpoint, "https://app.pangolin.net"),
+                NfsServer = await settings.GetAsync(SettingsService.NfsServer),
+                NfsExport = await settings.GetAsync(SettingsService.NfsExport),
+                NfsExportFolder = await settings.GetAsync(SettingsService.NfsExportFolder),
+                NfsExtraOptions = await settings.GetAsync(SettingsService.NfsOptions, string.Empty),
+            };
+
+            var composeYaml = composeRenderer.Render(templateConfig.Yaml, renderCtx);
+
+            // Deploy via SSH: pipe compose YAML to docker stack deploy
+            var sshHost = await GetSwarmSshHostAsync(settings);
+            using var sshClient = CreateSshClient(sshHost);
+            sshClient.Connect();
+
+            try
+            {
+                var safeYaml = composeYaml.Replace("'", "'\\''");
+                var deployCmd = $"printf '%s' '{safeYaml}' | docker stack deploy -c - {stackName}";
+                var result = RunSshCommand(sshClient, deployCmd);
+                return $"Stack '{stackName}' deployed. Output: {result}";
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }, ct);
+
+        // ── Step 6: credential-store ────────────────────────────────────────
+        await runner.RunAsync("credential-store", async () =>
+        {
+            // Bitwarden item structure:
+            //   Key:   "{abbrev}/mysql-password"    → MySQL password for xibo_{abbrev} user
+            //   Key:   "{abbrev}/mysql-database"    → Database name (for reference)
+            //   Key:   "{abbrev}/mysql-user"        → MySQL username (for reference)
+            // All stored in the instance Bitwarden project via IBitwardenSecretService.
+
+            await bws.CreateInstanceSecretAsync(
+                key: $"{abbrev}/mysql-password",
+                value: mysqlPassword,
+                note: $"MySQL password for instance {abbrev}. DB: {mysqlDbName}, User: {mysqlUserName}");
+
+            // Also persist in settings for Phase2 and future redeploys
+            await settings.SetAsync(
+                SettingsService.InstanceMySqlPassword(abbrev),
+                mysqlPassword,
+                SettingsService.CatInstance,
+                isSensitive: true);
+
+            // Clear in-memory password
+            mysqlPassword = string.Empty;
+
+            return $"Credentials stored in Bitwarden for instance '{abbrev}'.";
+        }, ct);
+
+        _logger.LogInformation("Phase1Pipeline completed for job {JobId} (abbrev={Abbrev})", job.Id, abbrev);
+    }
+
+    // ─────────────────────────────────────────────────────────────────────────
+    // Helpers
+    // ─────────────────────────────────────────────────────────────────────────
+
+    private static async Task<PipelineContext> BuildContextAsync(Job job, OrchestratorDbContext db, CancellationToken ct)
+    {
+        var customer = await db.Customers
+            .Include(c => c.Instances)
+            .FirstOrDefaultAsync(c => c.Id == job.CustomerId, ct)
+            ?? throw new InvalidOperationException($"Customer {job.CustomerId} not found for job {job.Id}.");
+
+        var instance = customer.Instances.FirstOrDefault()
+            ?? throw new InvalidOperationException($"No instance found for customer {job.CustomerId}.");
+
+        var abbrev = customer.Abbreviation.ToLowerInvariant();
+
+        return new PipelineContext
+        {
+            JobId = job.Id,
+            CustomerId = customer.Id,
+            InstanceId = instance.Id,
+            Abbreviation = abbrev,
+            CompanyName = customer.CompanyName,
+            AdminEmail = customer.AdminEmail,
+            AdminFirstName = customer.AdminFirstName,
+            InstanceUrl = instance.XiboUrl,
+            DockerStackName = instance.DockerStackName,
+            ParametersJson = job.Parameters,
+        };
+    }
+
+    /// <summary>
+    /// Reads SSH connection details for the Docker Swarm host from settings.
+    /// Expects settings keys: "Ssh.SwarmHost", "Ssh.SwarmPort", "Ssh.SwarmUser", "Ssh.SwarmKeyPath".
+    /// </summary>
+    private static async Task<SshConnectionInfo> GetSwarmSshHostAsync(SettingsService settings)
+    {
+        var host = await settings.GetAsync("Ssh.SwarmHost")
+            ?? throw new InvalidOperationException("SSH Swarm host not configured (Ssh.SwarmHost).");
+        var portStr = await settings.GetAsync("Ssh.SwarmPort", "22");
+        var user = await settings.GetAsync("Ssh.SwarmUser", "root");
+        var keyPath = await settings.GetAsync("Ssh.SwarmKeyPath");
+        var password = await settings.GetAsync("Ssh.SwarmPassword");
+
+        if (!int.TryParse(portStr, out var port)) port = 22;
+
+        return new SshConnectionInfo(host, port, user, keyPath, password);
+    }
+
+    private static SshClient CreateSshClient(SshConnectionInfo info)
+    {
+        var authMethods = new List<AuthenticationMethod>();
+
+        if (!string.IsNullOrEmpty(info.KeyPath))
+        {
+            authMethods.Add(new PrivateKeyAuthenticationMethod(
+                info.Username, new PrivateKeyFile(info.KeyPath)));
+        }
+
+        if (!string.IsNullOrEmpty(info.Password))
+        {
+            authMethods.Add(new PasswordAuthenticationMethod(info.Username, info.Password));
+        }
+
+        if (authMethods.Count == 0)
+        {
+            var defaultKeyPath = Path.Combine(
+                Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".ssh", "id_rsa");
+            if (File.Exists(defaultKeyPath))
+            {
+                authMethods.Add(new PrivateKeyAuthenticationMethod(
+                    info.Username, new PrivateKeyFile(defaultKeyPath)));
+            }
+            else
+            {
+                throw new InvalidOperationException(
+                    $"No SSH authentication method available for {info.Host}:{info.Port}. " +
+                    "Configure Ssh.SwarmKeyPath or Ssh.SwarmPassword in settings.");
+            }
+        }
+
+        var connInfo = new Renci.SshNet.ConnectionInfo(info.Host, info.Port, info.Username, authMethods.ToArray());
+        return new SshClient(connInfo);
+    }
+
+    private static string RunSshCommand(SshClient client, string command)
+    {
+        using var cmd = client.RunCommand(command);
+        if (cmd.ExitStatus != 0)
+            throw new InvalidOperationException(
+                $"SSH command failed (exit {cmd.ExitStatus}): {cmd.Error}");
+        return cmd.Result;
+    }
+
+    private static void RunSshCommandAllowFailure(SshClient client, string command)
+    {
+        using var cmd = client.RunCommand(command);
+        // Intentionally ignore exit code — used for idempotent cleanup operations
+    }
+
+    private static string GenerateRandomPassword(int length)
+    {
+        const string chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*";
+        return RandomNumberGenerator.GetString(chars, length);
+    }
+
+    private static void EnsureSuccess<T>(Refit.IApiResponse<T> response)
+    {
+        if (!response.IsSuccessStatusCode)
+            throw new InvalidOperationException(
+                $"API call failed with status {response.StatusCode}: {response.Error?.Content}");
+    }
+
+    /// <summary>SSH connection details read from Bitwarden settings.</summary>
+    internal sealed record SshConnectionInfo(
+        string Host,
+        int Port,
+        string Username,
+        string? KeyPath,
+        string? Password);
+}

OTSSignsOrchestrator.Server/Workers/Phase2Pipeline.cs

@@ -0,0 +1,479 @@
+using System.Security.Cryptography;
+using System.Text.Json;
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using Quartz;
+using OTSSignsOrchestrator.Core.Services;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+using OTSSignsOrchestrator.Server.Services;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// Phase 2 provisioning pipeline — Xibo CMS bootstrap. Handles <c>JobType = "bootstrap"</c>.
+/// Triggered after the stack deployed in Phase 1 becomes healthy.
+///
+/// Steps:
+///   1.  xibo-health-check       — Poll GET /about until 200 OK
+///   2.  create-ots-admin        — POST /api/user (superAdmin)
+///   3.  create-ots-svc          — POST /api/user (service account)
+///   4.  create-oauth2-app       — POST /api/application — IMMEDIATELY capture secret
+///   5.  create-groups           — POST /api/group (viewer, editor, admin, ots-it)
+///   6.  assign-group-acl        — POST /api/group/{id}/acl per role
+///   7.  assign-service-accounts — Assign admin + svc to ots-it group
+///   8.  apply-theme             — PUT /api/settings THEME_NAME=otssigns
+///   9.  delete-default-user     — Delete xibo_admin (after safety check)
+///  10.  schedule-snapshot       — Register Quartz DailySnapshotJob
+///  11.  authentik-invite        — Create Authentik user + invitation, send welcome email
+/// </summary>
+public sealed class Phase2Pipeline : IProvisioningPipeline
+{
+    public string HandlesJobType => "bootstrap";
+
+    private const int TotalSteps = 11;
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<Phase2Pipeline> _logger;
+
+    public Phase2Pipeline(
+        IServiceProvider services,
+        ILogger<Phase2Pipeline> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task ExecuteAsync(Job job, CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+        var xiboFactory = scope.ServiceProvider.GetRequiredService<XiboClientFactory>();
+        var authentikClient = scope.ServiceProvider.GetRequiredService<IAuthentikClient>();
+        var bws = scope.ServiceProvider.GetRequiredService<IBitwardenSecretService>();
+        var settings = scope.ServiceProvider.GetRequiredService<SettingsService>();
+        var emailService = scope.ServiceProvider.GetRequiredService<EmailService>();
+        var schedulerFactory = scope.ServiceProvider.GetRequiredService<ISchedulerFactory>();
+
+        var ctx = await BuildContextAsync(job, db, ct);
+        var runner = new StepRunner(db, hub, _logger, job.Id, TotalSteps);
+
+        var abbrev = ctx.Abbreviation;
+        var instanceUrl = ctx.InstanceUrl;
+
+        // The initial Xibo CMS ships with xibo_admin / password credentials.
+        // We use these to bootstrap via the API.
+        IXiboApiClient? xiboClient = null;
+
+        // Mutable state accumulated across steps
+        var adminPassword = GenerateRandomPassword(24);
+        var svcPassword = GenerateRandomPassword(24);
+        int otsAdminUserId = 0;
+        int otsSvcUserId = 0;
+        string otsOAuthClientId = string.Empty;
+        string otsOAuthClientSecret = string.Empty;
+        var groupIds = new Dictionary<string, int>(); // groupName → groupId
+
+        // ── Step 1: xibo-health-check ───────────────────────────────────────
+        await runner.RunAsync("xibo-health-check", async () =>
+        {
+            // Poll GET /about every 10 seconds, up to 5 minutes
+            var timeout = TimeSpan.FromMinutes(5);
+            var interval = TimeSpan.FromSeconds(10);
+            var deadline = DateTimeOffset.UtcNow.Add(timeout);
+
+            // Create a temporary client using default bootstrap credentials
+            // xibo_admin/password → OAuth client_credentials from the seed application
+            // The first call is to /about which doesn't need auth, use a raw HttpClient
+            using var httpClient = new HttpClient { BaseAddress = new Uri(instanceUrl.TrimEnd('/')) };
+
+            while (DateTimeOffset.UtcNow < deadline)
+            {
+                ct.ThrowIfCancellationRequested();
+                try
+                {
+                    var response = await httpClient.GetAsync("/api/about", ct);
+                    if (response.IsSuccessStatusCode)
+                    {
+                        return $"Xibo CMS at {instanceUrl} is healthy (status {(int)response.StatusCode}).";
+                    }
+                }
+                catch (HttpRequestException)
+                {
+                    // Not ready yet
+                }
+
+                await Task.Delay(interval, ct);
+            }
+
+            throw new TimeoutException(
+                $"Xibo CMS at {instanceUrl} did not return 200 OK from /about within {timeout.TotalMinutes} minutes.");
+        }, ct);
+
+        // Get a Refit client using the seed OAuth app credentials (xibo_admin bootstrap)
+        // Parameters JSON should contain bootstrapClientId + bootstrapClientSecret
+        var parameters = !string.IsNullOrEmpty(ctx.ParametersJson)
+            ? JsonDocument.Parse(ctx.ParametersJson)
+            : null;
+
+        var bootstrapClientId = parameters?.RootElement.TryGetProperty("bootstrapClientId", out var bcid) == true
+            ? bcid.GetString() ?? string.Empty
+            : string.Empty;
+        var bootstrapClientSecret = parameters?.RootElement.TryGetProperty("bootstrapClientSecret", out var bcs) == true
+            ? bcs.GetString() ?? string.Empty
+            : string.Empty;
+
+        if (string.IsNullOrEmpty(bootstrapClientId) || string.IsNullOrEmpty(bootstrapClientSecret))
+            throw new InvalidOperationException(
+                "Bootstrap OAuth credentials (bootstrapClientId, bootstrapClientSecret) must be provided in Job.Parameters.");
+
+        xiboClient = await xiboFactory.CreateAsync(instanceUrl, bootstrapClientId, bootstrapClientSecret);
+
+        // ── Step 2: create-ots-admin ────────────────────────────────────────
+        await runner.RunAsync("create-ots-admin", async () =>
+        {
+            var username = $"ots-admin-{abbrev}";
+            var email = $"ots-admin-{abbrev}@ots-signs.com";
+
+            var response = await xiboClient.CreateUserAsync(new CreateUserRequest(
+                UserName: username,
+                Email: email,
+                Password: adminPassword,
+                UserTypeId: 1, // SuperAdmin
+                HomePageId: 1));
+
+            EnsureSuccess(response);
+            var data = response.Content
+                ?? throw new InvalidOperationException("CreateUser returned empty response.");
+
+            otsAdminUserId = Convert.ToInt32(data["userId"]);
+            return $"Created user '{username}' (userId={otsAdminUserId}, type=SuperAdmin).";
+        }, ct);
+
+        // ── Step 3: create-ots-svc ──────────────────────────────────────────
+        await runner.RunAsync("create-ots-svc", async () =>
+        {
+            var username = $"ots-svc-{abbrev}";
+            var email = $"ots-svc-{abbrev}@ots-signs.com";
+
+            var response = await xiboClient.CreateUserAsync(new CreateUserRequest(
+                UserName: username,
+                Email: email,
+                Password: svcPassword,
+                UserTypeId: 1, // SuperAdmin — service account needs full API access
+                HomePageId: 1));
+
+            EnsureSuccess(response);
+            var data = response.Content
+                ?? throw new InvalidOperationException("CreateUser returned empty response.");
+
+            otsSvcUserId = Convert.ToInt32(data["userId"]);
+            return $"Created user '{username}' (userId={otsSvcUserId}, type=SuperAdmin).";
+        }, ct);
+
+        // ── Step 4: create-oauth2-app ───────────────────────────────────────
+        // CRITICAL: POST /api/application — NOT GET (that's blocked).
+        // The OAuth2 client secret is returned ONLY in this response. Capture immediately.
+        await runner.RunAsync("create-oauth2-app", async () =>
+        {
+            var appName = $"OTS Orchestrator — {abbrev.ToUpperInvariant()}";
+            var response = await xiboClient.CreateApplicationAsync(new CreateApplicationRequest(appName));
+
+            EnsureSuccess(response);
+            var data = response.Content
+                ?? throw new InvalidOperationException("CreateApplication returned empty response.");
+
+            // CRITICAL: Capture secret immediately — it cannot be retrieved again.
+            otsOAuthClientId = data["key"]?.ToString()
+                ?? throw new InvalidOperationException("OAuth application response missing 'key'.");
+            otsOAuthClientSecret = data["secret"]?.ToString()
+                ?? throw new InvalidOperationException("OAuth application response missing 'secret'.");
+
+            // Store to Bitwarden CLI wrapper
+            await bws.CreateInstanceSecretAsync(
+                key: $"{abbrev}/xibo-oauth-secret",
+                value: otsOAuthClientSecret,
+                note: $"Xibo CMS OAuth2 client secret for OTS application. ClientId: {otsOAuthClientId}");
+
+            // Store clientId ONLY in the database — NEVER store the secret in the DB
+            db.OauthAppRegistries.Add(new OauthAppRegistry
+            {
+                Id = Guid.NewGuid(),
+                InstanceId = ctx.InstanceId,
+                ClientId = otsOAuthClientId,
+                CreatedAt = DateTime.UtcNow,
+            });
+            await db.SaveChangesAsync(ct);
+
+            return $"OAuth2 app '{appName}' created. ClientId={otsOAuthClientId}. Secret stored in Bitwarden (NEVER in DB).";
+        }, ct);
+
+        // ── Step 5: create-groups ───────────────────────────────────────────
+        await runner.RunAsync("create-groups", async () =>
+        {
+            var groupDefs = new[]
+            {
+                ($"{abbrev}-viewer", "Viewer role"),
+                ($"{abbrev}-editor", "Editor role"),
+                ($"{abbrev}-admin", "Admin role"),
+                ($"ots-it-{abbrev}", "OTS IT internal"),
+            };
+
+            foreach (var (name, description) in groupDefs)
+            {
+                var response = await xiboClient.CreateGroupAsync(new CreateGroupRequest(name, description));
+                EnsureSuccess(response);
+                var data = response.Content
+                    ?? throw new InvalidOperationException($"CreateGroup '{name}' returned empty response.");
+                groupIds[name] = Convert.ToInt32(data["groupId"]);
+            }
+
+            return $"Created {groupDefs.Length} groups: {string.Join(", ", groupIds.Keys)}.";
+        }, ct);
+
+        // ── Step 6: assign-group-acl ────────────────────────────────────────
+        // POST /api/group/{id}/acl — NOT /features
+        await runner.RunAsync("assign-group-acl", async () =>
+        {
+            var aclMap = new Dictionary<string, (string[] ObjectIds, string[] PermissionIds)>
+            {
+                [$"{abbrev}-viewer"] = (XiboFeatureManifests.ViewerObjectIds, XiboFeatureManifests.ViewerPermissionIds),
+                [$"{abbrev}-editor"] = (XiboFeatureManifests.EditorObjectIds, XiboFeatureManifests.EditorPermissionIds),
+                [$"{abbrev}-admin"] = (XiboFeatureManifests.AdminObjectIds, XiboFeatureManifests.AdminPermissionIds),
+                [$"ots-it-{abbrev}"] = (XiboFeatureManifests.OtsItObjectIds, XiboFeatureManifests.OtsItPermissionIds),
+            };
+
+            var applied = new List<string>();
+            foreach (var (groupName, (objectIds, permissionIds)) in aclMap)
+            {
+                if (!groupIds.TryGetValue(groupName, out var groupId))
+                    throw new InvalidOperationException($"Group '{groupName}' ID not found.");
+
+                var response = await xiboClient.SetGroupAclAsync(groupId, new SetAclRequest(objectIds, permissionIds));
+                EnsureSuccess(response);
+                applied.Add($"{groupName} ({objectIds.Length} features)");
+            }
+
+            return $"ACL assigned: {string.Join(", ", applied)}.";
+        }, ct);
+
+        // ── Step 7: assign-service-accounts ─────────────────────────────────
+        await runner.RunAsync("assign-service-accounts", async () =>
+        {
+            var otsItGroupName = $"ots-it-{abbrev}";
+            if (!groupIds.TryGetValue(otsItGroupName, out var otsItGroupId))
+                throw new InvalidOperationException($"Group '{otsItGroupName}' ID not found.");
+
+            // Assign both ots-admin and ots-svc to the ots-it group
+            var response = await xiboClient.AssignUserToGroupAsync(
+                otsItGroupId,
+                new AssignMemberRequest([otsAdminUserId, otsSvcUserId]));
+            EnsureSuccess(response);
+
+            return $"Assigned ots-admin-{abbrev} (id={otsAdminUserId}) and ots-svc-{abbrev} (id={otsSvcUserId}) to group '{otsItGroupName}'.";
+        }, ct);
+
+        // ── Step 8: apply-theme ─────────────────────────────────────────────
+        await runner.RunAsync("apply-theme", async () =>
+        {
+            var response = await xiboClient.UpdateSettingsAsync(new UpdateSettingsRequest(
+                new Dictionary<string, string> { ["THEME_NAME"] = "otssigns" }));
+            EnsureSuccess(response);
+
+            return "Theme set to 'otssigns'.";
+        }, ct);
+
+        // ── Step 9: delete-default-user ─────────────────────────────────────
+        await runner.RunAsync("delete-default-user", async () =>
+        {
+            // First verify ots-admin works via OAuth2 client_credentials test
+            var testClient = await xiboFactory.CreateAsync(instanceUrl, otsOAuthClientId, otsOAuthClientSecret);
+            var aboutResponse = await testClient.GetAboutAsync();
+            if (!aboutResponse.IsSuccessStatusCode)
+                throw new InvalidOperationException(
+                    $"OAuth2 verification failed for ots-admin app (status {aboutResponse.StatusCode}). " +
+                    "Refusing to delete xibo_admin — it may be the only working admin account.");
+
+            // Use GetAllPagesAsync to find xibo_admin user — default page size is only 10
+            var allUsers = await xiboClient.GetAllPagesAsync(
+                (start, length) => xiboClient.GetUsersAsync(start, length));
+
+            var xiboAdminUser = allUsers.FirstOrDefault(u =>
+                u.TryGetValue("userName", out var name) &&
+                string.Equals(name?.ToString(), "xibo_admin", StringComparison.OrdinalIgnoreCase));
+
+            if (xiboAdminUser is null)
+                return "xibo_admin user not found — may have already been deleted.";
+
+            var xiboAdminId = Convert.ToInt32(xiboAdminUser["userId"]);
+
+            // Safety check: never delete if it would be the last SuperAdmin
+            var superAdminCount = allUsers.Count(u =>
+                u.TryGetValue("userTypeId", out var typeId) &&
+                Convert.ToInt32(typeId) == 1);
+
+            if (superAdminCount <= 1)
+                throw new InvalidOperationException(
+                    "Cannot delete xibo_admin — it is the only SuperAdmin. " +
+                    "Ensure ots-admin was created as SuperAdmin before retrying.");
+
+            await xiboClient.DeleteUserAsync(xiboAdminId);
+
+            return $"Deleted xibo_admin (userId={xiboAdminId}). Remaining SuperAdmins: {superAdminCount - 1}.";
+        }, ct);
+
+        // ── Step 10: schedule-snapshot ──────────────────────────────────────
+        await runner.RunAsync("schedule-snapshot", async () =>
+        {
+            var scheduler = await schedulerFactory.GetScheduler(ct);
+
+            var jobKey = new JobKey($"snapshot-{abbrev}", "daily-snapshots");
+            var triggerKey = new TriggerKey($"snapshot-trigger-{abbrev}", "daily-snapshots");
+
+            var quartzJob = JobBuilder.Create<DailySnapshotJob>()
+                .WithIdentity(jobKey)
+                .UsingJobData("abbrev", abbrev)
+                .UsingJobData("instanceId", ctx.InstanceId.ToString())
+                .StoreDurably()
+                .Build();
+
+            var trigger = TriggerBuilder.Create()
+                .WithIdentity(triggerKey)
+                .WithCronSchedule("0 0 2 * * ?") // 2:00 AM daily
+                .Build();
+
+            await scheduler.ScheduleJob(quartzJob, trigger, ct);
+
+            return $"DailySnapshotJob scheduled for instance '{abbrev}' — daily at 02:00 UTC.";
+        }, ct);
+
+        // ── Step 11: authentik-invite ───────────────────────────────────────
+        await runner.RunAsync("authentik-invite", async () =>
+        {
+            var adminEmail = ctx.AdminEmail;
+            var firstName = ctx.AdminFirstName;
+
+            // Create Authentik user
+            var userResponse = await authentikClient.CreateUserAsync(new CreateAuthentikUserRequest(
+                Username: adminEmail,
+                Name: $"{firstName}",
+                Email: adminEmail,
+                Groups: [$"customer-{abbrev}", $"customer-{abbrev}-viewer"]));
+            EnsureSuccess(userResponse);
+
+            // Create invitation with 7-day expiry
+            var inviteResponse = await authentikClient.CreateInvitationAsync(new CreateFlowRequest(
+                Name: $"invite-{abbrev}",
+                SingleUse: true,
+                Expires: DateTimeOffset.UtcNow.AddDays(7)));
+            EnsureSuccess(inviteResponse);
+
+            var inviteData = inviteResponse.Content;
+            var invitationLink = inviteData?.TryGetValue("pk", out var pk) == true
+                ? $"{instanceUrl}/if/flow/invite-{abbrev}/?itoken={pk}"
+                : "(invitation link unavailable)";
+
+            // Send welcome email
+            await emailService.SendWelcomeEmailAsync(adminEmail, firstName, instanceUrl, invitationLink);
+
+            return $"Authentik user '{adminEmail}' created, assigned to customer-{abbrev} + customer-{abbrev}-viewer. " +
+                   $"Invitation link: {invitationLink}. Welcome email sent.";
+        }, ct);
+
+        _logger.LogInformation("Phase2Pipeline completed for job {JobId} (abbrev={Abbrev})", job.Id, abbrev);
+    }
+
+    // ─────────────────────────────────────────────────────────────────────────
+    // Helpers
+    // ─────────────────────────────────────────────────────────────────────────
+
+    private static async Task<PipelineContext> BuildContextAsync(Job job, OrchestratorDbContext db, CancellationToken ct)
+    {
+        var customer = await db.Customers
+            .Include(c => c.Instances)
+            .FirstOrDefaultAsync(c => c.Id == job.CustomerId, ct)
+            ?? throw new InvalidOperationException($"Customer {job.CustomerId} not found for job {job.Id}.");
+
+        var instance = customer.Instances.FirstOrDefault()
+            ?? throw new InvalidOperationException($"No instance found for customer {job.CustomerId}.");
+
+        var abbrev = customer.Abbreviation.ToLowerInvariant();
+
+        return new PipelineContext
+        {
+            JobId = job.Id,
+            CustomerId = customer.Id,
+            InstanceId = instance.Id,
+            Abbreviation = abbrev,
+            CompanyName = customer.CompanyName,
+            AdminEmail = customer.AdminEmail,
+            AdminFirstName = customer.AdminFirstName,
+            InstanceUrl = instance.XiboUrl,
+            DockerStackName = instance.DockerStackName,
+            ParametersJson = job.Parameters,
+        };
+    }
+
+    private static string GenerateRandomPassword(int length)
+    {
+        const string chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*";
+        return RandomNumberGenerator.GetString(chars, length);
+    }
+
+    private static void EnsureSuccess<T>(Refit.IApiResponse<T> response)
+    {
+        if (!response.IsSuccessStatusCode)
+            throw new InvalidOperationException(
+                $"API call failed with status {response.StatusCode}: {response.Error?.Content}");
+    }
+}
+
+/// <summary>
+/// Quartz job placeholder for daily instance snapshots.
+/// Registered per-instance by Phase2Pipeline step 10.
+/// </summary>
+[DisallowConcurrentExecution]
+public sealed class DailySnapshotJob : IJob
+{
+    private readonly IServiceProvider _services;
+    private readonly ILogger<DailySnapshotJob> _logger;
+
+    public DailySnapshotJob(IServiceProvider services, ILogger<DailySnapshotJob> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task Execute(IJobExecutionContext context)
+    {
+        var abbrev = context.MergedJobDataMap.GetString("abbrev");
+        var instanceIdStr = context.MergedJobDataMap.GetString("instanceId");
+
+        _logger.LogInformation("DailySnapshotJob running for instance {Abbrev} (id={InstanceId})",
+            abbrev, instanceIdStr);
+
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+
+        if (!Guid.TryParse(instanceIdStr, out var instanceId))
+        {
+            _logger.LogError("DailySnapshotJob: invalid instanceId '{InstanceId}'", instanceIdStr);
+            return;
+        }
+
+        db.ScreenSnapshots.Add(new ScreenSnapshot
+        {
+            Id = Guid.NewGuid(),
+            InstanceId = instanceId,
+            SnapshotDate = DateOnly.FromDateTime(DateTime.UtcNow),
+            ScreenCount = 0,
+            CreatedAt = DateTime.UtcNow,
+        });
+        await db.SaveChangesAsync();
+
+        _logger.LogInformation("DailySnapshotJob completed for instance {Abbrev}", abbrev);
+    }
+}

OTSSignsOrchestrator.Server/Workers/ProvisioningWorker.cs

@@ -0,0 +1,127 @@
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// Background service that polls <see cref="OrchestratorDbContext.Jobs"/> for queued work,
+/// claims one job at a time, resolves the correct <see cref="IProvisioningPipeline"/>,
+/// and delegates execution. All transitions are logged and broadcast via SignalR.
+/// </summary>
+public sealed class ProvisioningWorker : BackgroundService
+{
+    private readonly IServiceProvider _services;
+    private readonly ILogger<ProvisioningWorker> _logger;
+
+    private static readonly TimeSpan PollInterval = TimeSpan.FromSeconds(5);
+
+    public ProvisioningWorker(
+        IServiceProvider services,
+        ILogger<ProvisioningWorker> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
+    {
+        _logger.LogInformation("ProvisioningWorker started — polling every {Interval}s", PollInterval.TotalSeconds);
+
+        using var timer = new PeriodicTimer(PollInterval);
+
+        while (await timer.WaitForNextTickAsync(stoppingToken))
+        {
+            try
+            {
+                await TryProcessNextJobAsync(stoppingToken);
+            }
+            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
+            {
+                break;
+            }
+            catch (Exception ex)
+            {
+                _logger.LogError(ex, "ProvisioningWorker: unhandled error during poll cycle");
+            }
+        }
+
+        _logger.LogInformation("ProvisioningWorker stopped");
+    }
+
+    private async Task TryProcessNextJobAsync(CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+
+        // Atomically claim the oldest queued job
+        var job = await db.Jobs
+            .Where(j => j.Status == JobStatus.Queued)
+            .OrderBy(j => j.CreatedAt)
+            .FirstOrDefaultAsync(ct);
+
+        if (job is null)
+            return;
+
+        // Optimistic concurrency: set Running + StartedAt
+        job.Status = JobStatus.Running;
+        job.StartedAt = DateTime.UtcNow;
+
+        try
+        {
+            await db.SaveChangesAsync(ct);
+        }
+        catch (DbUpdateConcurrencyException)
+        {
+            // Another worker already claimed this job
+            _logger.LogDebug("Job {JobId} was claimed by another worker", job.Id);
+            return;
+        }
+
+        _logger.LogInformation("Job {JobId} claimed (type={JobType}, customer={CustomerId})",
+            job.Id, job.JobType, job.CustomerId);
+
+        // Resolve the correct pipeline for this job type
+        var pipelines = scope.ServiceProvider.GetRequiredService<IEnumerable<IProvisioningPipeline>>();
+        var pipeline = pipelines.FirstOrDefault(p =>
+            string.Equals(p.HandlesJobType, job.JobType, StringComparison.OrdinalIgnoreCase));
+
+        if (pipeline is null)
+        {
+            _logger.LogError("No pipeline registered for job type '{JobType}' (job {JobId})", job.JobType, job.Id);
+            job.Status = JobStatus.Failed;
+            job.ErrorMessage = $"No pipeline registered for job type '{job.JobType}'.";
+            job.CompletedAt = DateTime.UtcNow;
+            await db.SaveChangesAsync(ct);
+            await hub.Clients.All.SendJobCompleted(job.Id.ToString(), false, job.ErrorMessage);
+            return;
+        }
+
+        try
+        {
+            await pipeline.ExecuteAsync(job, ct);
+
+            job.Status = JobStatus.Completed;
+            job.CompletedAt = DateTime.UtcNow;
+            await db.SaveChangesAsync(ct);
+
+            var summary = $"Job {job.JobType} completed for customer {job.CustomerId}.";
+            _logger.LogInformation("Job {JobId} completed successfully", job.Id);
+            await hub.Clients.All.SendJobCompleted(job.Id.ToString(), true, summary);
+        }
+        catch (Exception ex)
+        {
+            _logger.LogError(ex, "Job {JobId} failed: {Message}", job.Id, ex.Message);
+
+            job.Status = JobStatus.Failed;
+            job.ErrorMessage = ex.Message;
+            job.CompletedAt = DateTime.UtcNow;
+            await db.SaveChangesAsync(CancellationToken.None);
+
+            await hub.Clients.All.SendJobCompleted(job.Id.ToString(), false, ex.Message);
+        }
+    }
+}

OTSSignsOrchestrator.Server/Workers/ReactivatePipeline.cs

@@ -0,0 +1,229 @@
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using Renci.SshNet;
+using OTSSignsOrchestrator.Core.Services;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// Subscription reactivation pipeline — scales up Docker services, verifies health, resets
+/// payment failure counters. Handles <c>JobType = "reactivate"</c>.
+///
+/// Steps:
+///   1. scale-up       — SSH docker service scale web=1, xmr=1
+///   2. health-verify  — Poll GET /about every 10s up to 3 minutes
+///   3. update-status  — Customer.Status = Active, reset FailedPaymentCount + FirstPaymentFailedAt
+///   4. audit-log      — Append-only AuditLog entry
+///   5. broadcast      — InstanceStatusChanged via FleetHub
+/// </summary>
+public sealed class ReactivatePipeline : IProvisioningPipeline
+{
+    public string HandlesJobType => "reactivate";
+
+    private const int TotalSteps = 5;
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<ReactivatePipeline> _logger;
+
+    public ReactivatePipeline(
+        IServiceProvider services,
+        ILogger<ReactivatePipeline> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task ExecuteAsync(Job job, CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+        var settings = scope.ServiceProvider.GetRequiredService<SettingsService>();
+
+        var ctx = await BuildContextAsync(job, db, ct);
+        var runner = new StepRunner(db, hub, _logger, job.Id, TotalSteps);
+
+        var abbrev = ctx.Abbreviation;
+
+        // ── Step 1: scale-up ────────────────────────────────────────────────
+        await runner.RunAsync("scale-up", async () =>
+        {
+            var sshHost = await GetSwarmSshHostAsync(settings);
+            using var sshClient = CreateSshClient(sshHost);
+            sshClient.Connect();
+
+            try
+            {
+                var cmd = $"docker service scale xibo-{abbrev}_web=1 xibo-{abbrev}_xmr=1";
+                var result = RunSshCommand(sshClient, cmd);
+                return $"Scaled up services for xibo-{abbrev}. Output: {result}";
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }, ct);
+
+        // ── Step 2: health-verify ───────────────────────────────────────────
+        await runner.RunAsync("health-verify", async () =>
+        {
+            var timeout = TimeSpan.FromMinutes(3);
+            var interval = TimeSpan.FromSeconds(10);
+            var deadline = DateTimeOffset.UtcNow.Add(timeout);
+
+            using var httpClient = new HttpClient { BaseAddress = new Uri(ctx.InstanceUrl.TrimEnd('/')) };
+
+            while (DateTimeOffset.UtcNow < deadline)
+            {
+                ct.ThrowIfCancellationRequested();
+                try
+                {
+                    var response = await httpClient.GetAsync("/api/about", ct);
+                    if (response.IsSuccessStatusCode)
+                        return $"Xibo CMS at {ctx.InstanceUrl} is healthy (status {(int)response.StatusCode}).";
+                }
+                catch (HttpRequestException)
+                {
+                    // Not ready yet — keep polling
+                }
+
+                await Task.Delay(interval, ct);
+            }
+
+            throw new TimeoutException(
+                $"Xibo CMS at {ctx.InstanceUrl} did not return 200 OK from /about within {timeout.TotalMinutes} minutes.");
+        }, ct);
+
+        // ── Step 3: update-status ───────────────────────────────────────────
+        await runner.RunAsync("update-status", async () =>
+        {
+            var customer = await db.Customers.FirstOrDefaultAsync(c => c.Id == ctx.CustomerId, ct)
+                ?? throw new InvalidOperationException($"Customer {ctx.CustomerId} not found.");
+            var instance = await db.Instances.FirstOrDefaultAsync(i => i.Id == ctx.InstanceId, ct)
+                ?? throw new InvalidOperationException($"Instance {ctx.InstanceId} not found.");
+
+            customer.Status = CustomerStatus.Active;
+            customer.FailedPaymentCount = 0;
+            customer.FirstPaymentFailedAt = null;
+            instance.HealthStatus = HealthStatus.Healthy;
+            instance.LastHealthCheck = DateTime.UtcNow;
+            await db.SaveChangesAsync(ct);
+
+            return $"Customer '{abbrev}' status → Active, FailedPaymentCount → 0, instance health → Healthy.";
+        }, ct);
+
+        // ── Step 4: audit-log ───────────────────────────────────────────────
+        await runner.RunAsync("audit-log", async () =>
+        {
+            db.AuditLogs.Add(new AuditLog
+            {
+                Id = Guid.NewGuid(),
+                InstanceId = ctx.InstanceId,
+                Actor = "system/stripe-webhook",
+                Action = "reactivate",
+                Target = $"xibo-{abbrev}",
+                Outcome = "success",
+                Detail = $"Subscription reactivated. Services scaled to 1. Health verified. Job {job.Id}.",
+                OccurredAt = DateTime.UtcNow,
+            });
+            await db.SaveChangesAsync(ct);
+
+            return "AuditLog entry written for reactivation.";
+        }, ct);
+
+        // ── Step 5: broadcast ───────────────────────────────────────────────
+        await runner.RunAsync("broadcast", async () =>
+        {
+            await hub.Clients.All.SendInstanceStatusChanged(
+                ctx.CustomerId.ToString(), CustomerStatus.Active.ToString());
+            return "Broadcast InstanceStatusChanged → Active.";
+        }, ct);
+
+        _logger.LogInformation("ReactivatePipeline completed for job {JobId} (abbrev={Abbrev})", job.Id, abbrev);
+    }
+
+    // ─────────────────────────────────────────────────────────────────────────
+    // Helpers (shared pattern from Phase1Pipeline)
+    // ─────────────────────────────────────────────────────────────────────────
+
+    private static async Task<PipelineContext> BuildContextAsync(Job job, OrchestratorDbContext db, CancellationToken ct)
+    {
+        var customer = await db.Customers
+            .Include(c => c.Instances)
+            .FirstOrDefaultAsync(c => c.Id == job.CustomerId, ct)
+            ?? throw new InvalidOperationException($"Customer {job.CustomerId} not found for job {job.Id}.");
+
+        var instance = customer.Instances.FirstOrDefault()
+            ?? throw new InvalidOperationException($"No instance found for customer {job.CustomerId}.");
+
+        var abbrev = customer.Abbreviation.ToLowerInvariant();
+
+        return new PipelineContext
+        {
+            JobId = job.Id,
+            CustomerId = customer.Id,
+            InstanceId = instance.Id,
+            Abbreviation = abbrev,
+            CompanyName = customer.CompanyName,
+            AdminEmail = customer.AdminEmail,
+            AdminFirstName = customer.AdminFirstName,
+            InstanceUrl = instance.XiboUrl,
+            DockerStackName = instance.DockerStackName,
+            ParametersJson = job.Parameters,
+        };
+    }
+
+    private static async Task<SshConnectionInfo> GetSwarmSshHostAsync(SettingsService settings)
+    {
+        var host = await settings.GetAsync("Ssh.SwarmHost")
+            ?? throw new InvalidOperationException("SSH Swarm host not configured (Ssh.SwarmHost).");
+        var portStr = await settings.GetAsync("Ssh.SwarmPort", "22");
+        var user = await settings.GetAsync("Ssh.SwarmUser", "root");
+        var keyPath = await settings.GetAsync("Ssh.SwarmKeyPath");
+        var password = await settings.GetAsync("Ssh.SwarmPassword");
+
+        if (!int.TryParse(portStr, out var port)) port = 22;
+
+        return new SshConnectionInfo(host, port, user, keyPath, password);
+    }
+
+    private static SshClient CreateSshClient(SshConnectionInfo info)
+    {
+        var authMethods = new List<AuthenticationMethod>();
+
+        if (!string.IsNullOrEmpty(info.KeyPath))
+            authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(info.KeyPath)));
+
+        if (!string.IsNullOrEmpty(info.Password))
+            authMethods.Add(new PasswordAuthenticationMethod(info.Username, info.Password));
+
+        if (authMethods.Count == 0)
+        {
+            var defaultKeyPath = Path.Combine(
+                Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".ssh", "id_rsa");
+            if (File.Exists(defaultKeyPath))
+                authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(defaultKeyPath)));
+            else
+                throw new InvalidOperationException(
+                    $"No SSH authentication method available for {info.Host}:{info.Port}.");
+        }
+
+        var connInfo = new Renci.SshNet.ConnectionInfo(info.Host, info.Port, info.Username, authMethods.ToArray());
+        return new SshClient(connInfo);
+    }
+
+    private static string RunSshCommand(SshClient client, string command)
+    {
+        using var cmd = client.RunCommand(command);
+        if (cmd.ExitStatus != 0)
+            throw new InvalidOperationException(
+                $"SSH command failed (exit {cmd.ExitStatus}): {cmd.Error}");
+        return cmd.Result;
+    }
+
+    internal sealed record SshConnectionInfo(
+        string Host, int Port, string Username, string? KeyPath, string? Password);
+}

OTSSignsOrchestrator.Server/Workers/RotateCredentialsPipeline.cs

@@ -0,0 +1,274 @@
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Core.Services;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// OAuth2 credential rotation pipeline — deletes the old Xibo OAuth app, creates a new one,
+/// stores the new credentials, and verifies access. Handles <c>JobType = "rotate-oauth2"</c>.
+///
+/// CRITICAL: OAuth2 clientId CHANGES on rotation — there is no in-place secret refresh.
+/// The secret is returned ONLY in the <c>POST /api/application</c> response.
+///
+/// Steps:
+///   1. delete-old-app    — Delete current OAuth2 application via Xibo API
+///   2. create-new-app    — POST /api/application — IMMEDIATELY capture secret
+///   3. store-credentials — Update Bitwarden + OauthAppRegistry with new clientId
+///   4. verify-access     — Test new credentials via client_credentials flow
+///   5. audit-log         — Write AuditLog (or CRITICAL error with emergency creds if steps 3-4 fail)
+/// </summary>
+public sealed class RotateCredentialsPipeline : IProvisioningPipeline
+{
+    public string HandlesJobType => "rotate-oauth2";
+
+    private const int TotalSteps = 5;
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<RotateCredentialsPipeline> _logger;
+
+    public RotateCredentialsPipeline(
+        IServiceProvider services,
+        ILogger<RotateCredentialsPipeline> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task ExecuteAsync(Job job, CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+        var xiboFactory = scope.ServiceProvider.GetRequiredService<XiboClientFactory>();
+        var bws = scope.ServiceProvider.GetRequiredService<IBitwardenSecretService>();
+        var settings = scope.ServiceProvider.GetRequiredService<SettingsService>();
+
+        var ctx = await BuildContextAsync(job, db, ct);
+        var runner = new StepRunner(db, hub, _logger, job.Id, TotalSteps);
+
+        var abbrev = ctx.Abbreviation;
+
+        // Load current OAuth2 credentials
+        var currentReg = await db.OauthAppRegistries
+            .Where(r => r.InstanceId == ctx.InstanceId)
+            .OrderByDescending(r => r.CreatedAt)
+            .FirstOrDefaultAsync(ct)
+            ?? throw new InvalidOperationException(
+                $"No OauthAppRegistry found for instance {ctx.InstanceId}. Cannot rotate.");
+
+        var currentSecret = await settings.GetAsync(SettingsService.InstanceOAuthSecretId(abbrev))
+            ?? throw new InvalidOperationException(
+                $"OAuth secret not found in settings for instance '{abbrev}'. Cannot rotate.");
+
+        var currentClientId = currentReg.ClientId;
+
+        // Get a Xibo API client using the current (about-to-be-deleted) credentials
+        var xiboClient = await xiboFactory.CreateAsync(ctx.InstanceUrl, currentClientId, currentSecret);
+
+        // Mutable state — captured across steps
+        string newClientId = string.Empty;
+        string newSecret = string.Empty;
+
+        // ── Step 1: delete-old-app ──────────────────────────────────────────
+        await runner.RunAsync("delete-old-app", async () =>
+        {
+            await xiboClient.DeleteApplicationAsync(currentClientId);
+            return $"Old OAuth2 application deleted. ClientId={currentClientId}.";
+        }, ct);
+
+        // ── Step 2: create-new-app ──────────────────────────────────────────
+        // CRITICAL: After this step, the old credentials are gone. If subsequent steps fail,
+        // the new clientId + secret MUST be logged for manual recovery.
+        await runner.RunAsync("create-new-app", async () =>
+        {
+            // Need a client with some auth to create the new app. Use the instance's bootstrap
+            // admin credentials (ots-admin user) via direct login if available, or we can
+            // re-authenticate using the base URL. Since the old app is deleted, we need a
+            // different auth mechanism. The Xibo CMS allows creating apps as any authenticated
+            // super-admin. Use the ots-admin password from Bitwarden.
+            //
+            // However, the simplest path: the DELETE above invalidated the old token, but the
+            // Xibo CMS still has the ots-admin and ots-svc users. We stored the admin password
+            // in Bitwarden. Retrieve it and create a new factory client.
+            var adminPassBwsId = await settings.GetAsync(SettingsService.InstanceAdminPasswordSecretId(abbrev));
+            if (string.IsNullOrEmpty(adminPassBwsId))
+                throw new InvalidOperationException(
+                    $"Admin password Bitwarden secret ID not found for '{abbrev}'. Cannot create new OAuth app.");
+
+            var adminPassSecret = await bws.GetSecretAsync(adminPassBwsId);
+            var bootstrapClientId = await settings.GetAsync(SettingsService.XiboBootstrapClientId)
+                ?? throw new InvalidOperationException("Bootstrap OAuth client ID not configured.");
+            var bootstrapClientSecret = await settings.GetAsync(SettingsService.XiboBootstrapClientSecret)
+                ?? throw new InvalidOperationException("Bootstrap OAuth client secret not configured.");
+
+            // Re-create client using bootstrap credentials
+            var bootstrapClient = await xiboFactory.CreateAsync(ctx.InstanceUrl, bootstrapClientId, bootstrapClientSecret);
+
+            var appName = $"OTS Orchestrator — {abbrev.ToUpperInvariant()}";
+            var response = await bootstrapClient.CreateApplicationAsync(new CreateApplicationRequest(appName));
+
+            EnsureSuccess(response);
+            var data = response.Content
+                ?? throw new InvalidOperationException("CreateApplication returned empty response.");
+
+            // CRITICAL: Capture secret IMMEDIATELY — it is returned ONLY in this response.
+            newClientId = data["key"]?.ToString()
+                ?? throw new InvalidOperationException("OAuth application response missing 'key'.");
+            newSecret = data["secret"]?.ToString()
+                ?? throw new InvalidOperationException("OAuth application response missing 'secret'.");
+
+            return $"New OAuth2 application created. ClientId={newClientId}. Secret captured (stored in next step).";
+        }, ct);
+
+        // Steps 3-5: if ANY of these fail after step 2, we must log the credentials for recovery.
+        // The old app is already deleted — there is no rollback path.
+        try
+        {
+            // ── Step 3: store-credentials ───────────────────────────────────
+            await runner.RunAsync("store-credentials", async () =>
+            {
+                // Update Bitwarden with new secret
+                var existingBwsId = await settings.GetAsync(SettingsService.InstanceOAuthSecretId(abbrev));
+                if (!string.IsNullOrEmpty(existingBwsId))
+                {
+                    await bws.UpdateSecretAsync(
+                        existingBwsId,
+                        $"{abbrev}/xibo-oauth-secret",
+                        newSecret,
+                        $"Xibo CMS OAuth2 client secret (rotated). ClientId: {newClientId}");
+                }
+                else
+                {
+                    await bws.CreateInstanceSecretAsync(
+                        key: $"{abbrev}/xibo-oauth-secret",
+                        value: newSecret,
+                        note: $"Xibo CMS OAuth2 client secret (rotated). ClientId: {newClientId}");
+                }
+
+                // Update OauthAppRegistry with new clientId
+                db.OauthAppRegistries.Add(new OauthAppRegistry
+                {
+                    Id = Guid.NewGuid(),
+                    InstanceId = ctx.InstanceId,
+                    ClientId = newClientId,
+                    CreatedAt = DateTime.UtcNow,
+                });
+
+                // Also update the settings cache with the new client ID
+                await settings.SetAsync(
+                    SettingsService.InstanceOAuthClientId(abbrev),
+                    newClientId,
+                    SettingsService.CatInstance,
+                    isSensitive: false);
+
+                await db.SaveChangesAsync(ct);
+                return $"Credentials stored. ClientId={newClientId} in OauthAppRegistry + Bitwarden.";
+            }, ct);
+
+            // ── Step 4: verify-access ───────────────────────────────────────
+            await runner.RunAsync("verify-access", async () =>
+            {
+                var verifyClient = await xiboFactory.CreateAsync(ctx.InstanceUrl, newClientId, newSecret);
+                var aboutResp = await verifyClient.GetAboutAsync();
+                EnsureSuccess(aboutResp);
+                return $"New credentials verified. GET /about succeeded on {ctx.InstanceUrl}.";
+            }, ct);
+
+            // ── Step 5: audit-log ───────────────────────────────────────────
+            await runner.RunAsync("audit-log", async () =>
+            {
+                db.AuditLogs.Add(new AuditLog
+                {
+                    Id = Guid.NewGuid(),
+                    InstanceId = ctx.InstanceId,
+                    Actor = "system/credential-rotation",
+                    Action = "rotate-oauth2",
+                    Target = $"xibo-{abbrev}",
+                    Outcome = "success",
+                    Detail = $"OAuth2 credentials rotated. Old clientId={currentClientId} → new clientId={newClientId}. Job {job.Id}.",
+                    OccurredAt = DateTime.UtcNow,
+                });
+                await db.SaveChangesAsync(ct);
+
+                return $"AuditLog entry written. OAuth2 rotation complete.";
+            }, ct);
+        }
+        catch (Exception ex)
+        {
+            // CRITICAL: Steps 3-5 failed AFTER step 2 created the new app.
+            // The old credentials are already deleted. Log new credentials for manual recovery.
+            _logger.LogCritical(ex,
+                "CRITICAL — OAuth2 rotation for '{Abbrev}' failed after new app creation. " +
+                "Old clientId={OldClientId} (DELETED). " +
+                "New clientId={NewClientId}, New secret={NewSecret}. " +
+                "EMERGENCY RECOVERY DATA — store these credentials manually.",
+                abbrev, currentClientId, newClientId, newSecret);
+
+            // Also persist to a JobStep for operator visibility
+            var emergencyStep = new JobStep
+            {
+                Id = Guid.NewGuid(),
+                JobId = job.Id,
+                StepName = "emergency-credential-log",
+                Status = JobStepStatus.Failed,
+                StartedAt = DateTime.UtcNow,
+                CompletedAt = DateTime.UtcNow,
+                LogOutput = $"CRITICAL EMERGENCY RECOVERY DATA — OAuth2 rotation partial failure. " +
+                    $"Old clientId={currentClientId} (DELETED). " +
+                    $"New clientId={newClientId}. New secret={newSecret}. " +
+                    $"These credentials must be stored manually. Error: {ex.Message}",
+            };
+            db.JobSteps.Add(emergencyStep);
+            await db.SaveChangesAsync(CancellationToken.None);
+
+            throw; // Re-throw to fail the job
+        }
+
+        _logger.LogInformation(
+            "RotateCredentialsPipeline completed for job {JobId} (abbrev={Abbrev}, newClientId={ClientId})",
+            job.Id, abbrev, newClientId);
+    }
+
+    // ─────────────────────────────────────────────────────────────────────────
+    // Helpers
+    // ─────────────────────────────────────────────────────────────────────────
+
+    private static async Task<PipelineContext> BuildContextAsync(Job job, OrchestratorDbContext db, CancellationToken ct)
+    {
+        var customer = await db.Customers
+            .Include(c => c.Instances)
+            .FirstOrDefaultAsync(c => c.Id == job.CustomerId, ct)
+            ?? throw new InvalidOperationException($"Customer {job.CustomerId} not found for job {job.Id}.");
+
+        var instance = customer.Instances.FirstOrDefault()
+            ?? throw new InvalidOperationException($"No instance found for customer {job.CustomerId}.");
+
+        var abbrev = customer.Abbreviation.ToLowerInvariant();
+
+        return new PipelineContext
+        {
+            JobId = job.Id,
+            CustomerId = customer.Id,
+            InstanceId = instance.Id,
+            Abbreviation = abbrev,
+            CompanyName = customer.CompanyName,
+            AdminEmail = customer.AdminEmail,
+            AdminFirstName = customer.AdminFirstName,
+            InstanceUrl = instance.XiboUrl,
+            DockerStackName = instance.DockerStackName,
+            ParametersJson = job.Parameters,
+        };
+    }
+
+    private static void EnsureSuccess<T>(Refit.IApiResponse<T> response)
+    {
+        if (!response.IsSuccessStatusCode)
+            throw new InvalidOperationException(
+                $"API call failed with status {response.StatusCode}: {response.Error?.Content}");
+    }
+}

OTSSignsOrchestrator.Server/Workers/StepRunner.cs

@@ -0,0 +1,96 @@
+using Microsoft.AspNetCore.SignalR;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// Helper that wraps pipeline step execution with <see cref="JobStep"/> lifecycle management:
+/// creates the row, sets Running, captures output, marks Completed/Failed, and broadcasts
+/// progress via SignalR.
+/// </summary>
+public sealed class StepRunner
+{
+    private readonly OrchestratorDbContext _db;
+    private readonly IHubContext<FleetHub, IFleetClient> _hub;
+    private readonly ILogger _logger;
+    private readonly Guid _jobId;
+    private readonly int _totalSteps;
+    private int _currentStep;
+
+    public StepRunner(
+        OrchestratorDbContext db,
+        IHubContext<FleetHub, IFleetClient> hub,
+        ILogger logger,
+        Guid jobId,
+        int totalSteps)
+    {
+        _db = db;
+        _hub = hub;
+        _logger = logger;
+        _jobId = jobId;
+        _totalSteps = totalSteps;
+    }
+
+    /// <summary>
+    /// Execute a named step, persisting a <see cref="JobStep"/> record and broadcasting progress.
+    /// </summary>
+    /// <param name="stepName">Short identifier for the step (e.g. "mysql-setup").</param>
+    /// <param name="action">
+    /// Async delegate that performs the work. Return a log string summarising what happened.
+    /// </param>
+    /// <param name="ct">Cancellation token.</param>
+    public async Task RunAsync(string stepName, Func<Task<string>> action, CancellationToken ct)
+    {
+        _currentStep++;
+        var pct = (int)((double)_currentStep / _totalSteps * 100);
+
+        var step = new JobStep
+        {
+            Id = Guid.NewGuid(),
+            JobId = _jobId,
+            StepName = stepName,
+            Status = JobStepStatus.Running,
+            StartedAt = DateTime.UtcNow,
+        };
+
+        _db.JobSteps.Add(step);
+        await _db.SaveChangesAsync(ct);
+
+        _logger.LogInformation("Job {JobId}: step [{Step}/{Total}] {StepName} started",
+            _jobId, _currentStep, _totalSteps, stepName);
+
+        await _hub.Clients.All.SendJobProgressUpdate(
+            _jobId.ToString(), stepName, pct, $"Starting {stepName}…");
+
+        try
+        {
+            var logOutput = await action();
+
+            step.Status = JobStepStatus.Completed;
+            step.LogOutput = logOutput;
+            step.CompletedAt = DateTime.UtcNow;
+            await _db.SaveChangesAsync(ct);
+
+            _logger.LogInformation("Job {JobId}: step {StepName} completed", _jobId, stepName);
+
+            await _hub.Clients.All.SendJobProgressUpdate(
+                _jobId.ToString(), stepName, pct, logOutput);
+        }
+        catch (Exception ex)
+        {
+            step.Status = JobStepStatus.Failed;
+            step.LogOutput = ex.Message;
+            step.CompletedAt = DateTime.UtcNow;
+            await _db.SaveChangesAsync(CancellationToken.None);
+
+            _logger.LogError(ex, "Job {JobId}: step {StepName} failed", _jobId, stepName);
+
+            await _hub.Clients.All.SendJobProgressUpdate(
+                _jobId.ToString(), stepName, pct, $"FAILED: {ex.Message}");
+
+            throw; // re-throw to fail the job
+        }
+    }
+}

OTSSignsOrchestrator.Server/Workers/SuspendPipeline.cs

@@ -0,0 +1,195 @@
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using Renci.SshNet;
+using OTSSignsOrchestrator.Core.Services;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// Subscription suspension pipeline — scales down Docker services and marks status.
+/// Handles <c>JobType = "suspend"</c>.
+///
+/// Steps:
+///   1. scale-down     — SSH docker service scale web=0, xmr=0
+///   2. update-status  — Customer.Status = Suspended, Instance.HealthStatus = Degraded
+///   3. audit-log      — Append-only AuditLog entry
+///   4. broadcast      — InstanceStatusChanged via FleetHub
+/// </summary>
+public sealed class SuspendPipeline : IProvisioningPipeline
+{
+    public string HandlesJobType => "suspend";
+
+    private const int TotalSteps = 4;
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<SuspendPipeline> _logger;
+
+    public SuspendPipeline(
+        IServiceProvider services,
+        ILogger<SuspendPipeline> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task ExecuteAsync(Job job, CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+        var settings = scope.ServiceProvider.GetRequiredService<SettingsService>();
+
+        var ctx = await BuildContextAsync(job, db, ct);
+        var runner = new StepRunner(db, hub, _logger, job.Id, TotalSteps);
+
+        var abbrev = ctx.Abbreviation;
+
+        // ── Step 1: scale-down ──────────────────────────────────────────────
+        await runner.RunAsync("scale-down", async () =>
+        {
+            var sshHost = await GetSwarmSshHostAsync(settings);
+            using var sshClient = CreateSshClient(sshHost);
+            sshClient.Connect();
+
+            try
+            {
+                var cmd = $"docker service scale xibo-{abbrev}_web=0 xibo-{abbrev}_xmr=0";
+                var result = RunSshCommand(sshClient, cmd);
+                return $"Scaled down services for xibo-{abbrev}. Output: {result}";
+            }
+            finally
+            {
+                sshClient.Disconnect();
+            }
+        }, ct);
+
+        // ── Step 2: update-status ───────────────────────────────────────────
+        await runner.RunAsync("update-status", async () =>
+        {
+            var customer = await db.Customers.FirstOrDefaultAsync(c => c.Id == ctx.CustomerId, ct)
+                ?? throw new InvalidOperationException($"Customer {ctx.CustomerId} not found.");
+            var instance = await db.Instances.FirstOrDefaultAsync(i => i.Id == ctx.InstanceId, ct)
+                ?? throw new InvalidOperationException($"Instance {ctx.InstanceId} not found.");
+
+            customer.Status = CustomerStatus.Suspended;
+            instance.HealthStatus = HealthStatus.Degraded;
+            await db.SaveChangesAsync(ct);
+
+            return $"Customer '{abbrev}' status → Suspended, instance health → Degraded.";
+        }, ct);
+
+        // ── Step 3: audit-log ───────────────────────────────────────────────
+        await runner.RunAsync("audit-log", async () =>
+        {
+            db.AuditLogs.Add(new AuditLog
+            {
+                Id = Guid.NewGuid(),
+                InstanceId = ctx.InstanceId,
+                Actor = "system/stripe-webhook",
+                Action = "suspend",
+                Target = $"xibo-{abbrev}",
+                Outcome = "success",
+                Detail = $"Subscription suspended. Services scaled to 0. Job {job.Id}.",
+                OccurredAt = DateTime.UtcNow,
+            });
+            await db.SaveChangesAsync(ct);
+
+            return "AuditLog entry written for suspension.";
+        }, ct);
+
+        // ── Step 4: broadcast ───────────────────────────────────────────────
+        await runner.RunAsync("broadcast", async () =>
+        {
+            await hub.Clients.All.SendInstanceStatusChanged(
+                ctx.CustomerId.ToString(), CustomerStatus.Suspended.ToString());
+            return "Broadcast InstanceStatusChanged → Suspended.";
+        }, ct);
+
+        _logger.LogInformation("SuspendPipeline completed for job {JobId} (abbrev={Abbrev})", job.Id, abbrev);
+    }
+
+    // ─────────────────────────────────────────────────────────────────────────
+    // Helpers (shared pattern from Phase1Pipeline)
+    // ─────────────────────────────────────────────────────────────────────────
+
+    private static async Task<PipelineContext> BuildContextAsync(Job job, OrchestratorDbContext db, CancellationToken ct)
+    {
+        var customer = await db.Customers
+            .Include(c => c.Instances)
+            .FirstOrDefaultAsync(c => c.Id == job.CustomerId, ct)
+            ?? throw new InvalidOperationException($"Customer {job.CustomerId} not found for job {job.Id}.");
+
+        var instance = customer.Instances.FirstOrDefault()
+            ?? throw new InvalidOperationException($"No instance found for customer {job.CustomerId}.");
+
+        var abbrev = customer.Abbreviation.ToLowerInvariant();
+
+        return new PipelineContext
+        {
+            JobId = job.Id,
+            CustomerId = customer.Id,
+            InstanceId = instance.Id,
+            Abbreviation = abbrev,
+            CompanyName = customer.CompanyName,
+            AdminEmail = customer.AdminEmail,
+            AdminFirstName = customer.AdminFirstName,
+            InstanceUrl = instance.XiboUrl,
+            DockerStackName = instance.DockerStackName,
+            ParametersJson = job.Parameters,
+        };
+    }
+
+    private static async Task<SshConnectionInfo> GetSwarmSshHostAsync(SettingsService settings)
+    {
+        var host = await settings.GetAsync("Ssh.SwarmHost")
+            ?? throw new InvalidOperationException("SSH Swarm host not configured (Ssh.SwarmHost).");
+        var portStr = await settings.GetAsync("Ssh.SwarmPort", "22");
+        var user = await settings.GetAsync("Ssh.SwarmUser", "root");
+        var keyPath = await settings.GetAsync("Ssh.SwarmKeyPath");
+        var password = await settings.GetAsync("Ssh.SwarmPassword");
+
+        if (!int.TryParse(portStr, out var port)) port = 22;
+
+        return new SshConnectionInfo(host, port, user, keyPath, password);
+    }
+
+    private static SshClient CreateSshClient(SshConnectionInfo info)
+    {
+        var authMethods = new List<AuthenticationMethod>();
+
+        if (!string.IsNullOrEmpty(info.KeyPath))
+            authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(info.KeyPath)));
+
+        if (!string.IsNullOrEmpty(info.Password))
+            authMethods.Add(new PasswordAuthenticationMethod(info.Username, info.Password));
+
+        if (authMethods.Count == 0)
+        {
+            var defaultKeyPath = Path.Combine(
+                Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".ssh", "id_rsa");
+            if (File.Exists(defaultKeyPath))
+                authMethods.Add(new PrivateKeyAuthenticationMethod(info.Username, new PrivateKeyFile(defaultKeyPath)));
+            else
+                throw new InvalidOperationException(
+                    $"No SSH authentication method available for {info.Host}:{info.Port}.");
+        }
+
+        var connInfo = new Renci.SshNet.ConnectionInfo(info.Host, info.Port, info.Username, authMethods.ToArray());
+        return new SshClient(connInfo);
+    }
+
+    private static string RunSshCommand(SshClient client, string command)
+    {
+        using var cmd = client.RunCommand(command);
+        if (cmd.ExitStatus != 0)
+            throw new InvalidOperationException(
+                $"SSH command failed (exit {cmd.ExitStatus}): {cmd.Error}");
+        return cmd.Result;
+    }
+
+    internal sealed record SshConnectionInfo(
+        string Host, int Port, string Username, string? KeyPath, string? Password);
+}

OTSSignsOrchestrator.Server/Workers/UpdateScreenLimitPipeline.cs

@@ -0,0 +1,189 @@
+using System.Text.Json;
+using Microsoft.AspNetCore.SignalR;
+using Microsoft.EntityFrameworkCore;
+using OTSSignsOrchestrator.Server.Clients;
+using OTSSignsOrchestrator.Server.Data;
+using OTSSignsOrchestrator.Server.Data.Entities;
+using OTSSignsOrchestrator.Server.Hubs;
+using OTSSignsOrchestrator.Core.Services;
+
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// Updates the Xibo CMS screen limit and records a snapshot.
+/// Handles <c>JobType = "update-screen-limit"</c>.
+///
+/// Steps:
+///   1. update-settings — PUT /api/settings with new MAX_LICENSED_DISPLAYS
+///   2. update-snapshot — Record new screen count in ScreenSnapshots for today
+///   3. audit-log       — Append-only AuditLog entry
+/// </summary>
+public sealed class UpdateScreenLimitPipeline : IProvisioningPipeline
+{
+    public string HandlesJobType => "update-screen-limit";
+
+    private const int TotalSteps = 3;
+
+    private readonly IServiceProvider _services;
+    private readonly ILogger<UpdateScreenLimitPipeline> _logger;
+
+    public UpdateScreenLimitPipeline(
+        IServiceProvider services,
+        ILogger<UpdateScreenLimitPipeline> logger)
+    {
+        _services = services;
+        _logger = logger;
+    }
+
+    public async Task ExecuteAsync(Job job, CancellationToken ct)
+    {
+        await using var scope = _services.CreateAsyncScope();
+        var db = scope.ServiceProvider.GetRequiredService<OrchestratorDbContext>();
+        var hub = scope.ServiceProvider.GetRequiredService<IHubContext<FleetHub, IFleetClient>>();
+        var xiboFactory = scope.ServiceProvider.GetRequiredService<XiboClientFactory>();
+        var settings = scope.ServiceProvider.GetRequiredService<SettingsService>();
+
+        var ctx = await BuildContextAsync(job, db, ct);
+        var runner = new StepRunner(db, hub, _logger, job.Id, TotalSteps);
+
+        var abbrev = ctx.Abbreviation;
+
+        // Parse newScreenCount from Job.Parameters JSON
+        var newScreenCount = ParseScreenCount(ctx.ParametersJson)
+            ?? throw new InvalidOperationException(
+                "Job.Parameters must contain 'newScreenCount' (integer) for update-screen-limit.");
+
+        // Get Xibo API client via XiboClientFactory
+        var oauthReg = await db.OauthAppRegistries
+            .Where(r => r.InstanceId == ctx.InstanceId)
+            .OrderByDescending(r => r.CreatedAt)
+            .FirstOrDefaultAsync(ct)
+            ?? throw new InvalidOperationException(
+                $"No OauthAppRegistry found for instance {ctx.InstanceId}.");
+
+        var oauthSecret = await settings.GetAsync(SettingsService.InstanceOAuthSecretId(abbrev))
+            ?? throw new InvalidOperationException(
+                $"OAuth secret not found in settings for instance '{abbrev}'.");
+
+        var xiboClient = await xiboFactory.CreateAsync(ctx.InstanceUrl, oauthReg.ClientId, oauthSecret);
+
+        // ── Step 1: update-settings ─────────────────────────────────────────
+        await runner.RunAsync("update-settings", async () =>
+        {
+            var response = await xiboClient.UpdateSettingsAsync(new UpdateSettingsRequest(
+                new Dictionary<string, string>
+                {
+                    ["MAX_LICENSED_DISPLAYS"] = newScreenCount.ToString(),
+                }));
+
+            EnsureSuccess(response);
+            return $"Xibo MAX_LICENSED_DISPLAYS updated to {newScreenCount} for {ctx.InstanceUrl}.";
+        }, ct);
+
+        // ── Step 2: update-snapshot ─────────────────────────────────────────
+        await runner.RunAsync("update-snapshot", async () =>
+        {
+            var today = DateOnly.FromDateTime(DateTime.UtcNow);
+
+            // Upsert: if a snapshot already exists for today, update it; otherwise insert
+            var existing = await db.ScreenSnapshots
+                .FirstOrDefaultAsync(s => s.InstanceId == ctx.InstanceId && s.SnapshotDate == today, ct);
+
+            if (existing is not null)
+            {
+                existing.ScreenCount = newScreenCount;
+            }
+            else
+            {
+                db.ScreenSnapshots.Add(new ScreenSnapshot
+                {
+                    Id = Guid.NewGuid(),
+                    InstanceId = ctx.InstanceId,
+                    SnapshotDate = today,
+                    ScreenCount = newScreenCount,
+                    CreatedAt = DateTime.UtcNow,
+                });
+            }
+
+            // Also update Customer.ScreenCount to reflect the new limit
+            var customer = await db.Customers.FirstOrDefaultAsync(c => c.Id == ctx.CustomerId, ct);
+            if (customer is not null)
+                customer.ScreenCount = newScreenCount;
+
+            await db.SaveChangesAsync(ct);
+            return $"ScreenSnapshot recorded for {today}: {newScreenCount} screens.";
+        }, ct);
+
+        // ── Step 3: audit-log ───────────────────────────────────────────────
+        await runner.RunAsync("audit-log", async () =>
+        {
+            db.AuditLogs.Add(new AuditLog
+            {
+                Id = Guid.NewGuid(),
+                InstanceId = ctx.InstanceId,
+                Actor = "system/stripe-webhook",
+                Action = "update-screen-limit",
+                Target = $"xibo-{abbrev}",
+                Outcome = "success",
+                Detail = $"Screen limit updated to {newScreenCount}. Job {job.Id}.",
+                OccurredAt = DateTime.UtcNow,
+            });
+            await db.SaveChangesAsync(ct);
+
+            return "AuditLog entry written for screen limit update.";
+        }, ct);
+
+        _logger.LogInformation(
+            "UpdateScreenLimitPipeline completed for job {JobId} (abbrev={Abbrev}, screens={Count})",
+            job.Id, abbrev, newScreenCount);
+    }
+
+    // ─────────────────────────────────────────────────────────────────────────
+    // Helpers
+    // ─────────────────────────────────────────────────────────────────────────
+
+    private static int? ParseScreenCount(string? parametersJson)
+    {
+        if (string.IsNullOrEmpty(parametersJson)) return null;
+
+        using var doc = JsonDocument.Parse(parametersJson);
+        if (doc.RootElement.TryGetProperty("newScreenCount", out var prop) && prop.TryGetInt32(out var count))
+            return count;
+
+        return null;
+    }
+
+    private static async Task<PipelineContext> BuildContextAsync(Job job, OrchestratorDbContext db, CancellationToken ct)
+    {
+        var customer = await db.Customers
+            .Include(c => c.Instances)
+            .FirstOrDefaultAsync(c => c.Id == job.CustomerId, ct)
+            ?? throw new InvalidOperationException($"Customer {job.CustomerId} not found for job {job.Id}.");
+
+        var instance = customer.Instances.FirstOrDefault()
+            ?? throw new InvalidOperationException($"No instance found for customer {job.CustomerId}.");
+
+        var abbrev = customer.Abbreviation.ToLowerInvariant();
+
+        return new PipelineContext
+        {
+            JobId = job.Id,
+            CustomerId = customer.Id,
+            InstanceId = instance.Id,
+            Abbreviation = abbrev,
+            CompanyName = customer.CompanyName,
+            AdminEmail = customer.AdminEmail,
+            AdminFirstName = customer.AdminFirstName,
+            InstanceUrl = instance.XiboUrl,
+            DockerStackName = instance.DockerStackName,
+            ParametersJson = job.Parameters,
+        };
+    }
+
+    private static void EnsureSuccess<T>(Refit.IApiResponse<T> response)
+    {
+        if (!response.IsSuccessStatusCode)
+            throw new InvalidOperationException(
+                $"API call failed with status {response.StatusCode}: {response.Error?.Content}");
+    }
+}

OTSSignsOrchestrator.Server/Workers/XiboFeatureManifests.cs

@@ -0,0 +1,139 @@
+namespace OTSSignsOrchestrator.Server.Workers;
+
+/// <summary>
+/// Hardcoded Xibo feature ACL manifests per role.
+/// Used by Phase2Pipeline step "assign-group-acl" when calling
+/// <c>POST /api/group/{id}/acl</c>.
+///
+/// ObjectId is the feature key, PermissionsId is the permission level ("view", "edit", "delete").
+/// </summary>
+public static class XiboFeatureManifests
+{
+    /// <summary>Viewer role: read-only access to layouts, displays, media.</summary>
+    public static readonly string[] ViewerObjectIds =
+    [
+        "layout.view",
+        "media.view",
+        "display.view",
+        "schedule.view",
+        "report.view",
+    ];
+
+    public static readonly string[] ViewerPermissionIds =
+    [
+        "view",
+        "view",
+        "view",
+        "view",
+        "view",
+    ];
+
+    /// <summary>Editor role: view + edit for layouts, media, schedules.</summary>
+    public static readonly string[] EditorObjectIds =
+    [
+        "layout.view",
+        "layout.edit",
+        "media.view",
+        "media.edit",
+        "display.view",
+        "schedule.view",
+        "schedule.edit",
+        "report.view",
+    ];
+
+    public static readonly string[] EditorPermissionIds =
+    [
+        "view",
+        "edit",
+        "view",
+        "edit",
+        "view",
+        "view",
+        "edit",
+        "view",
+    ];
+
+    /// <summary>Admin role: full access to all features.</summary>
+    public static readonly string[] AdminObjectIds =
+    [
+        "layout.view",
+        "layout.edit",
+        "layout.delete",
+        "media.view",
+        "media.edit",
+        "media.delete",
+        "display.view",
+        "display.edit",
+        "display.delete",
+        "schedule.view",
+        "schedule.edit",
+        "schedule.delete",
+        "report.view",
+        "user.view",
+        "user.edit",
+    ];
+
+    public static readonly string[] AdminPermissionIds =
+    [
+        "view",
+        "edit",
+        "delete",
+        "view",
+        "edit",
+        "delete",
+        "view",
+        "edit",
+        "delete",
+        "view",
+        "edit",
+        "delete",
+        "view",
+        "view",
+        "edit",
+    ];
+
+    /// <summary>OTS IT group: full super-admin access (all features + user management).</summary>
+    public static readonly string[] OtsItObjectIds =
+    [
+        "layout.view",
+        "layout.edit",
+        "layout.delete",
+        "media.view",
+        "media.edit",
+        "media.delete",
+        "display.view",
+        "display.edit",
+        "display.delete",
+        "schedule.view",
+        "schedule.edit",
+        "schedule.delete",
+        "report.view",
+        "user.view",
+        "user.edit",
+        "user.delete",
+        "application.view",
+        "application.edit",
+    ];
+
+    public static readonly string[] OtsItPermissionIds =
+    [
+        "view",
+        "edit",
+        "delete",
+        "view",
+        "edit",
+        "delete",
+        "view",
+        "edit",
+        "delete",
+        "view",
+        "edit",
+        "delete",
+        "view",
+        "view",
+        "edit",
+        "delete",
+        "view",
+        "edit",
+    ];
+}

OTSSignsOrchestrator.sln

@@ -7,22 +7,70 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "OTSSignsOrchestrator.Core",
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "OTSSignsOrchestrator.Desktop", "OTSSignsOrchestrator.Desktop\OTSSignsOrchestrator.Desktop.csproj", "{B2C3D4E5-5555-6666-7777-888899990000}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "OTSSignsOrchestrator.Server", "OTSSignsOrchestrator.Server\OTSSignsOrchestrator.Server.csproj", "{C36D7809-5824-4AE0-912E-DBB18E05CF46}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "OTSSignsOrchestrator.Server.Tests", "OTSSignsOrchestrator.Server.Tests\OTSSignsOrchestrator.Server.Tests.csproj", "{452C671A-9730-44CF-A9B8-083CE36A4578}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
+		Debug|x64 = Debug|x64
+		Debug|x86 = Debug|x86
 		Release|Any CPU = Release|Any CPU
-	EndGlobalSection
-	GlobalSection(SolutionProperties) = preSolution
-		HideSolutionNode = FALSE
+		Release|x64 = Release|x64
+		Release|x86 = Release|x86
 	EndGlobalSection
 	GlobalSection(ProjectConfigurationPlatforms) = postSolution
 		{A1B2C3D4-1111-2222-3333-444455556666}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
 		{A1B2C3D4-1111-2222-3333-444455556666}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{A1B2C3D4-1111-2222-3333-444455556666}.Debug|x64.ActiveCfg = Debug|Any CPU
+		{A1B2C3D4-1111-2222-3333-444455556666}.Debug|x64.Build.0 = Debug|Any CPU
+		{A1B2C3D4-1111-2222-3333-444455556666}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{A1B2C3D4-1111-2222-3333-444455556666}.Debug|x86.Build.0 = Debug|Any CPU
 		{A1B2C3D4-1111-2222-3333-444455556666}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{A1B2C3D4-1111-2222-3333-444455556666}.Release|Any CPU.Build.0 = Release|Any CPU
+		{A1B2C3D4-1111-2222-3333-444455556666}.Release|x64.ActiveCfg = Release|Any CPU
+		{A1B2C3D4-1111-2222-3333-444455556666}.Release|x64.Build.0 = Release|Any CPU
+		{A1B2C3D4-1111-2222-3333-444455556666}.Release|x86.ActiveCfg = Release|Any CPU
+		{A1B2C3D4-1111-2222-3333-444455556666}.Release|x86.Build.0 = Release|Any CPU
 		{B2C3D4E5-5555-6666-7777-888899990000}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
 		{B2C3D4E5-5555-6666-7777-888899990000}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{B2C3D4E5-5555-6666-7777-888899990000}.Debug|x64.ActiveCfg = Debug|Any CPU
+		{B2C3D4E5-5555-6666-7777-888899990000}.Debug|x64.Build.0 = Debug|Any CPU
+		{B2C3D4E5-5555-6666-7777-888899990000}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{B2C3D4E5-5555-6666-7777-888899990000}.Debug|x86.Build.0 = Debug|Any CPU
 		{B2C3D4E5-5555-6666-7777-888899990000}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{B2C3D4E5-5555-6666-7777-888899990000}.Release|Any CPU.Build.0 = Release|Any CPU
+		{B2C3D4E5-5555-6666-7777-888899990000}.Release|x64.ActiveCfg = Release|Any CPU
+		{B2C3D4E5-5555-6666-7777-888899990000}.Release|x64.Build.0 = Release|Any CPU
+		{B2C3D4E5-5555-6666-7777-888899990000}.Release|x86.ActiveCfg = Release|Any CPU
+		{B2C3D4E5-5555-6666-7777-888899990000}.Release|x86.Build.0 = Release|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Debug|x64.ActiveCfg = Debug|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Debug|x64.Build.0 = Debug|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Debug|x86.Build.0 = Debug|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Release|Any CPU.Build.0 = Release|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Release|x64.ActiveCfg = Release|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Release|x64.Build.0 = Release|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Release|x86.ActiveCfg = Release|Any CPU
+		{C36D7809-5824-4AE0-912E-DBB18E05CF46}.Release|x86.Build.0 = Release|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Debug|x64.ActiveCfg = Debug|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Debug|x64.Build.0 = Debug|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Debug|x86.Build.0 = Debug|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Release|Any CPU.Build.0 = Release|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Release|x64.ActiveCfg = Release|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Release|x64.Build.0 = Release|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Release|x86.ActiveCfg = Release|Any CPU
+		{452C671A-9730-44CF-A9B8-083CE36A4578}.Release|x86.Build.0 = Release|Any CPU
+	EndGlobalSection
+	GlobalSection(SolutionProperties) = preSolution
+		HideSolutionNode = FALSE
 	EndGlobalSection
 EndGlobal

docker-compose.dev.yml

@@ -0,0 +1,15 @@
+services:
+  postgres:
+    image: postgres:16
+    restart: unless-stopped
+    environment:
+      POSTGRES_DB: orchestrator_dev
+      POSTGRES_USER: ots
+      POSTGRES_PASSWORD: devpassword
+    ports:
+      - "5432:5432"
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+
+volumes:
+  pgdata: