Troubleshoot Mideye Slow Startup Issues

Startup and initialization issues occur when Mideye Server’s service appears to be running but is not yet fully functional. This guide covers diagnosing and resolving timing-related startup problems.

Overview

When Mideye Server starts, it goes through several initialization phases:

1. Service start — Operating system starts the process
2. Configuration load — Reads configuration files
3. Database connection — Connects to the database and runs migrations
4. External service initialization — Connects to Switch, MAS, Shield
5. LDAP pool initialization — Establishes LDAP/AD connections
6. RADIUS listener start — Opens RADIUS ports for authentication
7. Web interface start — Web admin interface becomes available

Issues can occur at any phase, and later phases depend on earlier phases completing successfully.

---

Common Issues

1. Service Is Running but Not Responding

Problem: Service status shows “active (running)” but the web interface is unreachable and authentication doesn’t work.

Symptoms:

• systemctl status mideyeserver6 shows “active”
• Web interface at https://server:8080 doesn’t load
• RADIUS authentication times out
• No error messages in system logs

Root Cause: The service process started, but the application is still initializing (loading configuration, connecting to database, running migrations).

Diagnostic Steps:

Linux:

# Check service status
systemctl status mideyeserver6

# Check application log for startup progress
tail -f /opt/mideyeserver6/log/mideyeserver.log

# Check if web interface port is listening
ss -tlnp | grep 8080

# Check if RADIUS port is listening
ss -tlnp | grep 1812

Windows:

# Check service status
Get-Service MideyeServer6

# Check application log
Get-Content "C:\Program Files\Mideye Server\log\mideyeserver.log" -Tail 50 -Wait

# Check if ports are listening
Get-NetTCPConnection -LocalPort 8080 -ErrorAction SilentlyContinue
Get-NetTCPConnection -LocalPort 1812 -ErrorAction SilentlyContinue

Solution:

1. Wait for initialization — Startup can take 30–120 seconds depending on database size and network speed
2. Look for “Started MideyeServerApp” in the logs — this confirms full startup
3. If no progress after 5 minutes, check for errors in the log file
4. Restart the service if it appears stuck

---

2. External Service URLs Not Loaded

Problem: Switch or MAS connection fails at startup because URLs are not yet available from the database.

Symptoms:

• “Switch URL not configured” or “MAS URL not configured” in logs
• OTP delivery fails immediately after startup
• External service health checks fail
• Issue resolves itself after a few minutes

Root Cause: The service attempts to connect to external services before the database configuration is fully loaded.

Solution:

1. Wait for full initialization — External services typically become available within 1–2 minutes of startup

2. Verify configuration in the web interface:

   • Navigate to Mideye Authentication Service and verify MAS URL
   • Navigate to Mideye Switch and verify Switch URL

3. Check connectivity to external services:

   Linux:

   # Test Switch connectivity
   nc -zv primary.mideye.com 20460
   nc -zv secondary.mideye.com 20460
   
   # Test MAS connectivity
   curl -k https://mas.mideyecloud.se/health

   Windows:

   Test-NetConnection -ComputerName primary.mideye.com -Port 20460
   Test-NetConnection -ComputerName secondary.mideye.com -Port 20460

4. If services don’t become available after 5 minutes, check External Service Issues

---

3. LDAP Connections Not Ready

Problem: LDAP/AD lookups fail immediately after startup.

Symptoms:

• First few authentications fail with “User not found”
• LDAP errors in logs during first minutes
• Issue resolves after a few minutes

Root Cause: LDAP connection pool initializes lazily — connections are created on first use, not at startup.

Solution:

1. Wait for the first LDAP query to succeed — The connection pool establishes connections on first use

2. Verify LDAP configuration in LDAP Profiles

3. Test LDAP connectivity:

   Linux:

   nc -zv ldap.example.com 636

   Windows:

   Test-NetConnection -ComputerName ldap.example.com -Port 636

4. If LDAP issues persist, see LDAP Connection Issues

---

4. RADSEC Fails to Start

Problem: RADSEC (RADIUS over TLS) listener doesn’t start.

Symptoms:

• Port 2083 not listening after startup
• “Failed to start RADSEC server” in logs
• Regular RADIUS (ports 1812/1813) works fine

Root Cause:

• Certificate/keystore not available at startup
• Port 2083 already in use
• Certificate expired or invalid

Diagnostic Steps:

Linux:

# Check if port 2083 is in use
ss -tlnp | grep 2083

# Check if keystore exists
ls -la /opt/mideyeserver6/config/keystore.p12

Windows:

Get-NetTCPConnection -LocalPort 2083 -ErrorAction SilentlyContinue
Test-Path "C:\Program Files\Mideye Server\config\keystore.p12"

Solution:

1. Verify RADSEC certificate is configured in Certificate Management
2. Ensure port 2083 is available (not used by another process)
3. Check certificate validity
4. Restart Mideye Server

See Certificate and RADSEC Issues for detailed RADSEC troubleshooting.

---

5. Multi-Instance Race Conditions

Problem: In clustered or multi-instance deployments, multiple instances compete for resources during simultaneous startup.

Symptoms:

• Database migration lock contention (“Waiting for changelog lock”)
• One instance works, another doesn’t
• Intermittent startup failures in clustered environments

Solution:

1. Stagger startup: Start instances 60 seconds apart
2. Check for stuck database locks:

   SELECT * FROM DATABASECHANGELOGLOCK;

3. Release stuck lock if no migration is running:

   UPDATE DATABASECHANGELOGLOCK SET LOCKED=0, LOCKGRANTED=NULL, LOCKEDBY=NULL;

4. Restart the affected instance

---

Startup Verification Checklist

After starting Mideye Server, verify each phase:

Phase 1: Service Running

Linux:

systemctl is-active mideyeserver6
# Expected: active

Windows:

(Get-Service MideyeServer6).Status
# Expected: Running

Phase 2: Ports Listening

Linux:

ss -tlnp | grep -E "8080|1812|1813|2083"

Windows:

Get-NetTCPConnection -LocalPort 8080,1812,1813,2083 -ErrorAction SilentlyContinue |
  Select-Object LocalPort, State

Phase 3: Web Interface Accessible

Open https://your-server:8080 in a browser — login page should appear.

Phase 4: Application Fully Started

Check logs for the completion message:

• Look for “Started MideyeServerApp” in the Log Files page

Phase 5: External Services Connected

Navigate to external service configuration pages and verify connectivity:

• Mideye Switch — Connection status
• Mideye Authentication Service — Health status
• LDAP Profiles — Connection test

Phase 6: Authentication Test

Send a test RADIUS authentication to verify end-to-end functionality.

---

Health Check Script

Use these commands to quickly verify Mideye Server health:

Linux:

#!/bin/bash
echo "=== Mideye Server Health Check ==="

# Service status
echo -n "Service: "
systemctl is-active mideyeserver6

# Port checks
for port in 8080 1812 1813; do
  echo -n "Port $port: "
  ss -tlnp | grep -q ":$port " && echo "LISTENING" || echo "NOT LISTENING"
done

# Web interface
echo -n "Web UI: "
curl -sk -o /dev/null -w "%{http_code}" https://localhost:8080/login && echo "" || echo "UNREACHABLE"

# Log errors (last 5 minutes)
echo "Recent errors:"
grep -c "ERROR" /opt/mideyeserver6/log/mideyeserver.log 2>/dev/null || echo "No log file"

echo "=== Done ==="

Windows:

Write-Host "=== Mideye Server Health Check ==="

# Service status
$svc = Get-Service MideyeServer6 -ErrorAction SilentlyContinue
Write-Host "Service: $($svc.Status)"

# Port checks
foreach ($port in @(8080, 1812, 1813)) {
    $conn = Get-NetTCPConnection -LocalPort $port -ErrorAction SilentlyContinue
    $status = if ($conn) { "LISTENING" } else { "NOT LISTENING" }
    Write-Host "Port ${port}: $status"
}

# Web interface
try {
    $response = Invoke-WebRequest -Uri "https://localhost:8080/login" -SkipCertificateCheck -UseBasicParsing -TimeoutSec 5
    Write-Host "Web UI: $($response.StatusCode)"
} catch {
    Write-Host "Web UI: UNREACHABLE"
}

Write-Host "=== Done ==="

---

Expected Startup Times

Phase	Typical Time	Notes
Service start	5–10 seconds	Process initialization
Database connection	5–30 seconds	Depends on database location
Database migration	0–120 seconds	Only during upgrades
External services	10–30 seconds	Network dependent
Web interface	5–15 seconds	After database ready
Total cold start	30–120 seconds	Normal range

If startup consistently takes longer than 120 seconds, investigate database performance or network latency.

---

Getting Help

If startup issues persist, contact Mideye Support with:

• Service status output
• Application log from startup (from Log Files)
• How long the service has been stuck
• Whether this is a new installation or upgrade
• Deployment type (single instance or clustered)