Video Service Monitoring and Alerting: QoS Metrics, Log Analysis, Anomaly Detection

Video QoS Key Metrics

QoS (Quality of Service) is the core standard for measuring video service quality. Unlike regular web services, video services have their unique metric system that requires comprehensive evaluation from multiple dimensions such as playback experience, transmission quality, and system performance.

1. Stalling Rate / Buffer Starvation Rate

The stalling rate is the core metric for measuring video playback smoothness, referring to the number of buffering waits or the proportion of time during playback.

• Stall Count: Number of stalls per unit of playback time
• Stalling Rate: Total stall duration / total playback duration
• First Stall Time: Time from playback start to the first stall
• Excellent Standard: Stalling rate below 1%, fewer than 1 stall per hour

2. Time to First Frame (TTFP)

Time to first frame refers to the time from when the user clicks play to when the first video frame is displayed, directly impacting the user's initial experience.

• Components: DNS resolution + TCP connection + TLS handshake + request response + data buffering + decoding and rendering
• Excellent Standard: First frame time under 1 second, good at 1-2 seconds, poor over 3 seconds
• Influencing Factors: Network conditions, CDN distribution, player strategy, encoding parameters

3. Bitrate and Quality

• Average Bitrate: Average bitrate of actual playback
• Bitrate Switch Count: Frequency of adaptive bitrate switching
• Quality Experience: Average resolution distribution (1080p / 720p / 480p proportion)
• Peak Signal-to-Noise Ratio (PSNR): Objective video quality evaluation metric

4. Playback Success Rate

• Playback Success Rate: Proportion of requests that successfully start playback out of total requests
• Error Rate: Proportion of playback failures, categorized by error type
• Drop-off Rate: Proportion of users who exit within 5 seconds after playback starts
• Completion Rate: Proportion of users who watch the entire video

5. Live Streaming Specific Metrics

• Live Latency: Time difference from push end to playback end
• Online Viewers: Number of users currently watching the live stream
• Push Stream Quality: Frame rate and bitrate stability at the push end
• Instant Start Rate: Proportion of successful playback starts within 1 second

Log Collection and Analysis

Logs are the data foundation of video monitoring, requiring data collection from multiple levels such as client, server, and CDN.

Client-Side Instrumentation

Instrument at the player end to collect real user viewing experience data:

• Playback Events: play, pause, ended, seeking, buffering
• Error Events: network errors, decoding errors, format not supported
• Quality Data: current bitrate, buffer duration, frame rate, dropped frames
• User Behavior: playback duration, progress seeking, volume adjustment

Server-Side Logs

• Access Logs: Nginx / Apache access logs, recording detailed information for each request
• Application Logs: Business system logs, recording processing flows and errors
• Transcoding Logs: Detailed logs of video transcoding tasks, for troubleshooting processing failures
• Streaming Logs: RTMP/HLS server logs, recording push and pull stream situations

Log Collection Architecture

Client log collection flow:
1. Player SDK collects playback events and quality data
2. Local batch caching, periodic reporting (e.g., every 30 seconds)
3. Report to log collection service (e.g., Kafka)
4. Real-time computing layer performs aggregation and anomaly detection
5. Data written to time-series databases and data warehouses
6. Monitoring dashboard display and alert triggering

Anomaly Detection Algorithms

Traditional fixed-threshold alerting tends to produce false positives or false negatives. Using intelligent anomaly detection algorithms can identify problems more accurately.

1. Statistics-Based Methods

• 3σ Principle: Data deviating more than 3 standard deviations from the mean is considered abnormal
• Interquartile Range (IQR): Values beyond Q1-1.5*IQR or Q3+1.5*IQR are considered abnormal
• Exponential Moving Average (EMA): Detect deviations after smoothing time series

2. Period-over-Period Detection

Video service traffic usually has obvious periodicity. Comparing with historical periods can effectively detect anomalies:

• Year-over-Year (YoY): Compare with the same time on the same day last week
• Month-over-Month (MoM): Compare with the previous time period
• Dynamic Baseline: Automatically calculate normal fluctuation range based on historical data

3. Time Series Decomposition

Decompose time series into trend component, seasonal component, and residual component, then perform anomaly detection on the residuals:

• STL Decomposition: Seasonal and trend decomposition based on Loess
• Prophet: Facebook's open-source time series forecasting algorithm

4. Machine Learning Methods

• Isolation Forest: Suitable for anomaly detection on high-dimensional data
• LSTM Autoencoder: Learn normal patterns, with large reconstruction errors indicating anomalies
• Change Point Detection: Detect points where the time series distribution suddenly changes

Alerting Strategies

Alert Tiers

Divide alerts into different levels to avoid alert storms:

• P0 - Critical: Large-scale service unavailability, requires immediate response (within 5 minutes)
• P1 - Important: Partial function anomalies or small-scale failures, requires prompt handling (within 30 minutes)
• P2 - Normal: Minor issues or potential risks, handle during working hours
• P3 - Info: Informational notifications, for understanding system status

Alert Convergence and Aggregation

• Same-type Aggregation: Merge multiple alerts of the same category into one
• Time Window: Same alerts within a period only notify once
• Dependency Suppression: After a root cause alert triggers, suppress other alerts dependent on that service
• Escalation Mechanism: Alerts not handled for a long time automatically escalate to higher levels

Notification Methods

• Phone Call: Used for P0 level alerts
• SMS: Used for P0/P1 level alerts
• Instant Messaging: WeCom / DingTalk / Slack, used for P1/P2 levels
• Email: Used for P2/P3 levels

Monitoring Tool Recommendations

1. Prometheus + Grafana

The most popular open-source monitoring combination, suitable for metric-based monitoring:

• Prometheus: Time-series database with powerful PromQL query language
• Grafana: Visualization dashboard with rich chart types and alerting features
• Exporter: Metric collectors for various services, such as Node Exporter, Nginx Exporter

2. ELK / EFK Stack

Suitable for log-based monitoring and analysis:

• Elasticsearch: Distributed search engine for storing and searching logs
• Logstash / Fluentd: Log collection and processing pipelines
• Kibana: Log visualization and analysis interface

3. Video Quality Monitoring Tools

• Mux Data: SaaS service focused on video QoE monitoring
• Conviva: Intelligent video experience analytics platform
• NexGuard / Qligent: Broadcast-grade video quality monitoring

4. Other Common Tools

• Zabbix: Classic enterprise-level monitoring system
• Jaeger / Zipkin: Distributed tracing
• Sentry: Frontend error monitoring
• StatsD: Metric statistics service

User Experience Measurement

QoS metrics focus on the quality of the service itself, while QoE (Quality of Experience) focuses on the user's subjective perception.

QoE Evaluation Methods

• Subjective Rating: MOS (Mean Opinion Score) rating, scored by human viewers
• Objective Models: Predict subjective experience through algorithms, such as VMAF, SSIM, PSNR
• User Behavior: Infer experience from user behavior, such as drop-off rate, completion rate, rewatch rate

Key QoE Metrics

• Playback Success Rate: Whether playback can start successfully is the most basic experience
• First Frame Time: Faster entry into playback means better experience
• Stalling Rate: Smooth playback is the core experience
• Picture Quality: Balance between clarity and smoothness
• Interaction Latency: Timeliness of interaction in live streaming scenarios

Optimization Closed Loop

The ultimate goal of monitoring is to discover problems and continuously optimize, forming a complete closed loop:

1. Discover Problems

Discover anomalies through monitoring alerts, including:

• Real-time alert triggering
• Regular inspections and reports
• User feedback and complaints
• Competitor comparison analysis

2. Locate Problems

Quickly locate the root cause of problems:

• Troubleshoot layer by layer from client to server
• Comprehensive analysis combining metrics, logs, and tracing
• Use A/B testing to verify hypotheses

3. Solve Problems

• Develop and implement optimization solutions
• Verify effects with small traffic
• Full rollout and launch

4. Effect Validation

• Continuously observe metric changes after optimization goes live
• Compare data before and after optimization
• Summarize experience and improve the monitoring system

Summary

Video service monitoring is a systematic engineering effort that requires comprehensive construction from QoS metric definition, log collection, anomaly detection, and alerting strategies to optimization closed loop. The core is to build a quantifiable, monitorable, and optimizable quality assurance system centered around user experience.

It is recommended to start with core metrics (first frame time, stalling rate, playback success rate) and gradually improve monitoring coverage. Combine open-source tools (Prometheus, Grafana, ELK) with dedicated video quality monitoring services to build a monitoring system suitable for your own business. Monitoring is not the goal; continuously improving user experience is the ultimate goal of monitoring.

Want to learn more about video technology? Check out our HLS.js Usage Guide, or try our M3U8 Online Player to experience streaming media playback.