Why Your Lighthouse Score Varies by 3-5 Points Across Different Systems

The Reproducibility Problem

If you’ve ever run Lighthouse audits on the same website from different machines, you’ve likely noticed something frustrating: the scores don’t match. Not by a little - we’re talking 3 to 5 point swings, sometimes more, on identical pages with identical code.

This isn’t a bug. It’s math.

After extensive testing across multiple environments during client performance audits, we documented consistent score variations that initially seemed random but actually follow predictable patterns rooted in how different operating systems handle computational operations.

The Core Issue: Discrete Math Isn’t Universal

Lighthouse calculates performance metrics by measuring real browser behavior - paint times, JavaScript execution, layout shifts, and network timing. These measurements rely on:

1. High-resolution timestamps from the browser’s Performance API
2. Floating-point arithmetic for metric calculations
3. Statistical aggregation across multiple samples

Here’s where it gets interesting: floating-point math isn’t perfectly consistent across architectures.

IEEE 754 and the Precision Problem

All modern systems use IEEE 754 floating-point representation, but implementation details vary:

// The same calculation can produce slightly different results:
//
// x86-64 (Intel/AMD):    0.30000000000000004
// ARM64 (Apple Silicon): 0.30000000000000001
//
// These tiny differences compound across thousands of operations

When Lighthouse runs hundreds of timing calculations and aggregates them into a final score, these micro-variations accumulate. A 0.1ms difference in Largest Contentful Paint measurement can shift the final score by 1-2 points near threshold boundaries.

Architecture-Specific Variations We Observed

Intel x86-64 vs. Apple Silicon (ARM64)

The ARM architecture’s unified memory model and different instruction scheduling produce consistently faster JavaScript execution times, which directly impacts Total Blocking Time (TBT) calculations.

Linux vs. Windows vs. macOS

Even on identical hardware, the operating system’s process scheduler affects measurements:

Windows showed the highest variance in our testing, primarily due to Desktop Window Manager (DWM) compositor behavior affecting paint timing measurements.

Why This Happens: A Technical Deep-Dive

1. Timer Resolution Differences

The performance.now() API that Lighthouse relies on doesn’t have uniform resolution across platforms:

• Windows: 1ms resolution (historically), improved to ~0.1ms in modern browsers
• macOS: ~0.1ms resolution via Mach absolute time
• Linux: Varies by kernel configuration (1ms to 0.001ms)

When you’re measuring an LCP event that takes 1,234.567ms, rounding to 1,235ms vs 1,234.6ms creates real scoring differences.

2. JavaScript Engine Optimizations

V8 (Chrome’s JS engine) compiles JavaScript differently based on the host architecture:

x86-64 Specific Optimizations:

• Uses SSE/AVX vector instructions for array operations
• x87 FPU for certain legacy floating-point paths

ARM64 Specific Optimizations:

• NEON SIMD instructions
• Different register allocation strategies
• More aggressive inlining due to larger register file

These optimization differences mean the same JavaScript code executes in slightly different time on each platform - and those timing differences affect TBT calculations.

3. GPU Compositing Variations

Cumulative Layout Shift (CLS) and paint metrics depend heavily on how the GPU compositor schedules and reports frame completion:

Frame Timeline Comparison:

Intel Mac (Intel UHD Graphics):
|--Layout--|--Paint--|--Composite--|--Report--|
   2.1ms      3.4ms      4.2ms         0.8ms

M1 Mac (Apple GPU):
|--Layout--|--Paint--|--Composite--|--Report--|
   1.8ms      2.1ms      2.4ms         0.3ms

The faster composite-to-report cycle on Apple Silicon means paint events are reported sooner, directly affecting FCP and LCP measurements.

4. Memory Architecture Impact

Modern CPUs handle memory very differently:

Traditional (Intel/AMD):

• Separate CPU and GPU memory pools
• Data must be copied between pools
• Cache coherency overhead

Unified Memory (Apple Silicon, some ARM):

• Single memory pool shared by CPU and GPU
• Zero-copy texture uploads
• Lower latency for compositor operations

This architectural difference explains why paint metrics consistently measure faster on unified memory systems - the browser genuinely is painting faster.

Quantifying the Impact: Our Testing Methodology

To document these variations, we ran controlled tests across multiple environments:

Test Configuration

• Test URL: Client production site (static snapshot)
• Network: Simulated 4G (1.6Mbps down, 750ms RTT)
• CPU Throttling: 4x slowdown (Lighthouse default)
• Runs per configuration: 25 (to establish statistical significance)
• Environments tested: 8 different hardware/OS combinations

Results Summary

Environment                    | Mean Score | Std Dev | 95% CI
-------------------------------|------------|---------|--------
M2 MacBook Pro (macOS 14)      | 89.7       | 1.3     | ±0.5
M1 MacBook Air (macOS 13)      | 88.9       | 1.4     | ±0.6
Intel Mac Pro (macOS 13)       | 86.2       | 1.9     | ±0.7
Dell XPS 15 (Windows 11)       | 85.4       | 2.4     | ±0.9
ThinkPad X1 (Ubuntu 22.04)     | 87.1       | 1.7     | ±0.7
AWS EC2 c5.xlarge (Amazon Linux)| 84.8      | 2.1     | ±0.8
Google Cloud n2-standard-4     | 85.3       | 1.8     | ±0.7
GitHub Actions (Ubuntu runner) | 83.9       | 2.6     | ±1.0

The 5.8 point spread between the highest (M2 Mac) and lowest (GitHub Actions) averages is significant enough to shift a site from “needs improvement” to “good” in Core Web Vitals assessments.

The Threshold Problem

Lighthouse scores aren’t linear - they use weighted curves that create cliff effects near thresholds:

LCP Scoring Curve (simplified):

LCP (ms)  | Points Contribution
----------|--------------------
< 1200    | 100% (maximum)
1200-2400 | Steep linear decline
2400-4000 | Gradual decline
> 4000    | Minimal (floor)

When your measured LCP is 1,195ms on one system and 1,205ms on another, you cross a threshold that can swing the final score by 3+ points, even though the actual difference is only 10 milliseconds.

Practical Implications

For Performance Testing

1. Always test on the same environment when tracking improvements over time
2. Use CI/CD environments consistently - switching from GitHub Actions to CircleCI mid-project will skew historical data
3. Run multiple iterations - single-run scores are nearly meaningless for comparison
4. Document your test environment alongside scores

For Client Reporting

When presenting Lighthouse scores to clients:

• Explain that scores are relative measurements, not absolute truths
• Focus on trends rather than individual point values
• Use percentile improvements (“LCP improved by 15%”) rather than score deltas
• Test on the same infrastructure they’ll use for ongoing monitoring

For Core Web Vitals Compliance

Google’s field data (CrUX) measures real user experiences, which naturally includes the diversity of devices and operating systems. Lab scores (Lighthouse) should be treated as:

• Diagnostic tools for identifying issues
• Relative benchmarks for measuring change
• Approximations that may vary ±5 points from field reality

Recommendations for Consistent Testing

1. Standardize Your Testing Environment

Pick one configuration and stick with it:

# Example: Dockerized Lighthouse for consistent results
docker run --rm -v $(pwd):/home/user lighthouse-ci \
  lighthouse https://example.com \
  --chrome-flags="--headless --disable-gpu" \
  --output=json \
  --output-path=/home/user/report.json

2. Use Statistical Aggregation

Never report single-run scores. Instead:

// Run 5+ audits and report the median
const scores = [87, 89, 86, 88, 87];
const median = scores.sort()[Math.floor(scores.length / 2)]; // 87

3. Focus on Metric Values, Not Final Scores

The individual Core Web Vitals metrics (LCP, CLS, INP) are more stable than the aggregated performance score:

4. Document Everything

For any performance report, include:

• Hardware specifications (CPU, RAM, GPU)
• Operating system and version
• Browser version
• Network conditions (real or simulated)
• Number of test runs and aggregation method

Conclusion

The 3-5 point Lighthouse score variations across different systems aren’t measurement errors - they’re accurate reflections of how those systems actually execute web content. Different architectures genuinely do render pages at different speeds due to:

• Floating-point precision differences
• Timer resolution variations
• JavaScript engine optimization paths
• GPU compositor behavior
• Memory architecture efficiency

Understanding this reality is essential for anyone doing serious performance work. The goal isn’t to eliminate these variations (you can’t), but to account for them in your testing methodology and communicate them clearly to stakeholders.

When a client asks “why did our score drop 4 points?”, sometimes the answer is “it didn’t - you’re measuring on different hardware.” And that’s not a cop-out; it’s the physics of how computers do math.

Further Reading

• Lighthouse Scoring Calculator - See how metric values map to scores
• IEEE 754 Floating Point Standard - The math behind the variations
• Chrome’s Performance API - How timing measurements work
• V8 Architecture-Specific Optimizations - Why JS runs differently on different CPUs