Solar-VLM: Using Vision-Language Models to Forecast Solar Power by Fusing Satellite Images, Weather Text, and Time Series

A new framework called Solar-VLM applies large vision-language models to solar power forecasting, fusing three complementary data sources: time-series observations, satellite imagery, and textual weather information in a unified LLM-driven framework.

The Innovation

Previous AI-based solar forecasting methods typically use only numerical weather data and time series. Solar-VLM introduces multimodal fusion:

1. Time-series encoder — Patch-based design captures temporal patterns from multivariate observations at each solar site
2. Visual encoder — Built on Qwen vision backbone, extracts cloud-cover information from satellite images
3. Text encoder — Distills historical weather characteristics from textual weather descriptions

Why Multimodal?

Solar power generation is extremely sensitive to:

• Cloud cover — Satellite images provide spatial cloud motion prediction
• Weather patterns — Textual forecasts contain nuanced meteorological context
• Local conditions — On-site sensor time series capture microclimate effects

Combining all three provides complementary information that no single source offers.

Technical Approach

• Spatial dependency modeling across geographically distributed PV stations
• Modality-specific encoders for each data type
• Unified LLM framework for cross-modal reasoning
• Capture both local and regional patterns

Practical Impact

Accurate solar forecasting is critical for:

• Grid operators — Dispatch and stability management
• Energy markets — Bidding and trading strategies
• Renewable integration — Reducing curtailment of solar generation
• Battery optimization — Charge/discharge scheduling

This work shows that LLMs, traditionally used for text tasks, can effectively reason across visual and temporal modalities for physical world applications.