# PMI Data Sources and Update Timing
## Overview
PMI (Purchasing Managers Index) data is used for overlaying on HS300/SP500 valuation charts.
Three PMI series are tracked:
- **China PMI** (manufacturing + composite) from NBS
- **US Composite PMI** from S&P Global
- **US Manufacturing PMI** from S&P Global
## Data Source Hierarchy
### China PMI (monthly, NBS)
- **Source**: National Bureau of Statistics (stats.gov.cn)
- **Release**: Last day of each month (e.g., July 31 for July data)
- **Scraping**: `_try_update_cn_pmi_from_nbs()` in `quant_update.py` scrapes the NBS website
- **Cache**: `hs300/data/cn_pmi.csv` (columns: date, PMI010000=制造业, PMI030000=综合)
- **Initialization**: Tushare `cn_pmi` API for historical data, NBS scraping for latest month
- **Reliability**: ✅ Works well, NBS site is accessible from China
### US Composite PMI (monthly, S&P Global)
- **Primary source**: `pmi.spglobal.com` homepage (faster update, sometimes unstable)
- **Backup source**: Trading Economics `tradingeconomics.com/united-states/composite-pmi`
- **Cache**: `sp500/data/us_composite_pmi.csv` (columns: date, pmi)
- **Release**: Early each month (e.g., Aug 5 for July final)
- **TE lag**: TE can lag 1+ months behind S&P Global (as of 2026-08-11, TE still showed June while S&P Global had July)
### US Manufacturing PMI (monthly, S&P Global)
- **Primary source**: `pmi.spglobal.com` homepage
- **Backup source**: Trading Economics `tradingeconomics.com/united-states/manufacturing-pmi`
- **Cache**: `sp500/data/us_manufacturing_pmi.csv` (columns: date, pmi)
- **TE scraping bug**: TE Manufacturing PMI page regex `(\d+\.\d+)\s*points\s*in\s*(\w+)\s*of\s*(\d{4})` matches historical peaks (e.g., 63.4 July 2021) instead of current values. The TE page layout for Manufacturing PMI is different from Composite PMI. **Do not rely on TE for Manufacturing PMI.**
## S&P Global PMI Homepage Scraping Pattern
The S&P Global PMI homepage (`https://www.pmi.spglobal.com/`) contains current-month values in HTML spans:
```html
United States
COMPOSITE OUTPUT PMI
Jul: 53.6 ▲
United States
MANUFACTURING PMI
Jul: 53.8
```
**Regex pattern** (must account for HTML tags between country name and PMI type):
```python
# Composite
match = re.search(r'United States<.*?>COMPOSITE\s+OUTPUT\s+PMI<.*?>(\w{3}):\s*(\d+\.?\d*)', r.text)
# Manufacturing
match = re.search(r'United States<.*?>MANUFACTURING\s+PMI<.*?>(\w{3}):\s*(\d+\.?\d*)', r.text)
```
**⚠️ Site instability**: pmi.spglobal.com sometimes returns HTTP 202 with empty body (anti-bot). It works intermittently. Code must handle this gracefully and fall back to TE.
**⚠️ Proxy required**: pmi.spglobal.com is NOT directly accessible from China. Must use the user's HTTP proxy (`http://:10808`). The `quant_update.py` functions currently do NOT pass proxy, so they may fail silently — the TE fallback handles this.
## Trading Economics Scraping Pattern (backup)
TE pages are directly accessible from China without proxy:
```python
r = requests.get('https://tradingeconomics.com/united-states/composite-pmi',
headers={'User-Agent': 'Mozilla/5.0 ...'}, timeout=10)
# Pattern: "51.90 points in June of 2026"
match = re.search(r'(\d+\.\d+)\s*points\s*in\s*(\w+)\s*of\s*(\d{4})', r.text)
```
**Limitations**:
- TE lags 1+ months behind S&P Global for US PMI
- TE Manufacturing PMI regex matches wrong values (historical peaks)
- TE doesn't expose historical data without subscription
## ISM PMI vs S&P Global PMI
These are **different indicators** — do not mix them:
- **ISM Manufacturing PMI**: Based on ISM's own survey, released ~1st business day of month. Higher values (e.g., 55.6% in July 2026)
- **S&P Global Manufacturing PMI**: Based on S&P Global's survey (formerly Markit/IHS Markit). Lower values (e.g., 53.8 in July 2026)
- Our cache uses **S&P Global** series throughout (seed data + updates)
- ISM PMI can be found via Bing search: `"ISM Manufacturing PMI" "July 2026"` → ISM官网 confirms values like 55.6%
## Bing Search for Economic Data (requires proxy)
Bing from a Chinese IP returns Chinese-localized results even for English queries.
To get English financial search results:
```python
import subprocess, requests, re
# Get gateway for proxy
result = subprocess.run(['ip', 'route'], capture_output=True, text=True)
gateway = [l.split()[2] for l in result.stdout.split('\n') if 'default' in l][0]
proxy = f'http://{gateway}:10808'
proxies = {'http': proxy, 'https': proxy}
r = requests.get('https://www.bing.com/search',
params={'q': 'ISM Manufacturing PMI July 2026', 'cc': 'us', 'setlang': 'en'},
headers={'User-Agent': 'Mozilla/5.0 ...'},
timeout=15, proxies=proxies)
# Parse results as usual
```
Without proxy + `cc=us&setlang=en`, searching "ISM Manufacturing PMI July 2026" returns results about "ISM频段" and "us是什么意思" — completely useless.
## Update Functions in quant_update.py
Both `_try_update_us_pmi()` and `_try_update_us_mfg_pmi()` were updated (2026-08-11) to:
1. Try S&P Global homepage first (faster, needs proxy — currently no proxy passed, so may silently fail)
2. Fall back to Trading Economics (accessible from China, but may lag)
3. Log which source succeeded: `"added new data from S&P Global"` vs `"added new data from TE"`
## Manual Update Procedure
When automatic update fails (both sources unreachable or stale):
1. Check `pmi.spglobal.com` via proxy for latest month values
2. Cross-check with Bing search (proxy + `cc=us&setlang=en`)
3. ISM PMI values from Bing search can confirm direction (ISM and S&P Global move together)
4. Append to CSV: `pd.concat([df, new_row]).sort_values('date').to_csv(csv_path, index=False)`
5. Verify: `tail -5 sp500/data/us_composite_pmi.csv`
## Cache File Summary
| File | Series | Latest (as of 2026-08-11) | Next Update |
|------|--------|--------------------------|-------------|
| hs300/data/cn_pmi.csv | China Mfg+Composite | 2026-07 (49.2/49.3) | Aug 31 |
| sp500/data/us_composite_pmi.csv | US Composite (S&P Global) | 2026-07 (53.6) | Sep 1 |
| sp500/data/us_manufacturing_pmi.csv | US Mfg (S&P Global) | 2026-07 (53.8) | Sep 1 |