# ETF Combined CSV Data Quality — fund_share Zero-Day Anomalies

## Symptom

In `etf_*_combined.csv` (gold/tech/national-team/old-economy), `total_asset` shows a
one-day cliff (e.g. gold group 2025-07-08: 1421亿 → 544亿 → 1416亿 next day), while
`total_change` for that day is tiny (-0.08亿). User question pattern: "是不是份额拆分合并?"

## Diagnosis (split/merge vs data gap)

A share split/merge ALWAYS leaves a jump in NAV/close price. So:

1. Print each ETF's `asset_*` columns for ±3 days around the cliff.
2. If SOME ETFs show exactly 0 while others are normal → suspect data gap.
3. Check the `close_*` columns for the same days: if prices are CONTINUOUS (no jump),
   it is NOT a split/merge — it is tushare fund_share returning 0 for those ETFs that day.
4. The tiny `total_change` confirms it: it only sums the ETFs that returned real data;
   the zeroed ETFs contribute 0 change and 0 asset, deflating total_asset.

Gold case (2025-07-08): 518880/518800/517520/518850 zeroed; 159937/159934 fine;
518880 close 7.355→7.355→7.314 continuous ⇒ data gap, not split.

## Full-history scan for zero-days

```python
import pandas as pd
df = pd.read_csv('etf_gold_combined.csv', dtype={'trade_date': str})
asset_cols = [c for c in df.columns if c.startswith('asset_')]
anoms = []
for i in range(1, len(df)):
    for c in asset_cols:
        prev = pd.to_numeric(df[c].iloc[i-1], errors='coerce')
        cur  = pd.to_numeric(df[c].iloc[i],   errors='coerce')
        if pd.notna(prev) and prev > 1 and (cur == 0 or pd.isna(cur)):
            anoms.append((df['trade_date'].iloc[i], c.replace('asset_',''))
print(anoms)
```

## Fix (linear interpolation)

For each zeroed asset column on the bad day: set asset = (prev_day + next_day)/2,
set its `change_*` = asset - prev_day, then recompute
`total_asset = sum(asset_*)` and `total_change = sum(change_*)` for ALL rows
(or at least verify they are consistent). Back up the CSV first
(`cp etf_gold_combined.csv etf_gold_combined.csv.bak_YYYYMMDD`).

After fix the cliff disappears (gold 7/8: total_asset 1422.68亿, total_change -1.80亿)
and downstream panels (percentile, stacked flows, net-asset curve) are clean.

## Prevention

- The cron fetch scripts (fetch_*_etf_data.py) call fund_share next morning 08:30+;
  a zero-day that still shows 0 after re-fetch is a genuine API gap → interpolate.
- This is the same fund_share unreliability noted in memory
  ("tushare fund_share接口:返回0,更新延迟严重,不可靠"), just surfacing mid-history.

## Split detection: NAV is the authoritative signal, not close price (2026-08-15 bug)

`detect_and_fix_splits` (in ALL FIVE fetch scripts: tech/gold/national-team/
old-economy/sector) originally decided split-vs-subscription by checking whether the
fund_daily close dropped afterward. **Real bug found**: 588710.SH 2026-08-14 had a
genuine 3:1 split — unit NAV fell 3.1344→1.0532 (-66%) and shares jumped +196.7% —
but tushare's fund_daily close had NOT yet been ex-adjusted (still 3.16). The
close-based check saw no price drop, judged it "大额申购", skipped, and injected a
**fake +223亿 single-day inflow** into etf_tech_combined.csv.

Rules:
- **NAV (fund_nav unit_nav) is ALWAYS ex-adjusted on the split day; fund_daily close
  can lag.** So when shares jump >1.5x but close doesn't drop, cross-check NAV:
  `nav_ratio = pre_nav / post_nav` matches the share ratio within ±25-33% ⇒ split;
  NAV continuous ⇒ genuine subscription.
- Fix pattern (already applied 2026-08-15, function `nav_confirms_split` + inline
  merge branch): history shares × ratio, ALL close prices / ratio (unify to post-split
  basis since tushare close hasn't ex-adjusted yet). Merge case symmetric
  (post/pre NAV ratio ≈ 1/ratio).
- All five fetch scripts share this function — patched 2026-08-15. When any sibling
  script's copy is edited, sync the function body to the other four (regex-extract the
  block from `def nav_confirms_split`/`def detect_and_fix_splits` through `\n    return df`).

## Real large subscriptions ≠ anomalies — verify before "fixing"

A full scan of all 9 ETF group CSVs flagged many >40%-of-prior-asset single-day
inflows. NAV cross-validation showed they were REAL money events, not splits:
159682/159681 2024-10-09 (924行情, both 创业板50 ETFs same day), 512100.SH
2024-02-05 (国家队救市, +125%), 588200.SH 2024-10-09 (+122%). **Never clip or smooth
these — they carry real information.** Diagnostic scan pattern: CSV asset/change
columns are ALREADY 亿元 (do NOT /1e4), threshold |change|/prev_asset > 0.4 AND
prev_asset > 1亿 (excludes new-listings/zeros), then fund_nav continuity check per
candidate.