alpha_lab/stock_1d/d033/alpha158_beta/BUG_ANALYSIS_FINAL.md

Data Pipeline Bug Analysis - Final Status

Summary

After fixing all identified bugs, the feature count now matches (341), but the embeddings remain uncorrelated with the database 0_7 version.

Latest Version: v5

• Feature count: 341 ✓ (matches VAE input dim)
• Mean correlation with DB: 0.0050 (essentially zero)
• Status: All identified bugs fixed, but embeddings still differ

---

Bugs Fixed

1. Market Classification (FlagMarketInjector) ✓ FIXED

• Bug: Used instrument >= 600000 which misclassified 新三板 instruments
• Fix: Use string prefix matching with vocab_size=2 (not 3)
• Impact: 167 instruments corrected

2. ColumnRemover Missing IsN ✓ FIXED

• Bug: Only removed IsZt, IsDt but not IsN
• Fix: Added IsN to removal list
• Impact: Feature count alignment

3. RobustZScoreNorm Scope ✓ FIXED

• Bug: Applied normalization to all 341 features
• Fix: Only normalize 330 features (alpha158 + market_ext, both original + neutralized)
• Impact: Correct normalization scope

4. Wrong Data Sources for Market Flags ✓ FIXED

• Bug: Used Limit, Stopping (Float64) from kline_adjusted
• Fix: Load from correct sources:
  • kline_adjusted: IsZt, IsDt, IsN, IsXD, IsXR, IsDR (Boolean)
  • market_flag: open_limit, close_limit, low_limit, high_stop (Boolean, 4 cols)
• Impact: Correct boolean flag data

5. Feature Count Mismatch ✓ FIXED

• Bug: 344 features (3 extra)
• Fix: vocab_size=2 + 4 market_flag cols = 341 features
• Impact: VAE input dimension matches

---

Correlation Results (v5)

Metric	Value
Mean correlation (32 dims)	0.0050
Median correlation	0.0079
Min	-0.0420
Max	0.0372
Overall (flattened)	0.2225

Conclusion: Embeddings remain essentially uncorrelated with database.

---

Possible Remaining Issues

1. Different input data values: The alpha158_0_7_beta Parquet files may contain different values than the original DolphinDB data used to train the VAE.

2. Feature ordering mismatch: The 330 RobustZScoreNorm parameters must be applied in the exact order:

   • [0:158] = alpha158 original
   • [158:316] = alpha158_ntrl
   • [316:323] = market_ext original (7 cols)
   • [323:330] = market_ext_ntrl (7 cols)

3. Industry neutralization differences: Our IndusNtrlInjector implementation may differ from qlib's.

4. Missing transformations: There may be additional preprocessing steps not captured in handler.yaml.

5. VAE model mismatch: The VAE model may have been trained with different data than what handler.yaml specifies.

---

Recommended Next Steps

1. Compare intermediate features: Run both the qlib pipeline and our pipeline on the same input data and compare outputs at each step.

2. Verify RobustZScoreNorm parameter order: Check if our feature ordering matches the order used during VAE training.

3. Compare predictions, not embeddings: Instead of comparing VAE embeddings, compare the final d033 model predictions with the original 0_7 predictions.

4. Check alpha158 data source: Verify that stg_1day_wind_alpha158_0_7_beta_1D contains the same data as the original DolphinDB stg_1day_wind_alpha158_0_7_beta table.