Free indicators

It is not the Doji pattern that matters - it is trade management. Arbitrary Targets and Stops are meaningless. Set these to pivot and congestion/ranging areas. This follows market behavior. Python code used to create this will be available through paid subscriptions - I will take your suggestions to implement.

DojiGrid Pro: Technical Guide to Grid-Based Doji Signal Analysis Courtesy of MarketFragments.com – Open-Source Trading Tools for Advanced Users Generated on 2025-10-16 | Data Source: ES Full 5-Min Continuous (Last 1 Year) | Optimized with Hyperopt & Ensemble

ML Overview: Architecture and Output Generation

This guide details the implementation and analytical process of Doji Grid Pro, a grid-based indicator for identifying high-probability Doji reversal setups in futures data (e.g., ES 5-min). The system integrates technical feature engineering with hyperparameter optimization and multi-model validation to produce a ThinkOrSwim (ToS)-compatible script. Outputs include CSV rankings, full optimization logs, and a parameterized ToS study file, derived from vectorized computations on ~100K+ bars.

The pipeline processes data through:

1. Preprocessing and feature extraction (grids, ATR, zones).

2. Signal detection and setup filtering (Doji + RR validation).

3. Backtesting and ML evaluation (11 models, Sharpe-optimized).

4. Hyperopt for parameter tuning (TPE, 150 evals).

5. Script generation with 10 settings from top runs.

All computations use NumPy/Pandas for efficiency, with Joblib caching (XXHash keys) to handle large datasets. Resource management caps CPU/memory at 80% via psutil throttling. Analysis focuses on the last year of data (post-2024-10-16) to emphasize recent volatility regimes.

Core Components: Code Breakdown

1. Configuration and Validation Schema: Pydantic Config model enforces structure (GridConfig, ModelConfig, etc.). Defaults load from config.yaml or inline dicts.

* Key Params: Grid size (5x5/6x6), lookback=100, congestion touches=3, volume threshold=1.5, ATR mults (target=1.5, stop=1.0), min RR=2.0.

* Hyperopt Space: Uniform/quniform distributions for tuning (e.g., volume_threshold ~ U(1.0,2.0)).

* Loading: Merges user YAML with defaults; raises ValidationError on mismatches.

* Analysis Note: Ensures reproducibility; e.g., fallback RR ratios [2.0,3.0,4.0,5.0] prevent invalid setups.

2. Data Pipeline

* Ingestion (load_data): Chunked CSV read (500K rows, c-engine) with parallel processing (Joblib, n_jobs=-1). Filters: >0 volume, last 1 year, no duplicates.

* Output: Indexed DataFrame (Date as tz-naive Timestamp).

* ATR (calculate_atr_cached): Vectorized TR max + rolling mean (period=14, min=0.25). Cached via Memory decorator.

* Resource Handling: ResourceGovernor sleeps on excess CPU/mem; log_memory_usage tracks GB usage.

* Analysis Insight: On ES data, avg ATR ~0.75 points; filters out ~5% low-vol bars.

3. Feature Engineering: Grid and Zones

* Grid Computation (compute_grid_features): For each bar i >= lookback:

* Window: Rolling 100-bar OHLCV.

* Normalization: Divide range (maxH - minL) into grid_size zones; zone_size = range / grid_size.

* Population: Vectorized np.add.at accumulates touches (OHLC indices) + volume weights across columns (time axis).

* Congestion: Row sums >= touches AND total vol >= threshold * mean_vol * grid_size.

* Pivots: Standard formula (Pivot=(H+L+C)/3; S1=2Pivot-H; R1=2Pivot-L); flags zones within zone_size/2 of any.

* Output: Dict with grids (n x grid_size x grid_size), binary masks (congestion_zones, pivot_touches), OHLCV arrays, ATR.

* Shape Example: grids (100K,5,5); cached as Joblib (~50MB).

* Analysis Insight: Grids capture ~70% of touches in 2-3 rows (congestion bias); pivot flags align with 65% of zone hits in backtests.

4. Signal and Setup Generation

* Doji Detection (detect_doji_signals): |C-O| / (H-L) <= 0.005 AND H-L >0; direction from prior candle (up if C[1]>O[1]).

* Initial: Entry=C, Stop=C ± ATR, Target=C ± min_RR * ATR.

* Setups (compute_trade_setups): For each signal:Stop: Entry ± atr_multiplier_stop * ATR.

* Projected Target: Entry ± atr_multiplier_target * ATR.

* Snap: Nearest active zone (congestion OR pivot) in direction; else fallback min(valid RR >= min_RR).

* Filter: Calculated RR >= min_RR.

* Analysis Insight: ~200 raw signals/year; 60-70% qualify post-filter (higher in low-vol regimes). Scale-out (80% to trailing ATR stop) reduces DD by 20%.

5. Backtesting and ML Evaluation

* Simulation (_compute_backtest_stats): Bar-by-bar forward walk from setup index:

* Exits: SL, target, or scale-out (if on, trail after 80% profit via ATR mult).

* Metrics: Win rate (mean outcomes), avg return, Sharpe (mean/ret_std * sqrt(N)), max DD (cumsum peak-trough).

* Requires >=20 trades; else objective=-inf.

* Data Prep (prepare_data): 70/30 split; X = flattened grids + zones (shape: n x (25+5+5)); y = binary wins.

* Models (TradingModelBase subclasses):

* Tree-Based: RandomForest (200 est), XGBoost (100 est, logloss).

* DL: CNN (Conv2D 32@3x3 on grids), RNN/GRU/LSTM (50 units, seq=11), Transformer (2-head attn + GAP).

* RL: Q-Learning (tabular hash states), DQN (3-layer FC, 100 episodes, epsilon decay), PPO (ActorCritic, REINFORCE baseline).

* Hybrid: Avg CNN+XGBoost preds (>0.5 threshold).

* Regression: LSTM on targets (MSE/MAE).

* Eval: OOS accuracy, filtered Sharpe on y_pred==1 returns.

* TradingEnv (Gym): States = flat grid + norm OHLCV + pos + PnL; actions=[hold,long,short]; rewards=delta PnL - costs (0.01%).

* Analysis Insight: RF/XGB accuracy ~65%; RL (DQN) excels in trending (Sharpe +0.15); ensemble voting stabilizes at 68% OOS. 128 trades avg, win rate 62%.

6. Optimization and Outputs

* Hyperopt (run_optimization): TPE (n_startup=10), max_evals=150, early_stop if 5+ trials >Sharpe 0.5.

* Objective: -Sharpe (minimize loss).

* Trials: Parallel eval on 80% train split; top 10 serialized to CSV (loss, trades, win_rate, etc.).

* Best Params: Extracted (e.g., grid_size from choice idx); final eval on full data.

* ToS Script (generate_tos_script):

* Header: 10 settings from top rows (switch input).

* Logic: Recursive defs for zone touches/vol (per-zone, lookback fold); doji filter; snapped targets; RR check; scale-out trails.

* Plots: Arrows (entries/exits), dashed targets, cyan cloud (range), labels (grid/active zones).

* Saved: .ts/.txt with timestamp.

* Viz: Seaborn heatmap of final grid (Blues cmap, annot).

* Analysis Insight: Top run (Setting1): 5x5 grid, 3 touches, Sharpe=1.23, 128 trades, DD=0.3%. Logs in top10_results_YYYYMMDD.csv; rankings in doji_rankings_YYYYMMDD_HHMMSS.csv.

Validation and Extensions

* Tests: Unit coverage for load_data, ATR, grids (e.g., assert shapes, min values).

* Edge Handling: NaN drops, zero-range skips, cache invalidation.

* Extensions: Edit config.yaml for other symbols; add models via factory; scale to daily via BACKTEST_STEP.

* Performance: cProfile: ~2-5 min full run (150 evals); memory peak ~4GB.

## Author Notes: This is the first iteration into an entire trade management system. Don't think of the Doji, think of any signal. However, we were able to increase W/L ratio from 31% to 50%, which is 100% improvement. We are already on the 3rd interaction. Once complete, we move to adding dynamic intrade system. Page "AI Gameplan" has the prototype for a plugin. We will integrate this complete system. Please feel free to comment in this group. Any and all suggstions are greatly welcomed.