Forex Doombot — My application of Deep Reinforcement Learning and Graph-Tensor Neural Network for Forex Trading

I am inspired by great inventors, engineers, thinkers, and individuals who push the boundaries with their intelligence and character. I won’t deny the fact that some of these inspirations come from fictional characters. One of these is none other than Victor Von Doom, a.k.a Dr. Doom, the creator of Doombots — his advanced AI robots that contribute to his goals. My goal, create a Reinforcement Learning Agent (my “Doombot”) and for my “Doombot” to learn the optimal actions and policies of profitable trading behaviors and strategies in the live Forex market (with future implementations in other markets) and execute trades (Algorithmic Trading).

Algorithmic trading involves using advanced mathematical models, combined with human oversight, to make decisions about buying or selling financial securities to generate profit. Algorithmic trading in the foreign exchange markets presents a unique set of challenges (feature engineering, data preprocessing, etc.), and from my experience, it is difficult to find clear research on due to the sensitivity of algorithmic trading models. Factors like high volatility, non-stationarity, and an overwhelming amount of data available from various sources make it difficult for traditional machine learning models to capture the complex interactions between economic factors, price movements, and market sentiment. My solution to applying machine learning to the Forex market is to develop a custom reinforcement learning (RL) agent that combines techniques from Deep Q-Networks (DQN), Multi-Goal Training Networks (MGTN), and Q-learning.

I want to share what I am learning through building this project and provide information that I felt wasn’t available when I was starting out. This article is the first in a series detailing my approach, methodology, findings, and eventually the application process to the live market. The goal of this project is to create my own army of Doombots — not in the “take over the world” sense, but in the sense of utilizing our technologies (Machine Learning and AI) to improve aspects of life. The first aspect I am aiming to improve is financial profitability.

Below is a brief overview of the project and at the bottom you can find the other parts that will go into more detail on each aspect they cover.

Data ETL, Pipeline, and Time Series Analysis

Data Sources

• Historical Price Data: Collected from the Oanda API, covering various major currency pairs.
• Macroeconomic Indicators: Interest rates, GDP growth rates, and various economic factors from sources like FRED Economic Data.
• Market Sentiment Data: Extracted from news outlets and social media platforms using Natural Language Processing (NLP) techniques.

Data Pipeline Setup

• Real-time Data Ingestion: Implemented using streaming services to capture live market data.
• Batch Processing: For historical data, batch jobs processed and updated the database nightly.
• Feature Engineering: Calculated technical indicators like Moving Averages, RSI, and MACD.

Time Series Analysis Techniques

I am taking care of the fact that time-series data needs to be handled appropriately to get meaningful predictions and features for the model’s learning process.

• Autocorrelation Function (ACF): Used to find repeating patterns or periodicity.
• Partial Autocorrelation Function (PACF): Helps in identifying the order of autoregressive models.
• Augmented Dickey-Fuller Test: Conducted to test for stationarity.
• Additional Techniques: Employed as needed.

Proper time-series analysis techniques are crucial in understanding how variables change over time, especially in financial markets where trends, seasonality, and cyclicality significantly impact trading decisions.

Model Architecture

I came up with the idea of modifying Q-learning with a Deep Q-Network and also utilizing the strength of Graph-Tensor Networks’ capability of handling multi-modal data. The proposed architecture is a hybrid Deep Q-Network (DQN) enhanced by:

• Q-learning (DQN): For action-value function optimization.
• Multi-Graph Tensor Networks (MGTN): To handle multi-modal inputs and capture complex relationships between currencies and macroeconomic factors.

Components of the Architecture

• Feature Extraction Layer: Pulls features from raw market data and macroeconomic indicators.
• Graph Tensor Module: Constructs a graph of currency relationships (based on interest rates, economic news, etc) and compresses it using MGTN techniques.
• DQN Module: Uses tensorized graph representations to estimate Q-values and output optimal actions (Buy, Sell, Hold).

Training Process

• Environment: Creating and utilizing OpenAI’s Gymnasium, tailored for Forex market dynamics.

Data Segmentation

• High-Frequency Data (Minute-Level): 1 year (~525,600 data points per currency pair).
• Medium-Frequency Data (Hourly-Level): 3 years (~26,280 data points per pair).
• Low-Frequency Data (Daily-Level): 10 years (~3,650 data points per pair).

Expected Results and Use Cases

By integrating DQN, Q-learning, and MGTN, the model is expected to:

• Improve Profitability: Through more accurate predictions and optimal policy learning.
• Enhance Stability: Using experience replay and target networks.
• Efficiently Handle Complex Data: MGTN’s ability to process multi-modal inputs.

Future Deployments and Improvements

• Real-Time Deployment: Integrating the model with live data feeds for real-time trading.
• Expansion to Other Markets: Adapting the model for stocks, commodities, or cryptocurrencies.
• Incorporating Advanced Features: Adding improved sentiment analysis using NLP.
• Algorithmic Trading: Implementing the model for automated trading strategies.
• Financial Analysts: Using the model to inform decision-making with comprehensive data analysis.
• Academic Research: Serving as a foundation for further studies in financial machine learning.

Final Thoughts

With my experience in Quantitative Economics, Econometrics, Finance, Data Science, and Machine Learning, this project is a great showcase of my capabilities, allowing me to combine my experiences to make something, hopefully, great. I look forward to helping teach what I learn and any questions or feedback is welcomed!

The next parts of this project are as follows:

• Part 1 — Introduction to Reinforcement Learning in Forex Trading
• Part 2 — Data ETL, Pipeline Setup, and Time Series Analysis
• Part 3 — Understanding Stationarity and Time Series Modeling in Forex
• Part 4 — Model Architecture — Combining DQN, Q-learning, and MGTN
• Part 5 — Training the Hybrid Reinforcement Learning Model
• Part 6 — Experiments, Results, and Analysis
• Part 7 — Discussion — Improvements Over Existing Models
• Part 8 — Conclusion, Future Work, and Code Walkthrough
• Part 9 — Applying the Model in Real-Time Trading Systems

I will link these parts to the corresponding article as I write and publish them in the future if you would like to follow this series!

Feel free to reach out to me on LinkedIn as well!

Resources/ Reference Research Papers

• GitHub — https://github.com/ramruph
• Théate, T., & Ernst, D. (2020). An Application of Deep Reinforcement Learning to Algorithmic Trading. arXiv preprint arXiv:2004.06627. Available: https://arxiv.org/abs/2004.06627
• Carapuço, J., Neves, R. F., & Horta, N. (2018). Reinforcement Learning applied to Forex Trading. Applied Soft Computing, 73, 783–794. https://doi.org/10.1016/j.asoc.2018.09.014
• Xu, Y. L., Konstantinidis, K., & Mandic, D. P. Multi-Graph Tensor Networks. Department of Electrical and Electronic Engineering, Imperial College London.