REINFORCEMENT LEARNING IN EQUITY TRADING: EVIDENCE FROM DEVELOPED AND FRONTIER MARKETS

Introduction: Understanding whether algorithmic trading strategies can generate consistent risk-adjusted returns remains a central question in financial markets, particularly across environments with different levels of market efficiency. This study develops a deep reinforcement learning framework for systematic equity trading and evaluates its performance in both a developed market (S&P 500 ETF) and a frontier market (Mongolian Stock Exchange). The study is motivated by the need to examine whether reinforcement learning strategies behave differently across markets characterized by varying degrees of informational efficiency and structural frictions. Methods: The study implements a Deep Q-Network (DQN) reinforcement learning framework for sequential trading decisions. The model is trained and evaluated under identical experimental conditions using historical market data from SPY and a composite portfolio of actively traded Mongolian equities. The methodology incorporates chronological train-validation-test splits, technical indicator-based state representation, transaction cost modeling, and out-of-sample performance evaluation. Market efficiency is additionally examined through weak-form efficiency tests using return autocorrelation and Ljung–Box statistics. Results: The empirical results demonstrate a clear contrast between developed and frontier markets. In the developed market, the reinforcement learning strategy provides only modest improvements in risk-adjusted performance relative to passive benchmarks, consistent with the Efficient Market Hypothesis. In contrast, the frontier market results show substantially higher returns and improved performance metrics for the DQN strategy, indicating the presence of exploitable inefficiencies and supporting the Adaptive Markets Hypothesis. The findings further reveal that the reinforcement learning agent converges toward near-passive behavior in efficient markets while exploiting return persistence in less efficient environments. Discussion: The findings suggest that the effectiveness of reinforcement learning-based trading strategies is strongly dependent on market structure and informational efficiency. In highly efficient markets, adaptive strategies generate limited incremental value over passive investment approaches. Conversely, in frontier markets characterized by lower liquidity, weaker information dissemination, and structural inefficiencies, data-driven trading strategies may provide substantial advantages. From a practical perspective, the results highlight the potential value of reinforcement learning as a decision-support framework for investors and financial institutions operating in frontier market environments, while also emphasizing the importance of risk management due to higher volatility and drawdowns.