Financial Risk Manager Part 1

Get started today

Ultimate access to all questions.

Explanation:

A is correct. Using the MC approach with $\alpha = 0.05$ , the current Q-value $Q(2,2) = 0.9$ , and the total subsequent reward $R = 1.2$ :

$Q_{\text{new}}(2,2) = Q_{\text{old}}(2,2) + \alpha[R - Q_{\text{old}}(2,2)]$ $Q_{\text{new}}(2,2) = 0.9 + 0.05[1.2 - 0.9]$ $Q_{\text{new}}(2,2) = 0.9 + 0.05 \times 0.3$ $Q_{\text{new}}(2,2) = 0.9 + 0.015 = 0.915$

B is incorrect. The MC approach would only update $Q(2,2)$ to 1.200 if the learning rate $\alpha = 1$ , which would mean completely replacing the old Q-value with the new reward. With $\alpha = 0.05$ , the update is much smaller.

C is incorrect. The TD approach updates the Q-value of the current state where the agent is making the decision. Since the TD approach is initialized at S1, it would update $Q(1,A)$ — specifically $Q(1,2)$ — not $Q(2,2)$ . Additionally, the calculation does not produce 0.895.

D is incorrect. Similar to C, the TD approach would not update $Q(2,2)$ because the agent has already moved to S1. The TD update happens at S1, the new state, not at S2 where the action A2 was taken.

Key distinction between MC and TD learning:

Monte Carlo (MC) learning: Updates the Q-value of the state-action pair that was visited at the start of an episode, using the total actual reward observed from that point onward. It waits until the end of the episode to update.
Temporal Difference (TD) learning: Updates the Q-value of the current state-action pair incrementally using the immediate reward plus the estimated future reward (bootstrapping) from the next state. It does not wait until the end of the episode.

Explanation:

A is correct. Using the MC approach with $\alpha = 0.05$ , the current Q-value $Q(2,2) = 0.9$ , and the total subsequent reward $R = 1.2$ :

D is incorrect. Similar to C, the TD approach would not update $Q(2,2)$ because the agent has already moved to S1. The TD update happens at S1, the new state, not at S2 where the action A2 was taken.

Key distinction between MC and TD learning:

Monte Carlo (MC) learning: Updates the Q-value of the state-action pair that was visited at the start of an episode, using the total actual reward observed from that point onward. It waits until the end of the episode to update.
Temporal Difference (TD) learning: Updates the Q-value of the current state-action pair incrementally using the immediate reward plus the estimated future reward (bootstrapping) from the next state. It does not wait until the end of the episode.

Comments (0)

No comments yet.

Q-49. A data science team at the market-making division of a global financial institution is working to improve the division's risk-adjusted returns on options trading using machine-learning methods. The team creates a reinforcement learning (RL) algorithm that lets the learning agent develop price setting strategies. The strategies are intended to maximize profit and minimize inventory risk by dynamically adjusting bid and ask prices in the market maker's electronic limit order book.

The RL algorithm defines the following market states and market actions:

State 1: high transaction costs and high volatility

State 2: low transaction costs and low volatility

Action 1: take a market-neutral approach

Action 2: take a directional approach

The team considers using either the Monte Carlo (MC) approach, formulated as
$Q_{\text{new}}(S,A) \leftarrow Q_{\text{old}}(S,A) + \alpha[R - Q_{\text{old}}(S,A)]$
or the temporal difference (TD) learning approach, to update estimates for the Q-values, called $Q(S,A)$ or the value of taking an action in a particular state called, used to decide on the optimal strategy.

The current Q-values for the 2-state, 2-action scenario are given in the table below:

State 1 (S1) State 2 (S2)
Action 1 (A1) 0.7 0.8
Action 2 (A2) 0.5 0.9

In the next trial of the RL algorithm, the team notes that under an $\alpha$ parameter of 0.05, the MC approach samples a run taking A2 in S2 and estimates the total subsequent reward to be 1.2. Furthermore, the next decision on the trial is when the learning agent is on S1, with the jump between S2 and S1 generating a reward of 0.3. S1 is where the TD approach is initialized.

In using the two approaches, which of the following provides the correct update on the $Q(2,2)$ -value?

	State 1 (S1)	State 2 (S2)
Action 1 (A1)	0.7	0.8
Action 2 (A2)	0.5	0.9

Other

Community

UAnonymous

Last updated: July 10, 2026 at 05:44