Reward Projection and Feature Rank

Proofs and proof sketches on this page draw from Kang (2026), Rawat and Rust (2026), and the classical sources cited inline.

Read this page when a previous page says that a reward is recovered. A recovered reward function is not automatically a recovered parameter vector. EconIRL often reports a finite parameter \(\theta\), so one more step is needed: the recovered reward contrasts must live in the span of the supplied features.

Setup

This page starts after the dynamic part of identification has been handled: \(\beta\) is fixed, the relevant transition or continuation object is known or identified, and the previous pages have recovered reward contrasts on the covered state-action support. Let \(a_0(s)\) be the reference or anchor action in state \(s\). Suppose the reward model is linear:

\[ r_\theta(s,a)=\phi(s,a)^\top\theta. \]

The policy identifies action contrasts first, so define

\[ \Delta r(s,a)=r(s,a)-r(s,a_0(s)), \qquad \Delta\phi(s,a)=\phi(s,a)-\phi(s,a_0(s)). \]

Stack the covered non-anchor state-action pairs into a vector \(y\) and matrix \(X\):

\[ y_{s,a}=\Delta r(s,a), \qquad X_{s,a,\cdot}=\Delta\phi(s,a)^\top . \]

The parameter recovery equation is then

\[ y=X\theta. \]

This is the last algebraic bridge from behavior to finite parameters.

Action-Difference Rank Theorem

Theorem. If the recovered reward contrasts satisfy \(y=X\theta_0\) and \(X\) has full column rank, then \(\theta_0\) is uniquely identified from the recovered reward contrasts. If \(X\) is rank deficient, \(\theta_0\) is not uniquely identified.

Proof. If \(X\) has full column rank, then \(X^\top X\) is invertible. The normal equations have the unique solution

\[ \theta_0=(X^\top X)^{-1}X^\top y. \]

If \(X\) is rank deficient, there is a nonzero vector \(v\) with \(Xv=0\). Then

\[ X(\theta_0+v)=X\theta_0=y. \]

The two parameter vectors produce exactly the same recovered reward contrasts on the covered support. No estimator using only those contrasts can distinguish them.

Estimator consequence. The feature-rank condition in NFXP, CCP, TD-CCP, MCE-IRL, AIRL-Het, and GLADIUS is not a numerical preference. It is the identification condition that turns recovered rewards or reward contrasts into finite parameters.

Why State-Only Features Fail for Action Rewards

Suppose the feature map is copied across actions:

\[ \phi(s,a)=\psi(s) \quad\text{for every }a. \]

Then

\[ \Delta\phi(s,a)=\psi(s)-\psi(s)=0, \]

so \(X=0\). The action-difference equation becomes

\[ \Delta r(s,a)=0^\top\theta, \]

which carries no information about \(\theta\) unless every action contrast is actually zero.

Proof. This is the rank theorem with rank\((X)=0\). If \(\theta\) is feasible, then every \(\theta+v\) is also feasible because \(Xv=0\) for all \(v\).

Estimator consequence. State-only features can be useful when the model is explicitly a state-reward model, as in the original AIRL guarantee. They cannot identify action-dependent payoff differences unless the model supplies another source of action variation.

Projection When the Feature Span Is Wrong

If the recovered reward contrast \(y\) is not exactly in the column span of \(X\), there is no structural parameter \(\theta\) that reproduces the reward perfectly. The estimand becomes a weighted projection:

\[ \theta_W =\arg\min_\theta (y-X\theta)^\top W (y-X\theta), \]

where \(W\) is a positive semidefinite weighting matrix over covered state-action contrasts. If \(X^\top W X\) is invertible, then

\[ \theta_W=(X^\top W X)^{-1}X^\top W y. \]

Proof. Differentiate the quadratic objective:

\[ \frac{\partial}{\partial\theta} (y-X\theta)^\top W(y-X\theta) =-2X^\top W(y-X\theta). \]

Setting the derivative to zero gives \(X^\top W X\theta=X^\top W y\). Invert \(X^\top W X\) to obtain the displayed formula.

Estimator consequence. Under misspecification, the reported parameter is a best linear projection under the chosen weighting, not the literal primitive reward. This is why reward plots, action contrasts, and counterfactual checks matter even when a finite parameter is reported.

For counterfactual work, the projection has to preserve the reward directions that the new policy or transition law will use. A projection that is accurate on frequent observed contrasts can still be fragile if the counterfactual puts mass on poorly approximated states or actions.

Neural Rewards Are Function Objects First

For neural reward estimators, many parameter vectors can represent the same reward map. Hidden-unit permutations, redundant units, and flat directions in the network can change the raw weights while leaving \(r_\eta(s,a)\) unchanged on the covered state-action pairs.

Formally, if two neural parameter vectors \(\eta_1\) and \(\eta_2\) satisfy

\[ r_{\eta_1}(s,a)=r_{\eta_2}(s,a) \quad\text{for all covered }(s,a), \]

then every policy, value, reward-contrast, and Bellman-rearranged quantity on that support is the same. The raw parameter vector is therefore not the identified object. The identified object is the anchored reward map or its finite projection.

Estimator consequence. Neural MCE-IRL and GLADIUS should be interpreted at the level of anchored reward maps, action contrasts, induced policies, and projected parameters. Raw neural weights are implementation details.

What To Check

Question	Formal check	Why it matters
Are action rewards identified?	\(X\) has full column rank after anchoring.	Otherwise multiple \(\theta\) values produce the same reward contrasts.
Are state-only features being used for action rewards?	\(\Delta\phi(s,a)\) is not identically zero.	State-only features cannot explain action payoff differences.
Is the linear model exact?	\(y\) lies in the column span of \(X\).	If not, \(\theta\) is a projection.
Is a neural reward being reported?	Compare reward maps, not raw weights.	Network parameters are many-to-one.

The broader logic is simple: behavior identifies policy and action-value contrasts; anchors and Bellman equations identify reward contrasts; feature rank turns those contrasts into finite parameters.