This paper investigates statistical inference for semi-varying coefficient panel data models with fixed effects subject to linear restrictions. By incorporating fixed effects to control for time-invariant heterogeneity across individuals and employing varying coefficient functions to capture nonlinear covariate-response associations, the model mitigates two key limitations: inflexibility in traditional parametric models and the curse of dimensionality in nonparametric models. It further provides theoretical support for empirically prevalent linear restrictions such as economic structural hypotheses and policy intervention conditions. The study first establishes unrestricted estimators for parametric and nonparametric smooth coefficient functions via restricted profile least squares. Subsequently, individual dummy variables are introduced to isolate fixed effects, thereby eliminating their impact on the dependent variable. Finally, linear restrictions are imposed to derive restricted estimators for parameters, nonparametric functions, and error variances using the Lagrange multiplier method. Under regularity conditions, the restricted estimators for parameters, nonparametric functions, and error variance are rigorously proven to exhibit asymptotic normality. Monte Carlo simulations demonstrate that restricted estimation significantly enhances parameter estimation accuracy in finite samples. Moreover, when the correlation between fixed effects and covariates intensifies, the robustness advantages of restricted estimation become more pronounced.