Add TrainJob validation to Experiment webhook validator

[Krishna-kg732]

What would you like to be added?

The Experiment validating webhook currently only validates batch/v1 Job trial specs in validateTrialJob(). When the trial template uses a TrainJob (or any other supported Kubeflow job kind), the function returns nil without any structural validation:

https://github.com/kubeflow/katib/blob/master/pkg/webhook/v1beta1/experiment/validator/validator.go#L440-L459

func (g *DefaultValidator) validateTrialJob(runSpec *unstructured.Unstructured) error {
    gvk := runSpec.GroupVersionKind()
    // Only validates batch/v1 Job — everything else is silently skipped
    if gvk.GroupVersion() != batchv1.SchemeGroupVersion || gvk.Kind != consts.JobKindJob {
        return nil
    }
    // ...
}

This means a user can submit an Experiment with a malformed TrainJob trial template, and the webhook will accept it without error. The misconfiguration only surfaces at runtime when the controller tries to create the trial.

Proposed Changes

1. Add validation for TrainJob in validateTrialJob() — when kind: TrainJob, verify the unstructured spec can be converted to a valid TrainJob structure.
2. Extract "TrainJob" hardcoded strings in constants.go and experiment_defaults.go into a named constant (JobKindTrainJob).
3. Fix the copy-paste comment on DefaultTrainJobPrimaryPodLabels (line 49 of constants.go) — the comment incorrectly says DefaultKubeflowJobPrimaryPodLabels.

Why it matters

• TrainJob is already a supported trial kind in Katib (added in KubeflowJobKinds and with dedicated success/failure conditions and pod labels).
• The upcoming OptimizationJob CRD (#2605) will exclusively use TrainJob as trial workloads, making this validation even more important.
• Catches misconfigurations at admission time rather than at runtime.

/kind feature
/area api

[github-actions]

🎉 Welcome to the Kubeflow Katib repo! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Next steps:

• Our team will review your issue soon! cc @kubeflow/wg-automl-leads
• Check out the Contributing Guide and the Kubeflow Contributor Guide
• Join the Kubeflow Slack channels: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
• Join the AutoML & Training WG meetings: https://bit.ly/2PWVCkV

Feel free to ask questions in the comments. Thanks again for contributing! 🙏

[Krishna-kg732]

/assign

[aniket2405]

Awesome catch @Krishna-kg732!
Fixing this will help us reuse this validation in the new controller.
I'd like @andreyvelich and @akshaychitneni to help out with whether we want this fix as a standalone and update it to the old Experiment CRD, or we want to directly address this in the OptimizationJob CRD.

[Krishna-kg732]

I think it makes sense to keep this as a standalone fix for the Experiment webhook. Right now malformed TrainJob trial templates get accepted and only fail later at runtime, so adding validation here would help catch those errors earlier.

Also, since OptimizationJob will rely on TrainJob trials, having this validation in place now would let the new controller reuse the same logic instead of implementing it again.

cc: @andreyvelich , @akshaychitneni