You described AI, but I use XAI.

It's a different AI.
The field of Explainable AI (XAI) encompasses AI systems capable of providing insights into their decision-making processes, bridging the gap between input and output analysis. In my XAI, I use interpretable models. My model cross-checks decision trees, linear regression, and rule-based systems. People might use one or another, but my model exploits all of these methods to correct the errors of one with another. Permutation Importance and Partial Dependence Plots contribute to weighting the different models to create a new model. It's different than using any one of the mentioned techniques.
I share your view about the importance of post-hoc explanation techniques, and that's why I trained my XAI to do post-explanation with LIME (Local Interpretable Model-Agnostic Explanations) and SHAP (SHapley Additive exPlanations).
You mentioned drawdown. My XAI can explain Maximum Drawdown with two techniques I wrote above: LIME and SHAP. The XAI uses LIME to explain factors contributing to a specific maximum drawdown event, identifying influential features like market volatility or portfolio composition. SHAP quantifies the impact of each feature on the drawdown, revealing those most responsible for the loss.
In the end, I didn't burn my CFA certificate, and I didn't suggest it. There's a reason I've got companies with employed portfolio managers, but still, I'm here in person and writing my personal opinion in my free time. I love market analytics, and I'll do it no matter what. But AI helps me solve monotonous tasks. -
I can have fun trading.