Gotta turn this around.
If this is the situation, assignments coming from higher ups than through your "manager" your manager somehow lost respect along the way. Work shouldn't flow the way you describe.
Your manager has the problem, not you.
If your manager is showing frustration...