It's not a childish argument. I haven't followed the case, so I don't know the details, but if he's charged with any type of federal offense related to fraud, it does need at least 1 victim or intended victim. If there's no intent to defraud, then there cannot be a crime. But it sounds like in this case there's clearly a case of intent, or at least the judge wants the jury to decide intent based on the facts.
A second prong of any federal fraud offense is the amount of financial damage sustained by the victims. This is especially relevant for sentencing purposes as the entire federal sentencing guidelines is based on how much money his victims lost. The base offense only carries 6 months, then it goes up from there depending on the amount of loss. So the argument of "nobody lost money, so there's no crime" is a very valid argument.
Of course, the 2nd Circuit ruled that a mere intent to defraud, even if the scheme is unsuccessful, can satisfy the loss prong for however much money the defendant intended to defraud. I don't know specifics, but if it's true his investors got all their money back can be a strong argument against intent to defraud. But if these investors only got their money back AFTER civil/criminal investigations started against him, then his defense is out the window, and he's looking at a long prison sentence, and the judge doesn't have much authority to deviate from the below:
https://www.whitecollarcrimeresources.com/federal-fraud-sentencing-guidelines.html
He's looking at an offense level of 28 assuming no surprising upward or downward departures at sentencing trial.
Assuming no criminal history, the judge would be bound by the range of 78 to 97 months. See table here:
https://en.wikipedia.org/wiki/United_States_Federal_Sentencing_Guidelines
If he paid all his victims back after he got caught, which is enough to establish guilt, but would be strong argument for a downward departure, he might be able to get as much as 5 points knocked off his offense level, and he'd only be looking at 46-57 months, but downward departures of this type are extremely rare especially without accepting responsibility (guilty plea).