I understand your sentiments exactly. But think of it this way. If the dollar goes up, gold will go down and vice versa. Therefore, keeping some money in gold bullion or GLD is simply a hedge against a probable resumption of the dollar decline.
Personally, I got stopped out of all my metal...