This is a commonly repeated fallacy, it is patently untrue even in the absence of QE.
The Fed controls the value of money. Therefore it controls, to some extent, the price (in dollars) of all assets, including long bonds. A simple example would be if the Fed raised the price of money to 10%...