Worth taking into account: the amount of energy you need for even tiny amounts of matter is gigantic. E=mc^2, and c is a very big number.
For reference, the entire world energy consumption for 2008 was 474 exajoules( 474,000,000,000,000,000,000 Joules). If you turned all that energy into matter, you'd have
5274 kg, which Wolfram Alpha kindly reminds me is about one elephant or three small cars. Creating a single person would take a significant fraction of the US yearly energy production.
As an aside, I can't imagine how holodeck AI would be any more of an ethical dilemma than AI by itself, which is kind of independent of being able to make matter from energy.