Didn't know that. Did know that even before the Village it was General Lee's massive Roman mansion. The Quartermaster General, Meigs (who had lost a son to Lee's army) was given the gig of finding a spot to bury all the corpses produced by the war. Blaming Lee for the length of the conflict, and noting Lee's justification of the war as a defence of his land*, he decided to build the cemetery on that very land. Which is, I think, apt.
*IMO a bullshit justification.