In a decision that could reshape the way banks and high-tech firms protect their intellectual property, a federal appeals court ruled Thursday that a man’s business concept was too vague for patent protection.