Can one party of a contract...?

If your name is on the title, then no, he has no legal right to keep you there. You are an adult. A co signer's responsibility is to the lender, if you stop payments he is to pick them up.