This thread has gotten me thinking and tinkering again. I looked up the basics of digital signatures, and from what I gather, it works like this:
1. A hash function is run to get a hash of the source data
2. The results of the hash are encrypted
3. The encrypted hash is bundled with the data and delivered
4. The encrypted hash is decrypted at the destination
5. The hash function is run on the delivered data
6. If the calculated hash matches the decrypted hash, the signature is verified
So assuming this is the scheme the 3DO uses, there are a couple of ways to approach running homebrew. We could learn the encryption key so that we can create our own signatures, or we can create data that matches an existing signature. Depending on the complexity of the hashing algorithm, creating data to match an existing signature could be fairly straightforward.
Does anyone have more information about what changes to an ISO break a game and what changes don't? As Gir Draxa and 3DOkid have talked about, modifying data files and keeping them the same size doesn't break the game. I just did a test and verified that modifying the main executable (LaunchMe) and keeping it the same size doesn't break it, either.
Does adding a file break it? How about deleting a file? Or changing a file name? Do all files have to remain the same size, or just certain ones? Can you extract and rebuild the ISO and have it work, or do you have to directly edit the original?
I could do some more experiments to answer these questions, but if anyone knows off the top of their head, it would save me some trouble.