I have most often seen messages like this when the DBCA attempts to start up an instance and the startup fails. Most likely, you forgot to set up the required kernel parameters in /etc/system for shared memory and semaphores. Without the proper parameters, Oracle may fail to start.
Check the installation guide for Solaris and ensure that you've set the proper parameters. You can also check the $ORACLE_BASE/admin/$ORACLE_SID/create directory for logfiles from the DBCA that may contain more precise information about the error.
This was first published in July 2004