An XML-based DBMS is one that doesn't worry about relations. It stores the data in the database in XML format. I'm not sure why anyone would really want to do this. The XML format, by its very nature is hierarchical. The industry has long-since moved away from hierarchical database systems for many reasons (this is one reason why you don't see ADABAS being used too much any more). To me, the move to XML-based DBMS platforms is a step backwords.
The rules of normalization are written specifically for a RDBMS. As such, they do not apply to non-RDBMS systems. While you can try to extend these rules to XML-based systems, they won't be a perfect fit for these systems.