The good databases have not gone away, they just walk instead of run...
Adam Bosworth has pointed out an interesting set of requirements for database systems in his post titled “Where have all the good databases gone”:
- Dynamic Schema support
- Dynamic partitioning
- Better Indexing of non relational data
I think all of them are indeed important and based on my insight into the database industry, the major database vendors are working on addressing them. I think the recent addition of the XML datatype to SQL (and the major database vendors) to store data with more dynamic schemas and XQuery to query that data in a (hopefully) efficient way is certainly one of the main technologies to address some of the dynamic schema scenarios. Others are dynamic, online schema changes (ALTER TABLE) and other features.
There are also efforts underway to improve dynamic partitioning and indexing. For example, look at the investments into improving the full-text search capabilities of the database engines (SQL Server 2005's full-text search is considerably more performant than the previous version and more work is being done for future releases) and adding full-text support to XQuery.
However, adding such support has to be done carefully, if one wants to continue to provide the performance, scalability and maintainability that database systems normally provide (or should provide). For example, it is much harder to compile a query into an efficient query plan if the compiler and optimizer has less (reliable) schema information available. Frequent dynamic recompiles of a query due to frequent schema changes may be too costly if done wrong etc.. And for better or worse, database engines have become complex systems that need many components to provide the necessary integration of a new feature.
So doing this right takes time, but the vendors have started doing it over the last five years.
He also then asked the open source community to build it which lead to an interesting post by Kris (and a reply post by Adam and a comment by Dare). Interesting to note that all three of them have worked with me at Microsoft at some time :-). And note the interesting parallelism to Mr. Egelund's recent remarks and this exchange about contributing to the open-source world :-). My take: I think Kris and Dare have a point in that “web services” businesses prefer software to be free, because it lowers their cost and increases the number of people getting access (although, they should ask/provide for free hardware too, like the cell phone providers :-)). All that will be paid for will be services and servicing. OTOH, I don't think this part was the important aspect of Adam's post. These payment models are part of a set of competing business models with different trade-offs, which will look differently based on what you are working on :-).