The unholy legacy of databases

I have tried this sort of thing as well and I really really like the concept. The only thing that stops my poor, slow, small brain from really seeing it through to its logical conclusion is the sometimes-requirement to join attributes from one Thing stored with one Storage mechanism with the attributes of another Thing stored with another Storage mechanism. Do you have suggestions here?

Obviously, a Compass/Lucene-type search handles a huge number of cases–people tend to like to search by keywords, and so a coarse-grained search/locate strategy like that makes a lot of sense. But in some of the applications I work on, careful targeted queries that join bits of two entities together–a classic SQL join–are also needed.

Have you found a convenient way to expose a *common* SQL-like query mechanism across items that use different Search implementations?

I surely don’t. :-) My main problem is that I need both–the ability to join LDAP information (Student information, say), as it happens, with additional information relevant to it from a database (classes they’re taking).

The decidedly brute-force, ugly, smelly, hairy, nasty and yet strangely cool approach I took at one point was to make a kind of query builder that, in conjunction with some simple lookup and filtering facilities implemented on Storage instances–but again, not true searches–whittled down the sets of items from each Storage to be combined (so the Student Storage was able to filter using a simple where clause/predicate, and the Class Storage was able to do the same). Then I loaded those sets into a temporary database (H2–obviously could have been anything) and did the more complicated joining there (minus the simple where clauses/filters that were used to get me the candidate sets). (A tip of my hat to a former colleague for first exploring this approach.) The result, of course, was dog slow and could not be used on enormous datasets, but performance was never a priority and the client understood that they would pay dearly in performance costs for this approach. I was hoping that someone somewhere smarter than me had figured out how to bridge querying disparate systems in a better way.

Thanks for the links to the papers; very interesting reading.

Part of the point of splitting storage and query is that it becomes easier to do cross-storage queries. If you store objects in many places, but index/query them in one (again, the website vs Google analogy), it becomes supertrivial to query stuff in different places. The LDAP database problem you outline is one of the cases I had in mind when I designed these API’s in Qi4j, because I want to be able to do the same thing. In Qi4j our primary indexer is going to be Sesame2 (i.e. RDF), with SPARQL as the main query language (although it’s usually hidden under a domain-oriented Java API). Will be very interesting to see how it works out.