Querying Unstructured Data with Pyramid

The lack of effective methods for how to analytically use unstructured data is a common problem that we often hear from our customers. Getting the data in is proving to be far simpler than getting the analysis of the data out. Pyramid now has an answer – starting with its new data connector for MongoDB.

Unstructured vs Structured

It seems many organizations have found numerous data-related problems can be solved effectively with the simplicity and speed of unstructured data stores  – especially in the realm of IoT. Since the target system requires no formalized structure, it is significantly faster to store loads of incoming data in a framework that is less concerned with how it looks, its data types, structure, or even its size. This makes the ‘storing of data’ business significantly simpler (and usually more cost-effective) than the relational data business: where data needs to be carefully inserted into databases and tables; it needs to meet data type and sizing standards, and it needs to be committed carefully to ensure it can be rolled back in the event of failure. The unstructured universe, in many respects, is unconcerned with most of these ‘requirements’.

Indeed, it goes a step further. Related data elements need to be added to multiple tables in a relational world, while they can be stored as a simple parcel of data in a single addition to the unstructured system – using ‘nested’ data fragments and records. Again, this simplifies the data capture process dramatically.

The Catch

Anybody in the data business will comfortably admit that there is no value in collecting data if you cannot get some type of insight from it. That includes accounting data to see the status of a business; web logs to see which pages are most visited, and fitness device recordings to monitor your health. Or put simply, if we cannot analyze the data to derive some understanding, the collection of it is pointless. In business terms, there needs to be a way to monetize the information – usually done with numerous tools and applications; some specialized and others generic.

Using structured data in classic business analytics is the easiest example of such monetization – a very mature concept that is almost universally embraced. However, the same cannot be said for unstructured data. Marrying classic business analytic tools and technologies with the fluidity of unstructured data has proven to be a challenge.

And the reason is obvious: classic data analytic technologies require structure.

The Fix

There are various intermediate tools that address the gap, by using specialized tools for ‘scanning’ unstructured data; proposing a likely structure; and, allowing the analysis tools to query them in a common language. MongoDB’s BI Connector, included with MongoDB Enterprise, is one such example.

The connector, using a ‘probabilistic schema’ model, works with Pyramid to directly query the unstructured MongoDB data sources  – giving end users the ability to analyze the data in the same way as analyzing a relational/structured data stack.

Whether the data is stored as flat fragments, as separated entities or nested data fragments in more complicated structures, the tools are able to decipher the schemas and present them to the end users for analysis.

Pyramid and MongoDB

What makes the Pyramid implementation of this solution interesting, however, is that we have brought the same level of simplicity and elegance available on all our data sources to MongoDB.

Using Pyramid’s virtual schema and modeling framework, classic joins and metrics can be used to embellish the schema  – giving the end-user the simplicity and freedom to analyze MongoDB data the same way they might analyze data in MS SQL Server, Oracle, Teradata, or SAP Hana.

In Pyramid’s discovery tool, MongoDB data is treated like any other data source, with all associated analytic and visualization capabilities intact.  We can build custom formulas, lists, and KPIs against the data while putting it into presentations, dashboards, and analytic publications like any other data source.

By giving unstructured data a full seat at the proverbial main table, users can start unlocking the many interesting insights captured in the data itself.

To see it in action, watch the video of MongoDB and Pyramid working together here,