MongoDB makes it so easy to persist Entities with dynamic set of attributes without requiring any POJO/XSD/ORM layer

Goal :

  • Persist incoming stream of Objects into the Database.
  • Parse the incoming XML/Json data to extract “Object of similar type” (i.e. a particular Business Entity) .
  • The Business Entity may have a fixed set of attributes or a dynamically changing set of attributes.
  • We should not create/maintain ORM layer and no need for POJO conversions from/to wsdl/http streams.
  • We should keep the persistence service layer very very light-weight.
Setup :
  • mongodb2.0.1 (A database named “sampledb” needs to be created in MongoDB)
  • spring-data-mongodb 1.0.0.M4, mongo-java-driver 2.5.2 (and all other dependencies specified in pom.xml)
Development Environment : SpringSource Tool Suite – 2.8.1.RELEASE
Implementation Approach :
Lets first start with traditional JDBC
  • SimpleJdbcInsert Command make things really simple when Objects of same type has a fixed set of attributes.

Create the singleton instance of the SimpleJdbcInsert Command for the corresponding table.

Prepare the Map of key-value pairs by parsing the incoming data and simply push the data into the tables against the corresponding columns (keys in the map).

Number num = insertCommand.executeAndReturnKey(parameters);
Or insertCommand.executeBatch(…)

  • SimpleJadbcInsert can be extended and tweaked to leverage DatabaseMetadata and perform DDL on RDBMS programmatically.
  • But its very costly to Create/Update/Delete columns in relational data-store on-the-fly.

So finally we take resort to MongoDB

  • If the Business Entity has a pre-defined set of attributes we should leverage the well-known annotated document object.
  • Otherwise if Entity structure is changing constantly to accommodate newer attributes, then just create new DBOnjects and DBRefs.
MongoConfig creates a MongoTemplate against the database “mysampledb”
MongoDataAccessService provides all CRUD operations.
MongoRestController specifies all the Rest operations.
Sneak Peak into the code .. oxrg.jdom.Element rootElement = jdomDocument.getRootElement(); String rootElemName = rootElement.getName();

if (!mongoDataAccessService.collectionExists(rootElemName)) {
mongoDataAccessService.createCollection(rootElemName);
}
// we assume the element contains all the values for the table-columns as the attributes.
List list = rootElement.getAttributes(); DBObject parentDocument = new BasicDBObject();
// suppress _ClassName in the DbObject

for (Object attr : list) {
org.jdom.Attribute source = (org.jdom.Attribute) attr;
if (source.getName().equals(“Id”)) { parentDocument.put(“_id”, source.getValue());
// use the original incoming Id // candidate for shard key — TODO
} else { parentDocument.put(source.getName(), source.getValue()); }
}


// Now iterate through the Child Elements and store them as Database
References DBRef childDocumentRef = new DBRef( mongoDataAccessService.getDB(“mysampledb”),
elementName, elementId); parentDocument.put(elementName + “_ref”, childDocumentRef);

How to run the application :
  • target folder already contains a pre-built war file.
  • just drop the war inside an app server context root.
  • http://localhost:8080/dynamic-data-collection/mongo/entities/add
  • Run the MongoRestClient to dynamically extract an entity from XML fragment and add it to the corresponding Collection.
  • Simple MongoClient just creates instance of a predefined Mongo Document. This comes handy when Entity has a fixed set of attributes.

Code : https://github.com/kaniska/Dynamic-Object-Persistence

Take Away :
  • Analyze the Business requirements and accordingly categorize the types of Entities.
  1. If all Entities can have pre-defined schema :   then use SimpleJdbcInsert Command along with MySQL / Oracle for best performance.
  2. But in case the Business Process consists of both fixed and dynamic entities, then define annotated POJO (Mongo Document) for the Entities having a fixed structure.  (Example TestMongoService.java in github )
  3. If same type of Entity has different set of attributes for different instances then consider creating them dynamically (DBObject that holds bson map)
  • So the choice of database (MongoDB Vs RDBMS) – depends upon how much dynamism one needs to accommodate in the application.
Future Improvements
  1. Externalize the MongoDB configuration in external properties file.
  2. Use the latest Spring-MongoDB release
  3. Work with XPathOperations instead of parsing xml fragments manually.
  4. Figure out if Spring-Data MongoDB API leverages latest Morphia annotations for journal-sync, suppressing _ClassName in objects etc. (for reducing latency and improving performance )