The Scenario: Disconnected ML and web development
Located in London, Cognitiv+ streamlines the legal review process by helping companies and their employees in real estate, financial services, insurance, procurement, legal services, and construction navigate and understand their documents and make better decisions. Getting contracts right is a high-stakes process for customers: a typical Fortune 1000 company handles more than 40,000 ongoing contracts, and industry estimates put the cost of inefficient contracting anywhere from 5 to 40 percent of the total contract value.
To help customers avoid these risks, Cognitiv+ uses machine learning (ML) to automatically review and classify documents, extract the information professionals need, and present it in reports. It’s easier, cheaper, faster, and more efficient than a manual review. Although this may sound simple, all the document data is unstructured, and most of it comprises legal and contractual documents, often hundreds of pages long.
Cognitiv+ customer workflow.
Initially, the company’s data scientists built ML models that, while comprehensively categorizing almost every sentence in the lengthy documents, presented customers with too much extraneous information, resulting in a less-than-optimal experience.
The challenge was amplified because one team using Node.js and AngularJS developed the web app for user interactions. Another handled ML development in Python. The two teams worked autonomously with little cooperation or understanding.
Adding to the challenge, a monolithic backend was becoming a single point of failure that wouldn’t scale. The single database for the initial product was a NoSQL document-oriented database system that worked fine for information but fell short in how Cognitiv+ could extract and present information. Besides handling documents, there was also almost no way to control constantly-changing elements such as users, organizations, projects, and roles — or to manage the complex relationships among them.
The document-centric data model had to be improved. For instance, assigning roles took up too much space, and obtaining results required too many queries. The Cognitiv+ team also found that the key to analyzing documents and presenting the right results to customers, without irrelevant information or false positive results, was to get more granular with the data.
They realized they needed to break documents into a hierarchical structure of zones and sections, dividing them by, for example, titles, contents, paragraphs, and sentences. Documents also had to be categorized based on the entire document, and the sections, paragraphs, or text snippets within the document. Only then would they be passed to a specific ML model for processing.
How data is logically structured in Cognitiv+.
D’Anna and his team developed an architecture with a front-end and an API gateway with Redis handling authorization. ArangoDB runs the remaining services — authentication, user management, and asynchronous interactions via NATS — all with controlled access to specific collections within the database.
They are now experimenting with ArangoML to analyze information better, extract pertinent data, and store ML experiments. ArangoDB is also used to obtain targeted data quickly and feed it to the Ray framework to produce ML training materials. Once models are trained, the team uses GraphML to make sense of them and traverse extracted insights.
Reporting is also becoming more sophisticated with ArangoDB in place. For example, the team can target specific data, such as the risks of accepting a contract, helping customers make smarter decisions.
Below is what Cognitiv+ built for searching and analyzing legal documents. This implementation includes a front-end and an API gateway with Redis handling authorization. ArangoDB runs the other services — authentication, user management, and asynchronous interactions — all with controlled access to specific collections within the database.
Cognitiv+ application architecture.