This article was originally published in Intelligence Community News on December 02, 2019.
Deploy machine learning at the enterprise level in your agency by combining it with DevOps
Machine learning is a game-changer for the Intelligence Community. Unfortunately, many agencies only apply machine learning solutions to narrow challenges. The result? They see one-off successes in niche categories, but these wins are not tied to any broader goals or even to each other. Agencies fail to reap enterprise-wide benefits.
Specifically, machine learning models can “learn” on their own based on the continuous evaluation of data. This automated approach to analyzing data frees up your agency employees to perform more high-value creative analysis.
But to get to this point—leaving the machines to do the grunt work and your people to do the visionary work—your agency needs to approach machine learning from the enterprise standpoint. Here’s how.
Combine Machine Learning with DevOps
When you look across agencies, there are dozens of different tools and applications to complete missions. The abundance of data is voluminous. In fact, it staggers many agencies; they simply don’t know how to put it to practical use.
It’s our strong assertion that machine learning can’t be a standalone data science exercise. It needs to be baked into your DevOps (i.e., your software development and IT operations). This combined approach materially moves the ball forward for your agency.
How? Your systems and operations are linked to machine learning models—continually feeding the models new data that they continuously evaluate and re-evaluate to learn more and to learn better, providing your analysts and data scientists with the best information. In short, it’s a never-ending successful feedback loop.
This is called Machine Learning Operations (MLOps). Let’s take a closer look.
Set up an orchestrated pipeline for data
Right now, your agency is likely set up with a machine learning pipeline. A narrow dataset flows into a machine learning model designed to solve a narrow challenge. Yet your software deployment has a pipeline of data, too, as well as your software integration. As a result, you have multiple pipelines of data flowing through your agency.
To achieve maximum value of machine learning at an enterprise level, all your machine learning and DevOps pipelines of data need to converge into one MLOps pipeline.
The MLOps pipeline provides your team with:
- An iterative loop of sprints that cover data acquisition, business understanding, modeling, deployment, sensemaking, and data acquisition
- Scrum teams comprised of cross-functional groups: a scrum master, data engineers, data scientists, software engineers, system engineers, and product owners
- Constant curation, data labeling, and quality control, operations, and sustainment
- Continuous integration and continuous principles
- Repeated processes for model creation, model evaluation, model training, testing, deployment, and monitoring
Agile delivery of Machine Learning
To succeed with machine learning across the enterprise, you need a solution that scales. The MLOps pipeline provides a robust environment where data and models can be accessed by various team members. It’s important to note that many data scientists do not have computer science backgrounds. They don’t understand the discipline it takes to deliver a production-ready system––and that’s your goal in order to achieve better results.
Machine learning must be integrated with other traditional technology. This includes but is not limited to configuration, data collection, feature extraction, data verification, machine resource management, analysis tools, process management tools, visualization tools, server infrastructure, and monitoring.
“By combining an Agile approach to machine learning with DevOps, leveraging automation, delivery discipline, and a focus on quality, our solutions leveraging machine learning are delivered in a predictable and highly reliable manner,” says Keith Sheppard, CIO of NT Concepts.
To deliver on machine learning potential, it takes a solution architected using a purposeful approach. NT Concepts uses a proven methodology for each data science project.
Keep the end goal in mind
If you keep approaching machine learning solutions on a piecemeal basis, you’re going to be stuck with a lot of proofs of concepts that don’t point to a larger enterprise solution. In our experience, machine learning solutions have unique data dependencies and complexities that can impact the process of moving them into production.
With MLOps, models and results are always current because the process is iteratively run. The process can handle a wide range of data types (structured and unstructured) and tasks including object detection, natural language processing, and signal classification.
Machine learning is a broad field that includes a wide range of expectations. Using a MLOps approach can help your agency continuously iterate, refine, and deploy models at an enterprise level. Holistically, this leads to significantly better outcomes and results for your Intelligence analysts and decision makers.