Data scientists do experiments. They repeat. They are rehearsing again. They create insights that drive business decisions. They work with partners in IT to harden ML use cases into production systems. To work effectively, data scientists need flexibility in the form of access to enterprise data, streamlined tools, and infrastructure that just works. Agility and enterprise security, compliance and governance are often at odds. This tension leads to more friction for data scientists, more headaches for IT, and missed opportunities for businesses to maximize their investments in data and AI platforms.
Resolving this tension and helping you make the most of your current ecosystem investments is at the heart of the DataRobot AI platform. The DataRobot team is hard at work on new integrations that make data scientists more agile and meet enterprise IT needs, starting with Snowflake. In our 9.0 release, we’ve made it easy for you to quickly prepare data, design new features, and later automate model deployment and monitoring in your Snowflake data landscape, all with limited data movement. We’ve tightened the loop between ML data preparation, testing, and testing before putting models into production. Now, data scientists can be agile in the machine learning lifecycle thanks to Snowflake’s scale, security, and management benefits.
Why are we focusing on this? Because the current ML lifecycle process is broken. On average, 54% of AI projects make it from pilot to production. Consequently, almost half of AI projects fail. There are several reasons for this.
First, it is difficult to conduct experiments long enough to identify meaningful patterns and drivers of change. Prototyping the loop, especially preparing ML data for each new experiment, is tedious at best. It is difficult for data scientists to securely connect, browse and preview and prepare data for ML models, especially when the data is spread across multiple tables. From there, every time you do a new experiment, you go back to preparing the data. And once you do find a signal and build a great model, it’s hard to produce those ML models.
Models that go into production require time-consuming management through monitoring and replacement to maintain forecast quality. The lack of integrated tools throughout the process not only slows down the productivity of data scientists, but also increases the total cost of ownership, as teams must integrate tools to get through this process. The DataRobot AI platform is focused on making the entire ML lifecycle seamless, and today we’re doing even more with our new Snowflake integration.
Secure, seamless, and scalable ML data preparation and testing
Now DataRobot and Snowflake customers can maximize the return on their investment in AI and their cloud data platform. You can seamlessly and securely connect to Snowflake with support for external OAuth authentication in addition to basic authentication. DataRobot’s secure OAuth configuration sharing allows IT administrators to configure and manage access to Snowflake.
DataRobot will automatically inherit access controls, so you can focus on building value-driven AI and IT can simplify their backlog.
With our new integration, you can quickly browse and preview data in the Snowflake landscape to identify the data you need in your machine learning use case. Automated data preparation and well-defined APIs allow business problems to be rapidly modeled as training data sets. Downward integration minimizes data movement and allows you to use Snowflake for secure and scalable data preparation, as well as a feature engineering engine, so you don’t have to worry about computing resources or waiting for processes to complete. Now you can take full advantage of the scale and elasticity of your Snowflake instance.
With our DataRobot notebooks, you can use Snowpark for Python in conjunction with the DataRobot Python Client to quickly connect to Snowflake, explore, train, and create machine learning experiments with your Snowflake data. You can use both platforms in a way that makes the most sense for you, using Snowpark and the DataRobot developer framework, which has native support for Python, Java, and Scala. Because this integration is native to the DataRobot AI platform, you get your time back in one frictionless experience.
One-click model deployment and monitoring in Snowflake
Once the built models are ready for deployment, you can run them in Snowflake with a single click. Supported models can be directly deployed to Snowflake as a Java UDF by DataRobot. This functionality includes the ability to deploy models built outside of DataRobot to Snowflake. This means you can bring the model directly into Snowflake’s managed runtime, allowing enterprises to make accurate predictions at scale with sensitive data in the database, and without configuration noise. One-click model deployment also gives ML practitioners the flexibility to use normal queries or more advanced functions such as Stored Procedures from Snowflake to read evaluation data, evaluate data, and write predictions.
With the deployment of the one-click model comes more powerful monitoring capabilities that allow continuous monitoring of not only the health of the deployment service, but also the movement and accuracy. Model replacement is simplified with training and deployment workflows to ensure enterprise-grade reliability for production machine learning on Snowflake.
Snowflake and DataRobot. combining data and AI for business results
The new integration of Snowflake and DataRobot provides organizations with a unique and scalable enterprise platform for data and AI-driven business outcomes. We’ve reduced ML cycle time and made it easier for you to run more experiments, prepare datasets, and build ML models quickly, then get those models into production to drive value faster.
Try out the new integration and let us know what you like! Learn more from Torsten Grubbs, Director of Product Management at Snowflake, who will share more about these innovative capabilities at the DataRobot virtual on-demand event, From Vision to Value: Join us on March 16th and see more DataRobot and Snowflake integration first hand.
1: Gartner®, Gartner Survey Analysis. The most successful implementations of AI require a discipline, not a doctorate Eric Brethenau, Anthony Mullen Published 26 Aug 2022
About the writer
Senior Product Manager, DataRobot
Kian Kamyab is a Senior Product Manager at DataRobot. He developed his customer empathy and analytical edge as Managing Director of SAP’s New Ventures and Technologies group, Senior Data Scientist in the Enterprise Software Venture Studio, and founding team member of the James Beard Award-nominated Cocktail Bar. When he’s not creating AI/ML products that solve real-world problems, he’s handcrafting furniture and exploring the woods in and around San Francisco.
Meet Kian Kamyab