If you are unfamiliar with Databricks, it is an analytics service based on Apache Spark. It aims to simplify data information by streamlining workflows and allowing one-click setup. “Azure Databricks is designed in collaboration with Databricks whose founders started the Spark research project at UC Berkeley, which later became Apache Spark,” explains Rohan Kumar, corporate vice president, Azure Data said earlier this year. “Our goal with Azure Databricks is to help customers accelerate innovation and simplify the process of building Big Data & AI solutions by combining the best of Databricks and Azure.”
The platform was announced at Microsoft’s Connect() developer conference last year. In March, the company made Azure Databricks generally available. Microsoft was at the Spark + AI Summit yesterday when it announced the new Databricks abilities. With support for GPU enabled VMs, Microsoft is giving dev’s the tools to train and build artificial intelligence models at scale and then deploy them.
New Runtime
At the event, the company put the focus on a new machine learning runtime for Databricks. This runtime allows distributed multi-GPU training for deep neural networks. Using Horovod, developers can now create deep learning models without needing lots of code. “This runtime enables developers to build deep learning models with a few lines of code. Previously, developers had to invest considerable time and effort to leverage these toolkits. Now, they no longer have to write their own logic to load data, distribute training code to multiple clusters and validate model accuracy.”
