For many organisations, the prospect of realising functional applications in Azure databricks may seem little more than a distant possibility. For others, this will be something that has been achieved and taken as a given. In either scenario, there is always more that can be done to improve the performance of applications. While moving to the cloud is an important first step, there is a considerable amount more that needs to be done once that migration is completed. Jobs can often be inefficient leading to a gradual accumulation in cost that is ultimately felt by the business at a critical mass. No matter how finely-tuned those jobs are, they can generally be refined further to generate greater efficiencies and an optimised data ecosystem
Business Intelligence is Power
Optimising your workloads in Azure Databricks fundamentally boils down to operational intelligence. By having clear insights into what applications are running, how they are utilising resources, which users are accessing them, and at what times this is all occurring, a broad picture of the data landscape can be drawn. This kind of granular detail leads to faster diagnosis and resolution of application problems. For example, If an app is failing, rather than spending considerable time finding the metrics needed to inform your root cause analysis, we can instead draw these conclusions right away. This minimises downtime of what could be business-critical applications. So what kind of intelligence does the data team need to draw these insights? Importantly, they will need a detailed view of usage breakdown and trending over time across workspaces and users. Equally, it is essential to have visibility into data usage. This should cover things like what tables are being accessed when they’re being accessed and by who, and which applications. Consideration should also be given to the extent of custer usage

View Entire Article on