Introducing Genie Code | Databricks Weblog

We’re excited to announce Genie Code, the most recent addition to the Databricks Genie household. Up to now six months, agentic coding instruments have essentially modified software program engineering; Genie Code brings that very same transformation to knowledge groups. Genie Code can autonomously perform complicated duties reminiscent of constructing pipelines, debugging failures, transport dashboards, and sustaining manufacturing techniques.

Not like brokers that focus solely on writing code, Genie Code additionally operates as a proactive manufacturing agent. It displays your Lakeflow pipelines and AI fashions within the background, triaging failures, dealing with routine DBR upgrades, and investigating anomalies earlier than your staff even notices.

It does all this by deeply integrating with Unity Catalog in order that it understands your enterprise’s knowledge, semantics, and governance insurance policies. Genie Code considerably outperforms a number one coding agent by greater than 2x on real-world knowledge science duties.

Rise of Agentic Information Work

Agentic coding instruments have reworked software program engineering, transferring builders past autocomplete and towards agent-driven improvement. With a single immediate, engineers can now scaffold options, refactor code, and deploy prototypes in seconds. This shift has been pushed by advances in LLMs and by agentic techniques that may interpret the complicated context of recent software program codebases.

Most brokers in the marketplace concentrate on code as the ultimate product. Nonetheless, for knowledge groups, code is merely a automobile to govern and perceive the underlying knowledge. That is precisely why software-centric brokers usually wrestle with knowledge work. In an information ecosystem, context lives not simply within the script but additionally in utilization patterns, lineage, and enterprise semantics. 

Accessing this context is important as a result of the stakes are excessive. Dashboards drive enterprise selections, pipelines energy manufacturing techniques, and machine studying fashions affect real-world outcomes. For knowledge groups, the pace and leverage supplied by brokers have to be paired with absolute accuracy, reproducibility, and governance.

Genie Code is an AI agent constructed particularly for knowledge. It leverages Unity Catalog to mechanically curate essentially the most related knowledge and content material as you’re employed. It creates customized search indexes, customized directions, information shops, and extracts utilization patterns from lineage. Better of all, it will get smarter the extra your staff makes use of it. This deep integration into Unity Catalog is way superior to any system that merely reads the information from the skin.

We have seen the impression of Genie and Genie Code firsthand at Databricks, throughout each technical and non-technical customers. Our gross sales staff makes use of it to get a whole image of each buyer earlier than conferences, summarizing key consumption metrics, assist tickets, and up to date interactions in seconds. Product Managers use Genie Code to construct dashboards from a hand-drawn sketch of charts and graphs. Our finance staff runs budget-versus-actual evaluation and superior ROI modeling. Our management staff solutions knowledge questions in actual time throughout strategic discussions, lowering follow-up and accelerating complicated selections. Throughout the corporate, these instruments have modified how we work with knowledge.

What Genie Code Does:

• Acts as an knowledgeable machine studying engineer: Genie Code handles full ML workflows end-to-end. It causes via complicated issues to plan, write, and deploy fashions, whereas logging experiments to MLflow and fine-tuning serving endpoints for peak efficiency.
• Deep knowledge engineering experience: Whereas a novice engineer would possibly write a script that works on check knowledge, Genie Code designs like a senior architect. It accounts for the variations between staging versus manufacturing environments, builds workflows for change knowledge seize and applies knowledge high quality expectations.
• Proactively maintains and optimizes: Genie Code displays Lakeflow pipelines and AI fashions within the background to triage failures and examine anomalies. It autonomously analyzes agent traces to repair hallucinations and tunes useful resource allocation earlier than a human intervenes.
• Understands enterprise context: Built-in with Unity Catalog, Genie Code enforces current governance insurance policies and entry controls. It understands enterprise semantics and audit necessities and federates enterprise knowledge, together with knowledge from exterior platforms.
• Improves over time: Genie Code grows smarter the extra groups use it. Via persistent reminiscence, it mechanically updates inner directions based mostly on previous interactions and coding preferences. On inner knowledge science duties, Genie Code outperforms main coding brokers 77.1% to 32.1% on high quality.

With Genie Code, knowledge groups transfer from prompting a copilot to delegating actual work: constructing pipelines, debugging failures, transport dashboards, and sustaining manufacturing techniques — autonomously, finish to finish.

At SiriusXM, Genie Code helps every part from authoring notebooks and complicated SQL to reasoning via desk relationships and debugging pipelines. It acts as a hands-on improvement accomplice that helps our knowledge groups ship high-quality work in much less time. — Bernie Graham, VP Information Engineering, Sirius XM

Highest High quality Agent for Information and AI Work

Genie Code is just not powered by a single mannequin. It’s an agentic system that routes duties throughout a number of fashions and instruments, mechanically choosing the right mannequin for every job, whether or not that may be a frontier LLM, an open supply mannequin, or a customized mannequin hosted on Databricks. This eliminates the necessity for customers to manually change between fashions or guess which one will produce one of the best consequence.

Genie Code can also be deeply built-in with Databricks APIs, permitting it to establish the fitting knowledge property, assemble wealthy context, and generate greater high quality queries. Databricks Analysis constantly tunes the system, benchmarking the newest fashions from main AI labs alongside customized fashions operating on the platform.

In our current efficiency benchmarking on real-world knowledge science and analytics duties collected from inner customers, Genie Code considerably outperformed a number one coding agent geared up with the Databricks Mannequin Context Protocol (MCP) servers.

• Genie Code: 77.1% Solved duties
• Main Coding Agent + Databricks MCP: 32.1% Solved duties

Genie Code Helps the Full Lifecycle of Information Work

Prepare and Consider Machine Studying Fashions

Genie Code acts as a devoted ML engineer embedded in your workflow. Ask it to “practice a forecasting mannequin predicting gross sales in @sales_table” and it’ll purpose via the total pipeline: 

• Figuring out and profiling options
• Cut up coaching, validation, and check datasets appropriately
• Prepare a number of mannequin sorts and evaluate them, operating hyperparameter sweeps to coach the absolute best mannequin.
• Evaluates outcomes throughout metrics like AUC, F1, RMSE, and R²
• Generate plots for function significance, confusion matrices, and ROC curves
• Observe experiments in MLflow
• Advocate enhancements based mostly on mannequin diagnostics

As soon as deployed on Databricks Mannequin Serving, Genie Code stays within the loop: it will possibly examine endpoint well being, analyze traces, and advocate optimizations. You may learn extra on this within the “From Code to Manufacturing: Observability with Genie Code” part under.

Genie Code modifications how our knowledge groups function. As an alternative of sewing collectively notebooks, pipelines, and fashions manually, we will hand off complicated workflows to an AI accomplice that understands our knowledge, governance, enterprise context, and inner libraries reminiscent of Repsol Synthetic Intelligence Merchandise. It accelerates every part from time collection forecasting to manufacturing deployment, with out sacrificing rigor or management. — Emilio Martín Gallardo, Principal Information Scientist, Information Administration & Analytics, Repsol

Create Manufacturing-Prepared Information Pipelines

Genie Code is your knowledgeable knowledge engineer, constructed that can assist you design and evolve dependable knowledge pipelines.

• Create pipelines from pure language: Describe what you want and Genie Code generates a whole Spark Declarative Pipeline with ingestion, transformations, and knowledge high quality expectations in-built.
• Prolong current pipelines: Add datasets, modify transformations, write AutoCDC flows, configure Auto Loader, and apply knowledge high quality expectations, all inside the context of your present pipeline.
• Perceive pipeline habits: Examine outputs, hint knowledge circulation into downstream tables, and floor surprising modifications in row counts or schemas.

Genie Code has moved us past assisted coding into true agentic knowledge engineering. It might probably analyze our Lakeflow pipelines, suggest multi-file modifications with diffs, execute runs with safeguards, and iterate via failures till points are resolved. It feels much less like autocomplete and extra like a collaborator embedded in our workflow. — Nishit Gajjar, Tech Lead, World Infrastructure Know-how Supplier

Create Dashboards with Reusable Semantic Definitions

Genie Code can generate visualizations, configure filters, and arrange multi-page dashboard layouts, all with reusable semantic definitions. It connects these definitions to filters, calculations, and layouts that scale as dashboards develop, serving to groups transfer quicker whereas sustaining consistency.

With Genie Code, our groups are delivering AI-driven analytics and automatic workflows in weeks, not months. Low-code brokers assist us transfer quicker whereas staying aligned to governance, enabling mission and engineering groups to get natural-language insights from complicated knowledge with out slowing supply. — Russell Singer, Chief Information Architect, Bechtel Company

Autonomous Multi-Step Planning and Execution 

Present a high-level goal, reminiscent of “Determine flight delay dangers and construct a monitoring dashboard”. Genie Code causes via the necessities, formulates a multi-step plan, and executes it throughout all Databricks Notebooks, AI/BI Dashboards, and Lakeflow in a single dialog thread.

What we’re seeing at Danfoss is that Genie Code modifications the roles inside an information staff, supporting our strategic concentrate on digitalization and AI. Information scientists nonetheless present course and evaluate, however engineers, analysts, and area consultants can now actively work in notebooks with the assistant and contribute to superior analytics workflows. It turns knowledge science into a way more collaborative staff exercise. — Radu Dragusin, Principal Engineer, Information & AI, Danfoss

Exploratory Information Evaluation with Deep Contextual Search

Genie Code makes use of reputation, lineage, code samples, and Unity Catalog metadata to search out essentially the most related datasets for any evaluation. This deep contextual search eliminates the guide effort of looking for knowledge and ensures that your work is predicated on essentially the most correct and often used tables inside your group.

I’m genuinely mesmerized. Genie Code appears like a glimpse into the way forward for how knowledge work will get accomplished. — Sameer Yasser, Sr. Information Engineer, Sundt Building

Customization and Extensibility

Genie Code is a versatile platform designed to be tailor-made to your staff’s particular requirements and exterior tech stack. There are three main methods to increase its capabilities:

1. Exterior Tooling by way of Mannequin Context Protocol (MCP)
   Genie Code helps Mannequin Context Protocol (MCP), an open normal that permits it to securely work together together with your exterior instruments, APIs, and documentation. This permits autonomous workflows that reach past the Databricks workspace.

For instance, if you’re assigned a Jira job to coach a brand new ML mannequin, Genie Code can mechanically collect context from it, carry out the duty, and replace the ticket with the outcomes.

Join Genie to your inner Confluence, Google Drive, GitHub, or Notion by way of MCP so it will possibly reference your staff’s particular runbooks and knowledge dictionaries when troubleshooting.

2. Agent Abilities: Outline domain-specific capabilities to show Genie Code the right way to carry out complicated duties constantly. Whether or not it’s a selected approach your organization handles PII masking or a customized framework for knowledge validation, Abilities be certain that the AI follows your group’s greatest practices each time. Abilities observe the open Agent Abilities format.
3. Reminiscence: Genie Code grows smarter the extra you employ it. Via persistent reminiscence, the agent mechanically updates its inner directions based mostly in your previous interactions. It learns your coding preferences, remembers which datasets you employ most often, and retains context throughout periods.

From Code to Manufacturing: Observability with Genie Code

Writing code is barely step one. Sustaining it’s the actual problem. Genie Code acts as an observability agent to maintain your knowledge and AI workflows wholesome. Whereas hundreds of shoppers use Databricks to serve subtle AI functions, debugging these fashions in manufacturing is usually essentially the most time-consuming a part of the lifecycle.

Genie Code now integrates instantly with Databricks Mannequin Serving and MLflow 3.0 to automate this course of. As an alternative of manually looking out via logs and traces, you should use Genie for:

• Endpoint well being checks: Get a full standing report throughout compute, request dealing with, and server logs in a single immediate.

• Agent high quality evaluation: Floor refined points like hallucinations, incorrect software calls, and consumer frustration patterns throughout complicated agent traces in actual time.

• Manufacturing troubleshooting: When incidents happen, Genie cross-references server logs and metrics to automate the primary spherical of prognosis and scale back time to decision.
• Endpoint optimization: Get suggestions on provisioned concurrency, {hardware} configs, and auto-scaling based mostly on Databricks greatest practices.

Background Brokers that Maintain Workloads Wholesome

Genie Code is designed to work within the background in order that your knowledge stays wholesome even after you shut your laptop computer. You may deploy a number of brokers in parallel to deal with the operational work that sometimes consumes an information engineer’s week. These background brokers transfer past reactive assist towards proactive upkeep by dealing with repetitive duties reminiscent of responding to job failures and managing routine upgrades. When a pipeline breaks, the agent identifies the foundation trigger and suggests a repair solely after validating it in a safe sandbox surroundings. 

For instance, if a manufacturing pipeline fails because of a schema mismatch, reminiscent of a column altering from an INT (150) to STRING (“150 USD”), Genie Code will pinpoint the failure and mechanically repair the damaged pipeline. 

Background brokers are coming quickly. 

Grounded in Unity Catalog: Built-in Safety and Governance

Genie Code is constructed instantly on Unity Catalog. This integration ensures that the agent follows the identical safety and governance guidelines as the remainder of the Databricks platform.

When Genie Code searches for knowledge, it solely surfaces property the consumer is permitted to entry. When it builds a pipeline, it adheres to current lineage and entry controls.

• Native Revision Historical past: Each edit is tracked via the Databricks versioning system. You may roll again modifications throughout notebooks, queries, information, and Lakeflow pipelines with full confidence.
• Constructed-in Guardrails: Genie Code is designed to proactively ask for affirmation earlier than executing code that may modify underlying tables.
• Entry Management Enforcement: Genie Code by no means exposes knowledge property {that a} consumer is just not permitted to see.
• Complete Audit Logging: Your group maintains full visibility into how Genie Code is getting used via current audit infrastructure.

Obtainable in your Workspace immediately

Genie Code is Usually Obtainable in your Databricks workspace proper now. You’ll find the Genie Code panel in your notebooks, SQL editor, and Lakeflow Pipelines editor immediately—no complicated configuration required.

Be taught Extra

If you want to study extra about Genie Code:

• Go to our internet web page to know key Genie Code options and use instances and study the way it works throughout the Databricks platform
• Watch the demo to see Genie Code plan and execute actual knowledge workflows finish to finish
• Learn the documentation to begin utilizing Genie Code in your personal workspace immediately

We’re excited to see what you construct with Genie Code and the way autonomous brokers will reshape the way in which your knowledge groups work in Databricks.