Building an Operational Machine Learning Organization from Zero

•Anthony G. Tellez•3 min read
Machine LearningMLOpsDatabricksCryptocurrencyBlockchainConferenceBlockFi

The complete journey of building an operational machine learning organization from the ground up at BlockFi, a cryptocurrency financial services company.

Our Journey with Databricks

Building a Cross-Functional ML Team

Key steps in assembling an effective ML organization:

  • Recruiting data scientists and ML engineers
  • Creating collaboration frameworks
  • Establishing shared goals and metrics
  • Building bridges between business and technical teams

Scoping Business Problems for Executive Buy-In

Strategies for securing leadership support:

  • Identifying high-impact use cases
  • Quantifying potential ROI
  • Demonstrating quick wins
  • Building credibility through incremental success

Conveying a Strategic Vision

Communicating ML strategy effectively:

  • Long-term roadmap development
  • Technology stack decisions
  • Build vs. buy considerations
  • Integration with existing systems

Operationalizing ML & Data Science

Making ML production-ready:

  • MLOps infrastructure
  • Model lifecycle management
  • Monitoring and observability
  • Continuous improvement workflows

Building Clear Business Objectives

Aligning ML projects with business goals:

  • KPI definition
  • Success metrics
  • Stakeholder management
  • Measuring business impact

Blockchain Analytics for Security

Unique Security Problems in Crypto

Challenges specific to cryptocurrency:

  • 24/7 trading environments
  • Irreversible transactions
  • Regulatory compliance (OFAC sanctions)
  • Cross-chain tracking

Estimated Costs

Understanding the stakes:

  • Fraud losses - Direct financial impact
  • Account takeover - Customer trust and retention
  • Regulatory fines - OFAC sanctioned entities and compliance violations

Graph Theory and Blockchain Analysis at Scale

Technical deep dive:

  • Nvidia Rapids - GPU-accelerated graph analytics
  • Apache Arrow - High-performance data interchange
  • Graphistry - Visual graph analytics
  • Databricks - Unified analytics platform

Onboarding Business Teams

Democratizing data access:

  • Training programs for non-technical users
  • Self-service analytics
  • Collaborative notebooks
  • Knowledge sharing frameworks

Using ML to Improve Platform Stability

Crypto Never Sleeps: 24x7 Trading

Operational challenges:

  • Zero downtime requirements
  • Global user base across time zones
  • Peak load management
  • Incident response

Cost of an Outage

Business impact analysis:

  • Lost trading revenue
  • Customer satisfaction
  • Regulatory implications
  • Competitive disadvantage

Forecasting Techniques

Practical approaches:

  • Simple Regression - Baseline models
  • Prophet - Time-series forecasting for seasonal patterns
  • SARIMAX - Advanced statistical methods
  • Trade-offs - Accuracy vs. complexity vs. interpretability

Integrating Non-Traditional Indicators

Feature engineering for infrastructure forecasting:

  • Market volatility as a predictor
  • Social media sentiment
  • Blockchain network metrics
  • External market events

Operationalizing Predictions

Making forecasts actionable:

  • Early warning systems
  • Integration with O11y (observability) tools
  • Automated alerting
  • Capacity planning automation

Presentation Materials

Watch Video (Requires login)

Download Slides

Presented at Databricks Data & AI Summit 2022

← Back to Blog