Building Polyglot applications using MetaCall Core

Harsh Mishra

This talk on Github: HarshCasper/pycon-my-2021

Agenda

  • Introduction to Polyglot development
  • Getting started with MetaCall Core
  • Understanding the underneath of the Core
  • Walking through a Polyglot Machine Learning application
  • Building a Polyglot application using Core
  • Q&A

Problems at hand



  • Slow and painful migration of legacy codebases. Example: Java -> Node JS
  • Lack of interoperability between libraries across languages. Example: Python -> NodeJS
  • Embedding low level libraries or scripts in high-level environments.

Introduction to Polyglot development

Why Polyglot development?



  • Help developers find the best possible solution through any framework/library possible.
  • Mix toolings, scripts, libraries amongst multiple languages without any friction.
  • Embed low-level and high-level code without sacrificing any portability and interoperability.

What is MetaCall?

MetaCall is an Open-Source Polyglot runtime. It brings forward:

  • Cross-language imports.
  • Multiple language support through a CLI and runtime.
  • N:N interoperability with a transparent API.

Demonstration

sum.py

def sum(a, b):
  return a + b

main.js

const { sum } = require('./sum.py');

sum(3, 4); // 7

CLI

metacall main.js

Installation

# Installation using curl
curl -sL https://raw.githubusercontent.com/metacall/install/master/install.sh | sh

# Installation using wget
wget -O - https://raw.githubusercontent.com/metacall/install/master/install.sh | sh

# Installation using Docker
docker pull metacall/core

Architecture of MetaCall

Understanding Ports



  • Inner architecture of MetaCall Core is driven by ports.
  • Ports offer MetaCall to other languages.
  • C API wrappers expose the functions to other languages.
  • Monkey Patching is used to make the API transparent for users overall.

Understanding Meta-Object protocol

Understanding Loaders

  • Loaders are used for embedding languages into MetaCall.
  • Act as plugins for a common interface wrapping the runtimes.
  • Loading them dynamically during execution.

Introducting MetaCall FaaS

  • MetaCall FaaS allows us to deploy applications built on MetaCall Core
  • Provides automated API generation and high performance server over a Kubernetes cluster.
  • Zero vendor locking and nudge towards NoOps development

Benchmarks

Software Time Bandwidth Calls/sec
Python C API 544 ms 42.0545MB/s 1.75227M items/s
MetaCall Python Variadic Arguments Call 988 ms 23.1689MB/s 988.54k items/s
MetaCall Python Array Arguments Call 903 ms 25.353MB/s 1081.73k items/s
Check out the source for MetaCall benchmarks

Walking through a Polyglot Machine Learning application

This project on Github: metacall/ml-news-article-scraper-example

Building a Polyglot application

What are we going to do?

  • Write a simple Python script.
  • Inspect the script using MetaCall CLI.
  • Load a script module in an Express API.
  • Test the API

Projects built using MetaCall

  • Acid Cam: Video manipulation using OpenCV which uses MetaCall to embed Python scripts in plugin system.
  • Pragma: Language used to build GraphQL APIs which uses MetaCall to import and execute functions in multiple languages.
  • Polyglot Kernel: IPython wrapper kernel using MetaCall Core to embed multiple languages into the Jupyter notebook interface.

Q&A

Any questions?

Thanks!

Harsh Bardhan Mishra
This talk on Github: HarshCasper/pycon-my-2021

References