lfex/py lfex/py

  • Stars
    star
    221
  • Rank 179,773 (Top 4 %)
  • Language
    Erlang
  • Created almost 10 years ago
  • Updated over 8 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
lfex/py
github

lfex

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Distributed Python for the Erlang Ecosystem

py

Distributed Python for the Erlang Ecosystem

Table of Contents

Introduction β†Ÿ

This project provides two key features:

  1. An easier interface to Python, wrapping ErlPort calls. This lets you do the following very easily:
    • Make module-level calls
    • Get module-level constants
    • Instantiate objects
    • Call object methods
    • Get object attributes
    • Call builtins and operators with convenient wrappers
  2. A means of running Python in a simple distributed context using all the well-known strengths of Erlang (fault-tolerance, scalability, concurrency, soft real-time, etc.).

What LFE py is not:

  • A PaaS; if that's what you're interested in, please take a look at CloudI.
  • A framework for pipelining jobs across distributed data (e.g., mapreduce). For that, see the Disco project.
  • A language-agnostic, general-purpose ports server. LFE py is, in fact, built on one of those: ErlPort!

LFE py was originally part of the lsci project, but was split out due to the ErlPort/Python-wrapping code being generally useful for all sorts of projects, not just scientific computing in Erlang/LFE. This bit of background should give further insight into the use cases LFE py was intended to address: scientific and (more recently) general computing in Python from the Erlang VM, with a focus on interactive workflows common in academic, research, and startup R&D environments. Just the sort of thing that IPython excels at, minus the GUIs :-)

Requirements β†Ÿ

To use py, you need the following:

  • lfetool and rebar (used by make targets to automatically set ERL_LIBS for deps)
  • Python 3
  • wget (used to download get-pip.py)

Setup β†Ÿ

For now, just run it from a git clone:

$ git clone [email protected]:lfex/py.git
$ cd py
$ make

Activate the Python virtualenv that was created by the make command you just ran. Then start up the LFE REPL:

$ . ./python/.venv/bin/activate
$ make repl-no-deps

Note that the repl and repl-no-deps make targets automatically start up the py (and thus ErlPort) Erlang Python server. If you run the REPL without these make targets, you'll need to manually start things:

$ lfetool repl lfe -s py

API Usage β†Ÿ

Below we show some basic usage of py from both LFE and Erlang. In a separate section a list of docs are linked showing detailed usage of wrapped libraries.

Metadata β†Ÿ

First things first: let's make sure that you have the appropriate versions of things -- in particular, let's confirm that you're running Python 3:

> (py-util:get-versions)
(#(erlang "17")
 #(emulator "6.2")
 #(driver-version "3.1")
 #(lfe "0.9.0")
 #(erlport "0.9.8")
 #(py "0.0.1")
 #(python
   ("3.4.2 (v3.4.2:ab2c023a9432, Oct  5 2014, 20:42:22)"
    "[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]")))

Module Level β†Ÿ

The following sub-sections describe module-level operations.

Calling Functions β†Ÿ

> (py:func 'os 'getcwd)
"/Users/yourname/lab/erlang/py"
> (py:func 'datetime.datetime 'now)
#("datetime" #(2014 12 23 16 57 11 693773 undefined))

Note that strings in arguments need to be converted to binary:

> (py:func 'os.path 'isfile '(#b("/tmp")))
false
> (py:func 'os.path 'isdir '(#b("/tmp")))
true

Keyword arguments are passed as proplists, e.g., '(#(key1 val1) #(key2 val2)). In the next example we'll pass a string (as a binary) which represents a binary number. We'll give int the keyword of base, since we're not going to use the default decimal base (10):

(py:func 'builtins 'int '(#b("101010")) '(#(base 2)))
42

Module Constants β†Ÿ

> (py:const 'math 'pi)
3.141592653589793

Optionally, you may provide a type:

> (py:const 'math 'pi 'float)
3.141592653589793
> (py:const 'math 'pi 'int)
3
> (py:const 'math 'pi 'str)
"3.141592653589793"

Objects β†Ÿ

The following sections describe how to work with Python objects.

Instantiation β†Ÿ

With no arguments passed to the constructor:

> (py:init 'builtins 'dict)
#("dict" ())
> (py:init 'collections 'UserDict)
#("UserDict" ())

With args:

> (py:init 'datetime 'date '(1923 4 2))
#("date" #(1923 4 1))

Calling Methods β†Ÿ

To call a method, we need an object. Let's return to the date example above:

> (set now (py:func 'datetime.datetime 'now))
#("datetime" #(2014 12 23 23 14 37 677463 undefined))

The tuple representing a date time object has been saved as the now variable in the REPL. Let's call some methods:

> (py:method now 'strftime '(#b("%Y.%m.%d %H:%M:%S")))
"2014.12.23 23:14:37"

Attribute Values β†Ÿ

Continuing with that same object:

> (py:attr now 'year)
2014
> (py:attr now 'microsecond)
677463

Operations on Objects β†Ÿ

Let's get another time ... and give our other variable a better name:

> (set later (py:func 'datetime.datetime 'now))
#("datetime" #(2014 12 23 23 21 25 714474 undefined))
> (set earlier now)
#("datetime" #(2014 12 23 23 14 37 677463 undefined))

Let's use the two objects in a calculation:

> (set diff (py:sub later earlier))
#("timedelta" 0 408 37011)
> (py:attr diff 'seconds)
408

We can get that in minutes in LFE/Erlang:

> (/ (py:attr diff 'seconds) 60)
6.8

Experimental Dotted Name Support β†Ÿ

There is currently tentative support for dotted names in the following calls:

  • (py:const ...)
  • (py:func ...)
  • (py:init ...)

Examples:

> (py:const 'math.pi)
3.141592653589793
> (py:func 'datetime.datetime.now)
#("datetime" #(2014 12 29 0 47 30 334180 undefined))
> (py:func 'math.pow '(2 16))
65536.0
> (py:func 'builtins.int '(#b("101010")) '(#(base 2)))
42
> (py:init 'collections.UserDict)
#("UserDict" ())
> (py:init 'collections.UserDict '() '(#("a" 1) #("b" 2)))
#("UserDict" (#("b" 2) #("a" 1)))

Though this is offered, it isn't really encouraged, since there will necessarily be inconsistencies in usage. Dotted notation can be used with const, func, and init but not with objects (i.e., not with method and attr). This mixing of styles could get confusing and you may think you have a bug in your code when, in fact, you just can't use dotted names with instantiated objects (since in LFE it's just a variable name, not an actual object).

Finally, with this code in place, many more function calls are incurred regardless of whether dotted notation is used. For this reason and the discouragement against use above, this feature is on the short list for getting axed. Don't count on it being around ...

ErlPort Pass-Through β†Ÿ

If for any reason you would like to skip the LFE py wrappers and call directly to ErlPort, you may do so:

> (py:pycall 'datetime 'datetime.now)
("datetime" #(2014 12 25 20 44 4 673150 undefined))
> (py:pycall 'datetime 'datetime '(1923 4 2 0 0 0))
#("datetime" #(1923 4 2 0 0 0 0 undefined))

These make direct calls to ErlPort's python:call function, but supply the required Python server pid behind the scenes.

Builtins β†Ÿ

In several of the examples above, we made calls to the builtins module like so:

> (py:init 'builtins 'dict)
#("dict" ())
> (py:func 'builtins 'int '(#b("101010")) '(#(base 2)))
42

LFE py actually provides wrappers for these, making such calls much easier.

> (py:dict)
#("dict" ())
> (py:dict '(#("a" 1) #("b" 2)))
#("dict" (#("b" 2) #("a" 1)))
> (py:int #b("101010") '(#(base 2)))
42

More examples:

> (py:any '(true true false false false true))
true
> (py:all '(true true false false false true))
false
> (py:all '(true true true))
true
> (py:pow 6 42)
481229803398374426442198455156736
> (py:round 0.666666667 5)
0.66667
> (py:range 7 42)
#($erlport.opaque python
  #B(128 2 99 95 95 98 117 105 108 ...))
> (py:len (py:range 7 42))
35
> (py:pylist (py:range 7 42))
(7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
 31 32 33 34 35 36 ...)
> (erlang:length (py:pylist (py:range 7 42)))
35

Operators β†Ÿ

It will often be the case that you want to operate on the Python data structures obtained via the LFE py calls directly in Python, without translating them into LFE/Erlang. The Python operator module is wrapped for your convenience in these cases.

Examples:

> (py:add 37 5)
42
> (py:mul 7 6)
42
> (py:sub -108 -150)
42
> (py:truediv 462 11)
42.0
> (py:floordiv 462 11)
42

Equality:

> (py:gt 7 6)
true
> (py:le 7 6)
false
> (py:eq 42 42)
true

Bitwise operations:

> (py:and- 60 13)
12
> (py:or- 60 13)
61
> (py:xor 60 13)
49
> (py:inv 60)
-61
> (py:rshift 60 2)
15
> (py:lshift 60 2)
240

Non-Python Additions β†Ÿ

So as not to stomp on the LFE function (list ...), the Python list builtin has been aliased to the pylist function, e.g.:

> (py:pylist)
()
> (py:pylist '(1 2 3 4))
(1 2 3 4)

(py:dir ...) and (py:vars ...) return elided lists, so you won't see complete results that are longer than 28 elements. If you wish to see everything, you may call (py:pdir) and (py:pvars), respectively.

(py:repr) provides wrapping for the Python builtin repr. If you would like to see a representation of the pickled Python data in LFE, you may use the (py:prepr) function.

Missing Functions β†Ÿ

Any Python function that does in-place modification of objects is not included. LFE py will eventually provide analogs for in-place functions that return a new data set or object.

Erlang β†Ÿ

We can, of course, do all of this from Erlang:

$ make shell-no-deps
1> 'py-util':'get-versions'().
[{erlang,"17"},
 {emulator,"6.2"},
 {'driver-version',"3.1"},
 {lfe,"0.9.0"},
 {'lfe-py',"0.0.1"},
 {python,["3.4.2 (v3.4.2:ab2c023a9432, Oct  5 2014, 20:42:22)",
          "[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]"]}]
2> py:func(os, getcwd).
"/Users/oubiwann/Dropbox/lab/erlang/py"
3> py:func('datetime.datetime', now).
{"datetime",{2014,12,25,23,16,14,979696,undefined}}
4> py:func('os.path', isfile, [<<"/tmp">>]).
false
5> py:func('os.path', isdir, [<<"/tmp">>]).
true
6> py:func(builtins, int, [<<"101010">>], [{base, 2}]).
42
7> py:const(math, pi).
3.141592653589793
8> py:const(math, pi, float).
3.141592653589793
9> py:const(math, pi, int).
3
10> py:const(math, pi, str).
"3.141592653589793"
11> py:init(builtins, dict).
{"dict",[]}
12> py:init(collections, 'UserDict').
{"UserDict",[]}
13> py:init(datetime, date, [1923, 4, 2]).
{"date",{1923,4,2}}
14> Now = py:func('datetime.datetime', now).
{"datetime",{2014,12,25,23,16,57,146812,undefined}}
15> py:method(Now, strftime, [<<"%Y.%m.d %H:%M:%S">>]).
"2014.12.d 23:16:57"
16> py:attr(Now, year).
2014
17> py:attr(Now, microsecond).
146812
18> Later = py:func('datetime.datetime', now).
{"datetime",{2014,12,25,23,19,51,934212,undefined}}
19> Earlier = Now.
{"datetime",{2014,12,25,23,16,57,146812,undefined}}
20> Diff = py:sub(Later, Earlier).
{"timedelta",{0,174,787400}}
21> py:attr(Diff, seconds).
174
22> py:attr(Diff, seconds) / 60.
2.9
23> py:pycall(datetime, 'datetime.now').
{"datetime",{2014,12,25,23,24,46,525495,undefined}}
24> py:pycall(datetime, datetime, [1923, 4, 2, 0, 0, 0]).
{"datetime",{1923,4,2,0,0,0,0,undefined}}
25> py:dict().
{"dict",[]}
26> py:dict([{"a", 1}, {"b", 2}]).
{"dict",[{"b",2},{"a",1}]}
27> py:int(<<"101010">>, [{base, 2}]).
42
28> py:any([true, true, false, false, false, true]).
true
29> py:all([true, true, false, false, false, true]).
false
30> py:all([true, true, true]).
true
31> py:pow(6, 42).
481229803398374426442198455156736
32> py:round(0.666666667, 5).
0.66667
33> py:range(7, 42).
{'$erlport.opaque',python,
                   <<128,2,99,95,95,98,117,105,108,...>>}
34> py:len(py:range(7, 42)).
35
35> py:pylist(py:range(7, 42)).
[7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,
 27,28,29,30,31,32,33,34,35|...]
36> length(py:pylist(py:range(7, 42))).
35
37> py:add(37, 5).
42
38> py:mul(7, 6).
42
39> py:sub(-108, -150).
42
40> py:truediv(462, 11).
42.0
41> py:floordiv(462, 11).
42
42> py:gt(7, 6).
true
43> py:le(7, 6).
false
44> py:eq(42, 42).
true
45> py:'and-'(60, 13).
12
46> py:'or-'(60, 13).
61
47> py:'xor'(60, 13).
49
48> py:inv(60).
-61
49> py:rshift(60, 2).
15
50> py:lshift(60, 2).
240
51> py:pylist().
[]
52> py:pylist([1, 2, 3, 4]).
[1,2,3,4]

Architecture β†Ÿ

Overview β†Ÿ

Here is a high-level diagram of the LFE py architecture:

+-------------------------------------------+
|                                           |
| +-----------+ +-----------+ +-----------+ |
| | py Worker | | py Worker | | py Worker | |
| +------+----+ +-----+-----+ +-----+-----+ |
|        |            |             |       |
|        |            |             |       |
|        |       +----+---+         |       |
|        |       |        |         |       |
|        +-------+ py+sup +---------+       |
|                |        |                 |
|                +----+---+                 |
|                     |                     |
|                +----+---+                 |
|                |        |                 |
|                | py+app |                 |
|                |        |                 |
|                +--------+                 |
|                                           |
|                  LFE py                   |
+---------------------+---------------------+
|        LFE          |       ErlPort       |
+---------------------+---------------------+
|                 Erlang/OTP                |
+-------------------------------------------+

Each py Worker is actually a wrapper for an ErlPort gen_server which starts up Python interpreter. LFE py is only designed to work with Python 3. Both ErlPort and it provide Python 3 modules for use in the interpreters started by the workers. They have the following conceptual structure:

+----------------+
|                |
|  +----------+  |
|  | Encoders |  |
|  +----------+  |
|  +----------+  |
|  | Decoders |  |
|  +----------+  |
|                |
|   LFE py lib   |
+----------------+
|   ErlPort lib  |
+----------------+
|    Python 3    |
+----------------+

As depicted above, when the LFE py/ErlPort Python server starts, it brings up a Python 3 interpreter. LFE py configures the PYTHONPATH for ErlPort so that the custom encoder, decoder, and object helper Python modules are available for use by all Python calls issued to the workers.

Erlang Components β†Ÿ

Working our way up from the diagram, here are references for Erlang/OTP components of LFE py:

Python Components β†Ÿ

And here are references for the Python components in LFE py:

Controlling the Python Servers β†Ÿ

Erlang Configuration β†Ÿ

Python Configuration β†Ÿ

Start, Stop, and Restart β†Ÿ

Dynamically Adding More Python Servers β†Ÿ

Automatic Restarts β†Ÿ

Python Server Schedulers β†Ÿ

Executing Code in Parallel β†Ÿ

Identical Calls β†Ÿ

Scatter/Gather β†Ÿ

Distributed Python β†Ÿ

Starting Remote Python Servers β†Ÿ

Executing Python on Remote Servers β†Ÿ

More Repositories

1

lsci

Scientific Computing on the Erlang VM - An LFE Wrapper Library for SciPy, NumPy, etc.
Erlang
69
star
2

ltest

A Testing Framework for LFE (successor to lfeunit)
LFE
35
star
3

lmud

An Erlang and LFE MUD/MUSH Server.
Erlang
19
star
4

lutil

LFE Utility Modules (successor to lfe-utils)
LFE
18
star
5

dragon

A Web Framework for LFE
JavaScript
16
star
6

logjam

A custom formatter for the Erlang logger application that produces human-readable output
Erlang
11
star
7

sdl2-examples

LFE SDL2 Examples using LoΓ―c's Erlang NIF
Erlang
10
star
8

nx

LFE support for the Elixir Nx machine learning library
LFE
9
star
9

complex

LFE support for numbers both real and imagined
LFE
7
star
10

lanes

A slightly more general HTTP routing library than LFE's lfest
LFE
6
star
11

barista

Barista serves up hot lmugs of LFE for your simple HTTP needs.
LFE
6
star
12

dockerfiles

Dockerfiles for LFE based upon the official Erlang Docker images
Makefile
6
star
13

poise

An LFE Library for the Framework-agnostic Generation of Static HTML Content
Erlang
5
star
14

moneta

An LFE wrapper for mnesia and qlc
HTML
4
star
15

bombadil

A wrapper for the Erlang toml library that uses the map data structure
LFE
4
star
16

lmatrix

A Simple Linear Algebra Library for LFE
Erlang
3
star
17

lcfg

The LFE library for managing and using lfe.config files
LFE
3
star
18

tcp-client

A simple LFE TCP client connection manager
LFE
2
star
19

horatio

The rational and most trusted friend of the tragic hero, Floating Point
Erlang
2
star
20

lgtk

An LFE GTK 2.0 API that wraps the Erlang gtknode library
Erlang
1
star
21

moneta-tutorial

The LFE Mnesia Tutorial Adapted for Moneta
Erlang
1
star
22

lfe-gtknode-example

An LFE Port of the Erlang GTK+ "top" Example
Erlang
1
star
23

plottah

An Erlang/LFE gnuplot process server
LFE
1
star
24

terrain-gen

A Terrain Generator written in LFE
Erlang
1
star