Edinburgh-Genome-Foundry/Flametree

Python file operations made easy

Flametree is a Python library which provides a simple syntax for handling files and folders (no os.path.join, os.listdir etc.), and works the same way for different file systems.

Write a Flametree program to read/write files in disk folders, and your code will also be able to read/write in zip archives and virtual (in-memory) archives - which is particularly useful on web servers.

As an illustration, here is how to use Flametree to read a file texts/poems/the_raven.txt, replace all occurences of the word "raven" by "seagull" in the text, and write the result to a new file the_seagull.txt in the same folder:

from flametree import file_tree

with file_tree("texts") as root:
    poem_text = root.poems.the_raven_txt.read()
    new_text = poem_text.replace("raven", "seagull")
    root.poems._file("the_seagull.txt").write(new_text)

Even in this very simple use case, the syntax is clearer than the os way, which would write as follows:

import os

with open(os.path.join("poems", "the_raven.txt"), "r") as f:
    poem_text = f.read()
new_text = poem_text.replace("raven", "seagull")
with open(os.path.join("poems", "the_seagull.txt"), "w") as f:
    content = f.write(new_text)

Moreover, the same Flametree code also works for files inside a zip archive:

with file_tree("my_archive.zip") as root:
    poem_text = root.poems.the_raven_txt.read()
    new_text = poem_text.replace("raven", "seagull")
    root.poems._file("the_seagull.txt").write(new_text)

Now in hard mode: suppose that your server receives binary zip data of an archive containing poems/the_raven.txt, and must return back a new zip containing a file poems/the_seagull.txt. Here again, the syntax of the core operations is the same:

destination_zip = file_tree("@memory") # Create a new virtual zip
with file_tree(the_raven_zip_data) as root:
    poem_text = root.poems.the_raven_txt.read()
    new_text = poem_text.replace("raven", "seagull")
    destination_zip._dir("poems")._file("the_seagull.txt").write(new_text)
destination_zip_data = destination_zip._close()
# Now send the data to the client

See section Usage below for more examples and features.

Installation

Flametree should work on Windows/Max/Linux, with Python 2 and 3, and has no external dependency.

It can be installed by unzipping the source code in one directory and using this command:

python setup.py install

You can also install it directly from the Python Package Index with this command:

pip install flametree

Contribute

Flametree is an open-source software originally written by Zulko and released on Github under the MIT licence (Copyright Edinburgh Genome Foundry). Everyone is welcome to contribute! In particular if you have ideas of new kinds of file systems to add to Flametree.

Usage

Opening a file tree

Here is how you open different kinds of file systems:

from flametree import file_tree

# Open a directory from the disk's file system:
root = file_tree("my_folder/")

# Open a zip archive on the disk:
root = file_tree("my_archive.zip")

# Connect to a file-like object (file handle, StringIO...) of a zip:
root = file_tree(file_like_object)

# Create a virtual 'in-memory' zip file:
root = file_tree("@memory")

# Open some data string representing a zip to read
root = file_tree(some_big_zip_data_string)

In the two first examples, if my_folder or my_archive.zip do not exist, they will be automatically created. If they do exist, it is possible to completely overwrite them with the option replace=True.

Exploring a file tree:

Once you have created the root element with one of the methods above, you can display the whole file tree with root._tree_view() :

>>> print (root._tree_view())
texts/
  poems/
    dover_beach.txt
    the_raven.txt
    the_tyger.txt
  todo_list.txt
figures/
  figure1.png
  figure2.png
Readme.md

The attributes of a directory like root are its files and subdirectories. For instance to print the content of dover_beach.txt you would write:

print( root.texts.poems.dover_beach_txt.read() )

or even simpler:

root.texts.poems.dover_beach_txt.print_content()

Notice that the . before txt was replaced by _ so as to form a valid

attribute name.

This syntactic sugar is particularly useful to explore a file tree in IPython Notebooks or other editors offering auto-completion:

Alternatively, you can access files and directories using dictionary calls:

root["texts"]["poems"]["dover_beach.txt"]

To iterate through the subdirectories of a directory, use the _dirs attribute:

for subdirectory in root._dirs:
    print (subdirectory._name) # Will print 'texts' and 'figures'

To iterate through the files of a directory, use the _files attribute:

for f in root.figures._files:
    print (f._name) # Will print 'figure1.png' and 'figure2.png'

Finally, use _all_files to iterate through all files nested in a directory. The snippet below prints the content of all .txt files in the file tree:

for f in root._all_files:
    if f._name.endswith(".txt"):
        f.print_content()

Creating files and folders

To create a new subdirectory use _dir:

root._dir("data") # create a 'data' folder at the root
root.data._dir("reports") # create a 'reports' folder under `root/data`

To create a new file use _file:

root._file("joke.txt") # create a 'joke.txt' file at the root.
root.texts._file("hello.txt") # create 'hello.txt' in `root/texts`.

To write content in a file, use .write:

root.joke_txt.write("A plateau is the highest form of flattery.")

Writing to a file will use mode a (append) by default. To overwrite the file set the write mode to "w". Let's erase and rewrite that joke.txt:

root.joke_txt.write("'DNA' stands for National Dyslexic Association.", "w")

File and directory creation commands can be chained. Let us create some new folders data/ and data/test_1/, and write to file data/test_1/values.csv, all in a single line:

root._dir("data")._dir("test_1")._file("values.csv").write("1,15,25")

Beware that ._dir and ._file overwrite their target by default, which means that if you write:

root._dir("data")._file("values_1.csv").write("1,4,7")
root._dir("data")._file("values_2.csv").write("2,9,7")

The directory data will only contain values_2.csv, because the second line's _dir("data") erases the data directory and starts a new one. To avoid this, either use root.data in the second line:

root._dir("data")._file("values_1.csv").write("1,4,7")
root.data._file("values_2.csv").write("2,9,7")

Or use replace=False in _dir:

root._dir("data")._file("values_1.csv").write("1,4,7")
root._dir("data", replace=False)._file("values_2.csv").write("2,9,7")

Other operations

You can move, copy, and delete a file with .move(folder), .copy(folder), .delete(), and a directory with ._move(folder), ._copy(folder), ._delete().

root.data.values1_csv.delete() # delete file 'values1.csv'
root.data._delete() # delete directory 'data'
# Move folder `plots` from `root/figures` to `other_root/figures`
root.figures.plots._move(other_root.figures)
# Move file `fig.png` from `root/figures` to `other_root/figures`
root.figures.fig_png.move(other_root.figures)

Special rules for ZIP archives

It is not currently possible to modify/delete a file that is already zipped into an archive (because zips are not really made for that, it would be doable but would certainly be a hack).

When creating files and folders in a zip with Flametree, the changes in the actual zip will only be performed by closing the root with root._close() (after which the root can't be used any more). If it is an in-memory zip, root._close() returns the value of the zip content as a string (Python 2) or bytes (Python 3).

Here are a few examples:

root = file_tree("archive.zip")
root._file("hello.txt").write("Hi there !")
root._close()

# Equivalent to the previous, using `with`:
with file_tree("archive.zip") as root:
    root._file("hello.txt").write("Hi there !")

# Getting binary data of an in-memory zip file:
root = file_tree("@memory")
root._file("hello.txt").write("Hi there !")
binary_data = root._close()

Using file writers from other libraries

Some libraries have file-generating methods which expect a file name or a file object to write too. You can also feed Flametree files to these functions. for instance here is how to use Weasyprint to create a PDF pdfs/report.pdf

import weasyprint
from flametree import file_tree
root = file_tree(".") # or 'archive.zip' to write in an archive.
html = weasyprint.HTML(string="<b>Hello</b> world!", base_url='.')
html.write_pdf(root._dir("pdfs")._file("test.pdf"))

And here is how you would save a Matplotlib figure in a zip archive:

import matplotlib.pyplot as plt
from flametree import file_tree
fig, ax = plt.subplots(1)
ax.plot([1, 2, 3], [3, 1, 2])
with file_tree("archive.zip") as root:
    fig.savefig(root._dir("plots")._file("figure.png"), format="png")

That's all folks !