• Stars
    star
    164
  • Rank 230,032 (Top 5 %)
  • Language
    Python
  • License
    MIT License
  • Created almost 3 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A tool that embosses the needed dependencies on the top level executable

Shrinkwrap

main built with nix

A tool that embosses the needed dependencies on the top level executable

Introduction

It can be useful to freeze all the dynamic shared objects needed by an application.

shrinkwrap is a tool which will discover all transitive dynamic shared objects, and lift them up to the executable referenced by absolute path.

Here is an example where we will apply this to ruby.

Lets take a look at all the dynamic shared objects needed by the Ruby interpreter.

ldd $(which ruby)
	linux-vdso.so.1 (0x00007ffeed386000)
	/lib/x86_64-linux-gnu/libnss_cache.so.2 (0x00007f638ddf8000)
	libruby-2.7.so.2.7 => /lib/x86_64-linux-gnu/libruby-2.7.so.2.7 (0x00007f638da79000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f638d8b4000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f638d893000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f638d888000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f638d807000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f638d7ff000)
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f638d7c4000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f638d67f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f638de06000)

We can see also that the ruby application only lists a few needed shared objects itself.

patchelf --print-needed $(which ruby)
libruby-2.7.so.2.7
libc.so.6

Let's now apply shrinkwrap and see the results.

nix run github:fzakaria/shrinkwrap $(which ruby)

It automatically creates a _stamped copy of the filename if none provided and sets all the NEEDED sections.

patchelf --print-needed ruby_stamped
/lib/x86_64-linux-gnu/libm.so.6
/lib/x86_64-linux-gnu/libcrypt.so.1
/lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libgmp.so.10
/lib/x86_64-linux-gnu/librt.so.1
/lib/x86_64-linux-gnu/libpthread.so.0
/lib/x86_64-linux-gnu/libruby-2.7.so.2.7
/lib/x86_64-linux-gnu/libc.so.6ldd ruby_stamped
	linux-vdso.so.1 (0x00007ffe641f3000)
	/lib/x86_64-linux-gnu/libnss_cache.so.2 (0x00007f9cd4320000)
	/lib/x86_64-linux-gnu/libm.so.6 (0x00007f9cd41db000)
	/lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f9cd41a0000)
	/lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9cd419a000)
	/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f9cd4119000)
	/lib/x86_64-linux-gnu/librt.so.1 (0x00007f9cd410e000)
	/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9cd40eb000)
	/lib/x86_64-linux-gnu/libruby-2.7.so.2.7 (0x00007f9cd3d8c000)
	/lib/x86_64-linux-gnu/libc.so.6 (0x00007f9cd3bc7000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9cd4336000)

Motivation

Certain store based build tools such as Guix, Nix or Spack make heavy use of RUNPATH to help create reproducible and hermetic binaries.

One problem with the heavy use of RUNPATH, is that the search space could effect startup as it's O(n) on the number of entries (potentially worse if using RPATH). This can alo be expensive in stat syscalls, that has been well documented by in this blog post.

Secondly, shared dynamic objects may be found due to the fact that they are cached during the linking stage. Meaning, if another shared object requires the same dependency but failed to specify where to find it, it may still properly resolved if discovered earlier in the linking process. This is extremely error prone and changing any of the executable's dependencies can change the link order and potentially cause the binary to no longer work.

Lifting up the needed shared objects to the top executable makes the dependency discovery simple, quick and hermetic since it can no longer change based on the order of visited dependencies.

Pitfalls

At the moment this only works with glibc and not other Standard C Libraries such as musl. The reason is that other linkers seem to resolve duplicate shared object files differently when they appear in the traversal. Consider the following example:

              +------------+
              |            |
              | Executable |
              |            |
      +-------+------------+----+
      |                         |
      |                         |
+-----v-----+            +------v----+
|           |            |           |
| libbar.so |            | libfoo.so |
|           |            |           |
+-----+-----+            +-----------+
      |               /some-fixed-path/libfoo.so
      |
+-----v------+
|            |
| libfoo.so  |
|            |
+------------+

In glibc the cache is keyed by the soname value on the shared object. That allows the first found libfoo.so at /some-fixed-path/libfoo.so to be used for the one which libbar.so depends on.

Unfortunately, musl does not support this functionality and ongoing discussions of inclusing it can be followed on the mailing list.

Development

You must have Nix installed for development.

This package uses poetry2nix to easily setup a development environment.

> nix develop

A helping Makefile is provided to run all the linters and formatters.

> make lint

Note: I publish artifacts to cachix that you can use to develop faster.

> cachix use fzakaria

Experiments

Included in the flake are different experiments for evaluating Shrinkwrap. In most cases they provide a Docker image (tar.gz) which can be loaded.

emacs

Creates a stamped version of the popular emacs editor similarly to the Guix experiment outlined in the blog post.

You can build the Docker image and inside will be emacs-wrapped as well as emacs and strace to recreate the experiment.

> nix build .#experiments.emacs
> docker load < result
643ace721190: Loading layer [==================================================>]  786.9MB/786.9MB
Loaded image: shrinkwrap-emacs-experiment:7jjlknqq660x1crrw7gm4m2qzalp71qj
> docker run -it emacs-experiment:7jjlknqq660x1crrw7gm4m2qzalp71qj /bin/bash
> patchelf --print-needed /bin/emacs-stamped
/nix/store/m756011mkf1i0ki78i8y6ac3gf8qphvi-gcc-10.3.0-lib/lib/libstdc++.so.6
/nix/store/xif6gg595hgmqawrcarapa8j2r7fbz9w-icu4c-70.1/lib/libicudata.so.70
/nix/store/i6cmh2d4hbyp00rnh5rpf48pc7xfzx6j-libgpg-error-1.42/lib/libgpg-error.so.0
/nix/store/q39ykk5fnhlbnl119iqjbgaw44kd65fy-util-linux-2.37.2-lib/lib/libblkid.so.1
/nix/store/b1k5z0fdj0pnfz89k440al7ww4a263bf-libglvnd-1.3.4/lib/libGLX.so.0

If you'd like you can pull the image directly from DockerHub via fmzakari/shrinkwrap-emacs-experiment:7jjlknqq660x1crrw7gm4m2qzalp71qj.

> docker pull fmzakari/shrinkwrap-emacs-experiment:7jjlknqq660x1crrw7gm4m2qzalp71qj
> docker run -it fmzakari/shrinkwrap-emacs-experiment:7jjlknqq660x1crrw7gm4m2qzalp71qj /bin/bash
> patchelf --print-needed /bin/emacs-stamped
/nix/store/m756011mkf1i0ki78i8y6ac3gf8qphvi-gcc-10.3.0-lib/lib/libstdc++.so.6
/nix/store/xif6gg595hgmqawrcarapa8j2r7fbz9w-icu4c-70.1/lib/libicudata.so.70
/nix/store/i6cmh2d4hbyp00rnh5rpf48pc7xfzx6j-libgpg-error-1.42/lib/libgpg-error.so.0
/nix/store/q39ykk5fnhlbnl119iqjbgaw44kd65fy-util-linux-2.37.2-lib/lib/libblkid.so.1
/nix/store/b1k5z0fdj0pnfz89k440al7ww4a263bf-libglvnd-1.3.4/lib/libGLX.so.0

Contributions

Thanks to @trws for the inspiration and original version of this Python script.

More Repositories

1

HypeMachine-Extension

This is a Google Chrome Extension that injects a download button next to the songs.
JavaScript
168
star
2

sqlelf

Explore ELF objects through the power of SQL
Python
108
star
3

slf4j-timbre

SLF4J binding for Clojure's Timbre
Clojure
94
star
4

mvn2nix

Easily package your Maven Java application with the Nix package manager.
Java
92
star
5

transcoding

A basic transcoding Golang server that utilizes FFMPEG over the command line
Go
68
star
6

HypeScript

Python HypeMachine script downloader
Python
56
star
7

ebpf-mpls-encap-decap

Sample project demonstrating how to use eBPF to encap/decap packets with an MPLS label.
C
46
star
8

nix-harden-needed

Bubble up the correct paths to your shared object libraries in Nix
Nix
40
star
9

autopatchelf

Go
30
star
10

space-saving

Space Saving algorithm implementation (StreamSummary) in Java, used to solve heavy hitters / topk items.
Java
28
star
11

WaterFlow

WaterFlow is a non-magical / easy to understand / JDK8 framework for use with SWF
Java
20
star
12

nix-home

A nix home development environment
Shell
19
star
13

nix-http-binary-cache-api-spec

An OpenAPI specification for a Nix HTTP Binary Cache
HTML
17
star
14

ascii85

A Java library for working with Ascii85, also called Base85 - a form of binary-to-text encoding
Java
16
star
15

Akka-Camel-SQS

This is a sample project that ties Apache-Camel, Akka & SQS together.
Scala
13
star
16

calcite-git

A native Calcite Git adapter
Java
8
star
17

http4j

http4j is an HTTP toolkit written in Java (in the vein of http4k) that allows for functional HTTP services
Java
6
star
18

HypemFinderSite

A website to help people download songs from Hypem.com!
Python
6
star
19

nixos-maven-example

An example of how to use buildMaven with Nix to build a Maven project
Nix
6
star
20

simple-annotations-plugin

A simple Grafana annotations plugin that works with any datasource.
TypeScript
6
star
21

fzakaria.com

My personal blog & website
HTML
5
star
22

addressme

This is a new service where you can find addresses across the world to send mail to.
Scala
5
star
23

chisel-base64

A Chisel module that includes a base64 codec
Scala
4
star
24

old

The Other Link Editor
C++
4
star
25

HypeMachine-Java

Java GUI downloader for HypeMachine
Java
3
star
26

CSE231-Breaking-Shared-Object-Monolith

The course project for UCSC CSE231 to attempt to break the shared code monolith.
C
3
star
27

guide-dedupe

Sample code showing how to properly dedupe modules and bindings in Guice
Java
2
star
28

lametun

This is a very minimalistic demonstration of how to setup a VPN-like tunnel.
Go
2
star
29

dotfiles

My dotfiles (or rather aggregation of others!)
Perl
2
star
30

iHype

iPhone HypeMachine Application using Three20 UI
Objective-C
2
star
31

beatprice

Attempts to find cheaper alternatives to tracks on electronic dance music stores
JavaScript
2
star
32

Huffman-Compression

Basic Huffman Encoding
C++
2
star
33

elf2sql

Convert ELF files to SQLite databases
Shell
2
star
34

ChocoPy

An implementation of ChocoPy using ANTLR4
Java
2
star
35

circularfifoqueue

CircularFifoQueue is a first-in first-out queue with a fixed size that replaces its oldest element if full
Go
1
star
36

donothing

LD_PRELOAD override to have your program do nothing.
C
1
star
37

dx

An exchange, is a highly organized market where commodities are sold and bought. Exchanges bring together brokers and dealers who buy and sell these objects.
Python
1
star
38

BeerService

A sample pants + finatra + thrift codebase
Shell
1
star
39

RayTracer

Raytracer for CS488 (http://www.student.cs.uwaterloo.ca/~cs488/)
C++
1
star
40

lookml-parsec

A LookML parser in Haskell using Megaparsec
Haskell
1
star
41

rust-ebpf-demo

A simple pure hello world demo of writing an ebpf filter in rust
Rust
1
star
42

ninja-crater

A web service that helps users ideas for others they'd like to see
Java
1
star
43

liblist

Provide a simple list of the dynamic libraries found and print as a new line list using the dynamic linker.
C++
1
star