pydbattach - attach to running Python process

It probably should have been called pydbgattach or so but I didn't noticed until I went to bed so I keep this name now. :)

This works in several steps:

1. We are attaching to the running Python process and want to inject some C-code pyinjectcode.c.

   Originally I planned to develop a small tool for this by myself based on ptrace. For Mac, I also found mach_inject which may have been useful.

   However, to keep things simple for now, I just use GDB for this.

   See the file pydbattach.sh which basically does this step.

2. pyinjectcode.c creates a new Python thread and runs a Python script. In our case, it runs pyinjectcode.py.

3. pyinjectcode.py starts a Pdb instance and attaches it to another already running Python process (just the first it founds for now). This is done via a more generic sys.settrace function which is implemented in pythreadhacks.py.

4. pythreadhacks.py heavily uses (_)ctypes to access the underlying CPython objects for the thread state. This way, it reimplements PyEval_SetTrace in a more general way.

Similar alternatives, to attach to running CPython process without having it prepared beforehand. Those usually use the OS debugging capabilities (or use gdb/lldb) to attach to the native CPython process and then inject some code or just analyze the native CPython thread stacks.

• Google pyringe, last update 2014
• pyrasite, last update 2017
• py-spy, sampling only
• Pyflame, sampling only

There are other alternatives where you prepare your Python script beforehand to listen on some (tcp/file) socket to provide an interface for remote debugging and/or just a Python shell / REPL.

• IPython, by embedding an IPython kernel: background_zmq_ipython package
• winpdb (cross platform) remote debugger
• PyCharm IDE remote debugger, doc
• PyDev IDE remote debugger
• Twisted Conch Manhole, official example, lothar.com example, lysator.liu.se example, related StackOverflow question, blog.futurefoundries.com (2013)
• very simple manhole, has also some overview over related projects
• ispyd
• Eric IDE
• Trepan (based on pydb)
• rpdb
• rconsole (part of rfoo)
• pystuck
• pdb-clone

Some overviews and collected code examples:

• (QGIS) Example code for PyDev, Winpdb, Eric
• Python Wiki: Python debugging tools, Python Wiki: Python debuggers
• StackOverflow question about Python remote shells.

-- Albert Zeyer, http://www.az2000.de