Segment Any Medical-Model (SAMM): A 3D Slicer integration to Meta's SAM.

paper

Laboratory of Biomechanical and Image Guided Surgical Systems, Johns Hopkins University

Yihao Liu, Jeremy Zhang, Zhangcong She

Known issues

#13 Resolved with a not-so-elegant way. Tested work on MRHead, DZ-MR, MRBrainTumor2 and CTChest data from Slicer. Please report a bug if your data is not working. Note for the first few seconds when you start "Mask Sync", the server is not so stable, wait a few seconds then slide it up and down, the mask then will be updated. Note now you can only work on the RED view. Will update later to support all 3 views.

#17 4GB Nvidia GeForce RTX 3050Ti VRAM issue. On testing a larger test machine, the VRam usage is at around 3GB. Might have issues with smaller machines.

Switch out the model to smaller ones if you have VRAM limit.

Demo

https://www.youtube.com/watch?v=vZK45noZVIA

Motivation

Accurate image segmentation is crucial for medical image analysis as it enables clinicians to extract meaningful information from the image. It also allows for the detection of subtle changes in the tissue or organ of interest, which is essential for monitoring disease progression and treatment response.

Our project, Segment Any Medical-Model aims to develop an integration for Segment Anything Model (SAM) and 3D Slicer for future development and validation of the potentials of transferring Large Language Model to the medical image analysis field.

Installation and How-To-Use

TLDR version

Works both on Linux and Windows. Has Mac support, but not tested yet.

This assumes Cuda, cv2 and pytorch are in your environment.

git clone [email protected]:bingogome/samm.git
conda create --name samm
conda activate samm
pip install git+https://github.com/facebookresearch/segment-anything.git
pip install opencv-python pycocotools matplotlib onnxruntime onnx

If you are using Windows, it's okay if you don't install pycocotools.

Start 3D Slicer, in the Python Console:

slicer.util.pip_install("pyyaml")
slicer.util.pip_install("pyzmq")

SD Slicer -> Developer Tools → Extension Wizard.

Extension Tools -> Select Extension -> import the samm/samm folder.

Back to terminal, cd to samm (upper level)

Run ./samm-python-terminal/sam_server.py

If it throws an error missing "sam_vit_h_4b8939.pth", move segment-anything/notebooks/sam_vit_h_4b8939.pth to samm/samm-python-terminal/samm-workspace

Follow the demo and Segment Any Medical Model away!

Install samm

Install this repo:

git clone [email protected]:bingogome/samm.git

Create Virtual Environment

It's essential to have a clean virtual environment to avoid any potential conflicts. Therefore, you'd better to create a new environment for running the rest part of the code.

Install any version of anaconda to manage the virtual environment. Anaconda installation guide can be found here.

Create virtual environment and activate it:

conda create --name samm
conda activate samm

Note: The given python script in this folder has to be executed in samm venv.

Install SAM

The code requires python>=3.8, as well as pytorch>=1.7 and torchvision>=0.8. Please follow the instructions here to install both PyTorch and TorchVision dependencies. Installing both PyTorch and TorchVision with CUDA support is strongly recommended.

Install Segment Anything:

pip install git+https://github.com/facebookresearch/segment-anything.git

or clone the repository locally and install with

git clone [email protected]:facebookresearch/segment-anything.git
cd segment-anything; pip install -e .

Then,

pip install opencv-python pycocotools matplotlib onnxruntime onnx

Install 3D Slicer

Follow this page to download a compatible version of 3D Slicer and install it in your local environment.

Install the SAMM Extension to 3D Slicer

The source code of the extension is contained in samm.

In the GUI of 3D Slicer, expand the extension drop-down menu, and choose Developer Tools → Extension Wizard.

Then on the left side of the GUI, click the toggle bar named Extension Tools and click `Select Extension' button. It will prompt a navigation window where you can find, select and import the samm folder.

Citation

If you use SAMM in your research, please consider use the following BibTeX entry.

@article{liu2023samm,
  title={SAMM (Segment Any Medical Model): A 3D Slicer Integration to SAM},
  author={Liu, Yihao and Zhang, Jiaming and She, Zhangcong and Kheradmand, Amir and Armand, Mehran},
  journal={arXiv preprint arXiv:2304.05622},
  year={2023}
}