mev-inspect-py
Maximal extractable value inspector for Ethereum, to illuminate the dark forest
Given a block, mev-inspect finds:
- miner payments (gas + coinbase)
- tokens transfers and profit
- swaps and arbitrages
- ...and more
Data is stored in Postgres for analysis.
Install
mev-inspect-py is built to run on kubernetes locally and in production.
Dependencies
Set up
Create a new cluster with:
kind create cluster
Set an environment variable RPC_URL to an RPC for fetching blocks.
mev-inspect-py currently requires a node with support for Erigon traces and receipts (not geth yet
pokt.network's "Ethereum Mainnet Archival with trace calls" is a good hosted option.
Example:
export RPC_URL="http://111.111.111.111:8546"
Next, start all services with:
tilt up
Press "space" to see a browser of the services starting up.
On first startup, you'll need to apply database migrations with:
./mev exec alembic upgrade head
And load prices data
./mev prices fetch-all
Usage
Inspect a single block
Inspecting block 12914944:
./mev inspect 12914944
Inspect many blocks
Inspecting blocks 12914944 to 12914954:
./mev inspect-many 12914944 12914954
Inspect all incoming blocks
Start a block listener with:
./mev listener start
By default, it will pick up wherever you left off. If running for the first time, listener starts at the latest block.
Tail logs for the listener with:
./mev listener tail
And stop the listener with:
./mev listener stop
Backfilling
For larger backfills, you can inspect many blocks in parallel
To inspect blocks 12914944 to 12915044, run
./mev backfill 12914944 12915044
This queues the blocks in Redis to be pulled off by the mev-inspect-worker service
To increase or decrease parallelism, update the replicaCount value for the mev-inspect-workers helm chart
Locally, this can be done by editing Tiltfile and changing "replicaCount=1" to your desired parallelism:
k8s_yaml(helm(
'./k8s/mev-inspect-workers',
name='mev-inspect-workers',
set=["replicaCount=1"],
))
You can see worker pods spin up then complete by watching the status of all pods
watch kubectl get pods
To see progress and failed batches, connect to Redis with
./mev redis
For total messages, query:
HLEN dramatiq:default.msgs
For messages failed and waiting to retry in the delay queue (DQ), query:
HGETALL dramatiq:default.DQ.msgs
For messages permanently failed in the dead letter queue (XQ), query:
HGETALL dramatiq:default.XQ.msgs
To clear the queue, delete keys for the main queue and delay queue
DEL dramatiq:default.msgs
DEL dramatiq:default.DQ.msgs
For more information on queues, see the spec shared by dramatiq
Backfilling a list of blocks
Create a file containing a block per row, for example blocks.txt containing:
12500000
12500001
12500002
Then queue the blocks with
cat blocks.txt | ./mev block-list
To watch the logs for a given worker pod, take its pod name using the above, then run:
kubectl logs -f pod/mev-inspect-worker-abcdefg
(where mev-inspect-worker-abcdefg is your actual pod name)
Exploring
All inspect output data is stored in Postgres.
To connect to the local Postgres database for querying, launch a client container with:
./mev db
When you see the prompt:
mev_inspect=#
You're ready to query!
Try finding the total number of swaps decoded with UniswapV3Pool:
SELECT COUNT(*) FROM swaps WHERE abi_name='UniswapV3Pool';
or top 10 arbs by gross profit that took profit in WETH:
SELECT *
FROM arbitrages
WHERE profit_token_address = '0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2'
ORDER BY profit_amount DESC
LIMIT 10;
Postgres tip: Enter \x to enter "Explanded display" mode which looks nicer for results with many columns.
FAQ
How do I delete / reset my local postgres data?
Stop the system if running:
tilt down
Delete it with:
kubectl delete pvc data-postgresql-postgresql-0
Start back up again:
tilt up
And rerun migrations to create the tables again:
./mev exec alembic upgrade head
I was using the docker-compose setup and want to switch to kube, now what?
Re-add the old docker-compose.yml file to your mev-inspect-py directory.
A copy can be found here
Tear down docker-compose resources:
docker compose down
Then go through the steps in the current README for kube setup.
Error from server (AlreadyExists): pods "postgres-client" already exists
This means the postgres client container didn't shut down correctly.
Delete this one with:
kubectl delete pod/postgres-client
Then start it back up again.
Maintainers
Contributing
Flashbots is a research and development collective working on mitigating the negative externalities of decentralized economies. We contribute with the larger free software community to illuminate the dark forest.
You are welcome here <3.
- If you want to join us, come and say hi in our Discord chat.
- If you have a question, feedback or a bug report for this project, please open a new Issue.
- If you would like to contribute with code, check the CONTRIBUTING file.
- We just ask you to be nice.
Security
If you find a security vulnerability on this project or any other initiative related to Flashbots, please let us know sending an email to [email protected].
Made with