• Stars
    star
    673
  • Rank 67,060 (Top 2 %)
  • Language
    Python
  • License
    GNU General Publi...
  • Created about 5 years ago
  • Updated 8 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

⛏️ Extract accounts info from personal pages on various sites for OSINT purpose

socid_extractor

Extract information about a user from profile webpages / API responses and save it in machine-readable format.

Usage

As a command-line tool:

$ socid_extractor --url https://www.deviantart.com/muse1908
country: France
created_at: 2005-06-16 18:17:41
gender: female
username: Muse1908
website: www.patreon.com/musemercier
links: ['https://www.facebook.com/musemercier', 'https://www.instagram.com/muse.mercier/', 'https://www.patreon.com/musemercier']
tagline: Nothing worth having is easy...

Without installing:

$ ./run.py --url https://www.deviantart.com/muse1908

As a Python library:

>>> import socid_extractor, requests
>>> r = requests.get('https://www.patreon.com/annetlovart')
>>> socid_extractor.extract(r.text)
{'patreon_id': '33913189', 'patreon_username': 'annetlovart', 'fullname': 'Annet Lovart', 'links': "['https://www.facebook.com/322598031832479', 'https://www.instagram.com/annet_lovart', 'https://twitter.com/annet_lovart', 'https://youtube.com/channel/UClDg4ntlOW_1j73zqSJxHHQ']"}

Installation

$ pip3 install socid-extractor

The latest development version can be installed directly from GitHub:

$ pip3 install -U git+https://github.com/soxoj/socid_extractor.git

Sites and methods

More than 100 methods for different sites and platforms are supported!

  • Google (all documents pages, maps contributions), cookies required
  • Yandex (disk, albums, znatoki, music, realty, collections), cookies required to prevent captcha blocks
  • Mail.ru (my.mail.ru user mainpage, photo, video, games, communities)
  • Facebook (user & group pages)
  • VK.com (user page)
  • OK.ru (user page)
  • Instagram
  • Reddit
  • Medium
  • Flickr
  • Tumblr
  • TikTok
  • GitHub

...and many others.

You can also check tests file for data examples, schemes file to expore all the methods.

When it may be useful

  • Getting all available info by the username or/and account UID. Examples: Week in OSINT, OSINTCurious
  • Users tracking, checking that the account was previously known (by ID) even if all public info has changed. Examples: Aware Online
  • Searching by commonly used cross-service UIDs (GAIA ID, Facebook UID, Yandex Public ID, etc.)
    • DB leaks of forums and platforms in SQL format
    • Indexed links that contain target profile ID
  • Searching for tracking data by comparison with other IDs - how it works, how can it be used.
  • Law enforcement online requests

Tools using socid_extractor

Maigret - powerful namechecker, generate a report with all available info from accounts found.

TheScrapper - scrape emails, phone numbers and social media accounts from a website.

YaSeeker - tool to gather all available information about Yandex account by login/email.

Marple - scrape search engines results for a given username.

Testing

python3 -m pytest tests/test_e2e.py -n 10  -k 'not cookies' -m 'not github_failed and not rate_limited'

Contributing

Check separate page if you want to add a new methods of fix anything.

More Repositories

1

maigret

🕵️‍♂️ Collect a dossier on a person by username from thousands of sites
Python
10,061
star
2

osint-namecheckers-list

A list of tools to search accounts by username
433
star
3

counter-osint-guide-ru

Исчерпывающее руководство по приватности и контр-ОСИНТ для Рунета и всего СНГ 🇷🇺
358
star
4

counter-osint-guide-en

Comprehensive Counter OSINT and privacy guide (initially for CIS countries)
275
star
5

marple

📖 Collect links to profiles by username through search engines and analyze with various plugins
Python
209
star
6

telegram-bot-dumper

🔪 Dumper & ripper for Telegram bots by token
Python
162
star
7

username-generation-guide

A definitive guide to generating usernames for OSINT purposes
Python
144
star
8

gitcolombo

🧬 Extract and analyze contributors info from git repos
Python
71
star
9

maigret-tg-bot

Maigret Telegram bot
Python
70
star
10

maigret-maltego

Maltego transformation for searching of accounts by username
Python
65
star
11

osint-cli-tool-skeleton

Template for new OSINT command-line tools
Python
60
star
12

mailto_analyzer

PoC analyzer of your email address exposure on various services
Python
52
star
13

files-search-guide

Guide to searching in different file types (documents, breaches, databases, etc.)
45
star
14

FRAVIA

FRAVIA: The Art of Searching
35
star
15

bellingcat-hackathon-watchcats

📊 Adana - 1-click analytical dashboard for OSINT researchers
Jupyter Notebook
33
star
16

lingolens

Search in Google Lens in lingo!
HTML
27
star
17

castreaper

Reaper of screencast videos for OSINT purposes
Python
21
star
18

information-disclosure-writeups-and-pocs

The list of write-ups, articles, and PoC of various vulnerabilites suitable for OSINT
PHP
17
star
19

maigret-adapter

Connect Maigret with other tools
Python
16
star
20

soxoj

14
star
21

netlas-email-search

Search emails with Netlas.io
Python
14
star
22

maltego-transformation-template

A template for standard Maltego transformation
Python
13
star
23

1c-database-converter

Converts binary files of 1C (1CD, cf, epf, efd, etc.) to grepable CSV
Python
12
star
24

email-archive-analysis

Processing PST/EML/MBOX mail archives for Investigative Research
11
star
25

everyday-search-guide-ru

Инструкции по бытовому каждодневному поиску
10
star
26

osint-cases-ru

Решения различных OSINT-задач
7
star
27

scraping_protection_analysis

Some utils for determination sites protection mechanisms
Python
7
star
28

soxoj.github.io

Soxoj's homepage
CSS
6
star
29

tiff2pdf-guide

Convert multi-page TIFF files into PDF documents efficiently and maintaining high image quality.
Python
5
star
30

disinfo-laundromat-maltego

Maltego transforms for The Disinformation Laundromat
Python
3
star
31

slowhttpdos-docker-images

DC7494 Meetup #4
Dockerfile
3
star
32

test-osint-tool

Python
2
star
33

4p-method

The 4P method: exploratory framework description and materials
2
star
34

nginx-3rd-party-login

Demo of transparent login with some platform through other platforms you're logged in.
HTML
2
star
35

osint-talks-list

Доклады и подкасты OSINT mindset
2
star
36

osint-mindset-org-ru

Главная страница сайта OSINT mindset
2
star
37

goaccess-proxy

Dockerfile
1
star
38

test-pdf

1
star
39

scanners

scanners for metascan.ru platform
Python
1
star
40

interpol-notices-maltego

Actual version on the link below
Python
1
star
41

osint-forum-bot-docs

1
star
42

dc7495-mod-bot

Python
1
star