Python3WebSpider/DouYin Python3WebSpider/DouYin

  • Stars
    star
    642
  • Rank 70,096 (Top 2 %)
  • Language
    Python
  • License
    MIT License
  • Created about 6 years ago
  • Updated almost 5 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
Python3WebSpider/DouYin
github

Python3WebSpider

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

API of DouYin for Humans used to Crawl Popular Videos and Musics

ๆณจ๏ผš้กน็›ฎๅทฒไธๅฏ็”จ...

DouYin

API of DouYin App for humans.

Installation

pip3 install douyin

Usage

Here is the sample code:

import douyin
from douyin.structures import Topic, Music

# define file handler and specify folder
video_file_handler = douyin.handlers.VideoFileHandler(folder='./videos')
music_file_handler = douyin.handlers.MusicFileHandler(folder='./musics')
# define mongodb handler
mongo_handler = douyin.handlers.MongoHandler()
# define downloader
downloader = douyin.downloaders.VideoDownloader([mongo_handler, video_file_handler, music_file_handler])

for result in douyin.hot.trend():
    for item in result.data:
        # download videos of topic/music for 100 max per
        downloader.download(item.videos(max=100))

then you can get:

Item <Topic: <1565818716518401, panama>>
Processing <Video: <6616517521098935565, ็œŸ้ฆ™#panama>> ...
Processing <Video: <6500385230921141518, ๅ“ˆๅ“ˆๅ“ˆๅ“ˆๅ“ˆ>> ...
...
Processing <Video: <6479958542747962637, ๐Ÿ‘…เธเนˆเธญเธ™เธเธดเธ™เธ‚เน‰>> ...
Processing <Video: <6473811426107460878, ๐Ÿ˜>> ...
0%|                                                      | 0/10 [00:00<?, ?it/s]
Processing 1-10 of files
Processing <Video: <6616517521098935565, ็œŸ้ฆ™#panama>> ...
Saving <Video: <6616517521098935565, ็œŸ้ฆ™#panama>> to mongodb...
Processing <Video: <6500385230921141518, ๅ“ˆๅ“ˆๅ“ˆๅ“ˆๅ“ˆ>> ...
Saving <Video: <6500385230921141518, ๅ“ˆๅ“ˆๅ“ˆๅ“ˆๅ“ˆ>> to mongodb...
Processing <Video: <6562690160868199693, ็š‡ๅŸŽ็›ธๅบœ็‰ˆCๅ“ฉCๅ“ฉ่ทจ>> ...
....
Downloading <Video: <6580510322468064526, ็ฌฌไบŒ้ ๅŽ้ข็š„ๅฐๅ“ฅๅ“ฅ>> ...
Saved <Video: <6479958542747962637, ๐Ÿ‘…เธเนˆเธญเธ™เธเธดเธ™เธ‚เน‰>> to mongodb successfully
Downloading <Video: <6479958542747962637, ๐Ÿ‘…เธเนˆเธญเธ™เธเธดเธ™เธ‚เน‰>> ...
Saved <Video: <6473811426107460878, ๐Ÿ˜>> to mongodb successfully
Downloading <Video: <6473811426107460878, ๐Ÿ˜>> ...
Downloaded file to ./videos/6580510322468064526.mp4
10%|โ–ˆโ–ˆโ–ˆโ–ˆโ–Œ                                         | 1/10 [00:01<00:16,  1.84s/it]
Downloaded file to ./videos/6516746291806997763.mp4
20%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–                                    | 2/10 [00:01<00:10,  1.33s/it]
Downloaded file to ./videos/6600742831352974596.mp4
40%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–                           | 4/10 [00:02<00:05,  1.03it/s]
Downloaded file to ./videos/6484393014599879950.mp4
50%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ                       | 5/10 [00:02<00:04,  1.15it/s]
Downloaded file to ./videos/6616517521098935565.mp4
60%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–Œ                  | 6/10 [00:03<00:03,  1.27it/s]
Downloaded file to ./videos/6479958542747962637.mp4
70%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–             | 7/10 [00:03<00:01,  1.68it/s]
Downloaded file to ./videos/6472305134377372941.mp4
80%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–Š         | 8/10 [00:03<00:00,  2.05it/s]
Downloaded file to ./videos/6562690160868199693.mp4
90%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–    | 9/10 [00:04<00:00,  2.27it/s]
Downloaded file to ./videos/6500385230921141518.mp4
100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 10/10 [00:04<00:00,  2.33it/s]

Examples

See examples

More Repositories

1

ProxyPool

An Efficient ProxyPool with Getter, Tester and Server
Python
5,329
star
2

Python3WebSpider

Source File of My Book related to WebSpider
1,994
star
3

CookiesPool

Cookies Pool
Python
540
star
4

AdslProxy

Adsl Proxy Pool
Python
234
star
5

TaobaoProduct

Taobao Product Spider by Selenium
Python
191
star
6

ScrapyRedisBloomFilter

Scrapy Redis Bloom Filter
Python
173
star
7

DeepLearningSlideCaptcha2

Deep LearningImage Captcha 2
Python
161
star
8

CrackGeetest

Crack Geetest
Python
155
star
9

Weibo

Weibo Spider Using Scrapy
Python
136
star
10

MaoYan

MaoYan TOP 100 Movie
Python
128
star
11

Jiepai

Jiepai Pictures of Toutiao
Python
123
star
12

Weixin

Sougou Weixin Spider Using Proxy
Python
86
star
13

RecaptchaResolver

Recaptcha Resolver
Python
84
star
14

ScrapySeleniumTest

Scrapy Selenium on Taobao Product
Python
84
star
15

DeepLearningSlideCaptcha

Deep Learning Slide Captcha
Python
74
star
16

WeiboList

WeiboList of MaYun
Python
65
star
17

GithubLogin

Login Github with Requests
Python
64
star
18

Moments

Wechat Friends Circle Spider
Python
56
star
19

CrackWeiboSlide

Crack Weibo Slide Captcha
Python
55
star
20

ScrapyUniversal

Scrapy Universal Spider
Python
54
star
21

ScrapyTutorial

Scrapy Tutorial
Julia
49
star
22

ProxyTunnel

Proxy Tunnel using Squid LoadBalancer
Shell
48
star
23

ProxySettings

Proxy Settings
Python
43
star
24

Images360

Download Images From 360 Using Scrapy
Python
42
star
25

PyppeteerTest

Pyppeteer Demo
Python
41
star
26

HCaptchaResolver

HCaptcha Resolver
Python
38
star
27

ScrapeStatic1

Spider for https://static1.scrape.cuiqingcai.com/
Python
34
star
28

AccountPool

Account Pool
Python
33
star
29

WeiboCrawler

Weibo Crawler for All Sites
Python
32
star
30

MitmAppiumJD

MitmProxy and Appium to Crawl Comments in JD APP
Python
31
star
31

ScrapySplashTest

Scrapy Splash on Taobao Product
Python
30
star
32

AjaxHookSpider

Ajax Hook Demo
JavaScript
29
star
33

PlaywrightTest

Playwright Test
Python
28
star
34

CrackTouClick

Crack Touch Click
Python
27
star
35

CrackImageCaptcha

Crack Image Code by Tesserocr
Python
24
star
36

ScrapeSsr1

Scrape Ssr1
Python
24
star
37

ScrapyPyppeteerDeprecated

Scrapy Pyppeteer Demo
HTML
23
star
38

ElasticSearchTest

Elastic Search Code
Python
22
star
39

DeepLearningImageCaptcha

DeepLearning Image Captcha
Python
22
star
40

CrackSlideCaptcha

Crack Slide Captcha
Python
16
star
41

ScrapyCrawlSpider

Scrapy Crawl Spider
Python
16
star
42

JavaScriptObfuscate

Demo of JavaScript Obfuscate
JavaScript
16
star
43

ProxyTest

Proxy Demo
Python
15
star
44

UrllibTest

Urllib Demo
Python
11
star
45

ScrapyPyppeteer

Scrapy Pyppeteer Demo
Python
11
star
46

CodeServer

Code Server
Dockerfile
11
star
47

Scrape

Platform of Web Views to Scrape
Python
10
star
48

SeleniumTest

Selenium Demo
Python
10
star
49

ScrapeSpa5

Spider for https://spa5.scrape.center/
Python
9
star
50

IGetGet

IGetGet Books Spider by MitmDump
Python
9
star
51

ScrapydDocker

Docker for Scrapyd
Dockerfile
8
star
52

BookCrawler

Book Crawler
Python
8
star
53

Installation

Instructions for Installation of Software or Packages
8
star
54

AsyncTest

Async HTTP Demo
Python
8
star
55

Deobfuscate

Deobfuscate Samples
JavaScript
7
star
56

RequestsTest

Requests Test
Python
6
star
57

ScrapyDocker

Docker for Scrapy Demo
Python
6
star
58

Universal

Universal Scrapy
Python
6
star
59

ScrapeDynamic1

Spider for Dynamic1
Python
6
star
60

CaptchaPlatform

Captcha Platform Demo
Python
6
star
61

MovieCrawler

Movie Crawler
Python
6
star
62

ScrapeCaptcha3

Spider for https://captcha3.scrape.cuiqingcai.com/
Python
5
star
63

ScrapyCompositeDemo

Scrapy Composite Demo
Python
5
star
64

MySQLTest

MySQL Test Demo
Python
5
star
65

TestTess

Test Tesseract and Tesserocr
Python
5
star
66

Qunar

Qunar Spider
Python
5
star
67

ScrapyDownloaderTest

Test Downloader Middleware of Scrapy
Python
5
star
68

FileStorageTest

File Storage Test
Python
4
star
69

ScrapeSpa2

Spider for https://spa2.scrape.center
Python
4
star
70

XposedTest

Xposed Demo
Java
4
star
71

ScrapyCrawlSpiderTest

Scrapy Universal Spider
Python
4
star
72

ScrapeAntispider5

Scrape Antispier5
Python
4
star
73

WebDriverDetection

Detect WebDriver and Pretend Browser
JavaScript
4
star
74

BeautifulSoupTest

Beautiful Soup Test
Python
4
star
75

ScrapeLogin2

Spider for https://login2.scrape.cuiqingcai.com/
Python
3
star
76

ScrapySpiderMiddlewareDemo

Scrapy Spider Middleware Demo
Python
3
star
77

MultiprocessingTest

Demo of Multiprocessing
Python
3
star
78

ScrapeLogin3

Spider for https://login3.scrape.cuiqingcai.com/
Python
3
star
79

PyQueryTest

PyQuery Demo
Python
3
star
80

XPathTest

XPath Test
Python
3
star
81

UnidbgServer

Unidbg Server
Java
3
star
82

RegexTest

Regex Demo
Python
3
star
83

AppiumTest

Appium Test
Python
3
star
84

ScrapeSpa1

Scrape Spa1
Python
3
star
85

MitmProxyTest

Mitm Proxy Test
Python
2
star
86

PlaywrightIntercepting

Playwright Intercepting
JavaScript
2
star
87

NewsCrawler

News Crawler
Python
2
star
88

BrowserMobProxyTest

Browser Mob Proxy Test
Python
2
star
89

ScrapeSpa6

Spider for https://spa6.scrape.center
Python
2
star
90

ScrapeAntispider3

Scrape Antispider3
Python
2
star
91

RabbitMQTest

RabbitMQ Test
Python
2
star
92

LearnAST

AST Demo
JavaScript
2
star
93

ParselTest

Parsel Test Demo
Python
2
star
94

ScrapySeleniumDemo

Scrapy Selenium Demo
Python
2
star
95

ScrapydDeploy

Scrapyd Deploy Template for Azure
2
star
96

ScrapeSpa14

Scrape Spa14
Python
1
star
97

ScrapeSpa7

Scrape Spa7
JavaScript
1
star
98

HttpxTest

Scrape Spa16
Python
1
star
99

ScrapeApp9

Scrape App9
Python
1
star
100

ScrapeAntispider6

Scrape Antispider6
Python
1
star