This is a Python implementation of crawling Weibo data (e.g., text, images, live photos, and videos) of a Sina Weibo user from the Weibo Mobile Client. It simulates user login by a session (username and password).
Many thanks to Python Chinese Community for providing the source code SourceCode_weibocrawler.py
.
-
Crawling short text in original and retweeted Weibo posts.
-
Crawling large (preferred) or small JPG/GIF images in original and retweeted Weibo posts.
-
Crawling live photos (as JPG images, MOV videos, and/or GIF images) in original and retweeted Weibo posts.
-
Crawling HD (preferred) or SD videos in original and retweeted Weibo posts.
This code has been tested on Ubuntu 16.04 operating system. For Windows and macOS, the format of folder path should be changed accordingly.
- requests 2.21.0
- lxml 4.2.5
- cv2 4.1.0
- imageio 2.4.1
- PIL 5.3.0
-
Set
S_DATA
andS_HEADER
of the session for simulating Sina Weibo user login (see comments for details). -
Set
USER_URL
of the target Weibo user (see comments for details). -
Set the amount of pages (
PAGE_AMOUNT
) for crawling (see comments for details). -
Set the path (
PATH_FOLDER
) and the TXT file (PATH_FILE_TXT
) for saving Weibo data. -
Set the type of Weibo data (
IF_IMAGE
,IF_PHOTO
, andIF_VIDEO
as1
) for crawling. -
Set
IF_LIVE2GIF = True
if live photos (videos) need to be converted to GIF images. -
Set
TIME_DELAY
of the crawler to avoidConnectionError 104: ('Connection aborted.')
. -
If
ConnectionError 104: ('Connection aborted.')
occurs:-
Set
IF_RECONNECT = True
for running the crawler in reconnection mode. -
Set
TAG_STARTCARD
as the serial number of the stopping Weibo post (according to log information). -
Re-run
run_WeiboCrawler.py
to continue to crawl Weibo data. -
Set
IF_RECONNECT = False
if run the crawler in normal mode!
-
-
Run
run_WeiboCrawler.py
to crawl Weibo data of the target Sina Weibo user. -
See
Log_run_WeiboCrawler.txt
for log information of running the code.
-
The Weibo data will be saved in the pre-specified folder (e.g.,
Demo_WeiboData/
). -
The text of Weibo posts will be saved in the TXT file (e.g.,
Demo_WeiboData/Demo_WeiboPost_Records.txt
). -
The images, live photos, and videos will be saved in sub-folders (e.g.,
Demo_WeiboData/1/
,Demo_WeiboData/1_livephoto/
, andDemo_WeiboData/1_video/
).
Please report an issue if you have any question about this repository, I will respond ASAP.
Please star this repository if you found its content useful. Thank you very much. ^_^