Data Analysis of Lagou Job

Introduction

This repository holds the code for job data analysis of Lagou. The main functions included are listed as follows:

1. Crawling job data from Lagou, and get the latest information of jobs about Internet.
2. Proxies are collected from XiCiDaiLi.
3. Data analysis and visualization.
4. Crawling job details info and generate word cloud as Job Impression.
5. In order to train a NLP task with machine learning, the data of interviewee's comments will be stored in mongodb

Prerequisites

1. Install 3rd party libraries

   sudo pip3 install -r requirements.txt

2. Install mongodb and start mongodb service [optional]

   sudo service mongod start

How to Use

1. clone this project from github.
2. Lagou's anti-spider strategy has been upgrade frequently recently. I suggest you run proxy_crawler.py to get IP proxies and execute the code with PhantomJS.
3. run m_lagou_spider.py to crawl job data, it will generate a collection of Excel files in ./data directory.
4. run hot_words_generator.py to cut sentences, it will return TOP-30 hot words and wordcloud figure.

Analysis Results

Report

• For technical details, please refer to my answer at Zhihu.
• The PDF report can be downloaded from here.

Change Log

• [V2.0] - 2019.04. Upgraded to PhantomJS and IP proxies.
• [V1.2] - 2017.05. Rewrite WordCloud visualization module.
• [V1.0] - 2017.04. Upgraded to mobile Lagou.
• [V0.8] - 2016.05. Finish Lagou PC web spider.

LICENSE

Apache-2.0