Top Rating
- Top Contributors
  Discover the Top Open Source contributors by country or by language
- Interviews
  Discover real stories from Open Source developers
Discover

Discover your Favorite Language
Discover the top trending repositories and projects on Github. Explore the latest trends in your preferred languages.

Emacs Lisp

Rust

HTML

JavaScript

Nix

Elixir

C#

TypeScript

More Languages
Awesome

Awesome repositories
Discover the most awesome repositories and projects of your favorite languages. Inspired by the Awesome-* lists trend in GitHub.

Lua

Ruby

Shell

C

Erlang

PowerShell

R

Elm

More Languages
By Country

Rankings by Country
Discover the community of talented open source contributors in each country.

🇱🇻 Latvia

🇪🇸 Spain

🇦🇺 Australia

🇳🇬 Nigeria

🇮🇪 Ireland

🇸🇩 Sudan

🇧🇯 Benin

🇳🇱 Netherlands

All Countries Compare Countries

codingforentrepreneurs/Web-Scraping

Stars
113
Rank 310,115 (Top 7 %)
Language
Python
License
MIT License
Created over 6 years ago
Updated almost 6 years ago

codingforentrepreneurs/Web-Scraping

codingforentrepreneurs

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Learn how to leverage Python's amazing tools to scrape data from other websites. The end goal of this course is to scrape blogs to analyze trending keywords and phrases. We'll be using Python 3.6, Requests, BeautifulSoup, Asyncio, Pandas, Numpy, and more!

Learn how to leverage Python's amazing tools to scrape data from other websites.

The end goal of this course is to scrape blogs to analyze trending keywords and phrases.

We'll be using Python 3.6, Requests, BeautifulSoup, Asyncio, Pandas, Numpy, and more!

Section 1: Your First Scraping Program

Watch here

Final code is first-web-scraping-program.zip

Install Guides

Windows: https://kirr.co/6r8wr9

Mac: https://kirr.co/386c7f

Linux: https://kirr.co/c3uvuu

Goals of Your First Scraping Program:

Enter any url (webpage)
Open and scrape that webpage's words each word
Save that info into a csv

Third party Packages

Python Requests : http://docs.python-requests.org/en/master/
```
pip install requests
```
Basically, it opens the webpage for us in this one.
BeautifulSoup 4 : https://www.crummy.com/software/BeautifulSoup/bs4/doc/
```
pip install beautifulsoup4
```
This allows us to search & extract content from an HTML webpage

Section 2: Advancing Scraping

Goals of Advancing Scraping:

Refine scraping code
Scrape Links
Add Scrape Depth
Scrape & Parse words in a Post

2 - Get URL Input

3 - Regular Expression Validation

4 - Force Quit Program

8 - Extract Data

9 - Parse Links

10 - Get Local Paths

11 - Local Paths by Regular Expression

12 - Some Lookup Errors

13 - Scrape Local Paths

14 - Parse Words

15 - Python Set

16 - A Recursive Function

17 - Mock Fetching

18 - All together

Section 3: Asyncio & Web Scraping

code coming soon

30-Days-of-Python

Learn Python for the next 30 (or so) Days.

eCommerce

We're going to take you step-by-step to build a modern, fully open-source, eCommerce web application using Python, Django, Bootstrap, Javascript, and more.

Try-Django

Learn Django bit by bit in this series

Guides

Guides for learning + doing better web and app development. Created by Coding for Entrepreneurs.

Tweetme-2

Build a twitter-like app in Django, Bootstrap, Javascript, & React.js. Step-by-Step.

OpenCV-Python-Series

An OpenCV & Python Tutorial Series and Reference.

Try-Django-1.11

Learn the fundamentals behind one of the most popular web frameworks in the world: Django. We will teach you step-by-step how to implement concepts like Views, Template Rendering, Forms, Saving Data, URL routing, Deployment aka Going Live, and so much more. Django is a web-framework written in Python and runs the backend for many of the internet's most popular websites such as Instagram and Pinterest. Get started today!

Try-Django-2.2

Try Django 2.2 is step-by-step to build a modern, fully open-source, Blog web application using Python, Django, Bootstrap, Javascript, and more.

Django-Rest-Framework-Tutorial

Learn how to build REST APIs with Django & the Django Rest Framework.

Blog-API-with-Django-Rest-Framework

Blog API with Django Rest Framework

Try-Django-3.2

Learn the fundamentals behind one of the most popular web frameworks in the world by building a real project.

open-ecommerce

Coding for Entrepreneurs Tutorial for building an eCommerce web application with Django & Bootstrap (djangoproject.com, getbootstrap.com). Topics include Python, HTML, CSS, JQuery, AJAX.

try-django-19

Try Django 1.9 is an introduction to Django version 1.9 by creating a simple, yet robust, Django blog. This series covers a variety of Django basics as well as Django 1.9 specific material. Created by Team CFE @ http://joincfe.com.

REST-API

Learn how to build your own REST API with Python, Django, and the Django Rest Framework.

Reactify-Django

Integrate React & Django

30-Days-of-Python-3.6

This is a soon-to-be archived project version of 30 Days of Python. The original tutorial still works but we have an updated version in the works right now.

REST-API-Basics

This is a basic guide on how to build a REST API with Django & Python. For much deeper depth, check out our new course on REST API: (https://kirr.co/90kxtx)

ecommerce-2

eCommerce 2 is a step-by-step guide to creating your own custom eCommerce site using Django (v. 1.8), Bootstrap (v. 3.3), AJAX, jQuery (v. 1.11), Braintree Payments and more.

Try-Django-1.8

Try Django 1.8 is an introduction to Django version 1.8. This series covers a variety of Django basics as well as Django 1.8 specific material

Django-Kubernetes

Learn how to deploy a docker-based Django application into a Kubernetes cluster into production on DigitalOcean.

SaaS-Foundations

Django tutorial integrating TailwindCSS, Neon Postgres, Stripe API, User auth, deployment, & more.

Advancing-the-Blog

Building on top of Try Django 1.9 (https://github.com/codingforentrepreneurs/try-django-19), we're going to build a more advanced blog and learn about all kinds of topics as we go.

Angular-Django

Integrate Try Angular v4 to a Django backend using the Django Rest Framework and Django version 1.11.

Rapid-ChatXChannels

A rapid fire tutorial and introduction of Django Channels. To get more in depth check out the full course https://kirr.co/badl8e

video-membership

Create a Video Membership app using FastAPI & NoSQL

Try-Django-1.10

Try Django 1.10 is an introduction to Django version 1.10 by creating a URL shortening service. This series covers a variety of Django basics as well as Django 1.10 specific material. Created by Team CFE @ http://joincfe.com.

OpenCV-REST-API

Learn to create a REST API microservice for extracting faces from images using OpenCV, OpenCV-python, Flask, Docker, and Heroku

Tweetme

Build a Twitter-like web app step-by-step with Django, jQuery, and Bootstrap!

ecommerce-2-api

eCommerce 2 API is a step-by-step guide to implementing an RESTful API into your eCommerce 2 project (https://github.com/codingforentrepreneurs/ecommerce-2) using the Django Rest Framework, Django Rest Framework JWT, Django CORS Headers, and more.

Try-Reactjs

The new version is here: https://github.com/codingforentrepreneurs/Try-React.js

digital-marketplace

A Digital Marketplace where users can buy & sell digital goods like High Resolution Images, MP3s, Audiobooks, and eBooks. Learn how to build this project on https://codingforentrepreneurs.com/projects/digital-marketplace/

Direct-Messages-in-Django

A tutorial on how to create synchronous slack-inspired message channels and private messages using Django. Enjoy!

Scrape-Websites-with-Python-FastAPI-Celery-NoSQL

Learn how to scrape websites with Python, Selenium, Requests HTML, Celery, FastAPI, & NoSQL with Cassandra via AstraDB.

Jupyter Notebook

FastAPI-Microservice-for-Django

Deploy a microservice REST API endpoint for our Try Django 3.2 https://cfe.sh/projects/try-django-3-2 project using FastAPI, pytesseract, Docker, & DigitalOcean App Platform.

DjangoFlix

Create a netflix-like service using Django. Learn advanced Django techniques to achieve amazing results like never before.

Django-to-iOS

Django-Angular-Ionic

Leveraging a REST API built with Django, we'll build an Angular Web App. With the same API, we'll build a mobile app using the Ionic Framework.

Django-Chart.js

Learn how to integrate Chart.js with Django

Pi-Awesome

References and guides to using & setting up a raspberry pi for your projects.

matchmaker-2

Learn how to connect people based off common answers to questionnaires and provide suggested positions, locations, and employers.

AI-as-an-API

Learn to create & deploy a deep learning algorithm into a production REST API microservice using Python, Keras, FastAPI, & NoSQL.

Jupyter Notebook

python-desktop-app

Learn to build a Python Desktop GUI app using pywebview, Python, JavaScript, HTML, & CSS.

micro-ecommerce

Learn how to build a Micro eCommerce Web App with Python, Serverless Postgres, and Stripe Checkout. Live demo at url below.

srvup-membership

Srvup is a step-by-step tutorial to teach you how to create a video membership site much like CodingForEntrepreneurs.com.

The-Hello-World-of-Machine-Learning

Learn to build a basic machine learning model from scratch with this repo and tutorial series.

Jupyter Notebook

launch-with-code

Launch with Code

Django-Bootcamp-1

This bootcamp was hosted in October 2020.

Django-JS-Frontend-Integration

This tutorial shows you the basics of how to integrate Angular, React, Vue, or any other JavaScript front end you'd like.

Django-Likes

Learn how to create an AJAX-powered like button in Django.

Supercharged-Web-Scraping-with-Asyncio

Scrape websites asynchronously with Python 3.8+, Asyncio, & arsenic (aka Selenium for Async).

Jupyter Notebook

Try-Angular-v4

Try Angular v4 is an introduction to Angular version 4 by creating an application that highlights some of the powerful features behind the innovative client-side framework.

Raspberry-Pi-Awesome

Updated repo linked in readme. The goal of this repo is to have simple, easy-to-use scripts to get a Raspberry Pi ready for all kinds of awesome (Face Recognition, Open CV, Dlib, Django, and more).

recommender

Build a recommendation engine using Django & a Machine Learning technique called Collaborative Filtering.

ChatXChannels

Learn to Build Real Time Chat with Django Channels

Flask-Landing

Learn to build a landing page with the very popular Python Flask Microframework.

MVP-Landing

Learn to build & deploy a MVP landing page using Django, Python, & Bootstrap. The tutorial series covers the Django basics using Django 3, Python 3.8, and Bootstrap 4.5.

Jupyter-x-Docker-on-Heroku

Build a Python Jupyter Notebook Server with Docker & Heroku.

Time-Tasks-2

Turn your python projects into a task master and scheduling genie with the magic of Celery & Redis. This is Time & Tasks 2.

Jupyter Notebook

Hello-Linux

This tutorial series is all about deploying a Python web application on Ubuntu 18.04 LTS.

full-stack-python

Learn how to build web apps in pure Python using Reflex.

Serverless-Django

Learn to build and deploy a Serverless Django application to Google Cloud Run with Cloud SQL, Cloud Build, & Docker.

Django-on-Docker

Use Django with Docker. Related code for the post--->

jref.io

A tutorial on creating a url shortening service with Next.js, Vercel, and Neon.

ai-travel-agent

Learn to create an AI Travel Agent with FastAPI, Next.js, MariaDB, MindsDB, OpenAI, and more!

Jupyter Notebook

Try-Django-2.0

A Tutorial on building a real project with Django version 2.0

30-Days-of-Swift

30 Days of Swift is a step-by-step tutorial to teach you the Swift programming language by Apple and how to build iOS Apps.

deploy-fastapi-from-scratch

Deploy FastAPI from Scratch to a production server with this series.

Playing-with-Asyncio

This series shows you the basics of how to use the Asyncio Library in Python.

Autogenerate-Django-Models

Learn how to take a scraped dataset and turn it into Django models using inspectdb. You can also use this method for legacy databases.

Jupyter Notebook

Docking-Django

Django x Docker to Production on Heroku

Serverless-Container-Based-Python-App-on-Google-Cloud-Run

Serverless enables you to focus on code, not infrastructure. Deploy a Docker Container to Cloud Run using this series. Cloud Run is a service by the Google Cloud Platform.

iac-python

Boilerplate FastAPI Python including a Dockerfile, docker-compose, and nginx config.

srvup-rest-framework

Extending the Srvup Membership project, we build an API using the Django Rest Framework.

Django-AngularJS

Django + AngularJS :: In this step-by-step series we bring Django and AngularJS together by leveraging the Django Rest Framework's RESTFUL API service

Notebooks

Various notebooks we're working on from either our blog cfe.sh/blog or as general research.

Jupyter Notebook

Python-for-Text-Classification

Python for Text Classification with Machine Learning in Python 3.6.

Jupyter Notebook

Django-on-Docker-with-Heroku-and-OpenCV

Deploy Django on Docker to Heroku and include OpenCV

django-nextjs-backend-api

The Django Ninja API backend for the Django x Next.js course.

AI-as-an-API-Course-Reference

This repository is the exact code used in the course. For the complete & most up to date project go to https://github.com/codingforentrepreneurs/AI-as-an-API

Jupyter Notebook

Elastic-Django

Integrate Elastic Search on a Django project using elasticsearch-dsl elasticsearch-py and django-elasticsearch-dsl

AJAXify-Django-Forms

Convert your Django Forms into Asynchronous forms using our guide here: https://kirr.co/r1tsdb

Dive-into-AWS-Course---Direct-to-S3-via-Django-JavaScript

In this section, we'll be implementing our Reusable Boto3 Utility (https://cfe.sh/courses/aws/reusable-boto3-utility) to implement secure direct to S3 uploads as well as secure downloads via Django & JavaScript.

30-Days-of-JavaScript

For this tutorial, I'll have an accompanying eBook along with videos (coming soon) for each day.

cfe-django-blog

Boilerplate Django Blog for production deployments!

Dive-Into-AWS

Links to the Repos and Sections in our Dive into AWS Course.

Python-with-Google-Colab-Sheets-Drive

Learn how to use Python to load, adjust, and change data from both Google Sheets and Google Drive using Google Colab.

Jupyter Notebook

Django-AllAuth-Tutorial

Email, Username, and Social Authentication made easier.

try-zappa

Create a Serverless Web Application using Zappa, AWS Lambda, and Django.

Class-Based-Views

Class Based Views (CBVs) are an important aspect of making Django work better for you. This tutorial series helps you understand how to use class based views in a pre-existing Django project.

getting-started-with-typescript

Learn how to setup npm & Typescript with basics of Typescript.

Single-Page-Django-App

Learn how to create a single page (module) Django app with this quick tutorial.

Geolocator-2

Learn how to find and work with locations in Django, the Yelp API, and Google Maps api.

Try-Pandas

In this series, we're going to learn the fundamentals of the popular Python data science tool called Pandas.

Jupyter Notebook

fastapi-airtable

Learn how to integrate a minimal FastAPI project with Airtable as our data store.

Idea-Landing

A rapid-fire building of a dynamic landing page to introduce the series -- HiitStartup \\ https://kirr.co/kb0y88. Use this project to launch your idea today. Watch here: https://www.codingforentrepreneurs.com/projects/idea-landing/

DjangoGap

Coding for Entrepreneurs Tutorial on building a Django + PhoneGap application using Django Project (http://djangoproject.com), PhoneGap (http://phonegap.com/developer/) & Jquery Mobile (http://jquerymobile.com). Topics include Python, HTML, PhoneGap, CSS, JQuery, AJAX, and more.

Capture-Emails-with-Django-and-React.js

Learn to create a reusable email capture component by integrating Django, the Django Rest Framework, and React.js.

Django-User-Model-Unleashed

Learn about the Django user model in this reference series. Created using Django 1.10

CFE-Blank-Project

A blank Django Starter Project that includes Docker support.

Bootstrap-Basics-v4

A introductory tutorial on how to use Bootstrap for our front-end design via docs on getbootstrap.com