• Stars
    star
    267
  • Rank 151,440 (Top 4 %)
  • Language
    Python
  • Created over 11 years ago
  • Updated 20 days ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Fill HTML login forms automatically

loginform

https://secure.travis-ci.org/scrapy/loginform.png?branch=master Coverage report

loginform is a library for filling HTML login forms given the login url, username & password. Which form and fields to fill are inferred automatically.

It's implemented on top of lxml form filling, and thus depends on lxml.

Usage

Usage is very simple and best illustrated with an example:

>>> from loginform import fill_login_form
>>> import requests
>>> url = "https://github.com/login"
>>> r = requests.get(url)
>>> fill_login_form(url, r.text, "john", "secret")
([('authenticity_token', 'FQgPiKd1waDL+pycPH8IGutirTnP69SiZgm0zXwn+VQ='),
  ('login', 'john'),
  ('password', 'secret')],
 u'https://github.com/session',
 'POST')

And it is possible to use it as a tool to quickly debug a login form:

$ python -m loginform -u john -p secret https://github.com/login
url: https://github.com/session
method: POST
payload:
- authenticity_token: FQgPiKd1waDL+pycPH8IGutirTnP69SiZgm0zXwn+VQ=
- login: john
- password: secret

Testing

A collection of real-world samples is used to keep this library tested. Those samples are managed as follows:

First, you select a site to try, find out its login url, and run the following command to try loginform on it:

$ python test_samples.py https://github.com/login
[
   "https://github.com/login",
   [
      [
         [
            "authenticity_token",
            "NsdVWGpzxKmn7zSJSOdgnDcLIzIdJlCTO754LiEv2W4="
         ],
         [
            "login",
            "USER"
         ],
         [
            "password",
            "PASS"
         ]
      ],
      "https://github.com/session",
      "POST"
   ]
]

From the output you can judge if it worked or not. If it worked, great. If it didn't, you would hack loginform.py to make it work and then add the sample with:

$ python test_samples.py https://github.com/login -w github

Note that we gave the sample a name (github in this case).

To list all available samples use:

$ python test_samples.py -l

To run all tests, install tox and run:

$ tox

More Repositories

1

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.
Python
51,036
star
2

scrapyd

A service daemon to run Scrapy spiders
Python
2,857
star
3

scrapely

A pure-python HTML screen-scraping library
HTML
1,855
star
4

dirbot

Scrapy project to scrape public web directories (educational) [DEPRECATED]
Python
1,630
star
5

quotesbot

This is a sample Scrapy project for educational purposes
Python
1,275
star
6

parsel

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
Python
1,088
star
7

scrapyd-client

Command line client for Scrapyd server
Python
755
star
8

w3lib

Python library of web-related functions
Python
382
star
9

cssselect

CSS Selectors for Python
Python
284
star
10

queuelib

Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python
Python
261
star
11

slybot

224
star
12

scrapy.org

The scrapy.org website
HTML
60
star
13

itemadapter

Common interface for data container classes
Python
60
star
14

protego

A pure-Python robots.txt parser with support for modern conventions.
DIGITAL Command Language
51
star
15

itemloaders

Library to populate items using XPath and CSS with a convenient API
Python
43
star
16

scrapy-bench

A CLI for benchmarking Scrapy.
Python
30
star
17

scurl

Performance-focused replacement for Python urllib
Python
21
star
18

pypydispatcher

A fork of http://pydispatcher.sourceforge.net/ with PyPy support
Python
15
star
19

xtractmime

https://mimesniff.spec.whatwg.org/ implementation for Python
Python
13
star
20

base-chromium

base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
C++
7
star
21

scrapy-itemloader

[Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
Python
6
star
22

gsoc2014-integration-tests

GSoC2014 - Scrapy Integration tests project
Shell
3
star
23

url-chromium

url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url
C++
2
star