electron-microscope
Use electron to load websites and extract data. Intended for automation, testing, web scraping, etc.
Loads URLs inside an electron webview tag, allows you to execute code on them and stream data from the pages back to your main process.
Run this headlessly on Linux using xvfb-run
.
Please note this is intended to be a fairly low level library that tries to not add much on top of what Electron is doing under the hood, so things that you might think are simple to do can turn out to be relatively complex due to the way web browser events end up working.
usage
Use this in an electron app:
var electron = require('electron')
var createMicroscope = require('electron-microscope')
electron.app.on('ready', function () {
createMicroscope(function (err, scope) {
if (err) throw err
// use your new microscope
})
})
Run it with electron:
$ npm install electron-prebuilt -g
$ electron my-code.js
examples
See the test/
and examples/
folders
API
require('electron-microscope')(options, ready)
Requiring the module returns a constructor function that you use to create a new instance. Pass it an options
object and a ready
callback that will be called with (error, scope)
. scope
is your new instance all ready to go.
scope.window
The electon BrowserWindow instance, AKA the renderer, which contains the <webview>
that pages are loaded in.
Currently because there are three node processes at play (main, renderer, webview), to access webview
APIs you have to go through the window
, e.g.:
scope.window.webContents.executeJavaScript("document.querySelector('webview').goBack()")
scope.loadURL(url, cb)
Load a url
, and call cb
with (err)
when loading is done. If there was a problem loading the page err
will be the error, otherwise it means it loaded successfully
var outputStream = scope.run(code)
Run code
on the currently loaded page. Run this after calling loadURL
. Code must be a string, if it is a function
then .toString()
will be called on it. scope.run
returns a readable stream that emits data generated by your code.
Uses the webview.executeJavascript electron API, which doesn't provide an error handling mechamism. Electron microscope wraps your code in a try/catch
and if an error occurs it will be emitted on the stream. However if you have a syntax error it will likely not catch it so it may appear nothing is happening.
You code must be a function that has this template:
function (send, done) {
// put your custom code here
// call 'send(data)' to write data to the stream
// call 'done()' to end the stream
// calling send is optional, but you must eventually call done to end the stream
}
For example:
var code = `function (send, done) {
for (var i = 0; i < 5; i++) send(i)
done()
}`
var output = scope.run(code)
output.on('data', function (data) {
// will get called for every time send is called above
// data will be the value passed to send
// in this case 5 times: 1, 2, 3, 4, 5
})
output.on('error', function (error) {
// will get called if your code throws an exception
// error will be an object with .message and .stack from the thrown error object
})
scope.on('will-navigate', cb)
Emitted the page wants to start navigation. It can happen when the window.location object is changed or a link is clicked in the page.
Calls cb
with (url)
, forwarded from this event.
scope.on('did-finish-load', cb)
This event is like did-finish-load
, but fired when the load failed or was cancelled.
Calls cb
with no arguments, forwarded from this event.
scope.on('did-fail-load', cb)
This event is like did-finish-load
, but fired when the load failed or was cancelled.
Calls cb
with (error)
, forwarded from this event.
scope.on('did-start-loading', cb)
Corresponds to the points in time when the spinner of the tab starts spinning.
Calls cb
with no arguments, forwarded from this event.
scope.on('did-stop-loading', cb)
Corresponds to the points in time when the spinner of the tab stops spinning.
Calls cb
with no arguments, forwarded from this event.
scope.destroy()
Call when you don't want to use the scope anymore. Causes the browser-window
elecron-microscope uses internally to close, which may cause your electron app to exit if you do not have any other active windows.