Discover prolificinteractive/node-html-to-json Open Source project

Parses HTML strings into objects using flexible, composable filters.

Installation

npm install html-to-json

htmlToJson.parse(html, filter, [callback]) -> promise

The parse() method takes a string of HTML, and a filter, and responds with the filtered data. This supports both callbacks and promises.

var promise = htmlToJson.parse('<div>content</div>', {
  'text': function ($doc) {
    return $doc.find('div').text();
  }
}, function (err, result) {
  console.log(result);
});

promise.done(function (result) {
  //Works as well
});

htmlToJson.request(requestOptions, filter, [callback]) -> promise

The request() method takes options for a call to the request library and a filter, then returns the filtered response body.

var promise = htmlToJson.request('http://prolificinteractive.com/team', {
  'images': ['img', function ($img) {
    return $img.attr('src');
  }]
}, function (err, result) {
  console.log(result);
});

htmlToJson.batch(html, dictionary, [callback]) -> promise

Performs many parsing operations against one HTML string. This transforms the HTML into a DOM only once instead of for each filter in the dictionary, which can quickly get expensive in terms of processing. This also allows you to break your filters up into more granular components and mix and match them as you please.

The values in the dictionary can be htmlToJson.Parser objects, generated methods from htmlToJson.createMethod, or naked filters that you might normally pass into htmlToJson.parse. For example:

return getProlificHomepage().then(function (html) {
  return htmlToJson.batch(html, {
    sections: htmlToJson.createParser(['#primary-nav a', {
      'name': function ($section) {
        return $section.text();
      },
      'link': function ($section) {
        return $section.attr('href');
      }
    }]),
    offices: htmlToJson.createMethod(['.office', {
      'location': function ($office) {
        return $office.find('.location').text();
      },
      'phone': function ($office) {
        return $office.find('.phone').text();
      }
    }]),
    socialInfo: ['#footer .social-link', {
      'name': function ($link) {
        return $link.text();
      },
      'link': function ($link) {
        return $link.attr('href');
      }
    }]
  });
});

htmlToJson.createMethod(filter) -> function (html, [callback])

Generates a method that wraps the passed filter argument. The generated method takes an HTML string and processes it against that filter.

var parseFoo = htmlToJson.createMethod({
  'foo': function ($doc) {
    return $doc.find('#foo').bar();
  }
});

htmlToJson.createParser(filter), new htmlToJson.Parser(filter)

For the sake of reusability, creates an object with .parse and .request helper methods, which use the passed filter. For example:

var linkParser = htmlToJson.createParser(['a[href]', {
  'text': function ($a) {
    return $a.text();
  },
  'href': function ($a) {
    return $a.attr('href');
  }
}]);

linkParser.request('http://prolificinteractive.com').done(function (links) {
  //Do stuff with links
});

is equivalent to:

linkParser.request('http://prolificinteractive.com', ['a[href]', {
  'text': function ($a) {
    return $a.text();
  },
  'href': function ($a) {
    return $a.attr('href');
  }
}]).done(function (links) {
  //Do stuff with links
});

The former allows you to easily reuse the filter (and make it testable), while that latter is a one-off.

parser.parse(html, [callback])

Parses the passed html argument against the parser's filter.

parser.method(html, [callback])

Returns a method that wraps parser.parse()

parser.request(requestOptions, [callback])

Makes a request with the request options, then runs the response body through the parser's filter.

Filter Types

Functions

The return values of functions are mapped against their corresponding keys. Function filters are passed cheerio objects, which allows you to play with a jQuery-like interface.

htmlToJson.parse('<div id="foo">foo</div>', {
  'foo1': function ($doc, $) {
    return $doc.find('#foo').text(); //foo
  }
}, callback);

Arrays

Arrays of data can be parsed out by either using the .map() method within a filter function or using the shorthand [selector, filter] syntax:

.map(selector, filter)

A filter is applied incrementally against each matched element, and the results are returned within an array.

var html = '<div id="items"><div class="item">1</div><div class="item">2</div></div>';

htmlToJson.parse(html, function () {
  return this.map('.item', function ($item) {
    return $item.text();
  });
}).done(function (items) {
  // Items should be: ['1','2']
}, function (err) {
  // Handle error
});

[selector, filter, after]

This is essentially a short-hand alias for .map(), making the filter look more like its output:

var html = '<div id="items"><div class="item">1</div><div class="item">2</div></div>';

htmlToJson
  .parse(html, ['.item', function ($item) {
    return $item.text();
  }])
  .done(function (items) {
    // Items should be: ['1','2']
  }, function (err) {
    // Handle error
  });

As an added convenience you can pass in a 3rd argument into the array filter, which allows you to manipulate the results. You can return a promise if you wish to do an asynchronous operation.

var html = '<div id="items"><div class="item">1</div><div class="item">2</div></div>';

htmlToJson
  .parse(html, ['.item', function ($item) {
    return +$item.text();
  }, function (items) {
    return _.map(items, function (item) {
      return item * 3;
    });
  }])
  .done(function (items) {
    // Items should be: [3,6]
  }, function (err) {
    // Handle error
  });

Asynchronous filters

Filter functions may also return promises, which get resolved asynchronously.

function getProductDetails (id, callback) {
  return htmlToJson.request({
    uri: 'http://store.prolificinteractive.com/products/' + id
  }, {
    'id': function ($doc) {
      return $doc.find('#product-details').attr('data-id');
    },
    'colors': ['.color', {
      'id': function ($color) {
        return $color.attr('data-id');
      },
      'hex': function ($color) {
        return $color.css('background-color');
      }
    }]
  }, callback);
}

function getProducts (callback) {
  return htmlToJson.request({
    uri: 'http://store.prolificinteractive.com'
  }, ['.product', {
    'id': function ($product) {
      return $product.attr('data-id');
    },
    'image': function ($product) {
      return $product.find('img').attr('src');
    },
    'colors': function ($product) {
      // This is where we use a promise to get the colors asynchronously
      return this
        .get('id')
        .then(function (id) {
          return getProductDetails(id).get('colors');
        });
    }
  }], callback);
}

Dependencies on other values

Filter functions may use the .get(propertyName) to use a value from another key in that filter. This returns a promise representing the value rather than the value itself.

function getProducts (callback) {
  return htmlToJson.request('http://store.prolificinteractive.com', ['.product', {
    'id': function ($product) {
      return $product.attr('data-id');
    },
    'image': function ($product) {
      return $product.find('img').attr('src');
    },
    'colors': function ($product) {
      // Resolve 'id' then get product details with it
      return this
        .get('id')
        .then(function (id) {
          return getProductDetails(id).get('colors');
        });
    }
  }], callback);
}

Objects

Nested objects within a filter are run against the same HTML context as the parent filter.

var html = '<div id="foo"><div id="bar">foobar</div></div>';

htmlToJson.parse(html, {
  'foo': {
    'bar': function ($doc) {
      return $doc.find('#bar').text();
    }
  }
});

$container modifier

You may specify a more specific DOM context by setting the $container property on the object filter:

var html = '<div id="foo"><div id="bar">foobar</div></div>';

htmlToJson.parse(html, {
  'foo': {
    $container: '#foo',
    'bar': function ($foo) {
      return $foo.find('#bar').text();
    }
  }
});

Constants

Strings, numbers, and null values are simply used as the filter's value. This especially comes in handy for incrementally converting from mock data to parsed data.

htmlToJson.parse('<div id="nada"></div>', {
  x: 1,
  y: 'string value',
  z: null
});

Contributing

Running Tests

Tests are written in mocha and located in the test directory. Run them with:

npm test

This script also executes jshint against lib/ and test/ directories.

Style

Please read the existing code in order to learn the conventions.

prolificinteractive/node-html-to-json

prolificinteractive

Reviews

Repository Details