https://github.com/rubys/nokogumbo

A Nokogiri interface to the Gumbo HTML5 parser.
https://github.com/rubys/nokogumbo

Keywords from Contributors

libxml2 libxslt nokogiri ruby-gem sax xerces xslt rubygems activerecord activejob

Last synced: about 5 hours ago
JSON representation

Repository metadata

A Nokogiri interface to the Gumbo HTML5 parser.

Host: GitHub
URL: https://github.com/rubys/nokogumbo
Owner: rubys
License: apache-2.0
Archived: true
Created: 2013-08-17T22:06:55.000Z (over 12 years ago)
Default Branch: master
Last Pushed: 2023-08-10T16:56:40.000Z (over 2 years ago)
Last Synced: 2025-11-30T06:43:35.066Z (12 days ago)
Language: C
Size: 3 MB
Stars: 188
Watchers: 11
Forks: 46
Open Issues: 2
Releases: 3
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.txt

Nokogumbo - a Nokogiri interface to the Gumbo HTML5 parser.

NOTICE: End of life

Nokogumbo has been merged into Nokogiri v1.12.0. Please update to Nokogiri v1.12.0 or later, and remove Nokogumbo as an explicit dependency.

Please note that the final release of Nokogumbo, v2.0.5 (2021-03-19), will not support Ruby 3.2.0 and later. For Ruby 3.2 support, please upgrade to Nokogiri v1.12.0 or later and remove Nokogumbo as an explicit dependency.

Summary

Nokogumbo provides the ability for a Ruby program to invoke
our version of the Gumbo HTML5 parser
and to access the result as a
Nokogiri::HTML::Document.

Usage

require 'nokogumbo'
doc = Nokogiri.HTML5(string)

To parse an HTML fragment, a fragment method is provided.

require 'nokogumbo'
doc = Nokogiri::HTML5.fragment(string)

Because HTML is often fetched via the web, a convenience interface to
HTTP get is also provided:

require 'nokogumbo'
doc = Nokogiri::HTML5.get(uri)

Parsing options

The document and fragment parsing methods,

Nokogiri.HTML5(html, url = nil, encoding = nil, options = {})
Nokogiri::HTML5.parse(html, url = nil, encoding = nil, options = {})
Nokogiri::HTML5::Document.parse(html, url = nil, encoding = nil, options = {})
Nokogiri::HTML5.fragment(html, encoding = nil, options = {})
Nokogiri::HTML5::DocumentFragment.parse(html, encoding = nil, options = {})
support options that are different from Nokogiri's.

The three currently supported options are :max_errors, :max_tree_depth and
:max_attributes, described below.

Error reporting

Nokogumbo contains an experimental parse error reporting facility. By default,
no parse errors are reported but this can be configured by passing the
:max_errors option to ::parse or ::fragment.

require 'nokogumbo'
doc = Nokogiri::HTML5.parse('<span/>Hi there!</span foo=bar />', max_errors: 10)
doc.errors.each do |err|
  puts(err)
end

This prints the following.

1:1: ERROR: Expected a doctype token
<span/>Hi there!</span foo=bar />
^
1:1: ERROR: Start tag of nonvoid HTML element ends with '/>', use '>'.
<span/>Hi there!</span foo=bar />
^
1:17: ERROR: End tag ends with '/>', use '>'.
<span/>Hi there!</span foo=bar />
                ^
1:17: ERROR: End tag contains attributes.
<span/>Hi there!</span foo=bar />
                ^

Using max_errors: -1 results in an unlimited number of errors being
returned.

The errors returned by #errors are instances of
Nokogiri::XML::SyntaxError.

The HTML
standard
defines a number of standard parse error codes. These error codes only cover
the "tokenization" stage of parsing HTML. The parse errors in the
"tree construction" stage do not have standardized error codes (yet).

As a convenience to Nokogumbo users, the defined error codes are available
via the
Nokogiri::XML::SyntaxError#str1
method.

require 'nokogumbo'
doc = Nokogiri::HTML5.parse('<span/>Hi there!</span foo=bar />', max_errors: 10)
doc.errors.each do |err|
  puts("#{err.line}:#{err.column}: #{err.str1}")
end

This prints the following.

1:1: generic-parser
1:1: non-void-html-element-start-tag-with-trailing-solidus
1:17: end-tag-with-trailing-solidus
1:17: end-tag-with-attributes

Note that the first error is generic-parser because it's an error from the
tree construction stage and doesn't have a standardized error code.

For the purposes of semantic versioning, the error messages, error locations,
and error codes are not part of Nokogumbo's public API. That is, these are
subject to change without Nokogumbo's major version number changing. These may
be stabilized in the future.

Maximum tree depth

The maximum depth of the DOM tree parsed by the various parsing methods is
configurable by the :max_tree_depth option. If the depth of the tree would
exceed this limit, then an
ArgumentError is thrown.

This limit (which defaults to Nokogumbo::DEFAULT_MAX_TREE_DEPTH = 400) can
be removed by giving the option max_tree_depth: -1.

html = '<!DOCTYPE html>' + '<div>' * 1000
doc = Nokogiri.HTML5(html)
# raises ArgumentError: Document tree depth limit exceeded
doc = Nokogiri.HTML5(html, max_tree_depth: -1)

Attribute limit per element

The maximum number of attributes per DOM element is configurable by the
:max_attributes option. If a given element would exceed this limit, then an
ArgumentError is thrown.

This limit (which defaults to Nokogumbo::DEFAULT_MAX_ATTRIBUTES = 400) can
be removed by giving the option max_attributes: -1.

html = '<!DOCTYPE html><div ' + (1..1000).map { |x| "attr-#{x}" }.join(' ') + '>'
# "<!DOCTYPE html><div attr-1 attr-2 attr-3 ... attr-1000>"
doc = Nokogiri.HTML5(html)
# raises ArgumentError: Attributes per element limit exceeded
doc = Nokogiri.HTML5(html, max_attributes: -1)

HTML Serialization

After parsing HTML, it may be serialized using any of the Nokogiri
serialization
methods. In
particular, #serialize, #to_html, and #to_s will serialize a given node
and its children. (This is the equivalent of JavaScript's
Element.outerHTML.) Similarly, #inner_html will serialize the children of
a given node. (This is the equivalent of JavaScript's Element.innerHTML.)

doc = Nokogiri::HTML5("<!DOCTYPE html><span>Hello world!</span>")
puts doc.serialize
# Prints: <!DOCTYPE html><html><head></head><body><span>Hello world!</span></body></html>

Due to quirks in how HTML is parsed and serialized, it's possible for a DOM
tree to be serialized and then re-parsed, resulting in a different DOM.
Mostly, this happens with DOMs produced from invalid HTML. Unfortunately, even
valid HTML may not survive serialization and re-parsing.

In particular, a newline at the start of pre, listing, and textarea
elements is ignored by the parser.

doc = Nokogiri::HTML5(<<-EOF)
<!DOCTYPE html>
<pre>
Content</pre>
EOF
puts doc.at('/html/body/pre').serialize
# Prints: <pre>Content</pre>

In this case, the original HTML is semantically equivalent to the serialized
version. If the pre, listing, or textarea content starts with two
newlines, the first newline will be stripped on the first parse and the second
newline will be stripped on the second, leading to semantically different
DOMs. Passing the parameter preserve_newline: true will cause two or more
newlines to be preserved. (A single leading newline will still be removed.)

doc = Nokogiri::HTML5(<<-EOF)
<!DOCTYPE html>
<listing>

Content</listing>
EOF
puts doc.at('/html/body/listing').serialize(preserve_newline: true)
# Prints: <listing>
#
# Content</listing>

Encodings

Nokogumbo always parses HTML using
UTF-8; however, the encoding of the
input can be explicitly selected via the optional encoding parameter. This
is most useful when the input comes not from a string but from an IO object.

When serializing a document or node, the encoding of the output string can be
specified via the :encoding options. Characters that cannot be encoded in
the selected encoding will be encoded as HTML numeric
entities.

frag = Nokogiri::HTML5.fragment('<span>아는 길도 물어가라</span>')
html = frag.serialize(encoding: 'US-ASCII')
puts html
# Prints: <span>&#xc544;&#xb294; &#xae38;&#xb3c4; &#xbb3c;&#xc5b4;&#xac00;&#xb77c;</span>
frag = Nokogiri::HTML5.fragment(html)
puts frag.serialize
# Prints: <span>아는 길도 물어가라</span>

(There's a bug in all current
versions of Ruby that can cause the entity encoding to fail. Of the mandated
supported encodings for HTML, the only encoding I'm aware of that has this bug
is 'ISO-2022-JP'. I recommend avoiding this encoding.)

Examples

require 'nokogumbo'
puts Nokogiri::HTML5.get('http://nokogiri.org').search('ol li')[2].text

Notes

The Nokogiri::HTML5.fragment function takes a string and parses it
as a HTML5 document. The <html>, <head>, and <body> elements are
removed from this document, and any children of these elements that remain
are returned as a Nokogiri::HTML::DocumentFragment.
The Nokogiri::HTML5.parse function takes a string and passes it to the
gumbo_parse_with_options method, using the default options.
The resulting Gumbo parse tree is then walked.
- If the necessary Nokogiri and libxml2 headers
  can be found at installation time then an
  xmlDoc tree is produced
  and a single Nokogiri Ruby object is constructed to wrap the xmlDoc
  structure. Nokogiri only produces Ruby objects as necessary, so all
  searching is done using the underlying libxml2 libraries.
- If the necessary headers are not present at installation time, then
  Nokogiri Ruby objects are created for each Gumbo node. Other than
  memory usage and CPU time, the results should be equivalent.
The Nokogiri::HTML5.get function takes care of following redirects,
https, and determining the character encoding of the result, based on the
rules defined in the HTML5 specification for doing so.
Instead of uppercase element names, lowercase element names are produced.
Instead of returning unknown as the element name for unknown tags, the
original tag name is returned verbatim.

Flavors of Nokogumbo

Nokogumbo uses libxml2, the XML library underlying Nokogiri, to speed up
parsing. If the libxml2 headers are not available, then Nokogumbo resorts to
using Nokogiri's Ruby API to construct the DOM tree.

Nokogiri can be configured to either use the system library version of libxml2
or use a bundled version. By default (as of Nokogiri version 1.8.4), Nokogiri
will use a bundled version.

To prevent differences between versions of libxml2, Nokogumbo will only use
libxml2 if the build process can find the exact same version used by Nokogiri.
This leads to three possibilities

Nokogiri is compiled with the bundled libxml2. In this case, Nokogumbo will
(by default) use the same version of libxml2.
Nokogiri is compiled with the system libxml2. In this case, if the libxml2
headers are available, then Nokogumbo will (by default) use the system
version and headers.
Nokogiri is compiled with the system libxml2 but its headers aren't
available at build time for Nokogumbo. In this case, Nokogumbo will use the
slower Ruby API.

Using libxml2 can be required by passing -- --with-libxml2 to bundle exec rake or to gem install. Using libxml2 can be prohibited by instead passing
-- --without-libxml2.

Functionally, the only difference between using libxml2 or not is in the
behavior of Nokogiri::XML::Node#line. If it is used, then #line will
return the line number of the corresponding node. Otherwise, it will return 0.

Installation

git clone https://github.com/rubys/nokogumbo.git
cd nokogumbo
bundle install
rake gem
gem install pkg/nokogumbo*.gem

Related efforts

ruby-gumbo -- a ruby binding
for the Gumbo HTML5 parser.
lua-gumbo -- a lua binding for
the Gumbo HTML5 parser.

Owner metadata

Name: Sam Ruby
Login: rubys
Email:
Kind: user
Description:
Website: http://intertwingly.net/blog/
Location: Raleigh, NC
Twitter:
Company: fly.io
Icon url: https://avatars.githubusercontent.com/u/4815?v=4
Repositories: 91
Last ynced at: 2023-04-10T07:45:58.390Z
Profile URL: https://github.com/rubys

GitHub Events

Total

Watch event: 2

Last Year

Watch event: 2

Committers metadata

Last synced: 7 days ago

Total Commits: 445
Total Committers: 25
Avg Commits per committer: 17.8
Development Distribution Score (DDS): 0.688

Commits in past year: 0
Committers in past year: 0
Avg Commits per committer in past year: 0.0
Development Distribution Score (DDS) in past year: 0.0

Name	Email	Commits
Craig Barnes	Cr@i****s	139
Stephen Checkoway	s@p****g	129
Sam Ruby	r**s@i**t	120
Mike Dalessio	m**o@g**m	12
Rafael Masson	r**n@g**m	9
Ryan Grove	r**n@w**m	5
Sven Schwyn	s**n@b**m	3
Jeremy Daer	j**r@g**m	3
Joel Low	j**l@j**g	3
Robin H. Johnson	r**2@g**g	3
Fraudfilter	v**v@m**u	2
Matt Wildig	m**t@m**k	2
Ryan Liptak	s**2@h**m	2
William Entriken	g**m@p**t	2
Alexandre Bernard	a**c@a**m	1
Itay Duvdevani	d**v@f**m	1
Jakub Jirutka	j**b@j**z	1
Johan Smits	j**s@l**u	1
John Hawthorn	j**n@h**l	1
Kevin Rutten	K**n@K**m	1
Krzysztof Kotlarek	k**f@g**m	1
Michael Grosser	m**l@g**t	1
Mick Staugaard	m**k@s**m	1
jbotelho2-bb	j**2@b**t	1
mrasu	m**i@g**m	1

Committer domains:

Issue and Pull Request metadata

Last synced: 3 months ago

Total issues: 47
Total pull requests: 55
Average time to close issues: 10 months
Average time to close pull requests: about 1 month
Total issue authors: 38
Total pull request authors: 10
Average comments per issue: 6.47
Average comments per pull request: 1.51
Merged pull request: 51
Bot issues: 0
Bot pull requests: 0

Past year issues: 0
Past year pull requests: 0
Past year average time to close issues: N/A
Past year average time to close pull requests: N/A
Past year issue authors: 0
Past year pull request authors: 0
Past year average comments per issue: 0
Past year average comments per pull request: 0
Past year merged pull request: 0
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/rubys/nokogumbo

Top Issue Authors

stevecheckoway (8)
flavorjones (2)
dan42 (2)
agross (1)
nyarly (1)
voondo (1)
thisconnect (1)
tnir (1)
Ganesan-Apollo-TW (1)
caponecicero (1)
staycreativedesign (1)
fulldecent (1)
banureddy1947 (1)
am-root (1)
dsisnero (1)

Top Pull Request Authors

stevecheckoway (38)
flavorjones (7)
rafbm (2)
fulldecent (2)
jeremy (1)
jhawthorn (1)
tupaschoal (1)
lis2 (1)
jtarchie (1)
johan-smits (1)

Top Issue Labels

enhancement (1)

Top Pull Request Labels

Package metadata

Total packages: 2
Total downloads:
- rubygems: 92,216,639 total
Total docker downloads: 144,068,340
Total dependent packages: 22 (may contain duplicates)
Total dependent repositories: 7,649 (may contain duplicates)
Total versions: 102
Total maintainers: 3

gem.coop: nokogumbo

Nokogumbo allows a Ruby program to invoke the Gumbo HTML5 parser and access the result as a Nokogiri parsed document.

Homepage: https://github.com/rubys/nokogumbo/#readme
Documentation: http://www.rubydoc.info/gems/nokogumbo/
Licenses: Apache-2.0
Latest release: 2.0.5 (published over 4 years ago)
Last Synced: 2025-12-09T01:32:28.504Z (3 days ago)
Versions: 51
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 46,107,506 Total
Docker Downloads: 72,034,170
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 0.274%
- Downloads: 0.529%
- Docker downloads count: 0.566%
Maintainers (3)

rubygems.org: nokogumbo