Model Creation Helpers

webstruct.model contains convetional wrappers for creating NER models.

class webstruct.model.NER(model, loader=None, html_tokenizer=None, entity_colors=None)

Class for extracting named entities from HTML.

Initialize it with a trained model. model must have predict method that accepts lists of HtmlToken sequences and returns lists of predicted IOB2 tags. create_wapiti_pipeline() function returns such model.

extract(bytes_data)

Extract named entities from binary HTML data bytes_data. Return a list of (entity_text, entity_type) tuples.

extract_from_url(url)

A convenience wrapper for extract() method that downloads input data from a remote URL.

extract_raw(bytes_data)

Extract named entities from binary HTML data bytes_data. Return a list of (html_token, iob2_tag) tuples.

extract_groups(bytes_data, dont_penalize=None)

Extract groups of named entities from binary HTML data bytes_data. Return a list of lists of (entity_text, entity_type) tuples.

Entites are grouped using algorithm from webstruct.grouping.

extract_groups_from_url(url, dont_penalize=None)

A convenience wrapper for extract_groups() method that downloads input data from a remote URL.

build_entity(html_tokens)

Join tokens to an entity. Return an entity, as text. By default this function uses webstruct.utils.smart_join().

Override it to customize extract(), extract_from_url() and extract_groups() results. If this function returns empty string or None, entity is dropped.

annotate(bytes_data, url=None, pretty_print=False)

Return annotated HTML data in WebAnnotator format.

annotate_url(url, pretty_print=False)

Return annotated HTML data in WebAnnotator format; input is downloaded from url.