https://wiki.tei-c.org/index.php?action=history&feed=atom&title=TrafilaturaTrafilatura - Revision history2026-04-24T00:41:17ZRevision history for this page on the wikiMediaWiki 1.32.0https://wiki.tei-c.org/index.php?title=Trafilatura&diff=16799&oldid=prevPiotr Banski: text prepared by Adrien Barbaresi2020-08-06T13:30:15Z<p>text prepared by Adrien Barbaresi</p>
<p><b>New page</b></p><div>[[Category:Tools]]<br />
<br />
[[Category:Development tools]]<br />
[[Category:Conversion and preprocessing tools]]<br />
[[Category:Testing and QA tools]]<br />
[[Category:Analysis tools]]<br />
[[Category:All-in-one Tools]]<br />
<br />
[[Category:Discovering]]<br />
[[Category:Annotating]]<br />
[[Category:Sampling]]<br />
<br />
== Synopsis ==<br />
<br />
Trafilatura is a Python library and command-line tool which processes<br />
HTML documents and converts the output to plain text, CSV, JSON, XML and<br />
TEI-XML. It seamlessly downloads, parses, and scrapes web page data: it<br />
can extract metadata, the main body text and comments while preserving<br />
part of the text formatting and page structure. It also includes a<br />
parser and validator for TEI documents.<br />
<br />
Distinguishing between whole page and essential parts can help to<br />
alleviate many quality problems related to web texts as it deals with<br />
the noise consisting of recurring elements (headers and footers, ads,<br />
links/blogroll). The extractor has to be precise enough not to miss<br />
texts or discard valid documents, robust but also reasonably fast. It is<br />
designed to run in production on millions of web documents.<br />
<br />
This effort serves the development of methods for deriving information<br />
from web documents in order to build text databases for research,<br />
especially for linguistic analysis and natural language processing as<br />
part of projects by the Center for Digital Lexicography of German<br />
([https://zdl.org ZDL] and [https://www.dwds.de DWDS]). A significant<br />
challenge resides in the ability to extract and pre-process web texts to<br />
meet scientific expectations: Web corpus construction involves numerous<br />
design decisions, and this software package helps facilitate collection<br />
and enhance corpus quality.<br />
<br />
== Features ==<br />
<br />
* Robust extraction algorithm preserving<br />
** Metadata (title, author, date, site name, categories and tags)<br />
** Structural elements (paragraphs, titles, lists, quotes, code, line breaks, in-line text formatting)<br />
* Seamless, parallelized online (including page retrieval) or offline processing<br />
* TEI-XML supported as output format, including document validation<br />
* Management of download/URL lists (ATOM/RSS feeds, URL queues, blacklisting)<br />
<br />
== User commentary ==<br />
<br />
'''Please sign all comments.'''<br />
<br />
== System requirements ==<br />
<br />
The software is tested on MacOS and Linux, it is expected to work on<br />
Windows as well. It supports all common Python 3 versions (3.4 upwards).<br />
<br />
== Source code and licensing ==<br />
<br />
Open-source software under [https://www.gnu.org/licenses/gpl-3.0.en.html GNU General Public License v3.0].<br />
<br />
Source code homepage: see [https://github.com/adbar/trafilatura GitHub repository].<br />
<br />
== Support for TEI ==<br />
<br />
Support for TEI P5 out of the box.<br />
<br />
== Language(s) ==<br />
<br />
* Tool written in Python, use within Python or on the command-line.<br />
* The documentation is available in English.<br />
<br />
== Documentation ==<br />
<br />
* [https://trafilatura.readthedocs.io/ Documentation]<br />
* [https://trafilatura.readthedocs.io/en/latest/tutorials.html Tutorials]<br />
** [https://trafilatura.readthedocs.io/en/latest/tutorial2.html Production and validation of TEI files in Python]<br />
** [http://adrien.barbaresi.eu/blog/validating-tei-xml-python.html Validating TEI-XML documents]<br />
<br />
== Tech support ==<br />
<br />
Using this [http://adrien.barbaresi.eu/ contact info] or filing issues<br />
on the [https://github.com/adbar/trafilatura/issues dedicated page].<br />
<br />
== User community ==<br />
<br />
[https://github.com/adbar/trafilatura/blob/master/CONTRIBUTING.md Contributions] to code and documentation are welcome! These<br />
[https://github.com/adbar/trafilatura/graphs/contributors contributors] have already submitted features and fixes.<br />
<br />
Tutorial video in German by Simon Meier-Vieracker: [https://www.youtube.com/watch?v=9RPrVE0hHgI Content von Webseiten laden mit Trafilatura].<br />
<br />
== History of versions ==<br />
<br />
Current version: [https://pypi.org/project/trafilatura/ v0.5.1], 2020-07-15.<br />
<br />
* [https://github.com/adbar/trafilatura/blob/master/HISTORY.md Change log]<br />
* [https://github.com/adbar/trafilatura/releases GitHub releases]<br />
<br />
== Download and installation ==<br />
<br />
Trafilatura is packaged as a software library available from the package<br />
repository ''PyPI''. As such it can notably be installed with<br />
<code>pip</code> or <code>pipenv</code>: <code>pip install --upgrade<br />
trafilatura</code>.<br />
<br />
For more details please read the [https://trafilatura.readthedocs.io/en/latest/installation.html installation documentation].<br />
<br />
== Additional notes ==<br />
<br />
'''Evaluation and alternatives:''' The extraction focuses on the main<br />
content, which is usually the part displayed centrally, without the left<br />
or right bars, the header or the footer, but including potential titles<br />
and (optionally) comments. This task is also known as ''web scraping'',<br />
''boilerplate removal'', ''DOM-based content extraction'', ''main<br />
content identification'', or ''web page cleaning''.<br />
<br />
Reproducible results are published on the<br />
[https://trafilatura.readthedocs.io/en/latest/evaluation.html evaluation page].</div>Piotr Banski