While it is useful for de-duplicating web documents, other tasks related to content extraction also profit from a cleaner text base, as it makes work on the “real” content possible. While some large-scale algorithms can be expected to smooth out irregularities, uses requiring a low margin of error and close reading approaches (such as the search for examples in lexicographic research) imply constant refinements and improvements with respect to the building and processing of the dataset.ĭistinguishing between the whole page and the main text content can help alleviating many quality problems related to web texts: if the main text is too short or redundant, it may not be necessary to use it. Depending of the purpose of data collection, it may also require a substantial filtering and quality assessment. Web data mining involves a significant number of design decisions and turning points in data processing. Date Fri 13 September 2019 Category Tutorial Tags python
0 kommentar(er)
