tobast/mpri-webdam
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

We do not want to use settings

Browse source
This commit is contained in:
Rémi Oudin 2018-02-26 15:14:53 +01:00
parent 65f777f00f
commit bdfa285e6b
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
crawl/crawl.py
View file

@ -73,7 +73,7 @@ def url_getter(html, current_page, root_url):
# Works only with python >= 3.6
links_list = list(dict.fromkeys(links_list))
forbidden_words = ['login', 'agreement', 'mailto']
forbidden_words = ['login', 'agreement', 'mailto', 'settings']
links_list = [link for link in links_list if not any(word in link.lower()
for word in
forbidden_words)]