Commit graph

Select branches

Hide pull requests

histories_models

histories_tobast

histories_xml

master

rdf

tobast_last_minute

Mono Color

• 89d1f8301a Remove duplicated url in history master Rémi Oudin 2018-02-26 17:46:49 +0100
• 379b53e6ce Fix printing in gen_history Théophile Bastian 2018-02-26 17:24:22 +0100
• c94841c17b Add gen_history django-admin command Théophile Bastian 2018-02-26 17:20:59 +0100
• 97107d9bec Merge branch 'master' of git.tobast.fr:tobast/mpri-webdam Rémi Oudin 2018-02-26 17:12:26 +0100
• dedc66bb9d Bug fix Rémi Oudin 2018-02-26 17:12:19 +0100
• d3d04739e7 Add DuckDuckGo lite search engine to stock data Théophile Bastian 2018-02-26 17:10:18 +0100
• b88aeffd5a Helpful README Théophile Bastian 2018-02-26 17:09:05 +0100
• 7c8ec7351c Merge branch 'master' of git.tobast.fr:tobast/mpri-webdam Rémi Oudin 2018-02-26 17:04:09 +0100
• 392e16b797 Merge branch 'histories_models' Rémi Oudin 2018-02-26 17:03:27 +0100
• 2005c0f24f Add xml string gen Théophile Bastian 2018-02-26 17:03:27 +0100
• 185c1cf8a4 Fix XML generation tobast_last_minute Théophile Bastian 2018-02-26 16:59:18 +0100
• 9dd1954067 Partial runner fix Rémi Oudin 2018-02-26 16:37:51 +0100
• 04270e88c0 Bug fix Rémi Oudin 2018-02-26 16:37:32 +0100
• 6bc64ceb7a Add requirement for aiohttp Théophile Bastian 2018-02-26 16:38:16 +0100
• 15e0c2a11c Partial runner fix histories_models Rémi Oudin 2018-02-26 16:37:51 +0100
• 2b07779f5c Bug fix Rémi Oudin 2018-02-26 16:37:32 +0100
• 8cdc50c04e Fix stupid typo Théophile Bastian 2018-02-26 16:32:47 +0100
• 22fa039f1b Remove debug print Rémi Oudin 2018-02-26 16:23:14 +0100
• e4ad8c7ce6 Towards a working XML export Théophile Bastian 2018-02-26 15:58:30 +0100
• 67ad232533 Add a timeout to a single page retrieval Théophile Bastian 2018-02-26 15:42:36 +0100
• e140d4a8a7 Fix merge remanences Théophile Bastian 2018-02-26 15:37:05 +0100
• 98fe69ba62 Real async crawling Théophile Bastian 2018-02-26 15:27:57 +0100
• 968ff6d24c More robust crawling Théophile Bastian 2018-02-26 13:01:05 +0100
• 5d4bd30e20 Exception handling Rémi Oudin 2018-02-26 15:15:03 +0100
• bdfa285e6b We do not want to use settings Rémi Oudin 2018-02-26 15:14:53 +0100
• 65f777f00f Should get the objects and not the Manager Rémi Oudin 2018-02-26 15:04:26 +0100
• 236e40d359 Sanity check Rémi Oudin 2018-02-26 14:57:46 +0100
• 22017cea91 Typo in data u_u Rémi Oudin 2018-02-26 14:56:22 +0100
• 549c861908 Bug fixé Rémi Oudin 2018-02-26 14:38:26 +0100
• 517be1d822 Merge rdf branch Rémi Oudin 2018-02-26 14:11:06 +0100
• c4f63a92b2 Error in the merge, mea culpa Rémi Oudin 2018-02-26 14:01:29 +0100
• db067e56fc Typo Rémi Oudin 2018-02-26 13:59:34 +0100
• 33bdae96e4 merge commit from histories_tobast into histories_models Rémi Oudin 2018-02-26 12:59:38 +0100
• 526aad1364 Add interests Rémi Oudin 2018-02-26 12:33:23 +0100
• 02e91bb2b7 Fix function calls Théophile Bastian 2018-02-26 11:56:02 +0100
• 3e5fc2f9b3 Fix search engine URL generation Théophile Bastian 2018-02-26 10:19:51 +0100
• 45ddbff91a Crawling and histories: fix a lot of stuff Théophile Bastian 2018-02-26 00:24:54 +0100
• e6d587bffd Actually save to DB a created history Théophile Bastian 2018-02-25 20:00:34 +0100
• 8baf408e02 Use dict from data/nicknames_dict for nicknames Théophile Bastian 2018-02-25 19:59:57 +0100
• 6463e348ac Fix populate.sh exec path Théophile Bastian 2018-02-25 18:55:27 +0100
• 22064ebee3 Histories: xml import/export — untested Théophile Bastian 2018-02-25 18:06:59 +0100
• a4de51b84a Crawl: do not use global SEARCH_ENGINES Théophile Bastian 2018-02-26 11:45:08 +0100
• 4f0148cb63 Crawler: use a random fingerprint Théophile Bastian 2018-02-26 11:27:07 +0100
• 4a8bd32516 Fix tor_runner import Théophile Bastian 2018-02-26 11:12:16 +0100
• 90a6164861 Fix search engine URL generation histories_xml Théophile Bastian 2018-02-26 10:19:51 +0100
• b7be4f4df4 Crawling and histories: fix a lot of stuff Théophile Bastian 2018-02-26 00:24:54 +0100
• a6d7d6b62b Actually save to DB a created history Théophile Bastian 2018-02-25 20:00:34 +0100
• f33820a4dc Use dict from data/nicknames_dict for nicknames Théophile Bastian 2018-02-25 19:59:57 +0100
• 04fcc2b324 Fix populate.sh exec path Théophile Bastian 2018-02-25 18:55:27 +0100
• 6e4709ac91 Histories: xml import/export — untested Théophile Bastian 2018-02-25 18:06:59 +0100
• fd4e1d35c7 Crawl: do not use global SEARCH_ENGINES Théophile Bastian 2018-02-26 11:45:08 +0100
• 44cf26df8f It can be useful to save a new object Rémi Oudin 2018-02-26 11:42:45 +0100
• 8f1d69bc41 Crawler: use a random fingerprint Théophile Bastian 2018-02-26 11:27:07 +0100
• 38ccd04d31 Fix tor_runner import Théophile Bastian 2018-02-26 11:12:16 +0100
• adb892ab7d Check if crawling a search engine Rémi Oudin 2018-02-26 11:12:36 +0100
• 15db8b4697 Change option name due to downgrade of aiohttp Rémi Oudin 2018-02-26 10:23:32 +0100
• d6b26c0a46 Better use of history Rémi Oudin 2018-02-26 10:05:33 +0100
• 8f5c4f3f0f Use datetimes Rémi Oudin 2018-02-26 09:49:24 +0100
• 71d9e18eec Add headers support Rémi Oudin 2018-02-25 23:56:51 +0100
• 8ad46c0481 Bug fix, syntax erro Rémi Oudin 2018-02-25 21:59:29 +0100
• f66c978466 Tor runner has a run function to replay the history Rémi Oudin 2018-02-25 21:53:28 +0100
• 0a676a2f65 PEP8 Rémi Oudin 2018-02-25 21:34:20 +0100
• e074d96f02 tor_runner can make requests Rémi Oudin 2018-02-25 21:27:15 +0100
• 93b235cb6c Fix interests import histories_tobast Rémi Oudin 2018-02-25 21:20:52 +0100
• ae5699c089 Basic tor runner Rémi Oudin 2018-02-25 19:42:58 +0100
• f7313ff659 Add populate.sh script Rémi Oudin 2018-02-25 16:16:04 +0100
• 0661fe0f01 Fix path Rémi Oudin 2018-02-25 16:10:38 +0100
• 4b19febdf6 Add interests Rémi Oudin 2018-02-25 16:10:22 +0100
• 15323c3465 [REBASE ME] Crawl: enhance efficiency and output a tree Théophile Bastian 2018-02-25 15:08:06 +0100
• c3bcdea1eb Add tentative export to RDF rdf Théophile Bastian 2018-02-25 13:34:44 +0100
• 05a2e2ca3f Partial generation of profiles Rémi Oudin 2018-02-25 13:18:12 +0100
• d4aefb6bb7 Load the data Rémi Oudin 2018-02-25 13:17:44 +0100
• 3eb82a4a0b data for names and emails Rémi Oudin 2018-02-25 13:17:27 +0100
• 7c0fb7dda1 Better naming Rémi Oudin 2018-02-25 11:49:44 +0100
• ee32e5385b Finished data import Rémi Oudin 2018-02-25 11:49:11 +0100
• bc7348f677 Integration of crawl module in histories Rémi Oudin 2018-02-24 23:17:24 +0100
• 60bfc8cb77 Merge branch 'crawl' into histories_models Rémi Oudin 2018-02-24 18:44:27 +0100
• 12c8c652d7 Serialisation function Rémi Oudin 2018-02-24 18:40:27 +0100
• c58f42476f Missing script for 854481d Rémi Oudin 2018-02-24 17:22:52 +0100
• 854481dbd3 Import utilities Rémi Oudin 2018-02-24 17:21:41 +0100
• d19c2e8216 Add mailto adresses to forbidden list Rémi Oudin 2018-02-24 15:41:46 +0100
• e56c088632 Better filter Rémi Oudin 2018-02-24 11:39:04 +0100
• 2732e4115f Add RDF models export classes — untested Théophile Bastian 2018-02-23 13:32:32 +0100
• f0b8672c89 Silly me. (bis) Rémi Oudin 2018-02-23 10:44:51 +0100
• f6da179820 If robots.txt file is invalid, abort mission. Rémi Oudin 2018-02-23 10:36:14 +0100
• 0e02f22d08 Exception handling Rémi Oudin 2018-02-23 00:37:36 +0100
• 77ca7ebcb9 Silly me. Rémi Oudin 2018-02-22 15:35:46 +0100
• 9b78e268c9 Nearly working crawler Rémi Oudin 2018-02-22 14:33:07 +0100
• e19e623df1 Multiple bug fixes. TODO : remove <div id=footer>-like patterns Rémi Oudin 2018-02-22 14:07:53 +0100
• 5decd205fb Typos + improvements Rémi Oudin 2018-02-22 11:06:45 +0100
• 236e15296c It can be useful to return the links list Rémi Oudin 2018-02-21 23:11:57 +0100
• 4e6ac5ac7b Url getter function : retrieves the list of so-called relevant links Rémi Oudin 2018-02-21 22:51:05 +0100
• a907cad33d Start of url getter function Rémi Oudin 2018-02-21 19:06:46 +0100
• ad0ad0a783 Command to add browser fingerprint data Rémi Oudin 2018-02-21 16:50:27 +0100
• b05e642c79 Make the code somewhat readable Théophile Bastian 2018-02-21 11:54:41 +0100
• cd4d8a4c3f More generic code using @8f4458b Rémi Oudin 2018-02-21 11:50:28 +0100
• 8f4458b009 Url generation method, for more genericity Rémi Oudin 2018-02-21 11:37:44 +0100
• 5539f57139 Add missing docstrings Rémi Oudin 2018-02-21 11:35:53 +0100
• 4920de5838 Going on in the generation of history Rémi Oudin 2018-02-20 23:42:21 +0100
• c97acb22b5 Add tentative crawl file Théophile Bastian 2018-02-20 12:48:53 +0100