Commit graph

20 commits

Author SHA1 Message Date
9b78e268c9 Nearly working crawler 2018-02-22 14:33:07 +01:00
e19e623df1 Multiple bug fixes. TODO : remove <div id=footer>-like patterns 2018-02-22 14:07:53 +01:00
236e15296c It can be useful to return the links list 2018-02-21 23:11:57 +01:00
4e6ac5ac7b Url getter function : retrieves the list of so-called relevant links 2018-02-21 22:51:05 +01:00
a907cad33d Start of url getter function 2018-02-21 19:06:46 +01:00
b05e642c79 Make the code somewhat readable 2018-02-21 11:54:41 +01:00
c97acb22b5 Add tentative crawl file
Nothing functional, just tests
2018-02-20 12:48:53 +01:00
c05c2561d2 Add crawler settings and requirements 2018-02-20 12:48:16 +01:00
bef1fca5b9 Init app 'crawl' 2018-02-20 08:51:16 +01:00
924657abdb Generate profiles' migration 2018-01-24 22:49:34 +01:00
e9b3127226 Use profiles as an installed application in pinocchio 2018-01-24 22:49:08 +01:00
cbf1911fe7 Add models for Interest and Profile 2018-01-24 22:48:53 +01:00
37581fb96a Add models for Place and Event 2018-01-24 22:39:20 +01:00
6531415d63 Add model for a webpage and website 2018-01-24 14:09:33 +01:00
114c8a3d3e Add model for search engines 2018-01-24 13:52:43 +01:00
225742798b Add BrowserFingerprint model 2018-01-24 13:36:55 +01:00
a3e6308837 Init apps histories and profiles 2018-01-23 18:12:47 +01:00
397784a673 Add first version of requirements.txt
Mainly Django, by now
2018-01-23 18:11:07 +01:00
132b7250c8 Initialize Django 2018-01-23 18:11:00 +01:00
c1e3be346f Initial commit 2018-01-23 17:53:08 +01:00