New upstream version 1.9.0upstream/1.9.0

4 files changed, 175 insertions, 30 deletions

diff --git a/docs/configuration.rst b/docs/configuration.rst
index c606c6c..32a529a 100644
--- a/docs/configuration.rst
+++ b/docs/configuration.rst
@@ -217,7 +217,7 @@ extractor.*.user-agent
 ----------------------
 =========== =====
 Type        ``string``
-Default     ``"Mozilla/5.0 (X11; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"``
+Default     ``"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"``
 Description User-Agent header value to be used for HTTP requests.
 
             Note: This option has no effect on `pixiv` and
@@ -300,8 +300,9 @@ extractor.*.retries
 -------------------
 =========== =====
 Type        ``integer``
-Default     ``5``
-Description Number of times a failed HTTP request is retried before giving up.
+Default     ``4``
+Description Maximum number of times a failed HTTP request is retried before
+            giving up or ``-1`` for infinite retries.
 =========== =====
 
 
@@ -333,6 +334,22 @@ Description Controls whether to verify SSL/TLS certificates for HTTPS requests.
 =========== =====
 
 
+extractor.*.download
+--------------------
+=========== =====
+Type        ``bool``
+Default     ``true``
+Description Controls whether to download media files.
+
+            Setting this to ``false`` won't download any files, but all other
+            functions (postprocessors_, `download archive`_, etc.)
+            will be executed as normal.
+=========== =====
+
+.. _postprocessors: `extractor.*.postprocessors`_
+.. _download archive: `extractor.*.archive`_
+
+
 extractor.*.image-range
 -----------------------
 =========== =====
@@ -381,6 +398,40 @@ Description Like `image-filter`__, but applies to delegated URLs
 __ `extractor.*.image-filter`_
 
 
+extractor.*.image-unique
+------------------------
+=========== =====
+Type        ``bool``
+Default     ``false``
+Description Ignore image URLs that have been encountered before during the
+            current extractor run.
+=========== =====
+
+
+extractor.*.chapter-unique
+--------------------------
+=========== =====
+Type        ``bool``
+Default     ``false``
+Description Like `image-unique`__, but applies to delegated URLs
+            like manga-chapters, etc.
+=========== =====
+
+__ `extractor.*.image-unique`_
+
+
+extractor.*.date-format
+----------------------------
+=========== =====
+Type        ``string``
+Default     ``"%Y-%m-%dT%H:%M:%S"``
+Description Format string used to parse ``string`` values of
+            `date-min` and `date-max`.
+
+            See |strptime|_ for a list of formatting directives.
+=========== =====
+
+
 
 Extractor-specific Options
 ==========================
@@ -737,24 +788,9 @@ Description Retrieve additional comments by resolving the ``more`` comment
 extractor.reddit.date-min & .date-max
 -------------------------------------
 =========== =====
-Type        ``integer`` or ``string``
+Type        |Date|_
 Default     ``0`` and ``253402210800`` (timestamp of |datetime.max|_)
 Description Ignore all submissions posted before/after this date.
-
-            * If this is an ``integer``, it represents the date as UTC timestamp.
-            * If this is a ``string``, it will get parsed according to date-format_.
-=========== =====
-
-
-extractor.reddit.date-format
-----------------------------
-=========== =====
-Type        ``string``
-Default     ``"%Y-%m-%dT%H:%M:%S"``
-Description An explicit format string used to parse the ``string`` values of
-            `date-min and date-max`_.
-
-            See |strptime|_ for a list of formatting directives.
 =========== =====
 
 
@@ -831,6 +867,15 @@ Description Download blog avatars.
 =========== =====
 
 
+extractor.tumblr.date-min & .date-max
+-------------------------------------
+=========== =====
+Type        |Date|_
+Default     ``0`` and ``null``
+Description Ignore all posts published before/after this date.
+=========== =====
+
+
 extractor.tumblr.external
 -------------------------
 =========== =====
@@ -877,6 +922,15 @@ Description A (comma-separated) list of post types to extract images, etc. from.
 =========== =====
 
 
+extractor.twitter.content
+-------------------------
+=========== =====
+Type        ``bool``
+Default     ``false``
+Description Extract tweet text as ``content`` metadata.
+=========== =====
+
+
 extractor.twitter.retweets
 --------------------------
 =========== =====
@@ -945,6 +999,16 @@ Description Enable/Disable this downloader module.
 =========== =====
 
 
+downloader.*.mtime
+------------------
+=========== =====
+Type        ``bool``
+Default     ``true``
+Description Use |Last-Modified|_ HTTP response headers
+            to set file modification times.
+=========== =====
+
+
 downloader.*.part
 -----------------
 =========== =====
@@ -992,7 +1056,8 @@ downloader.*.retries
 =========== =====
 Type        ``integer``
 Default     `extractor.*.retries`_
-Description Number of retries during file downloads.
+Description Maximum number of retries during file downloads
+            or ``-1`` for infinite retries.
 =========== =====
 
 
@@ -1240,6 +1305,23 @@ Description Custom format string to build content of metadata files.
 =========== =====
 
 
+mtime
+-----
+
+Set file modification time according to its metadata
+
+mtime.key
+---------
+=========== =====
+Type        ``string``
+Default     ``"date"``
+Description Name of the metadata field whose value should be used.
+
+            This value must either be a UNIX timestamp or a
+            |datetime|_ object.
+=========== =====
+
+
 ugoira
 ------
 
@@ -1375,6 +1457,19 @@ Description Path of the SQLite3 database used to cache login sessions,
 =========== =====
 
 
+ciphers
+-------
+=========== =====
+Type        ``bool`` or ``string``
+Default     ``true``
+Description * ``true``: Update urllib3's default cipher list
+            * ``false``: Leave the default cipher list as is
+            * Any ``string``: Replace urllib3's default ciphers with these
+              (See `SSLContext.set_ciphers() <https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_ciphers>`__
+              for details)
+=========== =====
+
+
 
 API Tokens & IDs
 ================
@@ -1479,6 +1574,20 @@ Custom Types
 ============
 
 
+Date
+----
+=========== =====
+Type        ``string`` or ``integer``
+Examples    * ``"2019-01-01T00:00:00"``
+            * ``"2019"`` with ``"%Y"`` as date-format_
+            * ``1546297200``
+Description A |Date|_ value represents a specific point in time.
+
+            * If given as ``string``, it is parsed according to date-format_.
+            * If given as ``integer``, it is interpreted as UTC timestamp.
+=========== =====
+
+
 Path
 ----
 =========== =====
@@ -1508,7 +1617,7 @@ Logging Configuration
 =========== =====
 Type        ``object``
 
-Example     .. code::
+Examples    .. code::
 
                 {
                     "format": "{asctime} {name}: {message}",
@@ -1517,10 +1626,21 @@ Example     .. code::
                     "encoding": "ascii"
                 }
 
+                {
+                    "level": "debug",
+                    "format": {
+                        "debug"  : "debug: {message}",
+                        "info"   : "[{name}] {message}",
+                        "warning": "Warning: {message}",
+                        "error"  : "ERROR: {message}"
+                    }
+                }
+
 Description Extended logging output configuration.
 
             * format
-                * Format string for logging messages
+                * General format string for logging messages
+                  or a dictionary with format strings for each loglevel.
 
                   In addition to the default
                   `LogRecord attributes <https://docs.python.org/3/library/logging.html#logrecord-attributes>`__,
@@ -1587,16 +1707,18 @@ Description An object with the ``name`` of a post-processor and its options.
 .. |verify| replace:: ``verify``
 .. |mature_content| replace:: ``mature_content``
 .. |webbrowser.open()| replace:: ``webbrowser.open()``
+.. |datetime| replace:: ``datetime``
 .. |datetime.max| replace:: ``datetime.max``
+.. |Date| replace:: ``Date``
 .. |Path| replace:: ``Path``
+.. |Last-Modified| replace:: ``Last-Modified``
 .. |Logging Configuration| replace:: ``Logging Configuration``
 .. |Postprocessor Configuration| replace:: ``Postprocessor Configuration``
 .. |strptime| replace:: strftime() and strptime() Behavior
 
 .. _base-directory: `extractor.*.base-directory`_
 .. _skipped: `extractor.*.skip`_
-.. _`date-min and date-max`: `extractor.reddit.date-min & .date-max`_
-.. _date-format: extractor.reddit.date-format_
+.. _date-format: `extractor.*.date-format`_
 .. _deviantart.metadata: extractor.deviantart.metadata_
 
 .. _.netrc:            https://stackoverflow.com/tags/.netrc/info
@@ -1604,12 +1726,14 @@ Description An object with the ``name`` of a post-processor and its options.
 .. _requests.request(): https://docs.python-requests.org/en/master/api/#requests.request
 .. _timeout:           https://docs.python-requests.org/en/latest/user/advanced/#timeouts
 .. _verify:            https://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification
+.. _Last-Modified:     https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.29
 .. _`Requests' proxy documentation`: http://docs.python-requests.org/en/master/user/advanced/#proxies
 .. _format string:     https://docs.python.org/3/library/string.html#formatstrings
 .. _format strings:    https://docs.python.org/3/library/string.html#formatstrings
 .. _strptime:          https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior
 .. _mature_content:    https://www.deviantart.com/developers/http/v1/20160316/object/deviation
 .. _webbrowser.open(): https://docs.python.org/3/library/webbrowser.html
+.. _datetime:          https://docs.python.org/3/library/datetime.html#datetime-objects
 .. _datetime.max:      https://docs.python.org/3/library/datetime.html#datetime.datetime.max
 .. _Authentication:    https://github.com/mikf/gallery-dl#5authentication
 .. _youtube-dl:        https://github.com/ytdl-org/youtube-dl
diff --git a/docs/gallery-dl-example.conf b/docs/gallery-dl-example.conf
index a5270d2..04be5e6 100644
--- a/docs/gallery-dl-example.conf
+++ b/docs/gallery-dl-example.conf
@@ -148,8 +148,13 @@
     {
         "mode": "terminal",
         "log": {
-            "format": "{name}: {message}",
-            "level": "info"
+            "level": "info",
+            "format": {
+                "debug"  : "\u001b[0;37m{name}: {message}\u001b[0m",
+                "info"   : "\u001b[1;37m{name}: {message}\u001b[0m",
+                "warning": "\u001b[1;33m{name}: {message}\u001b[0m",
+                "error"  : "\u001b[1;31m{name}: {message}\u001b[0m"
+            }
         },
         "logfile": {
             "path": "~/gallery-dl/log.txt",
diff --git a/docs/gallery-dl.conf b/docs/gallery-dl.conf
index c792e9e..835ed17 100644
--- a/docs/gallery-dl.conf
+++ b/docs/gallery-dl.conf
@@ -8,7 +8,7 @@
         "proxy": null,
         "skip": true,
         "sleep": 0,
-        "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0",
+        "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0",
 
         "artstation":
         {
@@ -132,6 +132,7 @@
         },
         "twitter":
         {
+            "content": false,
             "retweets": true,
             "videos": false
         },
@@ -152,8 +153,19 @@
 
         "http":
         {
+            "mtime": true,
             "rate": null,
-            "retries": 5,
+            "retries": 4,
+            "timeout": 30.0,
+            "verify": true
+        },
+
+        "ytdl":
+        {
+            "format": null,
+            "mtime": true,
+            "rate": null,
+            "retries": 4,
             "timeout": 30.0,
             "verify": true
         }
@@ -164,6 +176,7 @@
         "mode": "auto",
         "progress": true,
         "shorten": true,
+        "log": "[{name}][{levelname}] {message}",
         "logfile": null,
         "unsupportedfile": null
     },
diff --git a/docs/supportedsites.rst b/docs/supportedsites.rst
index f47ed10..2a1a1ed 100644
--- a/docs/supportedsites.rst
+++ b/docs/supportedsites.rst
@@ -1,5 +1,7 @@
 Supported Sites
 ===============
+Unless otherwise known, assume all sites to be NSFW
+
 ==================== =================================== ================================================== ================
 Site                 URL                                 Capabilities                                       Authentication
 ==================== =================================== ================================================== ================
@@ -23,6 +25,7 @@ DeviantArt           https://www.deviantart.com/         |deviantart-C|
 Doki Reader          https://kobato.hologfx.com/reader/  Chapters, Manga
 Dynasty Reader       https://dynasty-scans.com/          Chapters, individual Images, Search Results
 e621                 https://e621.net/                   Pools, Popular Images, Posts, Tag-Searches
+EroLord.com          http://erolord.com/                 Galleries
 ExHentai             https://exhentai.org/               Favorites, Galleries, Search Results               Optional
 Fallen Angels Scans  https://www.fascans.com/            Chapters, Manga
 Fashion Nova         https://www.fashionnova.com/        Collections, Products
@@ -46,7 +49,7 @@ ImageFap             https://imagefap.com/               Images from Users, Gall
 imgbox               https://imgbox.com/                 Galleries, individual Images
 imgth                https://imgth.com/                  Galleries
 imgur                https://imgur.com/                  Albums, individual Images
-Instagram            https://www.instagram.com/          Images from Users, individual Images, Tag-Searches
+Instagram            https://www.instagram.com/          Images from Users, individual Images, Tag-Searches Optional
 Jaimini's Box        https://jaiminisbox.com/reader/     Chapters, Manga
 Joyreactor           http://joyreactor.cc/               |joyreactor-C|
 Keenspot             http://www.keenspot.com/            Comics
@@ -94,7 +97,7 @@ Sankaku Channel      https://chan.sankakucomplex.com/    Pools, Posts, Tag-Searc
 Sankaku Complex      https://www.sankakucomplex.com/     Articles, Tag-Searches
 Sen Manga            https://raw.senmanga.com/           Chapters
 Sense-Scans          http://sensescans.com/reader/       Chapters, Manga
-Sex.com              https://www.sex.com/                Boards, Pins, Search Results
+Sex.com              https://www.sex.com/                Boards, Pins, related Pins, Search Results
 Simply Hentai        https://www.simply-hentai.com/      Galleries, individual Images, Videos
 SlickPic             https://www.slickpic.com/           Images from Users, Albums
 SlideShare           https://www.slideshare.net/         Presentations