diff options
Diffstat (limited to 'docs/configuration.rst')
| -rw-r--r-- | docs/configuration.rst | 1615 |
1 files changed, 1615 insertions, 0 deletions
diff --git a/docs/configuration.rst b/docs/configuration.rst new file mode 100644 index 0000000..c606c6c --- /dev/null +++ b/docs/configuration.rst @@ -0,0 +1,1615 @@ +Configuration +############# + +Contents +======== + +1) `Extractor Options`_ +2) `Extractor-specific Options`_ +3) `Downloader Options`_ +4) `Output Options`_ +5) `Postprocessor Options`_ +6) `Miscellaneous Options`_ +7) `API Tokens & IDs`_ + + + +Extractor Options +================= + + +Each extractor is identified by its ``category`` and ``subcategory``. +The ``category`` is the lowercase site name without any spaces or special +characters, which is usually just the module name +(``pixiv``, ``danbooru``, ...). +The ``subcategory`` is a lowercase word describing the general functionality +of that extractor (``user``, ``favorite``, ``manga``, ...). + +Each one of the following options can be specified on multiple levels of the +configuration tree: + +================== ===== +Base level: ``extractor.<option-name>`` +Category level: ``extractor.<category>.<option-name>`` +Subcategory level: ``extractor.<category>.<subcategory>.<option-name>`` +================== ===== + +A value in a "deeper" level hereby overrides a value of the same name on a +lower level. Setting the ``extractor.pixiv.filename`` value, for example, lets +you specify a general filename pattern for all the different pixiv extractors. +Using the ``extractor.pixiv.user.filename`` value lets you override this +general pattern specifically for ``PixivUserExtractor`` instances. + +The ``category`` and ``subcategory`` of all extractors are included in the +output of ``gallery-dl --list-extractors``. For a specific URL these values +can also be determined by using the ``-K``/``--list-keywords`` command-line +option (see the example below). + +extractor.*.filename +-------------------- +=========== ===== +Type ``string`` +Example ``"{manga}_c{chapter}_{page:>03}.{extension}"`` +Description A `format string`_ to build the resulting filename + for a downloaded file. + + The available replacement keys depend on the extractor used. A list + of keys for a specific one can be acquired by calling *gallery-dl* + with the ``-K``/``--list-keywords`` command-line option. + For example: + + .. code:: + + $ gallery-dl -K http://seiga.nicovideo.jp/seiga/im5977527 + Keywords for directory names: + ----------------------------- + category + seiga + subcategory + image + + Keywords for filenames: + ----------------------- + category + seiga + extension + None + image-id + 5977527 + subcategory + image + + Note: Even if the value of the ``extension`` key is missing or + ``None``, it will filled in later when the file download is + starting. This key is therefore always available to provide + a valid filename extension. +=========== ===== + + +extractor.*.directory +--------------------- +=========== ===== +Type ``list`` of ``strings`` +Example ``["{category}", "{manga}", "c{chapter} - {title}"]`` +Description A list of `format strings`_ for the resulting target directory. + + Each individual string in such a list represents a single path + segment, which will be joined together and appended to the + base-directory_ to form the complete target directory path. +=========== ===== + + +extractor.*.base-directory +-------------------------- +=========== ===== +Type |Path|_ +Default ``"./gallery-dl/"`` +Description Directory path used as the base for all download destinations. +=========== ===== + + +extractor.*.skip +---------------- +=========== ===== +Type ``bool`` or ``string`` +Default ``true`` +Description Controls the behavior when downloading files whose filename + already exists. + + * ``true``: Skip downloads + * ``false``: Overwrite already existing files + + * ``"abort"``: Abort the current extractor run + * ``"abort:N"``: Skip downloads and abort extractor run + after ``N`` consecutive skips + + * ``"exit"``: Exit the program altogether + * ``"exit:N"``: Skip downloads and exit the program + after ``N`` consecutive skips +=========== ===== + + +extractor.*.sleep +----------------- +=========== ===== +Type ``float`` +Default ``0`` +Description Number of seconds to sleep before each download. +=========== ===== + + +extractor.*.username & .password +-------------------------------- +=========== ===== +Type ``string`` +Default ``null`` +Description The username and password to use when attempting to log in to + another site. + + Specifying username and password is + required for the ``pixiv``, ``nijie`` and ``seiga`` modules and + optional (but strongly recommended) for ``danbooru``, ``exhentai``, + ``sankaku`` and ``idolcomplex``. + + These values can also be set via the ``-u/--username`` and + ``-p/--password`` command-line options or by using a |.netrc|_ file. + (see Authentication_) + + Note: The password for ``danbooru`` is the API key found in your + user profile, not the password for your account. +=========== ===== + + +extractor.*.netrc +----------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Enable the use of |.netrc|_ authentication data. +=========== ===== + + +extractor.*.cookies +------------------- +=========== ===== +Type |Path|_ or ``object`` +Default ``null`` +Description Source to read additional cookies from. + + * If this is a |Path|_, it specifies a + Mozilla/Netscape format cookies.txt file. + * If this is an ``object``, its key-value pairs, which should both + be ``strings``, will be used as cookie-names and -values. +=========== ===== + + +extractor.*.proxy +----------------- +=========== ===== +Type ``string`` or ``object`` +Default ``null`` +Description Proxy (or proxies) to be used for remote connections. + + * If this is a ``string``, it is the proxy URL for all + outgoing requests. + * If this is an ``object``, it is a scheme-to-proxy mapping to + specify different proxy URLs for each scheme. + It is also possible to set a proxy for a specific host by using + ``scheme://host`` as key. + See `Requests' proxy documentation`_ for more details. + + Example: + + .. code:: + + { + "http": "http://10.10.1.10:3128", + "https": "http://10.10.1.10:1080", + "http://10.20.1.128": "http://10.10.1.10:5323" + } + + Note: All proxy URLs should include a scheme, + otherwise ``http://`` is assumed. +=========== ===== + + +extractor.*.user-agent +---------------------- +=========== ===== +Type ``string`` +Default ``"Mozilla/5.0 (X11; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"`` +Description User-Agent header value to be used for HTTP requests. + + Note: This option has no effect on `pixiv` and + `readcomiconline` extractors, as these need specific values to + function correctly. +=========== ===== + + +extractor.*.keywords +-------------------- +=========== ===== +Type ``object`` +Example ``{"type": "Pixel Art", "type_id": 123}`` +Description Additional key-value pairs to be added to each metadata dictionary. +=========== ===== + + +extractor.*.keywords-default +---------------------------- +=========== ===== +Type any +Default ``"None"`` +Description Default value used for missing or undefined keyword names in + format strings. +=========== ===== + + +extractor.*.category-transfer +----------------------------- +=========== ===== +Type ``bool`` +Default Extractor-specific +Description Transfer an extractor's (sub)category values to all child + extractors spawned by it, to let them inherit their parent's + config options. +=========== ===== + + +extractor.*.archive +------------------- +=========== ===== +Type |Path|_ +Default ``null`` +Description File to store IDs of downloaded files in. Downloads of files + already recorded in this archive file will be skipped_. + + The resulting archive file is not a plain text file but an SQLite3 + database, as either lookup operations are significantly faster or + memory requirements are significantly lower when the + amount of stored IDs gets reasonably large. +=========== ===== + + +extractor.*.archive-format +-------------------------- +=========== ===== +Type ``string`` +Example ``"{id}_{offset}"`` +Description An alternative `format string`_ to build archive IDs with. +=========== ===== + + +extractor.*.postprocessors +-------------------------- +=========== ===== +Type ``list`` of |Postprocessor Configuration|_ objects +Example .. code:: + + [ + {"name": "zip", "compression": "zip"}, + {"name": "exec", "command": ["/home/foobar/script", "{category}", "{image_id}"]} + ] + +Description A list of post-processors to be applied to each downloaded file + in the same order as they are specified. +=========== ===== + + +extractor.*.retries +------------------- +=========== ===== +Type ``integer`` +Default ``5`` +Description Number of times a failed HTTP request is retried before giving up. +=========== ===== + + +extractor.*.timeout +------------------- +=========== ===== +Type ``float`` or ``null`` +Default ``30`` +Description Amount of time (in seconds) to wait for a successful connection + and response from a remote server. + + This value gets internally used as the |timeout|_ parameter for the + |requests.request()|_ method. +=========== ===== + + +extractor.*.verify +------------------ +=========== ===== +Type ``bool`` or ``string`` +Default ``true`` +Description Controls whether to verify SSL/TLS certificates for HTTPS requests. + + If this is a ``string``, it must be the path to a CA bundle to use + instead of the default certificates. + + This value gets internally used as the |verify|_ parameter for the + |requests.request()|_ method. +=========== ===== + + +extractor.*.image-range +----------------------- +=========== ===== +Type ``string`` +Example | ``"10-20"``, + | ``"-5, 10, 30-50, 100-"`` +Description Index-range(s) specifying which images to download. + + Note: The index of the first image is ``1``. +=========== ===== + + +extractor.*.chapter-range +------------------------- +=========== ===== +Type ``string`` +Description Like `image-range`__, but applies to delegated URLs + like manga-chapters, etc. +=========== ===== + +__ `extractor.*.image-range`_ + + +extractor.*.image-filter +------------------------ +=========== ===== +Type ``string`` +Example | ``"width >= 1200 and width/height > 1.2"``, + | ``"re.search(r'foo(bar)+', description)"`` +Description | Python expression controlling which images to download. + | Files for which the expression evaluates to ``False`` + are ignored. + | Available keys are the filename-specific ones listed + by ``-K`` or ``-j``. +=========== ===== + + +extractor.*.chapter-filter +-------------------------- +=========== ===== +Type ``string`` +Description Like `image-filter`__, but applies to delegated URLs + like manga-chapters, etc. +=========== ===== + +__ `extractor.*.image-filter`_ + + + +Extractor-specific Options +========================== + + +extractor.artstation.external +----------------------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Try to follow external URLs of embedded players. +=========== ===== + + +extractor.deviantart.extra +-------------------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Download extra Sta.sh resources from description texts. + + Note: Enabling this option also enables deviantart.metadata_. +=========== ===== + + +extractor.deviantart.flat +------------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Select the directory structure created by the Gallery- and + Favorite-Extractors. + + * ``true``: Use a flat directory structure. + * ``false``: Collect a list of all gallery-folders or + favorites-collections and transfer any further work to other + extractors (``folder`` or ``collection``), which will then + create individual subdirectories for each of them. +=========== ===== + + +extractor.deviantart.folders +---------------------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Provide a ``folders`` metadata field that contains the names of all + folders a deviation is present in. + + Note: Gathering this information requires a lot of API calls. + Use with caution. +=========== ===== + + +extractor.deviantart.journals +----------------------------- +=========== ===== +Type ``string`` +Default ``"html"`` +Description Selects the output format of journal entries. + + * ``"html"``: HTML with (roughly) the same layout as on DeviantArt. + * ``"text"``: Plain text with image references and HTML tags removed. + * ``"none"``: Don't download journals. +=========== ===== + + +extractor.deviantart.mature +--------------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Enable mature content. + + This option simply sets the |mature_content|_ parameter for API + calls to either ``"true"`` or ``"false"`` and does not do any other + form of content filtering. +=========== ===== + + +extractor.deviantart.metadata +----------------------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Request extended metadata for deviation objects to additionally + provide ``description``, ``tags``, ``license`` and ``is_watching`` + fields. +=========== ===== + + +extractor.deviantart.original +----------------------------- +=========== ===== +Type ``bool`` or ``string`` +Default ``true`` +Description Download original files if available. + + Setting this option to ``"images"`` only downloads original + files if they are images and falls back to preview versions for + everything else (archives, etc.). +=========== ===== + + +extractor.deviantart.refresh-token +---------------------------------- +=========== ===== +Type ``string`` +Default ``null`` +Description The ``refresh_token`` value you get from linking your + DeviantArt account to *gallery-dl*. + + Using a ``refresh_token`` allows you to access private or otherwise + not publicly available deviations. +=========== ===== + + +extractor.deviantart.wait-min +----------------------------- +=========== ===== +Type ``integer`` +Default ``0`` +Description Minimum wait time in seconds before API requests. + + Note: This value will internally be rounded up + to the next power of 2. +=========== ===== + + +extractor.exhentai.limits +------------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Check image download limits + and stop extraction when they are exceeded. +=========== ===== + + +extractor.exhentai.original +--------------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Download full-sized original images if available. +=========== ===== + + +extractor.exhentai.wait-min & .wait-max +--------------------------------------- +=========== ===== +Type ``float`` +Default ``3.0`` and ``6.0`` +Description Minimum and maximum wait time in seconds between each image + + ExHentai detects and blocks automated downloaders. + *gallery-dl* waits a randomly selected number of + seconds between ``wait-min`` and ``wait-max`` after + each image to prevent getting blocked. +=========== ===== + + +extractor.flickr.access-token & .access-token-secret +---------------------------------------------------- +=========== ===== +Type ``string`` +Default ``null`` +Description The ``access_token`` and ``access_token_secret`` values you get + from linking your Flickr account to *gallery-dl*. +=========== ===== + + +extractor.flickr.videos +----------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Extract and download videos. +=========== ===== + + +extractor.flickr.size-max +-------------------------- +=========== ===== +Type ``integer`` or ``string`` +Default ``null`` +Description Sets the maximum allowed size for downloaded images. + + * If this is an ``integer``, it specifies the maximum image dimension + (width and height) in pixels. + * If this is a ``string``, it should be one of Flickr's format specifiers + (``"Original"``, ``"Large"``, ... or ``"o"``, ``"k"``, ``"h"``, + ``"l"``, ...) to use as an upper limit. +=========== ===== + + +extractor.gelbooru.api +---------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Enable use of Gelbooru's API. + + Set this value to `false` if the API has been disabled to switch + to manual information extraction. +=========== ===== + + +extractor.gfycat.format +----------------------- +=========== ===== +Type ``string`` +Default ``"mp4"`` +Description The name of the preferred animation format, which can be one of + ``"mp4"``, ``"webm"``, ``"gif"``, ``"webp"`` or ``"mjpg"``. + + If the selected format is not available, ``"mp4"``, ``"webm"`` + and ``"gif"`` (in that order) will be tried instead, until an + available format is found. +=========== ===== + + +extractor.imgur.mp4 +------------------- +=========== ===== +Type ``bool`` or ``string`` +Default ``true`` +Description Controls whether to choose the GIF or MP4 version of an animation. + + * ``true``: Follow Imgur's advice and choose MP4 if the + ``prefer_video`` flag in an image's metadata is set. + * ``false``: Always choose GIF. + * ``"always"``: Always choose MP4. +=========== ===== + + +extractor.kissmanga.captcha +--------------------------- +=========== ===== +Type ``string`` +Default ``"stop"`` +Description Controls how to handle redirects to CAPTCHA pages. + + * ``"stop``: Stop the current extractor run. + * ``"wait``: Ask the user to solve the CAPTCHA and wait. +=========== ===== + + +extractor.oauth.browser +----------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Controls how a user is directed to an OAuth authorization site. + + * ``true``: Use Python's |webbrowser.open()|_ method to automatically + open the URL in the user's browser. + * ``false``: Ask the user to copy & paste an URL from the terminal. +=========== ===== + + +extractor.photobucket.subalbums +------------------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Download subalbums. +=========== ===== + + +extractor.pixiv.ugoira +---------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Download Pixiv's Ugoira animations or ignore them. + + These animations come as a ``.zip`` file containing all the single + animation frames in JPEG format. +=========== ===== + + +extractor.plurk.comments +------------------------ +=========== ===== +Type ``bool`` +Default ``false`` +Description Also search Plurk comments for URLs. +=========== ===== + + +extractor.reactor.wait-min & .wait-max +-------------------------------------- +=========== ===== +Type ``float`` +Default ``3.0`` and ``6.0`` +Description Minimum and maximum wait time in seconds between HTTP requests + during the extraction process. +=========== ===== + + +extractor.readcomiconline.captcha +--------------------------------- +=========== ===== +Type ``string`` +Default ``"stop"`` +Description Controls how to handle redirects to CAPTCHA pages. + + * ``"stop``: Stop the current extractor run. + * ``"wait``: Ask the user to solve the CAPTCHA and wait. +=========== ===== + + +extractor.recursive.blacklist +----------------------------- +=========== ===== +Type ``list`` of ``strings`` +Default ``["directlink", "oauth", "recursive", "test"]`` +Description A list of extractor categories which should be ignored when using + the ``recursive`` extractor. +=========== ===== + + +extractor.reddit.comments +------------------------- +=========== ===== +Type ``integer`` or ``string`` +Default ``500`` +Description The value of the ``limit`` parameter when loading + a submission and its comments. + This number (roughly) specifies the total amount of comments + being retrieved with the first API call. + + Reddit's internal default and maximum values for this parameter + appear to be 200 and 500 respectively. + + The value `0` ignores all comments and significantly reduces the + time required when scanning a subreddit. +=========== ===== + + +extractor.reddit.morecomments +----------------------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Retrieve additional comments by resolving the ``more`` comment + stubs in the base comment tree. + + This requires 1 additional API call for every 100 extra comments. +=========== ===== + + +extractor.reddit.date-min & .date-max +------------------------------------- +=========== ===== +Type ``integer`` or ``string`` +Default ``0`` and ``253402210800`` (timestamp of |datetime.max|_) +Description Ignore all submissions posted before/after this date. + + * If this is an ``integer``, it represents the date as UTC timestamp. + * If this is a ``string``, it will get parsed according to date-format_. +=========== ===== + + +extractor.reddit.date-format +---------------------------- +=========== ===== +Type ``string`` +Default ``"%Y-%m-%dT%H:%M:%S"`` +Description An explicit format string used to parse the ``string`` values of + `date-min and date-max`_. + + See |strptime|_ for a list of formatting directives. +=========== ===== + + +extractor.reddit.id-min & .id-max +--------------------------------- +=========== ===== +Type ``string`` +Example ``"6kmzv2"`` +Description Ignore all submissions posted before/after the submission with + this ID. +=========== ===== + + +extractor.reddit.recursion +-------------------------- +=========== ===== +Type ``integer`` +Default ``0`` +Description Reddit extractors can recursively visit other submissions + linked to in the initial set of submissions. + This value sets the maximum recursion depth. + + Special values: + + * ``0``: Recursion is disabled + * ``-1``: Infinite recursion (don't do this) +=========== ===== + + +extractor.reddit.refresh-token +------------------------------ +=========== ===== +Type ``string`` +Default ``null`` +Description The ``refresh_token`` value you get from linking your + Reddit account to *gallery-dl*. + + Using a ``refresh_token`` allows you to access private or otherwise + not publicly available subreddits, given that your account is + authorized to do so, + but requests to the reddit API are going to be rate limited + at 600 requests every 10 minutes/600 seconds. +=========== ===== + + +extractor.sankaku.wait-min & .wait-max +-------------------------------------- +=========== ===== +Type ``float`` +Default ``3.0`` and ``6.0`` +Description Minimum and maximum wait time in seconds between each image + + Sankaku Channel responds with ``429 Too Many Requests`` if it + receives too many HTTP requests in a certain amount of time. + Waiting a few seconds between each request tries to prevent that. +=========== ===== + + +extractor.smugmug.videos +------------------------ +=========== ===== +Type ``bool`` +Default ``true`` +Description Download video files. +=========== ===== + + +extractor.tumblr.avatar +----------------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Download blog avatars. +=========== ===== + + +extractor.tumblr.external +------------------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Follow external URLs (e.g. from "Link" posts) and try to extract + images from them. +=========== ===== + + +extractor.tumblr.inline +----------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Search posts for inline images and videos. +=========== ===== + + +extractor.tumblr.reblogs +------------------------ +=========== ===== +Type ``bool`` or ``string`` +Default ``true`` +Description * ``true``: Extract media from reblogged posts + * ``false``: Skip reblogged posts + * ``"same-blog"``: Skip reblogged posts unless the original post + is from the same blog +=========== ===== + + +extractor.tumblr.posts +---------------------- +=========== ===== +Type ``string`` or ``list`` of ``strings`` +Default ``"all"`` +Example ``"video,audio,link"`` or ``["video", "audio", "link"]`` +Description A (comma-separated) list of post types to extract images, etc. from. + + Possible types are ``text``, ``quote``, ``link``, ``answer``, + ``video``, ``audio``, ``photo``, ``chat``. + + You can use ``"all"`` instead of listing all types separately. +=========== ===== + + +extractor.twitter.retweets +-------------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Extract images from retweets. +=========== ===== + + +extractor.twitter.videos +------------------------ +=========== ===== +Type ``bool`` +Default ``false`` +Description Use `youtube-dl`_ to download from video tweets. +=========== ===== + + +extractor.wallhaven.api-key +--------------------------- +=========== ===== +Type ``string`` +Default ``null`` +Description Your `API Key <https://wallhaven.cc/settings/account>`__ to use + your account's browsing settings and default filters when searching. + + See https://wallhaven.cc/help/api for more information. +=========== ===== + + +extractor.[booru].tags +---------------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Categorize tags by their respective types + and provide them as ``tags_<type>`` metadata fields. + + Note: This requires 1 additional HTTP request for each post. +=========== ===== + + +extractor.[manga-extractor].chapter-reverse +------------------------------------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Reverse the order of chapter URLs extracted from manga pages. + + * ``true``: Start with the latest chapter + * ``false``: Start with the first chapter +=========== ===== + + + +Downloader Options +================== + + +downloader.*.enabled +-------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Enable/Disable this downloader module. +=========== ===== + + +downloader.*.part +----------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Controls the use of ``.part`` files during file downloads. + + * ``true``: Write downloaded data into ``.part`` files and rename + them upon download completion. This mode additionally supports + resuming incomplete downloads. + * ``false``: Do not use ``.part`` files and write data directly + into the actual output files. +=========== ===== + + +downloader.*.part-directory +--------------------------- +=========== ===== +Type |Path|_ +Default ``null`` +Description Alternate location for ``.part`` files. + + Missing directories will be created as needed. + If this value is ``null``, ``.part`` files are going to be stored + alongside the actual output files. +=========== ===== + + +downloader.*.rate +----------------- +=========== ===== +Type ``string`` +Default ``null`` +Examples ``"32000"``, ``"500k"``, ``"2.5M"`` +Description Maximum download rate in bytes per second. + + Possible values are valid integer or floating-point numbers + optionally followed by one of ``k``, ``m``. ``g``, ``t`` or ``p``. + These suffixes are case-insensitive. +=========== ===== + + +downloader.*.retries +-------------------- +=========== ===== +Type ``integer`` +Default `extractor.*.retries`_ +Description Number of retries during file downloads. +=========== ===== + + +downloader.*.timeout +-------------------- +=========== ===== +Type ``float`` or ``null`` +Default `extractor.*.timeout`_ +Description Connection timeout during file downloads. +=========== ===== + + +downloader.*.verify +------------------- +=========== ===== +Type ``bool`` or ``string`` +Default `extractor.*.verify`_ +Description Certificate validation during file downloads. +=========== ===== + + +downloader.ytdl.format +---------------------- +=========== ===== +Type ``string`` +Default youtube-dl's default, currently ``"bestvideo+bestaudio/best"`` +Description Video `format selection + <https://github.com/ytdl-org/youtube-dl#format-selection>`__ + directly passed to youtube-dl. +=========== ===== + + +downloader.ytdl.logging +----------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description | Route youtube-dl's output through gallery-dl's logging system. + | Otherwise youtube-dl will write its output directly to stdout/stderr. + + Note: Set ``quiet`` and ``no_warnings`` in + `downloader.ytdl.raw-options`_ to ``true`` to suppress all output. +=========== ===== + + +downloader.ytdl.raw-options +--------------------------- +=========== ===== +Type ``object`` +Example .. code:: + + { + "quiet": true, + "writesubtitles": true, + "merge_output_format": "mkv" + } + +Description | Additional options passed directly to the ``YoutubeDL`` constructor. + | All available options can be found in `youtube-dl's docstrings + <https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/YoutubeDL.py#L138-L318>`__. +=========== ===== + + + +Output Options +============== + + +output.mode +----------- +=========== ===== +Type ``string`` +Default ``"auto"`` +Description Controls the output string format and status indicators. + + * ``"null"``: No output + * ``"pipe"``: Suitable for piping to other processes or files + * ``"terminal"``: Suitable for the standard Windows console + * ``"color"``: Suitable for terminals that understand ANSI escape codes and colors + * ``"auto"``: Automatically choose the best suitable output mode +=========== ===== + + +output.shorten +-------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Controls whether the output strings should be shortened to fit + on one console line. +=========== ===== + + +output.progress +--------------- +=========== ===== +Type ``bool`` or ``string`` +Default ``true`` +Description Controls the progress indicator when *gallery-dl* is run with + multiple URLs as arguments. + + * ``true``: Show the default progress indicator + (``"[{current}/{total}] {url}"``) + * ``false``: Do not show any progress indicator + * Any ``string``: Show the progress indicator using this + as a custom `format string`_. Possible replacement keys are + ``current``, ``total`` and ``url``. +=========== ===== + + +output.log +---------- +=========== ===== +Type ``string`` or |Logging Configuration|_ +Default ``"[{name}][{levelname}] {message}"`` +Description Configuration for standard logging output to stderr. + + If this is a simple ``string``, it specifies + the format string for logging messages. +=========== ===== + + +output.logfile +-------------- +=========== ===== +Type |Path|_ or |Logging Configuration|_ +Default ``null`` +Description File to write logging output to. +=========== ===== + + +output.unsupportedfile +---------------------- +=========== ===== +Type |Path|_ or |Logging Configuration|_ +Default ``null`` +Description File to write external URLs unsupported by *gallery-dl* to. + + The default format string here is ``"{message}"``. +=========== ===== + + +output.num-to-str +----------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Convert numeric values (``integer`` or ``float``) to ``string`` + before outputting them as JSON. +=========== ===== + + + +Postprocessor Options +===================== + + +classify +-------- + +Categorize files by filename extension + +classify.mapping +---------------- +=========== ===== +Type ``object`` +Default .. code:: + + { + "Pictures" : ["jpg", "jpeg", "png", "gif", "bmp", "svg", "webp"], + "Video" : ["flv", "ogv", "avi", "mp4", "mpg", "mpeg", "3gp", "mkv", "webm", "vob", "wmv"], + "Music" : ["mp3", "aac", "flac", "ogg", "wma", "m4a", "wav"], + "Archives" : ["zip", "rar", "7z", "tar", "gz", "bz2"] + } + +Description A mapping from directory names to filename extensions that should + be stored in them. + + Files with an extension not listed will be ignored and stored + in their default location. +=========== ===== + + +exec +---- + +Execute external commands. + +exec.async +---------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Controls whether to wait for a subprocess to finish + or to let it run asynchronously. +=========== ===== + +exec.command +------------ +=========== ===== +Type ``list`` of ``strings`` +Example ``["echo", "{user[account]}", "{id}"]`` +Description The command to run. + + Each element of this list is treated as a `format string`_ using + the files' metadata. +=========== ===== + + +metadata +-------- + +Write image metadata to separate files + +metadata.mode +------------- +=========== ===== +Type ``string`` +Default ``"json"`` +Description Select how to write metadata. + + * ``"json"``: all metadata using `json.dump() + <https://docs.python.org/3/library/json.html#json.dump>`_ + * ``"tags"``: ``tags`` separated by newlines + * ``"custom"``: result of applying `metadata.format`_ to a file's + metadata dictionary +=========== ===== + +metadata.extension +------------------ +=========== ===== +Type ``string`` +Default ``"json"`` or ``"txt"`` +Description Filename extension for metadata files. +=========== ===== + +metadata.format +--------------- +=========== ===== +Type ``string`` +Example ``"tags:\n\n{tags:J\n}\n"`` +Description Custom format string to build content of metadata files. + + Note: Only applies for ``"mode": "custom"``. +=========== ===== + + +ugoira +------ + +Convert Pixiv Ugoira to WebM using `FFmpeg <https://www.ffmpeg.org/>`__. + +ugoira.extension +---------------- +=========== ===== +Type ``string`` +Default ``"webm"`` +Description Filename extension for the resulting video files. +=========== ===== + +ugoira.ffmpeg-args +------------------ +=========== ===== +Type ``list`` of ``strings`` +Default ``null`` +Example ``["-c:v", "libvpx-vp9", "-an", "-b:v", "2M"]`` +Description Additional FFmpeg command-line arguments. +=========== ===== + +ugoira.ffmpeg-location +---------------------- +=========== ===== +Type |Path|_ +Default ``"ffmpeg"`` +Description Location of the ``ffmpeg`` (or ``avconv``) executable to use. +=========== ===== + +ugoira.ffmpeg-output +-------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Show FFmpeg output. +=========== ===== + +ugoira.ffmpeg-twopass +--------------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Enable Two-Pass encoding. +=========== ===== + +ugoira.framerate +---------------- +=========== ===== +Type ``string`` +Default ``"auto"`` +Description Controls the frame rate argument (``-r``) for FFmpeg + + * ``"auto"``: Automatically assign a fitting frame rate + based on delays between frames. + * any other ``string``: Use this value as argument for ``-r``. + * ``null`` or an empty ``string``: Don't set an explicit frame rate. +=========== ===== + +ugoira.keep-files +----------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Keep ZIP archives after conversion. +=========== ===== + +ugoira.libx264-prevent-odd +-------------------------- +=========== ===== +Type ``bool`` +Default ``true`` +Description Prevent ``"width/height not divisible by 2"`` errors + when using ``libx264`` or ``libx265`` encoders + by applying a simple cropping filter. See this `Stack Overflow + thread <https://stackoverflow.com/questions/20847674>`__ + for more information. + + This option, when ``libx264/5`` is used, automatically + adds ``["-vf", "crop=iw-mod(iw\\,2):ih-mod(ih\\,2)"]`` + to the list of FFmpeg command-line arguments + to reduce an odd width/height by 1 pixel and make them even. +=========== ===== + + +zip +--- + +Store files in a ZIP archive. + +zip.compression +--------------- +=========== ===== +Type ``string`` +Default ``"store"`` +Description Compression method to use when writing the archive. + + Possible values are ``"store"``, ``"zip"``, ``"bzip2"``, ``"lzma"``. +=========== ===== + +zip.extension +------------- +=========== ===== +Type ``string`` +Default ``"zip"`` +Description Filename extension for the created ZIP archive. +=========== ===== + +zip.keep-files +-------------- +=========== ===== +Type ``bool`` +Default ``false`` +Description Keep the actual files after writing them to a ZIP archive. +=========== ===== + + + +Miscellaneous Options +===================== + + +cache.file +---------- +=========== ===== +Type |Path|_ +Default |tempfile.gettempdir()|_ + ``".gallery-dl.cache"`` +Description Path of the SQLite3 database used to cache login sessions, + cookies and API tokens across `gallery-dl` invocations. + + Set this option to ``null`` or an invalid path to disable + this cache. +=========== ===== + + + +API Tokens & IDs +================ + + +All configuration keys listed in this section have fully functional default +values embedded into *gallery-dl* itself, but if things unexpectedly break +or you want to use your own personal client credentials, you can follow these +instructions to get an alternative set of API tokens and IDs. + + +extractor.deviantart.client-id & .client-secret +----------------------------------------------- +=========== ===== +Type ``string`` +How To * login and visit DeviantArt's + `Applications & Keys <https://www.deviantart.com/developers/apps>`__ + section + * click "Register your Application" + * scroll to "OAuth2 Redirect URI Whitelist (Required)" + and enter "https://mikf.github.io/gallery-dl/oauth-redirect.html" + * click "Save" (top right) + * copy ``client_id`` and ``client_secret`` of your new + application and put them in your configuration file +=========== ===== + + +extractor.flickr.api-key & .api-secret +-------------------------------------- +=========== ===== +Type ``string`` +How To * login and `Create an App <https://www.flickr.com/services/apps/create/apply/>`__ + in Flickr's `App Garden <https://www.flickr.com/services/>`__ + * click "APPLY FOR A NON-COMMERCIAL KEY" + * fill out the form with a random name and description + and click "SUBMIT" + * copy ``Key`` and ``Secret`` and put them in your configuration + file +=========== ===== + + +extractor.pawoo.access-token +---------------------------- +=========== ===== +Type ``string`` +How To +=========== ===== + + +extractor.reddit.client-id & .user-agent +---------------------------------------- +=========== ===== +Type ``string`` +How To * login and visit the `apps <https://www.reddit.com/prefs/apps/>`__ + section of your account's preferences + * click the "are you a developer? create an app..." button + * fill out the form, choose "installed app", preferably set + "http://localhost:6414/" as "redirect uri" and finally click + "create app" + * copy the client id (third line, under your application's name and + "installed app") and put it in your configuration file + * use "``Python:<application name>:v1.0 (by /u/<username>)``" as + user-agent and replace ``<application name>`` and ``<username>`` + accordingly (see Reddit's + `API access rules <https://github.com/reddit/reddit/wiki/API>`__) +=========== ===== + + +extractor.smugmug.api-key & .api-secret +--------------------------------------- +=========== ===== +Type ``string`` +How To * login and `Apply for an API Key <https://api.smugmug.com/api/developer/apply>`__ + * use a random name and description, + set "Type" to "Application", "Platform" to "All", + and "Use" to "Non-Commercial" + * fill out the two checkboxes at the bottom and click "Apply" + * copy ``API Key`` and ``API Secret`` + and put them in your configuration file +=========== ===== + + +extractor.tumblr.api-key & .api-secret +-------------------------------------- +=========== ===== +Type ``string`` +How To * login and visit Tumblr's + `Applications <https://www.tumblr.com/oauth/apps>`__ section + * click "Register application" + * fill out the form: use a random name and description, set + https://example.org/ as "Application Website" and "Default + callback URL" + * solve Google's "I'm not a robot" challenge and click "Register" + * click "Show secret key" (below "OAuth Consumer Key") + * copy your ``OAuth Consumer Key`` and ``Secret Key`` + and put them in your configuration file +=========== ===== + + + +Custom Types +============ + + +Path +---- +=========== ===== +Type ``string`` or ``list`` of ``strings`` +Examples * ``"file.ext"`` + * ``"~/path/to/file.ext"`` + * ``"$HOME/path/to/file.ext"`` + * ``["$HOME", "path", "to", "file.ext"]`` +Description A |Path|_ is a ``string`` representing the location of a file + or directory. + + Simple `tilde expansion <https://docs.python.org/3/library/os.path.html#os.path.expanduser>`__ + and `environment variable expansion <https://docs.python.org/3/library/os.path.html#os.path.expandvars>`__ + is supported. + + In Windows environments, backslashes (``"\"``) can, in addition to + forward slashes (``"/"``), be used as path separators. + Because backslashes are JSON's escape character, + they themselves have to be escaped. + The path ``C:\path\to\file.ext`` has therefore to be written as + ``"C:\\path\\to\\file.ext"`` if you want to use backslashes. +=========== ===== + + +Logging Configuration +--------------------- +=========== ===== +Type ``object`` + +Example .. code:: + + { + "format": "{asctime} {name}: {message}", + "format-date": "%H:%M:%S", + "path": "~/log.txt", + "encoding": "ascii" + } + +Description Extended logging output configuration. + + * format + * Format string for logging messages + + In addition to the default + `LogRecord attributes <https://docs.python.org/3/library/logging.html#logrecord-attributes>`__, + it is also possible to access the current + `extractor <https://github.com/mikf/gallery-dl/blob/2e516a1e3e09cb8a9e36a8f6f7e41ce8d4402f5a/gallery_dl/extractor/common.py#L24>`__ + and `job <https://github.com/mikf/gallery-dl/blob/2e516a1e3e09cb8a9e36a8f6f7e41ce8d4402f5a/gallery_dl/job.py#L19>`__ + objects as well as their attributes + (e.g. ``"{extractor.url}"``) + * Default: ``"[{name}][{levelname}] {message}"`` + * format-date + * Format string for ``{asctime}`` fields in logging messages + (see `strftime() directives <https://docs.python.org/3/library/time.html#time.strftime>`__) + * Default: ``"%Y-%m-%d %H:%M:%S"`` + * level + * Minimum logging message level + (one of ``"debug"``, ``"info"``, ``"warning"``, ``"error"``, ``"exception"``) + * Default: ``"info"`` + * path + * |Path|_ to the output file + * mode + * Mode in which the file is opened; + use ``"w"`` to truncate or ``"a"`` to append + (see `open() <https://docs.python.org/3/library/functions.html#open>`__) + * Default: ``"w"`` + * encoding + * File encoding + * Default: ``"utf-8"`` + + Note: path, mode and encoding are only applied when configuring + logging output to a file. +=========== ===== + + +Postprocessor Configuration +--------------------------- +=========== ===== +Type ``object`` + +Example .. code:: + + { + "name": "zip", + "compression": "store", + "extension": "cbz", + "whitelist": ["mangadex", "exhentai", "nhentai"] + } + +Description An object with the ``name`` of a post-processor and its options. + + See `Postprocessor Options`_ for a list of all available + post-processors and their respective options. + + You can also set a ``whitelist`` or ``blacklist`` to + only enable or disable a post-processor for the specified + extractor categories. +=========== ===== + + + +.. |.netrc| replace:: ``.netrc`` +.. |tempfile.gettempdir()| replace:: ``tempfile.gettempdir()`` +.. |requests.request()| replace:: ``requests.request()`` +.. |timeout| replace:: ``timeout`` +.. |verify| replace:: ``verify`` +.. |mature_content| replace:: ``mature_content`` +.. |webbrowser.open()| replace:: ``webbrowser.open()`` +.. |datetime.max| replace:: ``datetime.max`` +.. |Path| replace:: ``Path`` +.. |Logging Configuration| replace:: ``Logging Configuration`` +.. |Postprocessor Configuration| replace:: ``Postprocessor Configuration`` +.. |strptime| replace:: strftime() and strptime() Behavior + +.. _base-directory: `extractor.*.base-directory`_ +.. _skipped: `extractor.*.skip`_ +.. _`date-min and date-max`: `extractor.reddit.date-min & .date-max`_ +.. _date-format: extractor.reddit.date-format_ +.. _deviantart.metadata: extractor.deviantart.metadata_ + +.. _.netrc: https://stackoverflow.com/tags/.netrc/info +.. _tempfile.gettempdir(): https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir +.. _requests.request(): https://docs.python-requests.org/en/master/api/#requests.request +.. _timeout: https://docs.python-requests.org/en/latest/user/advanced/#timeouts +.. _verify: https://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification +.. _`Requests' proxy documentation`: http://docs.python-requests.org/en/master/user/advanced/#proxies +.. _format string: https://docs.python.org/3/library/string.html#formatstrings +.. _format strings: https://docs.python.org/3/library/string.html#formatstrings +.. _strptime: https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior +.. _mature_content: https://www.deviantart.com/developers/http/v1/20160316/object/deviation +.. _webbrowser.open(): https://docs.python.org/3/library/webbrowser.html +.. _datetime.max: https://docs.python.org/3/library/datetime.html#datetime.datetime.max +.. _Authentication: https://github.com/mikf/gallery-dl#5authentication +.. _youtube-dl: https://github.com/ytdl-org/youtube-dl |