Skip to content

Commit b50b4aa

Browse files
mxsashamxsasha
committed
Fix irrdnet#26 - Configuration file support with documentation. (irrdnet#140)
1 parent 174933f commit b50b4aa

30 files changed

+945
-418
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
.mypy_cache
55
.tmpdirs
66
.pytest_cache
7+
local_config.yaml
78

89
syntax: glob
910
__pycache__

‎alembic.ini

Lines changed: 0 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,3 @@
1-
# TODO: this is a temporary alembic config file that should be integrated with a general config system
2-
31
[alembic]
42
# path to migration scripts
53
script_location = irrd/storage/alembic/
6-
7-
# Logging configuration
8-
[loggers]
9-
keys = root,sqlalchemy,alembic
10-
11-
[handlers]
12-
keys = console
13-
14-
[formatters]
15-
keys = generic
16-
17-
[logger_root]
18-
level = WARN
19-
handlers = console
20-
qualname =
21-
22-
[logger_sqlalchemy]
23-
level = WARN
24-
handlers =
25-
qualname = sqlalchemy.engine
26-
27-
[logger_alembic]
28-
level = INFO
29-
handlers =
30-
qualname = alembic
31-
32-
[handler_console]
33-
class = StreamHandler
34-
args = (sys.stderr,)
35-
level = WARN
36-
formatter = generic
37-
38-
[formatter_generic]
39-
format = %(levelname)-5.5s [%(name)s] %(message)s
40-
datefmt = %H:%M:%S

‎conftest.py

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
"""
2+
Testing configuration, used by py.test.
3+
Fixtures defined here are available to all tests.
4+
"""
5+
import os
6+
7+
import pytest
8+
from dotted.collection import DottedDict
9+
from typing import Dict, Any
10+
11+
from irrd.rpsl.rpsl_objects import rpsl_object_from_text
12+
from irrd.utils.rpsl_samples import SAMPLE_KEY_CERT
13+
14+
15+
@pytest.fixture()
16+
def config_override(monkeypatch):
17+
"""
18+
Fixture to override part of the configuration, by providing a dict
19+
with new config data.
20+
21+
Note that subsequent calls override previous ones, so the entire
22+
override dict must be supplied every time.
23+
"""
24+
def _override(override_data: Dict[Any, Any]):
25+
monkeypatch.setattr('irrd.conf.testing_overrides', DottedDict(override_data))
26+
return _override
27+
28+
29+
@pytest.fixture()
30+
def tmp_gpg_dir(tmpdir, monkeypatch):
31+
"""
32+
Fixture to use a temporary separate gpg dir, to prevent it using your
33+
user's keyring.
34+
35+
NOTE: if the gpg keyring name is very long, this introduces a 5 second
36+
delay in all gpg tests due to gpg incorrectly waiting to find a gpg-agent.
37+
Default tmpdirs on Mac OS X are affected, to prevent this run pytest with:
38+
--basetemp=.tmpdirs
39+
"""
40+
os.environ['IRRD_AUTH_GNUPG_KEYRING'] = str(tmpdir) + "/gnupg"
41+
42+
43+
@pytest.fixture()
44+
def preload_gpg_key():
45+
"""
46+
Fixture to load a known PGP key into the configured keychain.
47+
"""
48+
# Simply parsing the key-cert will load it into the GPG keychain
49+
rpsl_text = SAMPLE_KEY_CERT
50+
rpsl_object_from_text(rpsl_text)
51+
52+
53+
def pytest_configure(config):
54+
"""
55+
This function is called by py.test, and will set a flag to
56+
indicate tests are running. This is used by the configuration
57+
checker to not require a full working config for most tests.
58+
Can be checked with:
59+
hasattr(sys, '_called_from_test')
60+
"""
61+
import sys
62+
sys._called_from_test = True

‎docs/usage/configuration.rst

Lines changed: 268 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,268 @@
1+
.. |br| raw:: html
2+
3+
<br />
4+
5+
=============
6+
Configuration
7+
=============
8+
9+
IRRd reads its configuration from a YAML file, in a specified location. Many
10+
configuration options can be changed without restarting IRRd, but not all.
11+
12+
Example configuration file
13+
--------------------------
14+
15+
.. highlight:: yaml
16+
:linenothreshold: 5
17+
18+
This sample shows most configuration options::
19+
20+
irrd:
21+
database_url: 'postgresql://localhost:5432/irrd'
22+
23+
access_lists:
24+
http_database_status:
25+
- '::/32'
26+
- '127.0.0.1'
27+
28+
server:
29+
http:
30+
access_list: http_database_status
31+
interface: '::0'
32+
port: 8080
33+
whois:
34+
interface: '::0'
35+
max_connections: 50
36+
port: 8043
37+
38+
auth:
39+
gnupg_keyring: /home/irrd/gnupg-keyring/
40+
override_password: {hash}
41+
42+
email:
43+
footer: 'email footer'
44+
from: [email protected]
45+
smtp: localhost
46+
47+
log:
48+
logfile_path: /var/log/irrd/irrd.log
49+
loglevel: DEBUG
50+
51+
sources_default:
52+
- AUTHDATABASE
53+
- MIRROR-SECOND
54+
- MIRROR-FIRST
55+
56+
sources:
57+
AUTHDATABASE:
58+
# Authoritative database, allows local changes, full export every 2h
59+
authoritative: true
60+
keep_journal: true
61+
export_destination: /var/ftp/
62+
export_timer: 7200
63+
MIRROR-FIRST:
64+
# Run a full import at first, then periodic NRTM updates.
65+
authoritative: false
66+
keep_journal: true
67+
import_serial_source: 'ftp://ftp.example.net/MIRROR-FIRST.CURRENTSERIAL'
68+
import_source: 'ftp://ftp.example.net/mirror-first.db.gz'
69+
nrtm_host: rr.ntt.net
70+
nrtm_port: 43
71+
object_class_filter:
72+
- as-set
73+
- aut-num
74+
- filter-set
75+
- inet-rtr
76+
- key-cert
77+
- mntner
78+
- peering-set
79+
- route
80+
- route6
81+
- route-set
82+
- rtr-set
83+
MIRROR-SECOND:
84+
# Every hour, a new full import will be done.
85+
authoritative: false
86+
import_source:
87+
- 'ftp://ftp.example.net/mirror-second.db.as-set.gz'
88+
- 'ftp://ftp.example.net/mirror-second.db.aut-num.gz'
89+
- 'ftp://ftp.example.net/mirror-second.db.filter-set.gz'
90+
- 'ftp://ftp.example.net/mirror-second.db.route-set.gz'
91+
- 'ftp://ftp.example.net/mirror-second.db.route.gz'
92+
- 'ftp://ftp.example.net/mirror-second.db.route6.gz'
93+
- 'ftp://ftp.example.net/mirror-second.db.route-set.gz'
94+
import_timer: 3600
95+
96+
97+
Loading and reloading
98+
---------------------
99+
100+
The configuration is loaded when IRRd starts. The path to the config file is read from the ``IRRD_CONFIG_PATH``
101+
environment variable. If the configuration is invalid, the daemon will refuse to start.
102+
While running, the configuration can be reloaded by sending a `SIGHUP` signal. Most settings will take effect
103+
immediately, but some require a full restart. If a `SIGHUP` is sent and the new configuration is invalid,
104+
errors will be written to the logfile, but IRRd will keep running with the last valid configuration.
105+
A successful reload after a `SIGHUP` is also logged.
106+
107+
.. important::
108+
109+
Not all configuration errors are caught when reloading, such as making IRRd bind to a TCP port that
110+
is already in use. An incorrect password for the PostgreSQL database is only detected when IRRd
111+
restarts and attempts to connect.
112+
113+
Configuration options
114+
---------------------
115+
116+
Database
117+
~~~~~~~~
118+
* ``database_url``: a RFC1738 PostgreSQL database URL for the database used by IRRd, e.g.
119+
``postgresql://username:password@localhost:5432/irrd`` to connect to `localhost` on port 5432, database `irrd`,
120+
username `username`, password `password`.
121+
|br| **Default**: not defined, but required.
122+
|br| **Change takes effect**: after full IRRd restart.
123+
124+
Servers
125+
~~~~~~~
126+
* ``server.[whois|http].interface``: the network interface on which the whois or HTTP interface will listen
127+
|br| **Default**: `::0`
128+
|br| **Change takes effect**: after full IRRd restart.
129+
* ``server.[whois|http].port``: the port on which the whois or HTTP interface will listen
130+
|br| **Default**: 43 for whois, 80 for HTTP.
131+
|br| **Change takes effect**: after full IRRd restart.
132+
* ``server.[whois|http].access_list``: a reference to an access list in the configuration, where only IPs in the access
133+
list are permitted access. If not defined, all access is permitted for whois, but all access is denied for HTTP.
134+
|br| **Default**: not defined, all access permitted for whois, all access denied for HTTP
135+
|br| **Change takes effect**: after SIGHUP.
136+
* ``server.whois.max_connections``: the maximum number of simultaneous whois connections permitted
137+
|br| **Default**: 50
138+
|br| **Change takes effect**: after SIGHUP. Existing connections will not be terminated.
139+
140+
Email
141+
~~~~~
142+
* ``email.from``: the `From` email address used when sending emails
143+
|br| **Default**: not defined, but required
144+
|br| **Change takes effect**: after SIGHUP, for all subsequent emails
145+
* ``email.footer``: a footer to include in all emails
146+
|br| **Default**: empty string
147+
|br| **Change takes effect**: after SIGHUP, for all subsequent emails
148+
* ``email.smtp``: the SMTP server to use for outbound emails
149+
|br| **Default**: not defined, but required
150+
|br| **Change takes effect**: after SIGHUP, for all subsequent emails
151+
152+
Authentication
153+
~~~~~~~~~~~~~~
154+
* ``auth.override_password``: a salted MD5 hash of the override password, which can be used to override any
155+
authorisation requirements for authoritative databases
156+
|br| **Default**: not defined, no override password will be accepted
157+
|br| **Change takes effect**: after SIGHUP
158+
* ``auth.gnupg_keyring``: the full path to the gnupg keyring
159+
|br| **Default**: not defined, but required
160+
|br| **Change takes effect**: after full IRRd restart
161+
162+
.. danger::
163+
164+
IRRd loads keys into the gnupg keyring when `key-cert` objects are imported. Their presence in the
165+
keyring is then used to validate requested changes. Therefore, the keyring referred to by
166+
``auth.gnupg_keyring`` can not be simply reset, or PGP authentications may fail.
167+
168+
169+
Access lists
170+
~~~~~~~~~~~~
171+
* ``access_lists.{list_name}``: a list of permitted IPv4 and/or IPv6 addresses and/or prefixes, which will be
172+
permitted access for any service that refers to access list ``{list_name}``
173+
|br| **Default**: no lists defined
174+
|br| **Change takes effect**: after SIGHUP, for all subsequent requests
175+
176+
Sources
177+
~~~~~~~
178+
* ``sources_default``: a list of sources that are enabled by default, or when a user selects all sources
179+
with ``-a``. The order of this list defines the search priority as well. It is not required to include
180+
all known sources in the default selection.
181+
|br| **Default**: not defined. All sources are enabled, but results are not ordered by source.
182+
|br| **Change takes effect**:
183+
* ``sources.{name}``: settings for a particular source.
184+
* ``sources.{name}.authoritative``: a boolean for whether this source is authoritative, i.e. changes are allowed
185+
to be submitted to this IRRd instance through e.g. email updates
186+
|br| **Default**: ``false``
187+
|br| **Change takes effect**: after SIGHUP, for all subsequent requests.
188+
* ``sources.{name}.keep_journal``: a boolean for whether a local journal is retained of changes to objects from
189+
this source. This journal can contain changes submitted to this IRRd instance, or changes received over NRTM.
190+
This setting is needed when offering mirroring services for this source. Can only be enabled when either
191+
``authoritative`` is enabled, or all three of ``nrtm_host``, ``nrtm_port`` and ``import_serial_source``.
192+
|br| **Default**: ``false``
193+
|br| **Change takes effect**: after SIGHUP, for all subsequent changes.
194+
* ``sources.{name}.nrtm_host``: the hostname or IP to connect to for an NRTM stream
195+
|br| **Default**: not defined, no NRTM requests attempted
196+
|br| **Change takes effect**: after SIGHUP, at the next NRTM update.
197+
* ``sources.{name}.nrtm_port``: the TCP port to connect to for an NRTM stream
198+
|br| **Default**: not defined, no NRTM requests attempted
199+
|br| **Change takes effect**: after SIGHUP, at the next NRTM update.
200+
* ``sources.{name}.import_source``: the FTP URL or list of URLs where the full copies of this source can be
201+
retrieved. You can provide a list of URLs for sources that offer split files.
202+
|br| **Default**: not defined, no imports attempted
203+
|br| **Change takes effect**: after SIGHUP, at the next full import. This will only occur if this source is
204+
forced to reload, i.e. changing this URL will not cause a new full import by itself in sources that use NRTM.
205+
For sources that do not use NRTM, every mirror update is a full import.
206+
* ``sources.{name}.import_serial_source``: the FTP URL where the file with serial belonging to the ``import_source``
207+
can be retrieved
208+
|br| **Default**: not defined, no imports attempted
209+
|br| **Change takes effect**: see ``import_source``.
210+
* ``sources.{name}.import_timer``: the time between two updates, either by full import or NRTM.
211+
This is particularly significant for sources that do not offer an NRTM stream, as they will instead run a
212+
full import every time this timer expires. The default is rather frequent for sources that work exclusively
213+
with periodic full imports.
214+
|br| **Default**: 300
215+
|br| **Change takes effect**: after SIGHUP
216+
* ``sources.{name}.object_class_filter``: a list of object classes that will be mirrored. Objects of other RPSL object
217+
classes will be ignored. Without a filter, all objects are mirrored.
218+
|br| **Default**: no filter, all object classes permitted
219+
|br| **Change takes effect**: after SIGHUP, at the next NRTM update or full import.
220+
* ``sources.{name}.export_destination``: a path to save full exports, including a serial file, of this source
221+
|br| **Default**: not defined, no exports made.
222+
|br| **Change takes effect**: after SIGHUP, at the next ``export_timer``
223+
* ``sources.{name}.export_timer``: the time between two full exports
224+
|br| **Default**: 3600
225+
|br| **Change takes effect**: after SIGHUP
226+
227+
.. caution::
228+
229+
**Journal-keeping is the only full object history that is kept of the database, and is therefore strongly
230+
recommended to enable on authoritative databases to be able to reconstruct history.**
231+
232+
Journal-keeping for NRTM streams is dependent on providing a single uninterrupted stream of updates.
233+
This stream is only kept while ``keep_journal`` is enabled. Disabling it while mirrors are dependent
234+
on it, even briefly, will cause the databases to go out of sync silently until the mirror
235+
runs a new full import.
236+
237+
.. note::
238+
239+
There are fundamentally two different ways to mirror other databases:
240+
241+
* **NRTM mode**: providing a location to download full copies of the database, the serial belonging to
242+
that copy, and then updating this using an NRTM stream. This method is recommended, as it is efficient
243+
and allows IRRd to generate a journal, if enabled, so that others can mirror the source from this
244+
IRRd instance too.
245+
246+
* **Periodic full import mode**: providing a location to download full copies of the database, and no
247+
other details. Every ``import_timer``, the entire database will be reloaded from the full copies.
248+
Journals can not be generated.
249+
250+
.. note::
251+
252+
Source names are case sensitive and must be an exact match to ``sources_default``, and the source
253+
attribute value in any objects imported from files or NRTM. E.g. if ``sources.EXAMPLE`` is defined,
254+
and ``sources_default`` contains ``example``, this is a configuration error. If an object is
255+
encountered with ``source: EXAMPLe``, it is rejected and an error is logged.
256+
257+
258+
Logging
259+
~~~~~~~
260+
* ``log.logfile_path``: the full path where the logfile will be written. IRRd will attempt to create the file if it
261+
does not exist. If the file is removed, e.g. by a log rotation process, IRRd will create a new file in the same
262+
location, and continue writing to the new file. Timestamps in logs are always in UTC, regardless of local machine
263+
timezone.
264+
|br| **Default**: not defined, logs will be sent to the console
265+
|br| **Change takes effect**: after full IRRd restart.
266+
* ``log.level``: the loglevel, one of `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. The recommended level is `INFO`.
267+
|br| **Default**: `INFO`
268+
|br| **Change takes effect**: after SIGHUP.

0 commit comments

Comments
 (0)