Commit b9588ede authored by Jan Koester's avatar Jan Koester
Browse files

Add master/slave replication 

Slaves poll their master over the authdb HTTP(S) port. The master
publishes a content hash per domain (admin db + all domains); a slave
fetches a full snapshot of every domain whose hash differs and replaces
its local content. One master serves any number of slaves; master and
slave can each be a single node or a cluster.

- /replication/status, /replication/snapshot/<id> (authenticated per
  slave by name:key), /replication/info
- slave is read-only: backend writes outside the replication path throw
- slave cluster: only the lowest reachable peer pulls
- domain storage on the slave is mapped to its own backend/path
- tests for snapshot apply, hashing and read-only mode

Co-Authored-By: default avatarClaude Opus 5.5 <noreply@anthropic.com>
parent f2635ea1

README.md

+49 −0
Changes for README.md: 49 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -4,6 +4,55 @@ Authentication Database system with json Rest/Api 
HTTP is available without TLS configuration. To enable HTTPS, set both

`AUTHDB/SSL/KEY` and `AUTHDB/SSL/CERT` in the config.



## Master/slave replication

A master can feed any number of read-only slaves. Master and slave can each be

a single node (file backend) or a cluster (`AUTHDB/CLUSTER/ENABLE`).



Slaves poll the master over the normal authdb HTTP(S) port. The master

publishes a content hash per domain (admin db and every domain); a slave

downloads a full snapshot of each domain whose hash differs and replaces its

local copy. Domains deleted on the master are dropped from the slave's cache.

On a slave all writes (API, admin UI, setup wizard, plugins) are rejected —

changes have to be made on the master.



Master:



```yaml

AUTHDB:

    REPLICATION:

        ROLE: "master"

        SLAVES:                       # name:key, one entry per slave

            - "slave-berlin:secret1"

            - "slave-hamburg:secret2"

```



Slave:



```yaml

AUTHDB:

    REPLICATION:

        ROLE: "slave"

        NAME: "slave-berlin"

        KEY: "secret1"

        MASTERS:                      # several entries if the master is a cluster

            - "https://master1:9090"

            - "https://master2:9090"

        INTERVAL: 30                  # poll interval in seconds

        SSLVERIFY: false

        DOMAINPATH: "/var/lib/authdb" # single node only: where domain files go

                                      # (default: directory of ADMINDB/PATH)

```



- Master cluster: every master node answers; the slave fails over between the

  `MASTERS` entries.

- Slave cluster: only the lowest reachable node in `CLUSTER/PEERS` pulls, the

  others get the data through the cluster store. On a slave cluster the

  domains are stored in the cluster, on a single slave in `DOMAINPATH`,

  independent of how the master stores them.

- `GET /replication/info` shows role, last sync and last error of a node.

- Use HTTPS (`AUTHDB/SSL/*`) between master and slave: the replication key

  and the full database are sent over this connection.



## Requirements

### Operating System

  - Linux

data/config.yaml

+13 −0
Changes for data/config.yaml: 13 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -20,3 +20,16 @@ AUTHDB: 
        PATH: "/tmp/admin.db"

    ADMIN_KEY: "changeme-authdb-admin-key"

    VACUUM_INTERVAL: 6

    # Master/slave replication (see README). ROLE: none | master | slave

    REPLICATION:

        ROLE: "none"

        # master:

        # SLAVES:

        #     - "slave1:changeme-replication-key"

        # slave:

        # NAME: "slave1"

        # KEY: "changeme-replication-key"

        # MASTERS:

        #     - "https://master:9090"

        # INTERVAL: 30

        # SSLVERIFY: false

src/CMakeLists.txt

+1 −0
Changes for src/CMakeLists.txt: 1 added line, 0 removed lines.
Original line number Diff line number Diff line
@@ -17,6 +17,7 @@ list(APPEND authdbSrcs 
    import.cpp

    plugin.cpp

    cluster.cpp

    replication.cpp

)



add_library( authobj OBJECT ${authdbSrcs} )

src/authdb.cpp

+117 −0
Changes for src/authdb.cpp: 117 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -61,6 +61,7 @@ 
#include "session.h"

#include "api.h"

#include "cluster.h"

#include "replication.h"

#include "gpo_default.h"

#include "plugin.h"

#include "log.h"
@@ -259,6 +260,11 @@ namespace authdb { 
                    return;

                }



                if(g_Replication && g_Replication->handleRequest(curreq)){

                    g_Log.access(clientAddr,method,curreq.getRequestURL(),200);

                    return;

                }



                if(_AdminBackend.end()<=sizeof(authdb::AuthHeader)){

                    // Force a prefetch to attempt retrieving domain data from the cluster.

                    // If we do not call lock_shared, prefetch is skipped, meaning transient
@@ -281,6 +287,14 @@ namespace authdb { 
                    }

                    // Cluster has data but not enough peers to reconstruct — don't

                    // show the wizard (would create conflicting admin data).

                    if(g_Replication && g_Replication->isSlave()){

                        libhttppp::HttpResponse rep;

                        rep.setState(HTTP503);

                        rep.setContentType("text/html");

                        const char *msg = "replication slave: waiting for initial sync from master";

                        rep.send(curreq,msg,strlen(msg));

                        return;

                    }

                    if(_AdminBackend.clusterDataExists()){

                        libhttppp::HttpResponse rep;

                        rep.setState(HTTP503);
@@ -912,6 +926,74 @@ int main(int argc,char *argv[]){ 
            backendPath = config.getValue(config.getKey("/AUTHDB/ADMINDB/PATH"), 0);

        }



        /* Master/slave replication */

        authdb::ReplicationConfig rcfg;

        if (auto *roleKey = config.getKey("/AUTHDB/REPLICATION/ROLE")) {

            std::string role = config.getValue(roleKey, 0);

            if (role == "master") {

                rcfg.role = authdb::ReplicationRole::Master;

            } else if (role == "slave") {

                rcfg.role = authdb::ReplicationRole::Slave;

            } else if (!role.empty() && role != "none") {

                std::cerr << "unknown /AUTHDB/REPLICATION/ROLE '" << role << "'" << std::endl;

                return 1;

            }

        }

        auto readList = [&config](const char *path) {

            std::vector<std::string> values;

            auto *key = config.getKey(path);

            if (!key)

                return values;

            for (int i = 0; ; ++i) {

                try {

                    std::string v = config.getValue(key, i);

                    if (v.empty()) break;

                    values.push_back(v);

                } catch (...) { break; }

            }

            return values;

        };

        if (rcfg.role == authdb::ReplicationRole::Master) {

            for (const auto &entry : readList("/AUTHDB/REPLICATION/SLAVES")) {

                auto colon = entry.find(':');

                if (colon == std::string::npos || colon == 0 || colon + 1 == entry.size()) {

                    std::cerr << "invalid replication slave entry (expected name:key)" << std::endl;

                    return 1;

                }

                rcfg.slaves[entry.substr(0, colon)] = entry.substr(colon + 1);

            }

            if (rcfg.slaves.empty())

                std::cerr << "[REPLICATION] master without /AUTHDB/REPLICATION/SLAVES: "

                             "every slave will be rejected" << std::endl;

        } else if (rcfg.role == authdb::ReplicationRole::Slave) {

            rcfg.masters = readList("/AUTHDB/REPLICATION/MASTERS");

            if (auto *k = config.getKey("/AUTHDB/REPLICATION/NAME"))

                rcfg.name = config.getValue(k, 0);

            if (auto *k = config.getKey("/AUTHDB/REPLICATION/KEY"))

                rcfg.key = config.getValue(k, 0);

            if (auto *k = config.getKey("/AUTHDB/REPLICATION/INTERVAL"))

                rcfg.interval = config.getIntValue(k, 0);

            if (rcfg.interval < 1)

                rcfg.interval = 1;

            if (auto *k = config.getKey("/AUTHDB/REPLICATION/SSLVERIFY"))

                rcfg.tls_verify = config.getValue(k, 0) == "true";

            rcfg.cluster = clusterEnabled;

            if (!clusterEnabled) {

                if (auto *k = config.getKey("/AUTHDB/REPLICATION/DOMAINPATH"))

                    rcfg.domain_path = config.getValue(k, 0);

                if (rcfg.domain_path.empty())

                    rcfg.domain_path = std::filesystem::path(backendPath).parent_path().string();

                if (rcfg.domain_path.empty())

                    rcfg.domain_path = ".";

                std::error_code ec;

                std::filesystem::create_directories(rcfg.domain_path, ec);

            }

            if (rcfg.masters.empty() || rcfg.name.empty() || rcfg.key.empty()) {

                std::cerr << "Replication slave requires /AUTHDB/REPLICATION/MASTERS, NAME and KEY" << std::endl;

                return 1;

            }

        }



        std::unique_ptr<libhttppp::HttpD> httpd;

        if (!cert.empty() && !key.empty()) {

            httpd = std::make_unique<libhttppp::HttpD>(
@@ -976,6 +1058,27 @@ int main(int argc,char *argv[]){ 


        authdb::AuthDB authdb(backend,httpd->getServerSockets());



        std::unique_ptr<authdb::Replication> replication;

        if (rcfg.role != authdb::ReplicationRole::None) {

            replication = std::make_unique<authdb::Replication>(rcfg, backend);

            replication->setDomainResolver([&authdb](const ::uuid::uuid &did) {

                return authdb.getDomainBackend(did);

            });

            replication->setDomainEvictor([&authdb](const ::uuid::uuid &did) {

                authdb.evictDomainCache(did);

            });

            if (replication->isSlave()) {

                authdb::AuthBackend::setReadOnly(true);

                authdb::Replication *r = replication.get();

                authdb::g_DomainStorageOverride = [r](const ::uuid::uuid &did, authdb::DomainInfo &info) {

                    r->overrideDomainStorage(did, info);

                };

            }

            authdb::g_Replication = replication.get();

            std::cerr << "[REPLICATION] role: "

                      << (replication->isMaster() ? "master" : "slave") << std::endl;

        }



        // netplus::event's default is just the core count — mediadb boosts

        // the same base class to max(cores*4, 16) (server.cpp) since request

        // handlers can block on cluster round-trips; authdb's session-lookup
@@ -1081,6 +1184,10 @@ int main(int argc,char *argv[]){ 
                            std::cerr << "chown " << ccfg.store_path << " failed: " << strerror(errno) << std::endl;

                    }

                }

                if (!rcfg.domain_path.empty()) {

                    if (chown(rcfg.domain_path.c_str(), pw->pw_uid, pw->pw_gid) != 0)

                        std::cerr << "chown " << rcfg.domain_path << " failed: " << strerror(errno) << std::endl;

                }

                if (setgid(pw->pw_gid) != 0) {

                    std::cerr << "setgid failed" << std::endl;

                    return 1;
@@ -1131,6 +1238,9 @@ int main(int argc,char *argv[]){ 
                vacuumIntervalHours = 1;

        }



        if (replication)

            replication->start();



        std::atomic<bool> vacuumRunning{true};

        std::mutex vacuumMtx;

        std::condition_variable vacuumCv;
@@ -1159,6 +1269,13 @@ int main(int argc,char *argv[]){ 
        vacuumRunning = false;

        vacuumCv.notify_all();



        /* Stop replication before the backends go away */

        if (replication) {

            replication->stop();

            authdb::g_Replication = nullptr;

            authdb::g_DomainStorageOverride = nullptr;

        }



        /* Shutdown cluster */

        authdb::g_Cluster = nullptr;

        cluster.stop();

src/authdb.h

+5 −0
Changes for src/authdb.h: 5 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -29,6 +29,7 @@ 
#include <iterator>

#include <cstring>

#include <mutex>

#include <functional>

#include <uuidp.h>



#include "backend.h"
@@ -158,6 +159,10 @@ namespace authdb { 
        bool        found = false;

    };



    // Set on replication slaves: rewrites the storage of a domain resolved

    // from the (replicated) admin db to this node's own storage.

    extern std::function<void(const uuid::uuid &did, DomainInfo &info)> g_DomainStorageOverride;



    class DomainBackend {

    public:

        DomainBackend(authdb::AuthBackend &adminbck);