admin
Implements the reva Admin API gRPC service (reva.admin.v1beta1), an authenticated operator surface for fleet introspection and bounded, audited actions.
- Package
github.com/cs3org/reva/v3/internal/grpc/services/admin- Source
- internal/grpc/services/admin · Go reference
Enable it
Add this table to the revad configuration file:
[grpc.services.admin]
# settings belowOverview
Package admin implements the reva Admin API gRPC service (reva.admin.v1beta1), an authenticated operator surface for fleet introspection and bounded, audited actions. It binds its own admin-only server block and requires the admin scope on every method except RequestAdmin — the step-up door.
The admin gRPC service is reva’s operator surface: an authenticated,
reva-specific API for looking at a running fleet and running bounded, audited
operations against it. It is defined inside reva (not in the CS3 APIs) because it
is tooling, and its Go stubs are exported from pkg/admin/adminpb
so external scripts integrate by a plain import.
It builds on three things: the service registry (which process runs what,
pkg/registry), the control channel (a way to
reach a running service instance, pkg/control/controlpb +
internal/grpc/control), and the invocation framework every
service can opt into (pkg/invoke). The Admin API is
the client that ties them together; the invocation framework is usable on its
own.
Architecture
The Admin API is off by default. It turns on when a process loads the admin
service in a gRPC block with an admin_group set — typically the process that
also runs the gateway, so a single co-located admin can see the whole fleet
through the shared registry.
reva CLI ──gRPC──► admin service (one gRPC block, admin-scoped)
│
reads ─────────────┤ registry: services, nodes, health, metadata
│
invokes ───────────┘ resolve selector ──► control channel(s)
│ (one gRPC port per process)
▼
route by node id ──► service instance
│
shared defaults (config) + service methods
Two planes:
- Introspection reads the registry directly (server info, per-service health, the fleet’s services with their nodes, a service’s redacted config). No service has to cooperate; the runtime already advertises everything in registry metadata.
- Invocations run an operation inside a service instance. The admin does not call the service’s business gRPC; it goes through the control channel, a separate small gRPC server every process stands up (see below), and the process routes the call to the addressed instance.
The control channel
Every reva process that hosts anything invokable stands up one control gRPC
server (reva.control.v1beta1), on its own port, independent of whether the
Admin API service is loaded. It hosts Invoke, its streaming twin
InvokeStream, and ListInvocations. It is
not registered as a service of its own — instead every real service node
advertises the channel’s address in its control registry metadata, and the
address is shared by all services in the process.
The control server routes by node id (host:port/service): the runtime
registers each loaded service as an instance under its node id, carrying the
service’s redacted config and its optional Invokable. Routing by id (not by
service name) means a process can run several instances of the same service and
each is still addressed precisely.
A serverless service has no listen address, so its node id uses the
process’s control address instead (control-addr/service) and its registry node
advertises an empty address. Everything else is identical — the control
metadata, the heartbeat, and the invocation routing — so the admin sees and
reaches it like any other instance.
Selectors
The Invoke and GetServiceConfig RPCs take a selector, resolved
server-side against the registry — the admin counterpart of how the gateway
selects a peer:
- a plain service name (
userprovider) fans out to every live instance, returning one result per instance; - a node id (
10.0.0.4:19003/userprovider, as shown byreva admin services -o wide/json) targets exactly that instance; and - a partial id widens the scope: an address (
10.0.0.4:19003) targets every instance bound to it, a bare host (10.0.0.4) every instance on that machine. A bare token that is both a service name and a host resolves as the service.
The admin resolves the selector to a set of (node id, control address)
endpoints, dials each process’s control channel, and passes the node id as the
routing target. A unary Invoke is synchronous: every addressed instance
runs and reports before the call returns.
Streaming invocations
Some operations produce a stream of results over time rather than a single one
(following logs is the built-in example). These ride InvokeStream, the
streaming twin of Invoke: same selector, same routing, but the admin
multiplexes each instance’s stream into one, labelling every item with its node,
until the client disconnects. A developer declares one by giving an invocation a
streaming handler (Set.Add(...).Stream().HandleStream(fn)); the catalog flags
it streaming so the CLI knows to open a stream. Multi-instance follows
interleave as items arrive (like kubectl logs -f).
Reading logs
Every reva process keeps its most recent log lines in a bounded in-memory ring
(pkg/logtail) that the logger tees into — the real
sink (stdout, a file) is unchanged. That buffer is exposed as a built-in logs
invocation on every service (like config), so it needs no per-service code and
rides the same selector + control fan-out as any invocation. reva admin logs
drives it:
reva admin logs userprovider # recent lines from every userprovider, merged
reva admin logs 10.0.0.4:19003/userprovider # one instance
reva admin logs 10.0.0.4:19003 # every service at that address
reva admin logs 10.0.0.4 # every service on that machine
reva admin logs userprovider -f # follow (InvokeStream) until Ctrl-C
reva admin logs userprovider -level warn -since 5m -grep timeout -n 500
Because the buffer is process-wide, lines are attributed to a service by their
service log field. It is stamped in two places: construction-time loggers get
it from the service loader, and per-request loggers get it from the appctx
interceptor — the gRPC server records which proto services each reva service
registers, so a request’s full method resolves to its owning service (the HTTP
router does the same by routed prefix). Process-level lines that belong to no
service (startup, registry) match no service filter; the raw stream, them
included, is reachable with admin invoke <node-id> logs all=true. The window
is bounded ([log] tail, default 2000 lines); there is no deeper history.
Draining instances
reva admin services drain <selector> takes the matched instances out of
rotation: the process starts advertising them as draining in the registry,
and the shared service selector already
excludes draining nodes from every Pick, so no new traffic is routed to them.
Established in-flight calls finish on their own — the selector has no per-request
handle to sever — so a drain is graceful by construction. enable returns them
to ready. Both ride the same selector fan-out, so drain 10.0.0.4:19003
drains a whole process and drain userprovider drains every instance of a
service.
This is a built-in rotation invocation flipping an in-memory per-node flag the
heartbeat reads (like logs level): it is not persisted — a restart or
config reload re-registers the node ready. A drained node stays alive and
control-reachable, so logs, stack, config and enable still work against
it; only offline nodes are hidden from the admin fan-out. The current state is
visible in admin services (the STATE column).
Request activity (is it safe to restart?)
reva admin services activity <selector> reports, per instance, how many
requests it is currently serving (in-flight), how many it has served
(total), and how long it has been idle. This is the companion to drain:
drain stops new traffic, and activity tells you when the in-flight traffic has
finished, so you know when a node has quiesced and is safe to restart.
admin services drain 10.0.0.4:19003 # out of rotation
admin services -wait -idle 5s activity 10.0.0.4:19003 # block until quiesced
# ... restart the process ...
admin services enable 10.0.0.4:19003
-wait polls until every matched instance has zero in-flight requests and
has been idle for at least -idle (default 5s), or -timeout (default 2m)
elapses — in which case it lists the still-active instances and exits non-zero,
so it drops into a shell procedure. (As with the other subcommands, flags come
before the activity token: services -wait … activity <selector>.)
-methods adds the per-RPC-method breakdown for gRPC services (busiest first),
so you can see what an instance is serving, not just how much:
10.0.0.4:19003/userprovider: in-flight=2 total=1841 idle=0s
GetUser in-flight=1 total=1203 idle=0s
GetUserByClaim in-flight=1 total= 512 idle=0s
GetUserGroups in-flight=0 total= 126 idle=3s
The method set is bounded, so this is cardinality-safe. The breakdown lives in
the same per-instance counter (a lazily-built per-method map alongside the
lock-free aggregate, so -wait’s quiescence check never pays for it). HTTP
requests count toward the aggregate only.
The count is kept always-on in a small counter (pkg/activity) created per
service per server in the runtime and shared by reference between the request
choke points that feed it — the same ones that stamp the service log field,
the gRPC appctx interceptor and the HTTP router — and the activity invocation
that reports it. There is no process-global state: because a server is one
listener, the counter is 1:1 with a node id, so the numbers are per
instance — unlike the shared logs/stack process buffers, two co-located
same-name instances (on different listeners) count separately. Admin API and
control-channel RPCs are excluded, so the activity query itself and long-lived
admin streams (e.g. logs -f) never register as traffic. It counts
registry-routed and directly addressed requests alike — a request reaching a
drained node through a pinned connection still shows up (correctly: it is
being served). Counters reset on restart (a fresh process starts idle).
Security and scope
The security model is sudo-style step-up, built on an admin auth scope that is
mutually isolated from user scopes (see
pkg/auth/scope/admin.go):
- An admin-scoped token satisfies Admin API and control requests and nothing else — the storage/share verifiers do not recognize admin messages.
- A user-scoped token can never satisfy an admin request — the always-true
user scope is guarded by
isAdminResource.
Getting an admin token is a deliberate step-up:
RequestAdminis the one method reachable with an ordinary user token (it is the elevation door). It checks the caller’s membership inadmin_groupand, only on success, mints a short-TTL, admin-only token (scope{admin}, nouserkey). Grant and deny are both audited.- Every other method (including
Invoke, and the control channel itself) requires that admin token. The control server runs the standard auth interceptor, so it is exactly as protected as the Admin API — an unauthenticated call to the control port is rejected. An admin token is authorized purely by its scope, so validating it never triggers a user-group lookup — its bearer may be a synthetic local-root identity (see below) that no user provider knows, and on askip_user_groups_in_tokenfleet that lookup would otherwise reject a valid admin token on every fan-out hop. Impersonatemints an ordinary user-scoped token for a target user (never admin+user), so downstream services apply their normal user checks. It is audited and needs the machine-auth api key configured.
Invocations are classified readonly | mutating | dangerous; the CLI prompts
before a dangerous one. Every invocation and action emits an audit event via
pkg/admin (audit=true on the context logger).
Local root
The step-up above answers “who are you?” with a token, which is right for remote
access. For an operator working on the box, the Admin API can also be served
on a Unix socket where the transport itself is the authentication — like
sudo, docker.sock, or PostgreSQL peer. There is no token and no login: a
permitted local user runs reva admin … and it just works.
On each connection the socket reads the peer’s OS credentials via SO_PEERCRED
(the kernel reports the connecting process’s uid/gid, so it cannot be forged). If
the uid is permitted, the interceptor mints an admin token internally and injects
it, so the request runs exactly like a remotely-elevated one — fan-out to other
processes’ control channels included. Two gates apply:
- the socket file’s permissions — who can even connect; and
- an optional
socket_group— with it set, the peer’s uid must belong to that Unix group (checked against the OS group database, supplementary groups included); without it, anyone who can open the socket is admin.
reva sets the socket ownership/mode to match: 0660 owned by socket_group (so
reva must be root or a member of it), else 0600 (only the process owner and
root). Every local-root request is audited with the OS user and uid.
It is on by default (Linux): the server binds a well-known path —
/run/reva/admin.sock, or $XDG_RUNTIME_DIR/reva/admin.sock when the former is
not writable (rootless) — and the CLI probes the same list, so reva admin …
just works locally with no flag and no login. Because the default is 0600, that
means only the reva process owner and root until a socket_group opens it.
Binding the default is best-effort (a missing runtime dir simply disables it);
set socket = "off" to turn it off, or an explicit path to override.
Configuration
The Admin API is enabled by loading the admin service with an admin_group.
The control channel is configured once per process in the [grpc] section.
[grpc]
# The per-process control channel's bind address. Empty => a random port on all
# interfaces (the auth interceptor, not the bind host, is what protects it).
control_address = "127.0.0.1:19700"
[grpc.services.admin]
address = "127.0.0.1:19010"
admin_group = "sailing-lovers" # required: members may step up. Unset => no Admin API.
admin_ttl = "15m" # lifetime of a minted admin token (default 15m)
# Lifetime of a token minted by Impersonate. Unset, it is the token manager's
# own (a day for jwt), as for signing in: an impersonated session is used like
# any other, long transfers included.
# impersonation_ttl = "12h"
# Optional: enables Impersonate.
machine_auth_apikey = "..."
# Local root over a Unix socket (Linux, ON BY DEFAULT). Omit to use the default
# path; "off" disables it; an explicit path overrides the default.
socket = "off" # e.g. "/run/reva/admin.sock"
socket_group = "reva-admin" # optional: restrict to this group (else 0600, owner-only)
A process that hosts an invokable service but not the admin service still
stands up its control channel (so a remote admin can reach it); it just reads the
same [grpc] control_address key.
The in-memory log buffer that backs reva admin logs is sized in the [log]
section:
[log]
tail = 2000 # recent lines kept in memory for `admin logs` (0 disables it)
Using the CLI
reva -insecure -host <gateway> login -username <u> -password <p> basic
reva admin elevate -admin-host <admin:port> # step up, stores a short-TTL admin token
reva admin services [-v] [-o wide|json] [service]
reva admin services drain <selector> [-y] # take instances out of rotation
reva admin services enable <selector> # return them to rotation
reva admin services [-wait [-idle D] [-timeout D]] activity <selector> # in-flight/idle
reva admin config <service|node-id> [-o toml|json] [-diff]
reva admin invocations <service|node-id>
reva admin invoke [-stream] <selector> <invocation> [key=val ...]
reva admin logs <selector> [-f] [-n N] [-level L] [-since D] [-grep P] [-o text|json]
reva admin logs level <selector> [trace|debug|info|warn|error] # report/set runtime level
reva admin trace <traceid> | -user <username> # one request/user across the fleet
reva admin stack <selector> [-grep P] # goroutine dumps, e.g. of a hung process
reva admin jobs list | active | runs | status <id> | run <job> [k=v] | trigger <job> | cancel <id> | stop <job>
reva admin impersonate <user>
# Local root: on the box, no login/elevate/flag — the CLI finds the socket. Only
# if it is absent (or denies you) does it need the network host + elevate above.
reva admin services
Over HTTP
Clients that only speak the public HTTPS surface (the web UI, the cernbox
CLI) reach the user-facing part of the Admin API through the admin HTTP
service. It sits behind the normal HTTP auth middleware, so the caller
authenticates as any other request does, and resolves the Admin API through the
service registry.
[http.services.admin]
# prefix = "admin" # the default
| Request | Answer |
|---|---|
GET /admin/status | {"admin": true|false}. Backed by CheckAdmin, which mints nothing and is not audited. |
POST /admin/impersonate {"user": "...", "reason": "..."} (reason optional) | {"token": "..."}, a user token for the target. |
impersonate runs RequestAdmin and Impersonate back to back on the
server, so the admin token never reaches the client: the client holds at most
a user token. Both steps are audited as usual, with reason when one is given. Errors
are {"message": "..."} with 401 (caller not identified), 403 (not an admin),
404 (no such user), 501 (impersonation not configured) or 503 (no Admin API).
Jobs
reva admin jobs drives the background jobs runner (pkg/rjobs),
and shows the general shape for adding typed Admin API methods to interact with
a service: the surface is typed RPCs on the AdminAPI, and each method’s body
reuses the invoke channel to reach the service — fanning out or targeting one
runner by where the state lives. The admin keeps no store or business
dependency; it reaches the runner the same way it reaches any service.
Two sources of truth, two access patterns:
The driver’s live internal state — registered jobs and what each is doing right now (running/idle, scope, worker pool), held in per-process memory and largely absent from any database.
InspectJobsfans out to every runner and merges.jobs listis the job-centric view,jobs activethe run-centric one:jobs list # NAME, KIND, SCHEDULE, SCOPE, STATUS (running-where) jobs active # per node: the runs executing now + worker poolThe durable run ledger — the persisted history of runs, in the shared store (nats queue + SQL status).
ListJobRuns/GetJobRuntarget one runner (any; the store is shared). The mutationsEnqueueJob/TriggerJob/CancelJobRun/CancelPeriodicJobalso target one — ingress ≠ execution: an enqueued run is later claimed by whatever worker is free (possibly a different runner), and cancels broadcast cluster-wide. They are audited.jobs runs [-job X] [-owner Y] [-state s,...] [-internal] [-n N] jobs status <run-id> jobs run <job> [k=v ...] [-owner U] # enqueue on-demand → run id jobs trigger <job> # fire a periodic job now jobs cancel <run-id> | jobs stop <job>
Under the hood the runner exposes an invoke.Set (inspect/runs/status/
enqueue/trigger/cancel/stop) that the admin calls over the control
channel — the admin↔runner transport, never a user-facing surface. Caveats:
all-nodes periodic jobs never touch the store so they have no runs history
but do show in list/active; and failed is not terminal — the framework
retries.
Adding an invocation to a service (for developers)
A service exposes admin operations by building an invoke.Set
— you declare each method once and the framework does the name→handler routing,
builds the catalog, and validates required arguments. There is no Invoke
switch to maintain. Every service also gets the built-in invocations (config,
logs, loglevel, stack, version) for free, so what you add is on top of
those. (Built-ins live in
pkg/invoke, one self-registering file each — see config.go.)
Two steps: embed the *invoke.Set (that makes the service Invokable, which the
runtime discovers), and declare your methods at construction.
type service struct {
*invoke.Set // makes the service Invokable
groupmgr group.Manager
}
func New(ctx context.Context, m map[string]any) (rgrpc.Service, error) {
// ...build groupManager...
svc := &service{groupmgr: groupManager}
svc.registerInvocations()
return svc, nil
}
// All a developer writes to add an operator method: name it, declare its args,
// hand it a handler.
func (s *service) registerInvocations() {
s.Set = invoke.NewSet()
s.Set.Add("is_member", "Report whether a user is a member of a group").
Arg("group", "the group id").
Arg("user", "the user id").
OptArg("idp", "the identity provider of the group and user").
Handle(s.invokeIsMember)
}
func (s *service) invokeIsMember(ctx context.Context, a invoke.Args) (invoke.Result, error) {
ok, err := s.groupmgr.HasMember(ctx,
&grouppb.GroupId{OpaqueId: a.String("group"), Idp: a.String("idp")},
&userpb.UserId{OpaqueId: a.String("user"), Idp: a.String("idp")},
)
if err != nil {
return nil, err
}
return invoke.Result{"member": ok}, nil
}
That is the whole thing. is_member now shows up in
reva admin invocations groupprovider, its required group/user args are
checked before the handler runs, and:
reva admin invoke groupprovider is_member group=sailing-lovers user=einstein idp=cernbox
# → 127.0.0.1:19004/groupprovider: {"member":true} (one line per instance)
Notes:
- Kinds:
.Mutating()or.Dangerous()afterAdd(...)change the kind (default readonly);dangerousmakes the CLI confirm. - Args: handlers receive
invoke.Argswith typed accessors —a.String(k),a.Bool(k),a.Int(k),a.Has(k). Arguments arrive as strings over the wire. - Result: return an
invoke.Result(amap[string]any); the framework redacts nothing for you, so do not put secrets in it. Return an error to surface a per-instance failure. - Streaming: for an operation that emits results over time, use
.Stream().HandleStream(fn)with a handlerfunc(ctx, invoke.Args, emit invoke.StreamEmit) errorthat callsemitper result untilctxis done oremitreports the client gone. It is reached viaInvokeStream; the built-inlogsinvocation is an example.
Configuration
| Key | Type | Default | Description |
|---|---|---|---|
admin_group | string | - | AdminGroup is the directory group whose members may step up. Unset disables the Admin API entirely. source |
admin_ttl | string | defaultAdminTTL | AdminTTL is the lifetime of a minted admin token (default 15m). source |
impersonation_ttl | string | - | ImpersonationTTL is the lifetime of a token minted by Impersonate. Unset, it is the token manager’s own, the same as a token from signing in: an impersonated session is used like any other, transfers included, and the admin token’s short lifetime would cut those off. source |
token_manager | string | "jwt" | source |
token_managers | map[string]map[string]any | - | source |
machine_auth_apikey | string | - | MachineAuthAPIKey enables Impersonate: the api-key the machine auth manager uses to mint a user-scoped token for a target user. Unset disables impersonation. source |
socket | string | - | Socket is the Unix socket path serving local root: callers authenticated by OS credentials (SO_PEERCRED) instead of a token. Empty uses the well-known default path; “off” disables it. source |
socket_group | string | - | SocketGroup restricts local root to members of this Unix group; unset leaves the socket’s file permissions as the only gate. source |