OpenSMTPD table protocol changes, now with the backstory

Regular readers will be aware that OpenBSD ships with its own mail server implementation, OpenSMTPD, in its base system.

In a recent message to the tech@ mailing list, Omar Polo (op@) asked for comments or oks for a patches implementing a change of table protocols. A little later, Gilles Chehade (gilles@) posted to the misc@opensmtpd.org mailing list with the backstory for this change.

The message follows in full below (apparently the otherwise fine marc.info archive site no longer archives the list):

Date: Fri, 03 May 2024 08:22:03 +0000
From: gilles@poolp.org
To: misc@opensmtpd.org
Subject: smtpd: change the table protocol

Hello,

This is a copy of a mail I sent to OpenBSD hackers a few days ago so you are aware of work
being done on OpenSMTPD by Omar Polo.

~~~

TL;DR: proposal to change table backends wire protocol to one that's closer to filters, it
       has proven to work for years now, comes with many benefits and it is a very trivial
       change that we can pull in a handful of hours:
       https://tmp.omarpolo.com/smtpd-tables.7.html

Longer mail with historical context.

A long time ago, OpenSMTPD shipped with two table backends for its table API: file and db.

Then we started seeing a need for ldap and SQLite, so we wrote backends for them and since
we had aldap.[ch] and SQLite in base this wasn't really a problem.

Then we started seeing a need for mySQL and postgreSQL which became problematic because we
didn't want OpenSMTPD to ship with backends that couldn't be built in base out of the box,
but also because if we had to fix a bug in table_mysql, it meant we had to do an OpenSMTPD
release to provide the fix for portable users.

To solve this, we introduced the "procexec" API, a mean for OpenSMTPD to spawn tables in a
separate process with socketpairs in place to perform lookups and obtain results via imsg.
This made it possible to release table backends separately as standalone programs, without
limitations to their dependencies and without requiring releases to be synchronised. Table
backends were releases as part of OpenSMTPD-extras, and we could do several releases of it
in the timespan of a single OpenSMTPD release.

Later, we did the same for filters, it turned out to be much more complex and we failed so
hard our first attempts that we reverted and waited two years to make a new proposal. That
new proposal replaced imsg with an stdio line-based protocol which was much simpler for us
to implement in the daemon and it had the benefit that users could easily interact with it
to handle use-cases we didn't care about using the languages they wanted. I have seen some
filters in awk, rust, C, golang, Perl, Python, ... users are happy with the simple API and
we didn't have to handle exotic use-cases in the daemon, which is good.

Recently there's been a few contributions to table-ldap in OpenSMTPD-extras, the first one
in years, and I think that the fact table backends still use "procexec" is a reason why we
can't rely on the community to contribute: you can't just read a line, parse it and reply,
instead you need to know how imsg works, you need to understand non-blocking async I/O and
you can't start a table backend of your own in a preferred language... so unless one of us
does the work, it stales (I wrote table_ldap ... I don't even use or care for ldap).

A while back I told eric@ we could switch tables to a filter-like protocol, he agreed, but
we never got to do it. Recently, as op@ began contributing more and more, I told him about
that idea again and my 3-steps plan to switch to it:

1- implement a table_stdio backend which translates imsg protocol to stdio protocol
2- convert all table backends to the new protocol
3- move the table_stdio code in smtpd so we no longer need table_stdio

These are all trivial changes and op@ actually already implemented 1- and 2-, we'd like to
start the work on 3- considering that all of this project is easy: we're changing the wire
protocol but the table API remains 100% the same from OpenSMTPD and filters perspective.

The new table protocol is identical to the smtpd-filters(7) protocol (draft by op@), it is
readable in HTML at the following link and I've included the man page at end of mail:

    https://tmp.omarpolo.com/smtpd-tables.7.html

The operations are exactly the same as with imsg, and since the imsg protocol already made
results available as serialised strings, we already parse them from string to structs, and
we'll re-use the same code. The change is mainly in not encapsulating them in imsg packets.

~~~

Note that we're past this as of today:

1- table_stdio was implemented as a PoC:
  - https://github.com/OpenSMTPD/table-stdio

2- all table backends from OpenSMTPD-extras were converted to the new interface:
  - https://github.com/OpenSMTPD/table-socketmap
  - https://github.com/OpenSMTPD/table-passwd
  - https://github.com/OpenSMTPD/table-ldap
  - https://github.com/OpenSMTPD/table-sqlite
  - https://github.com/OpenSMTPD/table-redis
  - https://github.com/OpenSMTPD/table-postgres
  - https://github.com/OpenSMTPD/table-mysql

3- Omar has sent a mail to tech@ with the diff to integrate protocol in smtpd:
  - https://marc.info/?l=openbsd-tech&m=171439783225288&w=2


There you go, now you know !