MTA-STS (SMTP MTA Strict Transport Security) is a standard that lets a domain require inbound mail to be delivered over authenticated TLS. Without it, SMTP encryption is opportunistic and can be stripped by an attacker on the network path without anyone noticing.
The three records
A working setup has a DNS TXT record at _mta-sts.yourdomain pointing at a policy version, a policy file at https://mta-sts.yourdomain/.well-known/mta-sts.txt that lists your MX hosts and a mode, and — strongly recommended — a TLS-RPT TXT record at _smtp._tls.yourdomain that names an address to receive TLS failure reports.
Modes
The policy mode is one of none, testing or enforce. In enforce, a sender that cannot establish valid TLS to a listed MX will not deliver the message. Begin in testing, read your reports, then switch to enforce.
Common mistakes
The most frequent problems are a policy file that is missing or served without a valid certificate, an mx list that does not match the domain’s real MX records, and forgetting to bump the policy id in DNS after editing the file. All of these are surfaced by the checker.