Custom SMTP Setup for Cold Email: A Step-by-Step Guide

Most cold email tools want to send through your mailbox one of two ways: a one-click OAuth connection, or a set of SMTP credentials you paste in by hand. The OAuth path is cleaner when your provider supports it. But plenty of setups don't fit that mold: a self-hosted mail server, a third-party relay, a niche hosting provider, or a Microsoft 365 account where the admin disabled modern auth. For all of those, custom SMTP setup for cold email is the path that actually works.

This guide is the credential-based version. If you run Google Workspace and just want the easiest connection, OAuth is usually the better call, and our walkthrough on setting up multiple Google Workspace accounts covers that flow end to end. Everything below is for when you need to wire SMTP directly: the five settings you need, the port and TLS choices that matter, the exact provider values for Gmail and Microsoft 365, and the bring-your-own-relay caveats that decide whether your mail lands in the inbox or the void.

Key Takeaways

• Custom SMTP setup needs five values: host, port, encryption type, username, and password. Get the port and encryption pair right and the rest is data entry.
• Use port 587 with STARTTLS or port 465 with implicit TLS for submission. Port 25 is for server-to-server relay, not for connecting an app, and most clouds block it outbound by default.
• Gmail needs a 16-digit app password (and 2-Step Verification on). Microsoft 365 needs SMTP AUTH explicitly enabled, since it's off by default for tenants created after January 2020.
• A bring-your-own SMTP relay can work, but a shared-IP pool means you inherit strangers' reputation. Verify deliverability on a small batch before you scale.

What custom SMTP setup for cold email actually means

SMTP (Simple Mail Transfer Protocol) is the language mail servers use to hand messages to each other. When your cold email tool "sends" an email, it isn't magic. It opens a connection to a mail server, authenticates, and submits the message for delivery. Custom SMTP setup just means you're pointing the tool at a specific server using credentials, instead of letting it broker the connection through an OAuth token.

You'd reach for custom SMTP in a handful of situations:

• You self-host mail or run your own mail server and want full control over the sending path.
• You use a third-party SMTP relay or a hosting provider whose mailboxes only expose SMTP credentials, not OAuth.
• Your provider supports OAuth, but it's blocked. A Microsoft 365 admin may have disabled modern auth integrations, leaving SMTP AUTH as the only door.
• You want one consistent connection method across mixed providers instead of managing separate OAuth grants per platform.

The tradeoff is honesty time: OAuth connections are scoped, revocable, and don't expose a reusable password. Custom SMTP hands a credential to the tool, so you're trusting that the credential is stored securely and that the connection is encrypted in transit. That second part is where the port and TLS choices come in, and where a lot of setups quietly go wrong.

One thing custom SMTP does not change: the deliverability fundamentals. Authentication records, warmup, and sending volume still govern whether your mail reaches the inbox. SMTP is the pipe. What flows through it is judged on its own. If your SPF, DKIM, and DMARC aren't aligned with the server you're sending through, custom SMTP won't save you.

The five settings you need before you start

Every SMTP connection comes down to five fields. Gather these before you touch the tool, and the actual setup takes two minutes.

The host and the username/password come straight from your provider. The two that trip people up are port and encryption, because they have to match. Picking port 465 with STARTTLS, or port 587 with implicit SSL, gives you a connection that either hangs or fails with a confusing handshake error. So before you fill in the form, it's worth understanding what each port actually is.

SMTP port 587 vs 465 vs 25: which to use and the TLS configuration

There are four ports you'll see in SMTP documentation, and only two of them belong in a cold email setup. Here's the full picture, then the rule.

Port 587 is the standard submission port. It's reserved specifically for clients submitting mail, and the standard requires that the server demand authentication by default, which is exactly what you want. The connection starts in the clear, then the STARTTLS command upgrades it to an encrypted channel before any credentials cross the wire. That's the smtp port 587 TLS configuration you'll select in most tools: "STARTTLS" or "TLS."

Port 465 does the same job a different way. It's registered as the "submissions" port for SMTP submission over implicit TLS, meaning the TLS handshake happens immediately on connect, before a single SMTP command is exchanged. There's no upgrade step to strip, which is why the standard expresses a preference for implicit TLS where both are available. In your tool, this maps to the "SSL/TLS" encryption option. Functionally, 587-with-STARTTLS and 465-with-implicit-TLS land you in the same place. Pick whichever your provider documents.

Port 25 is the one to avoid for this purpose. It's the original SMTP port, now used for server-to-server relay between mail transfer agents, and the standards explicitly steer user submission away from it. It gets worse in practice: most cloud providers block outbound port 25 by default. AWS, for example, restricts outbound traffic on port 25 for EC2 instances unless you request removal, while ports 465 and 587 aren't restricted. Many residential ISPs and firewalls do the same, because port 25 is the classic spam vector. If a setup guide tells you to use port 25 to connect an app, it's outdated.

Port 2525 is the escape hatch. It isn't an IETF standard, but several providers and hosts honor it as a mirror of 587 with STARTTLS, specifically for environments where 587 is blocked. Only use it if your host blocks 587 and your provider explicitly supports 2525. Otherwise, stay on 587 or 465.

The rule, simplified: start with 587 (STARTTLS). If your provider prefers it, use 465 (SSL/TLS). Reach for 2525 only when 587 is firewalled. Never use 25 to connect a tool.

Step-by-step: connect custom SMTP to your cold email tool

With your five values in hand, the actual setup is short. The labels vary between platforms, but the fields are universal. Here's how to connect custom SMTP to a cold email tool without the trial-and-error.

1. Open the account connection screen and choose the custom SMTP, IMAP/SMTP, or "connect via credentials" option rather than the one-click provider button.
2. Enter the SMTP host exactly as your provider documents it. Use the hostname (smtp.office365.com), never a raw IP address, so TLS certificate validation works.
3. Set the port and encryption as a pair. 587 with STARTTLS, or 465 with SSL/TLS. Don't mix them.
4. Enter the username, which is almost always the full email address of the sending mailbox, not just the local part.
5. Enter the password. For most providers this is an app password or relay key, not your everyday login password (more on that next).
6. Add the IMAP settings too if the tool asks. Cold email tools read replies over IMAP (typically port 993, SSL) so they can detect responses and pause sequences. SMTP sends; IMAP receives. You usually need both.
7. Send a test email to a mailbox you control, ideally on a different provider, and confirm it arrives and renders correctly.

If the test fails, the error message almost always points at one of two things: an authentication rejection (wrong password, or auth not enabled on the mailbox) or a TLS handshake failure (wrong port/encryption pair). We'll cover the specific fixes below.

A quick note on volume: connecting the mailbox is step one, not the finish line. How fast you ramp sending through that connection is what protects the mailbox's reputation. Set conservative throttling and ramp settings per inbox, and respect a safe daily volume well below the provider's hard cap.

Provider-specific SMTP credentials: Gmail and Microsoft 365

The hostnames and ports are easy. The credentials are where each provider adds its own rules. Here are the two you'll hit most often.

Gmail and Google Workspace. Google's documented SMTP server is smtp.gmail.com, using port 465 for SSL or port 587 for TLS, and it authenticates with your full email address plus an app password. You can't use your normal account password. Google removed the "less secure apps" option, so an app password is now the credential for SMTP. That's a 16-digit passcode, and you can only generate one if 2-Step Verification is turned on for the account. Generate it in your Google account security settings, paste it as the SMTP password, and you're connected. For Google Workspace specifically, OAuth is often the cleaner route, which is why we route that flow to the Workspace setup guide. Either way, Gmail and Workspace enforce their own daily send caps, covered in our Gmail sending limits breakdown.

Microsoft 365. Microsoft's client SMTP submission uses the host smtp.office365.com on port 587 with STARTTLS, and the device must support TLS 1.2 or TLS 1.3. Two gotchas here. First, Microsoft explicitly warns that if your tool defaults to port 465, it doesn't support the TLS versions required for client SMTP submission, so stick to 587. Second, SMTP AUTH is disabled by default for organizations created after January 2020, and it has to be enabled per-mailbox before any credential will authenticate. On top of that, Microsoft is deprecating Basic authentication for client SMTP submission and steering senders toward OAuth, so check the current state of your tenant before you build a workflow on a password. Microsoft also throttles this path at 10,000 recipients per day and 30 messages per minute per mailbox, which is plenty for one inbox but worth knowing when you plan volume.

For any other provider, the pattern is the same: find their documented SMTP host, the 587/465 port pair, and whether they want your mailbox password, an app password, or a generated API/relay key. When in doubt, their support docs will list it under "SMTP settings" or "send mail from an app."

BYO SMTP deliverability caveats for cold email

This is the section other guides skip, and it's the one that decides whether custom SMTP helps or hurts. Connecting a bring-your-own SMTP relay is trivial. Getting good deliverability out of it is not automatic. Here are the byo smtp deliverability cold email traps, in the order they bite.

You inherit the relay's IP reputation. If your SMTP provider routes you through a shared IP pool, your inbox placement rides on the behavior of every other sender in that pool. A clean list and perfect copy won't save you if a neighbor is blasting spam from the same IP. Mailbox providers judge the sending IP, and a poisoned shared pool drags everyone down with it. The decision of when a dedicated IP is worth it, and when it backfires, is its own topic, covered in our breakdown of dedicated vs shared IPs for cold email.

Cloud and ISP port-25 blocks aren't just an inconvenience. They're a signal. If you're self-hosting on a VPS and find port 25 throttled, that's the provider telling you the IP space wasn't built for sending. Even when you get 465 or 587 working, mailbox providers can and do block entire cloud IP ranges that send directly, which is why running production cold email straight off a generic compute instance is a fragile plan.

SPF alignment has to match the actual sending server. When you send through a custom relay, the message leaves from that relay's IP. Your SPF record has to authorize it, and your DKIM signing has to be set up on the domain you send as. A mismatch here means failed authentication on every message, and after Google and Yahoo tightened bulk-sender rules, failing authentication is a fast route to the spam folder. Re-verify your authentication setup against the new sending path, don't assume the records you wrote for a different server still hold.

Transactional relays and cold email are a poor fit. A relay tuned for password resets and receipts optimizes for speed and uptime, not for the engagement-driven reputation cold email needs. Some providers also prohibit cold outreach in their terms. Read the policy before you route prospecting mail through a transactional service, or you'll wake up to a suspended account.

The honest summary: custom SMTP gives you control, and control cuts both ways. A well-run dedicated setup can outperform a managed connection. A cheap shared relay on a blocklisted IP range will underperform a basic OAuth mailbox every time. Before you commit a campaign to a BYO relay, send a small batch and run an inbox placement check to see where it actually lands.

Test and troubleshoot your SMTP connection

Most SMTP setup failures throw one of a few errors. Here's how to read them.

• "Authentication failed" / 535 error. Wrong username or password is the usual cause. For Gmail, confirm you're using the app password, not the account password. For Microsoft 365, confirm SMTP AUTH is enabled on that specific mailbox, since it's off by default on newer tenants. Double-check the username is the full email address.
• TLS or handshake errors / connection hangs. Your port and encryption don't match. Switch 465 to SSL/TLS or 587 to STARTTLS. If you're on Microsoft 365 and set 465, change to 587, which is the supported pair there.
• "Connection timed out" on port 25. You're using the wrong port, or your network blocks it outbound. Move to 587. If 587 is also blocked, try 465, then 2525 as a last resort if your provider supports it.
• Mail sends but lands in spam. That's not an SMTP problem. It's a reputation or authentication problem. Check that the sending IP is authorized in SPF, that DKIM signs for your domain, and that the mailbox is properly warmed. Our deliverability audit checklist is the structured way to find the leak.

Once the connection is live and a test message inboxes cleanly on a separate provider, you're done with setup. From there, the job shifts to keeping the connection healthy: monitor the mailbox, warm it before you scale, and watch the reputation signals our guide on monitoring domain health lays out.

Common questions about custom SMTP setup

Should I use custom SMTP or OAuth for cold email?

If your provider supports OAuth and your tool offers it, OAuth is usually the better choice: it's scoped, revocable, and doesn't expose a reusable password. Use custom SMTP when OAuth isn't available, when you self-host or use a third-party relay, or when an admin has disabled modern auth and SMTP AUTH is the only path. Both can deliver well. The connection method matters less than the reputation behind it.

What port should I use for SMTP cold email?

Port 587 with STARTTLS is the default, standards-backed choice for message submission. Port 465 with implicit TLS is equally valid and often preferred when available. Avoid port 25, which is for server-to-server relay and is blocked outbound by most clouds and many ISPs. Use port 2525 only as a fallback when 587 is firewalled and your provider supports it.

Why does my Gmail SMTP login keep failing?

Almost always because you're using your account password instead of an app password. Gmail's SMTP server requires a 16-digit app password, and you can only generate one with 2-Step Verification turned on. Turn on 2-Step Verification, generate the app password in your account security settings, and use that as the SMTP password with your full email address as the username.

Does custom SMTP improve deliverability?

Not by itself. Custom SMTP is the delivery pipe, not the reputation. It can improve deliverability if it gives you a clean, dedicated sending IP with correct authentication. It can hurt deliverability if it routes you through a shared, blocklisted IP pool or a relay that blocks cold email in its terms. Test a small batch and check inbox placement before you trust a new SMTP path with a campaign.

The bottom line

Custom SMTP setup for cold email is five fields and one rule: match the port to the encryption. Use 587 with STARTTLS or 465 with implicit TLS, give the tool your host and credentials, and add IMAP so replies get detected. For Gmail that means an app password with 2-Step Verification on. For Microsoft 365 it means SMTP AUTH enabled on the mailbox and port 587, not 465.

The setup is the easy part. The judgment is in the relay you point it at. A dedicated, well-authenticated server can beat any managed connection. A cheap shared pool on a blocked cloud range will lose to a plain OAuth mailbox every time. Wire it up, send a test, verify placement, then ramp slowly. MailBeast supports both OAuth and custom SMTP connections, so you can use whichever fits each mailbox, and pair it with built-in warmup and deliverability monitoring so the connection you just configured keeps landing where it should.

Sources

• IETF, RFC 6409: Message Submission for Mail
• IETF, RFC 8314: Cleartext Considered Obsolete (TLS for Email Submission and Access)
• Google Workspace, Send email from a printer, scanner, or app
• Google, Sign in with app passwords
• Microsoft Learn, How to set up a device or application to send email using Microsoft 365
• AWS re:Post, Remove port 25 restrictions for instances and functions
• Mailgun, Which SMTP Port to Use? Understanding ports 25, 465, 587