I don't think there is anything equivalent to Treasury Direct. Treasury Direct offers access to securities from one issuer: The United States Treasury.
To have direct access to muni bonds, you would either have to have a platform that could host multiple municipal entities, or a few competing platforms, or... I dunno. I don't think such a thing exists.
New issue munis are offered through a system that is conceptually similar, but not identical, to an IPO for stock or corporate bonds. It isn't exactly underwriting, but certain brokers participate in the offering of new issues.
And you can enter an order for those new issues through one of those brokers.
Like, for example, Charles Schwab.
I'm the guy who pointed you to the agency bonds search function on schwab.com.
On that same page, the drop down menu also has an option for municipals. And once you select that option, a link will appear next to the drop down menu that says "New Issue Calendar"
And you can see a list of upcoming offerings. For each offering, there is a link to something called a POS, or Preliminary Official Statement (analogous to a prospectus).
Near the right edge of the screen, there is a link to a "New Issue User's Guide"
For some bonds you may have to call Schwab and place the order by phone.
I have never placed an order for a new issue. I have no idea what the commissions are, or what the minimum order size is.