ArghWebWorks

I’m working on a couple of projects; I’m keeping the web development projects I’m doing (like this site, for instance) in subversion repository. I’m trying to get better at this, but some of the instructions out there assume that you actually know what you’re doing.

So, here’s how to include another repository in your working copy; I’m building a ‘common library’ of code, that I update, but I plan to maintain as its own entity; that way I won’t have multiple copies of various things running around.

First, checkout your working copy (assumes you have a repository already set up. Happily, there are many tutorials on that).
$>svn checkout file:///var/subversion/repositoryname
Then go into this working copy and use ‘svn propedit’ to edit the properties of the root directory:
$>svn propedit svn:externals .
This’ll bring up an editor window for you. Place into it a tab delimition of directory and repository:
(in the file)
phplibs file:///var/subversion/libraryrepository
(out of the file)
now exit, and run svn update to import the externals.
Voila- you’re done! Pat yourself on the back.

So… you need to remove some addresses from your database…

Here’s how you do it.

Set the 'Errors-To' header in your email to something unusual... like 'zyxbounced@(your domain name ).' In your email server's virtuser's table, you want to pipe this to some script on your server. But you can't reference a script or other program in the virtuser table. So you use an alias: set in your virtusertable "zyxbounced@(your domain name): bounceremoval" and rebuild the virtusertable ( cd /etc/mail; make map hash virtusertable < virtusertable ). Then edit your aliases file (/etc/aliases) and you can set the forward to a script here. ( bounceremoval: "|/usr/local/bin/removalscript.pl" ) and rebuild the aliases database ( newaliases ).
Then write the /usr/local/bin/removalscript.pl script. You can probably look for the error message '5.7.1' and figure out who the message was to, and go from there. Once that's done, you need to soft link this file to the /etc/smrsh directory in order to give the sendmail restricted shell permission to run it.

Hurray! You’re done!

Wow. It’s taken me about six hours, all told, to get this thing working. But I’ve

1. upgraded snort to 2.4.4 and reinstalled (tip: make clean,./configure (with options), make, make uninstall, make install), using a unified (and thus fast) output plugin.
2. upgraded barnyard to 0.2.0 and reinstalled, reading the unified output from snort and outputting to a log file, to syslog and to a database on a second server.
3. reconfigured guardian to use the log file from barnyard and block files in there.
4. finally gotten BASE to read the alert entries from the database server on a third server ( the intranet server ).

A couple of tips along the way:

1. If your BASE can’t read the database, but there’s definately stuff in there, try taking the ‘detail full’ off of the _barnyard_ configuration output alert_acid_db entry. If that doesn’t work, try manually inserting an entry in the sensor table.
2. You don’t need to have both an alert_acid_db entry and a log_acid_db entry for the same ‘sensor.’
3. barnyard’s acid_db entries use the same database config as snort would if you were logging directly from snort. So use the snort source/schemas creation script (/usr/local/src/snort-2.4.4/schemas/create_mysql ) to create the tables in the database. When you set up BASE, it offers to create the extra ACID tables.

I was making a case yesterday that sendmail/mimedefang/clamd/spamassassin weren’t causing a heavy load on our systems. I had to back up my claims with some cold facts, so here’s what I put together.
First, I ran “sar” to get the system activity report; it’s some statistics that are taken every 10 minutes throughout a day on our Fedora Core 4 server. Then I grepped for today’s mail information in the maillog file (grep ^Feb\ 16 maillog) and saved it to another file. Then I ran this script to match total processed mail messages against the system activity report; I ran the script and saved the output in a file, then used “paste sarreport mailreport > foo” to get the whole report.

Sometimes my tools are just too much fun.


open( FH, "maillog.stripped");

$remote=$relay=$file=$local=$total=0;
$stattime=$checktime;
}
}
}
close( FH);
(END)


The server’s upward spiral for system load continued; we’re currently examining the qpopper process. What it does is copy the mailbox file over to a temporary place, parse the copy, and then if the user leaves mail on the server copies it back. Some of the users have over 20MB of saved mail. People, we’re not gmail here. So I’ve been writing a perl script to weed out some older files; I’m letting them keep 14 days of stuff, and only checking files over a certain size.

Here’s the completed script; obviously I’m not a perl master.


#!/usr/local/bin/perl

When you move to bring your archaic Named configuration files into a database, the mysql bind sdb patch looks like a terrific idea.

It’s pretty slick, but there are a couple of caveats; First, no wildcard entries are allowed; you can’t specify a wildcard server name and have mysql-bind track down the right server. As well, there’s no documentation as far as indexing the database files for better results. So, as the documentation specifies, having each domain in its own database table is probably the best (it offered me, on testing, a tenfold improvement over one main table).

Sendmail keeps its email in one big file, and as a mail-user reads their mail and elects to ‘leave messages on server,’ these files can get pretty large. I’d like to limit their size, but if I use quotas, it’ll bounce new messages. That won’t work on the mailboxes that are the worst offenders; one in particular is the destination of some automated scripts. So I need a way to remove the oldest messages.

Here’s the beginning of such a script

#!/usr/local/bin/perl

Our qmail client called up and asked me, since they had a huge backlog of mail and it all looked like spam to non-existent users, how they could just automatically round file it; sending it to /dev/null.

I recalled that you could specify /dev/null in the .qmail-default file, but that just putting in one line, commented out, was faster and just as good (It convinces qmaiil that all of the delivery options have been performed). So I instructed them to do that. A few hours later, the customer called back; they weren’t receiving any email for some aliased accounts, but others were working ok.

As it turned out, I needed to specify ‘| /home/vpopmail/bin/vdelivermail ” /dev/null’ rather than just “#”; because of the combination of qmail and vpopmail. So remember, if you’re using qmail+vpopmail, everything changes, including default delivery to /dev/null.

I’m not here to sing the death of Sendmail. Let’s just say that I’m tired of editing arcane files in order to compile even more arcane files. I’d like some closer sense of what the email server’s doing, and frankly, Sendmail and I aren’t the best of friends. We can work together, but we can’t really dance or sing.

My boss’s business hosts domains (like most do, these days), and so we have a whole bunch of pre-existing users, domains, forwards and autoresponders. We have been creating system users for each of these users, which can confuse the matter. I’ve already gotten ftp to read the database information for the users so we didn’t need shell accounts for that, and I wanted to do the same for email. Seems like a fairly simple idea; and yes, some people have walked this road before. I chose “qmail” for our system based on its speed and ease-of-configuration.

I strongly recommend the Life With Qmail site. He’s done a terrific job making it way too easy to install qmail. If you plan to incorporate qmail and mysql, however, first go through his setup. You can even patch vpopmail to use mysql. I don’t recommend patching both qmail and vpopmail; since both have hardcoded information in their configurations, it was very difficult to get them to talk to each other. I wound up receiving email ok for users in the database, but they couldn’t get their email. They could check it, but it wasn’t there. In order to keep from maintaining 3 database tables (the original hosting one, the vpopmail one and the qmail one). To simplify matters, I ripped out the qmail and started over without the mysql-patched version (but kept the mysql-patched vpopmail).

To be continued …

A client came to me with a need for a new webmail application. They were tired of constant telephone calls from their mail users, complaining of missing emails, lost connections, sending problems; they wanted to “fix” this by installing a new webmail service.

I looked at my trusty Squirrelmail and shook my head sadly. It’s dated and texty- kind of techy looking. I was looking for something a little more snazzy; maybe some DHTML, some graphics, you know, something to compare with the prettiness of gmail.

I stumbled across RoundCube and was totally blown away by their demo. Accompanying the download page was a slick,
thorough walkthrough. Ten minutes later, I was logging into their mail server with the roundcube interface. As an added touch, I installed squirrelmail into a seperate directory, to give them some other options. But I know which one I’d rather use.