We’re about to finish up a major infrastructure redesign here at Webmail and one of the last parts involved is creating a new and improved logging system. In a nutshell, we need a centralized log database that will allow us to easily search and manage enormous log files on all of our servers. After a few months of development, we finally implemented a solution last week.
Without getting too technical here is how this works: every night, a Perl script runs for about an hour—that generates all of the statistical data for the previous day using the log data in the database. There’s also an Apache web server running a PHP-based query tool that allows us to run live queries on the databases whenever we need information. Well I at least tried not to get too technical.
So, how awesome is this? Very! Now, all of our servers send their log data to a single server whose sole purpose is to log all email data. Once the data gets across the network to the log server, it is inserted at a very fast rate into the database. This database then allows us to do two things:
- Create daily statistical information about the number of emails received, the number of viruses detected, the number of spam emails caught, etc. This information is made available for email administrators inside the Control Panel.
- Provide our customer support team with the ability to troubleshoot every single email that ever enters or leaves our system.
- As importantly, we learned a TON about large data sets which will play a huge role in our next infrastructure project, code-named Mercury (more on that later).
By the way, we insert roughly 500,000 records in about 30 seconds of real time. Not too shabby.