Handling email notifications in PHP

12 Nov 2011

Twitter sends notifications of new followers and direct messages by email. While you can use the API to keep track of those things it wastes hits and it’s incredibly inefficient especially for accounts with minimal activity.

I run several Twitter-based services but they all share the same email address. This allowed me to use the same email handling setup to process all email coming in from Twitter. So far it’s working really well but it means this script is a bit more involved than it needs to be.

I’ve littered the code with comments – far more than should be necessary, but it saves me from going through it step by step here. Read, learn and enjoy. If you have any questions or comments feel free to contact me on Twitter.

incoming_mail.php

#!/usr/bin/env php
<?php
  // The email address notifications should be sent to
  $____email_address = 'contact@twitapps.com';

  // This array maps incoming emails to scripts
  $____processor_map = array(
    // For the ta_follows user there's a separate script for each notification type
    'ta_follows' => array(
      'is_following' => dirname( __FILE__ ).'/ta_follows/new_follower.php',
      'direct_message' => dirname( __FILE__ ).'/ta_follows/new_dm.php',
    ),
    // For this user one script handles all notifications
    'ta_replies' => dirname( __FILE__ ).'/ta_replies/process_email.php',
  );
  
  // CONFIGURATION ENDS - NOTHING BELOW THIS LINE SHOULD NEED CUSTOMISATION

  // OUTPUT HANDLER
  // It's important that this script doesn't output anything, so this chunk of code emails it somewhere instead
  ob_start('EmailOutput');
  register_shutdown_function('EmailOutput');
  function EmailOutput($str = false)
  {
    static $output = '';
    if ($str === false)
    {
      mail($GLOBALS['____email_address'], 'Output from incoming_mail.php', $output, 'From: Incoming Email <'.$GLOBALS['____email_address'].'>', '-f'.$GLOBALS[' ____ email_address']);
      $output = '';
    }
    else
    {
      $output .= $str;
    }
  }
  // END OF OUTPUT HANDLER

  // Get the email content
  $____data = file_get_contents('php://stdin');
  
  // Parse it
  $____msg = mailparse_msg_create();
  if (!mailparse_msg_parse($____msg, $____data))
  {
    echo "Failed to parse message\n\n".$____msg;
  }
  else
  {
    // Get the bits we need
    $____message = mailparse_msg_get_part($____msg, 1);
    $info = mailparse_msg_get_part_data($____message);
    
    if (!$____message or !$info)
    {
      echo "Failed to get message or info\n\n".$____msg;
    }
    else
    {
      $headers = array();
      $____prefix = 'x-twitter';
      $____prefixlen = strlen($____prefix);
      foreach ($info['headers'] as $____key => $____val)
      {
        $____key = strtolower($____key);
        if (substr($____key, 0, $____prefixlen) == $____prefix)
        {
          $headers[substr($____key, $____prefixlen)] = $____val;
        }
      }

      // Make sure it's a message from Twitter
      if (count($headers) == 0)
      {
        echo "Missing required headers\n\n".$____msg;
      }
      else
      {
        // Now get the message body and clean it up a bit
        ob_start();
        mailparse_msg_extract_part($____message, $____data);
        $body = ob_get_clean();
            $body = urldecode($body);
        if (!empty($info['charset'])) $body = iconv($info['charset'], 'UTF-8', $body);
            $body = html_entity_decode($body, ENT_NOQUOTES, 'UTF-8');
        
        // Now attempt to find a handler for this notification
        $____handler = false;
        // First do we handle this user at all?
        if (isset($____processor_map[$headers['recipientscreenname']]))
        {
          // Is it a single script or one per type?
          if (is_array($____processor_map[$headers['recipientscreenname']]))
          {
            // Do we have one for this specific type?
            if (isset($____processor_map[$headers['recipientscreenname']][$headers['emailtype']]))
            {
              // Set the handler
              $____handler = $____processor_map[$headers['recipientscreenname']][$headers['emailtype']];
            }
          }
          elseif (is_string($____processor_map[$headers['recipientscreenname']]))
          {
            // Single script, set the handler
            $____handler = $____processor_map[$headers['recipientscreenname']];
          }
        }
        
        // Did we find a handler?
        if ($____handler === false)
        {
          echo "No appropriate handler found\n\n".$____msg;
        }
        // Does it exist?
        elseif (!file_exists($____handler))
        {
          echo "Handler found but file missing: '".$____handler."'\n\n".$____msg;
        }
        // All good so do it!
        else
        {
          // We catch output so we can pre and postfix it with useful info
          ob_start();
          require $____handler;
          $____output = ob_get_clean();
          if (strlen(trim($____output)) > 0)
          {
            // Output caught, say it was from the handler script and include the raw message after it
            echo "Output from handler: '".$____handler."'...\n\n".$____output."\n\n".$____msg;
          }
        }
      }
    }
  }
  mailparse_msg_free($____msg);

new_follower.php

<?php
  // Note that we handle the case where we're already following a user and don't try to re-create them
  $user = Twitter::FollowByID($headers['senderid']);
  if (isset($user['error']) and stripos($user['error'], 'already on your list') === false)
  {
    // Something went wrong but it was not because we're already following this user
    echo 'Follow failed for "'.$headers['senderscreenname'].'" ('.$header['senderid'].'): '.$user['error'];
  }
  else
  {
    // So we're now following them - now create the user if they don't already exist
    if (!User::Exists($headers['senderid']))
    {
      $result = User::Create($headers['senderid']. $headers['senderscreenname']);
      if ($result === true)
      {
        // Created OK, send them a welcome DM
        if (!Twitter::DM($headers['senderid'], 'Welcome to Follows from TwitApps. Send your email address by direct message to @ta_follows to activate this service.'))
        {
          echo 'Failed to send DM to "'.$headers['senderscreenname'].'" ('.$header['senderid'].')';
        }
      }
      else
      {
        // Couldn't create the user!
        echo 'Failed to create user "'.$headers['senderscreenname'].'" ('.$header['senderid'].'): '.$result;
      }
    }
    else
    {
      // User already exists - this handler does nothing in this situation, but it could!
    }
  }

The script uses the mailparse extension so that must be available.

To use the script you just need to stick it somewhere accessible by the mail server, check the #! line will work, make it executable and add a piped alias to your aliases file – see your MTA documentation for details. My server runs Postfix and I have the following line in /etc/aliases…

twitmail: |/var/www/twitapps.com/common/incoming_mail.php

In my case I also have a line in the transport map to point the full email address including domain at that local alias. Your MTA requirements will vary.

!! Be sure to keep this email address secret !!

Due to the way email works there is no real way to validate that the email came from Twitter, and as a result nor is it reasonable to assert that the user mentioned in the email actually did what it says they did. The only real way to protect against fake emails at the moment is to ensure you use an email address that nobody else knows. Apply the same rules to this email address as you would to a password and you’ll be fine. It’s not great security but it’s what we’ve got and I’ve not heard of anyone having a problem so far but you don’t want to be the first!!

Writing handlers

The last piece of this solution is the actual handler scripts. There’s nothing special about them – they can do whatever you need them to do. The second script in the embedded gist above is an example is_following handler.

From within a handler script you have a few useful variables available from the main script…

$headers

This is an array of all the headers from the email that started with X-Twitter. At the time of writing this give you access to…

createdat
When the email was created

emailtype
What type of notification it is, e.g. is_following or direct_message

senderid, senderscreenname, sendername
The ID, screen name and name of the sender

recipientid, recipientscreenname, recipientname
The ID, screen name and name of the recipient

$body

The unmodified body of the message. Note that for direct messages this contains more than just the message itself.

$info

This variable contains the return value from mailparse_msg_get_part_data. The most useful part of this is the raw headers from the incoming email ($info[‘headers’]).

Errors and other output

If anything goes wrong during the processing of a message an error message along with the raw incoming email will be sent to the address in $____email_address (specified near the top of the file). This includes any output from the handler scripts themselves. The script will never bounce an email due because you don’t want Twitter to decide the address is not valid.

Caveat

There is one important point regarding email notifications from Twitter that you need to be aware of, and that’s the fact that you may not always get a notification for every event that happens.

The reason for this is spam. Twitter is still battling against a deluge of users and bots whose only purpose is to try an spam the legitimate users of their service. One of their defences against this is to recognise spamtastic patterns and take action to minimise the impact on other users. One of the actions they take is to not send email notifications from users flagged as possible spammers.

One important side-effect of this is that a user unfollowing you and then following again will not necessarily trigger an is_following notification.

For TwitApps I have a cron job that runs once a day and checks the followers using the API and “catches up” with any that have been missed. This means most users get an instant response but there’s a backup process that makes minimal use of the API to ensure that if anyone does get missed they only wait up to 24 hours for a response.