College football is very popular at universities throughout the US. It was so popular at my former university that the email system would randomly but silently redirect student email addresses to an alias named after the football team.
(For the purpose of this article, let's call the university Trinity and the mascot of the football team the Eagles, since this is statistically likely.)
While studying for my PhD at Trinity, my email address was [my-name]@utrinity.edu. However, towards the end of my fourth year, I started receiving emails addressed to a mysterious alias: [my-name]@eagles.utrinity.edu. Not only had I had never heard of this email alias, but I had never even been to a Trinity Eagles football game. Nevertheless, all of a sudden, a large fraction of my colleagues had simultaneously started sending my email to this new (and honestly somewhat unprofessional) email address. Where did this new email address come from? And why did my colleagues start using this new address on the same day, four years into my PhD? Were they subtly hinting I needed to get out of the lab and go see a sporting event? I decided to investigate.
My methodology was scarce. As a lowly PhD student, my access to the internals of Trinity's email system was about as abundant as my access to the university president's bank account. Additionally, I couldn't seem to reproduce the problem myself. When I sent an email from my Trinity account to a non-Trinity email address and responded, it came addressed to [me]@utrinity.edu. I tried several different external email addresses, multiple times, but it never came addressed to the "eagles" alias. I also asked a colleague with a Trinity account to send me an email, but that email also came through as [me]@utrinity.edu. Thus, it seemed like I was stuck with good old-fashioned "diggin' through my email history to find a pattern".
First, I wanted to see how many other "eagles" addresses there were in my social circle. I had noticed them several times before, but assumed it was a separate mail service some students used, or else an intentional display of school spirit. I looked through my email history and discovered hundreds of other "eagles" email addresses just within my personal contacts. This problem was not just affecting me: it was impacting potentially thousands of people at the university, and stretched back several years. But upon closer examination, I noticed a strange trend: "eagles" addresses primarily belonged to 4+ year PhD students. It seemed as if undergrads, faculty, and PhD students in their first three years were unaffected. Furthermore, when I looked at older messages sent by these people, the problem did not begin to impact them until their fourth year at Trinity. There were a few exceptions, though: a handful of undergrads, faculty, and junior PhD students also had this mysterious "eagles" address, and a few senior PhD students were unaffected. Investigating further, I found that the problem started to impact undergrads immediately before they graduated, i.e., after they had been at Trinity for almost four years. I also noticed that many of the junior PhD students with "eagles" addresses had done post-bac research at Trinity before they started their PhD. So for the most part, this problem mostly impacted people who had been at Trinity for four years.
So what was so special about the four year mark? This forced football obsession started during my fourth year, and it seemed to be the same for my colleagues. I took a closer look at the date it started happening to me. The problem began on March 3rd, 2020, which is almost exactly 4 years after I officially accepted my offer at Trinity (March 4th, 2016). That's pretty suspicious.
However, the strange thing was that not everyone was affected after four years, and not all of my emails were addressed to the "eagles" address. Perhaps, I thought, my address was changed within the official university address book, which was causing confusion when people would look up my address. However, I confirmed that an LDAP lookup gave me the expected @utrinity.edu address. Additionally, when people replied to my emails sent from my normal @utrinity.edu address, sometimes the reply would come addressed to @eagles.utrinity.edu. This means the reply was probably not just a change in the address book.
Another interesting piece of this puzzle was the fact that only some of my colleagues had started sending email to this "eagles" alias. It did seem to be consistent, though: the colleagues who sent email to my "eagles" alias always used this alias, starting on March 3rd. I will call these people eagle-senders. Non-eagle-senders never used this address at all (unless they were responding to a group email to which an eagle-sender had already responded). So it seemed like there were two groups of people: eagle-senders, who always used the "eagles" alias, and everyone else, who never used it.
Fascinatingly, I was unable to find a single example of someone who was both an eagle-sender and afflicted by the problem. In other words, if you consistently send email to other people's "eagles" alias, nobody sends email to your "eagles" alias. I compiled a list of all of the eagle-senders in my social circle and studied it carefully. After staring at it until my vision began to blur, I found another pattern: most eagle-senders were affiliated with the biomedical sciences.
Let's stop for a moment to reflect again on the absurdity of this situation. Whenever a biomedical scientist emailed someone who had been at Trinity for at least 4 years, the recipient's email address was silently converted into an alias named after the football team. Either there was a crazy bug in the Trinity email infrastructure, or else a bunch of biomedical scientists were secretly conspiring to increase attendance at the football games.
I got in touch with an eagle-sender colleague and asked her if she could help me debug. (It turns out she had also never been to a Trinity football game, so my conspiracy hypothesis was looking more unlikely.) I explained the problem and asked her to check her address book for my name. Strangely, it was still listed as [me]@utrinity.edu. I asked her to send me an email, specifically making sure to send it to [me]@utrinity.edu. When I received the email, it was addressed to [me]@eagles.utrinity.edu. So, somewhere in between her sending it and me receiving it, the address was changed. Progress!
I also noticed one additional peculiarity about the email. Not only was my email address listed in the "To" field as "[me]@eagles.utrinity.edu", but my name was listed as "[me]@utrinity.edu". Going back to my email history, this was consistent across all of the "eagles" emails and I and everyone else had received. Instead of the expected "Jane Smith smithj@utrinity.edu" to field, it was instead showing up as "smithj@utrinity.edu smithj@eagles.utrinity.edu". This is in contrast to my actual full name being listed, as it always had been. I confirmed again with my colleague that my full name and proper "@utrinity.edu" email address were visible in the address book. So, somewhere in between the transit between email sent by eagle-senders and accounts impacted by this bug, the "To" field was edited.
I dug around on Trinity's IT website for documents directed towards biomedical scientists. I found what I was looking for: in order to maintain HIPAA compliance, anyone working with patient data had an email address managed by a different company. Normal Trinity email is managed by Google, but some email addresses were managed by Microsoft. Both of these email systems used addresses in the format [name]@utrinity.edu, so there is no a priori way to tell whether a given Trinity email address uses Microsoft, Google, or both. However, based on this finding, it appeared that if a Trinity Microsoft account emailed a Trinity Google account which was at least four years old, the address was converted to the "eagles" alias.
So at this point, it became clear that somehow this bug was related to the way that Trinity was managing to serve both Microsoft accounts and Google accounts on the same domain. It also seemed clear that it was doing so by modifying the "To" header, changing both the name and the email address. But what wasn't clear was why this only impacted people who had been at the university for at least four years. It also wasn't clear why this "four year" rule was sometimes broken: the problem impacted some people who had been at Trinity for less than four years, and did not impact some people who had been at Trinity for over four years. By contrast, the Microsoft -> Google rule was true in 100% of the cases I observed.
I dug around some more on Trinity's IT website and found several documents about the setup of their email system (including an internal Confluence which was probably not intended to be public...). The two services were managed through an internal relay server. All mail went to the internal relay, which then routed it to either Google or Microsoft, depending on the type of mail account the user has and whether HIPAA access is required. Perhaps this routing was disrupted.
Email messages all contain hidden metadata, which offer a useful insight to how the email was created and relayed. Emails may make several "jumps" when going from the sending server to the receiving server, and each of the servers it passes through is recorded in a distinct header labeled "Received:". Thus, these can be used to show how the email was routed, and provide a window into how emails are handled by Trinity's internal systems. Because the buggy emails were being sent from a Microsoft server (from the sender's Microsoft account) and received by a Google server (for the receiver's Google account), the first "Received:" header should reference a Microsoft server, and the last one should reference a Google server, and in between should be the Trinity relay server.
It turns out that on the day the problem started, emails to me from Microsoft users started taking a different route. Previously, they had gone from the Microsoft server to a Trinity server and then to the Google server. Now, they were going straight from the Microsoft server to the Google server. I validated this by looking at the relevant MX records: indeed, eagles.utrinity.edu had an MX record which pointed to the Google servers, whereas utrinity.edu had an MX record which pointed to Trinity's own servers. So, this indicates that, when the problem started, Microsoft began routing mail through the eagles.utrinity.edu domain instead of the utrinity.edu domain.
At this point, there was not much more I could do to investigate, so I contacted Trinity IT support. Eventually, they were able to reproduce the problem, and identify the cause. They discovered that the change in routing by Microsoft was due to the addition of a Mail Contact for the given user in Exchange. Mail Contacts are supposed to be links from within your organization to outside your organization. However, in order to enable compatibility with some legacy systems, it was necessary to make a Mail Contact in Exchange for users who needed access to these legacy systems. This changed the routing of emails sent only by Microsoft. Normally, mail was routed through an internal Trinity mail server, and then sent to either the Microsoft or Google server. The Google server was aliased as eagles.utrinity.edu. If Exchange encountered a Mail Contacts address within its own domain (@utrinity.edu), it would not forward the message to the Trinity relay servers, since these are for addresses outside the organization. Thus, since it couldn't go through those servers, it had to be sent directly to the Google server at eagles.trinity.edu in order to avoid an infinite loop. Google did not rewrite the address or the To field as Exchange did, and therefore, it showed up as [me]@eagles.utrinity.edu when routed directly from Exchange instead of through the relay.
So what about the four year rule? It turns out my "four year rule" was slightly incorrect. While a Mail Contact was indeed added after four years for an administrative service, it could also be added for other reasons. For instance, if you are registered as an instructor for a class, an entry must be made for integration with the online learning system.
So, that is how a university's IT infrastructure subtly showed its support for the football team. They found a workaround and the problem was solved, but as the lowly PhD student who still can't access the president's bank account, I was never updated on the details. But in the end, their trick must have worked, because I finally attended a Trinity football game. They lost.
Thanks to Max S. for contributing this story. More such crazy stories.