And now, after nearly two weeks of intense analysis, detailed deliberation, and outright hand-to-hand combat, our esteemed judges emerge from their bunker, slightly bruised and battered, holding a single sheet of paper upon which, scrawled in blood, are the names of the winning entries to our annual Holiday Hacking Challenge.
In the background, a trumpet fanfare begins boldly proclaiming the announcement, with a scene that looks something like this:
Click here for the full video of the ceremony: http://www.youtube.com/watch?v=yixG8pfncOs
Go ahead and watch that video. It’s brilliantly cut and really fun. Don’t worry, we’ll wait until you are finished with it. OK? Now that you’ve seen it, we’d like to point out that, for a mapping of this here awards ceremony to that classic one, Ed Skoudis plays the part of the old guy in the grey beard, while Josh Wright plays the princess. Tom Hessman and Yori Kvitchko are the droids, and Tim Medin is the Wookie.
Competition was fierce among the dozens and dozens of really high quality entries we received. The judges read every one multiple times, teasing out the best of the best from some mighty fine contenders. Even those among the “Honorable Mentions” deserve high praise.
Grand Prize Winner
As you know, our grand-prize winner will receive a free course from our sponsor, SANS CyberCon, a virtual training event with several awesome classes held online Feb 10 through 15, a value of over $4,000. You can take week-long courses including Network Pen Testing (SEC560) and Web App Pen Testing (SEC542) without any travel at all, interacting with a live instructor. Our GRAND PRIZE WINNER (yes, the all-caps implies shouting), with the very best answer overall, the creme de la creme, the pick of the litter, the bestiest best of all bests is…drumrollllllll pllllleeeease…
GEBHARD ZOCHER, GRAND PRIZE WINNER, BEST OVERALL ANSWER
Gebhard’s answer contains four different components, each of which is included below.
Gebhard’s answer is incredible, with excellent technical details, clear explanations, detailed color pictures, and more. In fact, Gebhard’s entry is so good, the judges hereby deem it the official answers to the challenge. Congrats, Gebhard.
Next up, we’ve got our remaining winners, each of whom will receive an autographed copy of the Counter Hack Reloaded book, signed by Josh Wright and Ed Skoudis. Let’s start with…
Best Technical Answer
Our best technical answer came from…
PATRICK MOONEY, BEST TECHNICAL ANSWER
Patrick did a great job succinctly describing each attack, and he didn’t miss a beat in his excellent answers, available here. Nice job, Patrick!
Best Creative Answer
Next up is our best creative answer that was also technically correct. That prize, again an autographed copy of Counter Hack Reloaded, goes to:
PAUL HOSKING, CREATIVE WINNER
Paul’s answer, entirely in rhyme, is brilliant. It provides in-depth insight into each attack and associated defense, in an amazing sequence of rhyme that must be seen to be believed. It’s really beautiful, fun, and whimsical work. Check out Paul’s handiwork here.
As we mentioned, there were numerous really high-quality entries in the competition. Each of the names listed below deserves credit for a job well done. Their responses were incredibly detailed, discovered some subtlety that most others missed, provided a very clever narrative, included amazing innovation, and, in many cases, offered up a combination of each of these aspects. Truly, we applaud the following folks for their exemplary work (presented in no particular order whatsoever….each one of these is great and the order does not imply any are better or more spectacular than any other):
James Colgan: James’ submission was part of a class project, for which we are thrilled that instructors are able to integrate the challenge into course material. James hit an excellent balance of sophisticated analysis with well-illustrated yet concise results that was a pleasure to read.
Andrea Minigozzi and Giacomo Milani: Andrew and Giacomo had an excellent technical write-up with beautiful formatting, and even went the step further to ask “why” for each of the attacks (correctly citing that Mr. Potter wants to encourage the rapid growth of dental disease in Bedford Falls through manipulating drinking water fluoride levels). The team-of-two even went so far as to evaluate datestamp information in the “Firmware Update” phishing attack, identifying the 5-hour window between the upload of the ab-qfe.exe executable and the retrieval by Don Sawyer.
Annah Waggoner: Annah’s response was creative and fun, re-writing the lyrics to Burl Ives’ classic “Have a Holly Jolly Christmas” for SCADA hacking. I admit that I sang it out loud in my office, a cappella, much to my spouse’s concern. “Oh, bygolly I’ll control your PLC, Remotely”. Well done Annah, well done.
Bashar Ewaida: Bashar had both creative and technical chops in his submission, with a dramatic YouTube video (https://www.youtube.com/watch?v=sZx9KxlfpsU) and a write-up that may or may not have been professional formatted by a graphics designer. Bashar’s 56 figures are reminiscent of a report I just finished for a customer; apparently we both believe that a picture tells a story better than a thousand words.
Brian Chervenak: Brian had excellent technical results, but also a really engaging creative writing response, reminiscent of Kurt Vonnegut and James Joyce, with just a soupçon of Dan Brown. Brian’s write-up integrated the technical analysis with creative writing prose to make this a fun and enjoyable read.
Hecber Codrova: Hecber really took his time to thoughtfully chase down lots of interesting and precise details in his write-up, including noting the out-of-order VNC traffic and a top-notch analysis of the SQL injection attack. Hecber also correctly identified the victim account “ernie” as Ernie Bishop, the cab driver in “It’s a Wonderful Life”, which was awesome!
Matt Elliott: Matt (and his 6 year-old accomplice, who drew the technical network diagrams) delivered an impressive technical analysis, identifying subtleties in the attacks (including the use of the impostor phishing domain “valleyelectr1c.co.nw”, one of my personal favorite tricks). Matt also added suggestions that could help the attackers improve their foo, including several tool suggestions that were new to me. Thanks Matt!
Nathan Turnbull: Nathan really put himself in George’s shoes for the analysis of defenses that could have defeated Mr. Potter’s power grid attack, citing the Bailey family philosophy of operating the loan with little margin to best service the people of the community. The values of the business often influence the defenses that will be available to defenders, which we need to keep in mind as analysts. Thanks for this sage insight Nathan!
Nik Alleyne: Nik performed his analysis using command-line tools alone, with some CLKF (Command-Line Kung-Fu) that was so awesome that it almost brought a tear to my eye. For several of his responses he included parsed answers from different tools too, and regular expressions complex enough to make the screen become fuzzy (OK, that was me crying). Thanks Nik!
Aleksander Janusz: Aleksander’s analysis read like a professionally-written incident response report, with excellent use of detail, brevity, and illustrations in all the right places. Using the available data, Aleksander pulled in resources from publicly-accessible sites to discover the attacker’s intentions and actions throughout the attack, an excellent use of the available resources for analysis.
Rick El-Darwish: Rick’s analysis hit the correct technical aspects of the challenge, and even pointed out some of our dirty laundry. Normally we isolate systems in CyberCity, but had to open up the network a bit to accommodate some of components used for the attack including out use of Google public DNS server. Rick called us out on this in the “How Could George Do Better” section – I wholeheartedly agree Rick!
jatiki: Participant jatiki turned in a well-written and well-illustrated analysis, but his consideration of non-technical components was unique. Jatiki evaluated not only the attack details, but the potential motivators and assistance from insiders that could have made the attack possible. I must admit, Potter had some “insider” help for these attacks.
Rosinei Muniz Marinho: Rosinei is a funny person, who made me LOL while reading an excellent analysis. Your technical acumen was spot-on, and pointed out that employee training is an important component of a defense-in-depth program (that and access to the Samaritans hotline; that too).
John York: John’s answer includes an astonishingly detailed timeline, with all kinds of juicy info he teased out of the packet capture file. His write-up also includes in-line kudos and atta-boys for George as John describes each defense, which had me cheering along as I read it. Finally, John pointed out how working through a challenge like this is a fantastic way to learn, providing as much training as a 2-to-3 day class. Indeed, John! Great work.
Ahmed Adel: Ahmed’s answer was simultaneously clear, detailed, and succinct. He cited each attacker step with impressive precision, as well as described each tool he used in the analysis.
Christopher Byrd: Chris’ analysis of the Modbus traffic light attack, the SQL injection attack of the water reservoir, and the Allen Bradley system controlling the street lights is simply fantastic. Best of all, Chris was one of the few participants who noticed some time discrepancies between the HTTP header times and the packet capture times. When we built the challenge, we knew this discrepancy would exist, but left it in there to see how many people called us on it. Chris nailed that aspect, showing great kung fu in his evidence analysis skills.
Mark Jeffery: Mark’s answer laid out, step-by-step, all of the attacker’s moves in impressive detail. His defensive recommendations were likewise well thought-out.
Pradeep Kumar Gadde: Pradeep’s response was lavishly illustrated, with excellent Wireshark screenshots that describe the analysis very well. Pradeep also sent us a nice multi-colored Happy New Year 2014 greeting in an ICMP packet payload of a Wireshark capture. That really brought a smile to our faces.
Fadli B. Sidek: Fadli’s response was amazingly detailed, lavishly illustrated, and beautifully formatted. It’s an awesome entry from an obviously gifted information security analyst who knows how to convey information extremely effectively. This answer also pulls in the little lulzsec cartoon character near the end, to good comedic effect.
Guillermo Serrano: Guillermo’s answers regarding the power grid attack really shine through. He also provides a lot of good detail regarding recommended defenses.
Jack Radigan: Jack’s response was stellar, including a treasure trove of custom-created scripts to split up and automate the analysis of a large, multi-faceted packet capture file, perfect tools for a project like this challenge. His use of Snort was awesome and rare among the entries we received. And, finally, Jack’s detailed analysis and alignment of the timing of the video frames of the motion jpeg along with the observed events in the packet capture file was incredible. Very few other entries considered this very important time-based evidence, and Jack’s use of it to determine the playback speed of the video was really cool.
Mark Hillick: Mark’s answer was superb, highlighting excellent Wireshark, tshark, and command-line kung fu, along with a witty writing style that really draws in the reader and makes it fun. In reading it, it occurred to me that Mark’s answer was like sitting down with a good friend who was telling a fun and exciting story about an attack and how to analyze it. Nice work!
Nicolas Hochart: Nicolas’ answer was simply elegant. It was a tour de force of insightful analysis, well written, technically deep, and fantastically illustrated. The Force is strong with you, Nicolas!
Roy Luongo: Roy’s answer had an awesome series of creative twists, as, in his telling, Mr. Potter had recruited various holiday characters from the past, each embittered and twisted in some way, and used them as his goons. Rudolf, Hermie the Elf, and Santa’s vintage train conductor join forces as kind of a super villain legion of doom to undermine Bedford Falls’ infrastructure. It was really creative and was underpinned by really good technical analysis.
Tim Raidl: Tim’s answer is a great walk through of each step used by the bad guys to attack Bedford Falls. His analysis shows really good insight and skills.
And, last but in no way least, we get to the all-important random draw winner. Our judges reached deep into the stream of randomness provided by atmospheric static disturbances through the fine folks at random.org to unveil a magical mysterious number associated with the entry from…
KEVIN BEMIS, RANDOM DRAW WINNER
Kevin’s answer itself was absolutely fascinating, and a worthy entry in the quest for randomness.
Congrats to all our winners. You’ll hear from us soon regarding your prizes.
And finally, with our awards ceremony drawing to a close, we have a couple of final items we’d like to share. First off, Josh has a couple of comments on Modbus:
A few months ago I wrote a Modbus-TCP traffic light control system for CyberCity. We purchased some hobbyist traffic lights that are wired into several NetDuino+ controllers, allowing each traffic light to be controlled from a central PLC with standard traffic light patterns. Through this system, we wanted to teach CyberCity participants about the Modbus-TCP protocol, and challenge them to create their own tools to manipulate traffic light patterns to achieve various goals defined in a NetWars CyberCity challenge. We knew we wanted to use the system for the Holiday Hack Challenge as well, but we had a problem: how could George possibly defend a Modbus-TCP network, and still expose challenge participants to the Modbus-TCP traffic itself?
We could certainly firewall or airgap the Modbus-TCP network from other segments, but that doesn’t give the participant exposure to the protocol. I scoured the Modbus-TCP specification (http://www.modbus.org/docs/Modbus_Application_Protocol_V1_1b.pdf) for hints or clues on how people secure this unencrypted, unauthenticated protocol (zero hits for the search term “security” anywhere in the specification; Modbus-TCP is not exactly inspiring from a security perspective).
Ultimately, I had to re-write the behavior of the Modbus-TCP controllers, causing them to behave as if they were configured with an internal access control list for authorized devices, returning “Slave Device Failure” when the attacker changes the coil settings (which would have effectively changed the traffic lights themselves; consider the implications of city-wide “all red” or “all green” traffic lights). I was really happy to see that many of the responses caught this very small detail in the packet capture. Kudos to all! -Josh
And, to close this out, Ed has a little bit of behind the scenes action he’d like to share.
About a year ago, I had an idea to write our challenge based on a city under cyber siege. The city would be Santa’s workshop town at the North Pole, and people who hated Christmas would hack the town, all Grinch-like, to stop Christmas from coming. The whole thing could feature our CyberCity project, helping people learn to defend critical infrastructures and industrial control systems. But, we had already written How the Grinch Hacked Christmas years ago, so I left that idea on the shelf for about 10 months. In October 2013, Lynn Schifano (from Counter Hack) and I were debating over a burrito lunch about a theme for this year’s challenge. I mentioned the city idea, but we ruled it out. I googled for Holiday Themed TV shows and movies, bringing up a long list. Lynn and I went through each item, and decided that It’s a Wonderful Life might be pretty cool. It was then that I noticed an obscure holiday TV show from the 1990’s about Snowden the Snowman. “Oh, we have to figure out some way to use that.” I wasn’t sure how we could fit it in, but I knew there must be some angle.
Josh Wright and I began studying It’s a Wonderful Life, and sketched out the overall narrative and underlying attacks. We had the idea of showing numerous attacks that actually fail, so that people could explain defenses, a new twist we used for the first time in this challenge. Plus, we’d have one big hack, against a power grid, that would actually work. There would be some characters from the town acting in our packet capture, including George Bailey, Ernie (the cab driver from the movie), and a guy named Don. In a sense, we were going to build a little world inside a packet capture file for people to explore. We split up the work of writing, building various attacks, and so on among the Counter Hack team members, primarily Josh, Tom Hessman, and me, but also with some support from Yori Kvitchko and Tim Medin. We worked for several weeks building out assets.
The big hack that brings down the power grid required an attacker to control the GUI of a laptop running the Human Machine Interface (HMI) of the CyberCity power grid. The attacker would compromise this box, get VNC access to the machine with Metasploit, and use the GUI to cut power. I was working on a method for users to view the GUI. Rebuilding a VNC session from captured packets must be doable, right? I experimented a lot with the Chaos Reader tool, but realized that it only worked against some very specific old version of the VNC protocol, and that there were no free tools for rebuilding entire GUI sessions from packet captures using modern versions of VNC. Josh and I toyed around with the idea of building and releasing a tool to do so, but we decided that this was kinda cheating, plus our deadline for release loomed. We needed a way for users to view the GUI, but VNC and even RDP tools to do so were rare and buggy. We were stuck.
I went to bed one night thinking about this problem and how we needed to see the GUI of a machine controlling the power grid. I couldn’t sleep. Really really late that night (11 PM), I thought, “It’s almost like we need a video camera to show the laptop.” Then, it hit me. We’ve got 5 video cameras distributed around CyberCity! We could have a laptop in the view of one camera, with the city in the background watching the action. We’d make the laptop look like it was on a desk, with a window behind it and a coffee mug next to it, overlooking the city. Not only would people see the attacker using the GUI, but they could also see the impact of the power outage on the city at the same time. I quickly mocked up the scene for the camera in PowerPoint, and sent it to Lynn, asking her to build out the shot. Here is what I sent:
I wanted the white grating behind the laptop so that it looked like you were peering through a window. Otherwise, it would just look like we had Godzilla’s laptop, desk, and coffee mug towering over the city. For the window grating, I told Lynn to grab one from the front window of the house. I was in San Francisco at the time, emailing Lynn asking her to rip my house apart to build the scene.
As Lynn was assembling the scene, my wife saw her taking apart the front window to the house. “What are you doing?” my wife asked. “This is something for one of Ed’s challenges,” Lynn responded. “Oh… whatever,” was my wife’s entirely reasonable response after 17 years of marriage, “As long as I get that back before our holiday cookie baking party in a week, that’s fine.” I just thought that was AWESOME!
Lynn built the scene out, which looked perfect on camera. Tom Hessman, Lynn, and I had to do a lot of experimentation with lighting, contrast, frame rate, and more to get the laptop screen legible while having the city look ok in the background at the same time. After several hours of work, our scene came together, looking like this:
It does look rather like a laptop on a desk in a building, looking out a window over the scene of a small city. But, what I really want to show you is what it looked like from the other side. Here is what Lynn actually built to achieve that shot:
Note that the window grating is hanging from the ceiling with string. The laptop sits on a disembodied leaf from my dining room table. The laptop itself is a dual-NIC machine, as it is running several virtual machines, including the GUI of the power grid. An old lamp from my wife’s college dorm room provides ambient light so the desk is visible. In CyberCity, in the lower right, you can see our power substation, traffic lights, and the blue glow of our water reservoir. A video camera on the wall sits behind this contraption, capturing the action.
We gathered the actual streaming video of the GUI attack as a motion JPEG in the packet capture. Using a variety of tools (such as mplayer, hinted at in the packet capture with a user browsing to the mplayer website), you could rebuild that file and see the video of the successful attack against the power grid. One of our challenge takers (and honorable mention winner), Bashar loaded the ultimate video unto YouTube here (Bashar added the music and commentary, to good effect): https://www.youtube.com/watch?v=sZx9KxlfpsU
Everything was now ready, but we ran into some snags on some of the attack details. First off, we had to get some pivoting action in to take over the GUI. That took a couple of days to get right. But, during that time, we had to do a live demo of CyberCity for a military customer. Having a window lattice hanging from the ceiling would look pretty crappy for a demo, so we had Lynn dismantle the whole rig for about three hours. The demo went swimmingly. Lynn rebuilt the scene within minutes of the demo concluding so we could get back to work on the challenge.
And now, our final hurdle was a hole in the narrative. How could we explain a packet capture with packets from all over a city? Why, to accomplish that, you’d need monitoring stations all over town gathering packets. And then, it all came together, from our burrito lunch to our final step in creating the challenge. We could have Snowden the Snowman give the packets to the reader from a magical packet capture system he had access to. The revelations in the actual Snowden case were being announced on a daily basis as we were writing the challenge, so it seemed almost too perfect. We were very careful to write the narrative in a way that neither endorsed the real Snowden’s actions nor cursed him. We played it totally neutral. Furthermore, the narrative voice never even calls the snowman by the name of “Snowden”. He only calls himself that in his quotes. I read the challenge on the phone to several friends who are particularly sensitive to this type of thing to ensure we were remaining neutral on the controversial topic, and they gave us the thumbs up. We ironed out a few more issues with the hack, built, tested, rebuilt, retested, and rejiggered the packet capture file a couple dozen times, and then finally released the challenge! Oh, and we managed to reassemble the window just in time for my wife’s annual holiday cookie baking bash!
So, that’s the behind-the-scenes view of this year’s challenge creation. We’ve already started to think about next year’s challenge now. We’ve got a few more nifty ideas up our sleeves, and hope you’ll join us for all that fun. We’ll release it around the second week of December 2014. Please do stay tuned!
Congrats again to all our winners and honorable mention recipients.
And, for everyone who participated, we would like to give a deep, heartfelt thanks. Thank you for playing our game. We really appreciate all your efforts and were genuinely excited to see the amazing things you all did in working through it.
–Ed Skoudis, Josh Wright, & Tom Hessman