Defending New York City's IT infrastructure is a daunting task. With 8.6 million residents just within the city's five boroughs, the city hosts hundreds of web applications so residents can track and use services like street plowing, as well as the popular NYC.gov site. With more than 330,000 employees within the city and 400,000 endpoints to keep track of -- all within several, federated agencies such as the NYPD and Immigrant Affairs -- the attack surface is huge.
It's a responsibility that falls to New York City Cyber Command, an 18-month-old agency charged with defending the city from cyber threats and enabling New Yorkers to lead safe digital lives.
Given the scale and federation of New York CIty's IT infrastructure, the agency decided to build its own data pipeline. The agency wanted to build a secure, cloud-based security log aggregation platform for city systems -- one that enabled alerting, visualization and analysis for cybersecurity professionals. The pipeline also had to allow the agency to scale non-linearly as the demand on services grows and cybersecurity threats grow.
"We built it because we needed to solve a New York City-sized challenge... with a new, cutting-edge, cloud-first approach that enabled the latest tools and technology to be applied at scale against our problem," Colin Ahern, NYC's deputy CISO, told ZDNet. "One that would allow us to evolve and stay head of the threat."
Speaking at the Google Next Conference[1] in San Francisco, Ahern described