REN-ISAC Activities and REN-ISAC / Internet2 Focus Group Results Doug Pearson Technical Director, REN-ISAC Joint Techs, July 2005 How to Participate To – Join the vetted membership – Receive REN-ISAC information product – Participate in information sharing http://www.ren-isac.net/renisac-sec-l.html REN-ISAC Mission • The REN-ISAC – is an integral part of higher education’s strategy to improve network security through information collection, analysis, dissemination, early warning, and response; specifically designed to support the unique environment and needs of organizations connected to served higher education and research networks, and – supports efforts to protect the national cyber infrastructure by participating in the formal U.S. ISAC structure. • In this presentation: – Quick outline of REN-ISAC Activities – Quick look at REN-ISAC information resources – Quick look at REN-ISAC information product – Summary of the results of an Internet2 & REN-ISAC Focus Group study This isn’t a full presentation about what the REN-ISAC is or does, if you’d like to see that: http://ren-isac.net/docs/ren-isac.pdf Activities • Information products … • Incident response; broad-impact events and at request • 24x7 Watch Desk; ren-isac@iu.edu, +1.317.278.6630 • Vetted membership / security contacts • Tools (in conjunction with IU Advanced Network Mgmt Lab) • Security infrastructures work in specific communities; e.g. grid security working groups • Participate in mitigation communities • Participate in other HE efforts; e.g Internet2/EDUCAUSE Computer & Network Security Task Force, SALSA • Participate in other national activities, e.g. inter-ISAC, National Cyber Security Partnership, etc. Information resources • Network instrumentation – Abilene NetFlow – REN-ISAC Darknet – Abilene router ACL counters on common & threat ports – Global NOC operational monitoring systems • Daily security status calls with ISACs and US-CERT • Vetted network security collaborations, e.g. [XXXX] • Backbone and member security and network engineers • Vendors, e.g. monthly ISAC calls with vendors • Security mailing lists, e.g. EDUCAUSE, FIRST, etc. • Members – related to incidents on local networks Information products • Daily Weather Report • Daily Darknet Reports • Alerts • Notifications • Monitoring views • We don’t duplicate information flows provided by others, such as SANS ISC, US-CERT, etc. Rather, we provide unique product derived from our perspective and resources and provide value-add to existing information. • Some information products are shared to the broad vetted membership, others to individual institutions involved in incidents. Privacy is important. Daily Weather Reports • Contain observations at aggregate levels of network threat traffic based on – Abilene NetFlow and – REN-ISAC Darknet (Abilene and commercial Internet) – Information and perspective from daily Inter-ISAC Cybersecurity Status calls • Distributed to closed lists, including – REN-ISAC members and – Inter-ISAC plus DHS/US-CERT community • Example 1. highlights the Report structure Daily Darknet Reports • The REN-ISAC Darknet is a deployment of the University of Michigan Internet Motion Sensor project. • The Darknet contains a large block of dark IPv4 address space routed to a collector that records traffic inbound to the address space – it hears automated and manual scanning from malware (e.g. bots, worms) and perps. • REN-ISAC parses the information according to source member institutions, and sends reports of sources seen to the respective institution. • Institutions remediate the affected systems • Currently monitoring 41 Internet2 sites; growing • Example: Indiana University Alerts • Alerts contain critical actionable information alerting the broad membership to new or increasing network-based threat. • Alerts are sent as required, to: REN-ISAC members, and as appropriate to other network security groups. • Example: Dec 2004, increased TCP/5900; scanning for trojans with VNC backdoors? Notifications • Notifications contain actionable information about active network-based threat or incident involving a specific institution. • Notifications are sent to the involved (source and victim) institutions. • Typically contain identification of specific hosts. • Example: – March 2005; Keylogger botnet involving 46 EDU institutions (Internet2 and non-Internet2) Monitoring • Abilene NetFlow • Publicly available reports of Abilene traffic stats for common and threat vector ports, published to the REN-ISAC web pages – http://www.ren-isac.net/monitoring.cgi • Arbor PeakFlow DDOS • Darknet Abilene Traffic by Port Abilene Traffic by Port Abilene Traffic by Port Focus Groups • Goal: Determine what the REN-ISAC & Internet2 can do to help security and network practitioners better defend their local environments. • Two sessions: June 24 and 30; 9 universities, mostly Carnegie classification “Doctoral/Research Universities – Extensive”, i.e. medium-to-large research universities. • Method: – small groups of interviewees (< 6 per session) – prior to FG, each interviewee identified top 5 issues – single interviewer – list of questions, but free to roam; 1.5 hour session – group of experts communicated to interviewer via IM, prompting additional/exploratory questioning FG: How do you identify misbehaving hosts? • high: reliance on multiple methods, e.g. "use three tiers – active scanning, passive detection with netflow, and passive signature-based detection" • high: receive notifications and/or pull information from outside sources, including other EDUs, DShield, REN-ISAC darknet reports, etc. – med: initially verify reports locally, e.g. netflow, argus, etc., but when establish consistent veracity of source then take reports as gospel – high: find the reports very useful, and do follow-up • high: local abuse@[org.edu] contact point is actively monitored FG: How do you identify misbehaving hosts? • high: use netflow to identify misbehaving hosts – lots of local-custom scripts (most often used in conjunction with flow-tools) – on the fly inspection and stored-look-at-later – periodically run scripts looking for known indicators, such as scanning, connections to known bad internal and external hosts, top talkers on given IP ports, etc. – low: look for bandwidth per host anomalies – low: concern regarding sampling due to traffic level (led one to move to a commercial flow analysis system); other comments - still very useful even if sampling – mix of what's being looked at: more oriented to flows at the border than in the core; more oriented to outbound than inbound FG: How do you identify misbehaving hosts? • high: vulnerability and port scanning – vulnerability scanning using (in order of common use) (1) Nessus, (2) ISS; (3) Retina; some sites use more than one – and Nmap for port scanning – a mix of central and distributed (departmental) scanning – high: movement to centralize scanning resources and tools make the central scanners accessible to the departments and/or users – low: packaged scanning tools (i.e. on CD-ROM, etc.) provided to departments/system administrators – zero: policies that preclude departments from deploying their own scanners in their domains FG: How do you identify misbehaving hosts? • high: vulnerability and port scanning (CONTINUED) – mix of approaches • scan only when new a threat comes out • occasionally scan all hosts all ports (very time consuming) plus more frequent scans at known bad ports – look for banners, e.g. 220 messages (SMTP servers), at unusual port numbers – scanning becoming less useful as host-based firewalls come online; leading to more use of passive detection, but still useful for detecting backdoors – low: scan wireless and modem address space – med-low: automated scanning; most of those who are not automated are heading in that direction FG: How do you identify misbehaving hosts? • high: vulnerability and port scanning (CONTINUED) – med: tying scanning into network registration systems – with side benefit to solve the DHCP-created IP address- to-owner disconnect problem for scan reporting FG: How do you identify misbehaving hosts? • med-high: use of Snort – look for suspicious traffic patterns, loud talkers, hosts walking the address space, etc. – some paring-down of out-of-box rule sets in order to reduce false positives – issue with performance, e.g. capability to deal with network bandwidth, ability to keep up with scanning detection and packet inspection, high peaks of worm scanning, etc., leads some to: • multiple/distributed Snorts • move scanning detection to darknet, save Snort for packet inspection tasks • some to think about large commercial IDS/IPS – some use of BASE (Basic Analysis and Security Engine) FG: How do you identify misbehaving hosts? • low: use of commercial IDS/IPS – detect packet signatures, port-scanners, hosts with large number of session opens, etc. – high: concern regarding interfering with research- oriented non-standard traffic (mitigation: tune to block only the things that are well understood) – high: concern regarding ability to meet bandwidth requirements – med: concern regarding blocking of legitimate traffic at non-standard ports – comment: lots of human resource currently invested in getting good use out of Snort, open source tools, etc., a magic IPS box should reduce the resource requirements considerably, but haven't found the magic box yet FG: How do you identify misbehaving hosts? • med: system administrators inspect system logs for malicious activity • med: inspect DNS traffic; use of DNS query logs • low: router log analysis • low: bandwidth reports • low: darknet, but med: interest in darknet • low: QRadar • low: Argus • low: local users/system administrators identifying miscreant IRC activity FG: How do you identify misbehaving hosts? • low: locally-developed web spiders looking for institution or institution data-identifying information • low: automation to take notifications from sources (local sensors, outside reporters, etc.) directly into back-end system to generate alerts to LAN administrators or incident response teams FG: Incident Response / Investigation • high: capability for incident responders to quickly kill ports (some in direct control, others via quick process with NOC) • low: automatic blocking of ports/hosts (i.e. without a person in the middle) • med-high: when protected data is involved the central security office gets involved in investigation, forensics, etc. – low: policies that require reporting of incidents to the central security office; but most receive the reports anyway; and many have policies in the works – in some cases driven by state laws requiring notification of personal data compromise – a few notable exceptions • "We usually don't - we keep data from flows and logs, and the departments handle the investigation of incidents themselves", and FG: Incident Response / Investigation – a few notable exceptions (CONTINUED) • "Forensic response is a costly resource and is not encouraged. It's not policy to quiz people about data on the machine. Sometimes that's obvious and then the response takes a different path." • med: use of netflow to identify all flows for affected host(s) • med: forensics capability – med-low: formal forensics training – low: certified forensics personnel, e.g. GCFA – med: training expected soon • low: toolsets provided to departments/system administrators for system recovery and forensics; e.g. Knoppix/Helix, etc. FG: Prevention • med: blackhole known external sources of malware, hacking, etc. – typically only in extreme cases – typically don't do it just from external reports but confirm it on the local network – to trust external lists would need to have a really good definition of how/why hosts get on the list and how they get off; along with information about how critical a particular entry is; even then, many would still locally confirm the activity – mixed opinions within the institutions themselves FG: Internal Information Sharing • med: institutional private mailing lists where security matters, including tools and methods are discussed • med: incidents/compromises discovered at departments are reported to a central organization FG: External Information Sharing • high: local submissions of captured codes to antivirus vendors • high: try to notify other EDUs when have information regarding misbehaving machines, but – difficult to do - proper contact, time, methods, etc. – difficult because the number of hosts involved is typically very large • the [anticipated] REN-ISAC registry would be really helpful • if know clueful contacts, would probably report more • UNISOG and REN-ISAC are good resources • abuse@[org.edu] contact points usually work FG: Data Retention Policies • low: official policies regarding data retention • high: flow data kept for 14-90 days; low: keep forever • high: system logs, authentication records, mail records, etc. kept 6 months -> forever • high: don't store payloads (only for the duration of an investigation) FG: Tools • high: netflow • high: custom in-house developed scripts for netflow, mostly in conjunction with flow-tools • high: Nessus preferred over ISS, etc. – can look at the vulnerability checks and understand definitively what they're going to do and how they're doing it – easier to customize scripts to local environment – more flexible, e.g. for single vulnerability checks – community support is strong – low: ISS requires administrator rights on remote machines for some checks, and "we don't have and never will" FG: Tools • high: Nessus preferred over ISS, etc. (CONTINUED) – on the negative side, Nessus is more difficult for the non-professionals (i.e. departments, end-system administrators) to interpret results than ISS • high: Nmap • high: willingness to share locally developed tools – tools are discussed and shared at UNISOG, international ISP security communities, in regional security groups and conferences – but want to keep the sharing limited to white hats FG: Other Areas • Talked about the following in the FG, but not presented here due to time. Will be included in follow-on reports: • HOST/NETWORK REGISTRATION • WIRELESS • VPNs FG: Would like to see the REN-ISAC do… • help organizations to take a better and more strategic approach to network intelligence that's gathered, e.g. what needs to be collected, what the purposes are, where should the information go, policy for handling, retention, etc. • methods (including standards and policies) to share observations of misbehaving hosts that are external to the local institution • serve as an anonymization point for information sharing, e.g. "I'm seeing this sort of behavior" messages, Snort rules, etc., accepted from a trusted contact and distributed anonymously to the trusted community • serve as a trusted meeting point for peers, i.e. "I know that if they're here that they've been vetted according to XYZ" meeting point FG: Would like to see the REN-ISAC do… • reach out beyond the higher-ed community – where they can help us and we can help them • facilitate communications - make it easy for institutions to find each other and find the right contacts, quickly • tool repository • recommendations regarding best practices for border filtering – what to filter, what not to filter, and why; such community consensus guidelines would provide authority and backing to recommendations made to local decision makers • standardization and/or sharing of information around the use of flow tools • security contact information FG: Would like to see the REN-ISAC do… • rankings of the amount of misbehavior seen from institutions (while not making the rankings public) • coordinate alliance to acquire commercial products, e.g. Arbor for gigapops, etc. • information regarding state and local laws that bind institutions, e.g. legal precedents regarding log retention, etc. • DShield-like service • regular security workshop similar to EDUCAUSE, Jt. Techs • taxonomy of tools and pointers to people that have them • current security best practices guides ignore the open end- to-end concept, need credible best practices for security implementations that respect end-to-end openness FG: Would like to see the REN-ISAC do… • organize funding and grants for information sharing activities among the institutions FG: Path Forward Some of the preceding suggestions match very well to the REN-ISAC mission and some match better to other groups, such as EDUCAUSE Effective Practices, etc. REN-ISAC will work Internet2, EDUCAUSE, SALSA, and with its [to-be-formed] Technical and Executive Advisory Groups to determine paths forward on the results of the FGs. How to Participate To – Join the vetted membership – Receive REN-ISAC information product – Participate in information sharing http://www.ren-isac.net/renisac-sec-l.html Doug Pearson PGP: http://mypage.iu.edu/~dodpears/dodpears_pubkey.asc Research and Education Networking ISAC 24x7 Watch Desk: +1(317)278-6630 ren-isac@iu.edu http://www.ren-isac.net