Tag Archives: network

Supporting (& paying for) the network segments that support IoT Systems

Network segmentation is often promoted as the answer to IoT device and systems management and risk mitigation for an institution, city, or corporation. While segmenting networks is important, a subtle problematic aspect is that:

  1. segmenting networks takes work, energy, and resources in the form of initial investment and ongoing management and oversight
  2. numbers of network segments growth may well track with IoT device count growth – which, at least for the next few years, appears exponential
  3. cities and institutions may not be planning for increased network management resources to support IoT Systems deployments

Success criteria for an IoT System implementation

I use a two overarching component criteria to define a successful IoT System implementation for a city or institution —

1. ROI – does the system perform as expected for the actual (vs projected) costs of deployment and subsequent management

2. Cyber risk – did the implementation of the IoT System make the city or institution worse off in the course of deploying and operating the system?

A key aspect to both of these criteria is system manageability. An unmanageable or difficult to manage system costs more in terms of staffing, rework, repair/updating, and operational disruption. At the same time, a difficult to manage system can create cybersecurity vulnerabilities — both seen and unseen — and divert limited institutional resources from existing operational, cybersecurity and risk mitigation activities.

Similarly, as IoT Systems need to be manageable to positively (at least not negatively) affect ROI and institutional risk profile, the network segments supporting IoT Systems also need to be manageable.

Managing the network segments that support IoT Systems

Networks are no longer, “make this thing talk to that thing,” or “make these things talk to those things.” Network management requires a robust set of supporting system services that support consistent connectivity, resilience, real-time health reporting capabilities, and rapid network diagnostic capabilities.

In addition to these core network support services, for IoT Systems deployments, there is another overarching criteria needed to successfully support and manage networks that support IoT Systems —

The IoT System owner should be able to measure, monitor, and determine performance of the IoT System(s) at any particular point in time. This supports both the effort of determining ROI as well as providing visibility for cyber risk mitigation.

In the case of a city, the IoT System owner might be the city’s transportation department while the network provision is provided by or contracted through the city’s central IT organization. In the case of an institution, such as a research university, the IoT System owner might be an academic department purchasing and deploying an IoT System to support a research grant while the underlying supporting network segment is supported by the university’s central IT organization.

The IoT System owner should not be relying on the network provider to provide IoT application/system management and diagnostic services. IoT Systems are evolving so rapidly, it is very unrealistic to expect the network services provider to have the resources or wherewithal to keep up with the nuances of each new IoT System deployment, much less manage the performance expectations for a rapidly growing number of different IoT Systems.

Examples of IoT System/application-specific network services include:

  • enumeration – how many things/devices are on this network?
    • Is this count different from yesterday? By how much?
  • identification – how many of these devices belong to my IoT System? Have I seen them before? Yesterday? a month ago? How rapidly is this changing?
    • Is what I am seeing different from what I was expecting?
      • (Did I know what I was expecting?)
  • Application-specific network device awareness and health
    • Device heartbeat – are you there?
    • Device performance specifics –
      • Is device characteristic 1 returning a result within acceptable constraints? e.g. voltage level
      • Is device characteristic 2 returning a result within acceptable constraints? e.g. device temperature,outside air temperature (OAT)
      • Is device characteristic 3 returning a result within acceptable constraints? e.g. response time
      • Is device characteristic n returning a result within acceptable constraints?

Accomplishing these requirements is not free. Some technology investment is needed, but more importantly, an organizational framework that supports this activity is necessary.

Growth of devices & growth of network segments

While we don’t know what the rate of growth of network segments is or will be, we can be pretty sure that it will continue to grow for the foreseeable future. This growth in network segments stems from at least two reasons, 1) network segments whether VRF’s, VLAN’s or other are easier to implement than they used to be, and 2) network segmentation is currently a popular strategy for addressing IoT Systems risk mitigation. (Regarding the latter, I believe that this is in part because we don’t know what else to do — when all you have is a hammer, everything looks like a nail).

There are multiple projections that IoT device count is growing at an exponential rate, such as this Ericsson Mobility Report that suggests a 23% annual rate of growth between 2015 and 2021 and this McKinsey report that suggests 15% – 20% annual growth by 2020. Related projections on IoT market growth can be even higher with annual growth over of 50%.

This ongoing steady (or more) growth year after year appears to be exponential growth. While IoT devices don’t compound each other like dollars do, the growth count curve still appears exponential. To borrow from the idea of compounding (exponential) growth in finance — we add 1 to the rate of growth and raise that sum to the number of years out that we want to project and then multiple that whole thing by the starting count —

(Wikipedia)

So, let’s say that we start with 10,000 IoT devices in a hypothetical city or institution and that the growth rate in IoT device count over the next few years is 20% . That growth curve looks something like this —

exponential device count growth @ 20%

(As a quick side note, we can see the Rule of 72 providing a rough estimate of the time it takes to double, about 3 1/2 years, at work here).

Now let’s say that there are currently 1000 network segments (VLANs, VRFs, etc) in that city or institution and consider two hypothetical growth trajectories — one exponential at the same rate as the device count growth rate and one linear growing at 1000 network segments per year.

Hypothetical exponential growth at 20% and hypothetical linear growth at 1000 segments per year

While don’t know exactly what the network segment growth rate or trajectory will be for cities and institutions in particular or in aggregate, we can expect that count to continue to grow for the next several years. With that growth comes an increased demand on institutional and city resources. The question is, are we planning to resource that increase in demand?

Network segmentation management — impacts on ROI & institutional cyber risk

As mentioned earlier, to manage network segments for successful IoT systems implementations, at least two components are required — manageability of the network segment itself and manageability of the specific IoT System(s) on that network. Without both of these, the likelihood of an IoT System’s success for the city or institution is low. Without both of these, the city or institution can expect to feel negative impacts to ROI, the city or institution’s cyber risk profile, or more likely both.

Because both of these success components require current resourcing (eg staffing and tool investment) and planning for future resourcing, success is not guaranteed. If we’re not thoughtful about implementation, we could end up with broad portfolios of a rapidly growing number of unmanaged or under-managed networks. And that’s not good for any of us.

 

Internet2 Chief Innovation Office launches IoT Systems Risk Management Task Force

Internet2 has launched a national Task Force to study risk management needs around IoT Systems in Higher Education and research institutions. The Task Force is composed of Higher Education and research IT and Information Management leaders across the country and will explore the areas of IoT Systems selection, procurement, implementation, and management. At the end of 12 months, the IoT Systems Risk Management Task Force will deliver a set of recommendations for 3 – 5 areas of further in-depth work. (And in the interest of full disclosure, I am Chairing the IoT Systems Risk Management Task Force.)

Internet of Things Systems or IoT Systems offer great potential value to higher education, research, government, and corporate institutions. From energy management, to research automation systems, to systems that enhance student, faculty, staff, and public safety, to academic learning systems, IoT Systems offer great promise. However, these systems need to be implemented thoughtfully and thoroughly or the investment value won’t be realized. Further, because of the distributed computing and networking capabilities of IoT devices, poor IoT Systems implementations can even make things worse for institutions, corporations, or governments.

Internet2 Chief Innovation Office

i2logoThe mission of the Internet 2 Chief Innovation Office, led by Florence Hudson,  is to work with Internet2 members to define and develop new innovations around the Internet. The Innovation Program has three core working groups —

Internet2’s core offerings are its 100 gbps network and their NET+ services.  Their membership includes 300 Higher Education institutions and over 150 industry, lab, and national agency organizations.

Many IoT systems risk topics

Examples of topics that the Task Force will cover include IoT systems vendor management issues, network segmentation strategies and approaches, cost estimating tools and approaches for IoT systems, potential tool development and/or partnering with organizations that perform Internet-wide scanning for IoT-related systems, and the organizational and cultural issues encountered in transitioning to a data-centric organization.

IoT systems vendor management approaches

Organizations and institutions need to raise the bar with IoT systems vendors regarding what constitutes a successfully delivered product or service. For example, has the vendor delivered documentation showing the final installation architecture, have default logins & passwords been change on all devices (how is this demonstrated), have all unnecessary services been deactivated on all devices and systems and how is this demonstrated?

Development of common ‘backends’ for IoT systems

Current IoT systems (to include utility distribution, building automation systems, many others) vendor approaches require that institutions invest in separate and proprietary ‘backend’ architectures consisting of application servers, databases, etc for each different vendor. This is an approach that does not lend itself to manageability, extensibility, or scalability.  In this space, perhaps newer container and container management technologies offer solutions as well as other possibilities.

1200px-Internet_of_things_wilgengebroedDevelopment of network segmentation/micro-segmentation strategies and approaches for IoT Systems

Network segmentation seems to offer great promise for mitigating risk around IoT Systems implementations. However, without appropriate guidance for IoT network segmentation implementation and operation, institutions can end up with a full portfolio of poorly managed network segments. Exploration and development of institutional network segmentation best practices can serve to lower an organization’s risk profile.

Development of cost estimating tools and approaches for IoT Systems

There is little in the way of precedent for cost models for the rapidly evolving IoT systems space and, as such, planning for IoT Systems and trying to estimate Total Cost of Ownership is difficult and nuanced. Exploration of and development of IoT Systems cost models can be of real value to institutions making planning and resourcing decisions.

Development of risk language & risk categories around IoT systems

Currently it is difficult to discuss new risk brought on by IoT systems with enterprise risk managers because IoT systems themselves are difficult to describe and discuss.  Development and socializing IoT risk language, that incorporates existing familiar institutional risk language, would enhance the ability to discuss IoT systems risk at the enterprise level. This Task Force will explore this nuanced space as well.

Analysis tool development and partnering

The Task Force will explore tool development and/or partnerships with organizations that scan the Internet for industrial control systems and IoT systems and publish these results online. Exploring internal tool development of the same is also a possibility. Development of benchmarks and baselines of Internet-scanning results across different industries and market sectors will also be considered.

Organizational cultural barriers to successful implementation of IoT Systems

Changing from a traditional organization to a data centric organization is a non-trivial transition and not addressing these issues can be a barrier to successful implementations of IoT Systems in institutions, organizations, and cities. The Task Force will study this important space as well.

Early Task Force work will also include identifying and enumerating other independent and overlapping risk areas (operational, cyber, cultural, and others). Over the year, Task Force members will participate in phone conferences, listen to subject matter expert presentations, and identify, discuss, and prioritize IoT Systems issues. Finally, recommendations will be made for further focused work on the highest priority areas.  If you have questions, comments, further interest, please contact me ChuckBenson@longtailrisk.com or the Internet2 Chief Innovation Office at CINO@internet2.edu.

 

[IoT image above: By Wilgengebroed on Flickr – https://www.flickr.com/photos/wilgengebroed/8249565455/, CC BY 2.0, https://commons.wikimedia.org/w/index.php?curid=32745541]

Looking for a quick definition of IoT?

Defining IoT (image Wikipedia)

Defining IoT
(image Wikipedia)

 

Tired of looking for the right words when trying to impress your boss, friends, or potential future spouse with a description of the Internet of Things, aka IoT ? Well look no further !

The 10 word version

Here’s a 10 word version. An IoT device is one that:

1. Computes
2. Is networked
3. Interacts with the environment in some way

The 20 word version

And once you’ve impressed them with this knowledge that just rolled off your tongue, feel free take it further with the 20 word version!

1. Computes
2. Is networked
3. Interacts with the environment with the intention of collecting sensory data and/or manipulating the local environment

For example:

  • A FitBit device computes, is networked, & interacts with the environment (ie you)
  • An industrial  SmartGrid meter computes, is networked, & interacts with the environment (collects power data)
  • A residential Nest meter computes, is networked, and interacts with the environment (collects temperature data)
  • Chicago’s Array of Things devices compute, are networked, and interact with the environment (collect many environmental data points)
  • Blood glucose monitors compute, are networked, and interact with the environment (ie you)
  • and much much more !!

And then, while impressing those around you, you can bring it on home with the definition of an IoT System. An IoT system:

1. Is a set of IoT devices that
2. Communicate with each other and/or communicate with
3. A central server that aggregates data and/or provides control data

Congratulations on your assured future personal, social, and professional successes now that a handy definition of IoT is at your disposal!

Side effect of IoT growth – more attack platforms

iotgrowth

Rapid growth brings many good things, but also drives how we manage risk. [Image: theconnectivist.com http://bit.ly/1owv1dp]

The rapid growth of the Internet of Things (IoT) phenomenon, along with its corresponding rapid growth in device count, has been the talk about town over the past year or so. While IoT promises many good things, more conversation is being directed toward the risk brought about by the Internet of Things. Often this is in the form of someone will hack your web cams, steal your FitBit health information, hijack your routers and printers, or monkey with your thermostat remotely. While all important risks and concerns, I think that the bigger IoT risk has more to do with the sheer numbers of devices.

IoT devices as attack enablers

In all of the hoopla and coolness and excitement of the Internet of Things, we can sometimes forget the underlying subtle and amazing thing that they are all networked computing devices, many with well known and well understood operating systems. So, for a moment, forget that cool thing that the IoT device does in its local environment (capture video, audio, biometric authentication information, health information, temperature, humidity, refrigerator status, air composition, etc) and just remember that they are networked computing devices — many of these with substantial computing resources.

What this means is that IoT devices are not just targets themselves, but can also act as attack enablers or attack platforms. This can occur via direct hack or by unwitting participation in a botnet.

turkishpipelinehack

Baku-Tbilisi-Ceyhan (BTC) pipeline near the eastern Turkish city of Erzincan on Aug. 7, 2008.

From this recent analysis of a 2008 Turkish pipeline hack and sabotage:

“As investigators followed the trail of the failed alarm system, they found the hackers’ point of entry was an unexpected one: the surveillance cameras themselves.

The cameras’ communication software had vulnerabilities the hackers used to gain entry and move deep into the internal network, according to the people briefed on the matter.

Once inside, the attackers found a computer running on a Windows operating system that was in charge of the alarm-management network, and placed a malicious program on it. That gave them the ability to sneak back in whenever they wanted.”

So, the networked computing presence of the cameras themselves were used as a stepping stone (aka attack point) into the larger network. Some weakness in the operating system (OS) of the camera devices themselves provided a point of entry (‘vector’ in geek speak) into the pipeline’s operational network.

Big numbers

So, if we look at the growth in the number of IoT devices and consider them, for now, only as networked computing devices capable of being compromised, that’s a lot of new stepping stones for attacks.

These growing number of devices can enable & assist attacks by:

1) providing many more attack platforms, which …
2) provides more opportunities for indirection in attack, which …
3) makes attribution more difficult

buttonsLet’s get transitive – Kauffman’s buttons

At the risk of being a little bit tangential, all this reminds me of another network phenomenon, dealing with botnets, that I believe occurs. It is one that is exacerbated by the rapid increase in networked computing nodes, eg from IoT growth and has to do with how quickly the character of a network can change under fairly simple conditions.

I’ve always been intrigued with this ‘toy problem’ that Stuart Kauffman describes in his book, At Home in the Universe. He says to imagine that you have a bunch of buttons on the floor and some pieces of thread. You arbitrarily pick two buttons and then connect them with a piece of thread, a button at each end. Then you arbitrarily pick two more buttons and connect those two. (The original buttons are not excluded; they are still contenders. ) Keep doing this. While doing so, create a graph and plot the thread to number of buttons ratio on the X axis and the size of the largest cluster on the Y axis.

kauffman

Not too much happens at first. Early on, the largest button cluster stays pretty small. Then, at a certain point, the size of the largest cluster leaps. Logically, it’s not surprising. You can see how it happens. However, I still find myself staring at that big jump. That’s a real phase change for at least one aspect of that button network.

kauffman2

Quite a leap — https://keychests.com/media/bigdisk/pdf/16096.pdf

 

I think a similar thing happens with some botnets, particularly P2P botnets, as they grow in size. We can make the reasonable assumption that some botnet sizes are more effective than others at carrying out their varied nefarious tasks, eg 1000 is probably better than 10. While individual bots in botnets do not connect to all of the other bots on the network, they do connect to many.

IoT growth => More buttons

In this environment, I think Kauffman’s toy problem still applies. Namely, at some point, the largest cluster size grows very rapidly. Maybe not with the near-vertical drama of Kauffman’s problem where everything can be connected, but still with a significant acceleration in growth of the largest cluster once a critical point is reached. And if the largest cluster size suddenly meets or exceeds that putative optimal botnet size, well then, we’ve got ourselves an effective botnet.

So if the rapid growth in IoT provides many more buttons, then there are also many more buttons/potential botnet participants for the network. And the fact that these botnets can fairly suddenly (aka seemingly arbitrarily) reach their optimal effectiveness adds another air of uncertainty and difficult-to-predictness to the whole thing.

Not gloom & doom, but evolving risk picture

The sky is not falling and the Internet of Things holds much promise, but the way we look at risk will need to change. The advent and rapid growth of the Internet of Things will change some of the math on the Internet. More botnets will come online and they will do so in unpredictable ways. I’m not saying the end is near, but rather the way we look at risk will have to change.

Borrowing from search to characterize network risk

Most frequently occurring port is in outer ring, 2nd most is next ring in, ...

Most frequently occurring port is in outer ring, 2nd most is next ring in, …

Borrowing some ideas from document search techniques, data from the Shodan database can be used to characterize networks at a glance. In the last post, I used Shodan data for public IP spaces associated with different organizations and Wordle to create a quick and dirty word cloud visualization of exposure by port/service for that organization.

The word cloud idea works pretty well in communicating at a glance the top two or three ports/services most frequently seen for a given area of study (IP space).  I wanted to extend this a bit and compare organizations by a linear rank of the most frequently occurring services seen on that organization’s network.  So I wanted to capture both the most frequently occurring ports/services as well as the rank amongst those and then use those criteria to potentially compare different organizations (IP spaces).

Vector space model

I also wanted to experiment with visualizing this in a way that would give at a glance something of a ‘signature’.  Sooooo, here’s the idea: document search often uses this idea of a vector space model where documents are broken down into vectors.  The vector is a list of words representing all of the words that occur in that document.  The weight given to each word (or term or element) in the vector can be computed in a number of different ways, but one of the most popular is frequency with which that word occurs in that document (and sometimes with which it occurs in all of the documents combined).

A similar idea was used here, except that I used frequency with which ports/services appeared in an organization instead of words in a document. I looked at the top 5 ports/services that appeared.  I also experimented with the top 10 ports/services, but that got a little busy on the graphic and it also seemed that as I moved further down the ordered port list — 8th most frequent, 9th most frequent, etc — that these additional ports were adding less and less to the characterization of the network. Could be wrong, but it just seemed that way at the time.

I went through 12 organizations and collected the top 5 ports/services in each. Organizations varied between approximately 10,000 and 50,000 IP addresses. To have a basis for comparison of each organization, I used a list created by the ports returned from all of the organizations’ Top 5 ports.

Visualizing port rank ‘signatures’

A polar plot was created where each radial represents each port/service.  The rings of the plot represent the rank of that port — most frequently occurring, 2nd most frequently occurring, …, 5th most frequently occurring. I used a polar plot because I wanted something that might generate easily recognizable shapes or patterns. Another plot could have been used, but this one grabbed my eye the most.

Finally, to really get geeky, to measure similarity in some form, I computed the Euclidean distance between each possible vector pair. Two of the closest organizations of the 12 analyzed are (that is most similar port vectors):

 

mostsimilar

2 of the most similar organizations by Euclidean distance — ports 21, 23, & 443 show up with the same rank & port 80 shows up with a rank difference of only 1. This makes them close.  (Euclidean distance of ~2.5)

Two of the furthest way of the 12 studied are these (least similar port vectors):

 

leastsimilar

While port 80 aligns between the two (has the same rank) and port 22 is close in rank between the two, there is no alignment between ports 23, 3389, or 5900. This non-alignment, non-similar port rank, creates more distance between the two. (Euclidean distance of ~9.8)

Finally, this last one is some where in the middle (mean) of the pack:

 

midsimilar

A distance chosen from the middle of the sorted distance (mean). Euclidean distance is ~8.7. Because this median value is much closer to the most dissimilar, it seems to indicate a high degree of dissimilarity across the set studied (I think).

Overall, I liked the plots. I also liked the polar approach. I was hoping that I would see a little more of a ‘shape feel’, but I only studied 12 organizations.  I’d like to add more organizations to the study and see if additional patterns emerge. I also tried other distance measuring methods (Hamming, cosine, jaccard, Chebyshev, cityblock, etc) because they were readily available and easy to use with the scipy library that I was using, but none offered a noticeable uptick in utility over the plain Euclidean measure.

Cool questions from this to pursue might be:

1. For similar patterns between 2 or more organizations, can history of network development be inferred? Was a key person at both organizations at some point? Did one org copy another org?

2. Could the ranked port exposure lend itself to approximating risk for combined/multiprong cyber attack?

Again, if you’re doing similar work on network/IP space characterization and want to share, please contact me at ChuckBenson at this website’s domain for email.