Tag Archives: risk

Supporting (& paying for) the network segments that support IoT Systems

Network segmentation is often promoted as the answer to IoT device and systems management and risk mitigation for an institution, city, or corporation. While segmenting networks is important, a subtle problematic aspect is that:

  1. segmenting networks takes work, energy, and resources in the form of initial investment and ongoing management and oversight
  2. numbers of network segments growth may well track with IoT device count growth – which, at least for the next few years, appears exponential
  3. cities and institutions may not be planning for increased network management resources to support IoT Systems deployments

Success criteria for an IoT System implementation

I use a two overarching component criteria to define a successful IoT System implementation for a city or institution —

1. ROI – does the system perform as expected for the actual (vs projected) costs of deployment and subsequent management

2. Cyber risk – did the implementation of the IoT System make the city or institution worse off in the course of deploying and operating the system?

A key aspect to both of these criteria is system manageability. An unmanageable or difficult to manage system costs more in terms of staffing, rework, repair/updating, and operational disruption. At the same time, a difficult to manage system can create cybersecurity vulnerabilities — both seen and unseen — and divert limited institutional resources from existing operational, cybersecurity and risk mitigation activities.

Similarly, as IoT Systems need to be manageable to positively (at least not negatively) affect ROI and institutional risk profile, the network segments supporting IoT Systems also need to be manageable.

Managing the network segments that support IoT Systems

Networks are no longer, “make this thing talk to that thing,” or “make these things talk to those things.” Network management requires a robust set of supporting system services that support consistent connectivity, resilience, real-time health reporting capabilities, and rapid network diagnostic capabilities.

In addition to these core network support services, for IoT Systems deployments, there is another overarching criteria needed to successfully support and manage networks that support IoT Systems —

The IoT System owner should be able to measure, monitor, and determine performance of the IoT System(s) at any particular point in time. This supports both the effort of determining ROI as well as providing visibility for cyber risk mitigation.

In the case of a city, the IoT System owner might be the city’s transportation department while the network provision is provided by or contracted through the city’s central IT organization. In the case of an institution, such as a research university, the IoT System owner might be an academic department purchasing and deploying an IoT System to support a research grant while the underlying supporting network segment is supported by the university’s central IT organization.

The IoT System owner should not be relying on the network provider to provide IoT application/system management and diagnostic services. IoT Systems are evolving so rapidly, it is very unrealistic to expect the network services provider to have the resources or wherewithal to keep up with the nuances of each new IoT System deployment, much less manage the performance expectations for a rapidly growing number of different IoT Systems.

Examples of IoT System/application-specific network services include:

  • enumeration – how many things/devices are on this network?
    • Is this count different from yesterday? By how much?
  • identification – how many of these devices belong to my IoT System? Have I seen them before? Yesterday? a month ago? How rapidly is this changing?
    • Is what I am seeing different from what I was expecting?
      • (Did I know what I was expecting?)
  • Application-specific network device awareness and health
    • Device heartbeat – are you there?
    • Device performance specifics –
      • Is device characteristic 1 returning a result within acceptable constraints? e.g. voltage level
      • Is device characteristic 2 returning a result within acceptable constraints? e.g. device temperature,outside air temperature (OAT)
      • Is device characteristic 3 returning a result within acceptable constraints? e.g. response time
      • Is device characteristic n returning a result within acceptable constraints?

Accomplishing these requirements is not free. Some technology investment is needed, but more importantly, an organizational framework that supports this activity is necessary.

Growth of devices & growth of network segments

While we don’t know what the rate of growth of network segments is or will be, we can be pretty sure that it will continue to grow for the foreseeable future. This growth in network segments stems from at least two reasons, 1) network segments whether VRF’s, VLAN’s or other are easier to implement than they used to be, and 2) network segmentation is currently a popular strategy for addressing IoT Systems risk mitigation. (Regarding the latter, I believe that this is in part because we don’t know what else to do — when all you have is a hammer, everything looks like a nail).

There are multiple projections that IoT device count is growing at an exponential rate, such as this Ericsson Mobility Report that suggests a 23% annual rate of growth between 2015 and 2021 and this McKinsey report that suggests 15% – 20% annual growth by 2020. Related projections on IoT market growth can be even higher with annual growth over of 50%.

This ongoing steady (or more) growth year after year appears to be exponential growth. While IoT devices don’t compound each other like dollars do, the growth count curve still appears exponential. To borrow from the idea of compounding (exponential) growth in finance — we add 1 to the rate of growth and raise that sum to the number of years out that we want to project and then multiple that whole thing by the starting count —

(Wikipedia)

So, let’s say that we start with 10,000 IoT devices in a hypothetical city or institution and that the growth rate in IoT device count over the next few years is 20% . That growth curve looks something like this —

exponential device count growth @ 20%

(As a quick side note, we can see the Rule of 72 providing a rough estimate of the time it takes to double, about 3 1/2 years, at work here).

Now let’s say that there are currently 1000 network segments (VLANs, VRFs, etc) in that city or institution and consider two hypothetical growth trajectories — one exponential at the same rate as the device count growth rate and one linear growing at 1000 network segments per year.

Hypothetical exponential growth at 20% and hypothetical linear growth at 1000 segments per year

While don’t know exactly what the network segment growth rate or trajectory will be for cities and institutions in particular or in aggregate, we can expect that count to continue to grow for the next several years. With that growth comes an increased demand on institutional and city resources. The question is, are we planning to resource that increase in demand?

Network segmentation management — impacts on ROI & institutional cyber risk

As mentioned earlier, to manage network segments for successful IoT systems implementations, at least two components are required — manageability of the network segment itself and manageability of the specific IoT System(s) on that network. Without both of these, the likelihood of an IoT System’s success for the city or institution is low. Without both of these, the city or institution can expect to feel negative impacts to ROI, the city or institution’s cyber risk profile, or more likely both.

Because both of these success components require current resourcing (eg staffing and tool investment) and planning for future resourcing, success is not guaranteed. If we’re not thoughtful about implementation, we could end up with broad portfolios of a rapidly growing number of unmanaged or under-managed networks. And that’s not good for any of us.

 

Developing counter-drone technologies not so easy

It’s not easy countering drones.

As fast as the drone industry is growing with hobbyist and commercial drone counts in the low millions by 2021, some anticipate that the counter-drone industry will be even larger with a combined annual growth rate of almost 26% between 2017 and 2021.  Increasing the challenge, because of the multitude of factors involved in drone deployments and the many places of interest where drones can be deployed, whether individually or en masse, there are a surprisingly large number of scenarios to consider when developing and deploying counter-drone technologies and that in turn makes development and deployment of counter-drone technology particularly difficult.

There are obvious concerns amongst commercial, private, and government entities around risk stemming from drone use (to include property loss, privacy, injury, and potentially death) — such as drones over flying nuclear plants in France, drones snooping in the Olympics in Rio, drones with cameras outside 26th floor high rise apartments in Seattle, and many others. The challenge is that there are so many factors and so many interdependencies between those factors, that it can be hard to establish a shared conceptual framework around counter-drone approaches to even have a discussion about the concerns.

Already, many different approaches to counter-drone strategies

The broad and varied approaches of counter-drone technologies already on the market speak to the diversity of potential drone deployments and the areas in which they will operate – and which ostensibly there is a desire to be countered. For example, DroneShield’s DroneGun uses a rifle-like radio frequency jammer. Skywall uses a shoulder-fired drone-capturing net. The US Air Force is trying out net-filled shotgun shells by SkyNet and Malou Tech offers drone-nets-drone technology. And the US Navy and Marine Corps bring it on home with laser, aka directed energy, anti-drone approaches. The approaches are wide-ranging and I think we’ll continue to see even more evolution and many more companies in the space.

One approach for considering counter-drone systems

An approach discussed here splits out variables/attributes in drone deployments and operation from the variables/attributes of the operating area and what it contains into two separate sets of factors. This provides a broad base for conceptualizing counter-drone considerations. These two sets of factors are:

1. drone deployment attributes and
2. drone operating area attributes

In turn, each set of factors also has its own multiplicity of factors and interdependencies between them.

counter-drone sets of factors071017

many factors to consider in counter-drone systems development

 

Drone deployment attributes

When considering how to counter drones, there are many factors that contribute to any particular drone deployment. A few of them are:

  • Mass – How much does the drone weigh?
  • Altitude – Is there a characteristic altitude of the deployment?
    • < 50 feet?
    • 50 – 200 feet?
    • above 200 feet?
    • varies?
  • Potential Energy (PE)
    • derives from mass & altitude
  • Velocity
    • Is there a characteristic speed of the drone(s)
      • 5 mph?
      • 5 – 20 mph?
      • > 20 mph?
      • Varies?
  • Kinetic Energy (KE)
    • derives from velocity and mass of drone
    • rotor rotation also contributes to KE
  • Volume – What is the size of the drone?
    • cross-section?
    • diameter across opposing rotor tips?
  • Cardinality
    • Is it just one drone?
    • 2 – 25 drones?
    • 25 or more drones?
  • Behavior >1 drone
    • is the behavior coordinated?
    • are they operating independently?
    • sporadic?
  • Intent – Is the operator intent:
    • malicious?
    • benign?
    • negligent?
    • operator-impaired?
  • What is the payload of the drone?
    • nothing?
    • camera?
    • explosive
    • radiological?/chemical?
    • combination?

Effects of factor interdependencies (aka Potential Energy can become Kinetic Energy)

As discussed, these factors can have substantial interdependencies between them as well. One of the interdependencies that is the most basic and seems to be often forgotten in conversation is that any drone with any altitude has Potential Energy. Regardless of motors, batteries, wind, payload, etc — just the fact that a mass (drone) has been raised to an altitude, it now has Potential Energy. The larger the mass and the higher the altitude, the more Potential Energy is converted to downward directed Kinetic Energy if a drone stops flying – for whatever reason. This may be important to anyone in the path of the dead drone’s earth-seeking trajectory.

city2

Drone operating area attributes

The other part of the equation is what is the drone’s operating area? What are those things on the ground (or in the air) that have value? This could be the value that one puts on their individual privacy, to the value a corporation puts on a production facility, the value a city puts on a power substation, or the value of 50 people watching a little league game or 50,000 people watching an NFL game. Some operating area attributes/variables include:

stadium2

 

  • Does/do the drone(s) fly over a population?
    • Population size – 10’s, 100’s, 1000’s, more?
    • Density
      • rural
      • city
      • high density congregation (eg professional sports game, parade, concert, ..)
      • does the population density change, eg high when ball games & low when not
    • Does/do drone(s) fly over/near critical infrastructure?
      • power
      • water –  dams, reservoirs, …
      • gas
      • sewer
      • nuclear
    • Does/do drone(s) fly over/near schools, churches, hospitals …
    • Does/do drone(s) fly in area of critical radio spectrum?
      • broadcast
      • police, fire
      • aviation, marine
      • military

    ci2

    Counter-drone Combinatorics

    So let’s take a combination and create a constraint domain to illustrate the point.  Where there is a broad range or spectrum of possibilities for a particular attribute, I break them down into three arbitrarily chosen sub-ranges. This will allow us to make the numbers smaller and a little more manageable and yet still see how the number of possibilities to consider can become quite large. You can, of course, break them down into whatever sub-ranges you would like and into as many sub-ranges as you would like.

    the number of scenarios can scale quickly

    the number of scenarios can scale quickly

    Some counter-drone systems will naturally address multiple scenarios. However, to be effective and competitive in a rapidly growing market, the counter-drone systems developer has many flight profiles and many operating area profiles to consider.

    Another factor in counter-drone technology development is the legal aspect. Examples include laws regarding national aviation system navigational system protection, individual state law, and civil lawsuits. This will likely be a substantially evolving landscape as drone technologies evolve and damaging events occur with real loss.

    Dynamic landscape

    When considering the factors involved in developing or evaluating counter-drone technologies, one approach that can be helpful is to think about both the characteristics of the drone (or drones) potential flight as well as the area in which the flight will occur and what’s valuable in that space.

    With rapidly evolving and distribution of drone technologies, rapidly evolving counter-drone technological approaches, increased public awareness, and changes in local, state, and federal laws, I think investment, development, and deployment of counter-drone technology will continue to be a very dynamic realm.

FAA drone activity timeline

The FAA has found itself in a bold new role over the past few years regarding drone, aka Unmanned Aircraft System (UAS) policy, registration services, guidance, detection, establishment of permanent and temporary no drone zones, studies on drone/human collision risk and impact, and related areas. We can expect that this will continue to be a very active space over the next several years.

A timeline of some FAA drone activity since early 2015 is depicted below. The graph shows the total number of FAA News/Updates and Press Releases for that month. I used the FAA’s website news search engine to get these numbers. (I used the dates from search engine as the basis even though the actual event date might have differed by a few days in a couple of cases.) Some representative links to FAA News/Updates and Press Releases follow the graphic.

FAAdronenewsprtimeline2

Super Bowl XLIX
Super Bowl 50
Super Bowl LI

Other No Drone Zone announcements —

DC kicks off No Drone Zone
Papal visit No Drone Zone press release

Airport drone detection system updates —

Denver International Airport & FAA partner on drone safety
Testing FBI drone detection systems @ JFK
Evaluating drone detection systems in Denver area
Testing drone detection system @ DFW

Other notable updates —

Drone registration site launched
San Francisco 49ers launch drone safety effort
Registration requirements go into effect
FAA responds to Court of Appeals reversal of registration requirements

 

Clearly, drone policy development and enforcement, drone detection and management systems, other drone-related tech, and messaging the same to the public is likely to a substantial growth area and effort for the FAA over (at least) the next several years.

Costing IoT Systems in cities & institutions

Determining the total cost of ownership, operation, and stewardship of IoT Systems for an institution or city has a number of considerations. Some of these considerations are shared with traditional enterprise systems and some are unique to IoT Systems. Lack of realization and acknowledgement of these costs can lead to disappointing and costly IoT Systems outcomes. These disappointing outcomes manifest themselves in the lost ROI of an IoT investment as well as negative impacts to the cyber-risk profile to an institution.

Listing from least complex/nuanced to more complex/nuanced …

  1. Costs to host servers, databases, & redundancy — whether under desks (hopefully not), in local data centers, or in the cloud
  2. Costs to support large numbers of geographically distributed ‘Things’ & devices (the T in IoT) & the different institutional organizations that may be involved (some of which may not see or realize the potential benefit to the institution but may bear at least some of its support costs)
  3. Costs stemming from the natural friction, sometimes small & sometimes large, between multiple historically disparate organizations within an institution as they attempt to coordinate, collaborate, and address problems of understanding to support the system

1. The more familiar part — application server, database server, & redundancy costing

This costing is more closely aligned with traditional enterprise application costing than other IoT systems costs. Application licensing and support agreements are a part of this aspect. Important additional costs to include, though, are: what are the costs of hosting those applications and databases? And what are the costs of supporting that hosting (e.g. who manages the relationships, the tickets, the problems, etc).

The applications servers or virtual machines (VM’s), database servers/VM’s, and redundancy servers/VM’s can be hosted in a local data center, shared data center, the cloud, or similar. Hosting or otherwise servicing these applications and supporting databases have their own costs. In addition to fees for the hosting, there’s also some cost to managing the vendor relationship and agreement/contract.

2. A guy, a truck, and a ladder – a less obvious part of IoT System costing

 

this is not cheap

this is not cheap

Imagine an institution or city that implements 500 smart street lights that provide lighting, sense movement, maybe samples air quality, and possibly monitors and reports street or underground vibration. There will be some failure rate amongst the components in any single device/endpoint and some failure rate across the total number of installed devices/sensors – the T’s in IoT.

In this hypothetical scenario, troubleshooting and/or repair means:

  • deploying 1 or more skilled tradespeople
    • 1 or more for the work & possibly another as a safety observer
  • rolling a truck or trucks
    • with associated vehicle/fleet/fuel costs
  • spending 1-2 hours just to get to the point of troubleshooting/repair
  • 2 – 4 hours troubleshooting/repair
  • 1-2 hours wrap up and return

For this hypothetical example, let’s say the skilled tradespeople involved make $60/hour and their benefit load (expense to institution or city) is 25% for a total $75/hour expense. So, disregarding fleet and related costs, one estimate might be:

(2 hrs prep + 4 hrs on site troubleshooting/repair + 2 hrs recovery) x 2 tradespeople x $75/hr = $1200 per support event

Continuing on the thread, let’s say there’s a 10% / year failure rate (or required hardware/software update rate) of at least some component on a single device or Thing (T in IoT). That would be:

500 devices x 10% repair or troubleshoot rate/year x $1200/event = $60,000 / year

That dollar figure starts to become non-trivial. And that’s just one IoT System. Cities and institutions will have many, a portfolio, of well-managed or less-well-managed IoT Systems. Another hypothetical example is in the example  below —

IoTSysCostingExample050917

multiple costs involved

3. The insidious part — loss from organizational friction

The least apparent and possibly most costly aspect is the loss that occurs from the coordination and collaboration required between organizations and the oversight needed across all of them for a successful implementation. In the idealized scenario of institutional capacity, there is a homogeneous set of resources that include components such as available staffing levels (FTE), requisite skill sets (technical, operational, and interpersonal), support funding, political/institutional will to support the implementation and operation of the IoT System, vendor relationships, and other.

idealizedcapacity

idealized view of IoT System management capacity

In practice, however, this institutional capacity is comprised of many different organizations and the interrelationships between them.  While collaboration and inter-organizational cooperation is typically universally lauded, we all know from personal and professional experience that often collaboration between institutional organizations does not in fact work so well. Research has also been done on this phenomena where, “the discrepancy between the promise of collaboration and the reality of persistent failure” is studied (Koschmann).

Wherever two or more organizations interact with each other, some sort of friction or system loss is present. Metaphorically similar to the 2nd law of thermodynamics, not all of the time, energy, resources put into ostensible institutional IoT System management capacity will be used, or can be used, to do the expected work. In the organizational friction example, losses can come from a multitude of sources where the number of friction sources and the intensity of each can vary from organizational relationship to organizational relationship. Examples of this sort of friction/system loss and resultant loss in expected institutional capacity include:

real capacity stems from multiple organizations with multiple relationships and the friction between

real capacity stems from multiple organizations with multiple relationships and the friction between

The insidious part is that while the friction between any two organizations, may be small and possibly not obvious, these small instances of organizational friction aggregate across the whole institutional and IoT System implementation effort. Further, not all relationships are one-to-one. There are often many relationships where many organizations are involved. Likening to Newton’s Three Body Problem, adding a third planet, billiard ball, or organization can significantly increase the complexity of analysis, prediction/forecasting, and the ability to get work done.

capacityloss

 

Costing differently

In practice, capturing all of these costs can be challenging. #1 — application and database licensing and hosting costs is relatively the most straightforward and for which we generally have the most experience, but that doesn’t mean it’s effortless. We have less experience with #2 for IoT Systems — the guy, truck, & ladder costs — but that support cost can be estimated and the failure rates of devices and device components can be estimated. Costing #3 — aggregated inter-organizational friction is, by far, the most difficult and possibly the most impactful — in part because of its magnitude and in part because of the uncertainty it introduces.

The import thing, I believe, is to acknowledge all of these components, compute and estimate what we can, and work to allow for and hopefully prepare for the unique uncertainty that selecting, procuring, implementing, and managing IoT Systems brings. If we do this work, we have our best chance of reaching anticipated ROI and not degrading (possibly even enhancing) the risk profiles of our cities and institutions.

 

 

Can we manage what we own? — IoT in smart cities & institutions

The rate of growth of IoT devices and systems is rapidly outpacing the ability of an institution or city to manage those same devices and systems. The tools, capacities, and skill sets in institutions and cities that are currently in place were built and staffed for different information systems and technologies — centralized mail servers, file sharing, business applications, network infrastructure support, and similar. Some of these systems still exist within the enterprise and still need robust, effective support while others have moved to the cloud. The important consideration is to not assume that toolsets developed for traditional enterprise implementations are appropriate or sufficient for IoT Systems implementations.

What's manageable- 032217

things are increasing faster than the ability to manage those things

Working from the outside in

Starting with the outer ring, the number of ‘things’ — the T in IoT — is rapidly growing within institutions and cities. From my perspective, an IoT ‘thing’ is a device that computes in some way, is networked, and interacts with its local environment in some way. Further, these systems may be acquired via non-traditional methods. For example, a city’s transportation department may seek and acquire a sensor, data aggregation, and analysis system for predictive maintenance for a particular roadway. This system might have been selected, procured, implemented, and subsequently managed independently of the organization’s traditional central IT organization & processes. Complex and high data producing systems are entering the institution/city from a variety of sources and with little formal vetting or analysis.

Can we even count them?

Because of the rapid growth of IoT devices and systems in concert with alternative entry points into the city/institution, even counting (enumerating)  — these devices — which can compute with growing ability and are networked — is increasingly difficult. This lack of countability in itself is not so bad, it’s just a fact of life – the trouble comes when we base our management systems on the assumption that we can count, inventory, much less manage all of our devices.

What do we know about the devices?

Do we have documentation and clarity of support for the tens, hundreds, thousands (or more) of devices. What do they do? How are they configured? Have we set a standard for configuration? How do we know that that standard is being met? What services do we think should be running on the devices? Are those services indeed running on them? Are there more services than those required running? Are there processes for sampling and auditing those device services over the next 12 – 36 months?  Or did we install them, or have them installed, and simply move onto the next thing?

We can borrow from the construction industry and ask for as-built documentation. What actually got installed? What are the documents that we have to work with to support this system? Drawings? IP addresses? Configuration documents for logins, passwords, open ports/services?

What is manageable?

If we are in the fortunate position to be able to actually count these computing/networked/sensing devices with reasonable accuracy and we know some (enough) things about the devices, then the next question is — do we have the resources — staffing, time, skill sets, opportunity cost, etc — to actually support the devices? Suddenly in smart cities, smart institutions, smart campuses, we’re installing things, endpoints, in the field that may require regular updating (yearly, monthly, …) — and this occurs between the customer network with its protocols/processes and the vendor system that is proposed. Not all (possibly substantial) device updating can be accomplished effectively remotely.

Another challenge is that often the organizations that are charged with staffing, installing, and supporting these deployed IoT devices, such as smart energy meters or environmental monitoring systems, are more accustomed to supporting machines that last for years or decades. Such facilities management organizations have naturally built their planning, repair, and preventative maintenance cycles around longer periods. For example, a centrifugal fan in a building might have a projected lifespan of approximately 25 years, soft start electric motors 25 years, and variable air volume (VAV) boxes with expectancies of 25 years.

Similarly, central IT organizations generally are not accustomed to running out into the field with trucks and ladders to support 100’s, 1000’s, or more of computing, networked devices in a city or institution. So the question of who’s going to do the actual support work in the field is not clear in terms of capacity, skill sets, and costs.

device count vs mgmt ability 032217-3

Actually managing the things

So, if we have all of the above — and that subset gets smaller and smaller — have the decisions been made and priorities established to actually manage the devices? That is, to prioritize, risk manage, and develop process to manage the devices in practice? There’s a good chance that manageable things won’t actually be managed due to lack of knowledge of owned things, competing priorities, and other.

On not managing the things

It is my opinion that we will not be able to manage all of the ‘things’ in the manner that we have historically managed networked, computing things. While that’s a change, that’s not all bad either. However we do have to realize, acknowledge, and adjust for the fact that we’re not managing all of these things like we thought we could. Thinking we’re managing something we’re not is the biggest risk.

We’re moving into a world of potentially greater benefit to the populace via technology and information systems. However, we will have to do the hard work of being thoughtful about it across multiple populations and realize that we’re bringing in new risks with some known — and unknown — consequences.

Organizational-spanning characteristics of IoT Systems

Gaps between institutional organizations implementing & supporting IoT Systems create challenges

Gaps between institutional organizations implementing & supporting IoT Systems create challenges

One of the unique characteristics of IoT Systems, and one that adds to the complexity of a system’s deployment, is that they tend to span many organizations and entities within the institution. This is particularly true in Higher Education institutions with their city-like aspects, multiple service lines, and wide variety of activities in their buildings and spaces. While traditional enterprise systems, such as e-mail or calendaring, are likely to be owned and operated by one or two institutional organizations, IoT Systems involve many and are deployed in the ‘complex and material manifestations’ that characterize buildings and spaces.

A Higher Education institution example might be a research lab that incorporates an automation and environmental control system that involves the facilities organization, the central IT organization, maybe a local/distributed IT organization, the lead researcher (aka Principal Investigator or PI), her lab team, at least one vendor/contractor and probably several other vendors. Between each of these, a gap forms where system ownership and accountability can fall. Everyone sees their piece, but not much of the others. There’s no one monitoring the greater Gestalt of the IoT System. And that’s where the wild things are.

Traditional enterprise systems tend to fall within the domain of central IT with use of the system being distributed around the institution. So with traditional enterprise systems, use is distributed but ownership and operation is largely with one organization. IoT Systems, on the other hand, tend to have multiple parties/organizations involved in the implementation and management, but the ownership is unclear.

IoT Systems are systems within systems within systems ...

IoT Systems are systems within systems within systems …

This lack of ownership can lead to unfortunate assumptions. For example, the end user/researcher in the Higher Ed case is probably thinking, “central IT and the Chief Information Security Officer are ensuring my system is safe and secure.” The central IT group is thinking, “I’ve got no idea what they’re plugging into the network down there … I didn’t even know they bought a new system. Where did that come from? “ The facilities people might be thinking, “Okay I’ll install these 100 sensors and 50 actuators around this building and these two computers in the closet that the vendor said I had to install. The research people and central IT people will make sure it’s all configured properly.” No one is seeing the whole picture or managing the whole system to desired outcomes.

This implementation and management of IoT Systems is a part of what is being explored within Internet2’s IoT Systems Risk Management Task Force in support of  Internet2’s  Smart Campus Initiative.

Technology adoption in other aspects of the building industry

Research has been done in other areas of technology implementation in the building, space, and campus realms that might help shed some light on the multi-organizational challenge that IoT system implementations can bring to institutions.

Research in the Building Information Modeling (BIM) field suggests that buildings have a ‘complex social and material manifestation … [that requires] a shared frame of reference to create.’ Building Information Modeling seeks to codify or digitize the physical aspects of a space or place such that its attributes can be stored, transmitted, exchanged in a way that supports decision-making and analysis.

In their paper, “Organizational Divisions in BIM-Enabled Commercial Construction”, researchers Carrie Dossick and Gina Neff identify competing obligations within supporting/contributing organizations that limit technology adoption.  They point out that BIM-enabled projects are “often tightly coupled technologically, but divided organizationally.” I believe that their observations regarding BIM projects also share common aspects of deploying and managing IoT Systems.

The Dossick/Neff research suggests that mechanical, electrical, plumbing, and fire life safety systems can be as much as 40% of the commercial construction project scope.It is likely that this number will only increase as our buildings and spaces become more alive, aware, and aggregating of information of what goes on in and around these spaces.

For the BIM implementations studied, the research suggests that there are three obligations of the people and groups contributing to the effort and that these can be in conflict with each other.  The obligations are: scope, project, and company.

As I interpret the paper, scope obligation is what a person or group is supposed to accomplish for the effort – what are they tasked to do. That mission is not overarching organization and coordination of the project, but rather specific, often local, tasks that must be done in support of the effort. In the course of that work, participants in the work are naturally “advocating for their particular system.“

Projects bring together temporary teams for a particular purpose. These time boundaries and purpose boundaries create the environment for the project obligation. Specific timelines and milestones can drive the project obligation. This area can be particularly challenging as it can involve negotiations among different providers, the interests of owners, and design requirements.

Finally, obligations to company “emphasize the financial, legal, and logistical requirements” where ownership and management provide the environment and context in which work is done. This also makes sense intuitively as company is whence one’s paycheck comes. Performance today can influence today’s paycheck as well as paychecks down the road.

While not exactly the same, I think there are some parallels with IoT Systems implementation and management.

BIM implementation/adoption -> IoT Systems implementation & management

  • scope -> scope
  • project -> project
  • company -> department/organization

To me, organizational aspects of scope and project are very similar between IoT Systems implementation/management and those observed and analyzed by the research in BIM implementation.

For IoT Systems implementation and management within an institution, internal departments and internal organizations can closely parallel that of the company that the research addresses. A person’s or group’s directives, performance expectations, and paycheck approval ultimately comes from that department or organization so there will be natural alignment there.

Leadership ‘glue’ has its limits

Finally, the paper also points out the role of leadership in an effort where there is new technology to potentially be adopted or leveraged. While strong leadership is clearly desirable, the research suggests that even good leaders often cannot overcome structural organizational problems with great efficiency or effectiveness. The authors also note that research does not yet understand what “organizational resources to be in place for effective collaboration to occur after new technologies are introduced.”

IoT System ownership for implementation & management

Like BIM and related technology adoption, I believe successful IoT Systems implementations have similar institutional organizational challenges. IoT Systems implementations are themselves a part of larger system of institution, organizations, and people. IoT System implementation and management success will, I believe, also require learning to work with these multiple organizations that have inherent competing obligations. While there may be other approaches to evolve to solve these organizational challenges, a reasonable place to start in the near term might be to establish organizational-spanning system ownership and accountability.

Internet2 Chief Innovation Office launches IoT Systems Risk Management Task Force

Internet2 has launched a national Task Force to study risk management needs around IoT Systems in Higher Education and research institutions. The Task Force is composed of Higher Education and research IT and Information Management leaders across the country and will explore the areas of IoT Systems selection, procurement, implementation, and management. At the end of 12 months, the IoT Systems Risk Management Task Force will deliver a set of recommendations for 3 – 5 areas of further in-depth work. (And in the interest of full disclosure, I am Chairing the IoT Systems Risk Management Task Force.)

Internet of Things Systems or IoT Systems offer great potential value to higher education, research, government, and corporate institutions. From energy management, to research automation systems, to systems that enhance student, faculty, staff, and public safety, to academic learning systems, IoT Systems offer great promise. However, these systems need to be implemented thoughtfully and thoroughly or the investment value won’t be realized. Further, because of the distributed computing and networking capabilities of IoT devices, poor IoT Systems implementations can even make things worse for institutions, corporations, or governments.

Internet2 Chief Innovation Office

i2logoThe mission of the Internet 2 Chief Innovation Office, led by Florence Hudson,  is to work with Internet2 members to define and develop new innovations around the Internet. The Innovation Program has three core working groups —

Internet2’s core offerings are its 100 gbps network and their NET+ services.  Their membership includes 300 Higher Education institutions and over 150 industry, lab, and national agency organizations.

Many IoT systems risk topics

Examples of topics that the Task Force will cover include IoT systems vendor management issues, network segmentation strategies and approaches, cost estimating tools and approaches for IoT systems, potential tool development and/or partnering with organizations that perform Internet-wide scanning for IoT-related systems, and the organizational and cultural issues encountered in transitioning to a data-centric organization.

IoT systems vendor management approaches

Organizations and institutions need to raise the bar with IoT systems vendors regarding what constitutes a successfully delivered product or service. For example, has the vendor delivered documentation showing the final installation architecture, have default logins & passwords been change on all devices (how is this demonstrated), have all unnecessary services been deactivated on all devices and systems and how is this demonstrated?

Development of common ‘backends’ for IoT systems

Current IoT systems (to include utility distribution, building automation systems, many others) vendor approaches require that institutions invest in separate and proprietary ‘backend’ architectures consisting of application servers, databases, etc for each different vendor. This is an approach that does not lend itself to manageability, extensibility, or scalability.  In this space, perhaps newer container and container management technologies offer solutions as well as other possibilities.

1200px-Internet_of_things_wilgengebroedDevelopment of network segmentation/micro-segmentation strategies and approaches for IoT Systems

Network segmentation seems to offer great promise for mitigating risk around IoT Systems implementations. However, without appropriate guidance for IoT network segmentation implementation and operation, institutions can end up with a full portfolio of poorly managed network segments. Exploration and development of institutional network segmentation best practices can serve to lower an organization’s risk profile.

Development of cost estimating tools and approaches for IoT Systems

There is little in the way of precedent for cost models for the rapidly evolving IoT systems space and, as such, planning for IoT Systems and trying to estimate Total Cost of Ownership is difficult and nuanced. Exploration of and development of IoT Systems cost models can be of real value to institutions making planning and resourcing decisions.

Development of risk language & risk categories around IoT systems

Currently it is difficult to discuss new risk brought on by IoT systems with enterprise risk managers because IoT systems themselves are difficult to describe and discuss.  Development and socializing IoT risk language, that incorporates existing familiar institutional risk language, would enhance the ability to discuss IoT systems risk at the enterprise level. This Task Force will explore this nuanced space as well.

Analysis tool development and partnering

The Task Force will explore tool development and/or partnerships with organizations that scan the Internet for industrial control systems and IoT systems and publish these results online. Exploring internal tool development of the same is also a possibility. Development of benchmarks and baselines of Internet-scanning results across different industries and market sectors will also be considered.

Organizational cultural barriers to successful implementation of IoT Systems

Changing from a traditional organization to a data centric organization is a non-trivial transition and not addressing these issues can be a barrier to successful implementations of IoT Systems in institutions, organizations, and cities. The Task Force will study this important space as well.

Early Task Force work will also include identifying and enumerating other independent and overlapping risk areas (operational, cyber, cultural, and others). Over the year, Task Force members will participate in phone conferences, listen to subject matter expert presentations, and identify, discuss, and prioritize IoT Systems issues. Finally, recommendations will be made for further focused work on the highest priority areas.  If you have questions, comments, further interest, please contact me ChuckBenson@longtailrisk.com or the Internet2 Chief Innovation Office at CINO@internet2.edu.

 

[IoT image above: By Wilgengebroed on Flickr – https://www.flickr.com/photos/wilgengebroed/8249565455/, CC BY 2.0, https://commons.wikimedia.org/w/index.php?curid=32745541]

A potential IoT systems vendor checklist v2.0

In order to maximize the value of an IoT system to an institution or city, how that system is implemented is critical. Without a thoughtful and thorough implementation, the value of the investment will not be met and, possibly, the value can even be negative through the addition of unmitigated risk to the institution or city.

I’ve updated the IoT systems planning considerations list from an earlier post and created a more checklist-like document to use when working with IoT systems vendors. Earlier versions appeared in posts Institutional considerations for managing risk around IoT,  Developing an IoT vendor strategy, and Systems in the seam – shortcomings in IoT systems implementation. Ideally, this document could be used during the contract development and negotiation phases with the vendor.

The intent of the document is to help compute the Total Cost of Ownership for an IoT systems implementation as well as raise expectations of the vendor for a delivered system. In doing so, we can help mitigate some operational risk (suboptimal business decisions) as well as cybersecurity-related risk (bad guys wanting to use our assets in a malicious manner).

I’ve created rough categorizes of issues falling under operational risk, cybersecurity risk, and those falling in both categories to help provide some additional structure. However, many issues influence each other so it’s not critical to get tied up in the categorization.

IoTChecklistV2-032716

A starting point for an IoT systems vendor checklist

pdf version here

IoT & the Rule of 72

There are many different estimates regarding the growth rate of the Internet of Things (IoT). There are projections of number of connected devices, projections on market capitalization, projections on growth of semiconductor counts supporting those devices, and many others. Because the numbers of devices and systems are so high and these projections are around things that we typically don’t understand well, it’s hard to get a feel for what is actually increasing so rapidly. What is this thing that is growing so rapidly? How fast is it growing? If we can’t roughly understand the magnitudes involved, we can’t discuss, plan, assess, or begin to mitigate risk to our organizations and institutions involving these systems.

Going old school

summa

Summa de arithmetica – Wikipedia http://bit.ly/1MHOuxO

One way to better our ballpark understanding of this rate of growth can be with the old school method of applying the Rule of 72. Introduced by Pacioli in Summa de Arithmetica, the Rule of 72 has been around for over half of a millennium as a mental mechanism to quickly estimate how long it takes a value experiencing exponential growth to double. This works with systems that have parameters that are described by a percentage change over a period of time.  The classic example is interest on a loan or investment that compounds. Because we are used to seeing these kinds of measures in financial, economic,  and political systems, we will see them in IoT conversations also.

To apply the Rule of 72, you take the rate of growth for a period expressed as a percentage and then divide that into the number 72. The result is the number of time periods, typically expressed in years, that it takes for the doubling to occur.

For example, if you buy a house that increases in value by 6% per year, the time to double the value is:

72 / 6 = 12

or 12 years to double. So a $400,000 house purchased today that appreciates by 6% per year will see a value of around $800,000 in 12 years.

(72 is a convenient estimate that facilitates mental division with values such as 2, 3, 4, 6, 8, 12, etc. A more accurate, but less easy to mentally work with, value is closer to 69. This stems from the value for natural log 2, aka ln(2), which is .69314 … For our purposes, we’ll stick with 72.)

Making IoT growth estimates more understandable

As we all try to get our heads around IoT, what it is, and how fast it is growing, we are bombarded by a variety of estimates and figures. We know these numbers seem big, but we’re not really sure how to use these figures or compare them to something else. Being able to quickly compute how long it takes for something to double in quantity can have more meaning for us than trying to interpret growth expressed as a percentage.

In his book Grapes of Math, Alex Bellos does a great job of describing where the Rule of 72 comes from and how it works.  Further he reminds us that economic, financial, political, and other growth measures that describe sales, profits, stock prices, GDP, population, inflation, and more are often stated in percentage growth per year.  Because of our familiarity with communicating this way, we can expect at least some IoT growth projections to be stated this way as well.

Gartner Press Release http://www.gartner.com/newsroom/id/2905717

Gartner Press Release http://www.gartner.com/newsroom/id/2905717

Gartner’s installed IoT base estimate from late 2014 suggests exponential growth — 25% growth from 2013 to 2014, 30% growth from 2014 to 2015, and what looks like almost 40% annual growth from 2015 to 2020.  If this is the case, then we can estimate 72 / 40 = 1.8 years to double. So, if we started with the almost 5 billion devices indicated in the 2015 column, we’d have 10 billion in about 22 months, sometime in 2017 — 1.8 x 12 months.

GartnerPWCAnalysis

Analysis of IoT growth on semiconductor industry – http://pwc.to/1kwDuNc

This Gartner/PriceWaterhouseCoopers analysis shows a CAGR growth for sensors and actuators of approximately 10%.  Applying the Rule of 72 for an estimate, we can expect to see the number of sensors and actuators deployed in the world around us to double in ~72 / 10 = 7.2 years — less than 2 presidential terms. What will twice the number of sensors and actuators around us look like?

According to this IDC report, the IoT market will see 19% growth  for a market size doubling in a little under 4 years (72/19 = 3.8). The biggest growth area was 40% CAGR in the automotive sector for a market doubling in under 2 years.

BIIntelIotGrowth

Lots more connections … http://bit.ly/1msfrjG

This Business Insider report suggests a 45% year over year growth from 2 billion in 2014 to 9 billion in 2018 for connection count doubling in 72/45 = 1.6, a little over a year and a half.

And finally, ON World predicts a 250% growth in wireless light bulbs for a doubling in every ~ 3.5 months.

Limitations

It’s important to note that we don’t know what IoT growth will actually look like over several years. We have some initial data from the first few years that seem to suggest that this growth will be exponential versus linear growth, for example.  Also where the Rule of 72 was initially applied — money growth (compounding) —  is a recursive context — money grows because there is money to act on (and time). IoT growth will come from something else.  At least for now, it’s not obvious that IoT growth is or will be recursive* — we don’t know that many IoT deployments this year will cause even more deployments next year, and then that next year’s increased deployments will cause yet an even higher incremental increase the following year, and so on.

*[One frightening possibility, of course, is the Skynet scenario from Terminator where conscious machines build conscious machines and recursion in full play …]

If, however, IoT growth roughly mimics or correlates to compounding growth (for whatever reason), then we can use the Rule of 72 to help us quickly estimate magnitudes and time scales and add some context to our conversations. With more context around the phenomenon of IoT, the better are our chances for managing the risk to our organizations that comes from its proliferation.