New book on managing IoT Systems out this week

Leave a reply

My new book is coming out this week. Printed version out Thursday & electronic versions are out now on several sites including Amazon. “Managing IoT Systems for Institutions and Cities” is for institutional and city planners, managers, and operators as well as academics and IoT Systems vendors.

Topics include:

  • Differences between traditional enterprise IT and IoT System deployments
  • Establishing success criteria for IoT Systems deployments in your institution or city
  • Considerations for cities and institutions across the life cycle of IoT Systems:
    • IoT System selection
    • Procurement of IoT Systems
    • IoT System implementation and deployment
    • Managing and operating IoT Systems
    • Retiring IoT Systems
  • IoT Systems vendor relationships and vendor management
  • Importance of manageability of IoT Systems
  • Assessing the manageability of different IoT Systems
  • Managing multiple interdependent IoT Systems with the institution or city
  • Institutional strategies for managing these systems

Printed version available Thursday 7/17/2019. Electronic versions available now. Available from:

CRC Press

Amazon

Taylor and Francis direct

The Telegraph Bookshop

Routledge

and others …

Hearing transcript: China, the United States, and Next Generation Connectivity

Leave a reply

hearing transcript – IoT, 5G, US & China

The transcript of the hearing on “China, the United States, and Next Generation Connectivity” before the US-China Economic and Security Review Commission, addressing IoT and 5G technologies, came out this week and is available here. My testimony is early in the document. See earlier post for overview.

Testimony before US-China Economic & Security Review Commission re IoT & 5G

Leave a reply

Earlier this month, at the invitation of the US-China Economic and Security Review Commission, I submitted written testimony and subsequently testified at the China, the United States, and Next Generation Connectivity hearing regarding IoT Systems risk mitigation for institutions and cities as well as considerations regarding 5G deployments on the same.

A copy of the written testimony is here. A transcript of the oral testimony will be available in the next weeks.

The testimony discussed potential benefits of IoT Systems for US government, cities, universities, other institutions, and companies. It also discussed risks to those same entities from IoT Systems implementations. The risks discussed include:

  • Supply chain risks
  • Poor selection, procurement, implementation, and management of IoT Systems
  • Lack of institutional governance and lack of awareness of social-technical issues in IoT Systems deployments

Prior to the testimony, I was asked how the US government could help. I suggested these four areas in the testimony:

  • Standardized provenance vetting and reporting for IoT device components
  • Support for increased US labor force training in Operational Technology (OT) skill sets
  • Support for development of institutional and city IoT governance frameworks
  • Support for data ethnography and socio-technical research and application in context of IoT Systems

The testimony also included comments on supply chain risks:

Provenance of multiple components across thousands, millions (or more) devices is challenging

As well as aspects previously discussed in this blog such as the ability of an institution or city to manage their IoT systems:

where the wild things are [reference to Sendak]

Full written testimony is here:

https://www.uscc.gov/sites/default/files/Chuck%20Benson_Written%20Testimony.pdf

Bathtubs, manageability, & IoT

Leave a reply

The limited funding and staffing resources inherent in almost all institutions and cities creates a delicate balance between IT systems operations, managing institutional risk, and cybersecurity operations. A critical component to this balance is systems manageability. Implementing unmanaged/under-managed systems can quickly perturb this balance and cause reactionary spending, such as on cybersecurity incident response, institutional reputation damage control, unplanned systems repair dollars, as well as others.

IoT Systems — with their multi-organizational boundary spanning, unclear systems ownership and accountability, lack of precedence for implementation, and high number of networked computing devices (‘Things’) — are particular candidates for unmanaged/under-managed systems in a city or institution. 

Systems manageability

IT systems that tend to be more manageable allow for more predictability in an institution’s resource and cashflow planning.  Criteria for high systems manageability include:

  • having well-defined performance expectations
  • thoughtful and thorough implementation
  • accessible training and documentation
  • strong vendor support
  • others

Unmanaged or under-managed systems increase the likelihood of a cyber event such as device compromise or whole system compromise as well as facilitate potentially substantial operations disruption and unplanned financial burden. 

Bathtub modeling

We can use some concepts from stocks and flows diagrams where the stock is represented by a bathtub to create a basic model of resource availability in this delicate dance of balancing of resources for IT systems operations, cybersecurity operations, and managing institutional risk.

My understanding that the use of a bathtub to represent stocks and flows goes back to 2000 when John Sterman and Linda Booth Sweeney published results of an experiment on how people understand and interpret complex systems. On a related note, I found the book, Thinking in Systems, by the late Donella Meadows to be a very consumable and helpful introduction to stocks and flows diagrams.

bathtub metaphor for stocks and flows

The idea is that the ‘stock’ is the level of water in the tub. Water can flow into the tub, raising the tub level, and that amount can be varied by some mechanism(s) or external constraints. Similarly, water can flow out of the bathtub, draining the tub, and there is a mechanism for controlling the rate of that outflow. And, of course, both could happen at the same time.

Bathtub of bucks

inflows of $$ increase the tub level, outflows of $$ decrease the tub level

Now, imagine that instead of water, the tub holds metaphorical dollars. The tub can be thought of as an account, a set of funds, ‘budget number’, set of budget numbers, or similar. The inflows then are one or more sources for adding dollars to that tub with a mechanism or set of constraints that determines the rate of flow into the tub. Similarly, there is a mechanism for setting how much flows out of the tub (spending or investing).

City and institutional spending

Cities and institutions have multiple sources of inflows, most of which they probably don’t control. Those inflows have independent characteristics from each other as well as some interdependencies with each other. The main takeaway is that the city or institution probably does not control a whole lot regarding what’s coming in.

The spending from the top tub can go to multiple places, themselves other tubs. From the top bathtub, most organizations make decisions between operational dollars (running things) and capital dollars (buying or building big things).

splitting between operational $$ (running things) & capital $$ (buying or building things)

IT & cybersec resources & spending

From the operational dollars tub, some funding goes to IT operations, some goes to cybersecurity operations (eg CISO’s office), and other funding goes to many other traditional and important areas such as HR, finance, policy/law enforcement, and others.

Operational dollars, in turn, gets disbursed across multiple other operational tubs

In the interest of keeping the diagram simpler for our discussion, we won’t include capital spending or non-IT/cybersec spending in subsequent diagrams.

IT systems services and cybersecurity services

Funds from the IT operational bathtub are used to resource the management of various IT systems and sub-systems in the institution or city. This includes both on-premise systems as well as cloud-based systems. Examples include enterprise resource management (ERP) systems, institutional learning/training systems, calendaring and email systems, and others.

Systems that have known performance expectations and implementation precedents (either themselves or peer implementations) can provide the basis for a fairly reasonable calculation to be made on required staffing and funding support requirements.

Similarly, the city/institutional department/organization providing information security services  (usually the CISO’s office) also has a set of well-managed services that are planned for and delivered. Examples of these information security services might include: education and outreach, incident management capability, privacy policy guidance, intelligence analysis, and others. The CISO’s office will work to develop services and capabilities based on the IT systems that the city or institution is operating, known and evolving threats and vulnerabilities, existing risk levels, and others.

resourcing planned and reasonably well-managed systems and services

 

The trouble with unplanned, under-managed, and unmanaged systems

Managing and identifying management support resources can be challenging enough with known systems. Challenges and institutional risk quickly becomes exacerbated though when unplanned or weakly planned systems are added. For example, after the budget/planning cycle, an influential person or group may decide that the city or institution “must have” System X. And then later someone else with influence might insist on (unplanned) System Y.

When these unplanned or under-planned systems are added, several deleterious things can happen:

  • the unplanned system drains from the IT operational funding tub in the forms of implementation staffing, management staffing, and support tools  
  • planned systems now no longer have their expected resources and they themselves can become under-managed in addition to the add-on system that is very likely also to be under-managed
  • institutional/city risk increases because unmanaged/under-managed systems increase likelihood of system comprise due to misconfiguration, mismanagement, lack of oversight, failure of (or lack of application of) controls
  • things get worse as the problem also transmits to a different bathtub, ie the information security services provider for the city or institution, eg the CISO’s office
  • when compromise occurs — particularly on systems that the CISO’s office could not plan for — the CISO’s office is now forced to work in a reactionary mode. This is expensive and pulls resources from planned cybersecurity services

unplanned, under-managed systems also transmit their problems to the CISO’s office in the form of increased likelihood of systems compromise

IoT Systems often fall into the unplanned, under-managed category

Several aspects of IoT Systems deployments can contribute to them having high risk of being weakly planned and under-managed systems —

  • lack of precedent for implementation & management
    • cities/institutions don’t have deep experience with these systems
      • true for all phases – systems selection, procurement, implementation, & management
    • few, if any, peer cities/institutions from which to learn for systems expected to last years or decades (sufficient time hasn’t gone by)
  • accountability and ownership unclear
    • IoT systems span many organizations within a city or institution
    • most organizations are not familiar or practiced at coordinating with each other in this role
  • acquisition path – IoT Systems can come into the institution through many non-traditional paths
    • these IoT Systems are rarely acquired by central IT
    • even if acquired through central IT, traditional systems vetting approaches not sufficient
  • no established vetting of IoT systems prior to purchase
    • performance expectations unknown or unclear (see ownership above)
  • the city or institutional department acquiring the system might not be the one supporting the system
  • Newness and rapid evolution IoT Systems makes them hard to discuss, categorize, and plan for

the newness, novelty, and rapid evolution of IoT Systems will continue to make very susceptible to being under-managed systems

Rapid evolution of IoT Systems vs glacial pace of institutional change

While there are no silver bullets or magic technologies (and we shouldn’t spend much time looking for them) to address these added risks that IoT Systems bring, there are things that we can do now, or at least begin now, that can positively impact our risk exposure as institutions and cities. While we’re interested in mitigating risks that we have now from IoT Systems, the impact of IoT systems in our cities and institutions in the future will be much higher.

Some things that can be done now include —

  • establish a set of criteria for your city’s or institution’s for IoT Systems
    • circulate and share this across the organization
    • one starting point is here
    • a related document from the Internet2 IoT task force on IoT risk  is here
  • identify IoT Systems ownership and accountability
    • require before acquisition
  • identify institutional language used to communicate traditional risk & incorporate that into IoT risk conversations, guidelines, and planning
  • consider an IoT Systems oversight group for your city or institution

Making broad changes, perception changes, and policy changes in cities and institutions is arduous work that takes time, leadership, political capital, and patience.  It is important that we begin now because this level of institutional change will likely take some time and the impact of not making the changes is increasing rapidly.

 

Supporting (& paying for) the network segments that support IoT Systems

Leave a reply

Network segmentation is often promoted as the answer to IoT device and systems management and risk mitigation for an institution, city, or corporation. While segmenting networks is important, a subtle problematic aspect is that:

  1. segmenting networks takes work, energy, and resources in the form of initial investment and ongoing management and oversight
  2. numbers of network segments growth may well track with IoT device count growth – which, at least for the next few years, appears exponential
  3. cities and institutions may not be planning for increased network management resources to support IoT Systems deployments

Success criteria for an IoT System implementation

I use a two overarching component criteria to define a successful IoT System implementation for a city or institution —

1. ROI – does the system perform as expected for the actual (vs projected) costs of deployment and subsequent management

2. Cyber risk – did the implementation of the IoT System make the city or institution worse off in the course of deploying and operating the system?

A key aspect to both of these criteria is system manageability. An unmanageable or difficult to manage system costs more in terms of staffing, rework, repair/updating, and operational disruption. At the same time, a difficult to manage system can create cybersecurity vulnerabilities — both seen and unseen — and divert limited institutional resources from existing operational, cybersecurity and risk mitigation activities.

Similarly, as IoT Systems need to be manageable to positively (at least not negatively) affect ROI and institutional risk profile, the network segments supporting IoT Systems also need to be manageable.

Managing the network segments that support IoT Systems

Networks are no longer, “make this thing talk to that thing,” or “make these things talk to those things.” Network management requires a robust set of supporting system services that support consistent connectivity, resilience, real-time health reporting capabilities, and rapid network diagnostic capabilities.

In addition to these core network support services, for IoT Systems deployments, there is another overarching criteria needed to successfully support and manage networks that support IoT Systems —

The IoT System owner should be able to measure, monitor, and determine performance of the IoT System(s) at any particular point in time. This supports both the effort of determining ROI as well as providing visibility for cyber risk mitigation.

In the case of a city, the IoT System owner might be the city’s transportation department while the network provision is provided by or contracted through the city’s central IT organization. In the case of an institution, such as a research university, the IoT System owner might be an academic department purchasing and deploying an IoT System to support a research grant while the underlying supporting network segment is supported by the university’s central IT organization.

The IoT System owner should not be relying on the network provider to provide IoT application/system management and diagnostic services. IoT Systems are evolving so rapidly, it is very unrealistic to expect the network services provider to have the resources or wherewithal to keep up with the nuances of each new IoT System deployment, much less manage the performance expectations for a rapidly growing number of different IoT Systems.

Examples of IoT System/application-specific network services include:

  • enumeration – how many things/devices are on this network?
    • Is this count different from yesterday? By how much?
  • identification – how many of these devices belong to my IoT System? Have I seen them before? Yesterday? a month ago? How rapidly is this changing?
    • Is what I am seeing different from what I was expecting?
      • (Did I know what I was expecting?)
  • Application-specific network device awareness and health
    • Device heartbeat – are you there?
    • Device performance specifics –
      • Is device characteristic 1 returning a result within acceptable constraints? e.g. voltage level
      • Is device characteristic 2 returning a result within acceptable constraints? e.g. device temperature,outside air temperature (OAT)
      • Is device characteristic 3 returning a result within acceptable constraints? e.g. response time
      • Is device characteristic n returning a result within acceptable constraints?

Accomplishing these requirements is not free. Some technology investment is needed, but more importantly, an organizational framework that supports this activity is necessary.

Growth of devices & growth of network segments

While we don’t know what the rate of growth of network segments is or will be, we can be pretty sure that it will continue to grow for the foreseeable future. This growth in network segments stems from at least two reasons, 1) network segments whether VRF’s, VLAN’s or other are easier to implement than they used to be, and 2) network segmentation is currently a popular strategy for addressing IoT Systems risk mitigation. (Regarding the latter, I believe that this is in part because we don’t know what else to do — when all you have is a hammer, everything looks like a nail).

There are multiple projections that IoT device count is growing at an exponential rate, such as this Ericsson Mobility Report that suggests a 23% annual rate of growth between 2015 and 2021 and this McKinsey report that suggests 15% – 20% annual growth by 2020. Related projections on IoT market growth can be even higher with annual growth over of 50%.

This ongoing steady (or more) growth year after year appears to be exponential growth. While IoT devices don’t compound each other like dollars do, the growth count curve still appears exponential. To borrow from the idea of compounding (exponential) growth in finance — we add 1 to the rate of growth and raise that sum to the number of years out that we want to project and then multiple that whole thing by the starting count —

(Wikipedia)

So, let’s say that we start with 10,000 IoT devices in a hypothetical city or institution and that the growth rate in IoT device count over the next few years is 20% . That growth curve looks something like this —

exponential device count growth @ 20%

(As a quick side note, we can see the Rule of 72 providing a rough estimate of the time it takes to double, about 3 1/2 years, at work here).

Now let’s say that there are currently 1000 network segments (VLANs, VRFs, etc) in that city or institution and consider two hypothetical growth trajectories — one exponential at the same rate as the device count growth rate and one linear growing at 1000 network segments per year.

Hypothetical exponential growth at 20% and hypothetical linear growth at 1000 segments per year

While don’t know exactly what the network segment growth rate or trajectory will be for cities and institutions in particular or in aggregate, we can expect that count to continue to grow for the next several years. With that growth comes an increased demand on institutional and city resources. The question is, are we planning to resource that increase in demand?

Network segmentation management — impacts on ROI & institutional cyber risk

As mentioned earlier, to manage network segments for successful IoT systems implementations, at least two components are required — manageability of the network segment itself and manageability of the specific IoT System(s) on that network. Without both of these, the likelihood of an IoT System’s success for the city or institution is low. Without both of these, the city or institution can expect to feel negative impacts to ROI, the city or institution’s cyber risk profile, or more likely both.

Because both of these success components require current resourcing (eg staffing and tool investment) and planning for future resourcing, success is not guaranteed. If we’re not thoughtful about implementation, we could end up with broad portfolios of a rapidly growing number of unmanaged or under-managed networks. And that’s not good for any of us.

 

IoT & Smart Cities

Leave a reply

Just-released book on Smart Cities from College Publications and available on Amazon. (Chapter 4 – “IoT Systems – Systems Seams & Systems Socialization” is my contribution.)

From the back cover —

“This book brings together some of the best thinking about how cities work in today’s technology-enabled world. It reveals insight from many system perspectives to show in detail how cities can become smarter, be understood and meet the needs of the citizens. It explains how cities can sustain themselves under the increasing pressures of change.

An important thread between the authors is that from whatever point of view you start from in the city it is vital to be able to see the city as a complex adaptive system. Cities are extremely complex in their nature, being highly connected socially, economically and environmentally. The systems perspective allows underlying patterns, threats and opportunities to be understood and worked with in a constructive manner.

All of the systems perspectives recognise that people form the true fabric of the city. The architecture, the technology and the systems are transient and need constant management and renewal to continue to improve the lives of the citizens.

The city is a diverse system of systems that attracts the majority of the planets population. It is wholly dependent on the natural systems in which it resides to supply it with clean air, usable water, food, energy and a valuable quality of life. Without a systems perspective cities quickly become unsustainable. For the first time technology is now able to integrate the systems within cities making them more efficient and more effective while using resources wisely. Increasingly cities are working together across the globe to learn how make good use of technology to improve the lives of citizens. This book explains how this can be done.

A must read for people who operate city systems, design city policy and provide products and services to cities that aim to improve the quality of city life.”