Decomposing Multivariate Information in Complex Systems

In my talk I will discuss the information decomposition in the context of decision and game theory. I will restrict the discussion to the case of two sources and one target variable. I will contrast the "forward" case where the sources cause the target with the backward case that the target cause the predictors. In the latter case the source variables can be considered as an "information structure" in the sense of game theory. I will start with decision theoretic operationalisation of the BROJA measure and will then discuss potential extensions within a game theoretic context.

Currently there is no universally-accepted information decomposition. In fact, most existing proposals exhibit drawbacks such as counter-intuitive behavior on simple examples, lack of a clear operational interpretation, or being limited to two sources. I will show that these problems can be overcome by appropriately formalizing the analogy between information theory and set theory which originally inspired the approach of Williams and Beer (e.g., where information-theoretic redundancy is analogous to the size of set intersection, etc.). Specifically, I will show that one can define a multivariate information decomposition using any partial order that specifies when one source is "more informative" about the target than another. A natural choice of the "more information" relation is the Blackwell order, which leads to an information decomposition with intuitive algebraic, axiomatic, and operational meaning. Time permitting, I will also discuss the implicit and potentially problematic assumption of "the inclusion-exclusion principle" in the context of information decompositions.

One of the key principles of information processing is that it is substrate-independent – i.e. the same computation can be implemented by multiple systems obeying different physical laws. In this talk, I will illustrate how the principles of information decomposition (in particular, metrics of synergy and redundancy) can provide such a substrate-independent description of computation in neural systems by linking biological and artificial brains. First, I will show results obtained from fMRI data linking synergy in neural dynamics with consciousness and high-level cognitive function. Then, I will show results from a study of artificial neural networks, showing that synergy increases as neural networks learn novel tasks, possibly aiding in the process of generalizing learned representations; while redundancy helped the network sustain random perturbations. Together, these results suggest different functional roles of synergy and redundancy for computation in neural systems.

This talk will delineate how the framework of information decomposition reveals information processing principles that govern how brain networks navigate the fundamental trade-off between robustness and integration. Using multimodal neuroimaging data, we establish a hierarchical synergy-to-redundancy gradient across the human brain, which recapitulates the hierarchy from sensory to cognitive systems. Redundant interactions are associated with modular input-output systems, supported by an underlying backbone of structural connectivity. In contrast, synergistic interactions support integrative processes and are the fundamental drivers of complex human cognition. From microscale cellular, molecular, and genetic properties, to macroscale network organisation, convergent evidence delineates a complex architecture that is capable of leveraging synergistic information to a greater extent than other primates. Synergistic information is specifically more prevalent in humans than macaques, and high-synergy regions exhibit the highest degree of evolutionary cortical expansion compared with chimpanzees. I will further show that the capacity of brain networks to support synergistic dynamics is shaped by the network organisation of the structural connectome on which these dynamics unfold. Empirically, we find that patients with reorganised structural network topology due to brain injury exhibit less prevalence of synergistic interactions. To assess whether this observation is causally related to network reorganisation, we employed biophysical whole-brain computational modelling to simulate neurobiologically realistic brain dynamics based on empirical connectomes. Our model shows that loss of synergistic interactions can be simulated in silico based on perturbed network topology alone. To identify the topological determinants of this phenomenon we turn to network control theory, which characterise how the organisation of a structural network shapes its ability to support different dynamics. We show that injuries that diminish the controllability of structural brain networks, systematically diminish the prevalence of synergy in the dynamics taking place across the network. Finally, having established how network structure shapes synergistic dynamics, I will address how the PID framework can in turn shape our understanding of networks. I will delineate an approach to network decomposition inspired by PID, applicable at both local and global scales, and showcase its application to brain networks spanning over 100 mammalian species. Overall, my talk will outline the key role of synergistic neural interactions in the evolution and functioning of humans’ cognitive abilities. Combining information decomposition with computational approaches from network control theory and biophysical modelling, I will further demonstrate that the synergistic dynamics at the core of this functional architecture are causally dependent on the topology and controllability of large-scale structural brain networks. This work unveils the rich interplay of network structure and dynamics for information processing in complex systems.

High-order, beyond-pairwise interdependencies are at the core of biological, economic, and social complex systems, and their adequate analysis is paramount to understand, engineer, and control such systems. Firstly I will describe a framework to measure high-order interdependence that disentangles their effect on each individual pattern exhibited by a multivariate system. Then I will move to introduce the gradients of the O-information as low-order descriptors that can characterize how high-order effects are localized across a system of interest. Ref. [1] Gradients of O-information: Low-order descriptors of high-order dependencies T. Scagliarini, D. Nuzzi, Y. Antonacci, L. Faes, F. E. Rosas, D. Marinazzo, and S. Stramaglia Phys. Rev. Research 5, 013025 – Published 19 January 2023 [2] Quantifying high-order interdependencies on individual patterns via the local O-information: Theory and applications to music analysis Tomas Scagliarini, Daniele Marinazzo, Yike Guo, Sebastiano Stramaglia, and Fernando E. Rosas Phys. Rev. Research 4, 013184 – Published 4 March 2022

Network theory is often based on pairwise relationships between nodes, which is not necessarily realistic for modeling complex systems. Importantly, it does not accurately capture non-pairwise interactions in the human brain, often considered one of the most complex systems. In this work, we develop a multivariate signal processing pipeline to build high-order networks from time series and apply it to resting-state functional magnetic resonance imaging (fMRI) signals to characterize high-order communication between brain regions. We also propose connectivity and signal processing rules for building uniform hypergraphs and argue that each multivariate interdependence metric could define weights in a hypergraph. As a proof of concept, we investigate the most relevant three-point interactions in the human brain by searching for high-order "hubs" in a cohort of $100$ individuals from the Human Connectome Project. We find that, for each choice of multivariate interdependence, the high-order hubs are compatible with distinct systems in the brain. Additionally, the high-order functional brain networks exhibit simultaneous integration and segregation patterns qualitatively observable from their high-order hubs. Our work hereby introduces a promising heuristic route for hypergraph representation of brain activity and opens up exciting avenues for further research in high-order network neuroscience and complex systems.

Metrics of functional connectivity have been instrumental in advancing our knowledge of how the brain communicates information. Historically, the emphasis of previous studies has been on quantifying redundant functional connectivity, which entails determining when activity across multiple sites is similar. Recent research, however, has used Partial Information Decompositions (PID) to identify also synergistic connectivity, that is, connectivity that generates information that is not incorporated in either site alone. In this study, we disentangled the relationship between functional connectivity, stimulus-related synergy, and redundancy at the level of the individual neuron in the mouse primary auditory cortex during a perceptual discrimination task. Specifically, we first identified pairs of neurons whose functional connectivity was directed, as measured by Granger Causality. Using Partial Information Decomposition, we then quantified the amount of stimulus-related redundant and synergistic information conveyed by these neurons. Our results revealed that functionally connected pairs contain proportionally more redundancy and less synergy than unconnected pairs, indicating that their functional connectivity is predominantly redundant. Additionally, we observe that the proportion of redundant connectivity is greater for correct behavioral choices than for incorrect ones, supporting the notion that redundant connectivity is advantageous for behavior.

We offer a new approach to the information decomposition problem in information theory: given a ‘target’ random variable co-distributed with multiple ‘source’ variables, how can we decompose the mutual information into a sum of non-negative terms that quantify the contri- butions of each random variable, not only individually but also in combination? We define a new way to decompose the mutual information, which we call the Information Attribution (IA), and derive a solution using cooperative game theory. It can be seen as assigning a “fair share” of the mutual information to each combination of the source variables. Our decomposition is based on a different lattice from the usual ‘partial information decomposition’ (PID) approach, and as a consequence the IA has a smaller number of terms than PID: it has analogs of the synergy and unique information terms, but lacks separate terms corresponding to redundancy, instead sharing redundant information between the unique information terms. Because of this, it is able to obey equivalents of the axioms known as ‘local positivity’ and ‘identity’, which cannot be simultaneously satisfied by a PID measure.

Partial Information Decompositions (PIDs) are typically introduced by defining specific base-concepts of information, such as redundant information, union information, or synergistic information. However, a comprehensive understanding of the logical organization of these different based-concepts and their associated PIDs remains elusive. In this work, we apply the mereological approach to PID that we introduced in a recent paper to shed light on this problem. Within the mereological approach, base-concepts can be expressed in terms of conditions phrased in Boolean logic on the specific parthood relations between the PID components and the base-concept in question. We set forth a general pattern of these logical conditions of which all PID base-concepts in the literature are special cases and that also leads to novel base-concepts, in particular a concept we call "vulnerable information".

This would be a talk about recently published work available here: https://www.mdpi.com/1099-4300/25/4/648 The corresponding abstract reads: Information-theoretic quantities reveal dependencies among variables in the structure of joint, marginal, and conditional entropies while leaving certain fundamentally different systems indistinguishable. Furthermore, there is no consensus on the correct higher-order generalisation of mutual information (MI). In this manuscript, we show that a recently proposed model-free definition of higher-order interactions among binary variables (MFIs), like mutual information, is a Möbius inversion on a Boolean algebra, except of surprisal instead of entropy. This provides an information-theoretic interpretation to the MFIs, and by extension to Ising interactions. We study the objects dual to mutual information and the MFIs on the order-reversed lattices. We find that dual MI is related to the previously studied differential mutual information, while dual interactions are interactions with respect to a different background state. Unlike (dual) mutual information, interactions and their duals uniquely identify all six 2-input logic gates, the dy- and triadic distributions, and different causal dynamics that are identical in terms of their Shannon information content.

Even simply defined, finite-state generators produce stochastic processes that require tracking an uncountable infinity of probabilistic features for optimal prediction. For processes generated by hidden Markov chains, the consequences are dramatic. Their optimal predictive models, known as epsilon machines, are generically infinite state. Until recently, one could determine neither their intrinsic randomness nor structural complexity. However, new methods have been developed to accurately calculate the Shannon entropy rate (randomness) and to constructively determine their minimal set of predictive features. We also address the complementary challenge of determining how structured hidden Markov processes are by calculating their statistical complexity dimension—the information dimension of the minimal set of predictive features. This introduces a scaling law for the minimal memory resources required to optimally predict a broad class of truly complex processes.

Pyramidal neurons are neurons that have two separate trees of dendrites: apical and basal. These two distinct trees of dendrites are hypothesized to treat two classes of input with different roles, e.g. feedforward and feedback inputs. There is still, however, a lack of understanding of the roles of these distinct classes. Given that pyramidal neurons are abundant in the neocortex and can be found in various regions of the brain that serve inherently different tasks such as sensory, cognitive and motor tasks. It is safe to assume that pyramidal neuron architecture contributes in important ways to neocortical information processing capacity. Thus, a better understanding of the role of the two input classes promises to further our understanding of cortical computation and to also improve the design of artificial neural networks. To describe information processing in pyramidal cells independent of the semantics of the specific tasks above, their neural function can be formalized within the mathematical domain of information theory. Information theory is relevant to the description of pyramidal neurons with their input classes via its recent extension, partial information decomposition (PID). PID offers to characterize the information in the firing of the neuron as information that is (i) redundant in any of the two classes of input, (ii) unique to a specific class of input, (iii) synergistic requiring both classes of input jointly, and (iv) completely independent of the classes of input (i.e. driven by stochasticity). Thus, in this functional formulation, pyramidal neurons will tune their firing properties and synaptic weights to optimize types of information contributions relevant to their role. Therefore, building artificial neurons based on the PID formalism enables simulating various networks containing pyramidal-like neurons to gain new insights into the cortical computation. To design such an artificial neuron based on the PID formalism, we prefer a differentiable PID measure that quantifies the aforementioned types of information. This preference not only helps in facilitating the learning, but more importantly grants access to concrete learning rules that potentially helps in assessing the biological plausibility of “learning with PID”. We introduced such a measure in 2021 - enabling the design of “infomorphic” neurons inspired by pyramidal neurons that learn to fire by optimizing any prescribed combination of redundant, unique, synergistic information of their inputs and the stochastic information about their firing. In this presentation, we will explain the design of infomorphic neurons, show how to build various forms of learning scheme from first principles using PID, and demonstrate a minimalistic infomorphic network built as a Hopfield-like memory network that significantly outperforms the memorization capacity of classical Hopfield networks.

Neural networks rely on coordination among individual neurons to perform complex tasks, but in the brain, they must operate within the constraints of locality for both computation and learning. Our research uses an information-theoretic approach to better understand how locality affects neural networks' structure and operation. We employ Partial Information Decomposition (PID) to quantify unique, redundant, and synergistic information contributions to a neuron's output from multiple groups of inputs. Using this conceptualization, we derive a very general, parametric local learning rule that allows for the construction of networks that consist of locally learning neurons, which can perform tasks from supervised, unsupervised, and associative memory learning. We have recently scaled our approach, demonstrating its potential as an alternative to deep neural networks in machine learning. Our framework provides a powerful tool for investigating the information-theoretic principles underlying the operation of living neural networks and may facilitate the development of locally learning artificial neural networks that function more closely to the brain.

Despite their widespread adoption, the inner workings of Deep Neural Networks (DNNs) remain largely unknown. One key aspect of DNN learning is how the hidden layers' activation patterns encode the ground-truth label that the network is supposed to predict. This task-relevant information is represented jointly by groups of neurons within a layer. However, the specific way in which this mutual information about the classification label is distributed among the individual neurons is not well understood: While parts of it may only be obtainable from specific single neurons, other parts are carried redundantly or synergistically by multiple neurons. A better understanding of this representation may in the future help to identify problems during the training process or guide the selection of a network architecture. We show how Partial Information Decomposition (PID), a recent extension of information theory, can disentangle these different contributions. From this, we introduce the measure of "Representational Complexity", which quantifies the difficulty of accessing information spread across multiple neurons. The quantity reflects the smallest number of neurons that need to be considered jointly to retrieve a piece of information on average: A representational complexity of C=1 means that all pieces of information can be obtained from single neurons, while C is equal to the total number of neurons if the information is encoded synergistically between all of them. We show how this complexity is directly computable for smaller layers. For larger layers, we propose subsampling and coarse-graining procedures and prove corresponding bounds on the latter. Empirically, for quantized deep neural networks solving the MNIST and CIFAR10 image classification tasks, we observe that representational complexity decreases both through successive hidden layers and over training, and compare the results to related measures. Overall, we propose representational complexity as a principled and interpretable summary statistic for analyzing the structure and evolution of neural representations and complex systems in general.

I will present our research about developing and using multivariate information theoretic quantities to study how the interactions between neurons in the brain generate information processing functions.

The PID framework is based upon three axioms regarding the behaviour of a measure of redundant or intersection information. From these axioms, Williams and Beer derived a general structure for shared multivariate information called the redundant information lattice. Of course, these axioms do not uniquely determine a particular measure of redundant information, which has led to much debated about the exact measure of redundant information that should be used. In the first half of this talk, I will discuss some important details regarding these axioms which have important consequences for the functional form of redundant information. In the second half of the talk, I will discuss a modification of the Williams and Beer axioms that instead focus on a measure of the total marginal or union information. From these axioms I derive a second algebraic structure that can also be used to decompose multivariate information, which I call the total information lattice. I will explore the implications of demanding a consistency between these two decompositions, and in particular how requiring this consistency constrains the possible measures of redundant information, or equivalently, possible measures of the total marginal information. I will close by discussing how the total information lattice better aligns with the notions of higher-order interactions in network theory compared to the redundancy lattice.

Information theory provides a fundamental framework for the quantification of information flows through channels, formally Markov kernels. However, quantities such as mutual information and conditional mutual information do not necessarily reflect the causal nature of such flows.We argue that this is often the result of conditioning based on sigma algebras that are not associated with the given channels.We propose a version of the (conditional) mutual information based on families of sigma algebras that are coupled with the underlying channel. This leads to filtrations which allow us to prove a corresponding causal chain rule as a basic requirement within the presented approach.

In this presentation we review the approach to definition and quantification of the partition of mutual information among a set of variables into a sequence of the so-called connected informations (Schneidman, 2003) of increasing order k, capturing the dependence enforced by the constraints on the k-variate marginal distributions. In particular, we discuss the relation between the theoretically appealing (but practically prohibitively problematic to estimate) connected information constrained by the full marginal distributions, and some potentially more tractable approximations thereof. These include constraining only the k-th order moments, or constraining the k-th order entropies (Martin et al., 2016, 2017), or several heuristic approximations, that provide allow fast estimation from data, while potentially bounding the vanilla connected information (Studenicova, 2021). We shall discuss both theoretical relations between the introduced quantities, as well as illustrative synthetic and brain-activity data examples. Martin, E. A., Hlinka, J., & Davidsen, J. (2016). Pairwise Network Information and Nonlinear Correlations. Physical Review E, 94(4). https://doi.org/10.1103/PhysRevE.94.040301 Martin, E. A., Hlinka, J., Meinke, A., Děchtěrenko, F., Tintěra, J., Oliver, I., & Davidsen, J. (2017). Network Inference and Maximum Entropy Estimation on Information Diagrams. Scientific Reports, 7(1). https://doi.org/10.1038/s41598-017-06208-w Schneidman, E., Still, S., Berry, M. J., Bialek, W., & others. (2003). Network information and connected correlations. Phys. Rev. Lett., 91(23), 238701. https://doi.org/10.1103/PhysRevLett.91.238701 Studeničová, K. (2021). Properties of connected information variants, Bachelor Thesis, Czech Technical University, https://dspace.cvut.cz/handle/10467/97056

The detection of causality is a crucial point in the description of complex systems across many scientific disciplines. In practice, we try to reconstruct an oriented graph capturing the causal relationships in the system using methods like Granger causality or transfer entropy. However, it turns out that not all types of dependencies can be captured by a graph. A textbook example is the system where binary variable Z is given by XOR function of independent binary variables X and Y. This model of so-called synergy or second order dependency (which can be easily generalized to k-th order dependency) cannot be satisfactorily described by a graph but by a hypergraph in which there is an oriented hyperedge from the tuple (X,Y) to Z. In our work, we present a generalization of the information-theoretic concept of the definition of direct causal effect from a set of elements (variables) to an element. Next, we discuss the issue of so-called forced causal interactions and the possibility of solving this problem using partial information decomposition.

In this talk, we present a novel duality for scalar Gaussian multiple access channels and broadcast channels. The duality we explore is based on shared partial information quantities (e.g. synergy and redundancy). Using lattice theory, we establish a crossover correspondence of the synergistic and redundant components between these two channels. The dual channels are similar to the traditional pairs based on capacity regions, though the pairs we identify have equal transmission powers instead of a sum constraint relating transmission powers.

I will present two perspectives around networks and higher order informational multiplets. In the first one I will show how pairwise transfer entropy and synergy are modulated around the critical transition in a system of Ising spins living on networks of different topologies. In the second one I will present a way to reconcile higher order representations with the parsimonious representation of lower-order interactions as networks, focusing not on the informational content of multiplets of variables, but on the variation of information in the system when single variables, or pairs, or triplets, are added to the system to form these multiplets.

Using PIDs to map the flow of task-relevant information from human brain measurements in a perceptual decision-making task Hamed Nili, Marco Celotto, Alessandro Toso, Tobias Donner, and Stefano Panzeri How can we reverse engineer the information flow within the brain that leads humans to take appropriate decisions based on the available sensory evidence? Recordings of brain activity of subjects performing perceptual discrimination tasks have begun to reveal qualitatively the distribution across brain areas and frequency bands of neural signals encoding the sensory stimulus or the subject’s decision (Wilming et al., Nat. Commun. 2020), as well as mapping the transmission of activity between selected pairs of visual regions (Bosman et al., Neuron 2012; Ferro et al., PNAS 2021). Despite this progress, we still lack a quantitative framework to map the information processing underlying perceptual decision making. In this work, we develop this framework using Partial Information Decompositions (PID). We also demonstrate its power to reveal such computations in large brain networks recorded from the human brain. Specifically, our objective is to perform a complete identification of when and how sensory information is encoded in the cortical hierarchy and in the frequency activity space, how it is transmitted to other brain areas, and how is used to guide behavioral choice. We analysed published whole-brain MEG recordings (Wilming et al., Nat. Commun. 2020) from human participants performing a visual contrast discrimination task entailing a sequence of 10 discrete sample stimuli whose contrast had to be averaged and compared to a reference. We used mutual information to quantify stimulus and choice information in brain activity across time, frequency, and 180 regions per hemisphere covering the entire cortex. We used Intersection Information (II, Pica et al., NIPS 2017), which is based on PID, to quantify how much of the sensory information encoded in neural activity is used to inform choices. Additionally, using the same framework, we quantified the extent to which stimulus or choice information are redundant/synergistic in the homotopic ROIs. Moreover, we measured the amount of information about stimulus or choice transmitted between cortical areas using our newly developed Feature-specific Information Transfer (FIT) measure (Bim et al., BioRxiv, 2020) We identified frequency bands that carried stimulus, choice, and intersection information across cortex. Stimulus information used to inform choices was initially carried in the gamma-band, [40, 80) Hz, in visual, parietal and posterior cingulate regions. Choice signals later developed in the alpha-/beta-bands, [8, 40) Hz, in downstream regions of premotor and somato-motor cortex. The region-specific time-frequency patterns of information had distinguishable profiles for different anatomical groups and among the different measures, intersection information provided the highest level of discriminability. We also found that the pattern of redundancy/synergy of information in homotopic ROIs (i.e., across the left and right hemispheres) was different across regions, frequency bands and was content-dependent. Stimulus information was encoded redundantly between left and right homotopic ROIs in visual areas and gamma frequency band. Choice information was encoded synergistically across the two hemispheres in downstream areas and in lower frequencies. In some areas, like the Parietal cortex, both patterns were present. Investigating information transfer across regions, we found that stimulus-specific and choice-specific information were broadly transmitted across many areas. The transmission of stimulus information occurred predominantly in the feedforward direction in the gamma-band, and transmission of choice information predominantly in the feedback direction in the alpha-/beta-bands. In sum, using information theory, we mapped the encoding and transmission of task-relevant variables across frequency and cortical locations in the human brain during a perceptual discrimination task. This revealed a transformation over time of stimulus information in the gamma band in visual, parietal and cingulate areas into choice signals in the beta and alpha band in downstream areas, mediated by distributed patterns of feedforward gamma stimulus information and feedback alpha-beta choice information transmission. References Wilming, N., Murphy, P. R., Meyniel, F., & Donner, T. H. (2020). Large-scale dynamics of perceptual decision information across human cortex. Nature communications, 11(1), 5109. Bosman, C. A., Schoffelen, J. M., Brunet, N., Oostenveld, R., Bastos, A. M., Womelsdorf, T., ... & Fries, P. (2012). Attentional stimulus selection through selective synchronization between monkey visual areas. Neuron, 75(5), 875-888. Ferro, D., van Kempen, J., Boyd, M., Panzeri, S., & Thiele, A. (2021). Directed information exchange between cortical layers in macaque V1 and V4 and its modulation by selective attention. Proceedings of the National Academy of Sciences, 118(12), e2022097118. Pica, G., Piasini, E., Safaai, H., Runyan, C., Harvey, C., Diamond, M., ... & Panzeri, S. (2017). Quantifying how much sensory information in a neural code is relevant for behavior. Adv. Neural Inf. Process. Syst. (NeurIPS) 30: 3686–3696. Bím, J., De Feo, V., Chicharro, D., Hanganu-Opatz, I. L., Brovelli, A., & Panzeri, S. (2020) A Non-negative Measure Of Feature-specific Information Transfer Between Neural Signals. BioRxiv

The varied cognitive abilities and rich adaptive behaviors enabled by the animal nervous system are often described in terms of information processing. This framing raises the issue of how biological neural circuits actually process information, and some of the most fundamental outstanding questions in neuroscience center on understanding the mechanisms of neural information processing. Classical information theory has long been understood to be a natural framework within which information processing can be understood, and recent advances in the field of multivariate information decomposition offer new insights into the structure of computation in complex systems. In this presentation, I discuss recent empirical work we have done in this vein. I will focus particularly on the synergistic mode, which quantifies the “higher-order” information carried in the patterns of multiple inputs and is not reducible to input from any single source. Recent work in a variety of model systems has revealed that synergistic dynamics are ubiquitous in neural circuitry and show reliable structure–function relationships, emerging disproportionately in neuronal rich clubs, downstream of recurrent connectivity, and in the convergence of correlated activity. I will briefly touch on new directions we are moving in applying related methods for advancing the understanding and treatment of disorders of mental health.

One of the main drivers behind the initial research efforts related to partial information decomposition (PID) was the desire to find appropriate formulas to uniquely characterise the various modes of information. In this talk I'll present an alternative, complementary view towards PID in which the task of "solving PID" is not a priority, leaving the center stage to other research endeavours. For this purpose, I'll explore three different avenues for employing PID without solving it, and discuss examples of them from recent research.

The framework of information dynamics offers a set of tools, grounded within the solid basis of information theory, to quantify several aspects of the dynamic interactions observed at the nodes of complex network systems. These tools, including measures of information storage, Granger causality and higher-order interactions, are extensively used to assess regulatory mechanisms, coupled activity and emergent behaviors in physiological networks studied from multiple simultaneously measured biosignals. Nevertheless, the framework of information dynamics has been developed mostly to assess interactions in the time domain, and this constitutes a limitation for the analysis of physiological systems rich of oscillatory content. For instance, the standard formulation of information dynamics is inappropriate to study the frequency-specific interactions occurring among the physiological rhythms of different organ systems. This contribution illustrates some developments of the framework of information dynamics whereby we move from the static analysis of pairwise interactions between coupled variables to the dynamic analysis of higher-order interactions involving more than two processes, and from the time-domain to the frequency-domain analysis of physiological time series. Our developments exploit the spectral representation of linear vector autoregressive models to quantify pairwise and higher-order interactions for multivariate rhythmic processes interacting in distinct frequency bands. The new framework will be first formulated theoretically and implemented through data-efficient estimators. Then, its performance and peculiarities will be illustrated on simulations of stochastic oscillatory processes. Finally, applications to physiological networks explored at different levels of vertical and horizontal integration will be reported, including human cardiovascular, cardiorespiratory and cerebrovascular oscillations studied in different physio-pathological states, electrocorticographic signals acquired in an animal experiment during anesthesia, and multichannel EEG recordings acquired during a motor execution task.

Brain interdependencies are usually studied using Pearson's correlation, neglecting statistical causality, especially if the variables carry redundant and synergistic information about their future. Here, we quantify the redundant and synergistic interactions using Integrated Information Decomposition [1, 2] in functional MRI data in macaques before and after focused ultrasound stimulation (FUS), targeting different regions. Our preliminary results show disruptions in the synergy and redundancy after FUS while presenting heterogeneities across brain areas and macaques. References [1] Pedro A.M. Mediano, Fernando Rosas, Robin L. Carhart-Harris, Anil K. Seth, Adam B. Barrett Towards an extended taxonomy of information dynamics via integrated information decomposition. Preprint at https://arxiv.org/abs/2109.13186 (2021). [2] Luppi, A.I., Mediano, P.A.M., Rosas, F.E. et al. A synergistic core for human brain evolution and cognition. Nat Neurosci 25, 771–782 (2022). https://doi.org/10.1038/s41593-022-01070-0

Bivariate partial information decompositions (PIDs) characterize how the information in a "message" random variable is decomposed between two "constituent" random variables in terms of unique, redundant and synergistic information components. These components are a function of the joint distribution of the three variables, and are typically defined using an optimization over the space of all possible joint distributions. This makes it computationally challenging to compute PIDs in practice and restricts their use to low-dimensional random vectors. To ease this burden, we consider the case of jointly Gaussian random vectors. This case was previously examined by Barrett (2015), who showed that certain operationally well-motivated PIDs reduce to a closed form expression for scalar messages. We show that Barrett's result does not extend to vector messages in general, and characterize the set of multivariate Gaussian distributions that reduce to closed-form. Then, for all other multivariate Gaussian distributions, we propose a convex optimization framework for approximately computing the PID definitions of Bertschinger et al. (2014) and Banerjee et al. (2018). Using simplifying assumptions specific to the Gaussian case, we provide an efficient algorithm to approximately compute the bivariate PID for multivariate Gaussian variables with tens or even hundreds of dimensions. We also theoretically and empirically justify the goodness of this approximation.

Various requirements have been proposed that an information decomposition should satisfy. Curiously, however, only two of the original requirements that determined the Shannon information have been considered, namely monotonicity and normalization. Two other important properties, continuity and additivity, have not been considered. In this contribution, we focus on the "bivariate" case (i.e. on the decomposition of the mutual information of two finite variables $Y,Z$ about a third finite variable $S$) and check which of the decompositions satisfy these two properties. While most of them satisfy continuity, only one of them is both continuous and additive.

While the existence of a measure-theoretic (i.e. continuous and discrete-continuously mixed) partial information decomposition has been established recently, the work of Schick-Poland et. al was not concluded with an explicit form of the resulting generalized measure. In this talk, I will not only elaborate on the properties of the measure-theoretic measurfe of redundant information, but will also demonstrate an approximation of the measure-theoretic formulation and how to use it in the context of estimation of partial information quantities from real-world data.

We introduce the Redundant Information Neural Estimator (RINE), a method that allows efficient estimation for the component of information about a target variable that is common to a set of sources, known as the “redundant information”. We show that existing definitions of the redundant information can be recast in terms of an optimization over a family of functions. In contrast to previous information decompositions, which can only be evaluated for discrete variables over small alphabets, we show that optimizing over functions enables the approximation of the redundant information for high-dimensional and continuous predictors. We demonstrate this on high-dimensional image classification and motor-neuroscience tasks.

Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation. We introduce a new mathematical foundation for diffusion models inspired by classic results in information theory that connect Information with Minimum Mean Square Error regression, the so-called I-MMSE relations. We generalize the I-MMSE relations to exactly relate the data distribution to an optimal denoising regression problem. This connection has a number of interesting implications, including a novel non-negative decomposition of information. We will show preliminary results on decomposing which pixels in an image contain information about a text description.