Jinmeng Rao

2023

1. 

   ACM SIGSPATIAL

   Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models

   Jinmeng Rao, Song Gao, Gengchen Mai, and Krzysztof Janowicz

   In Proceedings of the 31st International Conference on Advances in Geographic Information Systems, 2023

   Highlight: Top Conference in Geographic Information Systems; Acceptance rate 37.2%

   Abs arXiv Bib

   In recent years we have seen substantial advances in foundation models for artificial intelligence, including language, vision, and multimodal models. Recent studies have highlighted the potential of using foundation models in geospatial artificial intelligence, known as GeoAI Foundation Models, for geographic question answering, remote sensing image understanding, map generation, and location-based services, among others. However, the development and application of GeoAI foundation models can pose serious privacy and security risks, which have not been fully discussed or addressed to date. This paper introduces the potential privacy and security risks throughout the lifecycle of GeoAI foundation models and proposes a comprehensive blueprint for research directions and preventative and control strategies. Through this vision paper, we hope to draw the attention of researchers and policymakers in geospatial domains to these privacy and security risks inherent in GeoAI foundation models and advocate for the development of privacy-preserving and secure GeoAI foundation models.

   @inproceedings{rao2023psgeoai,
     title = {Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models},
     author = {Rao, Jinmeng and Gao, Song and Mai, Gengchen and Janowicz, Krzysztof},
     booktitle = {Proceedings of the 31st International Conference on Advances in Geographic Information Systems},
     pages = {1--4},
     year = {2023},
     note = {Highlight: Top Conference in Geographic Information Systems; Acceptance rate 37.2\%},
   }

2. 

   IJGIS

   CATS: Conditional Adversarial Trajectory Synthesis for Privacy-Preserving Trajectory Data Publication Using Deep Learning Approaches

   Jinmeng Rao, Song Gao, and Sijia Zhu

   International Journal of Geographical Information Science, 2023

   Highlight: Top Journal in Geographic Information Science; SCI Q1 in CS, IS and Geography; Acceptance rate 12%

   Abs arXiv Bib Code

   The prevalence of ubiquitous location-aware devices and mobile Internet enables us to collect massive individual-level trajectory dataset from users. Such trajectory big data bring new opportunities to human mobility research but also raise public concerns with regard to location privacy. In this work, we present the Conditional Adversarial Trajectory Synthesis (CATS), a deep-learning-based GeoAI methodological framework for privacy-preserving trajectory data generation and publication. CATS applies K-anonymity to the underlying spatiotemporal distributions of human movements, which provides a distributional-level strong privacy guarantee. By leveraging conditional adversarial training on K-anonymized human mobility matrices, trajectory global context learning using the attention-based mechanism, and recurrent bipartite graph matching of adjacent trajectory points, CATS is able to reconstruct trajectory topology from conditionally sampled locations and generate high-quality individual-level synthetic trajectory data, which can serve as supplements or alternatives to raw data for privacy-preserving trajectory data publication. The experiment results on over 90k GPS trajectories show that our method has a better performance in privacy preservation, spatiotemporal characteristic preservation, and downstream utility compared with baseline methods, which brings new insights into privacy-preserving human mobility research using generative AI techniques and explores data ethics issues in GIScience.

   @article{rao2023cats,
     title = {CATS: Conditional Adversarial Trajectory Synthesis for Privacy-Preserving Trajectory Data Publication Using Deep Learning Approaches},
     author = {Rao, Jinmeng and Gao, Song and Zhu, Sijia},
     journal = {International Journal of Geographical Information Science},
     year = {2023},
     volume = {37},
     number = {12},
     pages = {1-33},
     note = {Highlight: Top Journal in Geographic Information Science; SCI Q1 in CS, IS and Geography; Acceptance rate 12\%},
   }

3. 

   Google Research Conference

   Evaluation and Mitigation of Agnosia in Multimodal Large Language Models

   Jinmeng Rao*, Jiaying Lu*, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang

   arXiv preprint arXiv:2309.04041, 2023

   Highlight: Trustworthy Multimodal Foundation Models; Multimodal Instruction Tuning; The poster version to be presented at Google Research Conference 2023

   Abs arXiv Bib

   While Multimodal Large Language Models (MLLMs) are widely used for a variety of vision-language tasks, one observation is that they sometimes misinterpret visual inputs or fail to follow textual instructions even in straightforward cases, leading to irrelevant responses, mistakes, and ungrounded claims. This observation is analogous to a phenomenon in neuropsychology known as Agnosia, an inability to correctly process sensory modalities and recognize things (e.g., objects, colors, relations). In our study, we adapt this similar concept to define "agnosia in MLLMs", and our goal is to comprehensively evaluate and mitigate such agnosia in MLLMs. Inspired by the diagnosis and treatment process in neuropsychology, we propose a novel framework EMMA (Evaluation and Mitigation of Multimodal Agnosia). In EMMA, we develop an evaluation module that automatically creates fine-grained and diverse visual question answering examples to assess the extent of agnosia in MLLMs comprehensively. We also develop a mitigation module to reduce agnosia in MLLMs through multimodal instruction tuning on fine-grained conversations. To verify the effectiveness of our framework, we evaluate and analyze agnosia in seven state-of-the-art MLLMs using 9K test samples. The results reveal that most of them exhibit agnosia across various aspects and degrees. We further develop a fine-grained instruction set and tune MLLMs to mitigate agnosia, which led to notable improvement in accuracy.

   @article{lu2023evaluation,
     title = {Evaluation and Mitigation of Agnosia in Multimodal Large Language Models},
     author = {Rao*, Jinmeng and Lu*, Jiaying and Chen, Kezhen and Guo, Xiaoyuan and Zhang, Yawen and Sun, Baochen and Yang, Carl and Yang, Jie},
     journal = {arXiv preprint arXiv:2309.04041},
     year = {2023},
     note = {Highlight: Trustworthy Multimodal Foundation Models; Multimodal Instruction Tuning; The poster version to be presented at Google Research Conference 2023},
   }

4. 

   NeurIPS SoLaR

   Tackling Vision Language Tasks Through Learning Inner Monologues

   Diji Yang, Kezhen Chen, Jinmeng Rao, Xiaoyuan Guo, Yawen Zhang, Jie Yang, and Yi Zhang

   arXiv preprint arXiv:2308.09970, 2023

   Highlight: Multi-Agent Collaboration. To be presented at NeurIPS SoLaR workshop. The poster version was presented at BayLearn 2023

   Abs arXiv Bib

   Visual language tasks require AI models to comprehend and reason with both visual and textual content. Driven by the power of Large Language Models (LLMs), two prominent methods have emerged: (1) the hybrid integration between LLMs and Vision-Language Models (VLMs), where visual inputs are firstly converted into language descriptions by VLMs, serving as inputs for LLMs to generate final answer(s); (2) visual feature alignment in language space, where visual inputs are encoded as embeddings and projected to LLMs’ language space via further supervised fine-tuning. The first approach provides light training costs and interpretability but is hard to be optimized in an end-to-end fashion. The second approach presents decent performance, but feature alignment usually requires large amounts of training data and lacks interpretability. To tackle this dilemma, we propose a novel approach, Inner Monologue Multi-Modal Optimization (IMMO), to solve complex vision language problems by simulating inner monologue processes, a cognitive process in which an individual engages in silent verbal communication with themselves. We enable LLMs and VLMs to interact through natural language conversation and propose to use a two-stage training process to learn how to do the inner monologue (self-asking questions and answering questions). IMMO is evaluated on two popular tasks and the results suggest by emulating the cognitive phenomenon of internal dialogue, our approach can enhance reasoning and explanation abilities, contributing to the more effective fusion of vision and language models. More importantly, instead of using predefined human-crafted monologues, IMMO learns this process within the deep learning models, promising wider applicability to many different AI problems beyond vision language tasks.

   @article{yang2023tackling,
     title = {Tackling Vision Language Tasks Through Learning Inner Monologues},
     author = {Yang, Diji and Chen, Kezhen and Rao, Jinmeng and Guo, Xiaoyuan and Zhang, Yawen and Yang, Jie and Zhang, Yi},
     journal = {arXiv preprint arXiv:2308.09970},
     year = {2023},
     note = {Highlight: Multi-Agent Collaboration. To be presented at NeurIPS SoLaR workshop. The poster version was presented at BayLearn 2023},
   }

2022

1. 

   ACM SIGSPATIAL GEOKG

   Measuring network resilience via geospatial knowledge graph: a case study of the us multi-commodity flow network

   Jinmeng Rao, Song Gao, Michelle Miller, and Alfonso Morales

   In Proceedings of the 1st ACM SIGSPATIAL International Workshop on Geospatial Knowledge Graphs, 2022

   Highlight: Food System; Network Resilience; Geospatial Knowledge Graphs

   Abs Bib HTML Code

   Quantifying the resilience in the food system is important for food security issues. In this work, we present a geospatial knowledge graph (GeoKG)-based method for measuring the resilience of a multi-commodity flow network. Specifically, we develop a CFS-GeoKG ontology to describe geospatial semantics of a multi-commodity flow network comprehensively, and design resilience metrics that measure the node-level and network-level dependence of single-sourcing, distant, or non-adjacent suppliers/customers in food supply chains. We conduct a case study of the US state-level agricultural multi-commodity flow network with hierarchical commodity types. The results indicate that, by leveraging GeoKG, our method supports measuring both node-level and network-level resilience across space and over time and also helps discover concentration patterns of agricultural resources in the spatial network at different geographic scales.

   @inproceedings{rao2022measuring,
     title = {Measuring network resilience via geospatial knowledge graph: a case study of the us multi-commodity flow network},
     author = {Rao, Jinmeng and Gao, Song and Miller, Michelle and Morales, Alfonso},
     booktitle = {Proceedings of the 1st ACM SIGSPATIAL International Workshop on Geospatial Knowledge Graphs},
     pages = {17--25},
     year = {2022},
     note = {Highlight: Food System; Network Resilience; Geospatial Knowledge Graphs},
   }

2021

1. 

   IEEE IGARSS

   Spatiotemporal contrastive representation learning for building damage classification

   Bo Peng, Qunying Huang, and Jinmeng Rao

   In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, 2021

   Highlight: Spatiotemporal Contrastive Representation Learning; Acceptance rate 34.2%

   Abs Bib HTML

   Automatic building damage assessment after natural disasters is important for emergency response. While existing supervised deep learning models achieved good performance on building damage classification, these models require massive human labels for training. Additionally, pre-trained models often fail to generalize well to new disaster events due to gaps between domains associated with training and testing data. In response, this study proposes a novel spatiotemporal contrastive representation learning model for learning features of building damages with big unlabeled data. Experimental results demonstrate superior performance of such features on classifying building damages resulting from various natural disasters (e.g., hurricanes, floods, wildfires, earthquakes, etc.) across different geographic locations worldwide, compared with the state-of-the-art supervised methods.

   @inproceedings{peng2021spatiotemporal,
     title = {Spatiotemporal contrastive representation learning for building damage classification},
     author = {Peng, Bo and Huang, Qunying and Rao, Jinmeng},
     booktitle = {2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS},
     pages = {8562--8565},
     year = {2021},
     organization = {IEEE},
     note = {Highlight: Spatiotemporal Contrastive Representation Learning; Acceptance rate 34.2\%},
   }

2. 

   ACM SIGSPATIAL GEOAI

   VTSV: A privacy-preserving vehicle trajectory simulation and visualization platform using deep reinforcement learning

   Jinmeng Rao, Song Gao, and Xiaojin Zhu

   In Proceedings of the 4th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery, 2021

   Highlight: Deep Reinforcement Learning; Personalized Driving Simulation [ Video Demo ]

   Abs Bib HTML

   Trajectory data is among the most sensitive data and the society increasingly raises privacy concerns. In this demo paper, we present a privacy-preserving Vehicle Trajectory Simulation and Visualization (VTSV) web platform (demo video: https://youtu.be/NY5L4bu2kTU), which automatically generates navigation routes between given pairs of origins and destinations and employs a deep reinforcement learning model to simulate vehicle trajectories with customized driving behaviors such as normal driving, overspeed, aggressive acceleration, and aggressive turning. The simulated vehicle trajectory data contain high-sample-rate of attributes including GPS location, speed, acceleration, and steering angle, and such data are visualized in VTSV using streetscape.gl, an autonomous driving data visualization framework. Location privacy protection methods such as origin-destination geomasking and trajectory k-anonymity are integrated into the platform to support privacy-preserving trajectory data generation and publication. We design two application scenarios to demonstrate how VTSV performs location privacy protection and customize driving behavior, respectively. The demonstration shows that VTSV is able to mitigate data privacy, sparsity, and imbalance sampling issues, which offers new insights into driving trajectory simulation and GeoAI-powered privacy-preserving data publication.

   @inproceedings{rao2021vtsv,
     title = {VTSV: A privacy-preserving vehicle trajectory simulation and visualization platform using deep reinforcement learning},
     author = {Rao, Jinmeng and Gao, Song and Zhu, Xiaojin},
     booktitle = {Proceedings of the 4th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery},
     pages = {43--46},
     year = {2021},
     note = {Highlight: Deep Reinforcement Learning; Personalized Driving Simulation <a href="https://www.youtube.com/watch?v=NY5L4bu2kTU">[ Video Demo ]</a>},
   }

3. 

   TGIS

   A privacy-preserving framework for location recommendation using decentralized collaborative machine learning

   Jinmeng Rao, Song Gao, Mingxiao Li, and Qunying Huang

   Transactions in GIS, 2021

   Highlight: Flagship Journal in Geographic Information Systems; Recognized as the first work on Federated Learning + Spatial Data Sciences (Graser, A. et al 2022)

   Abs Bib HTML Code

   The nowadays ubiquitous location-aware mobile devices have contributed to the rapid growth of individual-level location data. Such data are usually collected by location-based service platforms as training data to improve their predictive models’ performance, but the collection of such data may raise public concerns about privacy issues. In this study, we introduce a privacy-preserving location recommendation framework based on a decentralized collaborative machine learning approach: federated learning. Compared with traditional centralized learning frameworks, we keep users’ data on their own devices and train the model locally so that their data remain private. The local model parameters are aggregated and updated through secure multiple-party computation to achieve collaborative learning among users while preserving privacy. Our framework also integrates information about transportation infrastructure, place safety, and flow-based spatial interaction to further improve recommendation accuracy. We further design two attack cases to examine the privacy protection effectiveness and robustness of the framework. The results show that our framework achieves a better balance on the privacy–utility trade-off compared with traditional centralized learning methods. The results and ensuing discussion offer new insights into privacy-preserving geospatial artificial intelligence and promote geoprivacy in location-based services.

   @article{rao2021privacy,
     title = {A privacy-preserving framework for location recommendation using decentralized collaborative machine learning},
     author = {Rao, Jinmeng and Gao, Song and Li, Mingxiao and Huang, Qunying},
     journal = {Transactions in GIS},
     year = {2021},
     note = {Highlight: Flagship Journal in Geographic Information Systems; Recognized as the first work on Federated Learning + Spatial Data Sciences <a href="https://publications.ait.ac.at/en/publications/on-the-role-of-spatial-data-science-for-federated-learning">(Graser, A. et al 2022)</a>},
   }

2020

1. 

   GISCIENCE

   LSTM-TrajGAN: A deep learning approach to trajectory privacy protection

   Jinmeng Rao, Song Gao, Yuhao Kang, and Qunying Huang

   In 11th International Conference on Geographic Information Science (GIScience 2021) - Part I, 2020

   Highlight: Top Conference in Geographic Information Sciences; Acceptance rate 34%

   Abs Bib HTML Code

   The prevalence of location-based services contributes to the explosive growth of individual-level location trajectory data and raises public concerns about privacy issues. In this research, we propose a novel LSTM-TrajGAN approach, which is an end-to-end deep learning model to generate privacy-preserving synthetic trajectory data for data sharing and publication. We design a loss metric function TrajLoss to measure the trajectory similarity losses for model training and optimization. The model is evaluated on the trajectory-user-linking task on a real-world semantic trajectory dataset. Compared with other common geomasking methods, our model can better prevent users from being re-identified, and it also preserves essential spatial, temporal, and thematic characteristics of the real trajectory data. The model better balances the effectiveness of trajectory privacy protection and the utility for spatial and temporal analyses, which offers new insights into the GeoAI-powered privacy protection for human mobility studies.

   @inproceedings{rao2020lstm,
     title = {LSTM-TrajGAN: A deep learning approach to trajectory privacy protection},
     author = {Rao, Jinmeng and Gao, Song and Kang, Yuhao and Huang, Qunying},
     booktitle = {11th International Conference on Geographic Information Science (GIScience 2021) - Part I},
     volume = {177},
     pages = {12--1},
     year = {2020},
     organization = {Schloss Dagstuhl--Leibniz-Zentrum f$\{$$\backslash$"u$\}$r Informatik},
     note = {Highlight: Top Conference in Geographic Information Sciences; Acceptance rate 34\%},
   }

2. 

   ACM SIGSPATIAL

   Mapping county-level mobility pattern changes in the United States in response to COVID-19

   Song Gao, Jinmeng Rao, Yuhao Kang, Yunlei Liang, and Jake Kruse

   SIGSpatial Special, 2020

   Highlight: The first work on social distancing mapping during COVID-19; Media Coverage NBC, FOX, ABC

   Abs Bib HTML Code Website

   To contain the COVID-19 epidemic, one of the non-pharmacological epidemic control measures is reducing the transmission rate of SARS-COV-2 in the population through social distancing. An interactive web-based mapping platform that provides timely quantitative information on how people in different counties and states reacted to the social distancing guidelines was developed by the GeoDS Lab @UW-Madison with the support of the National Science Foundation RAPID program. The web portal integrates geographic information systems (GIS) and daily updated human mobility statistical patterns (median travel distance and stay-at-home dwell time) derived from large-scale anonymized and aggregated smartphone location big data at the county-level in the United States, and aims to increase risk awareness of the public, support data-driven public health and governmental decision-making, and help enhance community responses to the COVID-19 pandemic.

   @article{gao2020mapping,
     title = {Mapping county-level mobility pattern changes in the United States in response to COVID-19},
     author = {Gao, Song and Rao, Jinmeng and Kang, Yuhao and Liang, Yunlei and Kruse, Jake},
     journal = {SIGSpatial Special},
     volume = {12},
     number = {1},
     pages = {16--26},
     year = {2020},
     publisher = {ACM New York, NY, USA},
     note = {Highlight: The first work on social distancing mapping during COVID-19; Media Coverage <a href="https://www.nbc15.com/2020/10/07/tracking-movement-in-wisconsin-leading-up-to-latest-health-order/">NBC</a>, <a href="http://fox47.com/news/local/uw-madison-researchers-map-travel-data-to-combat-pandemic">FOX</a>, <a href="https://www.wisn.com/article/uw-madison-scientists-develop-social-distancing-tracker-map/32163419#">ABC</a>},
   }