https://cesbar99.github.io/blog/callm-judge-arxiv/Tue, 13 Oct 2026 00:00:00 +0000https://cesbar99.github.io/blog/callm-judge-arxiv/<p>“ ” is now available on <strong>arXiv</strong>.</p> <h2 id="abstract">Abstract</h2> <p>Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgments can exhibit systematic biases. Prior work has documented position, verbosity or model family of the candidate answers influencing the judge decision rather than their quality, while additional biases may remain undocumented. Multicalibration provides a natural framework to address this problem: each bias corresponds to miscalibration on an identifiable group, e.g., family bias manifests as overconfidence in answers generated by models belonging to the same family of the judge. The challenge is to fix miscalibration simultaneously across many, potentially unknown groups. To this end, we introduce CaLLM, a post hoc framework that multicalibrates an LLM judge, thereby bounding all such biases simultaneously. We prove that CaLLM bounds the bias of every group whose error structure is captured by an available representation, and that augmenting it with explicit bias groups further reduces the residual bias. We then derive theoretical conditions under which CaLLM improves the judge&rsquo;s accuracy, rather than trading it for calibration. Finally, we evaluate CaLLM across pairwise preference benchmarks, showing that it reduces the multicalibration error on both specified and unspecified bias groups, while improving predictive performance of the judge.</p>https://cesbar99.github.io/publications/callm-judge/Tue, 13 Oct 2026 00:00:00 +0000https://cesbar99.github.io/publications/callm-judge/<h2 id="abstract">Abstract</h2> <p>Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgments can exhibit systematic biases. Prior work has documented position, verbosity or model family of the candidate answers influencing the judge decision rather than their quality, while additional biases may remain undocumented. Multicalibration provides a natural framework to address this problem: each bias corresponds to miscalibration on an identifiable group, e.g., family bias manifests as overconfidence in answers generated by models belonging to the same family of the judge. The challenge is to fix miscalibration simultaneously across many, potentially unknown groups. To this end, we introduce CaLLM, a post hoc framework that multicalibrates an LLM judge, thereby bounding all such biases simultaneously. We prove that CaLLM bounds the bias of every group whose error structure is captured by an available representation, and that augmenting it with explicit bias groups further reduces the residual bias. We then derive theoretical conditions under which CaLLM improves the judge&rsquo;s accuracy, rather than trading it for calibration. Finally, we evaluate CaLLM across pairwise preference benchmarks, showing that it reduces the multicalibration error on both specified and unspecified bias groups, while improving predictive performance of the judge.</p>https://cesbar99.github.io/blog/selective-disclosure-arxiv/Wed, 30 Sep 2026 00:00:00 +0000https://cesbar99.github.io/blog/selective-disclosure-arxiv/<p>“ ” is now available on <strong>arXiv</strong>.</p> <h2 id="abstract">Abstract</h2> <p>In many high-stakes settings, human decision-makers can acquire support information before making a decision. However, acquiring information is costly, and disclosure may fail to improve human decisions or may even impair them. We tackle this problem by studying selective disclosure, i.e., the problem of learning when to reveal support information to a human decision-maker under a budget constraint. We first show that the optimal policy is a threshold rule on the Value of Information (VoI), i.e., the expected reduction in human decision risk induced by disclosure. Since VoI is unknown in practice, we estimate the regime-specific human risks and bound the possible degradation of the resulting plug-in policy relative to lack of disclosure, as well as its regret relative to the optimal policy. Experiments on benchmark datasets show that selective disclosure outperforms both no disclosure and full disclosure, regardless of whether the support information is beneficial or harmful. Two user studies show that human-AI team performance can improve when disclosure is led by our learned policy and not human-selected, although this advantage varies across tasks. A counterfactual benchmark, which replaces participants&rsquo; predictions with a machine-learning prediction when disclosure occurs, suggests that these differences might depend on lower adherence to advice when the information is automatically provided rather than self-requested.</p>https://cesbar99.github.io/blog/neurips-2026-divide-accepted/Wed, 02 Sep 2026 00:00:00 +0000https://cesbar99.github.io/blog/neurips-2026-divide-accepted/<p>“Divide et Calibra: Multiclass Local Calibration via Vector Quantization” was accepted at <strong>NeurIPS 2026</strong>.</p>https://cesbar99.github.io/publications/divide-et-calibra/Tue, 01 Sep 2026 00:00:00 +0000https://cesbar99.github.io/publications/divide-et-calibra/<h2 id="abstract">Abstract</h2> <p>Accurate and well-calibrated Machine Learning (ML) models are mandatory in high-stakes settings, yet effective multiclass calibration remains challenging: global approaches assume calibration errors are homogeneous across the latent space, while local methods often rely on latent-space dimensionality reduction, which leads to information loss. To address these issues, we propose a compositional approach to multiclass calibration, where region-specific calibration maps are constructed from shared codeword-dependent factors. We instantiate this idea via Vector Quantization (VQ), which induces a structured partition of the representation space, and an indexed parameterization of Dirichlet concentrations that enables parameter sharing across regions. Our approach learns heterogeneous calibration maps that generalize well even to sparse regions of the latent space. Experiments on benchmark datasets show significant improvements in local calibration while maintaining competitive global calibration and predictive performance.</p>https://cesbar99.github.io/blog/hldm-2026-conformal-anchors/Tue, 01 Sep 2026 00:00:00 +0000https://cesbar99.github.io/blog/hldm-2026-conformal-anchors/<p>I presented “Elementary, My Dear Classifier: Interpretable Hypothesis Exclusion with Conformal Anchors” at the <strong>HLDM workshop at ECML PKDD 2026</strong>.</p>https://cesbar99.github.io/publications/zoom-in-disparities/Wed, 17 Jun 2026 00:00:00 +0000https://cesbar99.github.io/publications/zoom-in-disparities/<h2 id="abstract">Abstract</h2> <p>Equitable access to reliable health information is vital when integrating AI into healthcare. Yet, information quality varies across languages, raising concerns about the reliability and consistency of multilingual Large Language Models (LLMs). We systematically examine cross-lingual disparities in pre-training source and factuality alignment in LLM answers for multilingual healthcare Q&amp;A across English, German, Turkish, Chinese (Mandarin), and Italian. We (i) constructed Multilingual Wiki Health Care (MultiWikiHealthCare), a multilingual dataset from Wikipedia; (ii) analyzed cross-lingual healthcare coverage; (iii) assessed LLM response alignment with these references; and (iv) conducted a case study on factual alignment through the use of contextual information and Retrieval-Augmented Generation (RAG). Our findings reveal substantial cross-lingual disparities in both Wikipedia coverage and LLM factual alignment. Across LLMs, responses align more with English Wikipedia, even when the prompts are non-English. Providing contextual excerpts from non-English Wikipedia at inference time effectively shifts factual alignment toward culturally relevant knowledge. These results highlight practical pathways for building more equitable, multilingual AI systems for healthcare.</p>https://cesbar99.github.io/publications/multiclass-local-calibration-jsd/Sat, 02 May 2026 00:00:00 +0000https://cesbar99.github.io/publications/multiclass-local-calibration-jsd/<h2 id="abstract">Abstract</h2> <p>Developing trustworthy Machine Learning (ML) models requires their predicted probabilities to be well-calibrated, meaning they should reflect true-class frequencies. Among calibration notions in multiclass classification, strong calibration is the most stringent, as it requires all predicted probabilities to be simultaneously calibrated across all classes. However, existing approaches to multiclass calibration lack a notion of distance among inputs, which makes them vulnerable to proximity bias: predictions in sparse regions of the feature space are systematically miscalibrated. In this work, we address this main shortcoming by introducing a local perspective on multiclass calibration. First, we formally define multiclass local calibration and establish its relationship with strong calibration. Second, we theoretically analyze the pitfalls of existing evaluation metrics when applied to multiclass local calibration. Third, we propose a practical method to enhance local calibration in Neural Networks, which enforces alignment between predicted probabilities and local estimates of class frequencies using the Jensen-Shannon distance. Finally, we empirically validate our approach against existing multiclass calibration techniques.</p>https://cesbar99.github.io/blog/aistats-2026-posters/Fri, 01 May 2026 00:00:00 +0000https://cesbar99.github.io/blog/aistats-2026-posters/<p>I attended <strong>AISTATS 2026</strong>, where I presented and discussed “Multiclass Local Calibration with the Jensen–Shannon Distance”, “On the Global and Local Calibration of Graph Neural Networks”, and a preliminary version of “Divide et Calibra: Multiclass Local Calibration via Vector Quantization” during the poster sessions.</p>https://cesbar99.github.io/blog/aistats-2026-divide-workshop/Sat, 03 Jan 2026 00:00:00 +0000https://cesbar99.github.io/blog/aistats-2026-divide-workshop/<p>A preliminary version of “Divide et Calibra: Multiclass Local Calibration via Vector Quantization” was accepted at <strong>Towards Trustworthy Predictions: Theory and Applications of Calibration for Modern AI</strong>, an AISTATS 2026 workshop.</p>https://cesbar99.github.io/blog/aistats-2026-gnn-workshop/Fri, 02 Jan 2026 00:00:00 +0000https://cesbar99.github.io/blog/aistats-2026-gnn-workshop/<p>“On the Global and Local Calibration of Graph Neural Networks” was accepted at <strong>Towards Trustworthy Predictions: Theory and Applications of Calibration for Modern AI</strong>, an AISTATS 2026 workshop.</p>https://cesbar99.github.io/blog/aistats-2026-jsd-accepted/Thu, 01 Jan 2026 00:00:00 +0000https://cesbar99.github.io/blog/aistats-2026-jsd-accepted/<p>“Multiclass Local Calibration with the Jensen–Shannon Distance” was accepted at <strong>AISTATS 2026</strong>.</p>https://cesbar99.github.io/blog/jsd-local-calibration-arxiv/Thu, 30 Oct 2025 00:00:00 +0000https://cesbar99.github.io/blog/jsd-local-calibration-arxiv/<p>“ ” is now available on <strong>arXiv</strong>.</p> <h2 id="abstract">Abstract</h2> <p>Developing trustworthy Machine Learning (ML) models requires their predicted probabilities to be well-calibrated, meaning they should reflect true-class frequencies. Among calibration notions in multiclass classification, strong calibration is the most stringent, as it requires all predicted probabilities to be simultaneously calibrated across all classes. However, existing approaches to multiclass calibration lack a notion of distance among inputs, which makes them vulnerable to proximity bias: predictions in sparse regions of the feature space are systematically miscalibrated. In this work, we address this main shortcoming by introducing a local perspective on multiclass calibration. First, we formally define multiclass local calibration and establish its relationship with strong calibration. Second, we theoretically analyze the pitfalls of existing evaluation metrics when applied to multiclass local calibration. Third, we propose a practical method to enhance local calibration in Neural Networks, which enforces alignment between predicted probabilities and local estimates of class frequencies using the Jensen-Shannon distance. Finally, we empirically validate our approach against existing multiclass calibration techniques.</p>https://cesbar99.github.io/blog/healthcare-llm-disparities-arxiv/Mon, 20 Oct 2025 00:00:00 +0000https://cesbar99.github.io/blog/healthcare-llm-disparities-arxiv/<p>“ ” is now available on <strong>arXiv</strong>.</p> <h2 id="abstract">Abstract</h2> <p>Equitable access to reliable health information is vital when integrating AI into healthcare. Yet, information quality varies across languages, raising concerns about the reliability and consistency of multilingual Large Language Models (LLMs). We systematically examine cross-lingual disparities in pre-training source and factuality alignment in LLM answers for multilingual healthcare Q&amp;A across English, German, Turkish, Chinese (Mandarin), and Italian. We (i) constructed Multilingual Wiki Health Care (MultiWikiHealthCare), a multilingual dataset from Wikipedia; (ii) analyzed cross-lingual healthcare coverage; (iii) assessed LLM response alignment with these references; and (iv) conducted a case study on factual alignment through the use of contextual information and Retrieval-Augmented Generation (RAG). Our findings reveal substantial cross-lingual disparities in both Wikipedia coverage and LLM factual alignment. Across LLMs, responses align more with English Wikipedia, even when the prompts are non-English. Providing contextual excerpts from non-English Wikipedia at inference time effectively shifts factual alignment toward culturally relevant knowledge. These results highlight practical pathways for building more equitable, multilingual AI systems for healthcare.</p>https://cesbar99.github.io/blog/to-ask-arxiv/Thu, 09 Oct 2025 00:00:00 +0000https://cesbar99.github.io/blog/to-ask-arxiv/<p>“ ” is now available on <strong>arXiv</strong>.</p> <h2 id="abstract">Abstract</h2> <p>Developing decision-support systems that complement human performance in classification tasks remains an open challenge. A popular approach, Learning to Defer (LtD), allows a Machine Learning (ML) model to pass difficult cases to a human expert. However, LtD treats humans and ML models as mutually exclusive decision-makers, restricting the expert contribution to mere predictions. To address this limitation, we propose Learning to Ask (LtA), a new framework that handles both when and how to incorporate expert input in an ML model. LtA is based on a two-part architecture: a standard ML model and an enriched model trained with additional expert human feedback, with a formally optimal strategy for selecting when to query the enriched model. We provide two practical implementations of LtA: a sequential approach, which trains the models in stages, and a joint approach, which optimises them simultaneously. For the latter, we design surrogate losses with realisable-consistency guarantees. Our experiments with synthetic and real expert data demonstrate that LtA provides a more flexible and powerful foundation for effective human–AI collaboration.</p>https://cesbar99.github.io/blog/hldm-2025-to-ask/Mon, 01 Sep 2025 00:00:00 +0000https://cesbar99.github.io/blog/hldm-2025-to-ask/<p>I presented “To Ask or Not to Ask: Learning to Require Human Feedback” at the <strong>HLDM workshop at ECML PKDD 2025</strong>.</p>https://cesbar99.github.io/publications/to-ask-or-not-to-ask/Mon, 01 Sep 2025 00:00:00 +0000https://cesbar99.github.io/publications/to-ask-or-not-to-ask/<h2 id="abstract">Abstract</h2> <p>Developing decision-support systems that complement human performance in classification tasks remains an open challenge. A popular approach, Learning to Defer (LtD), allows a Machine Learning (ML) model to pass difficult cases to a human expert. However, LtD treats humans and ML models as mutually exclusive decision-makers, restricting the expert contribution to mere predictions. To address this limitation, we propose Learning to Ask (LtA), a new framework that handles both when and how to incorporate expert input in an ML model. LtA is based on a two-part architecture: a standard ML model and an enriched model trained with additional expert human feedback, with a formally optimal strategy for selecting when to query the enriched model. We provide two practical implementations of LtA: a sequential approach, which trains the models in stages, and a joint approach, which optimises them simultaneously. For the latter, we design surrogate losses with realisable-consistency guarantees. Our experiments with synthetic and real expert data demonstrate that LtA provides a more flexible and powerful foundation for effective human–AI collaboration.</p>https://cesbar99.github.io/blog/hldm-2024-counterfactuals/Sun, 01 Sep 2024 00:00:00 +0000https://cesbar99.github.io/blog/hldm-2024-counterfactuals/<p>I presented “Interpretable and Efficient Counterfactual Generation with Disentangled Variational Autoencoders” at the <strong>HLDM workshop at ECML PKDD 2024</strong>.</p>https://cesbar99.github.io/experience/Tue, 24 Oct 2023 00:00:00 +0000https://cesbar99.github.io/experience/