0.33 pydataglobal2025 2025-12-09 2025-12-11 3 00:05 https://cfp.pydata.org UTC General Track Talk 2025-12-09T12:00:00+00:00 12:00 00:30 Python drives modern data workflows, yet Excel remains the lingua franca of business. Many Python-based data teams struggle when the “last mile” of delivery still involves exporting results to Excel for business users. This talk explores practical ways for Python users to automate, scale, and enhance Excel-heavy processes using open-source libraries. This talk will help you bridge the gap between code and the business-facing spreadsheet world. We will discuss real-world use cases for report generation, batch processing, and dashboard templating, all from a Python-first perspective. pydataglobal2025-78678-python-meets-excel-smarter-workflows-for-analysts-and-data-teams General Track /media/pydataglobal2025/submissions/HCURNN/pydata_P7HFqFA_hLWl0LJ.webp DR NISHA ARORA en This talk is designed for Python developers, analysts, and data scientists who routinely interact with Excel-based deliverables in their organization. It focuses on practical workflows that enhance productivity and reproducibility without requiring the audience to write or understand VBA or Excel formulas. The session begins by outlining common challenges Python users face when integrating with Excel, then introduces powerful Python tools that offer users seamless Excel file manipulation, specifically pandas, xlsxwriter, and xlwings. We will discuss some real-world use cases, such as generating reports, automating dashboards, creating custom functions in Excel and batch processing Excel files at scale. The talk concludes with a summary of tools, limitations, and best practices for integrating Python into Excel-centric workflows. This is a conceptual and strategic talk aimed at helping Python professionals work more effectively with Excel natives in the business ecosystem. false https://cfp.pydata.org/pydataglobal2025/talk/HCURNN/ https://cfp.pydata.org/pydataglobal2025/talk/HCURNN/feedback/ General Track Talk 2025-12-09T13:00:00+00:00 13:00 00:30 Contributing to open source isn’t just about code. Documentation, testing, community support, and issue triaging are critical but often overlooked. In this talk, I’ll share how Python developers — from junior to senior can make a meaningful, visible impact in open source. Whether you're new to open source or looking to expand your profile, this session will help you discover practical, beginner-friendly ways to contribute and stay engaged in the long term. pydataglobal2025-78662-python-beyond-the-code-unlocking-hidden-contributions-in-open-source General Track Iyanu Falaye en Open-source projects thrive on contributions, but those contributions don’t always come in the form of pull requests. In the Python community, roles such as documentation writing, bug reproduction, testing, onboarding, user feedback, and project coordination are vital to long-term sustainability. This talk aims to dispel the myth that only seasoned developers or prolific coders can contribute meaningfully to open-source projects. Through real-world examples and lessons from my own experience working with Python-based open-source communities, I’ll walk the audience through practical paths for getting involved — even if you're just starting or come from a non-traditional background like product, design, or DevRel. The session will outline the different ways contributions are recognized in the Python ecosystem, including the impact of GitHub discussions, contributing guides, documentation standards like reStructuredText or Markdown, and the importance of clear communication with maintainers. Expected audience: Python developers, career switchers, junior engineers, community managers, and anyone curious about participating in open source. Takeaway: You'll leave with an actionable roadmap to contribute beyond code and understand how to track and present your work to peers, employers, and the broader Python community. false https://cfp.pydata.org/pydataglobal2025/talk/GPFCXZ/ https://cfp.pydata.org/pydataglobal2025/talk/GPFCXZ/feedback/ General Track Talk 2025-12-09T13:30:00+00:00 13:30 00:30 The use of open-source models is rapidly increasing. According to Gartner, during the Magnetic Era, their adoption is expected to triple compared to foundational models. However, this rise in usage also brings heightened cybersecurity risks. In this lecture, we will explore the unique vulnerabilities associated with open-source models, the algorithmic techniques used to exploit them, and how our startup is addressing these challenges. pydataglobal2025-78576-open-source-models-security-adversarial-attacks-poisoning-sponge General Track /media/pydataglobal2025/submissions/QMUABM/FF_vcePwvh.png Natan Katz en In my lecture, I will discuss various methods for attacking machine learning models, including model poisoning, DDoS-style attacks, and the generation of adversarial examples—such as Projected Gradient Descent (PGD), Carlini-Wagner attacks, and others. We will also present defense strategies that are data-agnostic and focus on model-driven approaches to protecting AI systems, particularly those that use open-source models. We will also discuss the differentiation between protecting open-source models and regular LLM (what we are not OWASP LLM) false https://cfp.pydata.org/pydataglobal2025/talk/QMUABM/ https://cfp.pydata.org/pydataglobal2025/talk/QMUABM/feedback/ General Track Talk 2025-12-09T14:00:00+00:00 14:00 01:00 Keynote by Isabel Zimmerman pydataglobal2025-84929-opening-notes-keynote-by-isabel-zimmerman en Isabel is a Senior Software Engineer at Posit, PBC. false https://cfp.pydata.org/pydataglobal2025/talk/FHSZP7/ https://cfp.pydata.org/pydataglobal2025/talk/FHSZP7/feedback/ General Track Talk 2025-12-09T15:30:00+00:00 15:30 00:30 Data and Analytics Comedian Evan Wimpey is here to roast his own codebase! Enjoy the walk through of the worst Python habits. In this talk, you'll get to see: * Incomprehensible variable names * final_final_2.ipynb files * rerunning the same cell and hoping it works this time * imports that are never used * debugging with print * ML models that are validated on training data * code so poorly written that even ChatGPT can't understand it * and more! pydataglobal2025-78440-python-worst-practices-learn-from-the-expert General Track Evan Wimpey en This is meant to be comedy, but the best jokes always include a little bit of truth. Evan will share some hilarious stories of Python gone wrong, complete with code examples. That is, if the code even runs. false https://cfp.pydata.org/pydataglobal2025/talk/TTDNXY/ https://cfp.pydata.org/pydataglobal2025/talk/TTDNXY/feedback/ General Track Talk 2025-12-09T16:00:00+00:00 16:00 00:30 *Orkut* was once the emotional and cultural core of Brazil’s internet. Its scraps, testimonials, and communities gave users a way to publicly shape identity, build relationships, and engage with everything from music and religion to politics and humor. When Google shut it down in 2014, most of its data was deleted. What remains today is fragmented and buried in the Wayback Machine. In this talk, I use Python to recover and analyze limited traces of *Orkut*’s digital legacy. I scraped thousands of community names from archived HTML using `requests` and `BeautifulSoup`, processed them with multilingual sentence embeddings from `sentence-transformers`, and applied `scikit-learn` and `BERTopic` to cluster the data, surface major social themes, and quantify them. These techniques reveal how users created meaning, formed subcultures, and expressed identity through online interactions. Alongside the technical walkthrough, I draw on Cory Doctorow’s concept of *enshittification*, defined as the slow decline of platforms as they shift from serving users to exploiting them. *Orkut* is a case of *enshittification* by neglect: its shutdown led not just to the death of a platform, but to the erasure of a generation’s digital memory. According to Google's farewell announcement, over its 10 years of existence, *Orkut* hosted 51 million communities, 120 million discussion topics, and more than 1 billion interactions; most of which were permanently deleted. This talk is for Python users interested not only in working with social media text data but also in uncovering the cultural narratives embedded within it. It invites the audience to see datasets as more than technical artifacts, viewing them instead as living records of online social life. pydataglobal2025-78865-text-mining-orkut-s-community-data-with-python-cultural-memory-platform-neglect-and-digital-amnesia General Track Rodrigo Silva Ferreira en This talk explores how Python can be used to recover and analyze digital traces from a platform that once defined Brazil’s online culture. *Orkut*, active from 2004 to 2014, hosted millions of communities where users expressed identity, humor, politics, and emotion in public and often poetic ways. When the platform was shut down, nearly all of this user-generated data was deleted. Today, only fragmented pieces remain, preserved in the Wayback Machine. I present a data analysis project that extracts and categorizes *Orkut* community names using open-source Python tools. I use `requests` and `BeautifulSoup` to scrape data from archived HTML snapshots. I then apply multilingual sentence embeddings from the `sentence-transformers` library to generate vector representations of the text, followed by clustering techniques using `scikit-learn` and `BERTopic` to uncover and quantify recurring social themes. This technical walkthrough is grounded in a sociological lens. I draw on Cory Doctorow’s concept of *enshittification*, which describes how platforms degrade as they prioritize value extraction over user experience. *Orkut*'s case illustrates how platform neglect can result not only in product death but also in large-scale cultural erasure. By treating community names as social artifacts, I show how data science can help recover forgotten histories and highlight overlooked communities at the intersection of digital humanities, memorialization, and cultural heritage. Attendees will gain practical skills in web scraping, multilingual NLP, and unsupervised clustering. The talk also raises broader questions about data loss, platform decay, and the ethical role of data scientists, software engineers, and tech workers in preserving digital memory. No advanced data science, scraping, text mining, or NLP knowledge is required. The talk is best suited for data scientists and Python developers interested in working with real-world social data and approaching datasets with both technical rigor and cultural sensitivity. Regardless of background, this talk is accessible to anyone interested in data science, NLP, and text mining. **Time Breakdown (30 min)** | **Time** | **Section** | | --------- | ------------------------------------------------------------------------------------- | | 0–4 min | Introduction to *Orkut* and its cultural role in Brazil and in the Global South | | 4–7 min | Platform shutdown, data loss, digital memory and neglect | | 7–10 min | Project overview: goals, ethical framing, and data source (Wayback) | | 10–15 min | Scraping with `requests` and `BeautifulSoup` from archived HTML | | 15–20 min | Processing: multilingual embeddings with `sentence-transformers` | | 20–23 min | Clustering and theme discovery using `scikit-learn` and `BERTopic` | | 23–26 min | Insights: social themes, quantification, and what topic categories mattered to users | | 26–29 min | Reflection: *enshittification*, data loss, and cultural preservation | | 29–30 min | Final remarks and invitation to rethink data as memory + Q\&A | **Additional remarks:** 1) A GitHub repository containing the scraping scripts, archived HTML files, datasets, and analysis will be shared with attendees. 2) This project was inspired by both personal nostalgia and frustration over the loss of access to my *Orkut*'s profile, photos, testimonials, and communities. 3) Besides its overwhelming popularity in Brazil, *Orkut* also had a strong foothold in other countries across the Global South, such as India and China, reflecting its **broader appeal beyond the English-speaking tech centers**, typically prioritized in platform histories. This context would makes the talk proposal herein outlined even more interesting and compelling for a **PyData Global audience**. | Country | Traffic on Mar 31, 2004 | Traffic on Sep 30, 2014 | |----------------|--------------------------|--------------------------| | Brazil | 5.16% | 55.5% | | United States | 51.36% | 3.3% | | India | — | 18.4% | | China | — | 6.4% | | Japan | 7.74% | 2.7% | | Netherlands | 4.10% | — | | United Kingdom | 3.72% | — | | Other | 27.92% | 15.7% | Reference: https://web.archive.org/web/20140109153358/http://www.alexa.com/siteinfo/orkut.com.br false https://cfp.pydata.org/pydataglobal2025/talk/QHTA73/ https://cfp.pydata.org/pydataglobal2025/talk/QHTA73/feedback/ General Track Talk 2025-12-09T16:30:00+00:00 16:30 00:30 You may have seen the benchmark results and thought, "how the heck are the Julia ODE solvers on GPUs orders of magnitude faster than the GPU-accelerated Python libraries, that can't be true?" In this talk I will go into detail about the architectural differences between the Julia approaches to generating GPU-accelerated solvers vs the standard ML library approach to GPU usage. By the end of the talk you'll have a good enough understanding of models of GPU acceleration to understand why this performance difference exists, and the many applications that can take advantage of this performance improvement. pydataglobal2025-79202-why-julia-s-gpu-accelerated-ode-solvers-are-20x-100x-faster-than-jax-and-pytorch General Track Chris Rackauckas en This talk is about the results of the publication titled "Automated translation and accelerated solving of differential equations on multiple GPU platforms" which was published in 2024 demonstrating that the Julia GPU-based ODE solvers, specifically DiffEqGPU.jl, are 20x-100x faster than Jax (diffrax) and PyTorch (torchdiffeq). The publication goes into detail as to the architectural reasons for the performance difference, even going as far as recreating the ML style of GPU acceleration in Julia in order to demonstrate that such an approach loses the performance advantage, along with testing against alternative CUDA C++ implementations of a similar form to showcase exactly the effects of the architectural decisions on the resulting performance. However, as a highly technical article it can many times not be as easy to understand as it should. In this talk we're going to give a barebones "no HPC background required" explanation of how the Julia GPU stack enables a completely different approach from the "standard" ML libraries form of GPU acceleration, and how for some applications this can be majorly beneficial. We will note that the GPU design of the ML libraries is actually optimal for ML applications, but certain properties of some applications of ODE solvers make it require a completely different formulation. We will additionally talk about other projects which have seen similar results, such as solving nonlinear systems in Julia (with NonlinearSolve.jl), GPU-accelerated optimization with Optimization.jl, and new global optimizer methods in ParallelParticleSwarms.jl which all rely on this technique and the special aspects of the Julia GPU infrastructure. [1] https://www.sciencedirect.com/science/article/abs/pii/S0045782523007156 false https://cfp.pydata.org/pydataglobal2025/talk/FSTP8H/ https://cfp.pydata.org/pydataglobal2025/talk/FSTP8H/feedback/ General Track Talk 2025-12-09T17:30:00+00:00 17:30 00:30 Hybrid Execution is a new capability introduced in the open-source Modin library that lets developers write familiar pandas code while automatically selecting the most efficient execution backend. Small datasets run locally for fast, interactive development, while larger workloads are transparently pushed down to distributed backends for scalable, high-performance execution. This approach enables faster development for rapid prototyping and iteration and future-proofs pipelines as data volumes grow. pydataglobal2025-79374-bridging-interactive-data-science-and-big-data-with-hybrid-execution General Track Jonathan Shi en pandas is one of the most widely used tools in the Python ecosystem, but scaling it beyond memory limits has traditionally required significant refactoring or switching to other tools. In this talk, we introduce Hybrid Execution, a new capability powered by Modin that allows pandas code to seamlessly switch between local, in-memory execution and distributed backends. This approach preserves the familiar pandas API while enabling users to scale their workflows without rewriting code. We'll explore how Hybrid Execution works under the hood, how Modin enables backend flexibility, and what it means for building interactive, scalable data pipelines with pandas. false https://cfp.pydata.org/pydataglobal2025/talk/93KHNT/ https://cfp.pydata.org/pydataglobal2025/talk/93KHNT/feedback/ General Track Talk 2025-12-09T18:00:00+00:00 18:00 00:30 Most code and related workflows take place in "projects", directories with descriptive metadata. There are so many types of these around these days, it is hard to know what is contained where. projspec solves this for the majority of the python-data ecosystem, so that you can introspect your projects, act on them, and search across all your projects, local or remote. pydataglobal2025-78522-projspec-what-s-this-project-anyway General Track Martin Durant en Daily workflows in pydata usually occur in the context of projects - a directory tree of stuff, with special metadata files describing those contents. Many metadata specifications are in use for each of the many tools that operate on projects, storing information in small yaml, toml or json files, or in the pyproject.toml file for python-specific projects. This model encompasses not only the majority of the environment management tools and task runners in pydata (uv, pixi, poetry, etc) but other essential tools (e.g., git), definitions (e.g., hugging-face dataset), deployment (briefcase, helm, wheel) and workflow-specific metadata (e.g., pyscript). The range of possible metadata is bewildering! Most projects show how to invoke their functionality in README files, with the first step downloading some specific tool. In some way, all this flexibility has taken us backwards. There is no easy way to tell what type a project is and what definitions it contains without reading the supporting documentation and browsing specific files, or even downloading the whole thing and running a specific tool against it. projspec aspires to be a layer over the most common pydata related project types. It provides introspection of project type and contents from the metadata definitions, and this can be done on remote project directories too. For each project type, we infer a set of "contents" (things that are defined in the project and inherently part of it) and "artifacts" (things the project can make or do, usually by calling a subprocess). A project can be multiple types at once: a project designed to be executed with pixi, for instance, still likely contains git information and may also have dataset declarations, things that pixi is not concerned with. Projects may also contain sub-projects of the same or different type, e.g., a conda recipe alongside a code library. Projspec, due to be released in time for this talk, will provide a handy API to work with projects of many types, including introspection and effecting actions. It will have a way to index many projects locally or remotely, to allow for querying with complex criteria, to find the project that matches your needs - contains certain datasets, depends on specific library/versions or is capable of creating particular output types. We will demonstrate all of this! false https://cfp.pydata.org/pydataglobal2025/talk/JCXBBW/ https://cfp.pydata.org/pydataglobal2025/talk/JCXBBW/feedback/ General Track Talk 2025-12-09T18:30:00+00:00 18:30 00:45 Advances in large language models (LLMs) have propelled a recent flurry of AI tools for data management and operations. For example, AI-powered code assistants leverage LLMs to generate code for dataflow pipelines. RAG pipelines enable LLMs to ground responses with relevant information from external data sources. Data agents leverage LLMs to turn natural language questions into data-driven answers and actions. While challenges remain, these advances are opening exciting new opportunities for data scientists and engineers. In this talk, we will examine recent advances, along with some still incubating in research labs, with the goal of understanding where this is all heading, and present our perspective on what’s next for AI in data management and data operations. pydataglobal2025-84935-keynote-by-lisa-amini-what-s-next-in-ai-for-data-and-data-management en Dr. Lisa Amini leads IBM's Data & AI Platforms Research efforts globally, along with IBM's AI Horizons Network. She is also an IBM Distinguished Engineer (DE). The mission of the Data & AI Platforms Research theme is to infuse generative and agentic AI throughout IBM's Data Platform, to make it more intelligent, self-service, and autonomous, and to optimize its performance on AI workloads. false https://cfp.pydata.org/pydataglobal2025/talk/JVJZFT/ https://cfp.pydata.org/pydataglobal2025/talk/JVJZFT/feedback/ Machine Learning & AI Tutorial 2025-12-09T10:30:00+00:00 10:30 01:30 This workshop is designed to equip software engineers with the skills to build and iterate on generative AI-powered applications. Participants will explore key components of the AI software development lifecycle through first principles thinking, including prompt engineering, monitoring, evaluations, and handling non-determinism. The session focuses on using multimodal AI models to build applications, such as querying PDFs, while providing insights into the engineering challenges unique to AI systems. By the end of the workshop, participants will know how to build a PDF-querying app, but all techniques learned will be generalizable for building a variety of generative AI applications. If you're a data scientist, machine learning practitioner, or AI enthusiast, this workshop can also be valuable for learning about the software engineering aspects of AI applications, such as lifecycle management, iterative development, and monitoring, which are critical for production-level AI systems. pydataglobal2025-78637-building-llm-powered-applications-for-data-scientists-and-software-engineers Machine Learning & AI /media/pydataglobal2025/submissions/QREPPX/genai-sdlc_neLT9g1.png hugo bowne-anderson en This workshop is designed to equip software engineers with the skills to build and iterate on generative AI-powered applications. Participants will explore key components of the AI software development lifecycle through first principles thinking, including prompt engineering, monitoring, evaluations, and handling non-determinism. The session focuses on using LLMs to build applications, such as querying PDFs, while providing insights into the engineering challenges unique to AI systems. By the end of the workshop, participants will know how to build a PDF-querying app, but all techniques learned will be generalizable for building a variety of generative AI applications. If you're a data scientist, machine learning practitioner, or AI enthusiast, this workshop can also be valuable for learning about the software engineering aspects of AI applications, such as lifecycle management, iterative development, and monitoring, which are critical for production-level AI systems. **What You'll Learn:** * How to integrate AI models and APIs into a practical application. * Techniques to manage non-determinism and optimize outputs through prompt engineering. * How to monitor, log, and evaluate AI systems to ensure reliability. * The importance of handling structured outputs and using function calling in AI models. * The software engineering side of building AI systems, including iterative development, debugging, and performance monitoring. * Practical experience in building an app to query PDFs using multimodal models. **What is Unique About This Session:** This workshop uniquely bridges the gap between software engineering and generative AI development. While most AI workshops focus solely on model usage or tuning, this session emphasizes the entire AI software lifecycle — from prompt engineering to monitoring and tracing. Participants will learn how to manage non-determinism and create production-ready AI applications, giving them the knowledge to tackle the software engineering challenges of AI-powered apps. The hands-on approach ensures that attendees walk away with practical skills and a functional app. **Workshop Prerequisite Knowledge:** * Basic programming knowledge in Python. * Familiarity with REST APIs. * Experience working with Jupyter Notebooks or similar environments (preferred but not required). * No prior experience with AI or machine learning is required. * Most importantly, a sense of curiosity and a desire to learn! If you have a background in data science, ML, or AI, this workshop will help you understand the software engineering side of building AI applications. We will introduce you to certain modern frameworks in the workshop but the emphasis be on first principles and using vanilla Python and LLM calls to build AI-powered systems. [All tutorial material will be in this github repository](https://github.com/hugobowne/AI-for-SWEs). false https://cfp.pydata.org/pydataglobal2025/talk/QREPPX/ https://cfp.pydata.org/pydataglobal2025/talk/QREPPX/feedback/ Machine Learning & AI Talk 2025-12-09T12:00:00+00:00 12:00 00:30 AI systems are increasingly being integrated into real-world products - from chatbots and search engines to summarisation tools and coding assistants. Yet, despite their fluency, these models can produce confident but false or misleading information, a phenomenon known as **hallucination**. In production settings, such errors can erode user trust, misinform decisions, and introduce serious risks. This talk unpacks the root causes of hallucinations, explores their impact on various applications, and highlights emerging techniques to detect and mitigate them. With a focus on practical strategies, the session offers guidance for building more trustworthy AI systems fit for deployment. pydataglobal2025-78399-when-ai-makes-things-up-understanding-and-tackling-hallucinations Machine Learning & AI Aarti Jha en This session will unpack the problem of AI hallucination - not just what it is, but how it surfaces in everyday use. We’ll look at the common causes, ranging from incomplete context to over-generalisation, and walk through detection and prevention techniques such as grounding, prompt design and RAG. Whether you’re building AI products or evaluating outputs, this talk will give you the tools to recognise hallucinations and reduce their risk. ##### Outline: * Introduction to hallucinations in LLMs * Common causes behind hallucinated outputs * Impact on production applications * Techniques for detecting and evaluating hallucinations * Strategies to reduce hallucinations * Best practices for building trustworthy AI products * Key takeaways ##### Background Knowledge Required: Beginner-friendly - no prior knowledge needed. Familiarity with LLMs is a plus but not necessary. false PyData Global Slide Deck https://cfp.pydata.org/pydataglobal2025/talk/BQLTSH/ https://cfp.pydata.org/pydataglobal2025/talk/BQLTSH/feedback/ Machine Learning & AI Talk 2025-12-09T12:30:00+00:00 12:30 00:30 Discover how Insee (French National Statistics Institute) transitioned from fastText to a PyTorch-based model for text classification by developing and open-sourcing the torchTextClassifiers python package. This presentation will cover the creation, deployment, and practical applications of torchTextClassifiers in modernizing automatic coding systems, benefiting Insee and other European National Statistical Institutes (NSIs). pydataglobal2025-78661-torchtextclassifiers-modernizing-text-classification-for-french-national-statistics Machine Learning & AI Meilame TayebjeeCédric Couralet en Insee, France's National Institute of Statistics and Economic Studies, has long relied on fastText for automatic coding tasks. Recognizing the need to modernize and future-proof this critical functionality, we developed torchTextClassifiers — an open-source Python package that enables easy training and deployment of a PyTorch-based model for text classification, paving the way for further innovation in this domain. This session will delve into the motivations behind replacing the archived fastText package, the design and implementation of torchTextClassifiers , and its integration into Insee's production environment. We'll discuss the challenges faced during this transition, including model compatibility, performance optimization, and user adoption.​ Attendees will gain insights into:​ - The rationale for moving from fastText to a PyTorch-based model​ in production - Packaging a PyTorch-based model architecture and open-source collaboration - Key features and architecture of torchTextClassifiers ​ - Deployment strategies within a public administration (MLOps, cloud native tools, security) - Lessons learned and best practices for similar transitions​ This talk is intended for data scientists, machine learning engineers, and practitioners interested in NLP, model deployment, and open-source tool development. false https://cfp.pydata.org/pydataglobal2025/talk/JGSYEP/ https://cfp.pydata.org/pydataglobal2025/talk/JGSYEP/feedback/ Machine Learning & AI Talk 2025-12-09T13:00:00+00:00 13:00 00:30 This study is oriented to a synthetic non-life insurance premium dataset generated using several Generative Models. As a benchmark, a Conditional Gaussian Mixture Model has been employed. The validation of the generated data involved several steps: data visualisation, comparison with univariate analysis, PCA and UMAP representations between the trained data and the generated samples. In addition, check the consistency of data produced, the statistical Kolmogorov–Smirnov test and predictive modelling of frequency and severity with Generalised Linear Models (GLMs) exploited by Tweedie distribution as a measure of the generated data's quality, followed by the evidence of features importance. For further comparison, advanced Deep Learning architectures have been employed: Conditional Variational Autoencoders (CVAEs), CVAEs enhanced with a Transformer Decoder, a Conditional Diffusion Model, and Large Language Models. The analysis assesses each model’s ability to capture the underlying distributions, preserve complex dependencies, and maintain relationships intrinsic to the premium data. These findings provide insightful directions for enhancing synthetic data generation in insurance, with potential applications in risk modelling, pricing strategies with data scarcity, and regulatory compliance. pydataglobal2025-78761-harnessing-generative-models-for-synthetic-non-life-insurance-data Machine Learning & AI Claudio Giorgio Giancaterino en In classification and regression tasks, generative models aim to learn the joint probability distribution of data. These models focus on generating data points similar to the training data. Open insurance datasets are rare because they encode proprietary risk structures of the Company, limiting researchers’ access to comprehensive data for analysis and assessing new approaches. Generative models enable reproducible experimentation and innovation today. In the talk I explore several generative models used to produce synthetic data. 1) Conditional Gaussian Mixture Models used as a benchmark; 2) Conditional Variational Autoencoders; 3) Conditional Variational Autoencoders with a Transformer Decoder; 4) Conditional Diffusion Model; 5) Large Language Models. Finally, I gave the overall results, followed by different approaches. false https://cfp.pydata.org/pydataglobal2025/talk/GMWTUK/ https://cfp.pydata.org/pydataglobal2025/talk/GMWTUK/feedback/ Machine Learning & AI Talk 2025-12-09T15:00:00+00:00 15:00 00:30 Context Engineering for Agents involves getting relevant data into the LLM’s prompt and builds on in-context learning capabilities of LLMs. But LLMs have finite sized context windows, so you can't just dump unprocessed context data into your Agent's LLM prompt. You need to select the right data, process it into the correct format, and compress or summarize the data before its use as context data. In this talk, we will introduce techniques for selection, preprocessing, and compression of context data, taking inspiration from the tried and tested techniques used for feature engineering for ML. What goes around, comes around. pydataglobal2025-78780-from-feature-engineering-to-context-engineering-for-agents Machine Learning & AI Jim Dowling en Context Engineering for Agents involves getting relevant data into the LLM’s prompt and builds on in-context learning capabilities of LLMs. But LLMs have finite sized context windows, so you can't just dump unprocessed context data into your Agent's LLM prompt. You need to select the right data, process it into the correct format, and compress or summarize the data before its use as context data. In this talk, we will introduce techniques for selection, preprocessing, and compression of context data, taking inspiration from the tried and tested techniques used for feature engineering for ML. What goes around, comes around. false https://cfp.pydata.org/pydataglobal2025/talk/RL9RDQ/ https://cfp.pydata.org/pydataglobal2025/talk/RL9RDQ/feedback/ Machine Learning & AI Talk 2025-12-09T17:30:00+00:00 17:30 00:30 Training state-of-the-art Large Language Models (LLMs) increasingly rely on the availability of clean, diverse, and large-scale datasets. The traditional CPU-based preprocessing pipelines often become a bottleneck when curating datasets that span tens or hundreds of terabytes. In this talk, we introduce NeMo Curator, an open-source, GPU-accelerated data curation framework developed by NVIDIA. Built on Python and powered by RAPIDS, NeMo Curator enables scalable, high-throughput data processing for LLMs, including semantic deduplication, filtering, classification, PII redaction, and synthetic data generation. With support for multi-node, multi-GPU environments, the framework has demonstrated up to 7% improvement in downstream model performance on large-scale benchmarks. We will walk through its modular pipeline design, highlight real-world applications, and show how to integrate it into existing workflows for fast, reproducible, and efficient LLM training. pydataglobal2025-79352-scaling-data-processing-for-llms-with-nemo-curator Machine Learning & AI /media/pydataglobal2025/submissions/JVPL8S/NeMo_b40VXqv.png Allison Ding en The development and performance of Large Language Models (LLMs) increasingly rely on the availability of high-quality, diverse, and representative datasets. Scaling data preparation for LLMs remains a significant bottleneck in training pipelines, particularly when dealing with massive raw web-scale data. Traditional CPU-based preprocessing frameworks are often too slow and resource-intensive to meet the growing demand for efficiency, scalability, and compliance. This talk presents NeMo Curator, an open-source, GPU-accelerated data curation framework designed to accelerate and streamline the preparation of massive datasets across multi-node, multi-GPU infrastructures. NeMo Curator introduces a modular pipeline architecture that enables high throughput preprocessing with native integration of RAPIDS for GPU acceleration. Its functionality spans semantic deduplication, heuristic filtering, automated classification, personally identifiable information (PII) redaction, and synthetic data generation. These features work in tandem to reduce noise, eliminate redundancy, and enhance data quality, ultimately improving LLM training outcomes. With support for reward-based filtering and configurable augmentation modules, NeMo Curator can generate or enhance data in low-resource domains while maintaining quality and diversity. This talk will provide an informative walkthrough of NeMo Curator’s capabilities and show how its pipelines can be integrated into existing workflows to preprocess massive datasets efficiently. Attendees will see how to configure and execute the framework through Python APIs, leveraging both single-node and distributed environments. By the end of this talk, participants will become familiar with scalable data curation techniques and walk away with practical tools to enhance their own LLM training pipelines using GPU-accelerated infrastructure. Detailed Outlines: 1. Challenges in Scaling LLM Data Preparation (5 min) 2. Overview of NeMo Curator Framework (10 min) 3. Pipeline Modules and Functional Components (5 min) 4. Demonstration: Multi-GPU Pipeline Execution (5 min) 5. Case Studies and Performance Metrics (5 min) Targeted Audience: • Data Scientist, ML/AI Engineer, AI Researcher false https://cfp.pydata.org/pydataglobal2025/talk/JVPL8S/ https://cfp.pydata.org/pydataglobal2025/talk/JVPL8S/feedback/ Machine Learning & AI Talk 2025-12-09T18:00:00+00:00 18:00 00:30 Want to understand how transformers actually work without wading through 10,000 lines of framework code or drowning in tensor shapes? This talk walks you through building a transformer model from scratch — no pre-trained shortcuts, no black-box abstractions — just clean PyTorch code and good old-fashioned curiosity. You'll walk away with a clearer mental model of how attention, encoders, decoders, and masking really work. pydataglobal2025-78526-i-built-a-transformer-from-scratch-so-you-don-t-have-to Machine Learning & AI Jen Wei en Transformers power modern large language models, but their inner workings are often buried under complex libraries and unreadable abstractions. In this talk, we’ll peel back the layers and build the original Transformer architecture (Vaswani et al., 2017) step by step in PyTorch, from input embeddings to attention masks to the full encoder-decoder stack. This talk is designed for attendees with a basic understanding of deep learning and PyTorch who want to go beyond surface-level blog posts and get a hands-on, conceptual grasp of what happens under the hood. You'll see how each part of the transformer connects back to the equations in the original paper, how to debug common implementation pitfalls, and how to avoid getting lost in tensor dimension hell. This talk features: 🔍 A walkthrough of key components: attention, positional encoding, encoder/decoder stack 🧠 Visual explanations of attention masks, shapes, and residuals ⚠️ Common bugs and debugging strategies (like handling shape mismatches and masking errors) ✅ Real-world implementation tips and tricks that demystify the architecture By the end of the talk, attendees will: Understand the full forward pass of a transformer Know how each component connects to the original paper Feel more confident reading or writing custom model architectures The tone will be light-hearted and educational — ideal for those who are mathematically curious but don’t want to get bogged down in heavy theory. No prior experience building models from scratch required — just a working knowledge of Python and PyTorch. **Prior Knowledge Expected** Basic Python and PyTorch Some familiarity with neural networks (e.g., feedforward, softmax) No need for prior experience in building models from scratch false https://cfp.pydata.org/pydataglobal2025/talk/QWXTAN/ https://cfp.pydata.org/pydataglobal2025/talk/QWXTAN/feedback/ Analytics, Visualization & Decision Science Talk 2025-12-09T12:30:00+00:00 12:30 00:30 Tired of exact matches failing on messy data? This talk showcases how BM25, a powerful fuzzy search algorithm, tackles the challenge of enriching massive datasets with noisy product names. We'll compare practical, large-scale implementations using Python's bm25s library (accelerated by GPUs) and DuckDB's built-in full-text search. Join us to learn how to achieve fast, accurate data integration and discover the optimal tools for your fuzzy matching needs. pydataglobal2025-78697-scaling-fuzzy-product-matching-with-bm25-a-comparative-study-of-python-and-database-solutions Analytics, Visualization & Decision Science Aniket Abhay Kulkarni en **The problem at hand:** Are you constantly battling messy, inconsistent product names across massive datasets? Traditional exact matching just doesn't cut it when you're trying to integrate data from various sources (like a 1-million-row internal catalog with a 3.8-million-row external one like Open Food Facts). This talk addresses that exact problem: how to efficiently and accurately find fuzzy matches, saving you countless hours of manual reconciliation and enabling robust data enrichment. It's crucial for anyone working with real-world, imperfect data at scale. **Is this talk for me?** This talk is for data engineers, data scientists, and analytics professionals who work with large-scale datasets and face challenges with data integration, record linkage, or building robust search functionalities. A basic understanding of dataframes and SQL will be helpful, but no deep prior knowledge of search algorithms is required. This will be an informative and practical talk with a clear focus on real-world application. While we'll briefly cover the "why" behind BM25, the emphasis will be on "how" to implement and optimize it. We'll present concrete benchmarks and code examples, moving beyond theoretical concepts. **What will I learn?** By the end of this session, you will: - Understand why BM25 is a superior choice for fuzzy matching noisy product names compared to traditional methods. - See a practical, head-to-head comparison of implementing BM25 using Python libraries (specifically the optimized Cython bm25s) and DuckDB's native full-text search. - Gain insights into performance implications (speed and memory usage) for each approach on large datasets, including the benefits of GPU acceleration with Dask CuDF. - Learn production tips for persisting indexes, handling bulk queries, and managing memory effectively. - Be equipped to choose the most suitable BM25 implementation for your specific data enrichment and fuzzy matching needs, allowing you to build faster and more accurate data pipelines. **Any pre-requisite knowledge I should have?** - A medium level background in python - An introductory level information about DuckDB - An introductory level information into how BM25 works would be bonus! false https://cfp.pydata.org/pydataglobal2025/talk/8NYGXU/ https://cfp.pydata.org/pydataglobal2025/talk/8NYGXU/feedback/ Analytics, Visualization & Decision Science Talk 2025-12-09T13:30:00+00:00 13:30 00:30 In general, a Data Scientist spends significant efforts in transforming the raw data into a more digestible format before training an AI model or creating visualisations. Traditional tools such as pandas have long been the linchpin in this process, offering powerful capabilities but not without limitations. With numerous possible ways to write the same thing in pandas, often a user ends up selecting the uneconomical, inefficient ones, leading to large computational　costs　with the growth in data size. We introduce a couple of frequently occurring　intricate performance issues in pandas, and what we have learnt in solving the same using popular high-performance pandas alternatives: Polars, FireDucks and cuDF. The talk intends to highlight one of the best practices (breaking out of the loops) that one should follow while dealing with large-scale data analysis, while demonstrating the key advantages of the high-performance pandas alternatives based on different scenarios. pydataglobal2025-78768-lessons-learnt-in-optimizing-a-large-scale-pandas-application-using-polars-fireducks-and-cudf-go-smart-and-save-more Analytics, Visualization & Decision Science Sourav Saha en It is a known factor that pandas might be slow when dealing with large-scale data analysis, but the know-how of writing effective pandas application might save you a lot. For a data scientist who is primarily specialized in finding the key insights out of the data, it might be difficult to program from the perspective of runtime memory consumption, effective data flow optimization etc. High-performance pandas alternatives like Polars, FireDucks, cuDF etc. are designed to address these issues and can be very useful in saving a lot of operational cost (e.g., cloud cost, human cost etc.). We will talk about the key lessons we have learnt in optimizing a large-scale pandas application and the decision points in selecting the high-performance pandas alternatives. It can be very useful for the contemporary data professional who loves the flexible user APIs in pandas and wants to enhance the performance of their application without much effort when dealing with voluminous and complex data on a regular basis. The key takeaways would be as follows: 1. How the choice and execution order of API calls in writing an data-related application impacts its performance. 2. How to stop thinking the loop-based approach and design the algorithms using DataFrame APIs. 3. How the internal query optimizers in libraries like Polars, FireDucks etc, can be useful to bring SQL-like optimizations at python-level. 4. Whether to pay a large migration cost for optimizing an existing pandas-based application or to go smart with some minor modifications and save more operational cost. Here is the [presentation deck](https://github.com/qsourav/PyData-Global-2025/blob/main/docs/PyDataGlobal_20251209.pdf) used during the talk. false https://cfp.pydata.org/pydataglobal2025/talk/VHX7E7/ https://cfp.pydata.org/pydataglobal2025/talk/VHX7E7/feedback/ Analytics, Visualization & Decision Science Talk 2025-12-09T19:30:00+00:00 19:30 00:30 Ensuring and communicating data quality (DQ) is one of the most persistent challenges in data-driven organizations. Data scientists, engineers, and analysts often struggle not just with detecting DQ issues, but with presenting those issues in actionable ways for diverse stakeholders across an organization (e.g., pipeline owners, fellow developers, less-technical colleagues, etc). On top of this, DQ work has an image problem as it can be seen as tedious, opaque, or even adversarial. This talk introduces Pointblank, a Python package designed to make data quality validation and communication both robust and approachable. The library provides a comprehensive set of tools for profiling, validating, and reporting on data quality. There’s a strong focus on beautiful and actionable outputs as well. It can help you to generate tabular validation reports, data summaries, and granular error reporting that make it easy for anyone (technical or not) to understand what’s wrong and why. Attendees will learn how Pointblank can help their teams not only catch data issues early, but also communicate them effectively, fostering a culture of shared responsibility for data quality. The talk will include live demos of common DQ workflows, showing how Pointblank turns a traditionally painful process into something transparent, productive, and even a little bit fun. pydataglobal2025-78738-communicating-data-quality-making-the-invisible-visible-and-fun-with-pointblank Analytics, Visualization & Decision Science Richard Iannone en The overall goal of this talk is to get people excited about DQ and show how the Pointblank library makes DQ validation and communication easier, clearer, and more collaborative. I’ll demonstrate some practical workflows that will hopefully inspire attendees to treat DQ as a shared (yet approachable) responsibility. Here’s an outline for this talk: 1. The Data Quality Communication Problem - why DQ is hard: technical, social, and organizational barriers - the “last mile” problem: not just finding issues, but making them clear and actionable - the validation plan, execution, and report lifecycle 2. Introducing Pointblank - overview of the package and its philosophy: affordances for humans, not just machines - key features: validation, profiling, reporting, and workflow support 3. Making Data Quality Actionable - live demo: Python API for data profiling, validation, and missing value reports - nice-looking outputs: tabular report, step-by-step summaries, and crystal-clear DQ messaging how these outputs can help people get to the root of DQ problems faster 4. Flexible Workflows - using LLMs to draft a validation plan - creating a validation plan from YAML - integrating with CI/CD and data pipelines 5. Designing this Library for Collaboration and Fun - small design choices can make a big difference: easy-to-understand summaries, actionable extracts, and a user-friendly CLI - my personal goal: make DQ work less annoying and more rewarding I imagine the intended audience as being composed of data engineers, scientists, analysts, and anyone responsible for data quality. Also, this talk might interest team leads and managers looking to improve DQ culture in their organization. Insofar as skill level, this talk is suitable for Python users at any level. false https://cfp.pydata.org/pydataglobal2025/talk/9CMRXJ/ https://cfp.pydata.org/pydataglobal2025/talk/9CMRXJ/feedback/ Data Engineering & Infrastructure Tutorial 2025-12-09T12:30:00+00:00 12:30 01:30 **Abstract:** Blockchain generates millions of transactions daily, making it a rich yet complex source of data for developers, analysts, and researchers. While Google BigQuery offers public access to Solana’s historical data, repeated querying at scale can become costly and slow, especially during iterative exploration and analysis. In this talk, I’ll demonstrate a practical workflow that combines the power of BigQuery for data extraction with the speed and flexibility of DuckDB for local, in-memory analytics. We’ll show how to efficiently query Solana data in BigQuery, export it to partitioned Parquet files, and use DuckDB to run fast, repeatable SQL queries without incurring additional cloud costs. You'll learn: - Basic terms in blockchain data structure and how transactions are saved. - How to navigate and query Solana’s public datasets on BigQuery. - How to export filtered blockchain data to efficient Parquet files. - How DuckDB can serve as a lightweight analytics engine for on-chain data. - Tips for partitioning, enriching, and automating your Solana data pipeline. This demo would all run within Google collab to save time and also enable participant follow through the session. Whether you're working on blockchain analytics, wallet behavior analysis, or on-chain data engineering, this talk will equip you with a practical approach to blockchain data workflows using open tools. pydataglobal2025-78783-fast-cost-efficient-analytics-on-blockchain-data-using-duckdb-solana-as-a-case-study Data Engineering & Infrastructure Busirah Olaitan Hammed en This talk explores how to build a workflow for Solana blockchain data using BigQuery and DuckDB. You'll learn how to query Solana’s public datasets in BigQuery, export key data as Parquet files, and use DuckDB for high-speed, ideal for blockchain developers, data engineers, and analysts working with large on-chain datasets. false https://cfp.pydata.org/pydataglobal2025/talk/HKWFL8/ https://cfp.pydata.org/pydataglobal2025/talk/HKWFL8/feedback/ Data Engineering & Infrastructure Talk 2025-12-09T15:00:00+00:00 15:00 00:30 While commercial reverse geocoding APIs, such as Google Maps or Mapbox, are effective, they are also costly, have rate limitations, and are not appropriate for offline or privacy-sensitive settings. Using available datasets and Python modules like `cKDTree`, `shapely`, and `geopandas`, we will demonstrate how to create a quick, scalable, offline-capable reverse geocoding system in Python in this session. You will learn how to: - Convert geographic shapefiles into effective spatial indices - Perform location lookups in milliseconds using tree search and vector mathematics - Handle edge cases like unclear borders, cities with identical names, and GPS noise - Improve performance and memory usage through multiprocessing The system is fully open source and has been production-tested in a high-throughput environment. Whether you are developing applications for edge inference, mapping, or logistics, this talk will help you take control of your geospatial infrastructure without depending on costly commercial APIs. pydataglobal2025-79394-designing-a-fast-offline-capable-reverse-geocoder-in-python-an-open-source-alternative-to-big-geo-apis Data Engineering & Infrastructure Sooraj Sivadasan en Reverse geocoding — converting coordinates into readable place names — is a core building block of applications in logistics, mapping, mobility, and location intelligence. Yet developers are often locked into commercial APIs that are expensive, rate-limited, and unsuitable for offline or privacy-first use cases. In this talk, we’ll walk through the architecture and implementation of a fast reverse geocoding engine built entirely in Python using open-source tooling. You’ll see how spatial data (such as OpenStreetMap shapefiles) can be indexed efficiently using `scipy`'s `cKDTree`, queried with millisecond latency, and integrated into real-world systems. We’ll explore performance trade-offs, data preprocessing techniques, and methods for dealing with ambiguous or noisy GPS data. The session includes benchmarks and a live walkthrough of the code powering the reverse geocoder — which is lightweight enough to run on a laptop or edge device. Attendees will leave with a clear understanding of how to build and adapt this system for their own needs — and gain insight into how geospatial systems work behind the scenes. false https://cfp.pydata.org/pydataglobal2025/talk/CDEZQQ/ https://cfp.pydata.org/pydataglobal2025/talk/CDEZQQ/feedback/ Data Engineering & Infrastructure Talk 2025-12-09T15:30:00+00:00 15:30 00:30 In this talk, I will delve into the world of Apache NiFi 2.0 Python processors, exploring the capabilities they offer and demonstrating how to build custom processors to enhance your data processing pipelines. By the end of this talk, participants will have a comprehensive understanding of building and optimizing Apache NiFi 2.0 Python processors, enabling them to integrate Python seamlessly into their data processing workflows. This session is suitable for data engineers, architects, and anyone interested in harnessing the combined power of Apache NiFi and Python for efficient data integration and flow management. One of the main uses is to build prompts and call open LLM and AI. NiFi excels at integration, I will cover some interesting sources, sinks and enrichments and show when Python is helpful. pydataglobal2025-78844-enhancing-apache-nifi-2-x-with-python-processors Data Engineering & Infrastructure /media/pydataglobal2025/submissions/3BLRCH/2025-07-14_19-42-30_n0fcoVY.png Timothy Spann en In this talk, I will delve into the world of Apache NiFi 2.0 Python processors, exploring the capabilities they offer and demonstrating how to build custom processors to enhance your data processing pipelines. By the end of this talk, participants will have a comprehensive understanding of building and optimizing Apache NiFi 2.0 Python processors, enabling them to integrate Python seamlessly into their data processing workflows. This session is suitable for data engineers, architects, and anyone interested in harnessing the combined power of Apache NiFi and Python for efficient data integration and flow management. One of the main uses is to build prompts and call open LLM and AI. NiFi excels at integration, I will cover some interesting sources, sinks and enrichments and show when Python is helpful. false https://cfp.pydata.org/pydataglobal2025/talk/3BLRCH/ https://cfp.pydata.org/pydataglobal2025/talk/3BLRCH/feedback/ Data Engineering & Infrastructure Talk 2025-12-09T16:30:00+00:00 16:30 00:30 TIFF, HDF5, and Zarr represent a few choices to store large n-dimensional arrays which represent scientific and machine learning data. Trade-offs have to be considered when selecting one of these formats. While TIFF files are recognized by many applications particularly for imaging, they are limited in the number of dimensions, two, traditionally, or three in the case of GeoTIFF. HDF5 was created to support hierarchical scientific data with arrays up to 32 dimensions, but are mainly readable by scientific applications. Neither TIFF nor HDF5 were designed with the cloud in mind. Meanwhile, Zarr reimagined HDF5 in the era of cloud computing and key-value object stores. In retrospect, these disparate formats have many similarities. I will demonstrate how to take advantage of these similarities to combine the formats and make data accessible to a wide range of local and cloud-based application without duplicating the data itself. pydataglobal2025-79466-combining-zarr-hdf5-and-tiff-into-a-single-data-format Data Engineering & Infrastructure Mark Kittisopikul, Ph.D. en Choosing a standard format for high dimensional (N >= 2) array data is challenging in that one must consider trade-offs between compatible software packages, cloud optimization, and complexity, yet the need for such data has increased with recent advances in machine learning and volumetric imaging in the earth and biological sciences. The 927th installment of the XKCD comic series illustrates how standards proliferate: the existence of many prior and imperfect standards portends the creation of yet another standard to supplant the ones that came before often without considering similarities or compatability with prior standard formats. For n-dimensional data, TIFF, HDF5, and Zarr are now common formats in use across various fields and scientific domains. While TIFF and HDF5 were designed decades ago with flexible metadata structures, cloud optimization of these formats have helped to consolidate metadata in these formats and narrow the differences with the cloud-native file format Zarr. While Zarr has traditionally used individual keys for each compressed chunk, version 3 of the format introduces a sharding codec allowing multiple chunks to exist in the same file under a single key. The consolidation of chunks is reminiscent of tiles in TIFF files or chunked datasets in HDF5. Essentially each of these file formats have the capability to describe the location and sizes of individual blocks of data contained within. By taking advantage of metadata consolidation to achieve modularity, we can tailor and combine these formats to point to the same data blocks, avoiding duplication. The result is a hybrid file format that is simultaneously a TIFF, HDF5, and Zarr v3 shard. Readers of any of these formats can be used to read the same data blocks contained within this format. To illustrate the concept of a combined Zarr, HDF5, and TIFF format, I have created an example Jupyter notebook demonstrating a small Python library that can write data in this hybrid format. I then show how data can be read using libtiff, h5py, or tensorstore, manipulated by h5py, and then have the changes read using the same libraries. https://github.com/mkitti/simple_image_formats/blob/main/header_formats.ipynb false https://cfp.pydata.org/pydataglobal2025/talk/AAGRYV/ https://cfp.pydata.org/pydataglobal2025/talk/AAGRYV/feedback/ Data Engineering & Infrastructure Tutorial 2025-12-09T19:30:00+00:00 19:30 01:30 NVIDIA GPUs offer unmatched speed and efficiency for data processing and model training, significantly reducing the time and cost associated with these tasks. Using GPUs is even more tempting when you use zero-code-change plugins and libraries. You can use PyData libraries including pandas, polars and networkx without needing to rewrite your code to get the benefits of GPU acceleration. We can also mix in GPU native libraries like Numba, CuPy and pytorch to accelerate our workflows from end-to-end. However, integrating GPUs into our workflow can be a new challenge where we need to learn about installation, dependency management, and deployment in the Python ecosystem. When writing code, we also need to monitor performance, leverage hardware effectively, and debug when things go wrong This is where RAPIDS and its tooling ecosystem comes to the rescue. RAPIDS, is a collection of open source software libraries to execute end-to-end data pipelines on NVIDIA GPUs using familiar PyData APIs. pydataglobal2025-78723-gpu-python-for-the-real-world-practical-steps-to-gpu-accelerated-python-with-rapids Data Engineering & Infrastructure Jacob TomlinsonNaty Clementi en In this tutorial we will cover: - Introduction to cuDF, cuML and more that showcases a simple example of data processing and model training on GPUs. - Answers to questions like: “Where do I get a GPU?”, “How do I run a container on a VM with a GPU?”, “How do I install GPU packages into an existing environment?”, as well as follow along examples to get a GPU up and running. - Troubleshooting and monitoring: Examples of performance analysis, diagnostics, and debugging. This is a hands-on tutorial, with multiple examples to get familiarized with the RAPIDS ecosystem. Participants should ideally have some experience using Python, pandas and sci-kit learn. We'll use cloud-based VMs, so familiarity with the cloud and resource creation is helpful but not required. No prior GPU knowledge is needed. false https://cfp.pydata.org/pydataglobal2025/talk/9HUY9G/ https://cfp.pydata.org/pydataglobal2025/talk/9HUY9G/feedback/ Live from PyData Boston Talk 2025-12-09T15:00:00+00:00 15:00 00:40 Most data science projects start with a simple notebook—a spark of curiosity, some exploration, and a handful of promising results. But what happens when that experiment needs to grow up and go into production? This talk follows the story of a single machine learning exploration that matures into a full-fledged ETL pipeline. We’ll walk through the practical steps and real-world challenges that come up when moving from a Jupyter notebook to something robust enough for daily use. We’ll cover how to: - Set clear objectives and document the process from the beginning - Break messy notebook logic into modular, reusable components - Choose the right tools (Papermill, nbconvert, shell scripts) based on your workflow—not just the hype - Track environments and dependencies to make sure your project runs tomorrow the way it did today - Handle data integrity, schema changes, and even evolving labels as your datasets shift over time And as a bonus: bring your results to life with interactive visualizations using tools like PyScript, Voila, and Panel + HoloViz pydataglobal2025-84930-the-lifecycle-of-a-jupyter-environment-from-exploration-to-production-grade-pipelines Live from PyData Boston Dawn Wages en - (3 mins) Intro - I've been supporting various groups in their developer experience since 2020 after being a freelance Python consultant. I've worked on many many dozens of projects, unblocking users picking the right tools for the task at hand. - It works on my machine - What we're building today: ML pipeline with RAPIDS -> Snowflake - We're going to watch a real project grow up - (3 mins) Exploration - starting as a single messy notebook, sample data set. - Why RAPIDS? GPU - Large data sets - GPU availability - remote machine, local GPU - workflows that work well with GPU - Load Data cuDF / pandas - Quick EDA and data visualization - Train cuML / scikit-learn model - no-code change philosophy - (7 mins) Make it repeatable - Start with simple tried and true tools, explore where tools like Papermill help with flexibilty and reproducibility - common painpoints: operating cadence, specialized scenarios, manual execution is error prone - shell scripts versus papermill - reproducible environments - generate HTML reports - pass through parameters in your notebook - (8 mins) Make it reliable - Modular code & testing - common painpoints: data schema changes, debugging issues, testing & modularity - nbconvert + Python: turn your notebook into a script - turn a function into a module - dashboard with HoloViz / Panel, discuss choosing tools like Voila and PyScript - (5 mins) Snowflake integration - common painpoints: data volume, coordinate with other data systems, audits - picking the right tools: cost complexity tradeoff - RAPIDS preprocessing to Snowflake storage - self-service access for stakeholders - (3 mins) Conclusion - Start simple - Add complexity when you feel specific pain false https://cfp.pydata.org/pydataglobal2025/talk/8CFCDH/ https://cfp.pydata.org/pydataglobal2025/talk/8CFCDH/feedback/ Live from PyData Boston Talk 2025-12-09T16:15:00+00:00 16:15 00:40 Informed Consent Forms (ICFs) are critical documents in clinical trials. They are the first, and often most crucial, touchpoint between a patient and a clinical trial study. Yet the process of developing them is laborious, high-stakes, and heavily regulated. Each form must be tailored to jurisdictional requirements and local ethics boards, reviewed by cross-functional teams, and written in plain language that patients can understand. Producing them at scale across countries and disease areas demands manual effort and creates major operational bottlenecks. We used a combination of traditional AI and large language models to autodraft the ICF across clinical trial types, across countries and across disease areas at scale. The build, test, iteration and deployment offers both technical and non technical lessons learned for generative AI applications for complex documents at scale and for meaningful impact. pydataglobal2025-84933-using-traditional-ai-and-llms-to-automate-complex-and-critical-documents-in-healthcare Live from PyData Boston Aman BhandariLily Xu en Informed Consent Forms are highly complex documents that require high precision and quality. A phase 2 / 3 clinical trial can have almost 1000 different forms that takes considerable time to complete.We identified this challenge that directly impacts trial timelines and patient engagement. The automated AI solution: the “ICF Autodrafter”, a custom LLM-powered application that automates the drafting of ICFs. This tool ingests a clinical trial protocol and ICF template and outputs a complete draft in minutes, cutting document preparation time by 90%. This solution is not generic automation. The backend logic parses highly structured protocol documents, segments them, and feeds the relevant content into a carefully fine-tuned LLM that maps text to specific ICF fields. The front-end is designed for usability by clinical trial managers, with human-in-the-loop reviews. This system has already supported ICF creation for more than ten trials and has achieved near-perfect consistency (97%) with human-generated content, underscoring the speed, quality, and robustness of the solution. We rigorously test version with A/B comparisons, iterated with feedback from end-users, and anchored all development within regulatory and ethical guardrails. The impact extends beyond efficiency. By standardizing and accelerating ICF production, we can reduce delays in trial start-up and potentially get medicines to patients faster, without compromising safety, compliance, or clarity. Furthermore, it also lays down a scalable model for future AI-driven document workflows across other parts of life sciences and healthcare. false https://cfp.pydata.org/pydataglobal2025/talk/UHN9UX/ https://cfp.pydata.org/pydataglobal2025/talk/UHN9UX/feedback/ Live from PyData Boston Talk 2025-12-09T17:00:00+00:00 17:00 00:40 How exactly does one validate the factuality of answers from a Retrieval-Augmented Generation (RAG) system? Or measure the impact of the new system prompt for your customer service agent? What do you do when stakeholders keep asking for "accuracy" metrics that you simply don't have? In this talk, we’ll learn how to define (and measure) what “good” looks like when traditional model metrics don’t apply. pydataglobal2025-84934-where-have-all-the-metrics-gone Live from PyData Boston Dr. Rebecca Bilbro en In the good old supervised learning days, standard measures like accuracy, F1, and MSE were like blazes on the data science trail, showing us how to descend the gradient towards "better". But now we're in uncharted analytics territory, where our work increasingly involves unlabeled data and generative AI outputs, and metrics are either unavailable or undefined. The key to every successful trek is preparation. We have to move from thinking about “metrics as defaults” to “metrics as design choices." We also need to be ready to design those metrics before we even start testing, because when we devise metrics post-training, we risk HARKing (Hypothesizing After Results are Known) and losing our scientific footing. This talk will provide a field guide for translating different kinds of modern research questions into clearly-defined metrics, including: * Metrics of the past and why they aren't as useful now (~5 min) * Common failure modes when attempting to evaluate generative AI outputs and other unlabeled data (~8 min) * Techniques for identifying proxies when labels are missing (~8 min) * Defining criteria for open-ended outputs (~8 min) * Open source Python libraries (including new tools like [outlines](https://github.com/dottxt-ai/outlines) and [dspy](https://github.com/stanfordnlp/dspy) as well as old favorites like [hypothesis](https://hypothesis.readthedocs.io/en/latest/) and [pytest](https://docs.pytest.org/en/stable/)) to equip you for your next data science adventure (~8 min) Come learn how to define and adapt new metrics so that you'll be prepared for wherever your modeling journey takes you. false https://cfp.pydata.org/pydataglobal2025/talk/V7GSU7/ https://cfp.pydata.org/pydataglobal2025/talk/V7GSU7/feedback/ Live from PyData Boston Talk 2025-12-09T19:30:00+00:00 19:30 00:40 Why do male test takers consistently score about 30 points higher than female test takers on the mathematics section of the SAT? Does this reflect an actual difference in math ability, or is it an artifact of selection bias—if young men with low math ability are less likely to take the test than young women with the same ability? This talk presents a Bayesian model that estimates how much of the observed difference can be explained by selection effects. We’ll walk through a complete Bayesian workflow, including prior elicitation with PreliZ, model building in PyMC, and validation with ArviZ, showing how Bayesian methods disentangle latent traits from observed outcomes and separate the signal from the noise. No prior knowledge of Bayesian statistics is required; attendees should be familiar with Python and common probability distributions. pydataglobal2025-84936-the-sat-math-gap-gender-difference-or-selection-bias Live from PyData Boston Allen Downey en Overview This talk uses the SAT math gap as a case study to demonstrate modern Bayesian modeling in practice. For decades, male test takers have outperformed female test takers on the SAT math section by about 30 points. This outcome could reflect an actual difference in ability, or it could be explained by selection bias, if boys with weaker math skills are less likely to take the SAT than girls with comparable skills. I present a generative Bayesian model that explicitly incorporates this selection mechanism and estimates the fraction of the observed gap attributable to bias. The talk emphasizes workflow over theory: how to build, validate, and interpret Bayesian models using PyMC, ArviZ, and PreliZ. Audience The target audience includes data scientists, applied researchers, and engineers who: * Use Python for data analysis, * Have basic familiarity with probability distributions, * Are curious about Bayesian modeling but do not necessarily have prior experience with PyMC or Bayesian statistics. Learning goals Attendees will learn: * How to frame a substantive question as a Bayesian generative model, * How to use PreliZ for prior elicitation, PyMC for model building, and ArviZ for diagnostics and posterior predictive checks, * How to interpret results in terms of latent traits vs. observed outcomes, * How Bayesian models can provide a principled way to reason about confounding and bias. Outline (approx. 30–40 minutes) Introduction & background (5 min) – The SAT math gap and the debate over its causes – Why Bayesian inference is a good fit for this problem Model construction (10 min) – Latent efficacy distribution – Selection mechanism (logistic link) – Noise modeling for score perturbations Workflow demonstration (15 min) – Prior elicitation with PreliZ – Sampling and diagnostics with PyMC and ArviZ – Posterior predictive checks Results & interpretation (5–7 min) – Estimated contribution of selection bias to the observed gap – Broader implications for educational testing and applied modeling Takeaways (3–5 min) – Lessons about Bayesian workflow – Relevance to real-world problems of bias and confounding Materials All code and data preprocessing will be available in a public GitHub repository so attendees can reproduce the analysis and adapt it to their own work. false https://cfp.pydata.org/pydataglobal2025/talk/YHTMZY/ https://cfp.pydata.org/pydataglobal2025/talk/YHTMZY/feedback/ Live from PyData Boston Talk 2025-12-09T20:45:00+00:00 20:45 00:40 Do you feel lost in the jungle of GenAI frameworks and buzzwords? Here's a way out. Take any GenAI app, peel away the fluff, and look at its core. You'll find the same pattern: a boringly simple nested while loop. I will show you how this loop produces chat assistants, AI agents, and multi-agent systems. Then we'll cover how RAG, tool-calling, and memory are like lego bricks we add as needed. This gives you a first-principles based map. Use it to build GenAI apps from scratch; no frameworks needed. pydataglobal2025-84937-the-boringly-simple-loop-powering-genai-apps Live from PyData Boston Sebastian Wallkötter en ### Central Thesis We are at a point where talking about GenAI apps has become more complex than building them. Social media is obsessed with the "top 10 libraries for GenAI", search engines are swamped with shallow tutorials, and many devs I meet are rightfully confused what frameworks they should spend time on. The answer is "none, GenAI isn't all that complicated". However, that answer isn't sexy because it doesn't grab attention, doesn't sell consulting hours, and doesn't convince someone to buy an online course. Hence few people give it. That has to change! That's what this talk is about: The boringly simple basics of building GenAI apps and how you can use a simple nested while loop to build assistants, AI agents, or multi-agent systems. Sometimes less is more. ### Takeaways - Create prototypes of agentic apps from scratch using fundamental building blocks - Choose the right components (like RAG or MCP) for your specific problem - Debug agentic apps by spotting misconfigured context ### Target Audience This talk is for the software engineer and data professional that wants to get hands-on with GenAI. Medium and Substack taught you concepts like RAG and AI Agents, social media hyped you up, and now it’s time to build. The only problem: Where do you start? How do you turn "let's build something that does XYZ" into a concrete software product? If you feel like you are sitting with a pile of Lego pieces while everyone else is playing with a completed spaceship, this talk is for you. It's for builders who are ready to go from reading to coding. ### Prerequisites You should have working knowledge of Python and familiarity with LLM terminology (tokens, context window, system prompt, ...). If you're comfortable reading source code, you have everything you need. No prior experience with frameworks like LangChain, LlamaIndex, or others is necessary. ### Outline **Introduction** (2 min) **The core loop** (15 min) - Introduction to the fundamental pattern that orchestrates GenAI apps (the "core loop") - Definition of the terms "Turns" and "Traces" that are foundational to building and optimizing flows - Showcase on how to create assistants, workflows, AI agents, and multi-agent systems using this pattern **Context Engineering** (15min) - Introduction to the three parts of context engineering: Plans, Knowledge, and Tools. - Discussion on how these parts relate to the core loop and where to define them - Showcase how RAG, MCP, memory, etc. assist in setting up the system context **Q&A** (5min) **Buffer** (3min) ### Bio I'm an engineer and open-source maintainer with a PhD in Computer Science and over a decade of hands-on experience building with AI/ML. Having scaled ImageIO, a foundational Python library, from 2 to 35 million monthly downloads, I know what it takes to build robust, scalable software. I co-founded PyData Stockholm and am deeply integrated into our data community. My current focus is to bring first principles thinking to the GenAI landscape and help developers build more robust systems. false https://cfp.pydata.org/pydataglobal2025/talk/TZSWMW/ https://cfp.pydata.org/pydataglobal2025/talk/TZSWMW/feedback/ General Track Talk 2025-12-10T12:00:00+00:00 12:00 00:30 Scientific Python Ecosystem offers a wide variety of numerical packages, such as NumPy, CuPy, or JAX. One of the domains that also captures a lot of attention in the community is sparse computing. In this talk, we will present the current landscape of sparse computing in the Python ecosystem and our efforts to revive/expand it. Our main contributions to the Python ecosystem cover: (1) making a novel Finch sparse tensor compiler and Galley scheduler available for the community, (2) standardizing various aspects of sparse computing. We will show how to use the Finch compiler with the PyData/Sparse package and how it outperforms well-established alternatives for multiple kernels, such as MTTKRP or SDDMM. Real-world use-cases will show you how, step-by-step, Python practitioners can migrate their code to an Array API compatible version and benefit from tensor operator fusion and autoscheduling capabilities offered by the Finch compiler. Apart from the existing Julia implementation, the number of sparse backends offered by PyData/Sparse will grow in the future to provide a Python-native alternatives for scipy.sparse and Numba solutions. One of them that is currently under development is finch-tensor-lite, a pure Python rewrite of Finch.jl compiler, meant to make the solution lightweight by dropping Julia runtime dependency while providing the majority of features. pydataglobal2025-79476-pydata-sparse-finch-extending-sparse-computing-in-the-python-ecosystem General Track Mateusz SokółWillow Marie Ahrens en In this talk we're going to understand the current landscape of sparse computing in the Python ecosystem first. Then a high-level overview of the Finch technology and compiler's architecture will be presented together with other solutions vital for the project: Array API Standard and binsparse format. Next, we're going to present a selected set of benchmarks - also focusing on real world use-cases: how Finch impacts users' experience when writing sparse programs in Python. Last but not least a showcase of the current development will be shown - pure Python rewrite of Finch compiler. false Finch Tensor website https://cfp.pydata.org/pydataglobal2025/talk/B3QRQA/ https://cfp.pydata.org/pydataglobal2025/talk/B3QRQA/feedback/ General Track Talk 2025-12-10T13:00:00+00:00 13:00 00:30 Many notable PyData projects including Jupyter Hub, Matplotlib and JAX follow a versioning scheme called EffVer, where instead of making promises around backward compatibility they communicate the likelihood and magnitude of the work required to adopt a new version. In this talk we will dive into EffVer, what it is and what it means for developers and users. We will discuss how to apply EffVer to your own projects and how to depend on projects that use it. pydataglobal2025-78722-effver-versioning-code-by-the-effort-required-to-upgrade General Track Jacob Tomlinson en Intended Effort Versioning (EffVer), the version scheme where you just tell your users what order of magnitude to expect the upgrade effort to be. Version numbers are hard to get right. Semantic Versioning (SemVer) communicates backward compatibility via version numbers which often lead to a false sense of security and broken promises. Calendar Versioning (CalVer) sits at the other extreme of communicating almost no useful information at all. Many Python projects follow a looser scheme called EffVer where instead of making promises around backward compatibility they communicate the likelihood and magnitude of work required to adopt a new version. false https://cfp.pydata.org/pydataglobal2025/talk/NMYJM8/ https://cfp.pydata.org/pydataglobal2025/talk/NMYJM8/feedback/ General Track Tutorial 2025-12-10T13:30:00+00:00 13:30 01:30 As datasets grow, I/O becomes a primary bottleneck, slowing down scientific computing and data analysis. This tutorial provides a hands-on introduction to Blosc2, a powerful meta-compressor designed to turn I/O-bound workflows into CPU-bound ones. We will move beyond basic compression and explore how to structure data for high-performance computation. Participants will learn to use the python-blosc2 library to compress and decompress data with various codecs and filters, optimizing for speed and ratio. The core of the tutorial will focus on the Blosc2 NDArray object, a chunked, N-dimensional array that lives on disk or in memory. Through a series of interactive exercises, you will learn how to perform out-of-core mathematical operations and analytics directly on compressed arrays, effectively handling datasets larger than available RAM. We will also cover practical topics like data storage backends, two-level partitioning for faster data slicing, and how to integrate Blosc2 into existing NumPy-based workflows. You will leave this session with the practical skills needed to significantly accelerate your data pipelines and manage massive datasets with ease. pydataglobal2025-78708-hands-on-with-blosc2-accelerating-your-python-data-workflows General Track Francesc AltedLuke Shaw en ## Audience & Prerequisites This tutorial is for data scientists, engineers, and researchers who work with large numerical datasets in Python. Prerequisites: Attendees should have intermediate Python programming skills and be comfortable with the basics of NumPy arrays. No prior experience with Blosc2 is necessary. Setup: Participants will need a laptop and can follow along using a provided cloud-based environment (e.g., Binder) or a local installation of Python, Jupyter, and the python-blosc2 library. ## Learning Objectives By the end of this tutorial, attendees will be able to: * Understand the core concepts behind the Blosc2 meta-compressor. * Compress and decompress NumPy arrays, tuning parameters for optimal performance. * Create, manipulate, and slice Blosc2 NDArray objects for out-of-core processing. * Perform efficient mathematical computations directly on compressed data. * Store and retrieve compressed datasets using different storage backends. * Integrate Blosc2 into their existing data analysis workflows to mitigate I/O bottlenecks. ## Outline (90 minutes) ### Introduction & Setup (10 mins) * The I/O Bottleneck Problem. * Core Concepts: What are meta-compressors, chunks, and blocks? * Tutorial environment setup (Jupyter notebooks). ### Part 1: Compression Fundamentals (20 mins) * Hands-on: Using blosc2.compress() and blosc2.decompress(). * Exploring codecs (lz4, zstd), compression levels, and filters (shuffle, bitshuffle). * Exercise: Compressing a sample dataset and analyzing the trade-offs between speed and ratio. ### Part 2: The NDArray - Computing on Compressed Data (35 mins) * Hands-on: Creating NDArray objects from scratch and from NumPy arrays. * Storing arrays on-disk vs. in-memory. * Exercise: Slicing and accessing data from an on-disk NDArray. * Performing mathematical operations (arr * 2 + 1) and reductions (arr.sum()) on compressed data. * Exercise: Analyzing a dataset larger than RAM. ### Part 3: Advanced Features & Integration (20 mins) * Hands-on: Using two-level partitioning (meta-chunks) for faster slicing. * Brief overview of Caterva2 for sharing compressed data via an API. * Recap and Q&A. Repository: Tutorial materials including notebooks and datasets will be available at a public GitHub repository (link to be provided upon acceptance). false https://cfp.pydata.org/pydataglobal2025/talk/UXHBEZ/ https://cfp.pydata.org/pydataglobal2025/talk/UXHBEZ/feedback/ General Track Talk 2025-12-10T16:00:00+00:00 16:00 00:30 Machine learning teams today are drowning in massive volumes of raw, redundant data that inflate training costs, slow down experimentation, and degrade model quality. The core architectural flaw is that we apply control too late, after the data has already been moved into centralized stores or training clusters, creating waste, instability, and long iteration cycles. What if we could fix this problem right at the source? In this talk, we’ll discuss a playbook for shifting ML data filtering, transformation, and governance upstream, directly where data is generated. We’ll walk through a declarative, policy-as-code framework for building distributed pipelines that intelligently discard noise, balance datasets, and enrich signals before they ever reach your model training infrastructure. Drawing from real-world ML workflows, we’ll show how this “upstream control” approach can reduce dataset size, cut model onboarding time in half, and embed reproducibility and compliance directly into the ML lifecycle rather than patching them in afterward. Attendees will leave with: - A mental model for analyzing and optimizing the ML data supply chain. - An understanding of tools for declarative, source-level ML data controls. - Actionable strategies to accelerate iteration, lower training costs, and improve model outcomes. pydataglobal2025-86007-keynote-david-aronchick-from-pandas-to-policy-as-code-the-future-of-ml-data-engineering General Track David Aronchick en For over a decade, the Python ecosystem has given us a powerful arsenal to tame data. We started with the interactive magic of Pandas on a single machine, a revolutionary step that made complex analysis accessible. When our ambitions (and data) outgrew our laptops, we turned to Dask and Spark to scale our computations across clusters. More recently, projects like Apache Arrow began solving the critical problem of creating a standardized, efficient language for these distributed systems to speak. Each step in this journey solved a painful bottleneck. Yet, in our success, we've created a new one: the runaway cost and complexity of the "ingest-it-all-first" paradigm. Our cloud bills have become a tax on raw, unfiltered data, and our elegant downstream tools—from Airflow and dbt to our own ML models—are forced to waste expensive cycles sifting through noise just to find the signal. This talk argues for the next logical step in our stack's evolution: an Upstream Data Control Plane. We'll explore an playbook for applying intelligent filtering, transformation, and governance before data ever hits your expensive lakehouse. Just as Dask parallelized our processing and Arrow standardized our memory, this approach optimizes our data in motion, ensuring that our powerful downstream systems operate only on the high-value signals we care about. Join us to learn a declarative, policy-as-code framework that makes your entire data stack cheaper, faster, and more resilient. false https://cfp.pydata.org/pydataglobal2025/talk/NKQFBQ/ https://cfp.pydata.org/pydataglobal2025/talk/NKQFBQ/feedback/ General Track Tutorial 2025-12-10T16:30:00+00:00 16:30 01:30 Polars is a lightning fast DataFrame library that is taking the data science community by storm. Its elegant and expressive API makes analyses pleasant to write and efficient to run. In this workshop, we’ll demonstrate how Polars enables data scientists to go from raw data to reports–by reading, transforming, and visualizing data. pydataglobal2025-78694-python-polars-the-definitive-crash-course General Track Jeroen Janssens en Based on the book Python Polars: The Definitive Guide, we’ll teach the essentials of Polars to read, transform, and visualize data. While a hallmark of Polars is its speed, we’ll emphasize the benefits of its expression system for writing flexible, maintainable code. This hands-on workshop will cover: * Reading data from CSV, spreadsheets, Parquet, and databases * Common transformations such as selecting, filtering, sorting, and aggregating * Complex data types, including text, time, and nested structures * Expressions, the building blocks of every query * Visualizing data By the end of this workshop, attendees will have gained a solid understanding of Polars, and be equipped to start applying this lightning fast DataFrame library to their own datasets. No prior knowledge of Polars is required. false https://cfp.pydata.org/pydataglobal2025/talk/BSY9GA/ https://cfp.pydata.org/pydataglobal2025/talk/BSY9GA/feedback/ General Track Tutorial 2025-12-10T18:00:00+00:00 18:00 01:30 The complex nervous system provides a repertoire of evolutionary properties like neuron spiking, bursting, and chaos that are yet to be fully understood. One approach is to tackle these time-dependent properties using the technique of "dynamical systems”, such as ordinary differential equations. Since the popular work by Hodgkin and Huxley, many dynamical systems models of neurons have been proposed, of which FitzHugh–Nagumo and Morris–Lecar models draw special attention. The nervous system is made of a network of neurons, possessing a complex structural and functional topology. This topology is a function of different parameters, among which the coupling strength plays a major role. Our focus would be to systematically study the effect of various coupling strategies on the firing patterns exhibited by a collection of neurons. In this workshop, my goal is to popularize a reduced-order model of neuron dynamics known as the “denatured Morris–Lecar” system and to teach how Python can be efficiently used to perform research on time series analysis of coupled neurons. pydataglobal2025-78145-time-series-analysis-for-coupled-neurons General Track Indranil Ghosh en This is a tutorial on hands-on time series analysis of coupled neuron models. We will build mathematical models of coupled neurons, and then utilize tools from nonlinear dynamics to analyze simulated time series. We will discuss various empirically informed coupling strategies and statistically efficient time series measures. This workshop is 100% Jupyter notebook and will have room to openly brainstorm ideas to extend and improve the studies. Let’s unravel some complex dynamics together! The pipeline of this tutorial will be the following: (i) Start by building a coupled neuron system based on different coupling strategies, (ii) Simulate the system and generate time series data, (iii) Perform time series analysis by computing various metrics from the nonlinear dynamics literature, (iv) Finally, discuss what these metrics tell us about the temporal behavior of neurons. Coupling strategies we are going to look at: (i) Gap junction coupling (ii) Chemical coupling (iii) A hybrid coupling influenced by a superconductor model in physics (iv) Electromagnetic coupling (iv) Coupling, which is not pairwise but higher-order (A bit of background on graph theory is recommended) (v) A random coupling strategy We will implement the following methodologies/algorithms for time series analysis of coupled neuron models: (i) Hurst exponent: measuring persistence of time series, (ii) Sample entropy: measuring the complexity of time series, (iii) 0–1 test: measuring chaos, (iv) Kuramoto order-parameter: measuring synchrony between the neurons. This tutorial is 100% Python. And I will be utilizing Jupyter Notebooks to deliver the workshop. Packages that need to be downloaded beforehand are: (i) `matplotlib` for plotting, (ii) `numpy` and `scipy` for scientific computations, (iii) `nolds` for nonlinear measure for dynamical systems, (iv) `pandas` for data handling. The audience would find this interesting because it would be a hands-on introduction to how the mechanisms of neurons can be explored using different tools from the nonlinear dynamics literature. Mathematically modelling the dynamics of neurons has attracted several researchers in recent years because of the popularity of artificial intelligence. This field of neuron dynamics is booming, and delivering this workshop would be timely. I would also ensure to leave some room for brainstorming further ideas with the audience and how this study could be potentially extended and improved, thus an interactive session. The goal is to attract applied mathematicians, computer scientists, data scientists, engineers, and statisticians alike and provide them with a battery of tools to add to their knowledge base. The audience would then be able to apply these tools in domains other than neurodynamics, for example, climate, finance, or social science. The only technical background I would expect from the audience is familiarity with `matplotlib`, `numpy` and `pandas`, and some basic statistics (regression, correlation coefficient), linear algebra (matrix operations), and graphs (as in networks). After the tutorial, the audience will leave with a newly built insight into the mathematical modeling of neuron dynamics. Here is the breakdown of the tutorial: 0–15 mins: Introduction to neurons as dynamical systems and why we care about their behavior over time. We will talk about a single neuron's behavior and the selection of a mathematical model. We will also talk about the bursting phenomenon in neurons. 15-30 mins: We will then mathematically model a coupled system of neurons. We will cover the topic of “small networks” of neurons and what they teach us about the bigger picture: a complex, connected nervous system. 30-45 mins: Next, we will introduce various empirically informed coupling mechanisms. We will talk about how these couplings incorporate different firing patterns in the coupled neurons, ranging from regular behavior to chaotic firing. 45-75 mins: Finally, I will introduce time series analysis of neuron data. We will then implement the algorithms mentioned above to realize different dynamical properties of the neurons. 75-90 mins: Open the room to QA and brainstorm further ideas to improve/extend the analysis of neuron-time series data. All materials for the tutorial can be accessed via this repository link: https://github.com/indrag49/PyData-Global-Tutorial-2025 false https://cfp.pydata.org/pydataglobal2025/talk/N7EAFM/ https://cfp.pydata.org/pydataglobal2025/talk/N7EAFM/feedback/ Machine Learning & AI Talk 2025-12-10T11:30:00+00:00 11:30 00:30 Advanced statistics are transforming sports analysis, but many coaches and ex-players struggle to access meaningful insights due to complex data and jargon. Generative AI offers a solution. In this talk, I’ll demonstrate how I used Model Context Protocol (MCP) to turn Anthropic’s Claude Desktop into a football opposition analyst, making advanced performance data accessible and actionable. Topics include how MCP enables AI to interpret domain-specific knowledge and real examples of AI-generated football insights. pydataglobal2025-79349-using-mcp-to-turn-claude-into-a-football-opposition-analyst Machine Learning & AI Adam Cowley en Analysis in sports is changing. Advanced statistics like Wins Above Replacement (WAR) or Expected Goals (xG) are making their way into TV punditry and conversations in bars. But the people who need the information the most, ex-professionals and coaches without a background in statistics, often shun it. Not because they don't see the value, but because the language is impenetrable, the underlying data is overwhelming, and the insights are difficult to translate. Generative AI provides an opportunity to bridge the gap. In this talk, I'll share how I used Model Context Protocol (MCP) to turn Anthropic's Claude Desktop into a football opposition analyst by providing access to team and player performance event data, and in turn lower the barriers so anyone can turn a sea of numbers into actions. This talk will cover: - How MCP enables AI to access and interpret domain-specific knowledge - Real examples of AI-generated football insights in action false https://cfp.pydata.org/pydataglobal2025/talk/ETQTHC/ https://cfp.pydata.org/pydataglobal2025/talk/ETQTHC/feedback/ Machine Learning & AI Talk 2025-12-10T12:00:00+00:00 12:00 00:30 Building great AI-driven products starts with empowered teams. Hear proven strategies for leading, mentoring, and growing distributed engineering teams, with lessons in innovation, compliance, and diversity from global digital enterprises. pydataglobal2025-78515-the-human-side-leading-and-mentoring-global-data-teams-in-the-age-of-ai Machine Learning & AI amar naik en For engineering leaders, managers, and aspiring mentors. Session covers structures for remote work, upskilling, cross-cultural collaboration, promoting innovation, and embedding compliance and ethics in technical work—from real executive experience. If you would like hands-on tutorials for any of the 30-minute talks, or wish to tailor for a specific audience (engineering, product, executive), content can be customized to fit workshop/intermediate/advanced levels. false https://cfp.pydata.org/pydataglobal2025/talk/EKX7LV/ https://cfp.pydata.org/pydataglobal2025/talk/EKX7LV/feedback/ Machine Learning & AI Talk 2025-12-10T12:30:00+00:00 12:30 00:30 Building ML models for financial fraud detection sounds straightforward, until you have to evaluate, validate, and deploy them in real-world pipelines. This talk walks through the practical stack, metrics, and mindsets needed to build fraud detection systems with modern Python. We'll cover key challenges like concept drift, extreme class imbalance, false-positive overload, and why the usual ML workflows fall short. Along the way, we’ll explore a real-world architecture using classical ML, deep learning, and GNNs, plus the validation techniques and production patterns that make or break fraud systems. If you're tired of toy problems and want patterns that survive real money and real latency, this talk’s for you. pydataglobal2025-78692-realtime-financial-fraud-detection-with-modern-python Machine Learning & AI César Soto Valero en This talk distills a production‑tested path for real‑time financial fraud detection in Python (inc. choosing the right objective, validating in time, and shipping with guardrails). Core idea: Optimize the business decision (alerts under cost/latency constraints), not just the ML score. Outline (30 minutes): 1. Problem framing: Adversaries, label delay, extreme imbalance, and why “accuracy” lies. 2. Metrics that matter: Precision and recall, AUC‑PR vs ROC, cost‑weighted utility, calibration for decisions. 3. Validation done right: Temporal splits, rolling/blocked CV with gap, prequential test‑then‑train, leak and drift traps. 4. Modeling under latency budgets: Where XGBoost shines, when to add tabular DL, injecting graph signals without blowing latency (simple handcrafted graph stats + GNNs). 5. From notebook to service: Small, testable core, FastAPI endpoint, thresholds and shadow mode, alert quotas, analyst feedback loops. 6. Operations & monitoring: Drift indicators, calibration checks, label‑delay dashboards, canaries/rollbacks. 7. Wrap‑up/Q&A: Failure modes and a 1‑page runbook. Attendee outcomes: - A copy‑and‑adapt roadmap for deploying financial fraud detection services with Python. - A latency‑aware model selection heuristic. - A minimal deployment pattern (service, thresholds, monitoring) that scales from pilot to production. Prior knowledge expected: - Basic Python and DataFrames, ML classification basics, HTTP/JSON. false https://cfp.pydata.org/pydataglobal2025/talk/J7JK79/ https://cfp.pydata.org/pydataglobal2025/talk/J7JK79/feedback/ Machine Learning & AI Talk 2025-12-10T13:00:00+00:00 13:00 00:30 Text embeddings are a powerful tool for encoding the essence of unstructured text data into a structured, dense, multidimensional vector representation. Due to their inner structure, tree based models such as decision trees, gradient boosted decision trees and random forests struggle to effectively use text embeddings features. This is due to the fact that trees can use only one feature every time they split, so the number of used embedding dimensions is limited to the tree depth. Other models, such as linear models for example, can use text embeddings more effectively because they are able to use all of the embedding dimensions simultaneously. In this presentation we will present a novel approach to transform text embedding features into a format that tree-based models can effectively use. The proposed approach combines the strengths of non-tree based models with predictive power of tree based models to create a more effective feature representation for tree-based models. pydataglobal2025-78489-how-to-effectively-use-text-embeddings-in-tree-based-models Machine Learning & AI Claudio Salvatore Arcidiacono en The presentation is aimed at Data Science and Machine Learning practitioners who are already familiar with tree-based models and want to learn how to effectively incorporate text embeddings features to boost the performances of their models. The methodology showcased in the presentation is available in the sklearo open source package. The structure of the talk will be as follows: - **5 minutes** Overview of text embeddings, how tree-based models are built, and the challenges they face with text embeddings compared to linear models. - **5 minutes** Explanation of how can we leverage non-tree based models to transform text embeddings into a format that tree based models can effectively use. - **5 minutes** Explanation on *cross-fitting*, a technique used to avoid target leakage when generating features using the target variable. - **5 minutes** Code examples of how this technique can be used in practice using the `sklearo` open source library. - **5 minutes** Performance comparison of tree based models using text embeddings as-is vs using the transformed features. Prior knowledge about fundamental machine learning concepts such as overfitting, cross-validation, and feature engineering is recommended but not required. false https://cfp.pydata.org/pydataglobal2025/talk/W9RJKW/ https://cfp.pydata.org/pydataglobal2025/talk/W9RJKW/feedback/ Machine Learning & AI Talk 2025-12-10T13:30:00+00:00 13:30 00:30 In many regulated industries—finance, healthcare, insurance—logistic regression remains the model of choice for its interpretability and regulatory acceptability. Yet capturing non-linear effects and interactions often requires variable binning, and naive approaches (equal-width or quantile cuts) can either wash out signal or invite overfitting. In this 30-minute session, data scientists and risk analysts with a working knowledge of logistic regression and Python will learn to: -Diagnose the weaknesses of basic binning strategies. -Select and apply optimal-binning algorithms for different use cases. -Assess bin stability and guard against model overfit. All code, data samples, and a notebook will be available on GitHub. pydataglobal2025-79251-optimal-variable-binning-in-logistic-regression Machine Learning & AI Charaf ZGUIOUAR en Despite the rise of complex “black-box” models, regulated environments still demand transparency. Properly binned variables not only improve model fit but also yield coefficients that the business and auditors can interpret. However, determining cut-points that preserve true signal while avoiding data-snooping bias is non-trivial. By the end of this session, attendees will be able to: - Understand the basic idea behind binning (the what) - To know in which contexts variable binning makes sense (the when and why). - Choose among popular optimal-binning techniques (e.g., ChiMerge, MDLP, decision-tree-based) based on data size, feature type, and operational constraints (the how). Who Should Attend? Data scientists and risk analysts who use logistic regression in regulated settings and need a reproducible, explainable feature-engineering pipeline. Detailed 30-Minute Agenda | Time | Topic | | --- | --- | | 0–3 min | Context & Why Binning Matters in explainibility| | 3–8 min | Pitfalls of Naïve Binning (examples from real-life) | | 8–18 min | Binning as an optimization problem : Algorithms & Decision Criteria | | 18–26 min | Hands-On Python Demo: From Data to Defensible Bins | | 26–30 min | Q&A, Resources & Next Steps | Prerequisites & Materials - Prerequisites: Basic Python (pandas, scikit-learn) and logistic-regression familiarity - Materials: GitHub repo with notebook, data samples, will be shared during the talk You’ll leave equipped to choose the right optimal‐binning algorithm for your data. false https://cfp.pydata.org/pydataglobal2025/talk/BT7M3S/ https://cfp.pydata.org/pydataglobal2025/talk/BT7M3S/feedback/ Machine Learning & AI Talk 2025-12-10T14:00:00+00:00 14:00 00:30 Retrieval-Augmented Generation (RAG) systems are transforming how we interact with unstructured data using Large Language Models (LLMs). While it’s now relatively easy to stand up a basic RAG prototype, deploying a robust, customizable, and production-ready system remains challenging. In this talk, we present our open-source RAG blueprint through the lens of a real-world application: Bundestag Chat—a system that enables users to explore and converse with German parliamentary speeches. We’ll demonstrate how the blueprint streamlined development and scaling, and how its modular architecture allowed for seamless integration of components like LlamaIndex, Hugging Face embeddings, PGVector, Langfuse, and Ragas. Attendees will walk away with practical insights into customizing RAG pipelines for real use cases, whether building internal tools or user-facing applications. We’ll also explore build-vs-buy trade-offs, retrieval and scaling strategies, and considerations around privacy, evaluation, and monitoring. pydataglobal2025-78703-bundestag-chat-discovering-political-landscape-with-rag-systems Machine Learning & AI Piotr KalotaMatthias Boeck en Retrieval-Augmented Generation (RAG) systems are among the most impactful applications of LLMs, allowing for intelligent querying and contextual understanding of unstructured data. However, turning a prototype into a polished, scalable product is often where complexity sets in. In this talk, we walk through how our open-source RAG blueprint was used to create *Bundestag Chat*—a system that allows users to interact with over a decade of German parliamentary debates via a chat interface. This real-world use case illustrates the key benefits of our blueprint: modularity, observability, evaluation, and scalability. Our architecture includes: - **LlamaIndex** for document parsing and chunking, - **Hugging Face embedding models** stored in a **PGVector** vector database, - **Chainlit** for an intuitive chat UI, - **Langfuse** for logging, observability, and feedback collection, - **Ragas** for evaluating response quality across dimensions like faithfulness and relevance. What made this system successful was the flexibility to swap components, configure data flows, and monitor performance from day one. This modular design made it straightforward to go from an initial prototype to a system deployed in a privacy-sensitive environment. We’ll also contrast open-source and commercial RAG stacks, sharing insights on when to build versus buy. Topics include: - Estimating system requirements across different workloads, - Evaluating model performance and output reliability, - Ensuring data privacy and legal compliance, - Gathering and acting on human feedback to improve quality. false https://cfp.pydata.org/pydataglobal2025/talk/YPRZBE/ https://cfp.pydata.org/pydataglobal2025/talk/YPRZBE/feedback/ Machine Learning & AI Talk 2025-12-10T16:00:00+00:00 16:00 00:30 Academic research is often fragmented across dense PDFs, complex jargon, and scattered media articles, making it hard to access for students, interns, and the broader public. To address this, we introduce **Lab Lens**: an open-source Research AI Assistant that unifies a lab’s papers and media coverage into a conversational system, where anyone can ask natural language questions and receive structured answers with full source citations. This talk demonstrates how to build and deploy a production-ready RAG pipeline that uses Landing.AI for vision-based PDF parsing, Firecrawl for media extraction, and LangGraph for agentic orchestration. The entire system is containerized with FastAPI and Streamlit, launching with a single command: docker compose up. Attendees will learn how to turn scattered research artifacts into a transparent, queryable knowledge base, making lab insights accessible, reproducible, and conversational for all. pydataglobal2025-78626-building-production-ready-research-ai-assistants-with-one-command-setup Machine Learning & AI /media/pydataglobal2025/submissions/GS9GQP/system_design_aE48_PHvigwv.webp Cainã Max Couto da Silva en In this talk, we introduce Lab Lens: an open-source framework for Research AI Assistant that allows labs to ingest scientific papers and media coverage, build a vector database, and query it via natural language—all in one reproducible command. This 30-minute talk will explore: - **🧠 Architecture:** How LangGraph, FastAPI, and Streamlit are combined with agentic reasoning for document Q&A. - **📄 Multi-modal Ingestion:** How Lab Lens uses Landing.AI (vision agentic document extraction) and Firecrawl to intelligently extract content from complex PDFs and dynamic media pages. - **🤖 LLM Workflow:** How intents are classified, documents retrieved, and responses synthesized with structured JSON output and source attribution. - **🔄 Reusability and Extensibility:** How any lab or research group can plug in their own documents and deploy in minutes. - **⚙️ One-Line Setup:** How a single YAML config and docker compose up sets up ingestion, vectorization, API, UI, and Slack bot integration. We'll conclude with a live demo showing how Lab Lens answers real research questions using citation-backed reasoning, emphasizing transparency, reliability, and ease of use. Lab Lens is designed for reproducibility, minimal setup, and immediate utility. If you're interested in bringing GenAI to your research workflow—or your research to the world—this talk will show you exactly how. **Target Audience:** Researchers, students, and enthusiasts wanting practical AI tools. **Prerequisites:** - Python knowledge - Familiarity with containerization concepts. **Resources Provided:** Complete open-source codebase with Docker configuration for immediate deployment. Remember that the main goal/advantage here is to make it accessible for the whole lab's documents (papers and media coverage), so anyone can ask about it with a source citation. false https://cfp.pydata.org/pydataglobal2025/talk/GS9GQP/ https://cfp.pydata.org/pydataglobal2025/talk/GS9GQP/feedback/ Machine Learning & AI Talk 2025-12-10T16:30:00+00:00 16:30 00:30 The proliferation of AI/ML workloads across commercial enterprises, necessitates robust mechanisms to track, inspect and analyze their use of on-prem/cloud infrastructure. To that end, effective insights are crucial for optimizing cloud resource allocation with increasing workload demand, while mitigating cloud infrastructure costs and promoting operational stability. This talk will outline an approach to systematically monitor, inspect and analyze AI/ML workloads’ properties like runtime, resource demand/utilization and cost attribution tags . By implementing granular inspection across multi-player teams and projects, organizations can gain actionable insights into resource bottlenecks, identify opportunities for cost savings, and enable AI/ML platform engineers to directly attribute infrastructure costs to specific workloads. Cost attribution of infrastructure usage by AI/ML workloads focuses on key metrics such as compute node group information, cpu usage seconds, data transfer, gpu allocation , memory and ephemeral storage utilization. It enables platform administrators to identify competing workloads which lead to diminishing ROI. Answering questions from data scientists like "Why did my workload run for 6 hours today, when it took only 2 hours yesterday" or "Why did my workload start 3 hours behind schedule?" also becomes easier. Through our work on Metaflow, we will showcase how we built a comprehensive framework for transparent usage reporting, cost attribution, performance optimization, and strategic planning for future AI/ML initiatives. Metaflow is a human centric python library that enables seamless scaling and management of AI/ML projects. Ultimately, a well-defined usage tracking system empowers organizations to maximize the return on investment from their AI/ML endeavors while maintaining budgetary control and operational efficiency. Platform engineers and administrators will be able to gain insights into the following operational aspects of supporting a battle hardened ML Platform: 1.Optimize resource allocation: Understand consumption patterns to right-size clusters and allocate resources more efficiently, reducing idle time and preventing bottlenecks. 2. Proactively manage capacity: Forecast future resource needs based on historical usage trends, ensuring the infrastructure can scale effectively with increasing workload demand. 3. Facilitate strategic planning: Make informed decisions regarding future infrastructure investments and scaling strategies. 4.Diagnose workload execution delays: Identify resource contention, queuing issues, or insufficient capacity leading to delayed workload starts. Data Scientists on the other hand will gain clarity on factors that influence workload performance. Tuning them can lead to efficiencies in runtime and associated cost profiles. pydataglobal2025-78727-optimizing-ai-ml-workloads-resource-management-and-cost-attribution Machine Learning & AI Saurabh Garg en This abstract proposes a framework for systematically monitoring and analyzing AI/ML workloads to optimize resource utilization and effective cost attribution/management. By providing granular insights into resource consumption, the system helps identify cloud infra bottlenecks - leading to lower resource contention while promoting fairer use of resources. Built on Metaflow, this approach enables transparent usage reporting, improved performance, and strategic planning for future AI/ML initiatives. Ultimately, it empowers organizations to maximize ROI from their AI/ML investments while maintaining budgetary control and operational efficiency for both platform engineers and data scientists. false https://cfp.pydata.org/pydataglobal2025/talk/TXYJHL/ https://cfp.pydata.org/pydataglobal2025/talk/TXYJHL/feedback/ Machine Learning & AI Talk 2025-12-10T17:00:00+00:00 17:00 00:30 Ever wonder why structured LLM output doesn’t feel as reliable as its natural language responses? At Khan Academy, we asked ourselves the same thing—especially as we leaned heavily on JSON-based structured outputs to power our AI tutor, Khanmigo. Surprisingly, the root of the problem often lies in one of the most familiar tools in a Python developer’s toolbox: the humble `dict`. In this talk, we follow the story of how dictionary ordering can shape (and sometimes distort) structured LLM output. We’ll walk through how different frameworks—OpenAI, Claude, LangChain, OpenRouter, vLLM—handle structured responses, and why those differences matter more than you’d expect. Along the way, we’ll share practical best practices we’ve developed to improve structured output reliability, observe subtle failure cases, and debug weird edge behaviors. If you’re building LLM apps with structured output, you’ll leave with concrete tips—and a deeper appreciation for the details that make or break your system. pydataglobal2025-79296-let-me-structure-freely-how-to-improve-llm-structured-output-quality Machine Learning & AI Boris Lau en Structured output (like JSON) is increasingly used in LLM applications to enforce a predictable schema and simplify downstream parsing. However, developers often assume that structured output is deterministic and robust—until they run into subtle bugs. At Khan Academy, we’ve run Khanmigo on structured JSON output since before it was even a supported feature. Along the way, we’ve learned a lot about where things can go wrong. Our investigation began when we noticed inconsistent output quality across different LLM frameworks, even with identical prompts and models. The culprit? Python dictionary ordering and how different frameworks serialize JSON schemas. We'll explore: * How Python's evolution from unordered (pre-3.7) to insertion-ordered dictionaries affects LLM frameworks, or how it lingers through other frameworks in (post-3.7) * Framework-specific serialization behaviors in OpenAI SDK, Anthropic SDK, LangChain, OpenRouter, and vLLM * Measurable impact on output quality through A/B testing results Attendees should have basic familiarity with Python and JSON, but no deep LLM expertise is required. We'll explain technical concepts clearly while providing actionable insights for immediate application. false https://cfp.pydata.org/pydataglobal2025/talk/YBZLZK/ https://cfp.pydata.org/pydataglobal2025/talk/YBZLZK/feedback/ Machine Learning & AI Talk 2025-12-10T17:30:00+00:00 17:30 00:30 Tired of paying for cloud compute just to view your own data? Discover how to build a completely free, open-source personal data warehouse that runs entirely on your machine. – Import data from Excel, CSV, SQL Server, and Microsoft Fabric – Use AI-powered Python/C# code for advanced data transformations – Generate SSRS-style reports – no cloud required – Leverage local compute power to avoid cloud costs pydataglobal2025-79208-build-your-own-personal-data-warehouse Machine Learning & AI /media/pydataglobal2025/submissions/SFG8MV/443775241-b1fd8715-_9IxJogQ.png Michael Alan Washington en Typically, a data warehouse operates in the cloud. When you access your data—even just to view it—you incur compute costs (in other words, you pay). But you already have a computer with a CPU and memory capable of handling most tasks. Wouldn’t it be great to view, edit, and transform your data right on your own machine? Now you can! This free open source application allows you to import your data, transform it using AI to create python code to perform calculations, and report and export the results. In this talk, Microsoft MVP Michael Washington shows how to: – Import data from Excel, CSV, SQL Server, and Microsoft Fabric – Use AI-powered Python/C# code for advanced data transformations – Generate SSRS-style reports – no cloud required – Leverage local compute power to avoid cloud costs Whether you’re a developer, analyst, or data enthusiast, this session will help you take full control of your data with zero hosting fees. Live demos included! false https://cfp.pydata.org/pydataglobal2025/talk/SFG8MV/ https://cfp.pydata.org/pydataglobal2025/talk/SFG8MV/feedback/ Machine Learning & AI Talk 2025-12-10T18:00:00+00:00 18:00 00:30 LLMs have a lot of hype around them these days. Let's demystify how they work and see how we can put them in context for data science use. As data scientists, we want to make sure our results are inspectable, reliable, reproducible, and replicable. We already have many tools to help us in this front. However, LLMs provide a new challenge; we may not always be given the same results back from a query. This means trying to work out areas where LLMs excel in, and use those behaviors in our data science artifacts. This talk will introduce you to LLms, the Chatlas package, and how they can be integrated into a Shiny to create an AI-powered dashboard. We'll see how we can leverage the tasks LLMs are good at to better our data science products. pydataglobal2025-79565-llms-chatbots-and-dashboards-visualize-your-data-with-natural-language Machine Learning & AI Daniel Chen en This talk plans to provide data scientists the tools and techniques needed to integrate AI into their data products. Specifically around how to use APIs to work with chat providers and show where and how we can leverage tasks LLMs are good at to make sure we are confident with their output. Talk breakdown: 0-5: introduction and where we can push LLMs 5-10: Example of tasks where the LLM can do well, and where can fail (in a data science context) 10-15: brief introduction to the Chatlas package 15-20: brief introduction on Shiny dashboards and integrating chatlast into Shiny 20-25: demo + example of putting everything together and how we can create an LLM-powered data science product. 25-30: Q+A / overflow false https://cfp.pydata.org/pydataglobal2025/talk/ARAZTG/ https://cfp.pydata.org/pydataglobal2025/talk/ARAZTG/feedback/ Machine Learning & AI Talk 2025-12-10T18:30:00+00:00 18:30 00:30 As LLMs become increasingly embedded in critical applications across healthcare, legal, and financial domains, their tendency to generate plausible-sounding but false information poses significant risks. This talk introduces UQLM, an open-source Python package for uncertainty-aware generation that flags likely hallucinations without requiring ground truth data. UQLM computes response-level confidence scores from token probabilities, consistency across sampled responses, LLM judges, and tunable ensembles. Attendees will learn practical strategies for implementing hallucination detection in production systems and leave with code examples they can immediately apply to improve the reliability of their LLM-powered applications. No prior uncertainty quantification background required. pydataglobal2025-78742-uqlm-detecting-llm-hallucinations-with-uncertainty-quantification-in-python Machine Learning & AI /media/pydataglobal2025/submissions/8U7WLS/uqlm_graphic_NwE6pkZ.png Dylan BouchardMohit Singh Chauhan en ### Objective. Show how to add uncertainty-aware controls to LLM apps using UQLM so practitioners can detect and handle hallucinations at generation time without ground truth data. ### Context and Gap. Many hallucination detection methods assume existence of ground truth data, which is rarely available in production. Research has proposed ground-truth-free uncertainty quantification (UQ) techniques, but adoption suffers from fragmented tooling. UQLM packages these methods behind a simple API and provides a versatile suite of UQ-based confidence scorers that work across tasks. ### What you will see. - Black-box UQ via response consistency from multiple samples - White-box UQ from token log probabilities - LLM-as-a-judge scoring - Ensemble tuning and threshold selection for your use case - Patterns for routing: block, warn, or escalate to human review ### Outline (30 minutes total). - 0–4: Why hallucinations matter in production - 4–8: Limits of traditional hallucination detection approaches and where UQ fits - 8–20: UQLM walkthrough and code examples - 20–24: Choosing thresholds and tuning ensembles - 24–27: Results on several use cases and interpreting confidence - 27–30: Q&A ### Expected background. Basic familiarity with LLMs and machine learning. No prior uncertainty quantification knowledge required. ### Key takeaways. - When and why ground-truth-free hallucination detection is useful in production - How to add UQLM to a Python app in a few lines of code - Pros and cons of consistency-based, token-probability-based, and judge-based methods - Practical guidance on thresholds, ensemble tuning, and handling low-confidence outputs false https://cfp.pydata.org/pydataglobal2025/talk/8U7WLS/ https://cfp.pydata.org/pydataglobal2025/talk/8U7WLS/feedback/ Analytics, Visualization & Decision Science Talk 2025-12-10T13:00:00+00:00 13:00 00:30 Survival analysis tackles one of the oldest and most universal questions in data science: Can we learn from the past when something will happen in the future? I will introduce you to the core concepts of survival analysis, visualize time-to-event datasets with python and R, and introduce pertinent probability distributions. Classical analysis methods for fitting such datasets - some developed long before the age of modern computing - will be confronted to machine-learning approaches. Along the way, surprising paradoxes and counterintuitive results will reveal why survival analysis is not merely a blend of regression and classification, but an important prediction problem of its own. pydataglobal2025-78538-reviving-survival-analysis-timeless-yet-overlooked Analytics, Visualization & Decision Science Malte Tichy en Since at least 1693, when the first actuarial tables were used for calculating insurance premiums, survival (or "time-to-event") analysis has been relevant for many disciplines. Whether predicting when a mechanical component will fail, when a patient will recover, or when a customer will return a product, survival analysis has applications in nearly every domain - from engineering and medicine to finance and e-commerce. Despite its broad applicability and deep statistical foundations, survival analysis remains underappreciated in modern data science. I therefore want to give the audience, who does not need to have heard of survival analysis before, an impression about what survival analysis is about, what one needs to be careful with, and which analytical and computational tools to use to get to reliable predictions. In a step-by-step constructive approach, I will slowly guide the audience from the simplest flavor of the fully observed time-to-event-problem to the more intricate versions that include censoring and truncation, in which managing one's own ignorance becomes the most important and challenging aspect. Numerous code examples in python and R will make the talk hands-on, and allow listeners to replicate the numerical experiments and visualizations. At the same time, I will constantly recur to lucid everyday-examples (what age should the house that you buy have so you avoid problems? how long can you use your winter tires on your car? why is milk often still good after the best-before date?) - and thereby hopefully convince the audience: Survival analysis is almost always everywhere. Outline: - Motivation: The oldest problem in data science? [1 min] - Introduction: Prediction problems that are in fact survival problems? [3 min] - The simple case: Fully observed datasets. Visualization of the cumulative failure distribution. [3 min] - The Weibull distribution as the working horse of survival analysis: How to model early failures, constant risks and wear-outs. [4 min] - Why reporting another case of illness can be good news. [2 min] - Censoring: What can we learn from not having observed anything yet? [2 min] - The Kaplan-Meier estimator and the maximum-likelihood principle. [5 min] - Machine Learning approaches to the survival problem. [3 min] - Outlook: Which degree of individualized survival forecasts can we expect in the future? [2 min] After the talk, the audience will be able to recognize the time-to-event problem in their own domain, and use the appropriate tools in python and R to analyze and model it. false https://cfp.pydata.org/pydataglobal2025/talk/ZS37FH/ https://cfp.pydata.org/pydataglobal2025/talk/ZS37FH/feedback/ Analytics, Visualization & Decision Science Talk 2025-12-10T14:00:00+00:00 14:00 00:30 Switch or stay, what do you say? And more importantly, why? The Monty Hall Problem is a well-known brain teaser from which we can learn important lessons in decision making that are useful in general and in particular for data scientists. If you are not familiar with this problem, prepare to be perplexed 🤯. If you are, I hope to shine light on aspects that you might not have considered 💡. I introduce the problem and solve with three types of intuitions: Common, Bayesian and Causal. I summarise with a discussion on lessons learnt for better data decision making. pydataglobal2025-78336-lessons-in-decision-making-from-the-monty-hall-problem Analytics, Visualization & Decision Science /media/pydataglobal2025/submissions/S7PC89/monty-hall-goat-lap_tIcFNgr.png Eyal Kazin en Imagine you're a contestant on a game show. Three doors stand before you: behind one is a prize car, behind the other two are goats. You choose a door, and the host—who knows what's behind each—reveals a goat behind one of the doors you didn’t pick. Now you're asked: "Do you want to switch your choice or stay?" This is the essence of the Monty Hall Problem, a classic puzzle that famously baffles our intuitions about probability. While it may seem like just a fun brain teaser, it offers profound lessons for decision-making under uncertainty. In this talk, we'll break down the Monty Hall Problem, explore its counterintuitive nature, and uncover what it teaches us about probabilistic reasoning and critical thinking. Together, we'll navigate multiple perspectives. Key Topics: * The Monty Hall Problem: Origins, setup, and why it confuses even experts * Misconceptions and cognitive biases: Why our gut reactions often lead us astray * Bayesian thinking: The power of belief updating in uncertain scenarios * Information theory: How the host's actions reveal hidden information * Causal reasoning: A fresh lens for understanding the game's dynamics * Real-world takeaways: Applying these lessons to practical decision-making By the end of this session, attendees will gain: * A clear understanding of the Monty Hall Problem and its solution * Insights into the pitfalls of intuitive probability judgments * Strategies for approaching complex decisions and probabilistic reasoning This session is for data scientists, analysts, and decision-makers at all experience levels. No advanced math is required—just curiosity and a willingness to rethink what you know about probability. Join me to discover how a seemingly trivial game show puzzle can sharpen your decision-making skills and elevate your approach to statistics, data science, and beyond. I have summarised this talk in this publication: [bit.ly/mh-lessons](https://bit.ly/mh-lessons). false https://cfp.pydata.org/pydataglobal2025/talk/S7PC89/ https://cfp.pydata.org/pydataglobal2025/talk/S7PC89/feedback/ Analytics, Visualization & Decision Science Talk 2025-12-10T16:00:00+00:00 16:00 00:30 We often must make decisions under uncertainty—should you carry an umbrella if there's a 30 % chance of rain? Bayesian decision theory provides a principled, probabilistic framework to answer such questions by combining beliefs (probabilities), utilities (what matters to us), and actions to maximize expected gain. This talk: - Introduces key decision‑theoretic concepts in intuitive terms. - Uses a toy umbrella example to ground ideas in relatable context. - Demonstrates applications in Bayesian optimization (PoI/EI) and Bayesian experimental design. - Is hands‑on—with Python code and practical tools—so participants leave ready to apply these ideas to real‑world problems. pydataglobal2025-77955-decisions-under-uncertainty-a-hands-on-guide-to-bayesian-decision-theory Analytics, Visualization & Decision Science Quan Nguyen en This talk bridges everyday decision-making (umbrella example) with advanced techniques like Bayesian optimization and experimental design, and equips attendees with conceptual clarity and immediate code they can adapt to their data-driven workflows. ## Audience Primarily data scientists, ML practitioners, and statisticians who: - Have applied Bayesian models but want a broader decision-theory perspective. - Want actionable insight into uncertainty-aware decision frameworks. - Seek practical demos in Python. ## Outline ### Motivation & Core Concepts (5 min) - Frame real-world decision problems: rain or shine, clinical trials, A/B testing. - Introduce Bayesian decision theory: beliefs × utilities → action via expected utility maximization. ### Toy Example: Should I Bring an Umbrella? (8 min) - Define: Probabilityp of rain; utility/loss matrix | Action | Rain | No Rain | | ----------- | ------------ | ------------------ | | Umbrella | –1 (weight) | –1 (inconvenience) | | No Umbrella | –10 (soaked) | 0 | - Derive expected utility: ``` EU_umbrella = -1 EU_no_umbrella = -10p ``` So bring umbrella if p > 0.1. - Interactive Python demo: explore how p and utility values shift the decision point. ### Bayesian Optimization: PoI & EI (8 min) - Introduce Gaussian-process-based optimization and the need to trade off exploration vs. exploitation. - Define Probability of Improvement (PoI) and Expected Improvement (EI) - Show how they're derived from decision theory: choosing the next point to maximize expected gain. - Python demo using GPyTorch: fit GP, compute PoI/EI acquisition functions, visualize decision boundary—why one chooses a high-uncertainty point vs. one near known good values. ### Bayesian Experimental Design (BED): Minimizing Uncertainty (8 min) - Motivation: cost-sensitive data collection (labeling, surveys, medical tests). - Define an information-based utility (e.g., expected reduction in entropy). - Show how decision theory prescribes choosing the next experiment to maximize this expected utility. - Python demo using OptBayesExpt. ### Summary & Takeaways (1 min) - Reiterate the decision-theoretic arc: belief → utility → action. - Emphasize the unifying framework across umbrella example, optimization, and experimental design. - Share resources & practical tips: GPyTorch / scikit-optimize, OptBayesExpt false https://cfp.pydata.org/pydataglobal2025/talk/J9JCL9/ https://cfp.pydata.org/pydataglobal2025/talk/J9JCL9/feedback/ Analytics, Visualization & Decision Science Talk 2025-12-10T17:00:00+00:00 17:00 00:30 Fast interactive visualization remains a considerable barrier in analyses pipelines for large neuronal datasets. Here, we present fastplotlib, a scientific plotting library featuring an expressive API for very fast visualization of scientific data. Fastplotlib is built upon pygfx which utilizes the GPU via WGPU, allowing it to interface with modern graphics APIs such as Vulkan for fast rendering of objects. Fastplotlib is non-blocking, allowing for interactivity with data after plot generation. Ultimately, fastplotlib is a general purpose scientific plotting library that is useful for the fast and live visualization and analysis of complex datasets. pydataglobal2025-78560-fastplotlib-driving-scientific-discovery-through-data-visualization Analytics, Visualization & Decision Science /media/pydataglobal2025/submissions/8RUFNS/fastplotlib_SSa-R02_RTgO3OW.png Kushal KolarCaitlin Lewis en Over the past decade, advanced analyses pipelines have been developed for the analysis of large datasets. However, fast visualization and live interactivity during data collection remains challenging. While current tools within the Python plotting ecosystem allow for interactive data visualization, they either fail to leverage modern GPUs efficiently, lack intuitive APIs for rapid prototyping, or require users to write their own shaders. Additionally, other popular plotting libraries, such as bokeh and matplotlib, are not geared towards fast interactive visualization with millions of objects. Given these challenges with current visualization tools, the need for a modern GPU-driven interactive plotting library exists. In this presentation, we will go through the technical details, as well as a brief demo on how fastplotlib makes fast interactive visualization of complex datasets possible. We will demonstrate the broad applicability of fastplotlib as a fast, general-purpose plotting library. Fastplotlib is built on top of pygfx which is a cutting edge Python rendering engine that utilizes WGPU, which can efficiently leverage modern GPU and CPU hardware. WGPU is the successor to OpenGL and features a low overhead with respect to the amount of code per-draw-per-object allowing for speed even when rendering millions of objects. Pygfx is also non-blocking, which allows for interactivity and modification of already drawn objects. Fastplotlib utilizes the pygfx rendering library for fast visualization with an expressive API for scientific visualization. The benefits of fastplotlib are that it reduces boilerplate code which allows users to focus on their data without having to manage the underlying rendering process. Additionally, fastplotlib allows for animations as well as high-level interactivity among plots, which can be combined with lazy loading and lazy compute of very large datasets that are hundreds of gigabytes or terabytes in size. Furthermore, fastplotlib can be used in jupyter notebooks, allowing it to be used on cloud computing and other remote infrastructures for streaming visualizations of extremely large datasets. In total, these unique features and the underlying architecture create a plotting library that is fast, easy to use, and multifaceted. false https://cfp.pydata.org/pydataglobal2025/talk/8RUFNS/ https://cfp.pydata.org/pydataglobal2025/talk/8RUFNS/feedback/ Analytics, Visualization & Decision Science Tutorial 2025-12-10T17:30:00+00:00 17:30 01:30 This hands-on tutorial introduces practical Bayesian inference using PyMC, focusing on A/B testing, decision-making under uncertainty, and hierarchical modeling. With real-world examples, you'll learn how to build and interpret Bayesian models, evaluate competing hypotheses, and implement adaptive strategies like Thompson sampling. Whether you're working in marketing, healthcare, public policy, UX design, or data science more broadly, these techniques offer powerful tools for experimentation, decision-making, and evidence-based analysis. pydataglobal2025-78434-bayesian-decision-analysis-with-pymc-beyond-a-b-testing Analytics, Visualization & Decision Science /media/pydataglobal2025/submissions/NJNHQB/Screenshot_2025-07_dXbnZjS.webp Allen Downey en Bayesian methods offer a natural and interpretable framework for updating beliefs with data, and PyMC makes it easy to apply these techniques in practice. In this tutorial, we’ll walk through a series of examples that demonstrate the core concepts: 1. Bayesian A/B Testing with the Beta-Binomial Model * Represent prior beliefs with the beta distribution * Use binomial likelihoods to model observed outcomes * Understand posterior distributions and credible intervals 2. Bayesian Bandits and Thompson Sampling * Go beyond hypothesis testing: estimate the probability of one version outperforming another * Use Thompson sampling to guide decision-making * Simulate and visualize an adaptive email campaign 3. Hierarchical Models for Partial Pooling and Prediction * Learn how to share information across variants * Use posterior predictive distributions to quantify uncertainty * Understand second-order probabilities Hands-On Learning Participants will follow along in Jupyter notebooks (hosted on Colab — no installation required). Exercises are embedded throughout, with guided solutions. Code is based on PyMC, ArviZ, and standard scientific Python libraries. Prerequisites * Intermediate Python: basic familiarity with NumPy, plotting, and Jupyter notebooks * No prior experience with Bayesian statistics or PyMC is assumed * All materials run on Colab (no setup required) false The material for the tutorial is in this repository https://cfp.pydata.org/pydataglobal2025/talk/NJNHQB/ https://cfp.pydata.org/pydataglobal2025/talk/NJNHQB/feedback/ Data Engineering & Infrastructure Talk 2025-12-10T12:00:00+00:00 12:00 00:30 [OpenStreetMap](https://www.openstreetmap.org/) data is publicly available, but it's hard to get it downloaded at scale without domain knowledge and an external technology stack. With [QuackOSM](https://github.com/kraina-ai/quackosm), you can easily work with whole-country vector and tag data without installing additional dependencies - come and find out how you can use it in your next project! pydataglobal2025-78639-getting-big-openstreetmap-data-with-quackosm Data Engineering & Infrastructure Kamil Raczycki en [QuackOSM](https://github.com/kraina-ai/quackosm) is a powerful and user-friendly library that streamlines the process of accessing and manipulating OpenStreetMap (OSM) vector and tags data. It's using the [DuckDB](http://duckdb.org/) engine with its [Spatial extension](https://duckdb.org/docs/extensions/spatial/overview), and PyArrow library that enables users to efficiently retrieve large-scale OSM data in the GeoParquet format. It's similar in functionality to other available libraries, but it's faster, can work with bigger than memory datasets and doesn't require any additional dependencies. --- Target audience: Data engineers/analysts/scientists who have worked with or want to work with geospatial data. --- Outline: - Brief OpenStreetMap data introduction - Introduction to DuckDB and PyArrow - Why is it hard to work with big OSM datasets? Introduction to the OpenStreetMap data schema and PBF format. - QuackOSM overview: basic usage, data filtering, example use-cases + benchmark against available libraries (OSMnx, Pyrosm, PyDriosm and others). - Example of a simple ML model built on top of geospatial data false Slides https://cfp.pydata.org/pydataglobal2025/talk/EXUXFR/ https://cfp.pydata.org/pydataglobal2025/talk/EXUXFR/feedback/ Data Engineering & Infrastructure Talk 2025-12-10T13:00:00+00:00 13:00 00:30 [RDepot](https://rdepot.io) is a solution for the management of R package repositories in an enterprise environment. Python support has recently been implemented and this talk will introduce RDepot to the Python community. It allows to submit packages through a user interface or API and to automatically update and publish Python and R repositories. In this talk we will walk Python users and developers through different features of RDepot and demonstrate how these can be useful in different scenarios. pydataglobal2025-78691-rdepot-100-open-source-enterprise-management-of-python-and-r-repositories Data Engineering & Infrastructure /media/pydataglobal2025/submissions/YTYRLZ/rdepot_pydata_2025_w6QusCB.png Jonas Van Malder en [RDepot](https://rdepot.io) is a solution for the management of Python and R package repositories in an enterprise environment. It allows to submit packages through a user interface or API and to automatically update and publish Python and R repositories. Multiple departments can manage their own repositories and different users can have different roles in the management of their packages. With continuous integration infrastructure for quality assurance on Python and R packages, package uploads can be automated. All configuration is declarative and RDepot can be set up as infrastructure as code, which is especially relevant in regulated contexts, since it makes validation activities much easier. Packages from publicly available Python repositories such as [PyPi](https://pypi.org/) can be mirrored selectively in custom repositories for use behind a firewall, in internal networks and offline. Combined with [Crane](https://craneserver.net), authentication and fine-grained authorization (using [OpenID Connect](https://openid.net/developers/how-connect-works/)) can be configured per repository, which offers extra security when dealing with sensitive data or sensitive methodology. In this talk we will walk Python users and developers through different features of RDepot and demonstrate how these can be useful in different scenarios. The logic of the different workflows will be explained and live demos will be given to see the open source solution in action. We will make sure to address needs ranging from small research groups sharing a handful of packages up to multinational companies managing their Python (and R) code across the globe. false https://cfp.pydata.org/pydataglobal2025/talk/YTYRLZ/ https://cfp.pydata.org/pydataglobal2025/talk/YTYRLZ/feedback/ Data Engineering & Infrastructure Talk 2025-12-10T13:30:00+00:00 13:30 00:30 JSON support and interfaces vary widely across languages and Julia has been no different. As Julia has evolved as a language, patterns and best practices with regards to interfaces have also evolved with how to best leverage Julia's unique strengths: multiple dispatch, library composability, and zero-cost abstraction. The original JSON.jl package has been rewritten from scratch for a (finally!) 1.0 release bringing JSON support in Julia up to modern best practices and patterns, combining functionality from at least 3(!) existing JSON packages into one unified library offering. pydataglobal2025-79175-modernizing-json-for-julia Data Engineering & Infrastructure Jacob Quinn en Over Julia's history, there have been a number of JSON packages providing various forms of JSON support: * JSON.jl: oldest/original JSON package; very simple JSON support for reading/writing for mostly just core Julia data structures * LazyJSON.jl: package that attempted to provide "lazy" parsing support where JSON could be scanned without fully materializing objects in memory; never quite fully "finished" the package/functionality and was thus, never really widely adopted * JSON2.jl/JSON3.jl: Iterations on interfaces to support custom struct serialization/deserialization in Julia The new 1.0 release to the JSON package combines the functionality from all these packages in a single, unified, *and modern* interface. Package functionality now includes: * Same basic JSON support of reading/writing for core datastructures * Support for lazily processing JSON including extracting deeply nested values without intermediate materialization * A new JSON.Object structure that mimics a `Dict{Symbol, Any}` but preserves insertion (or in this case parse) order, allows dot access, and in most cases is faster with fewer memory allocations than Dict. * Custom struct serialization/deserialization support that includes specifying field defaults, custom field lower/lift functionality, or directly mutating fields (of mutable structs) while parsing This talk aims to cover the historical context leading to the JSON.jl 1.0 release, how the package leverages clean internal interfaces to provide a ton of functionality without exploding the codebase, and why the decision was made to ultimately rewrite the original JSON.jl package for a 1.0 release instead of yet-another-JSONX.jl type package. false https://cfp.pydata.org/pydataglobal2025/talk/CCRL7W/ https://cfp.pydata.org/pydataglobal2025/talk/CCRL7W/feedback/ Data Engineering & Infrastructure Talk 2025-12-10T14:00:00+00:00 14:00 00:30 The modern Python ecosystem shortens the distance between idea and implementation. This talk presents a focused workflow to move from a business question to a working prototype, fast. We'll explore reproducible environments (uv, Docker), quick data iteration with polars and duckdb, clean project scaffolding (pyproject.toml), and lightweight service layers with FastAPI and pydantic. Along the way, we’ll integrate tests (pytest), static checks (mypy), and fast linting (ruff). You’ll leave with a reusable structure, toolchain recommendations, and a mental model for optimizing feedback loops and development in modern Python projects. pydataglobal2025-78689-from-ideas-to-apis-delivering-fast-with-modern-python Data Engineering & Infrastructure /media/pydataglobal2025/submissions/ZXVYCB/Swiss_Army_Knife_of_MBfXDS5.png César Soto Valero en This talk outlines a practical, opinionated workflow for building real things quickly using modern Python without relying on heavy frameworks or over-engineering. Core idea: The shortest path from notebook to usable component is a repeatable, well-lit toolchain with the right structure. Attendees will learn how to: 1. Scaffold a clean project using pyproject.toml, deterministic environments (uv), and lightweight automation (e.g. Makefile or CLI scripts). 2. Explore data rapidly with polars and duckdb, capturing the business logic in small, testable functions. 3. Wrap the logic in a minimal FastAPI app with pydantic validation, creating clean contracts and boundaries. 4. Add fast feedback mechanisms: tests with pytest, type safety via mypy, and low-friction code hygiene using ruff and pre-commit. 5. Package a handoff-friendly interface (command-line entrypoints, minimal docs) for teammates or deployment pipelines. This talk isn’t a showcase of cutting-edge libraries. It’s a field guide on how to leverage modern Python tools and fostering repeatable software engineering habits to maximize value delivery. You’ll leave with: - A blueprint for rapid iteration. - Reusable patterns for API-bound prototyping. - A mindset that treats reproducibility as a first-class concern. Prior knowledge expected: Basic Python (functions, environments), familiarity with DataFrame operations, and HTTP/JSON fundamentals. false https://cfp.pydata.org/pydataglobal2025/talk/ZXVYCB/ https://cfp.pydata.org/pydataglobal2025/talk/ZXVYCB/feedback/ Data Engineering & Infrastructure Talk 2025-12-10T14:30:00+00:00 14:30 00:30 While many of us have adapted to work from home life, one major problem remains: finding an easy way to keep folks in your home away from your workspace when you’re on an important call. Dust off your Raspberry Pi––let’s build a custom on-air sign with Apache Kafka®, Apache Flink®, and Apache Iceberg™! We’ll begin by writing Python scripts to capture key events––such as when a Zoom meeting is running and when a camera is being used––and produce it into Kafka. The live data are then consumed by a Raspberry Pi script to drive the operation of a custom designed on-air sign. From there, you’ll be introduced to the ins and outs of FlinkSQL for stream processing as we wrangle the data into a better format for downstream use. And, finally, we’ll see Iceberg in action and learn how to use query engines to analyze meeting and recording trends. By the end of the session, you’ll be well-acquainted with this powerful trio of open source technologies and know how you could use the same scaffolding and scale out a simple, at-home project to millions of users and simultaneous events. pydataglobal2025-79503-quiet-on-set-building-an-on-air-sign-with-open-source-technologies Data Engineering & Infrastructure Danica Fine en Learn how to build a custom On-Air sign using Apache Kafka®, Apache Flink®, and Apache Iceberg™! See how to capture events like Zoom meetings and camera usage with Python, process data with FlinkSQL, analyze trends in your Iceberg tables, and bring it all together with a practical IoT project that easily scales out. false https://cfp.pydata.org/pydataglobal2025/talk/K38JGZ/ https://cfp.pydata.org/pydataglobal2025/talk/K38JGZ/feedback/ Impact Scholarship Program Talk 2025-12-10T16:00:00+00:00 16:00 01:00 Hosted by Inessa Pawson (NumPy Steering Council, OpenTeams Open Source Program) pydataglobal2025-86182-bof-from-data-to-decisions-leveraging-generative-ai-across-the-data-science-workflow Impact Scholarship Program Inessa Pawson en This Birds of a Feather session provides an opportunity for a cross-disciplinary dialogue about practical applications, challenges, ethical considerations, and emerging best practices for leveraging generative AI in data science. false https://cfp.pydata.org/pydataglobal2025/talk/BGK8N8/ https://cfp.pydata.org/pydataglobal2025/talk/BGK8N8/feedback/ General Track Talk 2025-12-11T13:00:00+00:00 13:00 00:30 At 1:23 a.m. on 26 April 1986, the RBMK-4 graphite-moderated reactor at Chernobyl exploded. Every dosimeter still working inside flat-lined at 3.6 R/h, its maximum reading, while lethal radiation raged unseen. That single detail from Chernobyl is the perfect allegory for what can go wrong in modern machine-learning pipelines: clipped features, hidden distribution shifts, missing logs, runaway feedback loops, and more. This talk unpacks key incidents from the disaster and map each one to an equivalent failure mode in production ML, showing how silent risk creeps into data systems and how to engineer for resilience. Attendees will leave with a practical set of questions to ask, signals to track, and cultural habits that keep models (and the businesses that rely on them) well clear of their own meltdowns. No nuclear physics required. pydataglobal2025-78785-when-the-meter-maxes-out-chernobyl-disaster-lessons-for-ml-systems-in-production General Track Idan Richman Goshen en Software engineers aren’t nuclear engineers, yet the patterns behind catastrophic failure are uncannily transferable. In Chernobyl’s control room, a radiation gauge pinned at 3.6 R/h masked lethal reality; in production we truncate floats, or hide exploding metrics behind poorly chosen histogram bins. Operators overrode the reactor’s emergency cooling “just for this test”; we disable schema validation to hurry a back-fill. Steam-void reactivity formed a positive feedback loop; recommenders amplify popularity bias until user engagement collapses. The session walks through several such parallels. Each mini-segment starts with the historical context, then immediately pivots into a modern use-case that demonstrates the ML analogue, for instance, an ad-ranking model whose session_depth feature is computed differently online than in training, yielding a negative CTR lift despite glowing offline metrics. While the historical narrative keeps the material memorable, the engineering focus stays firmly on actionable prevention: tools like great expectations, out-of-distribution gates, reproducible datasets, and perhaps most importantly - a culture that treats “impossible” as a probability, not a certainty. No specialized nuclear knowledge is assumed. Code examples (when present) use familiar PyData stack - NumPy, Pandas, scikit-learn. The use-cases, concepts and tools shown can appeal to both seasoned practitioners and those earlier in their ML journey. false https://cfp.pydata.org/pydataglobal2025/talk/NSWVT3/ https://cfp.pydata.org/pydataglobal2025/talk/NSWVT3/feedback/ General Track Talk 2025-12-11T14:00:00+00:00 14:00 00:30 The zarr-python 3.0 release includes native support for device buffers, enabling Zarr workloads to run on compute accelerators like NVIDIA GPUs. This enables you to get more work done faster. This talk is primarily intended for people who are at least somewhat familiar with Zarr and are curious about accelerating their n-dimensional array workload with GPUs. That said, we will start with a brief introduction to Zarr and why you might want to consider it as a storage format for the n-dimensional arrays (commonly seen in geospatial, microscopy, or genomics domains, among others). We'll see what factors affect performance and how to maximize throughput for your data analysis or deep learning pipeline. Finally, we'll preview the future improvements to GPU-accelerated Zarr and the packages building on top of it, like xarray and cubed. After attending this talk, you'll have the knowledge needed to determine if using zarr-python's support for device buffers can help accelerate your workload. pydataglobal2025-79453-gpu-accelerated-zarr General Track Tom Augspurger en This talk is targeted at users who have at least heard of zarr, but we will give a brief introduction of the basics. The primary purpose is to spread knowledge about zarr-python’s recently added support for device (GPU) buffers and arrays, and how it can be used to speed up your array-based workload. An outline: - Introduction - Brief overview of zarr (cloud-native format for storing chunked, n-dimensional arrays) - Brief example of how easy it is to use zarr-python’s native support for device arrays - Overview of GPU-accelerated Zarr workloads - We’ll some high-level examples of how Zarr fits into larger workloads (e.g. analyzing climate simulations, as part of a deep learning pipeline) - We’ll discuss the key factors to think about when trying to maximize performance - Overview of how it works - Show zarr’s configuration options for selecting between host and device buffers - An overview of the Zarr codec pipeline - Show how on-device decompression can be used, to accelerate decompression if that’s a bottleneck in your workload - Benchmarks showing the speedup users can expect to see from GPU acceleration - Preview of future work - Zarr-python currently only uses a single GPU, and doesn’t use any features like CUDA Streams. https://github.com/zarr-developers/zarr-python/issues/3271 tracks possible improvements for exposing additional parallelism. - We’ll look at a prototype of how CUDA streams enable asynchronous host-to-device memory copies, enabling you to start computing on one chunk of data while another chunk is being copied to the device. false https://cfp.pydata.org/pydataglobal2025/talk/PSNG8L/ https://cfp.pydata.org/pydataglobal2025/talk/PSNG8L/feedback/ General Track Talk 2025-12-11T15:00:00+00:00 15:00 01:00 Python is the number one language on GitHub, yet a git log of our shared future reveals a critical system failure. While Python adoption is exponential, women author only 2–3% of core-repository commits and comprise just 22% of the global AI talent pool. This talk moves beyond rhetoric to present a 5-Step Engineering Framework based on quantitative research from 24 global tech communities. We will introduce the VIM Model (Visibility, Invitation, Mechanism)—a proven architecture that drove 179% membership growth and 99% retention in pilot programs. pydataglobal2025-85077-keynote-noor-aftab-the-next-commit-building-inclusive-data-driven-ecosystems-for-responsible-ai General Track Noor Aftab en Python powers the global AI ecosystem, yet 78% of the talent pool is missing. We are building the most advanced systems in history with a critical "innovation debt." This is evidenced not just by the gender gap, but by biased algorithms and higher error rates in production models. This talk treats this gap as an engineering crisis and provides a research-backed solution. Drawing on published work from the SciPy Proceedings and a quantitative study of 24 global tech communities, we will introduce the 5-Step Engineering Framework. We will deconstruct the VIM Model (Visibility, Invitation, Mechanism), which drove 179% membership growth in the IBM Women in AI pilot. Attendees will walk away with three actionable tools: 1) The System Audit: A method to measure "innovation debt" in your own teams using specific retention and contribution metrics. 2) The VIM Patch: A blueprint for deploying high-yield mechanisms—such as hands-on Python labs (requested by 76% of members)—that statistically outperform generic networking. 3) The Retention Fix: A step-by-step guide to stabilizing the "leaky pipeline," specifically addressing the mid-career drop-off point where 50% of diverse talent currently leaves. This session is for builders and maintainers ready to stop admiring the problem and commit to the fix. false https://cfp.pydata.org/pydataglobal2025/talk/FLD9SR/ https://cfp.pydata.org/pydataglobal2025/talk/FLD9SR/feedback/ General Track Talk 2025-12-11T16:00:00+00:00 16:00 00:30 Your model might pass all the benchmarks—but can it survive a subpoena? In the race to ship AI, most teams are building workflows that look great in dashboards but fall apart under legal, regulatory, or ethical pressure. Because the real liability doesn’t live in your model weights—it’s buried in your data. pydataglobal2025-77389-garbage-in-lawsuit-out-building-compliant-and-reproducible-ml-pipelines General Track Itai Gilo en This session is a reality check for anyone shipping machine learning in production. We’ll walk through the dark corners of modern ML pipelines: mutable datasets with no history, mystery data sources with missing labels, and a forgotten column of PII that’s just been shipped to production. Then we’ll show how to fix it—without turning your data team into compliance officers. You’ll learn how to embed reproducibility, traceability, and policy enforcement into your pipeline without slowing it to a crawl: track every dataset change, version every experiment, validate against policy gates, and generate audit trails that actually mean something. Whether you’re dealing with GDPR, HIPAA, or just not wanting to get roasted by internal audit, this talk gives you the blueprint for ML you can defend in court—and still ship on time. false https://cfp.pydata.org/pydataglobal2025/talk/SBM8ZY/ https://cfp.pydata.org/pydataglobal2025/talk/SBM8ZY/feedback/ General Track Talk 2025-12-11T17:00:00+00:00 17:00 00:30 Access control is ultimately about relationships—between people, systems, and resources. In this talk, we’ll look at how modeling connected identities with a graph database unlocks a more efficient and transparent way to manage Identity and Access Management (IAM). Using Neo4j and Python, we’ll walk through a practical approach to building an IAM system that prioritizes clarity, performance, and portability. You’ll learn how to model users, roles, and permissions as a connected graph, write access logic in Cypher, and deploy a lightweight system that scales without adding complexity. In this fast-paced talk, you’ll learn how to : - Map users, roles, and permissions like a detective - Write smart queries to control access - Build a lightweight, graph-powered IAM engine No graph skills? No problem. Just bring Python and curiosity. pydataglobal2025-79433-connected-identities-rethinking-identity-and-access-management-with-neo4j-and-python General Track /media/pydataglobal2025/submissions/SSVDUG/role-user-group_GfdEY4N.png Irina Loghin en Access control: it sounds boring—until it breaks. In this talk, we’ll look at how to build a smarter Identity and Access Management (IAM) system using Neo4j and Python, and why graphs are a game-changer for modeling who can do what. You’ll get a crash course in graph-based thinking for IAM, see how to represent users, roles, and permissions as connected data, and learn how a few Cypher queries can uncover misconfigurations, rogue access, and hidden connections—all in real time. As systems scale and architectures grow more distributed, Identity and Access Management (IAM) often becomes a heavy, costly layer—difficult to maintain, expensive to scale, and slow to adapt. But it doesn’t have to be this way. This talk introduces an approach to IAM that is lightweight, portable, and cost-efficient, using Neo4j and Python. By leveraging the natural connectedness of identity data—users, roles, permissions, and resources—we can model access in a way that’s easy to manage, fast to query, and flexible to deploy. Attendees will learn how to build a graph-based IAM system that avoids complex cloud dependencies, offers real-time access insights, and supports role- and attribute-based access control without requiring massive infrastructure. Whether you're managing internal tools, building developer platforms, or scaling services, this approach provides strong access control without unnecessary overhead. false https://cfp.pydata.org/pydataglobal2025/talk/SSVDUG/ https://cfp.pydata.org/pydataglobal2025/talk/SSVDUG/feedback/ Machine Learning & AI Talk 2025-12-11T11:30:00+00:00 11:30 00:30 In this presentation, we demonstrate how Large Language Models (LLMs) can revolutionize safety log analysis in the oil and gas industry. Our research with a major operator involved processing 15,000 safety observations through a novel multi-stage pipeline. First, we developed a domain-specific categorical framework aligned with industry standards. We then implemented an unsupervised learning approach using sentence transformers to calculate semantic similarity between observations and predefined categories. This enabled multi-dimensional classification with weighted confidence percentages. Finally, we deployed a fine-tuned LLM to assign priority scores and enhance categorization accuracy, all while maintaining data privacy through on-premises processing. The resulting system streamlines real-time safety log processing, enabling more efficient identification of potential hazards and trends. Our implementation demonstrates significant improvements in classification accuracy and processing efficiency compared to traditional methods, providing actionable insights for proactive safety management. pydataglobal2025-78720-revolutionizing-safety-log-analysis-in-oil-and-gas-a-multi-stage-llm-approach-for-enhanced-hazard-identification Machine Learning & AI /media/pydataglobal2025/submissions/AJD8TU/PyData_Presentation_WfU6jwu.png Andrew YuleIain Docherty en This presentation explores a new application of Large Language Models (LLMs) in the oil and gas industry, specifically for safety log analysis. While oil and gas operators have traditionally been cautious in adopting LLM technologies, this project demonstrates a compelling use case that delivers tangible value through enhanced hazard identification and trend analysis. Attendees will learn how our multi-stage LLM pipeline processes safety observations to generate actionable insights while maintaining data privacy through on-premises processing. The presentation will showcase how this approach significantly improves classification accuracy and processing efficiency compared to traditional methods, providing a practical framework for organizations looking to leverage AI for safety management. false https://cfp.pydata.org/pydataglobal2025/talk/AJD8TU/ https://cfp.pydata.org/pydataglobal2025/talk/AJD8TU/feedback/ Machine Learning & AI Talk 2025-12-11T12:00:00+00:00 12:00 00:30 Small Language Models (SLMs) are designed to deliver high performance with significantly fewer parameters compared to Large Language Models (LLMs). Typically, SLMs range from 100 million to 30 billion parameters, enabling them to operate efficiently on devices with limited computational resources, such as smartphones and embedded systems pydataglobal2025-77504-how-big-are-slms Machine Learning & AI Jayita Bhattacharyya en The development of SLMs addresses the growing demand for AI solutions that are cost-effective, energy-efficient, and capable of running locally to ensure data privacy and reduce latency. Recent advancements have demonstrated that SLMs can rival or even surpass larger models in specific tasks, thanks to optimized architectures and training methodologies .​ A notable example is Google's Gemma 3, a multimodal SLM family with models ranging from 1 to 27 billion parameters. Gemma 3 introduces vision understanding capabilities, supports longer context windows of at least 128K tokens, and employs architectural changes to reduce memory usage . The 27B parameter version of Gemma 3 has achieved competitive performance, ranking among the top 10 models in the LMSys Chatbot Arena with an Elo score of 1339 . The shift towards SLMs signifies a paradigm change in AI development, focusing on creating models that are not only powerful but also accessible and adaptable to a wide range of applications. As the field evolves, SLMs are poised to play a crucial role in democratizing AI technology.​ false https://cfp.pydata.org/pydataglobal2025/talk/ATM79G/ https://cfp.pydata.org/pydataglobal2025/talk/ATM79G/feedback/ Machine Learning & AI Talk 2025-12-11T13:00:00+00:00 13:00 00:30 This Live demonstration shows how PyCaret, an open-source low-code machine learning library, can dramatically simplify model training and comparison workflows. PyCaret is democratizing machine learning by empowering anyone to train multiple algorithms and compare their performance with minimal code. Attendees will witness live demonstrations of training various ML algorithms and using automated comparison techniques to select the best performer based on key metrics. Perfect for data scientists, developers, and ML enthusiasts looking to spend less time coding and more time on model analysis and selection. pydataglobal2025-78505-automating-ml-with-pycaret-train-compare-multiple-models-to-find-the-best-performer Machine Learning & AI Manjunath Janardhan en Machine learning workflows often involve repetitive tasks, complex code, and time-consuming model comparisons. PyCaret changes this paradigm by democratizing machine learning - empowering anyone to train multiple algorithms and systematically compare their performance with low-code solutions. With PyCaret's philosophy of "spend less time coding and more time on analysis," this library transforms the model selection process by automating training and comparison across multiple algorithms. In this 30-minute session, you'll discover: ML and PyCaret Fundamentals (13 mins) 1. What is Machine Learning, Machine Learning Algorithms and workflows 2. What is PyCaret Live Demo: Multi-Algorithm Training & Comparison (10 mins) 1. Hands-on demonstration using the Diabetes Dataset 2. Training multiple algorithms simultaneously with minimal code 3. Automated model comparison using various performance metrics 4. Real-time exploration of model performance visualizations 5. Selecting the best performer based on key evaluation metrics Wrap-up & Resources (2 mins) 1. Key takeaways and next steps 2. Access to GitHub repository with slides and demo notebooks Q&A (5 min) Who Should Attend: 1. Data scientists looking to accelerate their workflow 2. Python developers interested in machine learning 3. ML practitioners seeking efficient model prototyping tools 4. Anyone curious about low-code ML solutions Prerequisites: 1. Basic understanding of Python 2. Familiarity with machine learning concepts (helpful but not required) 3. No prior PyCaret experience needed What You'll Take Away: 1. Practical knowledge of automated model training and comparison 2. Experience with systematic algorithm evaluation using PyCaret 3. Understanding of performance metrics for model selection 4. Ready-to-use code examples for multi-algorithm comparison 5. Confidence to choose the best ML algorithm for your specific projects Join us for this fast-paced, demo-heavy session that will transform how you approach machine learning projects! false https://cfp.pydata.org/pydataglobal2025/talk/ECCYVF/ https://cfp.pydata.org/pydataglobal2025/talk/ECCYVF/feedback/ Machine Learning & AI Talk 2025-12-11T13:30:00+00:00 13:30 00:30 Python users working on real-time analytics—from payment processing and fraud detection to AI-driven support—rely on message queues to keep data moving reliably and efficiently. Traditional message queues, however, can struggle with large-scale, concurrent workloads, especially when you need durability and replayability. In this session, we’ll show how Kafka 4.0 introduces robust queue semantics to distributed streaming, empowering Python applications to handle fair, concurrent, and isolated message processing at scale—using familiar Kafka Python clients and frameworks. But the power lies in what you can build next. We’ll demonstrate how Apache Flink can connect Kafka event streams to real-time Large Language Model (LLM) inference for tasks like sentiment analysis and summarization, all orchestrated via Python APIs and remote model endpoints for powerful, flexible AI inference. To complete the picture, we’ll cover how enriched results can be stored in popular data lake solutions—such as Apache Iceberg—enabling long-term analytics, time travel, and integration with downstream data science workflows. Support for Iceberg and other lakehouse formats is optional, giving you flexibility to choose the right data backend for your needs. pydataglobal2025-78603-streaming-ai-workflows-in-python-kafka-queues-and-flink-powered-llm-inference Machine Learning & AI Shekhar Prasad Rajakbhrathjatoth en This talk includes: Live Python-oriented demo and architecture walkthrough. Building an end-to-end pipeline: Kafka queue → Flink+LLM inference → (optional) Data lake storage (e.g., Iceberg). Python code samples, best practices, and design patterns for powering real-time, intelligent analytics on modern cloud-native stacks. Whether you’re developing in Jupyter Notebooks, Pandas, or PySpark, you’ll discover practical ways to combine Kafka, Flink, and LLMs in your Python data workflows—with or without a lakehouse backend. false https://cfp.pydata.org/pydataglobal2025/talk/7MEX7V/ https://cfp.pydata.org/pydataglobal2025/talk/7MEX7V/feedback/ Machine Learning & AI Talk 2025-12-11T14:00:00+00:00 14:00 00:30 Your notebooks are full of insights—but they’re scattered and hard to search. In this live-coding session I’ll show how to turn handwritten notes into a searchable, connected knowledge base using local AI and minimal Python. We start with AnythingLLM’s UI for quick wins, then move to Python agents that: • classify note types, • extract key ideas, • build a personal knowledge graph. The entire stack runs on your laptop with MLC-AI—no cloud, no data leaks. You’ll leave with a reusable agent blueprint you can drop into any data-processing workflow tomorrow. pydataglobal2025-78498-from-handwritten-notes-to-smart-knowledge-build-local-ai-agents-with-python Machine Learning & AI piotr stepinski en What you’ll learn • When to stay in a UI vs. when Python is essential • How to orchestrate agents with CrewAI and plug in custom logic • Clean patterns for local LLM inference with MLC-AI • A complete, copy-paste-ready pipeline for knowledge extraction & linking Live demos AnythingLLM quick-start (2 min) Python agent orchestration classifying & linking 10+ handwritten notes (15 min) Querying the resulting knowledge graph for recurring themes (3 min) Take-home repo GitHub repo + requirements.txt + Docker compose file so attendees can rerun everything on their own notes. Prerequisites Basic Python (functions, classes, pip install). No prior AI/ML knowledge required. false https://cfp.pydata.org/pydataglobal2025/talk/SPFEYP/ https://cfp.pydata.org/pydataglobal2025/talk/SPFEYP/feedback/ Machine Learning & AI Talk 2025-12-11T14:30:00+00:00 14:30 00:30 Financial and other real-world time series often experience abrupt regime changes that can break assumptions and invalidate models. This talk shows how to use k-nearest neighbor entropy estimators combined with clustering algorithms, implemented entirely in Python, to detect these change-points early. We’ll explore practical examples with financial market data, discuss strengths and limitations, and provide reusable open-source code. Attendees will leave with tools to make their time series models more robust to sudden structural changes. pydataglobal2025-77515-detecting-regime-shifts-in-time-series-with-python-entropy-based-change-point-detection Machine Learning & AI Sergei Nasibian en Time series data in finance, IoT, or sensor monitoring are rarely stationary — regime shifts happen suddenly, and failing to detect them early can lead to inaccurate predictions or large financial losses. This talk presents a practical, Python-based approach to change-point detection in multivariate time series using k-nearest neighbor entropy estimators combined with clustering techniques. This method uses open-source libraries like NumPy, scikit-learn, and pandas, and can be adapted to various domains. Takeaways: - How to implement entropy-based change-point detection with open-source Python tools. - How to identify and handle abrupt shifts in time series to make models more robust. - How to apply these techniques beyond finance to any time series with regime shifts. false https://cfp.pydata.org/pydataglobal2025/talk/RQSLXN/ https://cfp.pydata.org/pydataglobal2025/talk/RQSLXN/feedback/ Machine Learning & AI Talk 2025-12-11T16:00:00+00:00 16:00 00:30 In this talk I will cover frequent AI system problems caused by using prompts and opaque frameworks instead of a descriptive programmatic approach, using DSPy. pydataglobal2025-79526-future-proof-your-ai-product Machine Learning & AI Breno Brito en Most LLM frameworks are too opaque and obscure what they are doing. New state of the art models are released every week and different models respond differently to the same prompts. These framework's hardcoded prompts within the library make it difficult to debug, update and improve the system. Also, walls of text are a terrible way to program, and hardly maintainable. DSPy is a better way, using abstractions to code your intent into the LLM without defining the prompt, making it future proof. Changing one line, you can change models, tasks or inference strategy. false https://cfp.pydata.org/pydataglobal2025/talk/DYXWAV/ https://cfp.pydata.org/pydataglobal2025/talk/DYXWAV/feedback/ Machine Learning & AI Talk 2025-12-11T16:30:00+00:00 16:30 00:30 This talk discusses a hybrid recommender system implemented in Julia for preselecting job applicants. The recommender system is built using a neural network adopting a hybrid architecture that combines convolutional layers of a graph neural network and a transformer (both encoder and decoder). We discuss the preprocessing of applicant metadata and job adverts to generate a heterogeneous graph. Next, we present the recommender as a model and its training using an HPC. pydataglobal2025-78765-hpc-implementation-of-a-hybrid-recommender-system-in-julia Machine Learning & AI José Quenummarthin thomas en In this talk, we present the implementation of a hybrid recommender system that helps preselect candidates for a job application. We discuss the preprocessing of the data following NLP techniques and building on various libraries, including TextAnalysis, Embeddings and MLJ. The input information (applicant metadata and job adverts) is aggregated into a heterogeneous graph, later converted into a GNN using GraphNeuralNetworks. The underlying model supporting the recommendations combines several graph convolutional layers and a transformer (encoder and decoder). To make the model's training more efficient, we rely on the Distributed and ClusterManagers libraries. Note that our preprocessing and training steps are implemented using a supercomputer. We present the implementation and the job submission details. false https://cfp.pydata.org/pydataglobal2025/talk/EJJSKK/ https://cfp.pydata.org/pydataglobal2025/talk/EJJSKK/feedback/ Machine Learning & AI Talk 2025-12-11T18:30:00+00:00 18:30 00:30 Understanding customer behavior is essential in marketing. Traditionally, marketers rely on methods such as surveys, customer interviews, and focus groups to gather insights. However, these approaches can be expensive, time-consuming, and limited in scale and diversity. Recently, multi-agent simulation powered by Large Language Models (LLMs) is emerging as an innovative technique. TinyTroupe, for example, enables the creation of different personas (e.g., budget‑minded Gen‑Z shoppers, premium‑seeking parents), allowing marketers to predict and optimize advertising effectiveness or replace time-consuming interviews rapidly. In this talk, I will introduce the key concepts of LLM-powered multi-agent simulations, demonstrate their practical application in marketing through TinyTroupe, and share actionable insights and recommendations. pydataglobal2025-79471-tinytroupe-enhancing-marketing-insights-through-llm-powered-multiagent-persona-simulation Machine Learning & AI Hajime Takeda en **Agenda:** - Introduction - Business Context: customer understanding & traditional research - The Challenge: “Can’t we just use ChatGPT?” - TinyTroupe: LLM-powered multi-agent persona simulation - Code Walkthrough: end-to-end concept-test demo (running-shoe example) - Summary & practical tips **Code Walkthrough Part** https://github.com/takechanman1228/Effective-Persona-Simulation **Key Takeaways:** - Understand the core concepts and advantages of LLM-powered multi-agent persona simulation. - Learn how to leverage TinyTroupe for efficient and insightful marketing analytics. **Target Audience:** - Data analysts and data scientists interested in customer analytics and marketing. - Marketers, business analysts, and executives seeking innovative approaches to understanding customer behavior and optimizing marketing strategies. - IT specialists and developers interested in applying LLM and multi-agent simulation technologies to real-world business scenarios. false https://cfp.pydata.org/pydataglobal2025/talk/7PTYQX/ https://cfp.pydata.org/pydataglobal2025/talk/7PTYQX/feedback/ Analytics, Visualization & Decision Science Tutorial 2025-12-11T13:30:00+00:00 13:30 01:30 Computer vision, the field focused on enabling machines to interpret and understand visual data, tackles challenges like image recognition, object detection, and scene understanding. PyData tools play a critical role in solving these issues by offering robust libraries like TensorFlow, PyTorch, Keras, and Langchain for building and training machine learning models, performing image processing, and managing large datasets. This hands-on session will enable attendees to learn how to optimize computer vision projects with end-to-end version control baked in. pydataglobal2025-77401-computer-vision-data-version-control-and-reproducibility-at-scale Analytics, Visualization & Decision Science Joe Pringle en Petabytes of unstructured data stand as the cornerstone upon which triumphant Machine Learning (ML) models are built. One common method for researchers to extract subsets of data to their local environments is by simply using the age-old copy-paste, for model training. This method allows for iterative experimentation, but it also introduces challenges with the efficiency of data management when developing machine learning models, including reproducibility constraints, inefficient data transfer, alongside limited compute power. This is where data version control technologies can help overcome these challenges for computer vision researchers. In this workshop we'll cover: - How to use open source tooling to version control your data when working with data locally. - Best practices for working with data, preventing the need to copy data locally, while enabling the training of models at scale directly on the cloud. This will be demoed with an OSS stack: - Langchain - Tensorflow - PyTorch - Keras You will come away with practical methods to improve your data management when developing and iterating upon Machine Learning models, built for modern computer vision research. false https://cfp.pydata.org/pydataglobal2025/talk/WRSZRV/ https://cfp.pydata.org/pydataglobal2025/talk/WRSZRV/feedback/ Analytics, Visualization & Decision Science Talk 2025-12-11T16:30:00+00:00 16:30 00:30 This talk demonstrates how open-source Python tools like censusdis, pandas, and folium can be combined to create an interactive, time-enabled dashboard for visualizing economic vulnerability, housing affordability, and displacement risk in small communities. Using Oxford, NC as a case study, the talk showcases a multi-year, multi-indicator mapping project designed to support equitable local planning. pydataglobal2025-78718-animating-equity-python-dashboards-for-small-town-housing-and-displacement-risk Analytics, Visualization & Decision Science /media/pydataglobal2025/submissions/M78NZT/displacment_risk_tjiLd1Y.png Matthew Cox en How do you turn raw census tables into something a small town can actually use to guide housing policy? In this talk, I walk through the design and development of an animated spatial dashboard built entirely with Python, designed to help local residents and planners in Oxford, North Carolina understand where their most vulnerable neighbors live — and how that vulnerability is changing over time. Oxford is a rural town facing new development pressure, including non-contiguous annexation and suburban for-sale housing growth. While these changes promise tax base expansion, they also risk pushing out low-income renters, especially in historically underserved neighborhoods. My dashboard uses ACS 5-Year estimates and USDA Food Access data to visualize key indicators like rent burden, SNAP share, senior population, and a normalized displacement risk index — all animated from 2017 to 2023 using Leaflet.TimeDimension inside folium. The talk is both a case study in data storytelling for place-based equity and a practical demo of working with geospatial census data in Python — no proprietary software or expensive tools required. Outline (with time estimates) 0–5 min — Context: Why Oxford, NC? The risks of unchecked suburban growth for small cities 5–10 min — Data: ACS, USDA, and parcel-level value data via censusdis and publicly-available shapefiles 10–20 min — Dashboard architecture: Python data pipeline, Folium with TimeSliderChoropleth, adding map interactivity, overlays, and popups 20–25 min — Use case: Displacement risk and the intersection of rent burden, food access, and annexation 25–30 min — Q&A, tips for adapting the method to other communities Audience This talk is intended for: Data analysts, GIS specialists, and Python developers interested in civic tech or applied geospatial analysis Planners, advocates, and public servants exploring how open data and open tools can improve policy transparency Anyone working with small-area census data, especially at the block group or tract level Attendees should have a basic familiarity with Python and data visualization libraries (pandas, folium, etc.), but no prior experience with geospatial programming is required. Takeaways Attendees will learn: How to download and preprocess ACS data at the block group level using Python How to build time-animated choropleth maps using folium + Leaflet.TimeDimension How normalized composite indicators like a displacement risk index can help surface hidden patterns in small towns How interactive mapping can drive better community conversations around housing, equity, and development false https://cfp.pydata.org/pydataglobal2025/talk/M78NZT/ https://cfp.pydata.org/pydataglobal2025/talk/M78NZT/feedback/ Analytics, Visualization & Decision Science Talk 2025-12-11T17:00:00+00:00 17:00 00:30 Most ML models excel at prediction, answering questions like _"Who will buy our product?"_ or _"Which customers are likely to churn?"_. But when it comes to making actionable decisions, prediction alone can be misleading. Correlation does not imply causation, and business decisions require understanding causal relationships to drive the right outcomes. In this talk, we will explore how causal machine learning, specifically uplift modeling, can bridge the gap between prediction and decision making. Using a real-world use case, we will showcase how uplift modeling helps identify who will respond positively to interventions while avoiding those who they might deter. pydataglobal2025-79220-beyond-just-prediction-causal-thinking-in-machine-learning Analytics, Visualization & Decision Science Avik Basu en ## Audience This talk is for data scientists and ML engineers at any level. Basic familiarity with Python and machine learning concepts is helpful but not required. ## Objective Attendees will learn when to use causal thinking vs predictive modeling and how to implement uplift models using Python. They will also understand how to apply these techniques across different domains, such as marketing, healthcare, and other relevant fields. ## Details Predictive ML models are used everywhere for data-driven decision making across industries. However, accurate forecasts don't always translate to optimal actions. We will begin by exploring the fundamental challenges of deriving actions from model predictions, especially when determining the right audience to target. After that, we will dive into some fundamental concepts of causal inference and how it differs from traditional ML. We will then introduce uplift modeling and cover some key concepts, e.g., treatment effects, counterfactuals, meta-learning approaches, etc. We will see how these elements work together to create causal ML models. Finally, we will put theory into practice by building a sample uplift model in Python. We'll walk through each step using real-world intervention data (publicly available), demonstrating how this approach can dramatically improve decision-making and ensure that the interventions target the right audience for the right reasons. ## Outline - Introduction and motivation [1 min] - From correlation to causation [4 min] - Correlation vs Causation - When do we need a causal angle - Core causal concepts [4 min] - Treatment effects - Counterfactuals - Intervention problem - Uplift modeling concepts [5 min] - Four types of individual responses to a treatment - Meta learning approach - T-Learner and S-Learner comparison - Hands-on case study [10 min] - Problem explanation and formulation - Predictive model output - Causal uplift model in Python - Compare targeting strategies and intervention impact - Evaluation [4 min] - Why accuracy or F1 scores don’t work for uplift - Uplift curves - Qini coefficient - Explainability - Practical Considerations [2 min] - A/B testing treatment effects - Cross-domain applications false https://cfp.pydata.org/pydataglobal2025/talk/VY398A/ https://cfp.pydata.org/pydataglobal2025/talk/VY398A/feedback/ Analytics, Visualization & Decision Science Talk 2025-12-11T17:30:00+00:00 17:30 00:30 This session shows how Bayesian statistical modeling helps determine when you have collected enough data about new products, so that they are ready for competition. We'll explore: how this approach enables efficient decision-making with minimal data why we chose Bayesian over machine learning models how we covered for the required assumptions how this enables a risk-management approach while providing interpretable results that business stakeholders can understand and trust You will learn how to identify a Bayesian problem at your company and how to navigate the modelling with real-world data! pydataglobal2025-77428-enhancing-marketplace-competitiveness-a-bayesian-approach-to-modelling-the-cold-start-problem Analytics, Visualization & Decision Science Agustin Figueroa Nazar en In this session, we will explore the application of Bayesian methodology to address the cold start problem in a recommendation system: determining if there is enough data for a new product in a marketplace to be accurately ranked, or if the product should get further exposure to reach that stage. The target audience of this talk is data analysts of all levels, data practitioners interested in modelling, and professionals working in recommendation systems. Unlike traditional machine learning models, Bayesian statistical modelling offers a robust framework for updating probabilities with new evidence, making it particularly suited for dynamic environments like online marketplaces. That way, one can update the learnings on the performance of a new product daily, allowing for efficient decision-making around “should I keep on exploring this new product or not?” while minimising the traffic investment and enabling a risk-management-based approach. We will also cover how we control for the assumptions that Bayesian requires. Key takeaways: 1. Understanding Bayesian Methods: Learn how Bayesian statistics can be applied to real-world business problems, offering a flexible and interpretable approach to decision-making. 2. Benefits Over Machine Learning: Discover why statistical modelling can be more advantageous than machine learning in certain business contexts, particularly when managing risk, handling sparse data and providing interpretable results to the business. 3. Practical Application: Learn about the challenges of applying bayesian models in a real marketplace. Outline: Introduction to the cold-start problem (2 min) How we rank incoming activities at GetYourGuide and how modelling could make us more efficient (5 min) Explaining the model (15 min) Intro to a Bayesian binomial model (3 min) Controlling for independence among trials (3 min) Defining the prior (3 min) Designing a stop criteria (6 min) Risk-management: why Bayesian modelling over Machine Learning (5 min) Questions (3 min) Prerequisites Learn what the cold start problem in a recommender system is (https://en.wikipedia.org/wiki/Cold_start_(recommender_systems)). Get familiar with Bayesian thinking (https://www.countbayesie.com/blog/2022/2/19/how-to-read-the-news-like-a-bayesian). If you want to go fancy, read this paper: https://arxiv.org/pdf/2410.02126 false https://cfp.pydata.org/pydataglobal2025/talk/ESFUQB/ https://cfp.pydata.org/pydataglobal2025/talk/ESFUQB/feedback/ Data Engineering & Infrastructure Talk 2025-12-11T11:30:00+00:00 11:30 00:30 Get a firsthand look at how we built a lightweight feature store to accelerate electricity grid forecasting. We’ll cover our decision process, design choices, and implementation using Polars and Google Cloud Storage. Expect lessons learned, real-world bumps, and a clear view of the costs, trade-offs and benefits of our solution. pydataglobal2025-78648-building-a-lightweight-feature-store-for-electricity-grid-forecasts-with-polars Data Engineering & Infrastructure Robin Troesch en In this talk, we’ll share how we built a lightweight, production-ready feature store to support electricity grid forecasting. You'll hear a firsthand account of our journey—from identifying the need to accelerating model prototyping through feature standardization and flexibility. We’ll start with a high-level overview of our decision-making process: why we chose to build rather than buy, and the trade-offs we considered. Then, we’ll dive into the architecture of our custom feature store, detailing how we leveraged Polars for fast processing and Google Cloud Storage as a scalable backend. Expect an honest look at the challenges we faced, the benefits we gained, and the costs we encountered along the way. Whether you're considering building your own feature store or just curious about scaling ML for time series problems, this session will offer practical insights and real-world lessons. false https://cfp.pydata.org/pydataglobal2025/talk/SRCNAR/ https://cfp.pydata.org/pydataglobal2025/talk/SRCNAR/feedback/ Data Engineering & Infrastructure Talk 2025-12-11T12:30:00+00:00 12:30 00:30 Geospatial analysis often involves harmonizing and processing raster datasets from diverse sources with varying resolutions, coordinate systems, and data formats. This talk demonstrates how you can build efficient, scalable pipelines for zonal statistics extraction using Python’s scientific computing stack, xarray, and dask to handle rasters that would otherwise overwhelm traditional processing approaches. Through a real-world case study of processing multi-source geospatial data for small-area estimation of poverty, we’ll explore practical strategies for memory-efficient raster harmonization, parallel computing workflows, and automated statistical aggregation across administrative boundaries. pydataglobal2025-78709-engineering-large-scale-geospatial-raster-processing-with-xarray-and-dask Data Engineering & Infrastructure CLINTON OYOGO DAVID en This talk addresses a common challenge faced by data scientists, data engineers, researchers, and geospatial analysts working with large-scale geospatial data: how to efficiently process and harmonize raster datasets that exceed memory limits, while maintaining both data integrity and computational performance. Attendees are expected to have a basic familiarity with Python and an understanding of fundamental geospatial concepts. I will begin by outlining prevalent issues in geospatial data processing, such as memory constraints when working with large rasters, the difficulty of harmonizing datasets with varying resolutions and projections, and the computational cost of performing zonal statistics across multiple layers. To address these challenges, I will demonstrate how libraries like xarray and rioxarray offer elegant abstractions for geospatial data manipulation, while Dask facilitates out-of-core computation and parallel processing. A technical walkthrough will showcase a flexible pipeline designed to handle key data processing scenarios: downsampling, upsampling, masking, managing missing values, and other steps. I will do a live code demonstration from a project involving zonal statistics for small area poverty estimation. This will include processing layers such as population density, distance to healthcare, and nightlights to produce harmonized zonal statistics at administrative level three of a select country. To wrap up, we’ll briefly touch on optimization strategies, including chunking techniques and memory management. false https://cfp.pydata.org/pydataglobal2025/talk/YN7DYP/ https://cfp.pydata.org/pydataglobal2025/talk/YN7DYP/feedback/ Data Engineering & Infrastructure Talk 2025-12-11T13:30:00+00:00 13:30 00:30 ShinyProxy is 100% open-source software to deploy data science apps in an enterprise context. This talk will - for the first time - introduce ShinyProxy to the Python community. We'll start with a realistic example to explore what it takes to deploy a data science app for production use. Throughout the talk, you'll see how ShinyProxy addresses many of the common challenges faced when deploying apps. These include authentication, scaling, security (such as TLS), audit logging, version control, reproducibility, and more. The main goal of ShinyProxy is to ensure data scientists can focus on doing science instead of spending time on technical requirements, procedures and maintenance. This talk is tailored for both data scientists and anyone interested in setting up ShinyProxy. No deep technical knowledge is required to follow along. At the end of the talk, you'll know everything to get started with ShinyProxy and to deploy your first app. pydataglobal2025-78713-accelerate-deployment-of-your-python-data-science-apps-using-shinyproxy Data Engineering & Infrastructure Tobia De Koninck en ShinyProxy is already a well known tool to deploy apps built using R and Shiny. This talk will - for the first time - introduce ShinyProxy to the Python community. In the first part of the talk we present how Bob wrote a super useful Python app, but struggles to get it deployed at Bob's company. A first challenge is to get hold of a server with all dependencies and libraries installed. Next, Bob is informed that the app must be protected using TLS and integrated with the existing authentication system. After these first obstacles Bob learns that there are even more requirements and gets stuck. The second part of this talk demonstrates how Bob can solve all these problems using ShinyProxy. For example, using container technology (Docker), Bob has full control on installing dependencies and libraries, while at the same time improving the reproducibility of the setup. This talk is tailored for both data scientists and anyone interested in setting up ShinyProxy. No deep technical knowledge is required to follow along. At the end of the talk, you'll know everything to get started with ShinyProxy and to deploy your first app. ShinyProxy supports almost any web application, including Streamlit, Dash, Voila and Gradio. Therefore, we don't focus on a specific framework. Everything covered in this talk is applicable to your favourite framework. false https://cfp.pydata.org/pydataglobal2025/talk/VS8HWU/ https://cfp.pydata.org/pydataglobal2025/talk/VS8HWU/feedback/ Data Engineering & Infrastructure Talk 2025-12-11T16:30:00+00:00 16:30 00:30 Pandas is a popular library for data scientists but it struggles with large datasets; programs either become too slow or run out of memory. In this talk, we introduce Bodo DataFrames (https://github.com/bodo-ai/Bodo) as a drop-in replacement for the Pandas library that uses high performance computing (HPC) based techniques such as Message Passing Interface (MPI) and JIT compilation for acceleration and scaling. We give an overview of its architecture and explain how it avoids the problems of Pandas (while keeping user code the same), go over concrete examples, and finally discuss current limitations. This talk is for Pandas users who would like to run their code on larger data while avoiding frustrating code rewrites to other APIs. Basic knowledge of Pandas and Python is recommended. pydataglobal2025-78625-bodo-dataframes-a-fast-and-scalable-hpc-based-drop-in-replacement-for-pandas Data Engineering & Infrastructure Scott Routledge en Despite its popularity for data manipulation tasks, Pandas struggles at scale due to its single threaded execution and significant Python-based overheads. In this talk, we introduce Bodo DataFrames as a solution to scaling Pandas with a single line of code change; simply replace `import pandas as pd` with `import bodo.pandas as pd`. Bodo DataFrames transforms Pandas code into lazily evaluated plans, enabling database-quality query optimizations, and runs on a streaming, parallel backend using the Message Passing Interface (MPI) for fast worker-to-worker communication. This design avoids out-of-memory errors and is easily scalable from laptop to large cloud cluster. Unlike other data processing engines, Bodo DataFrames combine powerful techniques from high performance computing (HPC) and databases while remaining fully Pandas compatible. We will present multiple examples and benchmarks demonstrating how to use Bodo DataFrames. The first example will show how to scale a simple program covering functions like reading/writing Parquet files, Series-datetime, merge, and groupby-agg. The next example will demonstrate how to accelerate user defined functions (i.e. map and apply) using Bodo DataFrames builtin support for Just-In-Time (JIT) compilation. The final example will demonstrate how to use Bodo DataFrames support for the Apache Iceberg format, which provides schema evolution and time travel for ever-changing datasets. We will also discuss how Bodo DataFrames falls back to Pandas when it doesn't support all operations of a workload, and planned future work. This talk is designed for users of Pandas; data scientists, data engineers and AI/ML practitioners, who are interested in accelerating and scaling their workloads easily. In addition to a new tool under their belt, attendees will walk away with an understanding of techniques from HPC and databases, unlocking deeper insights into aspects of performance and memory utilization. false https://cfp.pydata.org/pydataglobal2025/talk/UKDKZ7/ https://cfp.pydata.org/pydataglobal2025/talk/UKDKZ7/feedback/ Impact Scholarship Program Talk 2025-12-11T12:00:00+00:00 12:00 01:00 Impact Scholars Program pydataglobal2025-86183-how-do-we-create-access-for-those-who-don-t-show-up-in-our-spaces Impact Scholarship Program Anita Ihuman en Impact Scholars Program false https://cfp.pydata.org/pydataglobal2025/talk/SGNMQM/ https://cfp.pydata.org/pydataglobal2025/talk/SGNMQM/feedback/ Impact Scholarship Program Talk 2025-12-11T17:00:00+00:00 17:00 01:00 During the BoF session, you’ll have the opportunity to submit discussion topics at the start of the meeting via the meeting chat. Discussions will take place in smaller, breakout rooms, where you are free to join or switch groups at any time. While we encourage topics related to Data Science, AI, career development, research, and Open Source, the group can explore other topics as long as they align with the Code of Conduct. Let’s collaborate, share ideas, and inspire each other in a dynamic, interactive environment! pydataglobal2025-86184-bof-networking-session Impact Scholarship Program en Impact Scholars Program false https://cfp.pydata.org/pydataglobal2025/talk/PXLTKU/ https://cfp.pydata.org/pydataglobal2025/talk/PXLTKU/feedback/