Roberto Arnetoli

Great article from Mark Freeman and Chad Sanderson. The answers to the questions at the end point to several other good reasons for moving data quality 'upstream' to where the business conversations happen. https://lnkd.in/gVUPCyAb?

9

6 Comments

An interesting study from Bill Schmarzo and Mark Stouse in Data Science Central "We have been vocal about my belief that organizations focusing on GenAI-driven productivity improvements rather than GenAI-driven innovation improvements are missing the bigger prize. GenAI tools can create value for an organization, but that increase in value is inconsistent. On one hand, GenAI tools can improve the productivity (efficiency) of below-average workers. On the other hand, they can significantly boost the innovation (effectiveness) of top-performing workers who better understand critical value-creation concepts such as data science, design thinking, and economics." #ai #genai #automation #productivity #innovation #teckedin https://lnkd.in/g7kgMnxf

8

1 Comment

You can't impose order on a complex adaptive system at the macro scale. The underlying behaviours will eventually bring it back to its strange attractor ("homeostatis") - the old order. We have to change the underlying behaviours. Macro changes start on the micro scale. More bluntly: lasting change happens bottom-up, not top-down.

21

5 Comments

One feature I have been looking for in #Quarto has been support for Scrollytelling. At posit::conf(2024), Andrew Bray has just presented #Closeread - a Quarto extension that enables quite a few scrollytelling techniques in Quarto. Closeread comes with different layouts and support for cross-fading plots, zooming in on code and images, and highlighted parts of a text. 🤯 Quarto is an open-source publishing system developed by Posit PBC that allows users to create and share dynamic, reproducible content like articles, presentations, dashboards, and books using #Python, #rstats, #Julia, and more, in various formats including HTML, PDF, and Word. The screengrab shows the example of Edward Tufte's description of Charles Minard's famous visualization of Napoleon's disastrous military campaign in Russia from the closeread documentation. I can't wait to try this out! #PositConf2024 #DataScience #dataviz

436

42 Comments

AI isn’t the end for data practitioners. Very much the opposite. Barry McCardel, the co-founder and CEO of Hex, is bullish on a new golden era of data teams, because, “There's an infinite demand for insight and an infinite demand for data work.” Barry joins Tristan Handy on the Analytics Engineering Podcast to discuss the hype cycle of data science, how AI is different, the experience of building AI products, and how AI will impact data practitioners. https://lnkd.in/d-hfmisr

53

You can’t properly treat a symptom without understanding its cause. Like a data doctor, you need to look at your data issues from all angles—identifying both the symptoms (what’s visibly wrong in your pipeline) and the causes (the underlying issues leading to those problems). That’s why we launched Fivetran observability to begin with: so you could start understanding which connectors were upstream of data quality issues. Today, we’re rolling out another huge enhancement as part of our Summer of Data Observability: observability on Fivetran metrics like connector duration and freshness. Our new dashboard puts connector performance front and center, so you can catch ETL issues as they happen. Now, you can: • Explore how long each connector takes • See how many rows Fivetran synced • Check for unusual connector performance spikes • Consolidate alerts in one place Check out the blog post in the comments. Can you guess what we’re launching next? 👀 #dataengineering #dataobservability #data

24

1 Comment

🦺 New Blog 👷 "Why a Semantic Layer is Critical in Manufacturing: Moving from Information to Insights with AtScale" The manufacturing industry is at a pivotal moment, with data volumes expected to grow 200-500% in the next five years. This surge in data presents both challenges and opportunities. 🌐 AtScale's Kieran O'Driscoll highlights in his recent blog that the most important revenue-generating or cost-saving workloads depend on massive data sets. This is particularly true for manufacturing, where data explosion outpaces other industries by 2-4x. Click here to read his blog: https://lnkd.in/dUJMaU-T

23

The value of vector indexes is rising for Generative AI ... why and what are vector indexes? As Gen AI gains traction, Retrieval-Augmented Generation (RAG) is emerging as a key approach for adding value. RAG enhances language models with external knowledge retrieval for improved accuracy and domain-focused applications. At the core of RAG lies its most critical component, efficient knowledge retrieval, which happens via vector indexes. In other words, how fast can the solution find the meaningful relevant information from the additional data provided? With vector databases, there are a number of options for indexing, including: 🌲 Tree-based indexing 📊 Graph-based indexing (e.g., HNSW) #️⃣ Hashing-based indexing (e.g., LSH) 🗜 Quantization-based indexing Each has its own set of benefits and challenges, which is why selecting the right vector index is crucial for RAG solutions. When selecting options, it is a challenge of balancing factors like dataset size, dimensionality, and accuracy-performance trade-offs. As Generative AI evolves, vector indexes will play a pivotal role in enabling accurate, knowledge-grounded AI applications. Read the full post on Vector indexing here: https://lnkd.in/egDjCxYy What has been your experience working with a RAG solution? #GenAI #AI #RAG #GenerativeAI #VectorIndexes #VectorDatabase #Data

12

If you've worked in data long enough, then you've likely seen a few different data modeling methods. It's probably ranged from 0 data modeling to a strict data modeling process that only allows new tables or columns in every 3rd week of the month. So how does all the theory you've learned apply to the various ways companies model their data. In this video I wanted to discuss the data modeling techniques I have seen used in companies that range from start-ups to big tech. If you enjoy this video consider re-posting! https://lnkd.in/gXwjDUxV

42

1 Comment

Ready to bring consistency to your metrics with the dbt Semantic Layer? Join one of our live, 2-hour interactive courses on Tuesday, April 30th at 12pm ET or Wednesday, May 1st at 1:30pm AEDT! Designed to improve your data project management, you'll gain a clear understanding of what the dbt Semantic Layer is, its purpose, and when to use it in your projects. Here's what you'll get by attending live:  ✅ Discover how to query the dbt Semantic Layer from Google Sheets and other BI tools, enhancing your analysis and data-sharing capabilities. 📈 Learn how to set up entities, dimensions, and measures in your semantic models and how to effectively configure and use metrics. 🤝 Get live support (because we've got your back) —ask questions, join discussions, and code along with us. Speakers include our resident dbt experts Faith Lierheimer, Alex Dinsmoor, Michelle Baird, and Joel Labes. Can't make it to the live event? No worries! Register anyway, and we'll send you the recordings and recap content so you can experience all the action on-demand ➡️ https://bit.ly/4aYKVqy

68

2 Comments

Why should Machine Learning teams use Feature Stores within Databricks? 🤔 This question and more were answered by Ben and Sepideh at Databricks when they spoke to Gavi in our most recent Advancing AI video: https://hubs.ly/Q02vyj4R0 #MachineLearning #ML #AI #FeatureStores #Databricks

19

1 Comment

With a bespoke MMM model, you risk running into a nasty problem: you can't collect enough data to understand what’s going on. So you have to make a choice: your data scientist has to 'choose' the right story. This makes your model fragile in the long-run and not representative of actual marketing dynamics in the short-term. It also makes it highly sensitive and unstable — all of which are bad outcomes. How can a generalised model like ours give you better, clearer and more reliable results? Read the full interview with Mutinex Co-Founder and CEO Henry Innis at the link below. It’s the best decision you’ll make. Discover the best way to predict outcomes from inputs. https://lnkd.in/gEaBQqWM

9

Data Engineering: Working with semi-structured data types. In our previous post, we demonstrated how to parse comma-separated data into a structured file format within the Snowflake project that utilizes the medallion architectural framework. In this post, we’ll introduce another file format that you’ll frequently encounter in your data engineering projects: the JSON File Format. JSON (JavaScript Object Notation) is an open standard file format used for sharing data. It employs human-readable text to store and transmit data objects, which consist of attribute-value pairs and arrays. This file format is commonly used for transmitting data between web applications and servers, making it very popular and suitable for our discussion and demonstration. To explain in more detail, we have three common types of JSON: JSON objects, JSON nested objects, and JSON arrays. JSON objects: “User_Name” :{ “Fist_Name”: “Henry”, “Last_Name”: “Godson” } Nested JSON objects: “User_Informations”: { “Date”: “2024-05-21”, “Sports”: {“Football”: “Good”, “Swimming”: “Excellent” } “Name”: “Henry } JSON arrays: (JSON are written inside the square brackets). “Employees”: [ { “First_Name”: “Henry”, “Last_Name”: “Godson” }, { “First_Name”: “Endy”, “Last_Name”: “Junior” }, ] We also have JSON Array in the simple format as “Favourit_Sports”: [“Football”, “Baseball”]. While certain data migration tools, such as Azure Data Factory, possess built-in intelligence for transforming data from one format to another, most data engineering ETL projects necessitate data engineers to perform similar transformations using either SQL or Python. To transform a JSON array using the Medallion Architectural Framework, we have two major approaches: Copy to Bronze and Transform: In this approach, we copy the JSON file into the Bronze layer and then transform the data to a structured format before moving it into the Silver table. Direct Transformation to Bronze: The second approach involves transforming the data directly to a structured format and depositing it into the Bronze layer straight from the staging environment. This approach is recommended, and we’ll demonstrate how to achieve it in our upcoming post. Stay tuned!

3

Mapping Workbench (MWB) has profoundly impacted our ability at Meaningfy to deliver high-quality solutions to our clients. But the implications of this tool go far beyond one organization. Therefore, we aim to transform how semantic engineers approach data mappings across industries. ✅ Increased productivity One of MWB's most significant benefits is increased productivity. By automating many aspects of the mapping process and providing a unified platform for creating and validating mappings, the tool has dramatically reduced the time it takes to complete complex mapping projects. ✅ Confidence in deliverables MWB has also increased confidence in the quality of the deliverables. Semantic engineers can be sure that their mappings are correct and conform to the target ontology, while business stakeholders can trust that the meaning of the data has not been altered during the transformation process. ✅ A solution for complex mappings Many semantic engineers face the complexity of mapping XML data to ontologies. MWB provides a solution tailored to these challenges, offering a tool that simplifies the process and ensures that the resulting mappings are accurate and reliable. ✅ Ensuring semantic interoperability MWB provides a structured methodology for creating and validating mappings and sets a new standard for how semantic engineers approach this challenging task. Mapping Workbench makes mapping XML/JSON data to OWL ontologies more efficient, accurate, and collaborative. Our tool helps organizations with large-scale semantic mapping projects, enabling domain experts and technical teams to collaborate smoothly. We invite you to try it for free and explore its capabilities 👇 https://lnkd.in/ecZ26-6h

24

Snowflake Agrees to Acquire Open Data Integration Platform, Datavolo. Empowering data engineers with hybrid, multimodal pipelines, and open source flexibility. Connectors incoming!

50