Everybody Lies

Prepare to have your understanding of human behavior fundamentally reshaped. In 'Everybody Lies,' data scientist Seth Stephens-Davidowitz reveals a hidden world of truth, unlocked not by traditional surveys or polls, but by the raw, unfiltered confessions found in our own Google searches. This book is your invitation to step beyond the polite fictions we tell each other and ourselves, and to confront the surprising, often uncomfortable, realities of who we truly are when no one is watching. You'll discover that the patterns we intuitively spot are the bedrock of data science, and that even the most arcane psychological theories, like those of Freud, can be tested and illuminated by the sheer volume of digital information. Stephens-Davidowitz demonstrates how Big Data isn't just about numbers; it's a powerful lens, offering granular, anthropological insights into everything from societal biases to personal desires. You'll learn to distinguish between the convenient narratives we present and the deeper truths revealed by our digital footprints, understanding why people lie and how to detect it. This journey will equip you with a new way of seeing the world, one that moves from vague correlations to clearer causation, and challenges conventional wisdom with empirical evidence. While acknowledging the limitations and ethical considerations of Big Data, the core promise of this book is profound: to offer you a more accurate, albeit sometimes shocking, portrait of humanity, empowering you with a deeper, data-driven comprehension of yourself and the society around you. The tone is intellectually exhilarating, consistently surprising, and ultimately, deeply illuminating.

Seth Stephens-Davidowitz, an expert in internet data, unveils a hidden landscape of human behavior, challenging conventional wisdom with the raw, unfiltered truths found in Google searches. He begins by recounting the surprise of the 2016 election, where polls predicted Donald Trump's defeat, yet the internet whispered a different story. This divergence, he explains, stems from a fundamental flaw in traditional data collection: people often say one thing and do another, especially on sensitive topics. His journey into this digital underground started with the 2008 election and the question of racial prejudice; while surveys suggested America had moved past race as a factor in voting, Google searches revealed a starkly different reality. The author discovered that search terms like 'nigger' were surprisingly common, often linked to racist jokes or expressions of hate, particularly in regions that polls overlooked, suggesting a hidden undercurrent of animosity. This stark contrast between public declarations and private searches extends to other areas, such as sexual behavior, where survey data wildly conflicts with the reality of condom usage, and marital satisfaction, where 'sexless marriage' is a far more frequent search than 'unhappy marriage.' Stephens-Davidowitz argues that Google searches act as a confessional, a space where people reveal their true desires, fears, and prejudices when they believe no one is watching. He illustrates this with surprising findings: economic insecurity doesn't necessarily correlate with increased racism, anxiety is higher in less educated rural areas, and humor is sought not in times of sadness, but in moments of contentment. These insights, initially met with skepticism by academic peers, gain powerful validation in the wake of events like Trump's election, where the 'secret racism' uncovered by his data played a significant role. The author posits that Big Data, particularly the vast trove of Google searches, offers a revolutionary lens through which to understand the human psyche, not just by its immense size, but by the honesty it often contains, provided the right questions are asked. He emphasizes that this new digital data is akin to a microscope for society, revealing profound truths previously hidden, and that understanding its power requires looking in promising places, like specific search queries, rather than just accumulating vast amounts of information. The chapter thus draws the outlines of a revolution in how we understand ourselves and our society, powered by the quiet confessions typed into search bars.

The author, Seth Stephens-Davidowitz, embarks on a journey to demystify data science, revealing that its core principles are not arcane mathematical formulas but rather an intuitive, pattern-spotting process that humans engage in daily. He illustrates this with the example of his grandmother, who, with a lifetime of observations, offers surprisingly astute relationship advice, acting as a human 'Big Data' analyst. This sets the stage for a central tension: if data science is so intuitive, why do we need computers and complex statistical tests? Stephens-Davidowitz posits that while our 'gut' can be remarkably insightful, especially in familiar territory, it often falters when confronted with the sheer scale and subtlety of real-world phenomena. He presents a striking study by Columbia University and Microsoft, where analyzing Bing search data revealed that a specific combination of symptoms—back pain followed by yellowing skin, or indigestion coupled with abdominal pain—could predict pancreatic cancer with remarkable accuracy, a pattern far too nuanced for unaided intuition to detect. This highlights a key insight: the power of Big Data lies not in replacing human intuition, but in augmenting it, providing a lens to see patterns invisible to the naked eye. Our gut can deceive us, as demonstrated by the common misconception that NBA players predominantly come from impoverished backgrounds; data, however, reveals a counterintuitive truth: a more stable, middle-class upbringing, with better healthcare and nutrition, actually provides a significant advantage. This is further supported by analyzing player names, which often reflect socioeconomic status, and observing that traits nurtured in stable environments, like discipline and social skills, are crucial for success, even for those with immense talent like Doug Wrenn, whose career was derailed by an inability to navigate relationships, unlike Michael Jordan, who benefited from strong parental guidance. The author concludes that while our intuition serves as a foundational tool, it is often imprecise and prone to biases, blinding us to the true workings of the world. Big Data, therefore, acts as a powerful amplifier, injecting rigor into our natural pattern-seeking tendencies and often revealing a reality far more complex and counterintuitive than our 'faulty gut' might suggest, urging us to embrace data not as an intimidating force, but as an extension of our own innate capacity to understand.

Seth Stephens-Davidowitz embarks on a fascinating journey to test the enduring theories of Sigmund Freud, not in the hushed confines of a therapist's office, but within the vast, often unfiltered landscape of Big Data. For decades, Freud's ideas about repressed desires manifesting in dreams and Freudian slips have been met with a knowing shrug, largely because they defied falsification – how could one definitively prove or disprove a subconscious thought? Karl Popper's critique echoes through the years, highlighting the 'he-said, she-said' nature of psychoanalytic interpretation. Yet, the digital age, with its torrent of anonymized information, offers a new frontier. Stephens-Davidowitz first turns his analytical gaze to dreams, examining a massive dataset of dream recordings. He discovers that the foods we dream about are overwhelmingly predicted by how frequently we consume them and how tasty we find them – think chocolate and pizza, not phallic symbols. Bananas and cucumbers, while potentially suggestive in shape, appear in dreams in direct proportion to their presence in our diets, a notion that feels decidedly un-Freudian. The author then pivots to Freudian slips, those 'penistrians' and 'sexurities' that seem to betray hidden desires. By creating an 'Error Bot' that mimics human typing errors based on statistical letter-switching frequencies, Stephens-Davidowitz demonstrates a crucial point: the bot, devoid of a subconscious, makes just as many 'sexual' errors as humans do. This suggests that many of these slips are not revelations of repressed urges, but rather the inevitable byproduct of making millions of tiny mistakes, like a monkey eventually typing Shakespeare. The author, however, finds a surprising echo of Freudian thought in a most unexpected place: the search data from PornHub. A significant number of searches, particularly among men, reveal a fascination with incestuous themes, especially mother-son relationships, echoing Freud's Oedipal complex. Google search data further supports this, showing a striking number of searches expressing attraction to mothers. While acknowledging that this data doesn't definitively prove fantasy versus reality, and that such desires are not widespread, it points to a persistent influence of childhood and maternal figures on adult sexuality. Stephens-Davidowitz posits that Big Data offers four unique powers: providing new types of data (like the raw honesty of porn and search logs), offering honest data by protecting anonymity, allowing us to zoom in on tiny subsets of the population, and enabling rapid, controlled experiments for causality. Ultimately, while many of Freud's specific interpretations may not hold up under the harsh light of Big Data, the author concludes that the Viennese psychologist was not entirely off the mark, particularly in his emphasis on the profound and lasting impact of childhood and maternal relationships on adult sexuality, a theme that modern data is beginning to illuminate with unprecedented clarity.

Seth Stephens-Davidowitz, in 'Data Reimagined,' invites us to witness a profound shift in how we understand the world, moving beyond traditional, often delayed, datasets to embrace the vast, unconventional streams of information now at our fingertips. He begins by illustrating the immense value placed on even milliseconds of data in the financial world, where firms spend fortunes to shave time off the delivery of unemployment figures—figures that are themselves weeks old by the time they're released. This stark inefficiency sparks a central question: must we always wait for slow, established methods when new, more immediate data exists? The author then unveils the power of Google searches, revealing how seemingly trivial queries like 'Slutload' or 'Spider Solitaire' can, in aggregate, offer a surprisingly accurate, real-time pulse on unemployment, demonstrating that the true power of big data lies not just in its size, but in its novelty. This principle extends beyond the digital realm, as seen in the story of Jeff Seder, who revolutionized horse racing by abandoning traditional pedigree analysis for rigorous data collection, discovering that the size of a horse's left ventricle was a far greater predictor of success than lineage. Seder’s journey, from measuring nostril size to dissecting internal organs, underscores a crucial insight: the most impactful data often resides in overlooked places, and the revolution is in reimagining what constitutes data itself. Similarly, the chapter explores how language, once a qualitative subject, has become quantifiable data through tools like Google Ngrams and sentiment analysis, allowing us to trace the evolution of national identity or predict the success of a first date by analyzing word choice and vocal patterns. Even visual information, from yearbook photos revealing shifts in social norms driven by marketing to satellite imagery tracking economic growth through night-time light, is now a rich source of data. The core tension throughout is the inertia of old systems versus the disruptive potential of new data; the resolution lies in embracing this reimagining, realizing that from the curvature of lips to the fullness of supermarket bins, everything is data, offering unprecedented insights and immense value, as exemplified by companies like Premise that leverage everyday photos to track economic performance long before official figures emerge.

The author, Seth Stephens-Davidowitz, unveils a profound truth: people lie. They lie to surveys, to friends, to family, and even to themselves, a phenomenon often driven by social desirability bias or a simple desire to present a polished, acceptable version of reality. This inherent human tendency, he explains, distorts traditional data collection, rendering surveys unreliable for understanding genuine thoughts and behaviors. Yet, in the vast, untamed wilderness of the internet, a 'digital truth serum' has emerged. Google searches, in particular, become a confessional, a place where individuals, alone and uninhibited, reveal their deepest curiosities, anxieties, and even prejudices. Stephens-Davidowitz illustrates this with striking examples: the disparity between reported and actual GPA, the underestimation of Donald Trump's support in polls, and the unexpected prevalence of searches related to personal struggles like depression or regret over having children. He then pivots to the complexities of human sexuality, using search data to offer a more accurate estimate of the gay population than surveys could provide, revealing the vastness of the 'closet' in less tolerant regions. The narrative shifts again to explore darker corners of the human psyche, analyzing searches for hate speech and prejudice, demonstrating how even presidential appeals for tolerance can, paradoxically, inflame online animosity. The author then challenges the common notion of internet-driven political segregation, presenting data that shows liberals and conservatives actually encounter opposing viewpoints online more frequently than offline, often driven by a desire to argue or simply understand the other side. He further reveals how Google data can expose hidden crises, such as the rise in child abuse searches during economic downturns or the increasing interest in self-induced abortions when legal access is restricted. Finally, Stephens-Davidowitz contrasts the curated self-presentation on social media, which he likens to 'digital brag-to-my-friends-about-how-good-my-life-is serum,' with the raw honesty of search queries. He concludes that this 'digital truth serum,' while often revealing uncomfortable realities about insecurity, prejudice, and suffering, offers invaluable insights. This knowledge, he argues, can provide comfort in shared vulnerability, alert us to hidden crises, and, most powerfully, guide us toward more effective solutions by understanding what truly motivates human behavior, both spoken and unspoken, thereby helping us address the world's problems with greater clarity and empathy.

The author, Seth Stephens-Davidowitz, invites us to explore the power of Big Data, not just as a tool for aggregation, but as a lens for granular, almost anthropological insight, revealing the hidden forces that shape our lives. He begins by contrasting his own lifelong obsession with the New York Mets, a passion he traces back to his childhood triumphs in 1969 and 1986, with his brother Noah’s complete indifference, a divergence that highlights a central question: what truly determines our adult identities? Through analyzing Facebook likes, Stephens-Davidowitz illustrates a profound principle: for males, the age of eight is a critical window for cementing lifelong sports allegiances, a phenomenon with parallels in how political views are formed, with age eighteen proving pivotal during periods of presidential popularity, echoing Winston Churchill's adage but offering a data-driven nuance. This ability to 'zoom in' on specific age cohorts, a capability only truly unlocked by Big Data, allows us to move beyond broad generalizations and uncover subtle yet powerful influences, much like how Raj Chetty’s team, by analyzing vast IRS tax records, discovered that the American dream of upward mobility is not uniform, but a patchwork of distinct regional opportunities. The data revealed that where you are born significantly impacts a child's chances of escaping poverty, with factors like educational spending, religiosity, and community demographics playing crucial roles, and surprisingly, that the presence of wealthy individuals in a city can extend the life expectancy of its poorer residents, possibly through the contagion of healthier habits. Further zooming in, the author delves into the surprising finding that violent movies, contrary to anecdotal evidence and lab experiments, appear to correlate with a drop in crime, a phenomenon explained by the simple fact that potentially violent young men are occupied in movie theaters rather than on the streets, and importantly, that the absence of alcohol sales in theaters contributes to this effect, demonstrating how Big Data can reveal counterintuitive truths by examining hourly crime patterns. The chapter also introduces the concept of 'doppelganger' searches, pioneered by Nate Silver in baseball analytics, which involves finding individuals with similar career trajectories to predict future performance, a method that correctly advised patience with David Ortiz despite his declining numbers, showcasing how identifying similar patterns can yield profound predictive power. This 'doppelganger' approach, when applied to broader datasets like Twitter profiles, can help us understand individual preferences and even uncover surprising connections, like the author's own 'doppelganger' being a bot, or how companies like Amazon and Netflix use similar techniques for personalized recommendations. Ultimately, Stephens-Davidowitz argues that Big Data's true power lies in its capacity to zoom in, to reveal the intricate, often invisible threads that connect us, from the arbitrary facts of our birth year shaping our loyalties, to the geography of our upbringing influencing our destinies, and even the subtle contagions of habit and information that spread through communities, transforming our understanding of human behavior from static surveys to dynamic, story-rich narratives of the human condition.

Seth Stephens-Davidowitz, in 'All the World's a Lab,' unveils a profound shift in understanding human behavior, moving from the murky waters of correlation to the crystal clarity of causation, a journey ignited by a simple idea at Google in 2000. The author explains that while we're inundated with studies suggesting links—like moderate drinking and good health—these correlations often mask deeper truths: reverse causation, where good health might lead to drinking, or omitted variables, where social engagement could drive both. The gold standard for truth, he reveals, is the randomized controlled experiment, a method that meticulously isolates variables by dividing participants into treatment and control groups, much like Esther Duflo's groundbreaking work in rural India, where paying teachers for attendance dramatically halved absenteeism and boosted student performance, especially for young girls. This rigorous approach, however, was historically resource-intensive, until the digital age transformed 'the world into a lab.' Google engineers, by randomly showing users ten versus twenty links, demonstrated that such experiments could be conducted cheaply and rapidly online, a practice now known as A/B testing. This revolutionized Silicon Valley, allowing platforms like Facebook to run thousands of tests daily, optimizing everything from ad colors to email timings, often leading to addictive designs, as Tristan Harris points out, where 'a thousand people on the other side of the screen whose job it is to break down the self-regulation you have.' Yet, the digital realm isn't the only laboratory; nature itself provides potent experiments. The unpredictable outcomes of sporting events, like the Patriots-Ravens game, can reveal the true causal impact of Super Bowl advertising, showing that companies are often dramatically underpaying for ads that yield significant returns. Even the arbitrary nature of life—a bullet narrowly missing a vital organ, or a slight difference in a test score—offers economists invaluable data. The comparison of leaders who survived assassination attempts versus those who didn't, or students just above and below a high school's admission cutoff, like Stuyvesant, reveals that inherent talent and drive, not just the 'brand' of an institution, are the primary drivers of success. This regression discontinuity design, for instance, showed that attending elite schools like Stuyvesant, or even Harvard, had no statistically significant causal effect on long-term earnings compared to similar students attending less prestigious institutions. Ultimately, Stephens-Davidowitz guides us through the tension between our intuitive assumptions and the often counterintuitive reality revealed by data, resolving that by embracing controlled and natural experiments, we can move beyond misleading correlations and 'shoddy correlations with what actually works—causally.'

The author, Seth Stephens-Davidowitz, recounts an intellectually exhilarating meeting with Lawrence Summers, former Treasury Secretary and Harvard President, who, after reviewing his paper on racism's impact on Obama's support, pivots to the tantalizing question: can Big Data, specifically Google searches, predict the stock market? This encounter forms the crux of the chapter, exploring the profound limitations of Big Data, a counterpoint to its celebrated power. Stephens-Davidowitz reveals that despite the allure of vast datasets, the stock market proved an elusive target, mirroring a common human desire to profit from expertise in areas like racism or child abuse, a desire that even his own father expressed. The core tension arises from the fierce competition in fields like finance, where massive resources are already dedicated to finding even the smallest edge, a reality that hedge funds like Renaissance are deeply immersed in. This leads to a central concept: the 'curse of dimensionality.' Imagine flipping a thousand coins for two years and finding one, Coin 391, that statistically predicts market movements – a phenomenon likely born of sheer chance when faced with numerous variables and limited observations. This insidious effect, where random correlations are mistaken for genuine predictors, ensnares many who claim to harness Big Data, as seen with a tweet-based hedge fund that quickly folded after its 'calmness' predictor, derived from too many tests on limited data, failed to deliver. The same pitfall affects genetic research, where initial claims of genes predicting intelligence, like Robert Plomin's finding about IGF2r, are often retracted when new data fails to replicate the correlation, a stark reminder that testing millions of genes against limited observations inevitably yields spurious results. The resolution to this 'curse' lies in humility and rigorous 'out-of-sample' testing, demanding more rigorous validation as the number of variables increases, and meticulously tracking every test performed. The narrative then shifts to the 'overemphasis on what is measurable,' illustrated by Zo Chance, a marketing professor so fixated on her pedometer's step count that she lost sight of her original goal of exercise, walking incessantly and even placing the device on her child. This anecdote underscores how easily numbers can become seductive, leading to the neglect of more important, less quantifiable aspects of life and work, much like the focus on test scores in schools overshadows critical thinking and curiosity, or how early baseball analytics undervalued defense because it was harder to measure. The chapter concludes that the solution is not abandoning Big Data, but complementing it with 'small data'—human judgment, intuition, and traditional surveys—as demonstrated by Facebook's use of social psychologists and anthropologists, and even the Oakland A's increasing their scouting budget. Ultimately, Stephens-Davidowitz posits that Big Data and small data are not adversaries but complementary forces, each filling the gaps left by the other, a synergy exemplified by horse guru Jeff Seder and his collaborator Patty Murray, who combines data-driven insights with traditional, hands-on horse evaluation to sniff out subtle problems his algorithms might miss, reminding us that while Big Data is revolutionary, it does not negate the millennia-old human methods of understanding the world.

Seth Stephens-Davidowitz, in 'Everybody Lies,' delves into the unsettling power of Big Data, revealing how its insights, while often impressive, can breed significant ethical dilemmas and societal challenges. The author explains how a study on loan applications, utilizing data from Prosper, uncovered surprising linguistic predictors of repayment; phrases like 'debtfree,' 'lower interest rate,' and 'graduate' correlated with a higher likelihood of repayment, while 'God promise,' 'will pay,' and mentioning family members in distress, particularly for hospital needs, were indicators of potential default. This highlights a core tension: the statistical accuracy of these predictions versus the fairness of judging individuals on abstract, albeit predictive, criteria, raising the specter of a dystopian world where even a mention of a sick relative could be a mark against one's financial trustworthiness. Moving beyond finance, Stephens-Davidowitz explores the implications for hiring practices, citing research showing correlations between Facebook 'likes'—such as Mozart or thunderstorms—and higher IQs, and others like Harley Davidson or 'I Love Being a Mom' with lower IQs. This presents another complex dilemma: the potential for subtle, yet intrusive, discrimination based on seemingly harmless digital footprints, a danger amplified as more of our lives are quantified. The narrative then pivots to the insidious nature of price discrimination, where Big Data allows corporations to pinpoint and extract the maximum price a customer is willing to pay, illustrated by casinos using 'doppelganger searches' to identify individual 'pain points'—the loss threshold before a customer departs—and strategically offering incentives, like free meals, to keep gamblers playing just shy of that point, a practice exemplified by Harrah's and their data partner Terabyte. This underscores the escalating power imbalance between corporations and consumers. The chapter then shifts to the even more profound dangers of empowered governments, using the tragic case of Adriana Donato, murdered by her ex-boyfriend James Stoneham, who had researched murder methods online. While acknowledging that online searches for criminal activity, like 'kill Muslims' or suicidal ideation, do correlate with real-world events on a societal level—allowing for resource allocation, such as increased patrols around mosques or suicide awareness campaigns—Stephens-Davidowitz draws a critical ethical and data-driven line at intervening with individuals. He argues that the leap from predicting city-level trends to targeting individuals is immense and fraught with peril; for every four thousand suicides, there are millions of searches, and for thousands of 'kill Muslims' searches, there were far fewer actual hate crimes. The author emphasizes that while a theoretical possibility exists for future data science to identify high-probability threats, as suggested by the statistics around 'how to kill your girlfriend' searches, the current reality is that most horrifying searches rarely lead to horrible actions, making individual-level government intervention based solely on search data ethically dubious and, for now, statistically unreliable. Ultimately, the chapter presents a dual-edged sword: Big Data empowers consumers with information like Yelp reviews and price comparison sites, creating a more equitable fight against corporate overreach, but it also arms corporations and governments with unprecedented tools for surveillance, prediction, and control, urging a cautious approach to ensure this powerful data remains a force for fairness rather than exploitation.

Seth Stephens-Davidowitz's "Everybody Lies" serves as a profound testament to the revolutionary power of Big Data, fundamentally altering our perception of human behavior and societal truths. The core takeaway is that traditional methods of data collection, reliant on self-reporting, are inherently flawed due to our innate tendency to deceive, whether consciously or unconsciously. In stark contrast, the anonymous and uninhibited nature of digital footprints, particularly Google searches, acts as a "digital truth serum," exposing our deepest desires, fears, and prejudices that we would never otherwise reveal. This revelation is not merely academic; it has tangible emotional weight, forcing us to confront uncomfortable truths about ourselves and society, such as the pervasive undercurrent of racism or the hidden suffering masked by polite façades. The book powerfully illustrates that while our gut instincts and intuitive patterns are a natural form of data science, they are often limited by personal bias and a lack of scale. Big Data, amplified by computational power, offers a corrective lens, revealing counterintuitive correlations and causal relationships that defy conventional wisdom and even psychoanalytic theories. The practical wisdom gleaned is multifaceted: we learn to question our assumptions, embrace empirical evidence over anecdote, and understand the limitations of intuition when faced with complex phenomena. The book cautions against the "curse of dimensionality" and the over-reliance on easily measurable metrics, emphasizing the need for rigorous validation and humility. Ultimately, "Everybody Lies" equips us with a new paradigm for understanding the world, one that acknowledges the often-uncomfortable honesty of our digital selves and compels us toward a more objective, albeit sometimes darker, comprehension of human nature and societal dynamics. It underscores that the true revolution lies not just in the volume of data, but in reimagining what constitutes data and in asking the right questions to unlock its profound insights.