15.S03: Metascientific examination of social & behavioral sciences reflection note

[hyunjimoon]

syllabus

[hyunjimoon]

Week 2 (02/13) - Intro to philosophy of science
Dienes, Z. Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. (Macmillan International Higher Education, 2008).
Chapters 1 & 2
Manzi, J. Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society. (Basic Books, 2012).
Chapters 1-9

reacting to prompts

• Difference between Positivists and Popper reminded me of the different motivation and outcome of opt in vs out system. Positivists define meaning through verifiability (empirical) whereas Popper sets falsifiability as the demarcation criterion for science.
• Popper's emphasis is on the process of conjecture and refutation. The usefulness of a model lies in its ability to generate testable predictions and to be subjected to falsification, thus contributing to the growth of knowledge.
• Null hypothesis testing aligns with Popper's philosophy to some extent, as it involves formulating a hypothesis that can be empirically tested and potentially falsified. However, Popper might warn against relying too much on math (e.g. p-value), less on the critical process of falsifiability and the search for counterexample.

logging excerpts/summaries

• one of the Popper’s three important  implication, "theory precedes experiments" (Manzi12)
• "only way to be sure in this thought way to be sure in this thought experiment would be to do a test drop in every square inch that is, by enumeration rather than a causal rule that permits prediction" (Manzi12)
• Lakatos's work on scientific research programmes offer a framework that captures the continuity and evolution of scientific theories over time. His approach seeks to balance the empirical rigor of falsificationism with an understanding of the historical and contextual dimensions of scientific practice. single experiment or observation cannot decisively falsify a research programme because of the programme's ability to adapt through modifications in its protective belt "Lakatos, like Popper, rejected the Bayesian approach, but thought, like Bayesians, that theories were often gradually worn down by evidence and not directly falsified in one hit"

building diagrams

Reproducible, replicable, coherent, generalizable, practical research (I'll improve understanding of these terms starting from "Reproducibility and Replication: Popper's focus on falsifiability underscores the importance of reproducibility and replication in science. By prioritizing theories that can be rigorously tested and potentially falsified, it promotes a more robust and reliable scientific process, Incoherency: Kuhn's paradigms suggest that incoherency in scientific theories is part of the normal process of scientific evolution, leading to paradigm shifts that resolve anomalies, Generalizability: Falsifiability as a criterion ensures that scientific theories have broad applicability and are not just ad hoc explanations, promoting theories with greater explanatory power and generalizability, Practical Utility: Popper's philosophy implies that theories with practical utility are those that can withstand rigorous testing and falsification, ensuring that scientific knowledge is reliable and useful"), would most likely be a form that balances theory, phenomenon, measurement.

Generally, we are taught to iterate between the three (e.g. you observe some phenomena you try to measure them, you try to develop some theory. Then you go back and you observe the phenomena again to see whether your theory seems to be fitting to what's going on. Maybe you try to quantify the theory and take some more measurement. You take some measurement, you do your analysis, you see some parts of the theory that seemed to be confirmed.). However, instead of bottom up, we should try top down. We should have the joint distribution of theory θ , measurement (from experiments) $\tilde{θ}$, phenomenon y (justification of each representation would be unveiled and developed throughout the course) in order to avoid local locked in from "one at a time" approach. Below figure shows relationship between functions where solid line are equivalent expressions of full joint and dotted lines are approximation from the independence assumption between combinations of theory, measure, phenomenon. Tradeoff exist, when choosing the level of factorization. The more we factorize (from $p(\theta, y, \tilde{\theta})$ to $p(\theta)p(y)p(\tilde{\theta})$ ) the more replicable it becomes but less generalizable.

[hyunjimoon]

review
Scientific truth and the null hypothesis.

• misuse of null hypothesis testing in social sciences, citing high causal density as issue.
  In 2019, Abdullah (A) was an instrumentalist, believing that science should be useful for prediction and control.However in 2019, he became realist, believing in the existence of objective truth.

Scientific perspectives and definitions.

• difference between realism and instrumentalism in science, arguing that both perspectives are motivated by different goals but have the same day-to-day practices.
• Robin raises a clarifying question about the definition of science, struggling to articulate a clear understanding of what it entails. definition of science may not matter in practical applications, but it can matter for understanding what science is and what it entails.
  A agrees definition of science can be important for clarity and consistency in discussions about science, but notes that different perspectives on science can lead to different meanings of what science is.

The definition of science and its limitations.
Positivist view of science: empirical verifiability and operational definitions are key.
A argues that Lord of the Rings is a meaningful pursuit that has changed who they are in the world, but it's not science because it doesn't make predictions that can be refuted. scholars in the field often conflate science with other meaningful pursuits, such as discovering the truth or preparing statements about how the world works.
A emphasizes the importance of subjective understanding in science, distinguishing it from metaphysical pursuits.
A's personal belief in the meaningfulness of metaphysical pursuits does not make them science, despite their utility and consistency in daily activities.

Scientific realism and instrumentalism.
Speaker 3 discusses the distinction between pneumonia(?) (something about itself) and phenomena (something we can operationalize or agree on), highlighting the importance of coherence between our representations and the world.
Speaker 3 suggests that valuing testing theories through verification or falsification can help align our representations with the world, rather than committing to an instrumentalist or realist approach.
Speaker 3 emphasizes the importance of predictive utility in science, prioritizing meta theories that provide conditions for using models in the right situations.
A balances instrumentalist and realist perspectives, recognizing the value of both approaches in different contexts and stages of science. discusses pragmatic methodology in science, prioritizing practicality over idealistic views.

The nature of science and its goals.
Speaker 3

• discusses the potential for a unifying theory in physics, citing the example of quantum field theory and the idea that all forces may be connected.
• expresses skepticism towards a single explanation for human behavior, emphasizing the importance of systematic testing and the limitations of making assumptions about reality.
• argues that defining science in terms of its goals may not be solvable, while Speaker 1 pushes for a more assertive stance on what science is and what it entails.

Scientific goals and perspectives.
Speaker 4

• distinguishes between learning in the sense of discovering new information (e.g., writing a fictional novel) and learning in the sense of discovering new things that no one else knows (e.g., scientific research). A suggests that writing a novel can be a form of science if it involves learning new things that no one else knows, such as creating a new world with its own morals and lessons.
• shares their thoughts on the goal of science, considering both realist and relativist views.
  A reveals shift from realism to instrumentalism in social sciences, finding distinction less relevant in day-to-day work.
  Philosophy of science and prediction.
  Speaker 5 asks Joanna and Abdullah follow-up questions related to their views on the changing nature of rules and the compatibility of relativism and instrumentalism in dealing with this dynamic nature.
  Speaker 4 expresses uncertainty about how to reconcile relativism and realism in the context of changing rules, but acknowledges that a realist view may be more consistent with a system of relative truths.
  A emphasizes the importance of making accurate predictions in science, citing it as the primary way to evaluate the truth of a theory. distinguishes between their day-to-day activity as an instrumentalist and their long-term views on reality, highlighting the importance of prediction and control in scientific inquiry. emphasizes the importance of understanding the goal of science

---

• just "rejecting" is Popperian, not entire NHTS
• used correctly, it's not useful (high causal density e.g. software, psychology) and when it's used incorrectly, it is useful (low causal density e.g. hardware, physics)
• beacons stances, which is to give us power to do things. So Silas is useful to the extent that it gives us prediction and control. I predict what will happen, and I can control things I can intervene and change things to change, what will happen.
• so there is a real world and there are statements for which they are about the world that are true or not.
• some point my current position is that it doesn't matter, and maybe you convinced me otherwise, it doesn't matter whether whether an instrumentalist or a realist.
• How do I know that my theory corresponds to reality, what mechanism, do I have that will tell me.

---

• does science require story (account of past events in someone's life or in the evolution of something) - understood by human cognition

• generalizability (explanation should be understood)

• not satisfied with humility (nature is so complicated and nuanced) how it works (given input -> output)

• gravity acts on donkey whether they understand

• capability, no use
  science and explaination comprehensible
  (systematic study of the structure and behavior of the physical and natural world)
  realist (alignment) vs relativist (no truth, consensus) vs instrumentalist
  goal of science vs non-science (testability; misaligned)
  realist (objective truth); verifiability (assuming induction - )
  doesn't falsify null hypothesis rejects
  0 the goal of science is giving scientist a job (socialogical, like entrepreneurs who make jobs for themselves as they enjoy entrep.'s lifestyle); people willing sign up to be scientist - science has a value; to provide humanity joy of learning (ALGORITHM GAMES evolutionary is bridge; game, challenge)

• https://www.cs.virginia.edu/~robins/YouAndYourResearch.html
  internal and external coordinate, whether the agent is doer or thinker

2. "human has tendency to overfit" - who judges?? (liar's paradox)

wouldn't there be a way to frame scientist hat as helpful for engineering goal? e.g. escape from local optima via longer shot goals?
scientist thinker (realist) vs engineer do-er (instrumentalist, local optimal)
popper (context of discovery and justification), kuhn, lakatos (indifferent to theory generation)
envy - tradeoff (complexity of theory and correctness of prediction - go with the later)
quantum: complicated, accurate
map is not territory how much we can simplify (to be useful)
simpler theory better diffusion (out of distribution generalizability)
complexity of theory and make out of dist prediction (meta scientist);
utility of like doing it, useful and
science (verb) - action coming up with representation of how word works
science is intrinsically social (what and how to measure)

[hyunjimoon]

raw

measurement issue (instrument; realist (limit of prediction))

realist psychi (learn about world), behaviro (instrument)

mohammend (instructemnatlist),

jaeyoon (realist acknowledging social view is hard to measure)

encompasses the best test for a science is engineering (sulitan) - the more (figure out; closer to pre)

engineering before physics (intuit) - construction; generalizing the engineering

theory (characteristic, can be simple, mechanistic, observational statements)

[hyunjimoon]

Week 3 (02/20) - What is wrong with our sciences? (part 1)
Meehl, Paul E. "Theory-testing in psychology and physics: A methodological paradox." Philosophy of science 34.2 (1967): 103-115.
Newell, A. You can’t play 20 questions with nature and win. in Visual Information Processing (ed. Chase, W. G.) (1973).
Meehl, P. E. Why Summaries of Research on Psychological Theories are Often Uninterpretable. Psychol. Rep. 66, 195–244 (1990)
Lykken, D. T. What’s wrong with psychology anyway. Thinking clearly about psychology 1, 3–39 (1991)

Extending "Testing Entrepreneurial Hypothesis and Experiment" #159 (comment) I fleshed out need, solve, approximate (NSA) triplet framework. (phenomenalize, theorize, measure) is one example of NSA relationship and so are (invent, startup, commercialize), (need, customer, desire), (solve, technology, make feasible). My hypothesis is "the more uncertainty level of need, solution, approximation are balanced, the higher the product's quality becomes". Sequence of NSF actions, affect the uncertinaty level. The product can be research/service/commodity of individual/community.
Product development process consists of first identifying the current state then identifying action sequences. To visualize, imagine a unit cube where each axis is need, solve, approximate. Vector from its state to diagonal where need=sol=fulfill is be one example of action sequence.

I conducted a data analysis to illustrate product development process under NSF. Product is "community's science/knowledge evolution". Using this prompt, which interpreted data from this blog on Meehl's philosophy of science piece + 1k lines of comment with NSF framework, I got the table evaluating how much each group (related to the challenges in scientific research) is based on phenomena, theory, and measurement:

Group	Phenomena	Theory	Measurement
1. Scientific Integrity and the Evolution of Standards	High	High	Medium
2. Economic and Institutional Drivers	High	Medium	Low
3. Impact Factors and the Measurement of Scientific Impact	Medium	Low	High
4. The Role of Education and Training in Shaping Scientific Practices	Medium	High	Low
5. Moving Forward	High	High	Medium

Main prompt is

could you summarize Andrew Gelman's ##blog and comments? classify them based on whether comment is mainly based on theory or measurement or phenomena. for instance,

below is theory based as it uses logic to argue everything is correlated.comment "One of Meehl’s repeated points was that nearly everything is somewhat correlated – at least, in real systems with some degree of complexity. I just did a little exploring that illuminates that.Take two straight line x-y plots, and let line 2 have 1/10 the slope of line 1. Their correlation coefficients (both Pearson and Spearman rank) are 1.0. Square line 2 so it becomes a quadratic. Now Spearman’s rank correlation is still 1.0, and Pearson’s is 0.968. Adding noise will reduce these values, of course, but not eliminate them. Since nearly all curves will have some slope, then nearly all quasi-monotonic curves will show correlation. The actual value will depend mainly on the noise. Thus weak (or even not so weak) correlations have little to do with anything more than an practically inevitable non-zero slope of a line."

Below is phenomena based as it is based on a quote which is interpretation of several observations
"Lydia on April 3, 2017 12:21 PM at 12:21 pm said:
Not exactly an answer to your question but Popper quotes a chemist – Joseph Black (Lectures on the Elements of Chemistry) as saying this: “A nice adaptation of conditions will make almost any hypothesis agree with the phenomena. This will please the imagination but will not advance our knowledge.”"

Below is measurement as it is based on a concrete fact.
"Keith O'Rourke on May 12, 2016 9:10 AM at 9:10 am said:
Meehl does seem to set the Hubble hypothesis aside, but here is I think an excellent example of the Hubble hypothesis http://www.sciencedirect.com/science/article/pii/S0895435610001381
What happened here is that the supposedly preferred adjustment method in SAS to remove artifacts from the plot was an implementation of a misconstrued formula that actually added artifacts. Human error had resulted from a lack of mathematical/statistical understanding of the technique.
Now I had a real challenge convincing the authors of the paper I worked with that something was wrong with the math as they had done a literature review of papers using this technique (correspondence analysis) and ever paper had used the preferred adjustment (I think they mentioned they reviewed over 100 papers) – not everyone could be wrong – right?
The collection of published papers that used SAS and the preferred adjustment all have the Hubble’s lens flaw and have studied illusory phenomena.
If they provided what is called a Burt matrix, which is a sufficient statistic, the Hubble’s lens flaw can be corrected.
If they did not, they are a relevant subset of published papers that are 100% known for sure not to be reproducible!
"

Some noteworthy comments are:

• it started with an influx of very large quantities of mostly Government money into research. people disbursing the money didn’t have the inclination nor incentives to really care about the quality of the product that came out. The problem became worst in fields where judging the quality of the product was indirect and hard. e.g. Social Sciences.
• Upton Sinclair: “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”

[hyunjimoon]

Building on last week's figure; below is a cube (of our community's science/knowledge evolution) on three dimensions. Each vertex is low and high levels of phenomena, theory, measurement. I placed five above inside the cube. Purple, green, red represent phenomena, theory, measurement.

[hyunjimoon]

(TBC)
can NSF triplet be expressed and tested with Bayesian workflow?
Imagine observed data as need, approximator (computational algorithms) as fulfillment, statistical model p(theta, y) as need solution pair. Using computational representation via SBC code (architecture) and two examples I learned:

(a) we should conceptually separate function (for need and solution, approximate, evaluate) and their object (paired need-sol fulfilled need-sol evaluated fulfilled need-sol)
(b) need, fulfillment, evaluating function can be developed in parallel

Based on the second row, we can draw precedence diagram:

got the precendence graph as below:

The last line on "research product" was the goal of this excercise but need to be completed after class today!

[hyunjimoon]

review

• temporal validity; theory of confirmation bias (evol, rl, etc)
• don't avg over method treatment (heterogenity of treatment eff)

And what do they suggest? So like? The replication crisis is mostly driven by type one error, things that are not true are being published as to
reproduction crisis (positive results, can't replicate - high false positive (typ1 err), p-hacking, publication bias, pre-registraation (decide what ) not conditional on data you use; robustness, open materials

I can't reject the null hypothesis, because my P value is large. I can't reject the null hypothesis. But actually there is an effect
increasing sampling error
is making at least two subtheory on A1: Xhat measuring X well and A2: Yhat measuring Y well (TA1A2)
about is intelligence, then I'm claiming that IQ measures is a valid measurement of intelligence. And it's more valid measure of intelligence
pysics are point prediction, no true zero

Because of high costs and density. We take that kind of to be true. And that has big consequences on the practice of null hypothesis testing as you increase the available sample size, precision of our experiments measurement on all of these other things, it has this weird effect on it, and everyone tend to fall I talked to them a sacrament or any social science here. Of course, everything affects everything. Of course there is high cause and density. Of course, there isn't like one cause and one effect.

1. rejection of null != support of substan. theory
2. null hypothesis is never true (high causal density; every thing affects everything (so NO one at a time))

And then over as you increase, or you reduce your type2 error, you increase your precision. converging to 5

Psychology, sampling, and scientific methods.
Students find it challenging to engage with suggested readings, despite their value in generating discussions.
Speaker 1 discusses sampling in psychology, arguing that it's important to be aware of potential biases and selection effects.
Psychologists have stronger methodological and meta views on doing science compared to sociologists and political scientists.
Generalizability and weak theory in social sciences.
Speaker 1 discusses the relevance of sociology and political science to the topic of generalizability in science, mentioning a particular exchange between Duncan Watson and Healy.
Speaker 4 discusses the importance of understanding recent solutions to problems, suggesting that sometimes having a basic understanding of the issue at hand is more important than delving into the complexities of the past.
Speaker 1 agrees, noting that while people have been criticizing null hypothesis testing for years, there are still few solutions proposed to address the issue.
Speaker 1 acknowledges the diversity of solutions to a problem and the need for creativity and exploration, rather than converging too early on a single solution.
Speaker 1 reflects on their own sabbatical experience, realizing that while there isn't much more to be said about the problem, there is much more to do in terms of demonstrating feasible alternatives and solutions.
Using machine learning to improve human decision-making.
Speaker 1 emphasizes the need for solutions-oriented thinking, rather than just discussing problems.
Speaker 4 suggests that doers may be more effective than thinkers in addressing complex problems, and that error management in space exploration can provide insights for how to approach problems.
Discussion on using AI to improve productivity in social and behavioral sciences, with examples from 2024.
Explaining AI predictions can improve human AI systems' performance.
Bayesian modeling in clinical decision-making.
Clinicians' mental models influence productivity, not just Bayesian updates.
The conversation revolves around the idea that people's biases and heuristics can lead to suboptimal decision-making, and how Bayesian modeling can help address this issue.
Speakers discuss the relationship between Bayesian updating and observable variables in AI research.
Speaker 1 discusses the limitations of measuring productivity using a specific task, as the task may not accurately capture the underlying concept of performance.
Speakers 9 and 10 discuss the potential for sampling error in generalizing results to a larger population, and the importance of considering both the subject side and object side of the experiment when evaluating measurement instruments.

Controlling type one error in scientific research.
Researchers discuss the null and alternative hypotheses for AI-assisted decision-making, including the potential for AI explanations to improve performance.
Speaker 1 discusses type one error in research, including how to reduce false positives and the replication crisis.
Researchers discuss ways to address type one error and the file drawer problem in psychology research, including preregistration, robustness checks, and lowering the significance level.
Speaker discusses challenges with P-value and replication in social and behavioral sciences, including limitations of assuming perfect replication and reducing type two error through standard error reduction.
Speaker 1 clarifies that holding one constant while changing another refers to compensating for potential errors, not ignoring them.
The importance of accurate measurement in scientific experiments.
Speaker 3 argues that changing operationalization can allow for more precise testing of a theory, but it can also change the theory itself.
Speaker 1 suggests that reducing the number of errors in an experiment can increase confidence in the results, but it may also lead to false conclusions if the errors are at odds with the data.
Speaker 1 argues that measurement devices are often flawed, leading to imperfect predictions of the construct being measured.
Speaker 1 and others discuss the importance of using better measures to reduce noise and improve precision in testing theories.
Validity of psychological measurements.
Speaker 1 argues that measuring gender and intelligence is complex due to the constructive nature of these concepts, and that self-report measures of gender may be subject to error.
Speaker 1 emphasizes the importance of validating measurement theories to ensure that the independent variable (x) is accurately measuring the theoretical construct of interest, and that the outcome variable (y) is a valid measure of the desired outcome.
Speaker 3 expresses concern about trusting background knowledge without proper validation.
Speaker 1 argues that background knowledge is essential for scientific inquiry, but it can also lead to problems if not properly managed.
Speaker 1 and Unknown Speaker discuss the challenges of locating and validating background knowledge in scientific research.
Scientific theories and their falsifiability.
Speaker 4 suggests decomposing the relationship between T and a into conditional parts to test for falsifiability.
Speaker 1 argues that falsifying a single prediction does not necessarily disprove a theory, as the problem of induction persists.
Speaker 1 criticizes colleague's theory for reducing falsifiability with ad hoc amendments.

Measuring intelligence and reducing bias in experiments.
Speaker 1 argues that IQ measures intelligence validly but not reliably.
Experimenters improve techniques to control variables that contribute to intergroup variation, such as randomizing order and controlling for demographics.
The difference in approach to null hypothesis testing between physics and social sciences is qualitatively different, leading to different conclusions.
Null hypothesis testing in social sciences.
Speaker 1 argues that social science lacks precision in theories and predictions, leading to uncertainty and unreliable results.
Precision in theories and predictions is a major difference between physics and social science, with physics making numerical predictions and social science making directional predictions.
Speaker 1 argues that rejecting the null hypothesis does not necessarily falsify the statement that the null is always false, as the precision of the experiment needs to be increased to infinity.
Speaker 1 discusses the limitations of null hypothesis testing in social sciences, including the conflation of null hypothesis testing with support for a substantive theory, and the subjective nature of the null hypothesis.
Experimental design and analysis.
Speaker 4 argues that increasing sample size and reducing standard errors can lead to falsification of a theory, while Speaker 2 suggests that everything in red could be interpreted as possible, in theory, increasing the sample size and decreasing the standard errors or the red area becomes.
Speaker 1 mentions "design hacking" in the analysis stage to show desired results (20:10).
Speakers discuss selective bias in reporting and editorial processes (21:12).
Psychology theories and their relevance over time.
Participants discuss challenges in psychology research, including low citation counts and the omission of problematic theories.
Speaker 1: Social sciences face challenges in keeping up with rapid change.
Theories in psychology are often redundant and explain the same phenomenon, with new theories emerging to replace old ones.
Treatment effect heterogeneity in social science research.
Theorist argues that scientific theories should be based on evidence, not just ticking boxes.
Speaker 1 criticizes meta-analysis methods for hiding heterogeneity of treatment effects, emphasizing the importance of considering individual differences in treatment outcomes.
Speaker 4 agrees, highlighting the need to multiply probabilities of different conditions and add them using the law of total probability to account for heterogeneity.
Speaker 5 suggests that simply pointing out problems is not enough to drive change, and that taking a contrarian approach can be more enjoyable than solving the problem.
Speakers 1 and 3 express frustration that potential solutions are not being presented with enough passion and conviction, and that the problems are being characterized with too much fire and little actionable advice.

[hyunjimoon]

Category	Aspect	Physics	Psychology
Nature of Problem
	State Space	Limited, well-defined dimensions	High-dimensional, unclear boundaries
	Reference System	Fixed spatial/temporal coordinates	No universal framework
	Transition Rules	Deterministic/clear probabilistic	Complex, context-dependent
Individual Factors
	Prior Impact	Minimal on observations	Shapes interpretation significantly
	Observer Effect	Independent of observer	Observer-belief dependent
	Measurement	Direct access to phenomena	Partial observation of internal states
Institutional/Social
	Training Data	Consistent across contexts	Varies by culture, time, situation
	Validation Method	Falsifiable point predictions	Directional effect detection
	Error Handling	Narrows with precision	Expands with complexity
Statistical Properties
	Null Hypothesis	Point prediction (x = x0)	Directional (x ≤ 0)
	Type 1 Error	False point prediction rejection	False directional effect detection
	Type 2 Error	Missing true deviation	Missing true effect
	Precision Impact	Reduces both error types	May increase Type 1 error
	Success Criterion	Surviving stringent tests - everything in red is interpreted as falsifying the theory - if xbar != xbar_0, probability we will find a supportive finding approaches 0 with increasing sample size	Any non-zero difference - everything in red is interpreted as validation of the theory - if x has no effect on y, probability we will find a supportive finding approaches 0.5 with increasing sample size

image for success criterion row:

[hyunjimoon]

Week 4 (02/27) - What is wrong with our sciences? (part 2) [slides], hier_need_sol_tool_table
Meehl, P. E. Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. The restoration of dialogue: Readings in the philosophy of clinical psychology. 523–555 Preprint at https://doi.org/10.1037/10112-043 (1992)
(Documentary) Richters, J. E. Incredible Utility: The Lost Causes and Causal Debris of Psychological Science. Basic Appl. Soc. Psych. 43, 366–405 (2021)
Oktar, Kerem. "Psychology remains marginally valid." Nature Reviews Psychology (2024): 1-1.
Feynman, R. P. Cargo Cult Science. in The Art and Science of Analog Circuit Design (ed. Williams, J.) 55–61 (Elsevier, 1998).

gpt_process given needs.pdf and solutions.pdf

my evaluation measure of today's class is Y/N of the last two columns (?/38)

#	Need (Summary from "needs.pdf")	Solution (Suggested in "solutions.pdf")	Example Tool	does angie understand Need, Solution column enough to accept/reject?	does angie understand Example Tool column (abstracted sequence of unit operations that has input as need, output as solution) to accept/reject?
1	Responsive Class Problem Summary: Difficulties in segmenting behavioral data into meaningful categories based on causally relevant attributes.	Solution: Adopt a more solution-oriented approach, directly tackling practical problems to refine theoretical models and methodologies.	A machine learning classification system that dynamically updates its criteria based on emerging data patterns.
2	Situation Taxonomy Problem Summary: Challenges in adequately classifying and sampling environments and situations for study.	Solution: Emphasize interdisciplinary collaboration to enhance the robustness and applicability of situational classifications.	A collaborative platform for cross-disciplinary researchers to contribute and refine situational taxonomies using consensus algorithms.
3	Unit of Measurement Summary: Disagreements on the necessity of genuine interval or ratio scales for certain types of statistical inference.	Solution: Leverage advancements in computational methods and big data to explore new metrics and scales.	A digital measurement framework that uses AI to propose new scales based on data analysis and pattern recognition.
4	Individual Differences Summary: The complexity of accounting for how individual dispositions are shaped and organized.	Solution: Utilize large-scale data and machine learning to better understand and predict individual variance.	A predictive analytics engine that integrates diverse datasets to model and forecast individual behavior patterns.
5	Polygenic Heredity Summary: The complexity of traits influenced by multiple genes, making causal inference challenging.	Solution: Integrate genetic data with social science research to elucidate the polygenic nature of behavioral traits.	A bioinformatics platform that correlates genetic markers with behavioral traits, facilitating interdisciplinary studies.
6	Divergent Causality Summary: Systems where minor differences amplify over time, making long-term predictions challenging.	Solution: Develop dynamic models that can adapt to initial conditions and incorporate stochastic elements.	A simulation software that models complex systems with feedback loops and varying initial conditions to predict divergent outcomes.
7	Idiographic Problem Summary: The unique aspects of individual personalities that challenge broad generalizations.	Solution: Encourage qualitative methodologies that capture the nuanced nature of individual experiences.	A narrative analysis tool that uses natural language processing to distill unique personal stories into broader themes.
8	Unknown Critical Events Summary: Critical but unobservable events that significantly influence personality development.	Solution: Enhance longitudinal studies with technologies that can capture real-time data, providing insights into critical events.	A mobile app that collects continuous, real-time psychological and environmental data for longitudinal research.
9	Nuisance Variables Summary: Variables that are systematic but difficult to measure or control, complicating causal analysis.	Solution: Implement complex statistical techniques like path analysis to better account for nuisance variables.	An advanced statistical analysis software that uses AI to identify and control for nuisance variables in complex datasets.
10	Feedback Loops Summary: The impact of an individual's behavior on others, creating complex interdependencies.	Solution: Use network analysis to study social interactions and feedback mechanisms.	A social network analysis platform that identifies and models feedback loops within communities or groups.
11	Autocatalytic Process Summary: Processes where the output is capable of accelerating the process itself, such as in depression.	Solution: Apply non-linear modeling to capture the dynamics of autocatalytic processes.	A computational modeling tool designed for studying non-linear dynamics and autocatalytic processes in psychological and social phenomena.
12	Random Walk Summary: The role of chance in determining life paths, complicating predictive modeling.	Solution: Integrate random walk models into social science research to better account for stochastic life events.	A Monte Carlo simulation tool that applies random walk theories to explore various life path outcomes based on chance events.
13	Sheer Number of Variables Summary: The overwhelming number of variables that can influence outcomes, making analysis difficult.	Solution: Employ high-dimensional data analysis techniques to manage and interpret the vast number of variables.	A big data analytics platform that employs machine learning algorithms to sift through and make sense of datasets with a high number of variables.
14	Importance of Cultural Factors Summary: The significant but hard-to-quantify impact of culture on individual development and behavior.	Solution: Promote cross-cultural studies and comparative research to elucidate cultural impacts.	A comparative analysis software that allows researchers to systematically compare and contrast cultural factors across different societies.
15	Context-Dependent Stochastologicals Summary: Statistical dependencies that vary significantly across different contexts.	Solution: Develop context-aware models that adjust based on the specific characteristics of the data set.	An adaptive modeling system that adjusts its parameters based on the specific context of the data being analyzed.
16	Open Concepts Summary: The challenge of dealing with concepts that cannot be strictly operationally defined.	Solution: Foster a more open, iterative approach to concept development, allowing for refinement as new data becomes available.	A collaborative conceptual mapping tool that allows for the dynamic definition and refinement of concepts based on user input and data integration.
17	Intentionality, Purpose, and Meaning Summary: The difficulty of capturing the subjective aspects of human experience in research.	Solution: Integrate psychological and social research with insights from cognitive science and philosophy.	An interdisciplinary research platform that combines qualitative and quantitative data to explore the dimensions of human intentionality and meaning.
18	Rule Governance Summary: Human behavior is influenced by perceived rules, which are not always empirically observable.	Solution: Utilize mixed-methods research to better understand how rules influence behavior.	A mixed-methods research tool that combines quantitative data analysis with qualitative insights to understand the impact of rules on behavior.
19	Uniquely Human Events and Powers Summary: Aspects of human life that are not shared with other		A specialized study framework that integrates anthropological, psychological, and sociological perspectives to examine events and capabilities unique to humans.

[hyunjimoon]

A thinks: 1. evaluate (eliminate bad idea, help guide refinement); only mutation without selection;

• doesn't include

M: 1. falsification should be the function your understanding of mechanism, 2. regulator should be included the model (plane model is inside the model; reverse engineer)

• joseph hendricks evolution of belief; toxin of fish and tabo (doing right thing for the wrong reason)

• falsibiable by forcing to make prediction

• A tried team-wide sampling p(theta|phi_1, phi_1) cube (phi_1=team size, phi_2=cognitive diversity)

• assumption is kernel (phi1, phi2)is designed orthogonal and relatively linear exprapolative

• AM commented although gibbs sampling is ineffective, it has better cognitive fitness

[hyunjimoon]

Most Grant applications are bad
Most manuscripts submitted to journals are bad
Most actually published research is bad
Most published articles are not read anyway
Theories in psychology are like old soldiers: they are not refuted or replaced - they don’t die-they only fade away
Research in psychology does not tend to replicate
Science is supposed to be a cumulative endeavour but psychologist build mostly castles in the sand

• high causal density -> nuisance variance, individual diff,
• disease:
• symptoms: reproducibility,
• solution would need

Creating a tree that organizes 19 different needs based on their relationships, especially in the context of entrepreneurship research, requires identifying overarching themes or problems and how they branch out into more specific issues. The example given suggests starting with "High Causal Density" as a broader category, which affects other issues such as "Nuisance Variance" and "Individual Differences."

Given this structure, let's conceptualize a simplified tree that might represent how various research needs in entrepreneurship could be related, focusing on broad categories that branch into more specific needs.

[hyunjimoon]

1. High Causal Density
   • 1.1 Nuisance Variance
     • 1.1.1 Sampling Variability
     • 1.1.2 Measurement Error
   • 1.2 Individual Differences
     • 1.2.1 Personality Traits
     • 1.2.2 Cognitive Styles
   • 1.3 Environmental Factors
     • 1.3.1 Market Dynamics
     • 1.3.2 Regulatory Impact
2. Operationalization Issues
   • 2.1 Definitional Ambiguity
     • 2.1.1 Entrepreneurship Conceptualization
     • 2.1.2 Innovation Metrics
   • 2.2 Measurement Challenges
     • 2.2.1 Outcome Assessment
     • 2.2.2 Process Tracking
3. Theory-Practice Gap
   • 3.1 Applicability of Research
     • 3.1.1 Practical Implementation
     • 3.1.2 Policy Relevance
   • 3.2 Knowledge Transfer
     • 3.2.1 Academic-Industry Collaboration
     • 3.2.2 Entrepreneurship Education
4. Responsive Class Problem
   • 4.1 Segmentation Difficulties
     • 4.1.1 Behavioral Categorization
     • 4.1.2 Causally Relevant Attributes
   • 4.2 Dynamic Market Adaptation
     • 4.2.1 Agility and Flexibility in Entrepreneurial Strategy
     • 4.2.2 Response to Market Volatility
5. Methodological Diversification
   • 5.1 Quantitative Methods
     • 5.1.1 Statistical Modeling
     • 5.1.2 Econometric Analysis
   • 5.2 Qualitative Methods
     • 5.2.1 Case Studies
     • 5.2.2 Ethnographic Research

[hyunjimoon]

#	Need	Solve	Example Tool	Need_ENT	Solve_ENT	Example Tool in ENT	Literature Supplying Need	Literature Failed to Deliver Solution to Need
5	Polygenic Heredity	Solution: Develop interdisciplinary research models that integrate genetic data, social factors, and behavioral studies across large populations.	A bio-social research platform facilitating data sharing and analysis of genetic markers alongside behavioral and environmental data (longitudinal studies)	The need to understand the interplay of genetic predispositions and socio-environmental factors in shaping entrepreneurial tendencies and outcomes.	Provides a framework to disentangle complex nature-nurture influences on entrepreneurship, aiding research into targeted interventions.	A secure data repository and analysis suite for multidisciplinary teams to study genetic predispositions, lived experiences, and entrepreneurial behaviors	"The Biology of Entrepreneurs: The Impact of Biomarkers on Entrepreneurial Decision-Making and Performance" by Michael Frese & Michael M. Gielnik (2014)	"Sociogenomics of Entrepreneurship" by Daniel J. Benjamin, Philipp D. Koellinger, Jan-Emmanuel De Neve, Danielle Posthuma, and Aysu Okbay (2012)
6	Divergent Causality	Solution: Develop dynamic, adaptive models that incorporate stochastic elements and feedback loops to simulate divergent outcomes.	A simulation platform enabling researchers to model entrepreneurial ventures with varying initial conditions, stochastic events, and feedback loops to explore long-term trajectories.	The need to understand how seemingly small differences in entrepreneurial ventures (resources, decisions, etc.) lead to vastly different outcomes over time.	Helps researchers and practitioners anticipate potential trajectories, identify critical decision points, and understand the role of chance and adaptive responses in entrepreneurial success.	A sensitivity analysis dashboard for entrepreneurial ventures, allowing users to simulate the impact of minor initial variations (e.g., access to capital, team dynamics) on long-term outcomes.	"Evolution and Revolution as Organizations Grow" by Larry E. Greiner (1972)	"Competitive and Institutional Spillover Dynamics" by Joel A. C. Baum and Jitendra V. Singh (1994)
7	Idiographic Problem	Solution: Facilitate qualitative, in-depth studies alongside quantitative methods, prioritizing insights into individual narratives.	A mixed-methods research platform that integrates qualitative data collection (interviews, observations) with quantitative analysis to explore commonalities and unique patterns in entrepreneurial stories.	The need to capture the richness of individual entrepreneurial journeys, motivations, and decision-making beyond broad statistical trends.	Provides a holistic approach to research, enabling the identification of individual-level factors that impact success, failure, and resilience alongside broader trends.	An entrepreneurial storytelling repository connected to an analytical engine that extracts themes, mindsets, and critical turning points from first-person narratives.	"Entrepreneurship: A Phenomenological Perspective" by Duane Ireland, David C. Webb, and Jeffrey B. Covin (2001)	"Toward a Theory of Entrepreneurial Passion: A Phenomenological Inquiry" by Dean A. Shepherd and Justin S. Patzelt (2011)
8	Unknown Critical Events	Solution: Integrate real-time data collection (wearables, mobile) into longitudinal studies to capture context-rich data surrounding turning points.	A wearable sensor and data analysis platform that combines physiological, environmental, and self-report data to identify patterns and triggers around critical events in long-term studies.	The need to understand how specific experiences (failures, successes, serendipitous encounters) impact entrepreneurial mindsets and future behaviors.	Provides a method to link critical events to long-term trajectories, enabling researchers to design interventions that build resilience or capitalize on turning points.	An integrated physiological sensor platform for entrepreneurial studies that links heart rate variability, location data, and activity tracking with critical event self-reporting.	"The Nature of Entrepreneurial Experience, Startup and Growth" by William B. Gartner (1985)	"Entrepreneurial Grit: Durable Goal Pursuit, Interest, and Effort" by Angela L. Duckworth, Katherine Milkman, and David Laibson (2011)
9	Nuisance Variables	Develop robust causal inference techniques (e.g., instrumental variables, mediation analysis) to isolate variable effects.	A causal modeling toolkit for entrepreneurial datasets, offering various methods for handling nuisance variables.	The need to identify and control for the influence of confounding variables in entrepreneurial research for accurate insights.	Enables researchers to determine the true impact of key variables on entrepreneurial outcomes, improving theory development and intervention design.	A sensitivity analysis dashboard for entrepreneurial models, allowing researchers to simulate the impact of potential nuisance variables on their findings.	"Entrepreneurial Risk and Market Entry" by Timothy Bates (1995)	"Measuring Entrepreneurial Activities" by David J. Storey (1991)
10	Feedback Loops	Utilize network analysis to map social and professional networks, modeling feedback mechanisms.	A social network mapping and analysis platform tailored for entrepreneurial ecosystems or communities.	The need to understand how feedback loops within entrepreneurial networks (investors, mentors, teams) influence individual and collective outcomes.	Allows researchers to map the flow of resources, information, and influence, identifying critical nodes and feedback mechanisms in entrepreneurial success or failure.	A network visualization tool for entrepreneurial ecosystems, allowing researchers to track resources, support, and information flows and their effect on venture trajectories.	"Network Dynamics of Entrepreneurial Ecosystems: A Review and Research Agenda" by Colin Mason and Ross Brown (2014)	"Entrepreneurship: A Social Network Perspective" by Howard E. Aldrich and Caroline Zimmer (1986)
11	Autocatalytic Process	Apply time-series analysis and non-linear models

[hyunjimoon]

Week 5 (03/05) - What is still wrong with our sciences? (part 3)
Jolly, E. & Chang, L. J. The Flatland fallacy: Moving beyond low-dimensional thinking. Top. Cogn. Sci. 11, 433–454 (2019)
Cronbach, L. J. Beyond the two disciplines of scientific psychology. Am. Psychol. 30, 116–127 (1975)
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J. & Reinero, D. A. Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences 113, 6454–6459 (2016)
Yarkoni, T. The generalizability crisis. Behav. Brain Sci. 1–37 (2020)
Muthukrishna, Michael, and Joseph Henrich. "A problem in theory." Nature Human Behaviour 3.3 (2019): 221-229.

reflection note on causal diagram of researcher's pairing, producing, delivering + connection with Matt's model in #190 (comment)

[hyunjimoon]

during the break

• narrowed belief and goal prior with abudllah (A)

• after receiving informative prior on problem solving process in A's domain, updated my prior in startup domain and further pooled prior from disease treatment (raw data: 15.363 W5.txt) where we can have more information gain on solution process.

• shared the above with A and prompted for input on informative prior for solution rows of Social Science column.

Step	Substep	Disease Treatment	Social Science	Startup Social Science
1		Disease is treated only at the symptom level without addressing underlying causes.	Social science is non-cumulative.	Startups fail at a higher rate than necessary.
2		Lack of understanding of the fundamental biological processes leading to disease.	High causal density and methodological incompatibility.	Inadequate adaptation to rapidly changing environments and market signals.
	2.1	Everything is a symptom of deeper level cause.	High causal density (many things matter).	Idiosyncratic signal from dynamic environment.
	2.2	Traditional approaches focus on symptom relief rather than root cause.	Incommensurability (design choices); focus on not hypothesis testing.	Too reactive from conflated effects of belief and goal (confusion of misalignment), agent and environment's uncertainty, and overconfidence and optimism (value of misalignment).
	2.3	Regulatory frameworks prioritize demonstrable efficacy over understanding complex biological interactions.	Existing methods ignore root causes.	Non-cumulative learning from success and failed startup cases.
3		Develop targeted therapies based on precise understanding of disease mechanisms.	Develop cumulative and integrative research methodologies.	Implement adaptive learning frameworks for rapid response to market changes.
	3.1	Invest in genomic and proteomic research to understand disease at a molecular level.	Facilitate compatibility and commensurability across studies.	Show agents potential choices through higher quality information with dashboard 🖥️, offering training 🏋️, and testing their belief with interactive simulation 🖱️.
	3.2	Adopt a precision medicine approach to develop treatments targeting specific molecular pathways.	Promote hypothesis testing and evidence-based research.	Enhance decision-making process.
	3.3	Implement adaptive clinical trial designs to allow for iterative learning about the disease.	Implement integrative research methodologies.	Develop comprehensive platforms for iterative learning and adaptation.
4		Targeting molecular pathways allows for treatment of the underlying causes of diseases, not just the symptoms.	By developing cumulative methodologies and promoting compatibility, science can build upon previous knowledge and become more integrative and cumulative.	With dashboard, training, and simulation, startups can better interpret market signals, make informed decisions, and reduce unnecessary failures.
5		Molecular understanding leads to targeted drug development; adaptive trials test these drugs efficiently, leading to effective treatments.	Implement cross-disciplinary platforms and standardized protocols to ensure research integration and accumulation.	Develop and implement a comprehensive platform that includes a real-time dashboard, targeted training programs, and interactive simulations to support startups in decision-making processes.

[hyunjimoon]

Week 9 (04/02) - What is still wrong with our sciences? (Part 4)
IJzerman, H. et al. Use caution when applying behavioural science to policy. Nat Hum Behav 4, 1092–1094 (2020)
Lin, H., Werner, K. M. & Inzlicht, M. Promises and perils of experimentation: The mutual-internal-validity problem. Perspect. Psychol. Sci. 1745691620974773 (2021)
Bryan, C. J., Tipton, E. & Yeager, D. S. Behavioural science is unlikely to change the world without a heterogeneity revolution. Nat Hum Behav 5, 980–989 (2021)
Kenny, D. A. & Judd, C. M. The unappreciated heterogeneity of effect sizes: Implications for power, precision, planning of research, and replication. Psychological Methods 24, 578–589 (2019)

Use caution when applying behavioral science to policy

• i'm builing startup genome and i shared "field of genetics started in a position similar to the position that many behavioural sciences find themselves in now, with small, independently collected samples that produced unreliable findings. Attempts to identify candidate genes for many constructs of interest kept stalling at TRL1/ERL4. In one prominent example, 52 patients provided genetic material for an analysis of the relationship between the 5-HTT gene and major depression, a finding that spurred enormous interest in the biological mechanisms underlying depression. Unfortunately, as with the current situation in psychology, these early results were contradicted by failed replication studies." with my team to persuade the validity of my analogy (:= theory that cannot be experiment?)

• as i found humility (:=a modest or low view of one's own importance)'s value in facilitating action by underestimating one's capability, the word "epistemic humility" was interesting but i don't think there's humility that's not epistemic.

Promises and Perils of Experimentation: The Mutual-Internal-Validity Problem

• given that we read Feyman's piece on integrity, I wonder what he'd say to have integrity in social science ("SS is an eg. of science which is not science. they collect data but they don’t get any laws, they haven’t found out anything")

• i knew belief pooling is easier than goal pooling "experiments designed to accomplish one goal (e.g., test theories of dopamine function) often have limitations (e.g., limited external validity)" but perhaps decomposing the long end2end into smaller phases (e.g. startup life cycle to nail, scale, sail), pool-ability of goals would increase, decreasing the heterogeneity of actions which are sequence of optimal solution of value function. in short, the smaller the value function space is, the smaller the optimal action sequence is, the more accumulative findings become within that homogenous phases.

• importance of separating the agents evaluating theory, phenomena, experiment from "phenomena elicited by experiments are used to develop theories that, in turn, influence the design of theory-testing experiments, experiments and theories can become wedded to each other and lose touch with reality"

• i think "tell us about" vs "resembleance" are not different. i wish to represent this using statistical structure (defined as having non-zero mutual information with the context variable in https://arxiv.org/pdf/1704.01087.pdf). "if the goal is to test theories, experiments should be evaluated by how much they tell us about the underlying theories and not by their resemblance Lin et al. to phenomena in real life"

[hyunjimoon]

1. realism, instrumental; if the measure is more concrete and generalizable,

• for those dying from cancer, trueness of scientific kwdg on dementia is of less use
• If you had to choose between being intelligent and being kind, what would you choose?
• in data and parameter set, bayesian proceed vertically and frequentist approach horizontally (hetereogenity is increasing in observation/generator level whereas heterogenity is increasing in theory/estimator)

predicted gravitational wave (couldn't have observed) - truer than (ability to compute more)

2. ability to evaluate truth: ran the two models and compare predicted phenomenon set

theories should make prediction (loose derivation chain) - relativity (instrumental)
blackholes colliding (liger paper - 1k scientist) - if we were able to test,

• we agreed holy grail (either you're R or I): share criteria - predict unseen phenomena

• Abdullah introduce different

Goal Type	Why People Do Science	Why People Do Startups
Realist	To accurately describe the real world and seek objective truth and knowledge.	To understand how the world works.
Instrumentalist	To use science as a tool for practical goals and predictions, rather than to describe reality accurately.	To understand how to create value.
Social Constructivist	To acknowledge that science is a product of social processes and is defined by societal consensus.	To explore social dynamics and understand how the money system works.
Aesthetic	To engage in science for intellectual stimulation and enjoyment, appreciating the beauty of discovery.	To pursue passion and innovation, finding joy and fulfillment in creating something new.
Science Relativist	To consider scientific rationality as dependent on specific data and parameters, without absolute truth.	To challenge and redefine existing paradigms and rationalities.

[hyunjimoon]

otter
The conversation revolved around the challenges and opportunities in scientific research, particularly in the context of heterogeneity. Speaker 1 emphasized the need for practical utility in social science research, while Speaker 2 highlighted the importance of recognizing and addressing heterogeneity. Speaker 3 offered a critical perspective on the issue, and Speaker 5 provided a more prescriptive approach. Unknown Speaker raised questions about the relevance of a paper on heterogeneity, and Speaker 6 inquired about senior colleagues' reactions to challenging papers. The speakers collectively underscored the need for more systematic and prescriptive solutions to address the challenges of heterogeneity in scientific research. They also discussed the importance of considering both internal and external validity in experimental design, and the potential benefits of applying TRL in social technologies. Speakers 1 and 3 debated the nature of scientific truth, with Speaker 1 advocating for instrumentalism and Speaker 3 defending scientific realism. Overall, the conversation highlighted the need for a more nuanced understanding of the challenges and opportunities in scientific research, and the importance of carefully considering experimental design and methodology to ensure valid and generalizable results.

Transcript

https://otter.ai/u/bz9d8JIMe5R-xCHLNxWqO_OsS-o?view=transcript

Action Items
[ ] Consider removing one of the readings and replacing it with two papers calling for practical utility and two highlighting heterogeneity to improve the session design.
[ ] Share thoughts on which paper should be removed from the session.
[ ] Articulate personal preferences and philosophies on the goals of science to facilitate discussion.
[ ] Explain perspectives on the distinction between realism and instrumentalism in science.
[ ] Discuss views on evaluating truth and uncovering reality in science from a realist perspective.
[ ] Consider how predictive ability has enabled evaluating the truthfulness of scientific theories over time.
[ ] Share examples of theories that seemed unimportant initially but led to useful discoveries to illustrate a counterpoint.
[ ] Articulate personal philosophies on the goals of science and how they relate to practices.
[ ] Send paper on scientific readiness to inform policy decisions to a government official as an example of its usefulness.
[ ] Consider the degree to which scientists lie to themselves or have mixed motivations related to the goals driving their work.
Outline
The practicality of scientific research.
Speaker 1 advocates for more practical utility in science, while Jason calls for removing one paper from the session to address heterogeneity.
Speaker 2 expresses reservations about the readiness of a paper on cooperation with designing in public good games, suggesting it may not be ready for policy advice.
Professor suggests removing paper from course readings due to lack of relevance.
Validity concerns in behavioral science research.
Speaker 3 highlights the importance of evaluating interventions using a framework, while Speaker 4 raises concerns about the relevance of research in the context of the mutual vulnerability problem.
Speaker 4 highlights the importance of considering internal validity in research, particularly in terms of ecological validity (10:05).
Speaker 1 raises questions about the paper's message and implications, suggesting that it may be overfitting or oversimplifying complex issues (10:20).
Speakers discuss the relevance of a research paper on behavioral science and its application in tribal organizations.
Science goals and research practices.
Speaker 6 highlights the importance of comparing NASA missions with science, and discusses the value of heterogeneity in research.
The speaker questions the purpose of science, suggesting it can be driven by multiple goals, including intellectual curiosity, advancing a science, predicting and controlling the world, and aligning research practices with personal goals.
Speaker 1 and Speaker 2 discuss the importance of considering heterogeneity in research, with Speaker 1 finding a particular paper on the topic unhelpful due to a lack of focus on practical implications.
Research methods and heterogeneity in statistics.
Speaker 1 expresses frustration with a paper's lack of clarity on how to address heterogeneity in research planning.
Speaker 1 proposes multiple solutions to a problem, seeking diverse perspectives and empirical evaluation.
Participants express relief at finding a paper they initially found depressing to be useful and engaging.
Limitations of social science in informing policy.
Speaker 2 suggests using caution when applying Behavioral Science to Policy, as most participants come from Western societies and effects may be barely perceptible.
Speaker 1 discusses the tension between social science research and policy implementation, particularly in the context of foreign policy.
Speaker 1 discusses the idea of "pianism," which refers to the belief that science is not about predicting and controlling the world, but rather about understanding it through research and discovery.
Unknown Speaker suggests that the goal of science is not always to inform policy, but rather to advance knowledge and understanding for its own sake.
Scientific method and its application in various fields.
Speakers debate the usefulness of social psychology research in understanding the world, with some finding it unhelpful and others seeing it as valuable for uncovering truth.
Speaker 1 explains that Facebook's research scientists prioritize predicting and controlling ad clicks over uncovering the truth, as their goal is to maximize ad revenue rather than generalize findings to other contexts.
Speaker 3 acknowledges that policy decisions are similar to Facebook's approach, prioritizing predictive modeling over generalizability, as the goal is to achieve specific outcomes rather than uncover broader truths.
Realists prioritize uncovering truth over practical use, which can lead to overclaiming and disruptive implications for public sector applications.
Science, truth, and policy.
Analysts prioritize generalizability in policy recommendations, while social psychologists focus on utility prediction and control.
Speaker 1 expresses dissatisfaction with others blaming them for misunderstandings in their work.
Speaker 1 questions the pursuit of truth in academic research, suggesting it may be driven by personal agendas rather than a genuine desire for knowledge.
Scientific truth and generalizability.
Instrumentalists predict and control through fictional entities, while realists seek accurate representations of the world.
Speaker 1 emphasizes the importance of generalizability in science, arguing that a theory must be able to make predictions in yet unseen situations, including policy questions.
Speaker 1 believes that complete representational accuracy is necessary for informing every policy question, as it allows for generalization to every situation, including the one relevant to the policy at hand.
Truth and knowledge in science.
Speaker 6 argues that realists and instrumentalists may overlap, and that the value of truth depends on the situation.
Speaker 6 argues that our ability to compute and simulate phenomena has improved our evaluation of scientific theories, allowing us to determine which theories are more true.
Speaker 1 agrees that predictions made by theories should be testable, but acknowledges that some predictions may not be able to be tested until future technological advancements are made.
Science goals and evaluation methods.
Speaker 1 argues that science is not only about uncovering the truth, but also about pursuing useful knowledge, even if it's not scientifically accurate.
Speaker argues that fundamental and applied science are not mutually exclusive, and both have their own value.
Speaker 1 discusses the limitations of science, including the difficulty of evaluating instrumentalism and relativism.
The nature of science and economics.
Speaker argues that science is a social construct, with no reference to truth or reality.
Speaker 1 clarifies that their model is normative, not descriptive, and assumes axioms that may not be true empirically.
Scientific Goals and Motivations.
Speaker 1 grapples with the purpose of science, questioning the role of normative models and intrinsic motivation.
Speaker 1 prioritizes fun and intellectual stimulation in their work, despite not being a scientist.
Speaker discusses five goals for math education, including problem-solving and aesthetics, and questions the legitimacy of the fifth goal, relativism.
Goals and motivations in scientific research.
Speaker 4 argues that serendipity can lead to discoveries, but also acknowledges the importance of prioritizing goals and making trade-offs between them.
Speaker 1 agrees with Speaker 4 and emphasizes the importance of holding multiple goals simultaneously, while also recognizing the need for prioritization and trade-offs.
Participants discuss self-deception in pursuit of goals, with one speaker acknowledging the ease of fooling oneself.
The challenges of scientific progress and the need for coherent research.
Speaker 1 struggles to connect their work to clear goals, while Speaker 5 hopes for magic and communication to make a positive impact.
Speaker reflects on past predictions of psychology's future, noting lack of progress in 50 years.
Speaker expresses concern about the lack of guaranteed success in their field, despite achieving notable accomplishments.
Readiness levels for social technologies.
Abdullah's framework for understanding why people do startups resonated with Speaker 6, who applied it to five categories: realists, instrumentalists, social constructivists, and aesthetes.
Speaker 1 expresses frustration with the lack of application of a paper on technology readiness levels, suggesting it would be useful to have a website with readiness levels for different social technologies.
Speaker 3 explains the NASA Technology Readiness Level (TRL) scale, which is used to measure the maturity of a technology, and how it has been applied in the context of entrepreneurship.
Funding for research and infrastructure.
Speaker 1 suggests that researchers are not incentivized to implement new ideas due to a lack of financial reward.
Speaker 1 argues that the NSF's funding process is too burdensome and requires too much work to be completed before applying.
Professor discusses NSF funding with colleague, mentioning heat map of distribution and KPIs.
Scientific theory and experimentation.
Matt reached out to the group to join them for dinner and one-on-one meetings.
Speakers discuss the limitations of experimentation in developing and testing theories, with a focus on the potential for researchers to conflate internal validity with external validity.
Validity concerns arise when measuring intelligence, as operationalization can lead to mistakes and lack of mapping to the construct.
Experiment validity and external generalizability.
Experiment validity concerns: randomization failure, confounding, and design space issues.
Speaker 1 argues that internal validity is crucial in experiments, as a lack of variance in the outcome can invalidate the results.
Speaker 1 also emphasizes the importance of external validity, or generalizability, in determining the practical relevance of the findings.
Speaker 1 argues that external validity is about the region and specified what the region is, but lacks ecological validity when the experiment does not generalize to the real-world context.
Speaker 1 also distinguishes between external validity and ecological validity, with the former referring to the size of the area for which the result holds, and the latter referring to the relevance to the specific situation or context.
Experimentation validity in research.
Expert emphasizes importance of demo validity in experiments, regardless of goal.
Speaker struggles to understand the distinction between internal and external validity in theory testing, finding it confusing and difficult to apply in practice.
Speaker emphasizes the importance of external validity in testing theories, arguing that a valid experiment must have some level of generalizability beyond a single data point.
Internal and external validity in psychology research.
Speaker 1 argues that relying solely on cognitive diversity can lead to a narrow understanding of a problem, while ignoring the broader context.
Speaker 1 argues that triangulation is necessary to validate research findings, avoiding bias and ensuring external and internal validity.
Speaker 2 questions whether internal validity should be prioritized over external validity.
Speaker 1 argues that internal validity is important but cannot be quantified, leading to discussions about acceptable thresholds.
Academic papers and their impact.
Participants discuss the importance of considering heterogeneity in scientific research, with a focus on a paper by Brian Tipton and Gager.
Speaker 1 expresses frustration with a paper's timidity and lack of assertiveness in addressing important issues (2:11:13).
Speaker 1 suggests that the paper could have benefited from more passion and prescriptive advice on how to tackle its subject (2:13:20).
Grumpy author's personality shines through in prospective pieces, despite criticism.
Science collaboration and data analysis.
Speaker 1 struggles with understanding the implications of heterogeneity in effect sizes, leading to confusion and frustration.
Scientists discuss potential solutions to improve the success of genetics research, including collective effort and better questions.
Speaker 6 notes difficulty in defining units of behavior due to complexity of mind and brain, increasing causal density as subject matter progresses from physics to biology.

[hyunjimoon]

Matt, Angie
vensim model of knowledge production system in nsp_academia_v2.mdl.zip

lower right fig. on startup operation's higher resolution in AMoon_test_template.pdf - would love to productize this concept as paper!

matt's talk

The conversation centered around the challenges of producing and disseminating validated knowledge in the digital age. Speaker 1 highlighted the issue of fake knowledge being published in medical journals, while Unknown Speaker raised concerns about the lack of attention to original research. Speaker 1 advocated for a shift in focus towards producing high-quality, validated knowledge that is accessible and useful to the broader public. Maria and Speaker 1 discussed the need for a paradigm shift in management education, emphasizing the importance of double loop learning and a more coordinated approach to organizational scholarship. Speaker 1 also emphasized the significance of organizational science in understanding how organizations affect society and how they can be improved. Finally, Speaker 1 discussed the difficulties of interdisciplinary research and proposed a departmental structure to facilitate collaboration.

Transcript

https://otter.ai/u/UAMOPXU00Q0ajD9nu6iPODbGwJc?view=transcript

Outline
The flooding of knowledge with fake research.
The speaker discusses the problem of "brickmaking" in the production of knowledge, where the focus on creating bricks (i.e., facts) leads to a lack of attention to the ultimate goal of constructing edifices (i.e., knowledge).
The speaker also mentions the issue of "paper mills" in medical journals, where fake knowledge is produced and sold to contaminate actual knowledge.
Improving academic research and its impact.
Speaker 1 discusses the challenges of generating original knowledge and disseminating it effectively, citing the example of a study on the benefits of drunk creative problem solving.
Speaker 1 highlights the problem of rewarding production without addressing the issue of replication, leading to a lack of credibility in academic research.
Speaker 1 argues that the field of management needs a perspective change to prioritize substantive things over market-based solutions.
Speaker 1 proposes a model for changing incentives to support this perspective change, but emphasizes the difficulty of doing so.
The importance of organizational science and effective decision-making.
Speaker 1 emphasizes the importance of organization science, citing its impact on society and the need for careful, cumulative, and collaborative research.
Speaker 1 proposes a marketplace for ideas as a solution to the problem of knowing what to do in a specific context, while acknowledging the complexity of particulars and the need for ongoing research.
The importance of specialization and coordination in scientific research.
Speaker 1 emphasizes the importance of a solid theoretical foundation, authority, and connection to the broader community in the field of stonemasonry.
Speaker 1 highlights the interconnectedness of various functions within the field, including invention, validation, and building, and notes that each function has numerous subfunctions that need to be specialized and coordinated.
The speaker discusses the limitations of group brainstorming and the importance of overcoming social loafing, evaluation apprehension, and blocking.
The speaker highlights the success of the nominal group technique, a teachable, usable skill that can help groups brainstorm more effectively, but notes that it's not enough to be creative.
The limitations of market research and the importance of understanding in product development.
Speaker 1 expresses frustration with the "market for lemons" in which people can't easily identify good products or services, and influencers often prioritize sounding usable over being truthful.
Speaker 1 suggests that design thinking should focus on creating products that are both useful and desirable, rather than simply competing on ease of use or likability.
Speaker discusses the limitations of understanding and using information, citing examples of how people's understanding of a concept can be influenced by their prior knowledge and how understanding something is harder than we think.
Speaker references a historical example of how engineers used irregularities in metal conductivity to develop components that could reliably change signals, leading to modern theories that explain those anomalies.
Scholarly research and its limitations.
Speaker discusses the development of the transistor and its impact on society, highlighting the limitations of current educational practices.
Scholar entrepreneurs must move from confederation to specialized enterprise, avoiding misleading information.
Research methods and expertise in various fields.
Speaker 1 discusses the importance of understanding how products across different quadrants fit together and how people working in each quadrant can help each other.
Speaker 1 emphasizes the need to teach people how to think about problems and the importance of considering the end user's perspective when creating products.
The speaker discusses the importance of understanding the relationship between lab studies and real-world applications, highlighting the need for a bridge between the two.
The speaker suggests that it's difficult for one person to be an expert in all the various methods and techniques used in scientific research, and instead proposes a collaborative approach to addressing the challenges of interdisciplinary research.
Speaker 1 suggests specializing in specific skills or topics to avoid the "zero sum" problem of trying to be good at everything (30 words)
Speaker 1 proposes distributing tasks based on individual strengths and interests to maximize efficiency and effectiveness (29 words)
Aligning efforts in a field with multiple scholarly vehicles.
Speaker 1 argues that prioritizing competition over collaboration in tech industry undermines innovation.
Speaker 1 discusses the importance of understanding how different scholarly vehicles fit together in the field of medical research, and how they can be used to incentivize collaboration rather than competition.
Speaker 1 highlights the need for more education and application-focused research, and suggests using a systems model to visualize how different parts of the field work together.
Improving academic publishing and education.
Speaker discusses the limitations of current research practices and suggests changes to promote more accessible and replicable studies.
Speaker highlights underutilized PhDs in academia and the impactful role of teaching in connecting people to information.
Improving management education.
Maria I taught that measuring impact is crucial for government programs, and that real science is needed to demonstrate effectiveness, not just cute stories or gimmicks.
The speaker suggests that double loop learning is necessary to truly understand the impact of management education, and that legitimacy should not be begged for, but rather earned through shared effort and collaboration.
Speaker 1 discusses the importance of cultural and socialization in organizational management, emphasizing the need for scholars to work together to improve societies and lives.
Speaker 2 mentions the opportunity for online questions and discussion, highlighting the interactive nature of the event.
Legitimizing management field through shared understanding.
Speaker 2 discusses the need for a shared understanding of how to work towards credible knowledge in a field, and the importance of mapping out the system and understanding the different stakeholders' needs.
Speaker 3 raises the question of why certain fields like guilds have emerged organically without the need for legitimacy, and what properties these fields share that may be relevant to management.
Speaker 1 discusses the importance of belief in one's work, particularly in fields like medicine and law, where practitioners are often deeply invested in their work and believe in its value.
Speaker 1 and others consider the role of culture and governance in shaping beliefs and practices within organizations, and how this can impact the effectiveness of these organizations.

chat with matt

In product development, understanding the needs of the target audience and taking a user-centered approach is crucial. Speakers emphasized the importance of balancing desirability and feasibility, and addressing delivery capacity challenges. They also discussed the significance of optimizing startup operations for scalability, integrating knowledge in cognitive science, and segmenting the academic market to better understand the capacity of scholars and researchers. Empirical validation and education are key in the candidate development process, and early-stage scholars face challenges in research production due to lack of financial support and inadequate institutional support.

Transcript

https://otter.ai/u/aCamLRhMCnwZBNjltCYfXcErNw4?view=transcript

Action Items
[ ] Update the conceptual model discussed with the new terminology suggested.
[ ] Send over the slides used in the discussion.
[ ] Incorporate the feedback into refining the conceptual model, focusing on narrowing down the scope.
[ ] Modify the conceptual model based on the discussion, updating elements like "candidates", "pairing", "products", and "delivery".
[ ] Attend the dinner meeting.
Outline
Developing a tool for teaching negotiation.
Organizations use incentives to educate and train employees, but there is a gap in practitioner knowledge.
Speaker 2 suggests targeting entrepreneurs and educators for a tool that can be useful and usable.
Speaker 1 emphasizes the importance of desirability in addition to usefulness and usability.
Product design and development strategies.
Colin and Speaker 2 discuss the importance of desirability and feasibility in product design, with Speaker 1 emphasizing the need for both in product adoption.
Speaker 1 and Speaker 2 discuss candidates, which they define as ideas or concepts that could be developed or implemented in a radically different way.
They also discuss method capacity, which refers to the ability to produce ideas or solutions more quickly and efficiently, and how this can be increased through statistical or theoretical writing.
Candidates are formed through pairing, then vetted through delivery of a product (paper) that undergoes peer review.
Modeling product development and delivery capacity.
Speaker 1 and 2 discuss the development of a product from potential to actual use, with a focus on increasing capacity in each stage.
Speaker 2 suggests that having more ideas or candidates does not necessarily speed up the production process.
Speaker 1 disagrees, arguing that the number of candidates determines the throughput, not the production time.
Mathematical modeling and material limits.
Speaker 2 suggests automating the pairing process to improve efficiency.
Expert discusses how mathematical models can be used to understand complex biological processes.
Knowledge synthesis and capacity building in education.
Speaker 1 discusses the concept of "synthesis" and how it relates to knowledge and capacity building.
Speaker discusses the importance of knowledge integration and coherence in education, citing a specific example of a researcher in England who has developed a framework for understanding innovation through reading academic journals.
Speaker 1 discusses the importance of capacity building and the need to reconcile incompatibilities in research.
Speaker 2 suggests that giving scholars a way to specialize in certain areas could help address the problem of too much information.
Market segmentation and solution development.
Speakers discuss the importance of segmentation in understanding academic market needs.
Speaker 1 expresses confusion over the random pairing of candidates, suggesting it may not be entirely random and could be influenced by unconscious biases.
Speaker 2 proposes defining segments based on shared needs and values to create more effective solutions, rather than relying on random pairing.
Speaker 1 discusses functional analysis, using the example of abduction and induction to explain the difference between reasoning and empirical validation.
Speaker 1 highlights the importance of language in understanding segmentation and the different ways people can specialize in the field, including education and entrepreneurship.
Entrepreneurship, innovation, and collaboration.
Speaker 2 explains the four functions in a startup, including desirability collector, CMO or CSO, salesperson, and CEO or CTO.
Speaker 2 discusses the A, B, C loud concept, where desirability and feasibility are matched to implement ideas and make a profit.
Speaker 2 discusses the importance of a concrete theory, measurement, and phenomena in developing a good research product.
Speaker 2 proposes dividing and conquering the research process to reach the desired outcome.
Speaker 1 highlights the coherence in a book's blending of physics, culture, space, and science fiction, making it engaging and enjoyable.
Knowledge sharing and product development.
Speaker 2 shares their experience with creating a product and the challenges they faced in communicating its value to others.
Speaker 1 emphasizes the importance of segmenting knowledge and delivering it to the right audience, using the term "routing" to describe this process.
Speaker 2 agrees, suggesting that products can be tailored to specific customers for easier delivery and better results.
Product development and testing.
Speaker 1 clarifies the difference between speculation and testing in knowledge application.
Speaker 1 discusses the importance of identifying needs and creating value, citing the example of the artist discovering a need for a product that doesn't yet exist.
Speaker 2 agrees, highlighting the value of segmentation in making push more effective, and shares a related web resource.
Innovation and delivery capacity in intrapreneurship.
Speaker 1 discusses delivery time and production capacity with Unknown Speaker.
Speaker 1: "More products = harder to find the right one" (search capacity)
Speaker 2: "Recombination of existing knowledge creates new ideas" (absorptive capacity)
Speaker 1 argues that the field needs to do a better job of delivering on its promises, citing the need for better routing and delivery capacity.
Speaker 2 agrees that there is a need for better delivery capacity, but is hesitant to pivot to a new approach without good evidence.
Research production strategies for early-stage scholars.
Speaker 2 seeks feedback on production plan, focusing on customizing strategies for early-stage scholars.
Speakers discuss challenges with research and publication in their field, including lack of institutional support and inadequate market segmentation.
Speakers discuss inefficiencies in resource allocation and demand management in a growing enterprise.
Speaker 1 emphasizes the importance of resonance in entrepreneurship, citing the analogy of a startup as a fraction.
Speaker 2 seeks to theorize this concept without getting too far out of their lane of startup operations.

[hyunjimoon]

Step	Substep	Operationalization
1. Need		Management science lacks systematic accumulation and integration of knowledge that can reliably inform practice
2. Root Cause	Nature	High causal density in social sciences makes simple one-at-a-time experiments insufficient
	Individual	Scholars try to do everything rather than specialize and coordinate
	Institution	Current academic incentives favor novel, attention-grabbing findings over careful integration
3. Solution	System	Build integrated enterprise with four core functions: - Generation: Systematic mapping of design spaces - Synthesis: Integration of findings across conditions - Education: Development of teaching frameworks - Application: Context-specific implementation
	Individual	Enable specialization in specific functions: - Methodological expertise - Theory building - Educational design - Practice translation
	Institution	Create coordination mechanisms: - Cross-functional teams - Shared theoretical frameworks - Knowledge exchange platforms - Long-term cultural change
4. How Solution Addresses Root Cause		- Systematic design space approach handles causal density - Specialization and coordination replace individual heroics - Cultural shift from novelty to cumulative knowledge
5. Implementation Plan		1. Map design spaces for key phenomena 2. Develop specialization pathways 3. Build coordination infrastructure 4. Foster cultural change through education 5. Measure progress through knowledge accumulation

---

ref
Enhancing mail to the enterprise with Operationalization Principles cld with input:
1.Mark Hansen's slides Hansen23_WhyHow_egRBV.pdf summarizing his work on Bayesian operationalization of the resource-based view which Breaks down abstract theoretical concepts into measurable components, Generate probabilistic insights for specific cases rather than just averages, Enable scenario analysis through combining effects
2. matt's slide (the enterprise) and abdullah's paper

[hyunjimoon]

W12
goal of today's class:

• finish figure summary
• understand "case for formal methodology in scientific reform(2021)" paper's plot

Stone	Capability and Character	Paper	gpt's Reason for Connection	angie's class-taking goal
Power Stone 💜	Grants the user ability to control energy; enhancing other stones destroying entire planets; unleashing massive energy blasts. Character's use: Ronan the Accuser used it to destroy the planet Xandar but was thwarted by the Guardians of the Galaxy.	Treating stimuli as a random factor in social psychology (2012) • Addresses biases in the treatment of stimuli in studies. • Proposes randomizing stimuli to improve study validity. | The Power Stone's capacity to influence and enhance other forces aligns with the paper's theme of enhancing research validity by random	The Power Stone's capacity to influence and enhance other forces aligns with the paper's theme of enhancing research validity by randomizing stimuli, thus exerting control over the 'energy' of the study's variables.
Reality Stone 🔴	Grants the user to alter reality; creating illusions, manipulating matter. Character's use: Thanos uses it to change the state of matter at will.	Constraints on Generality: A Proposed Addition to All Empirical Papers (2017) • Critiques overgeneralization in research. • Recommends adding COG to papers to define research applicability.	The Reality Stone's ability to reshape perception mirrors the paper's aim to reshape how research findings are perceived and applied, adding clarity and 'reality checks' to the generalization of results.
Space Stone 🔵	Grants the user ability to control space; teleportation; altering distances. Character's use: Loki used it for teleportation and summoning an invasion.	Psychological measures aren’t toothbrushes (2023) • Discusses the misuse of universal applications for psychological tools. • Advocates for tailored measures to specific contexts.	Like the Space Stone's manipulation of spatial relations, the paper encourages adapting psychological tools to fit the 'space' of their application, ensuring that measures are appropriately tailored to their research environment.
Soul Stone 🔶	Capabilities: Grants the user the ability to control souls (manipulate?) - steal, control, manipulate, and alter living and dead souls The Soul Stone is also the gateway to an idyllic netherworld known as the Soulworld. Character - Thanos sacrifices Gamora to obtain the Soul Stone on Vormir, as it requires a sacrifice of what one loves most	The case for formal methodology in scientific reform(2021) • Advocates for formal methodologies in scientific reform • Calls for rigorous statistical proofing for methodological claims • Aims to improve the reliability and validity of scientific findings - Claim 1: reproducibility is the cornerstone of, or a demarcation criterion for, science - Claim 2: using data more than once invalidates statistical inference - Claim 3: exploratory research uses ‘wonky’ statistics	The Soul Stone's requirement of sacrifice for the attainment of power mirrors the paper's call for a formal methodology in scientific reform, suggesting that true progress in science may come with the 'sacrifice' of untested claims and biases for rigorous, validated approaches.	understand this plot understand algorithm in simulation-based calibration context
Mind Stone 😲	Capabilities: Grants the user mental and psionic abilities - control the minds of others - grant sentience to artificial intelligence 😲 - enhance individual powers Character's use, Stone's use: Vision is created through with 😲	Small Effects: The Indispensable Foundation for a Cumulative Psychological Science (2022) • Emphasizes the importance of small effects in psychological research • Argues for a cumulative understanding of psychological phenomena • Advocates for a nuanced approach to scientific knowledge	The Mind Stone's enhancement of mental and psionic abilities parallels the paper's emphasis on the cumulative nature of science, where even small effects can significantly contribute to a greater understanding, akin to how small enhancements can lead to a greater cumulative power in the Mind Stone's character.	- push the connection with genes and genome-wide association studies further. - revisit two diagrams below

[hyunjimoon]

the-robust-beauty-of-improper-linear-models-in-decision-making.pdf
improper linear models mentioned in the paper:

The weights may be chosen to be equal across all predictor variables.
The weights may be chosen based on the intuition of the person making the prediction, rather than being mathematically optimized.
The weights may be chosen at random.
The weights may be derived by analyzing the predictions of human judges, a technique called "bootstrapping", rather than being derived from the actual relationship between predictor and criterion variables.

• reference class: 1. how to define (construct) 2. how to sample
• abdullah has made a plot with flexible, ease of use axis

[hyunjimoon]

• Duncun watt's A simple model of fads and cascading failures

The conversation centered around the challenges and limitations of current scientific practices, including the persistence of problems, lack of incentives for sharing ideas, and the need for a more deliberate approach to scientific contributions. Speakers discussed the importance of theoretical commensurability and community of practice, and the need for clear criteria for evaluating systems. They also highlighted the limitations of linear models in social science research and the importance of considering interactions and causal density in their analyses. Finally, speakers raised questions about the point where predictability becomes high enough to have any value in a chaotic system.

Transcript

https://otter.ai/u/ovYaCtIrednO9AjB-zwq3Xgm2po?view=transcript

Outline
Why some solutions to persistent problems are adopted while others are not.
Speakers discuss potential solutions to persistent problems in AI, but acknowledge broader structural issues remain unaddressed.
Speaker 1: Why are some solutions not adopted? Structural issues hinder progress.
Speaker 2: Persistence of problems due to lack of understanding of adoption barriers.
Challenges in psychological science, including lack of community and inadequate filtering of research findings.
Speaker 3 discusses the importance of understanding the needs of the end-user when building tools and products.
Speaker 3 highlights the potential value of building things that are needed, rather than trying to predict the future needs of consumers.
Speaker 3 argues for explicit incentives to produce and share scientific ideas for cumulative progress.
Speaker 1 argues lack of community in psychological science hinders progress.
Implementing Open Science measures to address reproducibility crisis, with concerns about potential abuse and the need for good faith in scientific research.
Speaker 4 discusses the challenges of implementing Open Science measures, including the potential for bad actors to abuse the system.
Speaker 5 connects knowledge domains to visualize ARC of scientific reform through formal methodology and simulation-based methods.
Whether science should prioritize cumulative knowledge or fast, responsive progress.
Speaker 5 discusses testing tool "simulation based calibration" and its connection to internal consistency, diversity, and heterogeneity in gene sequencing.
Speaker 5 questions whether cumulative science is necessary, suggesting adaptive approach instead.
Speakers debate the necessity of simulation in generalization, with some arguing it's necessary for cumulative consistency.
The limitations of comparing scientific theories due to lack of a shared framework.
Speaker 1: Community is backdrop canvas for different results in different contexts.
Speaker 2: Bounded generality within each car is necessary for competitiveness.
Speakers discuss need for shared mental model or framework to integrate diverse theories in psychology.
Speaker 1 explains that theories are incommensurate but their predictions are commensurate, meaning they both make predictions about the same phenomenon despite having different internal structures.
Speaker 1 discusses the idea of a "backdrop of space-time" that helps determine when to use which theory, based on the scale and speed of the phenomenon being observed.
Science predictions, theories, and falsification.
Speaker 1 emphasizes the importance of cumulative findings and multiple theories for predicting outcomes.
Speaker 1 discusses the concept of "strange coincidence" in defense and oceanography.
Speaker 1 discusses the importance of accumulating evidence in support of a theory before defending it against falsification.
Speaker 1 argues that making accurate predictions and accumulating money in the bank can demonstrate a theory's usefulness and warrant its defense.
Density in models and its implications for understanding psychology as a science.
Speaker 1 discusses the N K model fitness landscape, explaining the importance of costs and density in living landscapes.
Unknown Speaker shares a surprising perspective on costs and density, connecting it to the income model of living landscapes.
Speaker 1 discusses the concept of "extreme density" in modeling complex systems, arguing that it can lead to misleading models.
Speaker 1 and others cite papers on the effectiveness of "improper linear models" in predicting outcomes, despite weak interactions between variables.
The importance of accounting for interactions in statistical analysis.
Speaker 1 discusses the importance of interactions in psychology research, highlighting that even main effects change as other factors are introduced.
Interactions between variables are not included in current models, and documenting these changes is crucial for understanding how relationships work in the world.
Speaker 1 argues that pure linear models are often the best predictors, but ignoring interactions can lead to overfitting and poor predictions.
Speaker 6 suggests that decision-making processes can affect the interpretation of interactions, and ignoring them can lead to biased predictions.
Speaker 1 and 3 discuss the concept of "causal density" in social and behavioral sciences, which refers to the idea that many factors contribute to an outcome, each with a small effect size.
The discussion highlights the importance of considering interactions and higher-order effects in statistical analysis, as they can have significant consequences for determining the outcome.
Predictability in complex systems, with references to organic growth and chaos theory.
Speaker 1 argues that high-cost density is central to many issues, despite claims that individual-level interventions are sufficient.
Speaker 1 argues that small differences in initial conditions lead to vastly different outcomes in complex systems.
Speakers discuss predictability in music, questioning when it can be accurately predicted.
Evaluating systems using Lego blocks as a metaphor.
Speaker 5 proposes evaluator for edifice, faces criticism from Speaker 1.
Speaker 1: Building a house should meet criteria for shelter, durability, and predictive ability.
Speaker 5: User-based innovation is important for understanding criteria for practical use.
Speaker 5 argues that building a house requires a programmatic theory that is internally consistent, while Speaker 6 suggests that the meaning of a house is relative to the space it occupies.
Speakers discuss challenges with using Lego pieces and canvas in a building project.
15s.03

The conversation centered around improving research design and explanation in psychology. Speaker 1 emphasized the importance of defining parameters, while Speaker 3 highlighted the need for better integration of prediction and explanation. Speaker 6 brought up the issue of cultural differences in research, and Speaker 7 discussed the challenges of working with licorice flavor. The speakers also discussed potential solutions, such as labeling contributions, structured problem-solving, and pre-registration. In the second half of the conversation, Speakers 1, 3, and 9 discussed the role of formalism in scientific research, while Speaker 7 discussed the challenges of defining ideal experiments. The speakers also highlighted the need to improve social science research by addressing limitations such as unclear study bounds, lack of replication, and practical considerations.

Transcript

https://otter.ai/u/Pro67UfMRtMhDfr7st3tO5LeW6s?view=transcript

Action Items
[ ] Prioritize proposed solutions from papers discussed on dimensions of feasibility and impact. Assign solutions to quadrants on a 2D graph based on these dimensions.
[ ] Read additional papers suggested for future classes, including Dr. Cohn's blog posts on incentives in science and the pre-registration of illusion paper.
[ ] Revisit prioritization framework periodically to add new solutions as they come up in future readings.
[ ] Consider bringing proposed solutions from other papers discussed in past classes into the prioritization framework.
[ ] Save and share screenshot of solutions prioritization exercise with annotations for future reference. Add to course syllabus.
Outline
Using a platform for conducting experiments, with emphasis on parameterization and comparability between experiments.
Speaker defines parameters for experiment, including design decisions that affect results.
Speaker 1 explains how their platform makes it easy to compare experiments on different phenomena.
Speaker 1 expresses frustration with lack of resources at MIT, while Speaker 4 mentions discretionary funding.
Discussion of food preferences and cultural influences among a group of people.
Improving research transparency and reproducibility through formal models and standards.
Speaker 3 highlights the importance of labeling and categorizing tasks to improve research design and downstream analysis.
Speaker 1 suggests integrating prediction and explanation methods to enhance the interpretability of research findings.
Speaker 3 proposes evaluating solutions based on feasibility and impact, while Speaker 1 suggests using formal models to fix science.
Speaker 3 argues that solutions to disability issues are not panaceas and may not always produce desired results.
Scientific method reform, disagreements on approach.
Main figures in meta science reform movement argue that defaults are improving science.
Speakers debate the need for scientific reforms, with one advocating for a return to basics and another suggesting new solutions.
Speaker 3 criticizes paper's examples as strawman arguments, not accurately representing what people mean by replication.
Formalizing scientific methods to improve reproducibility.
Speaker 3 argues that formalism can be used to improve the scientific process, but must be accompanied by critical analysis.
Commons points out limitations of examples used, emphasizing need for more focus on validity of methods.
Speaker 3 highlights the limitations of a recent paper on reproducibility, arguing that it doesn't consider human behavior and its impact on results.
Speaker 1 agrees with the figure in the paper but notes that the examples of reforms are not convincing and need more formalism.
Treating stimuli as random factors in experiments, with implications for type I error rates and generalizability.
Speaker 1 discusses treating stimuli as a random factor in social psychology research.
The paper addresses the underestimation of variation in participants and stimuli, leading to higher type one error rates.
Speaker 3 and Speaker 4 discuss the importance of accounting for correlation structure in stimulus response data.
Speaker 5 suggests that precision and generalizability are trade-offs in sampling correlated stimuli, and that encoding this structure into the statistical procedure is crucial.
Representative design in psychology, sampling, and generalizability.
Speaker highlights 1947 paper on representative design of psychology experiments, still relevant today.
Speaker 1 discusses the concept of representative design, citing Brunswick's work as an example.
Commentators criticize the idea for being too expensive and difficult to implement, particularly in defining the reference class.
Research practices and generalizability of results.
Speakers discuss mixed models in econometrics, including Brunswick's idea of analyzing data from different samples.
Speaker 3 argues that researchers often fail to specify the bounds of their expected generality, leading to unrealistic replication expectations.
Speaker 1 suggests that the issue is not just about time, but also about participant naivety, which should be taken into account when replicating studies.
Incentives and structure in academic publishing.
Speaker 3 expresses frustration with authors' optimistic views on peer review and code sharing.
Speaker 1 mentions Dr. Cohn's blog posts as a potential solution to the problem of over-claiming in social science research.
Speaker 1 argues that incentives can influence how researchers present their findings, and that it's important to be transparent about the limitations of their research.
Speaker 3 agrees that incentives can be a problem, but suggests that it's also important to consider the broader context and potential consequences of research findings.
Speaker 3 suggests authors should be incentivized to be specific without being penalized for not meeting standards.
Low-cost solutions can easily become subverted or performative, negating their value.
Effect size and relevance in research, with a focus on small effects and their importance.
Speaker 3: People use effect size as proxy for relevance, leading to false positives and overlooked smaller effects.
Speaker 3: Discusses how this relates to causal density, highlighting the problem of inflated effects and overlooked smaller effects.
Speakers discuss the importance of small effects in statistical analysis, with one speaker arguing that small effects matter and another speaker questioning the definition of "small."
Speakers also discuss the implications of small effects for scientific research, including the potential for realist and instrumentalist views on science.
Effect size in research, power analysis, and measurement issues.
Speaker 1 argues that even small effects can have meaningful societal harm or benefit if they are frequent or widespread.
Speaker 3 raises the conceptual issue of determining clinical significance or cost-benefit analysis in the absence of foresight.
Speakers discuss power analysis for prediction in research design.
Francesca Gino's large effect sizes in journal publishing may be due to her preference for studies with significant results.
Designing research instruments for predicting treatment effects.
Speaker 3 questions journals' role in enforcing policies, while Speaker 1 discusses design choices for eliciting prediction estimates.
Speakers struggle with instrument validation and access, prioritize slides for class activity.
Speakers discuss feasibility of implementing research methods, including coordination and acceptance by editors and reviewers.
Designing experiments and pre-registering results to improve research quality.
Speakers discuss feasibility of impact on science, business, and design thinking.
Speaker 1 discusses designing documents and requirements with ease.
Pre-registration of research can improve quality and reduce bias, but challenges remain in implementing it effectively.
Operationalizing solutions for social systems.
Speakers discuss challenges in operationalizing ideal experiment in social systems, with small effects identified as most feasible.
Speaker 1 discusses integrating solutions for predictive modeling, valuing snowball effects.

[hyunjimoon]

W13 incentive of individual
note: solution tree in https://miro.com/app/board/uXjVKNmmc6A=/?share_link_id=701713907784

• connect market (environment, improve incommensurability by adding axis) and product (agent, improve incompatible)commensurable system (need), measurement system (method), compatible system (solutaion; paper)
  implicit assumption of high causal density

Meehl: We never seem in the experimental literature to put the results of al the experiments together. Innumerable aspects of the situations are permited to be suppressed. Thus, no way exists of knowing whether the earier studies are in tad commensurale with whatever ones are under present scrutiny, or are in fad contradicory"

• project intro of each: https://otter.ai/u/ofjOqOPB_M3ExrvPOFKFNdPrYiU?utm_source=copy_url

• https://otter.ai/u/OSFDd1giB3JiwTJ5qfylvjGfR3A?utm_source=copy_url

[hyunjimoon]

W14

• Temporal validity as meta-science

• Watts, Duncan. "Response to Turco and Zuckerman’s “Verstehen for Sociology”." American Journal of Sociology 122.4 (2017): 1292-1299.

• Meehl90_appraise_amend_theory.pdf

• Almaatouq24_beyond20.pdf

• Yarkoni, T. et al. Enhancing and accelerating social science via automation: Challenges and opportunities. http://dx.doi.org/10.31235/osf.io/vncwe (2019) doi:10.31235/osf.io/vncwe

• DellaVigna, S., Pope, D. & Vivalt, E. Predict science to improve science. Science vol. 366 428–429 Preprint at https://doi.org/10.1126/science.aaz1704 (2019)

• Protzko, J. et al. High replicability of newly discovered social-behavioural findings is achievable. Nat Hum Behav (2023) doi:10.1038/s41562-023-01749-9

• Bak-Coleman, J. B. & Devezer, B. Causal claims about scientific rigor require rigorous causal evidence. (2023) doi:10.31234/osf.io/5u3kj
  Behaviouris, Nature Human. "Promoting reproduction and replication at scale."
  No, it’s not The Incentives—it’s you

---

i'll prepare my presentation (mock title: prior, bias, time) focusing most on the first four and synthesize them with
geometric deep learning
-- induction prior / bias

-- algorithmic reasoning

• parameter taxonomy 2. Typify parameters in syntax-semantic with colors, units, bounds #7 (comment)
• def(geppetto_playbook)
• temporal validity as meta science
• exact inference (sum product network)

goal of this presentation is expressing need to question

• can we find parameterization / canvas on which to place collaborative hypothesis testing?
• relation between high causal density and temporal validity

---

linearity

The following from R. M. Murray's lec.note reminded me of my inference on Abdullah's recent interest
based on recommended paper on
the-robust-beauty-of-improper-linear-models-in-decision-making.pdf which explains how even improper linear models sometimes outperforms tend to outperform human clinical judgment in predicting numerical criteria from numerical predictors.

• improper linear model is one in which the weights given to the predictor variables are obtained by some non-optimal method, in contrast to a proper linear model where the weights are chosen to optimize the relationship between the prediction and the criterion. e.g. of improper linear models mentioned in the paper:

• equal weight across all predictor variables.

• chosen based on the intuition of the person making prediction, rather than being mathematically optimized.

• randomly chosen

• weights derived by analyzing predictions of human judges, a technique called "bootstrapping", rather than being derived from the actual relationship between predictor and criterion variables.

https://en.wikipedia.org/wiki/Linear%E2%80%93quadratic%E2%80%93Gaussian_control might also be relevant

[hyunjimoon]

TODO

• read The Epistemic and Pragmatic Function of Dichotomous Claims Based on Statistical Hypothesis Tests

randomness, high causal density, path dependence (pens, ) -> low prediction

• absencent the theory, it's unlikely to happen
• prior abscent the theory

low prior in any of the theory ()

either it's a money in a bank or coincidence
but if prior is 50% (nothing is damn coincendence)
directional -> functional,

randomness, (effect of consequence of time; value of those values)change in values of contextual values
reading epistemic lakotos based on statistical testing sense

surrogate (theoretical construct);

15s.03

The conversation centered around the challenges and limitations of utilizing generative AI in research, particularly in the context of language models. Speaker 1 highlighted the reproducibility issue, while Speaker 2 agreed that university settings may provide more access to resources but emphasized the challenges of deploying large LLMs for scale. Speaker 3 stressed the complexity of achieving reproducibility, and Speaker 4 mentioned a recent paper on the impact of generative AI on peer review. The conversation then shifted to the importance of experimentation in understanding market dynamics, with Speaker 6 emphasizing the need to set prior probabilities and use quantified ways to measure surprises to defend theories. Speakers 3 and 2 provided insights from Jerome's work on superstitious learning and conditional sampling, while Unknown Speaker and Speaker 4 explored the relationship between popularity and accuracy in predictions, and the importance of explanations in scientific research.

Transcript

https://otter.ai/u/YnDrKsLmzkw5IvZ_nz-LjPaPmII?view=transcript

Action Items
[ ] Further develop conceptual framework for research project on Notion board and solicit feedback from others
[ ] Develop research proposal applying organizational theories to questions about persistence of problematic research practices
[ ] Read papers on categorizing types of explanations and dimensionalizing parameters for explanations
[ ] Extend existing academic publishing incentive models to account for issues like recording negative findings
Outline
Challenges in using large language models for social science research.
Speaker 1 argues that it's difficult to reproduce LLM results due to changes in the model or API, and limited visibility into the model's inner workings.
Speaker 3 highlights the challenges of achieving reproducibility in LLM research, including version control, hardware convergence, and automatic payment issues.
Speaker 3 discusses challenges in reproducing findings with LLMs, while Speaker 1 and 4 discuss potential solutions and new directions for research.
Speaker 1: "This is a manual arrow. Yeah, I mean, that's the other thing."
Speaker 5: "They offer a way to think about it like a software paper."
AI tools and their limitations.
Speaker 3 laments the rise of research papers without substance, while Speaker 6 questions the difference between LM and empirical methods.
Speaker 3 argues that increasing skill in AI leads to emergent behaviors that are qualitatively different from previous designs.
Speaker 3 discusses controlling tools for generating theories and experiments.
Using generative AI to analyze market dynamics and update predictions in real-time.
Speaker 6 explains how dynamic business canvas helps entrepreneurs navigate high-dimensional space.
Speaker 6 discusses dynamic market analysis and product market fit, using rural and suburban markets as examples.
Speaker 6 updates product and market beliefs based on sales data and predictive modeling, prioritizing product updates over market changes.
Experiment design and validity in the context of behavioral economics.
Speaker 3 discusses challenges in testing experiments, including low stakes and difficulty changing theories.
Speaker 3 suggests using market dynamics to understand how people make decisions, but acknowledges it's harder to change people's minds about rational choice theory.
Speaker 3 discusses the challenges of defending a theory after it has failed an experiment, and the importance of directly addressing the context and population of the experiment.
Speaker 4 mentions Jerry Kirk's work on superstitious learning and conditional sampling, which may provide insights for framing and solving problems in the space.
Organizing and structuring a messy project with various suggestions and ideas.
Speaker 2 seeks to formalize project organization, while Speaker 3 suggests clustering issues by theme.
Speaker 3 suggests using MLM to cluster concepts and create initial candidates for things to be made.
Generating explanations for complex phenomena.
Speaker 5 discusses using logical models to generate explanations for human behavior, with a focus on the popularity of certain connections.
Speaker 5 aims to incentivize accurate predictions by paying based on how much explanation is likely to convince others.
Experimental results show people prefer explanations with clear causes, not just correlations.
Speakers discuss generating explanations for complex events, focusing on accuracy and framing.
Why social sciences persist in using bad practices despite knowing their flaws.
Speaker 7 discusses their project on modeling incentives in academic publishing, including issues with institutional factors and reproducibility.
Speaker 4 explores why social sciences persist in using bad practices despite knowing they're bad, using organizational theory to identify sources of persistence.
Speakers discuss using different frameworks to analyze a topic, with a focus on sociology of science and immunology.
Sociologists use existing social theory to study nation's institutional structures.

15s.03

The conversation revolved around the challenges of predicting outcomes in complex systems, with speakers highlighting the importance of considering the underlying causal structure, time, and aggressiveness in model accuracy. Speakers also discussed the difficulties in measuring economic concepts and intelligence, with a particular focus on the distinctions between theories and measurements in economics. Throughout the conversation, speakers emphasized the need for a systematic workflow and careful consideration of the interconnected factors that impact prediction and measurement in complex systems.

Transcript

https://otter.ai/u/VJm4t4rZsIiQwhew-7K-vdbsa7g?view=transcript

Action Items
[ ] Continue exploring distinctions between substantive theories, auxiliary hypotheses, and measurements/constructs. (Multiple speakers) [Throughout]
[ ] Share relevant papers on distributional shift and domain adaptation in machine learning.
[ ] Discuss examples comparing desire/act verbs to predict/prediction nouns in relation to falsifiability.
[ ] Consider modeling interactions between individual priors/inferences and their propagation through communities.
Outline
Statistical estimation and workflow, with a focus on prior and bias.
Speaker 1 invites significant others to dinner, while Speaker 2 discusses LLM workflow and potential issues with estimation.
Speakers discuss tools for organizing and grouping concepts, with a focus on understanding definitions and relationships.
Speaker discusses how past experiences influence beliefs and decision-making in a dynamic environment.
Causality and time dimensions in a changing context.
Speaker 3 questions the separation of causal identity and time dimension in a paper, while Speaker 1 revises their thinking after reading a manga paper that emphasizes time dependency.
The speaker discusses the concept of "smartphone" as a variable that existed but had zero value before its introduction, and now has a value that interacts with other determinants of civil unrest.
The speaker uses the example of COVID-19 to illustrate how the value of a dimension can change over time, affecting the measurement of a phenomenon, and how this change can be difficult to anticipate beforehand.
Predicting outcomes with uncertain parameters and shifting distributions.
Speaker 1 explains that predicting outcomes is challenging due to the unpredictability of context values.
Speaker 1 and Speaker 3 discuss the limitations of predicting outcomes when the values of parameters are unknown.
Speakers discuss covariant shift, distribution shift, and concept shift in machine learning.
Causality, generalizability, and time in machine learning and social science.
Speaker 1 discusses sabbatical, representation, and model selection in innovation environments.
Speaker discusses distributional shift in machine learning, sharing insights from papers and a book on the topic.
Speaker discusses using loops in causal inference to represent time and avoid mathematical difficulties.
Causality and prediction in social sciences, using smartphone adoption as an example.
Speakers discuss the relationship between time and causality in social science research.
Speakers discuss predicting outcomes with different variables, including smartphones.
Causality, prediction, and randomness in a social science context.
Speaker 1: We can't predict values of x (e.g., family member's death) that impact outcome, but we can specify parameters of x's relationship to outcome.
Speaker 5: Even without knowing x's value, we can articulate and specify its parameter, which is related to outcome.
Speaker 1 argues that causal relationships between variables can change over time due to random trends or irreducible randomness.
Speaker 1 suggests that predicting the outcome of a random event (like winning the lottery) is difficult due to lack of knowledge, but dynamics can still occur even with perfect information.
Predictability in machine learning, funding, and social sciences.
Speaker 1 explains that the outcome of a funding process is uncertain due to unpredictable life outcomes.
Speaker 6 highlights the limitations of using a model to predict the outcome of the funding process.
Speakers discuss challenges in predicting system behavior due to path dependence, density, and unknown unknowns.
Using Bayesian inference to make predictions and defend theories.
Speakers discuss prior probabilities in statistical inference, highlighting their role in decision-making and theory evaluation.
Speaker 1 argues that prior knowledge of a theory can influence the interpretation of data, leading to unexpected findings.
Speaker 3 adds that direction hypothesis testing in social sciences can lead to low prior probabilities for any theory, making it difficult to defend any circumstances.
Speaker 1 discusses predicting the direction of a function, citing examples from physics and experiments.
Speaker 3 questions the ease of estimating prior probabilities, particularly in cases with complex relationships.
Predictive modeling, preregistration, and out-of-sample validation.
Speaker 1 emphasizes the importance of separating the scientist from the observed phenomenon.
Speakers discuss the importance of preregistering predictions to ensure they are truly out-of-sample.
Predicting future events with uncertainty and time components.
Speaker 1 predicts their child will be asleep at different times based on current circumstances.
Key takeaway: Time affects values and context, making predictions difficult.
Speaker 1 explains how pandemic changed higher-order interactions, while Speaker 2 discusses time, inference, and prediction in AI.
Causality in sociology and statistical inference, with a focus on policy parameters and workflow.
Speaker 2 argues that soulstone sacrifice is necessary for AI sentience, while others disagree.
Speaker 2 discusses policy parameters and workflow in Bayesian inference.
Testing and measuring in philosophy of science.
Speakers discuss Bayesian analysis, simulation-based calibration, and probabilistic programming in the context of philosophy of science.
Speaker discusses testing theories through cause-effect switching experiments and the impact of individual and group biases on predictions.
Speaker 2 struggles with determining the hierarchy of T and A in their research.
Unknown Speaker suggests construct validity may be relevant to understanding T and A.
Measuring constructs, surrogacy, and profit prediction.
Speakers discuss the differences between substantive theory and auxiliary hypotheses in psychometrics.
Speakers discuss the limitations of measuring profitability in startups.
Speakers discuss the limitations of measuring intelligence, social perceptiveness, and other constructs.
Speaker 1 argues for incrementalist approach to measuring intelligence, using surrogate metrics like stock prices.

15s.03

Optimizing metrics was a major challenge for the group, with Speaker 1 acknowledging the existence of hidden variables and Speaker 4 emphasizing the need for better measurements. Speaker 2 stressed the importance of clear thinking about root causes, while Speaker 3 inquired about representing uncertainty. Speaker 5 offered an alternative perspective by suggesting that incorporating proxy variables can lead to better operationalization of the underlying phenomena.

Transcript

https://otter.ai/u/ks4z4U4xT-BZzPB-I-TpuFXMLfs?view=transcript

Action Items
[ ] Create a draft dictionary of key concepts with definitions, examples and references.
[ ] Consider dimensions to cluster concepts before including in framework.
[ ] Represent uncertainty and randomness at different levels (individual, institutional etc.).
Outline
Causality, uncertainty, and reductionism in social science.
Speaker 1 discusses using fictional mechanisms to improve prediction and control in science.
Speakers discuss clustering dimensions for validity and utility in natural language processing.
Speaker 1 distinguishes between reducible and irreducible sources of uncertainty in social science.
Adding new columns to a model to better predict civil unrest.
Speaker 1: Family death experience affects data set, leading to biased results.
Speaker 4: Hidden events, such as interpersonal relationships, are critical but underpopulated.
Speaker 4 suggests using columns to predict civil unrest, controlling for variables like information transmission.
Speaker 1 argues that existing columns may not accurately capture the underlying phenomena, and proposes alternative measurements.

[hyunjimoon]

three takeaways from this class

1. classification of problem in three agent level and mapping each with solutions

• problem listed above subsumes @chasfine's thoughts on four limitation of scientific method here

2. applying "one gets to lakatos via popper" with pivoting

is from Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles That Warrant It

• categorized different approaches for "Appraising and Amending Theories"as one table: pivoting model (bayesian, behavior, evolutionary) by system level (idea/belief, individual/firm, industry/ecosystem) in abductive classification of ent_learning/pivoting model #242

• Abdullah's guidance on what literature review I should do to build on scientific hypothesis/theory driven approach for pivoting in entrepreneurship (detailed here, using "Appraising and Amending Theories" would be greatly appreciated.

3. applying integrative design (prediction accuracy under intervention in yet unseen situation) to entrepreneurial experiment

Hofman21_integ_exp_pred_css.pdf

New Perspectives on Experimental Strategy in AOM conference (panelist: Scott Stern)

A central finding in the recent work on business experimentation in entrepreneurship and strategy is that experimentation is, fundamentally, a strategic decision. This work has directed attention to several critical choices actors face when crafting an experimental strategy including the design of the experiment, what to do after running the experiment, and the role of stakeholders, theories, the nature of the decision, and the quality of the idea. In this session, we aim to present N leading scholars who are working in this domain, allowing each to share emerging insights from their particular point of view on the topic; conduct a dialogue with the participants as a collective; and field questions from attendees. The purpose of the presentation will be to showcase areas of active research and illuminate opportunities for future work, in particular the elaboration of new theory and how current theory could be further developed through empirical insight. We anticipate that the first part of our panel discussion will be driven primarily through structured questions that encourage dialogue and constructive engagement between the panelists, orchestrated by the organizers. The overarching goal is to use the session to support the development of participant papers and to promote intellectual interchange on the topic of experimental strategy from the most nascent of entrepreneurs to the most established large firms.